 Daniel Nashed | Daniel Nashed 3 July 2022 21:18:40 If you are looking for a simple way to run HCL Nomad, here is a simple to use Docker container for SafeLinx and Nomad Web. I just merged it to the main branch and wrote some basic documentation. https://opensource.hcltechsw.com/domino-container/safelinx/ Oliver wrote a nice blog post about the container, which gives you a different view --> https://oliverbusse.notesx.net/hp.nsf/blogpost.xsp?documentId=22F6 Thanks Oliver for your great write up! -- Daniel  Daniel Nashed 18 June 2022 18:54:39  ZFS is one of my favorite file-systems. And I posted before about using it as a backup target. The integration is pretty simple with Domino backup, because it is a simple file backup. Now that we have the new VSS Writer for Domino 12.0.2 on Windows, it is time to look into ZFS snapshots. For now this is native Linux for now, because you need OS level calls to create snapshots and more important for mounting the backup to the server for a restore operation. But for native Domino on Linux this is pretty cool! ZFS has many advantages including sending and receiving snapshots in remote locations. This also includes encryption! ZFS has quite a history and with the move to OpenZFS it's now available in Linux distributions. One of the best integration is SUSE Linux Leap 15.3 and higher, where ZFS can be installed out of the box. Here is a must watch video if you are interested in ZFS and there is also a presentation: https://papers.freebsd.org/2020/linux.conf.au/paeps_the_zfs_filesystem/ Now working on the DNUG lab and final presentation preparations, I thought it would time to get this implemented. I will demo it at #DACHNUG 49 conference next week. And depending on feedback, I will make the configuration available via a DXL file. If you are at #DACHNUG 49 conference next week, stop by at the DNUG Lab booth. I have setup another server native on Linux running on ZFS, which runs the OpenZFS snapshot integration. We can look into all details live. I have prepared many different integrations running servers in the lab environment. -- Daniel Additional note: There is some optimization potential, if the Domino Backup application would provide the full restore file including the .DELTA file. I have worked around this in multiple configurations and I think it would make sense the default restore file would already contain the .DELTA extension.  Daniel Nashed 16 June 2022 10:04:37  openSUSE Leap is one of the platforms I really care about. Not just because they are German and it was the first distribution I used very long time ago, when software was distributed on floppy disks. They do a lot of things right and I have a mix of servers. I have not used the on-line update function. And I would wait for that for a while. But they already released the Docker base image and I had to install a new lab machine on my notebook for travel anyway. Here is where you get the full ISO. And also the Network Image (173.0 MiB) will work.
In earlier versions, there have been issues using the smaller images. You had to configure the repositories manually. This is now working very well and it the best setup wizard on Linux I know of. https://get.opensuse.org/leap/15.4/ Here is the current kernel version as of last night's update: Linux localhost 5.14.21-150400.22-default #1 SMP PREEMPT_DYNAMIC Wed May 11 06:57:18 UTC 2022 (49db222) x86_64 x86_64 x86_64 GNU/Linux OpenSSL 3.0.1 support SUSE added OpenSS 3.0.1. But in contrast to Redhat who moved to Openssl 3.0.1 with RHEL/CentOS Stream 9.0 completely, SUSE offers it in a separate package "openssl-3". This might be helpful for some software. Also OpenSSL 1.1.1 has been updated and I wonder why SUSE Leap 15.3 is not getting this update ( my Leap 15.3 server is still at OpenSSL 1.1.1d 10 Sep 2019)- Here are the current versions as of today. openssl-3 openssl-3 version OpenSSL 3.0.1 14 Dec 2021 (Library: OpenSSL 3.0.1 14 Dec 2021) openssl version
OpenSSL 1.1.1l 24 Aug 2021 SUSE release 150400.5.14 ZFS Support I have not managed to get ZFS installed last night. And I probably wait a bit until the official repo is listed. Building Docker images An easy start to look into it, would be a Docker container. I have updated our DNUG Lab environment last night via ./build.sh domino 12.0.2 -capi -verse -from=opensuse/leap:15.4 And I added new tags leap15.4 and leap15.3 in the develop branch to make it easier to select. -from=leap continues to point to whatever SUSE decides to be latest. Daniel Nashed 14 June 2022 23:22:08 There are is always one more thing to add ... I just introduced a Docker based Minio S3 server on the SUSE Leap server. The underlying file system is ZFS with deduplication enabled. Domino Backup S3 with ZFS This is an example for my backup and storage optimization session. I just took the S3 backup integration and configured a Bucket for Domino Backup. After 3 full backups of the server the storage looks like this: zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT zfs-pool 19.5G 195M 19.3G - - 0% 0% 2.80x ONLINE - DAOS T2 and DAOS shared encryption Now that I have S3 storage available, a DAOS T2 repository was easy to add. DAOS T2 is always encrypts NLOs when pushing them to S3. Therefore I added AES 256 shared key encryption to the server. So some reason the bucket stats don't update. But all the data is there. Backup and DAOS T2 along in the same ZFS pool ..  Daniel Nashed 14 June 2022 11:04:03
Reporting potential security issues is very important for software quality.
Every software has bugs -- As we have seen even in the Linux world in the couple of last month. Yes and there are even Linux kernel security bugs.
Reporting security issues in the open source world is a separate topic.
But how do you report security issues or potential security issues to a commercial vendor?
Customer Support
If you are a customer with support, you should always open a support ticket.
Those support teams know best about the actual problem and how to flag tickets for fast security reviews in the right team.
How do you report if you don't have support?
First of all, if security is important to you, you should have maintenance for all your software products to update to the latest versions to get security fixes!
But if you run into an issue and have no support, there are usually special accounts at software companies to report bugs in a safe way.
14.06.2022: Update from Martin (huge thanks) There is a blog post describing how to open tickets. There is a separate category for security and reporting security issues.
And there is even a guest form, in case you have no support account available. https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0010164
Security TXT
There is an initiative which some companies are following --> https://securitytxt.org/
This "standard" allows each Domain to provide information about their security incident reporting process.
Take a look for example at --> https://www.ibm.com/.well-known/security.txt
Product Security Incident Response Team
There is an other standard term you should know about -- "PSIRT".
Many companies have special teams and accounts to report security issues to.
In case you are having issues with your support account or your maintenance expired, this would be probably the best way to report a security incident.
For example HCL Software has the following web page with all the details about security incident reporting:
https://www.hcltechsw.com/resources/psirt
Why is reporting security incidents in private is important?
First all, a security concern needs to be evaluated by a vendor.
If you are not a professional researcher, it can be quite difficult to get it right and there can be false positives, due do a misconfiguration or misinterpretation of logs etc.
If you report in public -- like in a blog -- this can have negative effects for the product you care about and want to help to improve.
Blog posts
So even it might not be a bug, others known less then yourself might get the wrong impression.
Also if it turns out to not be a bug, it is difficult to correct the first impression someone had about this issue.
Adding another blog post with an update on an existing post where you raised the concern, would be even less desirable.
Because readers of your blog might only read your initial post -- not the updated information in a follow-up post.
Getting the fame for finding a bug
Money should not be the main incentive to report a bug.
But getting proper credit for a bug you found is something that even ethical hackers are striving for.
If you want the credit, you let the software company name you in their CVE instead of being the first one to blog about it.
A blog post should be the last step in the process after the problem has been confirmed, the bug is fixed and the fix is available.
Unless there s a simple work-around, there is no point in making it public early.
I thought this would be common knowledge. But I had some discussions in the last week, which really surprised and disappointed me.
This lead to this blog post and I hope this helps others if not the one I tried to discuss with in private.
-- Daniel Daniel Nashed 12 June 2022 10:53:46 Wouldn't it be cool to have a SafeLinx Docker image with Nomad Web included with auto configuration? Maybe having a docker-compose.yml with just some basic parameters to get SafeLinx and Nomad up and running?
Docker container configuration: A configuration could look like this: CONTAINER_HOSTNAME=nomad.acme.com DOMINO_ORG=acme LDAP_HOST=ldap.acme.com And just running "docker-compose up" could get SafeLinx and Nomad Web up and running ..
Certificate for the SafeLinux server But what about getting a certificate for your server?
If your server is behind a load balancer, you can get away with automatic created certificates just for the container. So it could include a small CA creating EDCDA keys for you.
CertMgr auto certificate updates If SafeLinx isn't behind a reverse proxy updating official certificates and keys could be just be dropping PEM files into a mount and let the container do all the work for you.. Maybe it would be a good idea to teach SafeLinx to auto update certificates from a CertMgr server directly if the existing private key matches the new certificate retrieved via HTTPS SNI? So wishful thinking would be just to just specify like CERTMGR_HOST=certmgr.acme.com to let the container update certificates automagically? Hmmmmm .... I really wanted a Nomad Web configuration for our new DNUG Lab environment, we want to showcase at DNUG. And configuring it via the old fashioned remote admin GUI wasn't an option for me... OK as you know once I have an idea and start building, I am like in a coding tunnel until it is all done .. So at #DACHNUG 49 I will demo the new HCL SafeLinx Community image in combination with Domino CertMgr functionality in my Domino 12.0.x security session. There isn't any documentation yet and I am working on some fit & finish. But it does already exactly what I described above and available in the develop branch of the Domino community image. Building the image works very similar to the Domino, Traveler and Volt image builds. And it builds in less then 2 minutes. The software download information is included in the software.txt like for any other image. ./build.sh safelinx +nomadweb A docker-compose.yml with .env setup file example file is also included. docker-compose up Creating network "safelinx_safelinx_net" with driver "bridge" Creating volume "safelinx_data" with default driver Creating safelinx ... done Attaching to safelinx safelinx | safelinx | HCL SafeLinx Community Server safelinx | safelinx | Configuration safelinx | ------------------------------------------------------------ safelinx | DOMINO_ORG : [acme] safelinx | NOMAD_HOST : [nomad.acme.com] safelinx | CONFIG_BASE : [o=local] safelinx | CERTMGR_HOST : [] safelinx | (CHECK_INTERVAL) : [30] safelinx | TRUSTED_ROOTS : [/opt/hcl/SafeLinx/datastore/trusted_roots.pem] safelinx | LDAP_HOST : [ldap.acme.com] safelinx | LDAP_PORT : [389] safelinx | LDAP_SSL : [0] safelinx | LDAP_USER : [] safelinx | LDAP_BASEDN : [acme] safelinx | ------------------------------------------------------------ safelinx | safelinx | safelinx | Configuring SafeLinx safelinx | safelinx | NomadServer Available safelinx | LDAP-Server Available safelinx | LDAP-Authentication Available safelinx | nomad-web-proxy0 Available safelinx | safelinx | Generated PEM import password: x3+SfroADK48vI2SHAzinLLHxAohqh/cMuoyJOX0WS4= safelinx | safelinx | Write down the password, if you plan to import password protected PEM files (e.g. from HCL Domino CertMgr) safelinx | safelinx | safelinx | Waiting for mounted cert ... safelinx | safelinx | Startup: Timeout waiting for initial certificate safelinx | safelinx | Creating new certificate for nomad.acme.com safelinx | safelinx | Signature ok safelinx | subject=O = acme, CN = nomad.acme.com safelinx | Getting CA Private Key safelinx | safelinx | Export Password: pZtC9IJh1h8RyMrSCFp23igSZtyo6msOLqwtkMC6phw= safelinx | safelinx | safelinx | safelinx | HCL SafeLinx Version 1.3.0.0 (5724-R20) safelinx | safelinx | safelinx | safelinx | Certificate safelinx | ----------- safelinx | safelinx | SAN : DNS:nomad.acme.com safelinx | Subject : O = acme, CN = nomad.acme.com safelinx | Issuer : O = acme, CN = SafeLinxCA safelinx | Expiration : Jun 9 08:14:06 2032 GMT safelinx | Fingerprint : C0:AB:7F:F5:3C:56:00:9E:EA:0C:6B:54:CA:68:44:13:3D:7B:3E:24 safelinx | Serial : 1FBAA17407B2CEFB2DA48C413797934983A2D044 safelinx | safelinx | Daniel Nashed 11 June 2022 09:56:13 There is always someone who might already have done, what you are looking for -- Specially on Linux
I found the following genius line via Google when I was looking for a unmatched quote in a bash script This like gives you the line numbers where you have unmatched quotes: tr -cd "\"\n" < install_dir_safelinx/entrypoint.sh | awk 'length%2==1 {print NR, $0}' This really made my day!! Very very cool!! -- Daniel Daniel Nashed 10 June 2022 20:52:42
This will be the most complete Domino lab environment you have seen prepared for a conference.
I took mot of the new Domino 12.0.x features -- including 12.0.2 EA1 into three servers.
Come and see Domino 12.0.2 live in action, get your own demo account for the conference, ask questions.
You will see some special configurations described in the last couple of month on my blog.
And I just finished a first version of a SafeLinx Nomad Web container image, which we will use and showcase at the conference.
I just got the OK from the board, that we get our own mini booth for the lab.
So beside the sessions I plan to spend a lot of time at the lab booth.
If anything is missing on the list you want to see about Domino 12.0.x.. It's still time to add it.. Let me know ..
We can walk thru all the features and specially my favorite topics Domino Backup, CertMgr and the new ICAP Antivius integration.
Have a great weekend and I hope to see many of you at #DACHNUG 49 soon.
-- Daniel
Domino Lab Setup
-- 3 Servers -- - Domino 12.0.2 EA1 first/additional server with OneTouch setup
- Servers hosted @ Hetzner --> Installation via Hetzner Cloud and DNS REST API using a Notes application demoed at Domino 12.0 launch event
- Linux servers run the current HCL Domino community image
- using dominoctl for containers -- Container start script for Docker and Podman (with systemd service)
- OneTouch templating and automation on Linux
- Access to Windows server via SSH tunnel with Ed25519 key for RDP access
linus.lab.dnug.eu - CentOS Stream 9 with Podman
- OneTouch templating and automation on Linux
- Traveler 12.0.2 on Podman
- c-icap server with ClamAV integration providing ICAP for mailscan behind NGINX to offload TLS
- Domino and Fail2Ban Integration
- SpamGeek with SPF
ray.lab.dnug.eu - SUSE Leap 15.3 with Docker
- CAPI 12.0.1 development environment in Domino container
- Verse 2.2.0a in Domino container image
- SafeLinx 1.3 with Nomad Web 1.0.3 in a separate Docker container
- KeyCloak server on Docker
- Minio server on Docker
- NGINX for port 443 to dispatch to different applications via SNI
bill.lab.dnug.eu - Windows 2022
- Veeam Backup & Replication 11
-- Main features configured --
Security - CertMgr with HTTP-01 & DNS-01 (free provider: deSEC e.V.)
- ECDSA keys
- TOTP
- SAML with KeyCloak
- Domino CA with Lotus Script
- Internet lockout with IP based blocking
- ID Vault
Message Security - MailScan with c-ICAP and ClamAV
- DKIM with RSA and Ed25519
- DKIM and SPF configured in DNS
TCO: Backup and storage optimization etc - VSS Writer Backup with Veeam
- Linux Domino Backup to ZFS
- Domino 3 way cluster with cluster repair
- DAOS
- NIFNSF
- Translog
- DBMT best practices configuration
- DDM setup
Daniel Nashed 6 June 2022 06:44:28 CentOS Stream 9 is an awesome Linux distribution.
But it still has no good ZFS support. And there are also other benefits using SUSE.
Sadly Hetzner does not allow to create virtual servers using SUSE Leap.
But they added the DVD ISO image for SUSE Leap 15.3.
You can just boot from the ISO and install your server on your own...
Works like a charm.. I just have to redo it, because I missed up ZFS and btrfs snapshots ..
Hetzner does some really cool things. They use DHCP and usually Linux and also Windows comes up with the right IP address configured.
Even if you attach a private network, the network is automatically detected.
The other DNUG Lab server is using Podman. SUSE Leap comes with Docker.
-- Daniel
 Daniel Nashed 3 June 2022 18:51:05
This is going to be awesome. This is my new weekend project and it will be a full featured lab @DACHNUG conference this month.
I already setup a cluster running Domino 12.0.2 EAP1 with most of the new features. And it is running in production style with best practices.
We will have a full lab environment to show & tell and for hands-on during DACHNUG conference.
And it will continue to be a lab environment the different DNUG focus groups will work with.
We are planning to add SafeLinx and Nomad web next week.
And I just created a registration database leveraging the Domino CA process to create users in ID Vault.
All other new features like TOTP and DKIM outbound are added step by step over the couple of next days.
It is built all on best practices also in the back-end with a SSH tunnel from the Linux (CentOS Stream 9) machine to the Windows 2022 server securing the RDP port.
This environment will have all cool new features.
See previous post for more details..
Have a great weekend!
-- Daniel
 | |
![[IBM Lotus Domino]](../lotus-domino.gif){kind=small}
![[Domino on Linux]](../domino-linux.gif){kind=small}
![[Nash!Com]](../nashcom.gif){kind=small}
![[Daniel Nashed]](../daniel-nsh.gif){kind=small}