Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...


Daniel Nashed


Domino ZFS Snapshot Backup

Daniel Nashed  18 June 2022 18:54:39

Image:Domino ZFS Snapshot Backup

ZFS is one of my favorite file-systems. And I posted before about using it as a backup target.
The integration is pretty simple with Domino backup, because it is a simple file backup.

Now that we have the new VSS Writer for Domino 12.0.2 on Windows, it is time to look into ZFS snapshots.

For now this is native Linux for now, because you need OS level calls to create snapshots and more important for mounting the backup to the server for a restore operation.
But for native Domino on Linux this is pretty cool! ZFS has many advantages including sending and receiving snapshots in remote locations. This also includes encryption!

ZFS has quite a history and with the move to OpenZFS it's now available in Linux distributions.
One of the best integration is SUSE Linux Leap 15.3 and higher, where ZFS can be installed out of the box.

Here is a must watch video if you are interested in ZFS and there is also a presentation:

Now working on the DNUG lab and final presentation preparations, I thought it would time to get this implemented.
I will demo it at #DACHNUG 49 conference next week.
And depending on feedback, I will make the configuration available via a DXL file.

If you are at #DACHNUG 49 conference next week, stop by at the DNUG Lab booth.

I have setup another server native on Linux running on ZFS, which runs the OpenZFS snapshot integration.

We can look into all details live. I have prepared many different integrations running servers in the lab environment.

-- Daniel

Additional note:

There is some optimization potential, if the Domino Backup application would provide the full restore file including the .DELTA file.
I have worked around this in multiple configurations and I think it would make sense the default restore file would already contain the .DELTA extension.

Image:Domino ZFS Snapshot Backup

openSUSE Leap 15.4 released -- works well with Domino and Docker images

Daniel Nashed  16 June 2022 10:04:37

Image:openSUSE Leap 15.4 released -- works well with Domino and Docker images

openSUSE Leap is one of the platforms I really care about. Not just because they are German and it was the first distribution I used very long time ago, when software was distributed on floppy disks.

They do a lot of things right and I have a mix of servers.

I have not used the on-line update function. And I would wait for that for a while.

But they already released the Docker base image and I had to install a new lab machine on my notebook for travel anyway.

Here is where you get the full ISO. And also the Network Image (173.0 MiB) will work.
In earlier versions, there have been issues using the smaller images. You had to configure the repositories manually.

This is now working very well and it the best setup wizard on Linux I know of.

Here is the current kernel version as of last night's update:

Linux localhost 5.14.21-150400.22-default #1 SMP PREEMPT_DYNAMIC Wed May 11 06:57:18 UTC 2022 (49db222) x86_64 x86_64 x86_64 GNU/Linux

OpenSSL 3.0.1 support

SUSE added OpenSS 3.0.1. But in contrast to Redhat who moved to Openssl 3.0.1 with RHEL/CentOS Stream 9.0 completely, SUSE offers it in a separate package "openssl-3".
This might be helpful for some software. Also OpenSSL 1.1.1 has been updated and I wonder why SUSE Leap 15.3 is not getting this update  ( my Leap 15.3 server is still at OpenSSL 1.1.1d  10 Sep 2019)-

Here are the current versions as of today.


openssl-3 version
OpenSSL 3.0.1 14 Dec 2021 (Library: OpenSSL 3.0.1 14 Dec 2021)

openssl version

OpenSSL 1.1.1l  24 Aug 2021 SUSE release 150400.5.14

ZFS Support

I have not managed to get ZFS installed last night. And I probably wait a bit until the official repo is listed.

Building Docker images

An easy start to look into it, would be a Docker container.

I have updated our DNUG Lab environment last night via

./ domino 12.0.2 -capi -verse -from=opensuse/leap:15.4

And I added new tags leap15.4 and leap15.3 in the develop branch to make it easier to select. -from=leap continues to point to whatever SUSE decides to be latest.

New arrivals in the DNUG LAB for next week: Minio for DAOS T2 and Domino Backup

Daniel Nashed  14 June 2022 23:22:08

There are is always one more thing to add ...
I just introduced a Docker based Minio S3 server on the SUSE Leap server.

The underlying file system is ZFS with deduplication enabled.

Domino Backup S3 with ZFS

This is an example for my backup and storage optimization session.
I just took the S3 backup integration and configured a Bucket for Domino Backup.

After 3 full backups of the server the storage looks like this:

zpool list
zfs-pool  19.5G   195M  19.3G        -         -     0%     0%  2.80x    ONLINE  -

DAOS T2 and DAOS shared encryption  

Now that I have S3 storage available, a DAOS T2 repository was easy to add.

DAOS T2 is always encrypts NLOs when pushing them to S3.
Therefore I added AES 256 shared key encryption to the server.  

So some reason the bucket stats don't update.

But all the data is there. Backup and DAOS T2 along in the same ZFS pool ..

Image:New arrivals in the DNUG LAB for next week: Minio for DAOS T2 and Domino Backup

How to report security related problems to a vendor?

Daniel Nashed  14 June 2022 11:04:03

Reporting potential security issues is very important for software quality.
Every software has bugs -- As we have seen even in the Linux world in the couple of last month. Yes and there are even Linux kernel security bugs.

Reporting security issues in the open source world is a separate topic.
But how do you report security issues or potential security issues to a commercial vendor?

Customer Support

If you
are a customer with support, you should always open a support ticket.
Those support teams know best about the actual problem and how to flag tickets for fast security reviews in the right team.

How do you report if you don't have support?

First of all, if security is important to you, you should have maintenance for all your software products to update to the latest versions to get security fixes!
But if you run into an issue and have no support, there are usually special accounts at software companies to report bugs in a safe way.

14.06.2022: Update from Martin (huge thanks)

There is a blog post describing how to open tickets. There is a separate category for security and reporting security issues.
And there is even a guest form, in case you have no support account available.

Security TXT

There is an initiative which some companies are following -->
This "standard" allows each Domain to provide information about their security incident reporting process.

Take a look for example at  -->

roduct Security Incident Response Team

There is an other standard term you should know about -- "PSIRT".

Many companies have special teams and accounts to report security issues to.

In case you are having issues with your support account or your maintenance expired, this would be probably the best way to report a security incident.
For example HCL Software has the following web page with all the details about security incident reporting:

Why is reporting security incidents in private is important?

First all, a security concern needs to be evaluated by a vendor.

If you are not a professional researcher, it can be quite difficult to get it right and there can be false positives, due do a misconfiguration or misinterpretation of logs etc.

If you report in public -- like in a blog --  this can have negative effects for the product you care about and want to help to improve.

Blog posts

So even it might not be a bug, others known less then yourself might get the wrong impression.

Also if it turns out to not be a bug, it is difficult to correct the first impression someone had about this issue.

Adding another blog post with an update on an existing post where you raised the concern, would be even less desirable.

Because readers of your blog might only read your initial post -- not the updated information in a follow-up post.

Getting the fame for finding a bug

Money should not be the main incentive to report a bug.
But getting proper credit for a bug you found is something that even ethical hackers are striving for.

If you want the credit, you let the software company name you in their CVE instead of being the first one to blog about it.

A blog post should be the last step in the process after the problem has been confirmed, the bug is fixed and the fix is available.

Unless there s a simple work-around, there is no point in making it public early.

I thought this would be common knowledge. But I had some discussions in the last week, which really surprised and disappointed me.

This lead to this blog post and I hope this helps others if not the one I tried to discuss with in private.

-- Daniel

SafeLinx Nomad Server Community project?

Daniel Nashed  12 June 2022 10:53:46

Wouldn't it be cool to have a SafeLinx Docker image with Nomad Web included with auto configuration?
Maybe having a docker-compose.yml with just some basic parameters to get SafeLinx and Nomad up and running?

Docker container configuration:

A configuration could look like this:

And just running "docker-compose up" could get SafeLinx and Nomad Web up and running ..

Certificate for the SafeLinux server

But what about getting a certificate for your server?

If your server is behind a load balancer, you can get away with automatic created certificates just for the container.
So it could include a small CA creating EDCDA keys for you.

CertMgr auto certificate updates

If SafeLinx isn't behind a reverse proxy updating official certificates and keys could be just be dropping PEM files into a mount and let the container do all the work for you..

Maybe it would be a good idea to teach SafeLinx to auto update certificates from a CertMgr server directly if the existing private key matches the new certificate retrieved via HTTPS SNI?
So wishful thinking would be just to just specify like to let the container update certificates automagically?

Hmmmmm ....

I really wanted a Nomad Web configuration for our new DNUG Lab environment, we want to showcase at DNUG.
And configuring it via the old fashioned remote admin GUI wasn't an option for me...

OK as you know once I have an idea and start building, I am like in a coding tunnel until it is all done ..
So at #DACHNUG 49 I will demo the new HCL SafeLinx Community image in combination with Domino CertMgr functionality in my Domino 12.0.x security session.

There isn't any documentation yet and I am working on some fit & finish. But it does already exactly what I described above and available in the develop branch of the Domino community image.

Building the image works very similar to the Domino, Traveler and Volt image builds.
And it builds in less then 2 minutes. The software download information is included in the software.txt like for any other image.

./ safelinx +nomadweb

A docker-compose.yml with .env setup file example file is also included.

docker-compose up

Creating network "safelinx_safelinx_net" with driver "bridge"
Creating volume "safelinx_data" with default driver
Creating safelinx ... done
Attaching to safelinx
safelinx    |
safelinx    | HCL SafeLinx Community Server
safelinx    |
safelinx    | Configuration
safelinx    | ------------------------------------------------------------
safelinx    | DOMINO_ORG       : [acme]
safelinx    | NOMAD_HOST       : []
safelinx    | CONFIG_BASE      : [o=local]
safelinx    | CERTMGR_HOST     : []
safelinx    | (CHECK_INTERVAL) : [30]
safelinx    | TRUSTED_ROOTS    : [/opt/hcl/SafeLinx/datastore/trusted_roots.pem]
safelinx    | LDAP_HOST        : []
safelinx    | LDAP_PORT        : [389]
safelinx    | LDAP_SSL         : [0]
safelinx    | LDAP_USER        : []
safelinx    | LDAP_BASEDN      : [acme]
safelinx    | ------------------------------------------------------------
safelinx    |
safelinx    |
safelinx    | Configuring SafeLinx
safelinx    |
safelinx    | NomadServer Available
safelinx    | LDAP-Server Available
safelinx    | LDAP-Authentication Available
safelinx    | nomad-web-proxy0 Available
safelinx    |
safelinx    | Generated PEM import password: x3+SfroADK48vI2SHAzinLLHxAohqh/cMuoyJOX0WS4=
safelinx    |
safelinx    | Write down the password, if you plan to import password protected PEM files (e.g. from HCL Domino CertMgr)
safelinx    |
safelinx    |
safelinx    | Waiting for mounted cert ...
safelinx    |
safelinx    | Startup: Timeout waiting for initial certificate
safelinx    |
safelinx    | Creating new certificate for
safelinx    |
safelinx    | Signature ok
safelinx    | subject=O = acme, CN =
safelinx    | Getting CA Private Key
safelinx    |
safelinx    | Export Password: pZtC9IJh1h8RyMrSCFp23igSZtyo6msOLqwtkMC6phw=
safelinx    |
safelinx    |
safelinx    |
safelinx    | HCL SafeLinx Version (5724-R20)
safelinx    |
safelinx    |
safelinx    |
safelinx    | Certificate
safelinx    | -----------
safelinx    |
safelinx    | SAN         :
safelinx    | Subject     : O = acme, CN =
safelinx    | Issuer      : O = acme, CN = SafeLinxCA
safelinx    | Expiration  : Jun  9 08:14:06 2032 GMT
safelinx    | Fingerprint : C0:AB:7F:F5:3C:56:00:9E:EA:0C:6B:54:CA:68:44:13:3D:7B:3E:24
safelinx    | Serial      : 1FBAA17407B2CEFB2DA48C413797934983A2D044
safelinx    |
safelinx    |

Bash command of the week: Find unmatched quotes in a shell script - very very helpful

Daniel Nashed  11 June 2022 09:56:13

There is always someone who might already have done, what you are looking for -- Specially on Linux
I found the following genius line via Google when I was looking for a unmatched quote in a bash script

This like gives you the line numbers where you have unmatched quotes:

tr -cd "\"\n" < install_dir_safelinx/ | awk 'length%2==1 {print NR, $0}'

This really made my day!! Very very cool!!
-- Daniel

#DACHNUG 49 conference lab mission completed

Daniel Nashed  10 June 2022 20:52:42

Image:#DACHNUG 49 conference lab mission completed

This will be the most complete Domino lab environment you have seen prepared for a conference.

I took mot of the new Domino 12.0.x features -- including 12.0.2 EA1 into three servers.

Come and see Domino 12.0.2 live in action, get your own demo account for the conference, ask questions.

You will see some special configurations described in the last couple of month on my blog.
And I just finished a first version of a SafeLinx Nomad Web container image, which we will use and showcase at the conference.

I just got the OK from the board, that we get our own mini booth for the lab.

So beside the sessions I plan to spend a lot of time at the lab booth.

If anything is missing on the list you want to see about Domino 12.0.x.. It's still time to add it.. Let me know ..

We can walk thru all the features and specially my favorite topics Domino Backup, CertMgr and the new ICAP Antivius integration.

Have a great weekend and I hope to see many of you at #DACHNUG 49 soon.

-- Daniel

Domino Lab Setup

-- 3 Servers --
  • Domino 12.0.2 EA1 first/additional server with OneTouch setup
  • Servers hosted @ Hetzner --> Installation via Hetzner Cloud and DNS REST API using a Notes application demoed at Domino 12.0 launch event
  • Linux servers run the current HCL Domino community image
  • using dominoctl for containers -- Container start script for Docker and Podman (with systemd service)
  • OneTouch templating and automation on Linux
  • Access to Windows server via SSH tunnel with Ed25519 key for RDP access
  • CentOS Stream 9 with Podman
  • OneTouch templating and automation on Linux
  • Traveler 12.0.2 on Podman
  • c-icap server with ClamAV integration providing ICAP for mailscan behind NGINX to offload TLS
  • Domino and Fail2Ban Integration
  • SpamGeek with SPF
  • SUSE Leap 15.3 with Docker
  • CAPI 12.0.1 development environment in Domino container
  • Verse 2.2.0a in Domino container image
  • SafeLinx 1.3 with Nomad Web 1.0.3 in a separate Docker container
  • KeyCloak server on Docker
  • Minio server on Docker
  • NGINX for port 443 to dispatch to different applications via SNI
  • Windows 2022
  • Veeam Backup & Replication 11

-- Main features configured --

  • CertMgr with HTTP-01 & DNS-01 (free provider: deSEC e.V.)
  • ECDSA keys
  • TOTP
  • SAML with KeyCloak
  • Domino CA with Lotus Script
  • Internet lockout with IP based blocking
  • ID Vault

Message Security
  • MailScan with c-ICAP and ClamAV
  • DKIM with RSA and Ed25519
  • DKIM and SPF configured in DNS

TCO: Backup and storage optimization etc
  • VSS Writer Backup with Veeam
  • Linux Domino Backup to ZFS
  • Domino 3 way cluster with cluster repair
  • DAOS
  • Translog
  • DBMT best practices configuration
  • DDM setup

SUSE Leap @ Hetzner

Daniel Nashed  6 June 2022 06:44:28

CentOS Stream 9 is an awesome Linux distribution.
But it still has no good ZFS support. And there are also other benefits using SUSE.

Sadly Hetzner does not allow to create virtual servers using SUSE Leap.

But they added the DVD ISO image for SUSE Leap 15.3.

You can just boot from the ISO and install your server on your own...

Works like a charm.. I just have to redo it, because I missed up ZFS and btrfs snapshots ..

Hetzner does some really cool things. They use DHCP and usually Linux and also Windows comes up with the right IP address configured.

Even if you attach a private network, the network is automatically detected.

The other DNUG Lab server is using Podman. SUSE Leap comes with Docker.

-- Daniel

Image:SUSE Leap @ Hetzner

DNUG Lab @DACHNUG conference running on Domino 12.0.2

Daniel Nashed  3 June 2022 18:51:05

Image:DNUG Lab @DACHNUG conference running on Domino 12.0.2

This is going to be awesome. This is my new weekend project and it will be a full featured lab @DACHNUG conference this month.

I already setup a cluster running Domino 12.0.2 EAP1 with most of the new features.  And it is running in production style with best practices.

We will have a full lab environment to show & tell and for hands-on during DACHNUG conference.

And it will continue to be a lab environment the different DNUG focus groups will work with.

We are planning to add SafeLinx and Nomad web next week.

And I just created a registration database leveraging the Domino CA process to create users in ID Vault.

All other new features like TOTP and DKIM outbound are added step by step over the couple of next days.

It is built all on best practices also in the back-end with a SSH tunnel from the Linux (CentOS Stream 9) machine to the Windows 2022 server securing the RDP port.

This environment will have all cool new features.

See previous post for more details..

Have a great weekend!

-- Daniel

Image:DNUG Lab @DACHNUG conference running on Domino 12.0.2

DACHNUG - Domino 12.0.2 Lab @DNUG Conference next month

Daniel Nashed  31 May 2022 19:36:00

Image:DACHNUG - Domino 12.0.2 Lab @DNUG Conference next monthImage:DACHNUG - Domino 12.0.2 Lab @DNUG Conference next month

The DNUG focus groups for Domino, Administration and Communications have been very active in the last two years.

Even we had no conference we had full day events and also remote handls-on workshops.

The communications team did a hands on ST workshop and created their own environment to be used by DNUG and also a lab environment with k3s.
The Domino group had a couple of workshops as well. The topics I covered in full day hands-on workshops have been Domino on Docker & K8s, Domino Certificate Manager and Domino Backup.
Each of them have been a lot of work, but also a lot of fun. And we really want to leverage what we did for all members.

Hetzner Cloud based lab environment

We are using the Hetzner cloud -- a great way for us to host on-line workshops.

The Notes database I built around the Hetzner Cloud and DNS API to build lab environments can be used by all members for their workshops.
The database we use has been show-cased by HCL for the Domino 12.0 launch event and allows to to setup any number of servers in minutes.
Since then we used it multiple times in different DNUG focus groups and I am offering the database for free for any user group for hands-on workshops.

DNUG Domino 12.0.2 EAP full featured Lab for DNUG conference

For the upcoming conference next month we came up with a new idea.
We always wanted a full lab environment for our members to look into for new functionality and best practices.
Now that Domino 12.0.2 Early Access 1 shipped, I created a new server cluster with a Linux and Windows machine as the base.

All my presentations will be prepared using this new cloud. Both servers have been setup with OneTouch setup.

The Linux box runs on CentOS Stream 9 on Podman in a container leveraging the HCL Domino Community container image project along with my start scripts.
The server have been deployed on Hetzner with my lab setup database. This includes generation of template JSON files for One-Touch setup.

We are planning to show-case everything that Domino 12.0 - 12.0.2 has shipped on the server side.
But also existing features like ID-Vault used for SAML integration and new features like TOTP.

You can expect a best practices implementation also enabling new features like CertMgr, Domino backup VSS Writer integration with Veeam. S3 Mino backup, DKIM, and much more.
All the security features will be in my presentations at DNUG. And I will use this implementation as a reference as well.

Nomad and SafeLinx planned

Beside that the admin team will help me to setup SafeLinx for Nomad web. And we already have Antivirus with ICAP enabled.

ICAP Antivirus Domino 12.0.2 Lab

Ulrich Krause ( just finished a very nice lab environment for ICAP using ClamAV.
And I took his setup and put it behind a NGINX for proper TLS termination with a ECDSA wild card certificate from Let's Encrypt, requested by CertMgr...

I could continue to list all the new features I have already implemented and which we will add before the conference.
This includes also Linux best practices and SSH configuration with ed25519 keys and fail2ban integration into Domino and a SafeLinx VPN to protect the Windows server.

You will recognize many of the deployment patterns and ideas I posted over the last two years.
And I plan to bring all to life in this demo environment, which we plan to continue to support and have available for all members as a test lab in future.

For the upcoming conference, this will be a great full featured demo place to show the features in Domino 12.0.x :-)
If we find some spot in the agenda we could also present how this is being build and what others can reuse for their own lab environments.

I hope to finally meet many of you in person again!
And I am really looking forward to the first German speaking conference since a long time!

-- Daniel

Image:DACHNUG - Domino 12.0.2 Lab @DNUG Conference next month

Image:DACHNUG - Domino 12.0.2 Lab @DNUG Conference next month



    • [IBM Lotus Domino]
    • [Domino on Linux]
    • [Nash!Com]
    • [Daniel Nashed]