Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

DNUG Deep Dive: HCL Domino auf Docker & Kubernetes – Online Hands-On Workshop

Daniel Nashed  3 December 2020 16:10:33
This workshop will be in German, but we can do a repeat anytime -- also for customer specific work-shops on-line.
I am using my Hetzner Lab automation I just wrote (see one of the last blog posts) using a Notes database with REST APIs, Lotus Script  HTTP request class and JSON parser ...

-- Daniel

Donnerstag, 14.01.2020  - 09 – 17 Uhr

Domino auf Docker und anderen Container-Plattformen wird gerade auch vor dem Hintergrund Domino V12 und neuer Features im Bereich vereinfachter Konfiguration interessant.
​Im Fokus dieses Workshops stehen die Plattformen Docker und Kubernetes mit einer Einordnung und Einführung, die auch für andere Applikationen, wie Sametime relevant sind.
Im Hands-On Teil wird jeder Teilnehmer sowohl Docker als auch eine kleine Kubernetes Umgebung mit Domino als Applikation installieren.
Bei diesem Workshop verwendet jeder Teilnehmer eine eigenen Virtuellen CentOS Linux Server und braucht nur eine SSH Verbindung von seinem eigenen Rechner ins Internet. 

Für den Workshop wird u. A., das Domino V12 Early Access Docker Image zum Einsatz kommen.
Aber auch das Open Source Domino Docker Image als Beispiel für die Erstellung angepaßter Docker Images. 

Sprecher: Daniel Nashed (Nash!Com)
09:00 Uhr Begrüßung, Vorstellungsrunde & Logistik
10:15 Uhr Docker & Kubernetes
​                   Grundlagen, Einführung und Einordnung unterschiedlicher Plattformen
​                   Warum „Containerization“ & Automatization
​                   Domino und andere HCL Produkte in der „Container-Welt“

11:15 Uhr Q&A und kurze Pause
11:30 Uhr Hands-on: Installation Docker & Domino auf dem „eigenen“ Cloud-Server
13:00 Uhr Mittagspause
14:00 Uhr Einführung Kubernetes
14:30 Uhr Hands-on Installation Kubernetes auf dem eigenen Server
15:45 Uhr Q&A und kurze Pause
16:00 Uhr Domino V12 Early Access Feature Preview und Hands-on
16:45 Uhr Ausblick, Diskussion, Einordnung und Fragen
17:00 Uhr Ende des offiziellen Teils

ACME providers beside Let’s Encrypt do you use?

Daniel Nashed  3 December 2020 00:51:34
Is someone using other ACME providers then Let's Encrypt to request web server certificates?
I have been playing with all applications/services I found...

Pebble and Boulder are just test servers for ACME client developers.
You can't really use them for anything production.
The certificates are not trusted and when you restart the server, all your accounts are gone.

But SmallStep CA is a pretty interesting project and I have blogged about some weeks ago.

There are two other ACME enabled CAs which provide freemium services.

ZeroSSL needs an account which you can register for free. And than the ACME client needs to support external account binding (EAB).
In this case an API token generated with your account, which is used by the ACME protocol when registering an ACME account.

BuyPass has also free SSL certificates.

I found the following limitations and functionality so far when playing with those two providers:

- No ACME account rollover
- Maximum NIST P-384
- Does support certificate revocation

- Only RSA for ACME account
- Does support ACME account rollover
- Maximum NIST P256 certs
- Does support certificate revocation
- Certificate is valid for 6 month -- which is great for testing but for production you want short certificate life time and we have automatic renewal via ACME anyway.

Here is the list of all implementation I looked into.

Let's Encrypt Production

Let's Encrypt Staging

Let's Encrypt Boulder

Let's Encrypt Pebble

ZeroSSL - requires external accout binding (EAB)


SmallStep ACME CA

Domino on Docker remote workshop with hosted servers

Daniel Nashed  22 November 2020 18:00:50

Last week Thomas Hampel and myself did the first part of a remote hands-on workshop. Each participants is sitting @home on their own computer.
Last year at the DNUG workshop we had everyone install their Linux server on their notebook on the VM product of their choice. That was a bigger challenge that we thought.
I prepared a whole server with local lab network with CentOS mirror, Docker registry, DNS, local clone of Docker documentation, own Git server, it took still quite a while to have everyone up and running.

Standardized hosted infrastructure in the cloud
I had that idea a while ago and last weekend I finally took the time to write it up for this workshop.
It took some time, but this will not be the last workshop where we are using it.
For sure we will have more workshops and make that concept available for others like DNUG here in Germany.

Hetzner Cloud

The ISP Hetzner in Germany really isn't what they used to be in their early days.
They offer very modern and flexible services with a very nice modern UI, to manage servers and DNS for a very moderate price.
The smallest server with 1 CPU core, 2 GB RAM and 20 GB disk costs around 3 euro per month. And you pay for it by the minute!

In addition they provide a simple to use REST API for server management and also DNS management.

Creating new servers on the fly with a Notes database with Lotus Script
I could have used their management website to create servers and DNS entries manually.
But because I wanted to have a deeper look into the newer Lotus Script classes for  HTTP requests and for JSON, I wrote a Lotus Script class "HetznerServerFactory".

It can be used to create servers, DNS entries, reset passwords and later on remove the servers and DNS entries after the workshop.
There are two different APIs needed. Both use API authorization tokens and have a well defined interface:

Cloud server API:


So the notes database creates custom servers, gets the IP address back and uses the DNS API to create matching DNS records on the fly.
There is also an agent sending notification mails including hostname and login information.

Benefits of this approach:

Every participant got their own standard server, with their own name and proper DNS configuration.
And we can focus from the first minute on the topic -- not on server installation.

The master server contains all the work-shop data like the HCL Domino V12 November code drop Docker image and we just used rsync in the same network to get all the data to each server.
Also updates and additional packages can be installed quickly -- independent of the network speed in each home office, where only a SSH connection was needed.
For remote control we use the same ECDSA key for all servers. So we could help each other in the cloud lab environment.

This setup might be even useful for workshops at conferences, where limited network connectivity can be an issue.

Hetzner Cloud Projects

Hetzner also provides their own command-line API and there are additional projects for other use cases.

And there are also official projects with a command line interface to the Hetzner cloud.

But in our case the notes database with the REST calls made more sense.

Next steps

This week we are going to deploy a Kubernetes server on all of those server.
After looking into OpenShift (which is really a resource hungry dog) I decided to use, which can bring up a small Kubernetes environment in 1 minute -- on a machine with 1 CPU core and 2 GB RAM!.

And I created an own TLS and password protected Docker registry to host the container images we will use :-)

I think we should have a lot more Docker & Kubernetes enablement in the community.
And we are probably adding more examples and howto information to our Domino Docker project.
But I think even from home those type of workshops can work with the right preparation.

-- Daniel

Domino DB Directory Cache changes in Domino 10 -- No dbdirman.nsf not needed any more

Daniel Nashed  20 November 2020 16:28:21

Some of you might even not know the database ever existed. But I got question today and I wasn't aware of the change.

The dbdirman.nsf was used to cache the database meta information on shutdown and read it again when the server starts.

With Domino 10 and higher the database is not needed any more and is disabled by default, because Domino 10 is optimized to work faster even without it.

So for a new Domino 10 server you will not find the database and this is perfectly OK (the template still exists).

The internal cache is still working in a similar way in memory like it did before. But there is just no need for a database to cache start for the restart case.

-- Daniel

    Traveler 11.0.2 released -- including Support for MySQL in Traveler HA mode

    Daniel Nashed  18 November 2020 22:53:53
    Beside some important JVM out of memory fixes and room reservation support for ESA 16.x with Apple calendar, there is one more big new thing!

    Support for MySQL !! This is big news for many customers.

    Today the DB2 license is still bundled with the Traveler HA license. This is still covered by an agreement with IBM.
    But this support will end as some point and if a customer uses DB2 just for Traveler HA, this might be overkill..

    A SQL server needs to be separate licensed -- if a customer doesn't have licences anyway, this can be some reasonable amount to pay.

    MySQL is a free, open source relational databases, which has shown in tests, that it can compete with a SQL server in a Traveler HA environment.

    Don't ask me for a migration from SQL server to MySQL yet .. But a quick Google search has shown resources that don't look like mission impossible. I found a YouTube video demoing it.

    DISCLAIMER: Currently MySQL is only supported for fresh setups or new migrations from stand-alone to HA mode.

    There is no official migration path between different databases used in HA mode.
    The migration from a stand-alone server using Derby is fully supported.

    My info was more about that could be possible in general.
    If someone is interested in testing what happens in a test environment, I would be very interested in the results.

    But until HCL comes out with an official supported migration path, I would not try this at home ;-)

    I also found an interesting TCO article when searching for the migration -->
    Depending on what you do, this could make a difference :-)

    Of course MySQL can run in a Docker container -->

    OK to be fair, SQL Server also has a Docker container on Linux! -- Which surprised me a while ago.

    But having an open source database on Docker is a different animal  ;-)

    HCL Technote for Traveler 11.0.2

    Notes/Domino 11.0.1 FP2 has been released

    Daniel Nashed  18 November 2020 17:16:47
    If you are a Mac user, you have to install ASAP! There is a known issue with big sur, which is fixed in FP2!
    There are a couple of issues on the Mac fixed.

    There are a couple of important stability fixes and the JVM has been updated.

    And there is an ID Vault fix adding multiple ~ symbols, which was a regression in FP1

    +SPR# SSARBTRH9T - Server - Archived users in the ID Vault who already have a ~ prepended to their name due to consecutive bad passwords will no longer have an extra ~ prepended to their name after they are archived. This regression was introduced in 11.0.1 FP1.

    Our Docker image als has been updated in the develop tree -->

    - Daniel

    See you @ HCL Digital Week next week

    Daniel Nashed  7 November 2020 17:37:44
    This year (like so many other events) the legendary HCL Factory tour, part of digital week for other parts of the HCL software portfolio, is an on-line event.
    I start to miss to meet with friends in the community face to face. But it will be still a great event!

    The last Factory tours have been a quite limited local event. Now many more people around the world have the opportunity to participate.

    I am looing forward to it and will be joining the security round table @Factory tour in my role as HCL Ambassador.

    See you virtually next week ..

    -- Daniel

    Image:See you @ HCL Digital Week next week

    The new magic number of the year is 398

    Daniel Nashed  4 November 2020 20:05:22

    Wow I almost missed that. I spoke with a friend who mentioned it today.

    I did know about all the discussions for certificate life time. But I missed the official statements.

    So basically we cuts the time by 50%.

    Before the mangic number was 825 days. Now it will be reduced to 398 day!

    For everyone running Let's Encrypt certs today, this will not a big deal, because they have a 90 days lifetime anyway and you automatically renew them.

    For everyone else without certificate management in place, this will be a big deal!

    Update 5.11.2020. I had a call with Christian Henseler who really looked into the details what Apple and Google wrote. In contrast to what we have seen with earlier Apple statements for the old limit of around 2 year, this does not affect private/corporate CAs if we read it correctly.

    The old statement from Apple --> is not limited CAs in the Apple trust store.
    Still that year will apply everyone running public websites with official certs.

    --snip --
    Apple: "This change will not affect certificates issued from user-added or administrator-added Root CAs."

    Google: "This will only apply to TLS server certificates from CAs that are trusted in a default installation of Google Chrome, commonly known as “publicly trusted CAs”, and will not apply to locally-operated CAs that have been manually configured. "

    --snip --

    What does that mean for Domino?

    Today for many of my Domino customers I always have to help with openssl and kyrtool.

    And I know all the commands when you wake me up at night.

    In future with Domino V12 most of it will all work "automagical".

    Automated certificate management is one of the cool new features in Domino V12.

    The first code drops of the early access program delivered Let's Encrypt support.

    But there are other provides like ZeroSSL ( or Bypass ( using the ACME protocol that we can expect to be supported in future.

    What is also already available is manual certificate operations -- and you can use all of this functionality today by pushing certs to older servers.

    "Manual" certificate operations

    You can paste a complete pem file including private key, cert, cert chain and trusted root into a field and submit a request which automatically reads all certificate information.

    But the preferred way would be

    - configure your host names, organisation name etc
    - let the new CertMgr create a private key and CSR for you

    - Use the copy action button to copy the CSR to your CA

    - Finally paste the certificates back using the right fields

    Today this is still automatically convered into a kyr file.
    But the October code drop already allowed you to specify a host name instead of a kyr file in your internet site / server doc configuration and the internet tasks have been reading it from there.

    In future code drop we can expect more automation and integration.

    So this replaces the need for openssl and the kyrtool. And this can be even user for older servers to generate the kyr files.

    I have posted an agent in the early access forum to push kyr files to a server.

    Given that certificate lifetime will be more and more reduced over the years, certificate management becomes more and more important!

    Domino V12 Early Access

    If certificate management in Domino is your topic, you should take the time to look into the Domino V12 early access code drops!
    It's really early access with code shipped once a feature ready to get feedback! Not just for this product area.

    -- Daniel

    PS: I don't want to confuse you with Certificate Transparency which replaces HTTP Public Key Pinning (HPKP) implies shorter lifetimes for certificates, because they want at least 2 certs in their database.

    But I at least want to mention and reference it below.


    Podman vs Docker on CentOS 8

    Daniel Nashed  2 November 2020 11:17:21

    As posted earlier it isn't that easy to get the current Docker 19 version installed on CentOS 8, because RedHat is shipping an older with CentOS 8 than on CentOS7.

    Podman ( becomes more and more interesting and also since a couple of releases supports integration with K8s / OpenShift.
    It's no secret that RedHat is in favor of Podman. They don't ship the latest version. And Podman recommends using the one shipped with CentOS 8 -- currently 1.6.4.

    There are newer versions available. And I think the way podman allows you to use K8s files instead of what docker-compose is doing, will be a great bridge between podman and K8s / OpenShift environments.

    Using the podman play command you can use the K8s pod definitions.Thats better than using docker-compose.yml with Docker Compose.

    And there is also a new podman play kube command since podman v2.0 which looks promising ->

    I just installed the most recent podman version 2.1.1 on a CentOS 8.2 machine and will do some testing.

    But it really sounds like Docker will have challenges keeping up with Podman is doing.

    And I would also say that Podman is already ahead! I will spend more time looking into Podman.

    The Domino Docker Open Source community project already supports Podman for a while as a build and run-time environment along with K8s and OpenShift support


    HCL doesn't officially support Podman today -- but it works. If you are runnng the Domino V12 Early Access Code Drop and you are running with CentOS 8.x, my preference would be Podman!

    Podman uses the same syntax that Docker uses. I am always using the Podman commands natively. But there is also a compatiblity mode, where Podman supports the Docker commands almost 1:1.

    -- Daniel

    sudo yum install podman

    Last metadata expiration check: 1:12:51 ago on Mon 02 Nov 2020 09:32:11 AM CET.

    Dependencies resolved.


    Package                                        Architecture   Version                                                  Repository         Size



    podman                                         x86_64         1.6.4-10.module_el8.2.0+305+5e198a41                     AppStream          12 M

    Installing dependencies:

    conmon                                         x86_64         2:2.0.6-1.module_el8.2.0+305+5e198a41                    AppStream          37 k

    containernetworking-plugins                    x86_64         0.8.3-5.module_el8.2.0+305+5e198a41                      AppStream          20 M

    containers-common                              x86_64         1:0.1.40-11.module_el8.2.0+377+92552693                  AppStream          50 k

    dnf-plugin-subscription-manager                x86_64         1.26.20-1.el8_2                                          BaseOS            278 k

    fuse-overlayfs                                 x86_64         0.7.2-5.module_el8.2.0+305+5e198a41                      AppStream          60 k

    fuse3-libs                                     x86_64         3.2.1-12.el8                                             BaseOS             94 k

    libvarlink                                     x86_64         18-3.el8                                                 BaseOS             44 k

    python3-ethtool                                x86_64         0.14-3.el8                                               BaseOS             45 k

    python3-iniparse                               noarch         0.4-31.el8                                               BaseOS             49 k

    python3-inotify                                noarch         0.9.6-13.el8                                             BaseOS             57 k

    python3-subscription-manager-rhsm              x86_64         1.26.20-1.el8_2                                          BaseOS            348 k

    slirp4netns                                    x86_64         0.4.2-3.git21fdece.module_el8.2.0+305+5e198a41           AppStream          88 k

    subscription-manager-rhsm-certificates         x86_64         1.26.20-1.el8_2                                          BaseOS            247 k

    usermode                                       x86_64         1.113-1.el8                                              BaseOS            202 k

    Installing weak dependencies:

    subscription-manager                           x86_64         1.26.20-1.el8_2                                          BaseOS            1.1 M

    Transaction Summary


    Install  16 Packages

    Total download size: 35 M

    Installed size: 132 M

    Is this ok [y/N]:

    Domino V12 ACME for company CAs using smallstep

    Daniel Nashed  17 October 2020 10:46:26

    The Let's Encrypt CA only works for web servers exposed to the internet (or at least public Domains in combination with your providers DNS).
    But the smallstep CA does now support the ACME protocol (RFC 8555) -- which is the underlying standard used by Let's Encrypt.
    I was looking for a way to deploy internal web server test certificates for my lab and ran into this.
    The whole setup took me like 10 minutes and it just works!

    Here is the main entry point for their documentation to ACME support-->
    The project is pretty interesting and well done! Beside web server certificates it does also support client certs for SSH etc.
    You can run it inside your company as a CA or sub-CA and it works with Domino V12 Let's Encrypt.

    I just took the Domino V12 October Early Access Docker image and configured it to use smallstep over ACME..
    [You find the full documentation for Domino V12 certmgr here -->]

    The smallstep CA is also available as a Docker image and very easy to deploy -->
    You just need to add a provisioner for the ACME protocol -->

    I just took one of my existing Domino servers and moved it to Docker on the same machine by pointing the local volumes to a new Docker containers running the current Domino V12 code drop.

    It just works !!  -  the only small limitation I found so far is the missing certificate revocation for the ACME protocol -- which the Domino certmgr supports since the October code drop.

    On the Domino side you just create a new ACME account document pointing to your smallstep ACME directory URL.

    Image:Domino V12 ACME for company CAs using smallstep

    With that you are ready to issue your first certificates selecting your local smallstep CA...

    Image:Domino V12 ACME for company CAs using smallstep


    • [IBM Lotus Domino]
    • [Domino on Linux]
    • [Nash!Com]
    • [Daniel Nashed]