Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

Domino V12 ACME for company CAs using smallstep

Daniel Nashed  17 October 2020 10:46:26

The Let's Encrypt CA only works for web servers exposed to the internet (or at least public Domains in combination with your providers DNS).
But the smallstep CA does now support the ACME protocol (RFC 8555) -- which is the underlying standard used by Let's Encrypt.
I was looking for a way to deploy internal web server test certificates for my lab and ran into this.
The whole setup took me like 10 minutes and it just works!

Here is the main entry point for their documentation to ACME support-->
The project is pretty interesting and well done! Beside web server certificates it does also support client certs for SSH etc.
You can run it inside your company as a CA or sub-CA and it works with Domino V12 Let's Encrypt.

I just took the Domino V12 October Early Access Docker image and configured it to use smallstep over ACME..
[You find the full documentation for Domino V12 certmgr here -->]

The smallstep CA is also available as a Docker image and very easy to deploy -->
You just need to add a provisioner for the ACME protocol -->

I just took one of my existing Domino servers and moved it to Docker on the same machine by pointing the local volumes to a new Docker containers running the current Domino V12 code drop.

It just works !!  -  the only small limitation I found so far is the missing certificate revocation for the ACME protocol -- which the Domino certmgr supports since the October code drop.

On the Domino side you just create a new ACME account document pointing to your smallstep ACME directory URL.

Image:Domino V12 ACME for company CAs using smallstep

With that you are ready to issue your first certificates selecting your local smallstep CA...

Image:Domino V12 ACME for company CAs using smallstep

HCL Domino 12.0 on Docker Early Access October 2020

Daniel Nashed  13 October 2020 20:35:03
The new code drop is available with more preview features.

One of the highlight features is TOTP. You can now use two factor authentication for HTTPS connections.

Here is the list of focus features for code drop 3.

If you are interested in Domino security, you should really have a look into the recent features.

You should join the Early Access forum to give early feedback.

Update: Link to the official blog post -->

-- Daniel

Easy kyr file creation with Early Access V12 in production

Daniel Nashed  10 October 2020 23:40:52

The kyr format is a really old propritary IBM format.
Since Domino 9 the only way to create kyr files is to use the command-line kyrtool.

It can only import existing key pairs + certificates.
So the current flow is often to use OpenSSL to create a key pair and a CSR or to import existing key pairs with the matching certificates.

That flow is going to change with Domino V12 completely.
The CertMgr servertask and the cerstore.nsf will completely simplify the operation and remove the need for kyr files.

But it will still allow to generate kyr files for older servers ..
And you can use it today to generate kyr files for your existing servers ;-)

-- Daniel

Let’s Encrypt Domino Early Access V12 in production

Daniel Nashed  10 October 2020 23:03:49

We got the first two code drops for the early access program.
The October code drop show be available soon. With new features ..
This is a great opportunity for an early look and to provide feedback.

And you can leverage the Let's Encrypt functionality on an internal test server connected to your production server today!
Let me show you how it just did it on my existing production server running.
The scenario is supported and documented as one of the current preview deployment scenarios.
Your existing server just needs the DSAPI filter copied from your Domino V12 server.
If you are working with Windows on your existing servers, HCL could provide a Windows version as well.

Here are the basic steps and I added a simple agent for automatically remote deploying the kyr/sth file.

The DSAPI intercepts the Challenge requests and replies with the challenge stored in certstore.nsf.
So your CertMgr Server running Domino V12 does not need to be exposed to the internet).
This will work on any machine that can connect to your existing server over NRPC.

Copy DSAPI from Container to local disk

docker cp domino12:/opt/hcl/domino/notes/latest/linux/ .

Transfer DSAPI to existing machine
  • upload file via ssh/winscp/MobaXterm etc
  • cp /home/notes/ /opt/ibm/domino/notes/latest/linux
  • chmod 755 /opt/ibm/domino/notes/latest/linux/

Add DSAPI to internet sites/server doc and restart HTTP

You should see the following line when it is loads:

29.08.2020 08:24:58   CertMgr: CertMgr / ACME & Let's Encrypt DSAPI

Point certmgr to your existing server
  • Create a certstore.nsf replica on your existing server
  • Set notes.ini on your Domino V12 server e.g. to point to your existing server.
    (don't be confused this is really the CN of my Server -->  -- to avoid DNS issues)
  • Restart of the certmgr server task. From there on certmgr only looks into this database for requests/challenges etc

Create a new request for your existing server and let certmgr process it.

Image:Let’s Encrypt Domino Early Access V12 in production

Deploy the kyr file

Certmgr automatically deploys kyr files only on the CertMgr machine.
In future Domino V12 servers should not need a kyr file. So deployment would be reading certs from the keyfile document directly.

But for your convenience you could my small agent, with uploads the kyr file to the current server for now.
Actually it is one trigger agent and a small run on server agent to deploy the kyr/sth files (see below).

You just run it and restart your HTTP task to have your new certificate ready:


DNS-Names   :
Common Name :
Expiration  : 2020-11-27 05:35:05 UTC
Days valid  : 89
Subject     : /
Issuer      : /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
PubKeyAlg   : rsaEncryption
PubKeySize  : 4096 bit
Sign Alg    : sha256WithRSAEncryption
Curve       :

StatusCode  : 0
StatusText  : ok

HCL Ambassador nominations open until end of October

Daniel Nashed  5 October 2020 11:24:50

Image:HCL Ambassador nominations open until end of October

It's time again for nominations.
Even you haven't seen much of us in person. Many of us have been really active on-line.
There have been and there will be more on-line user group events this year.

It is time to say thank you and vote for people in the community, who deserve it most.

We are an outstanding community and I am proud & happy to be part of it.

Click here to nominate

Thanks to Tim Clark the nomination application is based on HCL Volt!
Tim, thanks for all the hard work (and fun) you have with us over the years.

-- Daniel

In case you are wondering about the HCL Ambassador program. Here are 3 of the main statements and a link for more information.

"HCL Ambassador is a distinction that HCL awards select members of the community that are both experts in their field and are passionate about sharing their HCL knowledge with others."

"HCL Ambassadors are exactly that, ambassadors. Importantly they are not employees, but their commitment to sharing their expertise has a huge impact on the HCL community. Whether they are blogging, writing books, speaking, running workshops, creating tutorials and classes, offering support in forums, or organizing and contributing to local events – they help make HCL’s mission of making technology play nice, possible."

"HCL Ambassadors are eager to bring their technical expertise to new audiences both in person and online around the world."

Domino HTTP show kyr file used

Daniel Nashed  28 September 2020 07:40:53
Just ran into "tell http show security"
I did not notice it before.
Can be useful to show the currently configured kyr file per website.

Here is an example ..

-- Daniel

 tell http show security
  Web Site: CSI Domino Cloudflair Internet Site (
     SSL enabled
     Key file name: /local/notesdata/cf_csi.kyr

  Web Site: CSI Domino SAN Punycode (
     SSL enabled
     Key file name: /local/notesdata/csi.kyr

Install Docker 19.03 on CentOS 8+

Daniel Nashed  25 September 2020 10:55:57

RedHat is still shipping an older containerd version with CentOS 8 than what they shipped with CentOS 7.

This blocks Docker 19+ installations. The only way to get Docker installed, is to use the -nobest option which will install Docker 18.x.

But you really want to install Docker 19.03

Here is how it looks like when you installed with -nobest option. An update will not work and also block some other updates.

yum update

Last metadata expiration check: 0:31:00 ago on Fri 25 Sep 2020 10:14:12 AM CEST.


Problem: package docker-ce-3:19.03.13-3.el7.x86_64 requires >= 1.2.2-3, but none of the providers can be installed

- cannot install the best update candidate for package docker-ce-3:18.09.1-3.el7.x86_64

- package is filtered out by modular filtering

- package is filtered out by modular filtering

- package is filtered out by modular filtering

- package is filtered out by modular filtering

- package is filtered out by modular filtering

- package is filtered out by modular filtering

- package is filtered out by modular filtering

- package is filtered out by modular filtering

- package is filtered out by modular filtering

(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

You can download the newer package needed, here ->

The current version can be installed linke this (this will change over time)

dnf install -y

Once you updated containerid, you just run an update via yum:

yum update

In my case this updated 154 packages!

So apparently this blocked other updates as well..

Stay with CentOS 7 if you can for now...

Side note: not everything is better when you install a new OS version.
If you can, you should still stay with CentOS 7 for your Docker servers.
There are some network changes which could impact depending on your setup.
For example I had issues connecting Docker containers in the same defined Docker network.

- Daniel

    Rethink What You Know about HCL Domino

    Daniel Nashed  20 September 2020 22:32:15
    Wow! Those are statements we would never have heard from IBM.

    I would really wish we would have seen HCL taking over Domino and all the other collaboration products earlier ...

    For me Notes and Domino is still the best collaboration and rapid application development platform. Volt leverages Domino and it is evolving!

    Many customers are staying on older versions of Domino even compared to other platforms Domino is easy to update in place!
    Older Domino versions work to good to have the need for an upgrade. Even when upgraded new functionality is often not leveraged.

    Image:Rethink What You Know about HCL DominoImage:Rethink What You Know about HCL DominoImage:Rethink What You Know about HCL Domino

      HCL Domino Early Access presented at DNUG event

      Daniel Nashed  14 September 2020 22:00:44
      The slides have been posted here ->

      Image:HCL Domino Early Access presented at DNUG event

      You find the link in the offical Domino Early Access post:

      The slide deck gives you a background about the Early Access Program and also useful tips how to start (including a longer appendix with installation tips).

      You can directly start with the September code drop available on FlexNet.

      -- Daniel

      HCL Domino V12 Early Access September Code Drop available

      Daniel Nashed  14 September 2020 11:19:08

      The new code drop is available ...

      Image:HCL Domino V12 Early Access September Code Drop available


      • [IBM Lotus Domino]
      • [Domino on Linux]
      • [Nash!Com]
      • [Daniel Nashed]