Engage 2026 Agenda is live -- My CertMgr session got accepted

Just before the weekend the agenda got published --> https://engage.ug/pages/session2026

Bill, Martijn and myself are going to have a couple of sessions about Domino on Linux (Installfest & Roundtable sessions).

And there is also a Linux desktop session from Bill. We might have surprises for admins & developers who want to run Linux on their notebook as you might guess from earlier Linux on USB stick posts.
But beside the shared Linux sessions I am also very happy that my Domino CertMgr session got approved.
The session will go way beyond the standard use cases and I am working on some integration options which I will go thru and publish before or at Engage.

Domino CertMgr

If you have specific use cases that go beyond the standard use cases, I want to hear from you.
I have plenty of ideas and ready extensions. But I want to make this to make a practical experience.
You can ping me either directly or thru the HCL Domino CertMgr GitHub repository by opening an issue --> https://github.com/HCL-TECH-SOFTWARE/domino-cert-manager
There is already a lot of extra material in the repository and there is also OTS integration for CertMgr.

But the session will go far beyond that, explains technical background and how components work hand in hand together.
There are also some new CertMgr features in Domino 14.5.1 we will go thru.

I am looking forward to Engage conference!

-- Daniel

• Comments [0]

CertMgrUtils Usecase

Here is the use case I had in mind first when building the helper script lib.
A flow where my CertMgrUtils class is used.

Earlier the MicroCA was added as a trusted root.
From there the Script Lib copies the trusted root from CertStore to to names.nsf to make sure the Lotus Script NotesHTTPRequest can connect to the K8s service.

Flow

• Create a key outside Domino using OpenSSL with a password
• Add it via OTS to a setup document in an application including the password
• A setup agent uses the Lib to import the key directly from the document to certstore.nsf
• Request a MicroCA document by filling in the right fields
   
• Create a K8s pod which has the private key assigned to a secret
• The pod runs a Go process using the key and gets the matching certificate from CertMgr via HTTP before starting the listener
• At run-time CertMgr is queried over HTTP with SNI for a matching new certificate matching the private key
• The certificate and key is updated on the fly and the HTTP listener reloads on the fly
   

• Comments [0]

Picking the right cloud server hardware

When choosing cloud server hardware, the devil is in the detail.

CPU performance

Cost effective hardware usually uses older CPU models which don't have hardware support for hash and crypto operations.
In modern application communication requires TLS. Also ZFS and other components require hardware support for SHA operations and crypto.

A simple test shows the difference.

openssl speed -seconds 3 -bytes 16384 sha256 2>/dev/null | awk '/^sha256/ {printf "%.2f MB/s\n", $2/1000}'

My older VM cost efficient server at Hetzner doesn't have hardware support for thos operations:

332.13 MB/s

A more modern machine shows dramatically better performance:

1762.05 MB/s

My local Proxmox host on new Intel hardware even has better performance.

2347.82 MB/s

For larger servers with higher load the new modern CPU makes a lot of sense and is good invested money.


Disk performance

Specially for Domino I/O response time is very important for NSF.
Domino uses many small random I/Os and is more read than write bound.

I/O writes are usually very well cached. Reading data can also be well cached if you have sufficient RAM.

But fast SSDs still make a difference as you can see in an earlier test.

Latency is most important for Domino I/O.
Backup operations require high I/O transfer rates.

Cloud providers often provide different levels of disk performance.

• Different storage classes with different disk types
• Sometimes performance is limited by to certain number of IOPS and thruput unless you pay extra
   

When choosing hardware you really need to balance performance vs. price.



--- NVMe internal disk on an older notebook  ---

Disk  Random 16.0 Read                       321.10 MB/s
Disk  Sequential 64.0 Read                   433.78 MB/s
Disk  Sequential 64.0 Write                   97.33 MB/s
Average Read Time with Sequential Writes       0.620 ms
Latency: 95th Percentile                       1.839 ms
Latency: Maximum                              14.415 ms
Average Read Time with Random Writes           0.591 ms
Total Run Time 00:00:40.08


--- NVMe internal disk on my new notebook  ---

Dramatic increase in read and write performance.
Another 10 times better latency as well!

Disk  Random 16.0 Read                       1508.35 MB/s
Disk  Sequential 64.0 Read                   4414.35 MB/s
Disk  Sequential 64.0 Write                  1138.50 MB/s
Average Read Time with Sequential Writes        0.081 ms
Latency: 95th Percentile                        0.152 ms
Latency: Maximum                                1.208 ms
Average Read Time with Random Writes            0.084 ms
Total Run Time 00:00:07.33

• Comments [0]

NotesClass CertMgrUtils - Export / Import / Copy trusted roots into Domino Directory

As part of an application I am working on, I wrote a Notes Class to manage TLS Credentials.
What was of special interest is to copy trusted roots from CertStore into Domino Directory because it is needed for the NotesHTTPRequest in Lotus Script since 14.5 by default on servers.

There isn't any automation to import Trusted roots. But there is a simple way to import Trusted Roots into certstore.nsf by generating a request.
The resulting document can be copied into the Domino directory -- But the document needs to be mangled a bit.

Function CopyTrustedRootToDominoDirectory (doc As NotesDocument, DominoDirectoryDb As NotesDatabase) As Integer

In addition to Trusted Root functionality I also added the export and import functionality as easy to use functions.
There is a C-API call designed explicitly for use via LS2CAPI used in a ScriptLib

All functionality requires an existing document in certstore.nsf
I might add more functionality over time.

Function CertStoreCreateExportableKey (doc As NotesDocument, ExportPassword As String) As String
Function CertStoreImport (doc As NotesDocument, ImportFilePath As String, CurrentPassword As String, ExportPassword As String) As String
Function CertStoreExport (doc As NotesDocument, ExportFilePath As String, CurrentPassword As String, ExportPassword As String) As String

For now it is mainly intended for my application. But if you need this type of functionality, I am happy to share the ScriptLib which contains the CertMgrUtils class.

• Comments [1]

Are you running Domino on Kubernetes?

With the current new challenges for virtualization platforms to find an alternate solution for VMware work-loads Kubernetes (K8s) might get more attraction.
Some virtualization platforms are even built on K8s. In those cases running a container vs. running a VM could become more interesting.
Running on K8s brings new challenges. A container also on K8s runs a Linux instance with a very thin Linux based virtualization layer.

A container scales to the limits of the underlying Linux platform.
I wrote up a document with some details how to run in production -> https://opensource.hcltechsw.com/domino-container/concept_run-production/

The main challenge is the storage because Domino NSF files require a very stable and connection with low latency and many smaller I/Os.
A standard cloud based K8s environment usually isn't a good fit for larger Domino servers.
You really need special I/O tuned infrastructure. Like an enterprise SAN CSI connection.

If you are running on K8s I would like to understand what type of infrastructure you are running in which scale and how you manage it.
I am currently working on automated container and K8s deployments.


What is of special interest is the storage back-end and backup.
I am looking into different type of scenarios either using NFS based file targets or snapshot flows levering CSI driver snapshots.
Is anyone using Rancher as  management interface?

• Comments [0]

Run workloads on the same IP using NGINX Stream and HTTP configurations in parallel

Today I am working on a K8s lab environment where I only have one external IP.
That means I can only have one NGINX instance listening to the public IP on 443.

But what if I have different type of work-loads.

Some need TLS termination like the Rancher admin interface
Others would work well to send the traffic dispatched on TCP level like Domino requesting it's own certificates via CertMgr


Both are possible at the same time when thinking outside the box.

Every HTTPS request will first hit NGINX on the "stream" configuration
The stream configuration gets the SNI name using a SNI NGINX stream configuration

In case it matches a host that needs TLS termination the request is dispatched to a local port 8443 on the same NGINX instance
The NGINX instance on 8443 terminates the traffic and sends the request to the backend
The backend is still HTTPS but with a private certificate


Using this type of setup you can use a single NGINX DaemonSet to dispatch all your traffic.

The same kind of configuration would also work on Docker. But in my case this is sitting behind a K8s MetalLB to receive all the K8s cluster lab traffic.








 


 

• Comments [0]

Domino 14.5 is not supported on Windows Core Server

Microsoft offers two different installation modes for Windows Servers for the two different editions (Standard & Datacenter).

• "Windows Core server" an installation mode without full graphically UI
• The full server installation with a full administration GUI

Windows Core Server Standard Edition is meanwhile the default when you run the installer.
But even the Domino System Requirements does not explicitly exclude Windows Core server it is not a supported configuration yet.
There is an AHA idea to vote for, which only has 7 votes today.

https://domino-ideas.hcltechsw.com/ideas/DOMINO-I-2524


I am not sure if nobody has the requirement or nobody knows it is not supported yet.

Windows Core Server needs less resources and is the recommended installation mode by German BSI because also from security point of view.
Specially for smaller servers since Windows 2025 it is important to reduce the footprint of the Windows server, because it tends to use more memory out of the box.
It comes with a simple configuration menu for the most important tasks and works well with Domino.
The installer, the classical configuration wizard and also the Jconsole GUI just works unchanged.

Only the service.exe doesn't launch. But to start and stop services you can either use sc command line or use the services part of taskmgr.
RDP works as well. But Windows 2025 also comes with OpenSSH installed and you can just enable for administration via a SSH session -- But that's maybe something for another post.

This post is to raise awareness and to see if someone is currently using it or plans to use it.

If you have a requirement to install it, please vote for my AHA idea.

• Comments [3]

Domino IQ Mail Request Requirements and Troubleshooting

Domino IQ offers two simple to use actions which are integrated into the Mail 14.5 and 14.5.1 mail template.

• Summarize
• Reply with Domino IQ

They are a bit hidden and might not show up if not all requirements match.
The question what is required came up a couple of times.
Therefore I added a diagnostic form to the Domino IQ lab database a while ago.
It checks all the requirements I found in the mail template for the hide when formula around the actions.

Here is an example which also shows the requirements:

• Notes 14.5 or higher standard client
• Use the Notes Client theme Notes 11 or higher
• Use the 14.5 mail template or higher
• Have a Domino IQ server available in your domain (the actions will use the default server in your Domino domain, because the commands don't pass in a Domino IQ server)
• Have the two actions configured and available (The code gets the list of actions available)

You find the details on the Detail tab listed in the last screen shot below.

• Comments [0]

Getting sudo permissions right

sudo can be used in multiple ways. In general it is away to delegate root access for certain operations or to switch to root.
On Ubuntu traditionally no user can login directly with root remotely and you have to switch from your account to root using "sudo su -".
But it can be also used to delegate individual commands or even just some parameters of a command.

Narrowed whitelist

What is really important is that the list is as narrowed as possible.
You should only white list commands.
Allowing all commands and just have a black-list does not work well!

If you would only exclude some operations an admin could for example copy the "bash" binary and just run it to gain root access.

When using sudo to get a root bash session, you should make sure the session can only be opened asking for a password at least.
If you read one of my previous posts, there could be even a time limited authorization to use root using singed SSH keys. But this would need some planning and an application to securely issue those SSH certs after validating the user and approving the access.

Running distinct commands are usually OK without specifying a password.

Here is a simple example how it can look for a "notes" application user.
Starting and stopping all services on a Domino server should be perfectly OK for a Domino admin.
Eventually you also want the Domino user to reboot a machine or similar commands which can be only performed by root.

Also applying updates could be an operation potentially delegated to an application admin.


visudo

notes  ALL= NOPASSWD: /usr/bin/systemctl start *, /usr/bin/systemctl stop *, /usr/bin/systemctl restart *, /usr/bin/systemctl status *, /usr/sbin/reboot

Even allowing all systemctrl operations could be used to gain higher access then intended. You could install your own services for example, which would run with root permissions.


Check other ways to allow permissions -- Example Docker

Think about every operation to allow and check if there are other ways to allow an operation.
For example for Docker you can just add a "notes" user to the "docker" group to allow a Domino admin to run all Docker commands.

usermod -aG docker notes

A good way to operated is to provide everything an application admin has to do to the application user (in our case notes) and let the admins perform restricted operation on need to do level.
Normal application administration does usually not require root.

You would need root permissions for example when updating Domino servers.

But there is also a way around that.

For a native installed Domino server there is AutoUpdate since Domino 14.5 which is authorized by an autoinstall binary which uses with SUID to gain root access to shutdown Domino and install an update.
Those operations are also narrowed down to this exact use case and the software to install is verifying the software to install end to end using a software.jwt agains a public key backed into the binary.

With Docker you can build an image with "notes" permissions and if you Git clone the repositories with the "notes" user there is no need for "root" to build or operate Domino on Docker.

• Comments [0]

Domino 14.5+ Backup supports .ind files

Domino 14.5+ switched to the same back-end DBMT and the design task switched to in 14.5.
This offers out of the box support for more than 20000 databases without a notes.ini parameter.
The number of entries is now dynamic.
Switching to the new way to process databases also introduces support for .ind files.

This should still say a special purpose configuration.
Configuring a full backup with an exclude list is still the preferred way to ensure you are not missing any new added databases in another directory.
But it can be a helpful configuration for some special use cases.


Here is a simple example using the new system database list.
You should note that the exclude configuration is still in place.

lo backup dominosystemdbs.ind

Backup: Starting backup for 29 database(s)

• Comments [0]