Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...


Daniel Nashed

 Domino  AWS 

Domino 14.0 is available on AWS marketplace

Daniel Nashed – 17 May 2024 07:20:45

Domino 14.0 is available on AWS marketplace

The AMI comes wtih 100 licenses and is more intended for new customers.
But it is great news to have Domino available on the AWS market place.

It is based on Redhat Enterprise Linux 9 and comes fully installed including the start menu we have built for the community image.
The installation is actually using the new HCL Domino container community project build files in combination with the Domino Start Script.

This means you could build your own image in the same way -- even with additional build options.
You also get the same menu driven setup and run-time functionality.

I still need to change the documentation for the One Touch Domino install on the start script documentation page.
It is now using logic from the container build and supports the Domino download script.

See details about the new One Touch Domino install script here ->

-- Daniel

Image:Domino 14.0 is available on AWS marketplace

Image:Domino 14.0 is available on AWS marketplaceImage:Domino 14.0 is available on AWS marketplace

 Notes  Domino 

Notes/Domino 12.0.2 FP4 shipped on all portals

Daniel Nashed – 16 May 2024 21:16:06

Personally I am running most of my servers on 14.0 FP1. But it's important to also have 12.0.2 code stream updates.

12.0.2 FP4 is available on Flexnet (FNO), My HCLSoftware download portal (MHS).

The AutoUpdate software.jwt also has been updated including the AUT Catalog enabled client WebKits.

I also updated the develop branch of the container project with 12.0.2 FP4 fixpack data.

But I would still recommend using 14.0 FP1 for containers ;-)

Technote with release notes -->

-- Daniel

Image:Notes/Domino 12.0.2 FP4 shipped on all portals
 Linux  Systemd 

A closer look at Linux Systemd

Daniel Nashed – 14 May 2024 07:27:58

Long time ago Linux switched from init.d scripts to the new systemd service.

Today systemd is the standard for most Linux servers. But it is still a mystery for many admins.
Systemd is much more than a simple replacement. But this also makes it a bit more complex.

I had to look into systemd to stop, update and start a server with an application and ran into a couple of interesting details and useful commands.

The following information uses the Domino Start Script systemd service for reference.

For more details about systemd check the main man page of the project -->

-- Daniel

Listing systemd scripts

The Domino Start script is a systemd script and uses a well defined service.

Service files can be listed with a single command:

systemctl cat domino.service

Description=HCL Domino Server (notes)






ExecStart=/opt/nashcom/startscript/rc_domino_script start

ExecStop=/opt/nashcom/startscript/rc_domino_script stop








Checking properties of a systemd service

The Domino Start script is a systemd script and uses a well defined service.
But only uses a fraction of the available configuration options.
To list the complete configuration including some status like the current main PID, there is a show command:

systemctl show domino.service

To query just one setting the --property option is useful

systemctl show domino.service --property PIDFile

systemctl show domino.service --property MainPID=24334

Systemd Services use Control Groups (cgroups)

When starting a service the main process and all child processes are assigned to the same cgroup as you can see from the following output.

systemd-cgls -u domino.service shows the Domino service only

All child processes are always part of the same cgroup. On shutdown systemd takes care of cleaning up processed, if not terminated correctly.
A future start of the service requires all previous processes to be terminated.


Control group /docker/85b1e6835ef25a9e71ceec49c04849102436f23a9030f017e95a694dfa3647e4:


├─20158 bash

├─25761 systemd-cgls

├─25762 (pager)

├─init.scope (#601)

│ └─1 /sbin/init

└─system.slice (#614)

 ├─dbus-broker.service (#835)

 │ → user.invocation_id: 88585e258446434cac7d034f53a92d94

 │ → trusted.invocation_id: 88585e258446434cac7d034f53a92d94

 │ ├─10648 /usr/bin/dbus-broker-launch --scope system --audit

 │ └─10649 dbus-broker --log 4 --controller 9 --machine-id 8b3fdf8329b14614896d034706fba6d4 --max-bytes 536870912 --max-fds 4096 --max-matches 16384 --audit

 ├─domino.service (#2252)

 │ → user.invocation_id: c7e98a3ae0fe4856a082f1e7afb9f99f

 │ → trusted.invocation_id: c7e98a3ae0fe4856a082f1e7afb9f99f

 │ ├─24334 /bin/bash /opt/nashcom/startscript/rc_domino_script start

 │ ├─24391 /opt/hcl/domino/notes/latest/linux/server

 │ ├─24399 /opt/hcl/domino/notes/latest/linux/logasio NOTESLOGGER reserved

 │ ├─24407 /opt/hcl/domino/notes/latest/linux/event

 │ ├─24435 /opt/hcl/domino/notes/latest/linux/amgr -s

 │ ├─24436 /opt/hcl/domino/notes/latest/linux/adminp

 │ ├─24438 /opt/hcl/domino/notes/latest/linux/certmgr

 │ ├─25296 /opt/hcl/domino/notes/latest/linux/amgr -e 1

 └─systemd-journald.service (#692)

   → user.invocation_id: 43b26fdee15c48938fcbce780d931c8e

   → trusted.invocation_id: 43b26fdee15c48938fcbce780d931c8e

   └─19 /usr/lib/systemd/systemd-journald

Static linking C/C++ applications with OpenSSL

Daniel Nashed – 12 May 2024 09:15:31

Usually applications are dynamically linked to OpenSSL.

There are pros and cons about dynamically linking OpenSSL.

Some applications dynamically link with OpenSSL and provide their own version of the OpenSSL run-time DLLs/shared libs.

This causes more issues than it solves, because you end up with multiple instances and depending on the lib path set, this can cause the wrong instance to be loaded.

Using the OS level OpenSSL share libs on the other side might leave you with an older OpenSSL version.

OpenSSL is an essential part of Linux. But not every distribution updates OpenSSL at the same time.

This means usually it is best providing OpenSSL with your application.
To avoid conflicts with other applications statically linking is a good idea.
This allows full control of the OpenSSL version and makes OpenSSL really glue into the application.

Note: Current Notes/Domino releases are statically link very current OpenSSL versions as well.

How to statically link

The two static link libs required for OpenSSL are

  • libcrypto.a
  • libssl.a

On Linux the is an easy way to build them for your platform and version (glibc dependency) is to start a container and build it on your own from the sources once.

The resulting static link libs can be then used for your applications.

In my build environment I have the sources of OpenSSL and switch to the right branch to build the latest release version once.

My build scripts for my applications automatically detect if static link libs are available and link OpenSSL statically (part of the makefile).

Run a new container based on Redhat UBI 9 to ensure a stable build environment (glibc 2.34).

docker run --rm -it bash

Install git, compiler and make

dnf install git gcc perl make -y

Create a directory, switch to it and clone the OpenSSL GitHub project

mkdir -p /local/github

cd /local/github

git clone
cd openssl

Switch to the right branch to get the latest stable version (in my case 3.3.0).

git checkout openssl-3.3.0

Finally configure and make OpenSSL



Once the build completes, libcrypto.a and libssl.a are available in your current directory.

Clone nsh-tools and make nshcipher

cd ..

git clone
cd /nsh-tools/nshmailx


Checkout the resulting binary

All newer NashCom tools have a --version command-line option to dump the version and OpenSSL information if the application uses OpenSSL.

/nshmailx --version

Nash!Com SMTP Mail Tool 0.9.8

Copyright 2024, Nash!Com, Daniel Nashed

OpenSSL 3.3.0 9 Apr 2024

(Build on: OpenSSL 3.3.0 9 Apr 2024)

You can see that the build and the run-time OpenSSL is the same.
Also the below dependencies don't show any OpenSSL dynamic lib.

ldd nshmailx (0x00007fff771a0000) => /lib64/ (0x00007f26638f3000) => /lib64/ (0x00007f26636ea000)

      /lib64/ (0x00007f266390c000)

Building applications in a build container

Daniel Nashed – 12 May 2024 08:17:00

Specially when developing for different target versions of an OS or an application a build container can be very helpful.
But build containers are also really helpful in larger teams when everyone should use the exact same build environment.

The Domino container project supports adding the Notes/Domino C-API SDK to the container image.
In case of Domino is required. Therefore compiling requires at least an installed Domino server with the same or newer version than the SDK version.

I built a Domino 14.0 FP1 image including C-API 14.0 and tagged it hclcom/domino:build.

For this blog post I am using the simple test program in the container projects automation test directory -->
The directory is defined as a volume inside the container /build.

To ensure the binary can be written I am running the build using the root.
Alternative you can ensure that the 1000:1000 has write permissions in the directory to write the object and binary.

The following command starts a temporary build container and invokes make  (for better readability the line is split into separate lines).
finds the makefile and uses the compiler inside the container to compile

docker run -it --rm -w /build --entrypoint= -v $(pwd):/build -u 0
-e LOTUS=/opt/hcl/domino
-e Notes_ExecDirectory=/opt/hcl/domino/notes/latest/linux
-e LD_LIBRARY_PATH=/opt/hcl/domino/notes/latest/linux
-e INCLUDE=/opt/hcl/domino/notesapi/include
hclcom/domino:build make

Here is the break down of the command into it's components.

I have added the OpenSSL Development environment packages to the C-API development environment.
Most Domino add-on applications don't require OpenSSL. But adding this option I can use this container build environment also for native C/C++ applications which use OpenSSL.


Runs a new container based on the specified image


The specified command is invoked interactively instead of detached (-d)


Once the command is executed and the container is stopped, remove it

-w /build

Switch to the build directory as the working directory to ensure the make command is executed in the right directory


The Domino container has an entrypoint script configured to start Domino automatically when the container is started.
Specifying an empty entrypoint, skips invoking the container image defined entrypoint.

-v $(pwd):/build

"Mount" the current directory into the container at /build.

-u 0

Run as root to ensure the output can be written. Can be omitted when user 1000:1000 has write permissions to this directory.

-e LOTUS=/opt/hcl/domino

Export the main LOTUS environment variable pointing to the server, which is used in makefiles.

-e Notes_ExecDirectory=/opt/hcl/domino/notes/latest/linux

Export the Domino binary directory location used in makefiles.

-e LD_LIBRARY_PATH=/opt/hcl/domino/notes/latest/linux

Set the LIB path to allow the linker and runtime to find the notes share objects

-e INCLUDE=/opt/hcl/domino/notesapi/include

Finally set the makefile INCLUDE directory to the C-API expected header file definition to find the header files.


Image name of the build container


finally the make command executed inside the container.

Using Containers for software testing

Daniel Nashed – 5 May 2024 15:27:50

Using Containers for software testing

Containers are great to deploy software and also to test container images before deploying them.
But sometimes you want to just install software manually inside a "sandbox".
On Windows the Sandbox is a great way to test. On Linux you can just run a container.

In my current test case I am using a Redhat UBI Init image, which provides systemd support.
Usually systemd isn't what would be the right choice for a container image.
Containers usually run a single application/process and a entrypoint script is the right choice.

But for testing a container with systemd is an interesting option.
It requires a privileged container to support systemd -- which should be avoided for production environments.
For testing this is perfectly OK.

Run a new container

docker run -d --name init --privileged

Jump into the container

docker exec -it init bash

In my case I amend to end testing Domino installations.
I have updated the Domino native install script to work in this environment.
But there are many different use cases.

curl -sL | bash -

Domino Container Project: software.txt link & new start script version

Daniel Nashed – 5 May 2024 09:06:11


The container project contains a software.txt file with all the web-kits used to build images.
For some technical reasons the file was located in two places.

1. The script uses software.txt for checking web-kits before the image build starts.
2. The actually image build process uses software.txt to verify the downloaded web-kits before installing them.

software.txt and current_version.txt can also be added to a custom software directory (SOFTWARE_DIR) or remote download location (DOWNLOAD_FROM).

Git can handle symbolic links on Linux platforms. But converts them to a text file on Windows.
But Windows isn't a build platform for the container image and I changed one of the two locations to a link:

The file in /local/github/domino-container/software is now pointing to a relative symbolic link

ls -l software.txt -> ../dockerfiles/install_dir_common/software.txt

Domino Start Script 3.9.2 update

The start script is part of the Domino container, but maintained in a separate repository.
From time to time it is synced with the container repository when important corrections or feature additions are pending.

The latest version of the start script provides a new menu, which is also useful inside a container.

Changes are already merged to the develop branch for feedback.

Image:Domino Container Project: software.txt link & new start script version

HCL SafeLinx 1.4.2 available -- New best friend "Domino CertMgr"

Daniel Nashed – 29 April 2024 20:15:44

This was a quite high rated AHA idea. So the SafeLinx team and Domino team teamed up to implement it.
The flow is integrated into the SafeLinx UI and SafeLinx also allows ACME challenge "passthru".

There isn't any change in CertMgr needed. It is implemented in a way that you could implement your own integration flows.
If you have a specific integration idea, ping me. I can point you to the right direction.

-- Daniel

You find all the details about the SafeLinx integaration in help and the related KBase article.

Setting Up CertManager with Domino and SafeLinx Server

Configuring Automatic Certificate Updates in HCL SafeLinx

Domino container project base image switch from CentOS Stream 9 to Redhat UBI 9 Minimal

Daniel Nashed – 26 April 2024 10:03:26

For a container project the Linux base image is a very important decision to take.
The base image is really the Linux base you pick to base your container on.

HCL uses the RedHat Universal Base Image (UBI) for a long time.
The community project selected CentOS Stream as the base image because of the following reasons:

  • Separate glibc language packs per language (in contrast UBI only has one big pack of 200 MB if you need anything else then EN).

  • epel-release for additional software available out of the box (this means for example direct Borg Backup download. but there are other ways to install it and the build script can take care of it meanwhile).

On the other side there are good reasons to use UBI

  • UBI provides a working minimal base image which allows to better control which packages are installed for a Domino server.
    In combination with the change introduced a while ago to use gdb-minimal package to avoid python, the container has much less packages installed (not just size relevant but also from CVE point of view).

  • UBI base images are on the Redhat registry ( and also the package updates are coming from Redhat directly.
    With CentOS Stream the updates are coming from a random mirror that is "close" to you.
    If you are in a restricted network where you have proxy settings that require to limit the target hosts, this is an important difference.

  • There have been issues accessing mirrors an getting packages recently, which causes container build failures.


Meanwhile many more corporate admins are leveraging the Domino container project.
So it looks like a good move to change the default to UBI Minimal and be more aligned to the HCL base image as well.

All existing container base images will continue to be supported.
This is just a change of the default base image.
You can still us the CentOS Stream 9 base image by just selecting it at image build time via -from=centos9.

Avoiding Docker registry at build time for NGINX

Once we switch, you can consider only getting signed images from Redhat.
There is also a NGINX build option (-nginx=dom-nginx) to avoid accessing the Docker registry completely.
This options builds a NGINX image based on Redhat UBI and does not download the NGINX image from Docker Hub either.

Unless anyone raises valid concerns, I would introduce the changed default in the develop branch and then merge it to the main branch next week.

-- Daniel

Discord Webhooks -- Sending messages via REST-API

Daniel Nashed – 21 April 2024 18:31:51

I have been looking for a simple way to send messages to a messaging application.

In Discord it is pretty simple to create a simple webhook.

You just go to your channel and create the hook.
The webhook URL contains the authentication and the target.

Here is how the result looks like. Below you find the simple bash script and the configuration.

-- Daniel

Image:Discord Webhooks -- Sending messages via REST-API





curl -H "Content-Type: application/json" -X POST -d "{\"content\": \"$1\"}" $discord_url


send_discord_msg "Hello, World!"

Image:Discord Webhooks -- Sending messages via REST-API



    • [HCL Domino]
    • [Domino on Linux]
    • [Nash!Com]
    • [Daniel Nashed]