What you should know about Domino "res" files on Linux and AIX

res files actually come from Windows and are used to translate strings for UI and errors.
Those res files are usually linked to the Windows binary.

Linux and AIX also use "res" files in a res/ directory below the binary directory.

The files are essential for a server. All the core code string resources are in strings.res.
Most Domino native servertasks also use string resources.
C-API SDK applications can also use string resources, but usually use hard coded strings and not need external res files.

• When installing Domino the locale of the root user is used.
• When running Domino the locale of the "notes" user is used.

Those two locales should be the same. If there is a mismatch, no error texts might be displayed.

"en_US.UTF-8" is truncated to "en_US" and then searched on disk.


Special logic for English locales

• There is specific logic in the internal start script to map English to the system locale "C" (you can see the logic copied below)
• The system locale "C" is the "POSIX" locale and is a special case.
• The directory is also the fallback if no locale is found.

But the following logic is only used for English at run-time and install time.
For other languages the full locale de_DE.UTF-8 is used to install the res files.

But it turns out that at run-time only the first 5 chars of the locale (e.g. de_DE) are used to lookup the resource directory.


Potential problems

In case you had other settings for the root/notes user at install time than the previous install, it can happen that you install into for example the de_DE.UTF-8 directory and the run-time uses the "C" directory.

This would cause potentially wrong res files to be loaded.
Using the wrong res files will not have new string resources added in the new version.
In addition it will show the wrong Domino version, because strings.res ordinal 1 is the Domino version.

It took until now to find out about the details and I had to trace the system calls to find out.
This isn't new at all and I had questions from admins about locale issues.

The Domino container project always uses the POSIX locale during installation.
That means the resulting directory is always called "res/C".
But beside adding the full locale the container project now sets the short locale without UTF-8 for English and German.

When you run a Domino native installation outside a container, you should be aware of those details.
A future Domino version might address this in the installer. But today you should be aware of those details.
In case your Domino version is not updated, you might need to check the "res" files.

Here are the details from a trace and the Domino internal start script called instead of the binaries.

I hope this explains how it works and what to look for when you run into a mismatch.

Today's server all uses UTF-8 locales, which causes effects we might not have seen in earlier versions.

I hope this helps to solve one or another ticket.

-- Daniel



Logic in internal start script -- invoked to start all servertasks

if [ -z "$LANG" ]; then
     LLANG="C"
else
     LLANG=$LANG
     case ${LLANG} in
             POSIX   |\
             posix   |\
             en_US*  |\
             En_US*  |\
             en_GB*  |\
             En_GB*  )
             LLANG="C"
             ;;
     esac
fi
export LLANG


Logic in the installer perl script

# if LANG isn't set or we're in English, move to res/C.  Otherwise, use the language name
if (
  (! $ENV{'LANG'}) ||
  ($ENV{'LANG'} =~ /^posix$/i) ||
  ($ENV{'LANG'} =~ /^en_us/i) ||
  ($ENV{'LANG'} =~ /^en_GB/i)
  )
{
  $langpath = "res/C";
}
else
{
  $langpath = "res/$ENV{'LANG'}";
}

--- systrace ---

access("/opt/hcl/domino/notes/latest/linux/res/en_US", R_OK) = -1 ENOENT (No such file or directory)
access("/opt/hcl/domino/notes/latest/linux/res/C", R_OK) = 0
access("/opt/hcl/domino/notes/latest/linux/res/C/strings.res", R_OK) = 0
openat(AT_FDCWD, "/opt/hcl/domino/notes/latest/linux/res/C/strings.res", O_RDONLY) = 3


Here is what we are adding in the container project

C
de_DE -> C
de_DE.UTF-8 -> C
en_US -> C
en_US.UTF-8 -> C

• Comments [0]

Wolfi OS - Secure base layer for containers

This project is pretty cool. It's a container only OS using the kernel from the host.
But it has a couple of really interesting design goals.

https://github.com/wolfi-dev/

They build container base images with the minimum number of packages and "CVE free" as much as possible.
So their own containers for NGINX for example really only have NGINX and nothing around it -- not even a shell unless you install a :latest-dev container.

In contrast to Alpine Linux this project supports glibc based applications.

So in theory it could also run Domino. When trying it out the NSD script had issues, but the Domino server itself worked.
The NSD script has some history starting with HP-UX, Solaris and AIX and requires certain Linux command-line tools.

But this container Linux is interesting for many other use cases building containers.
I tested all my C/C++ based code projects, which do not use any Domino code. Including applications with OpenSSL code.
There are also ready to use containers like NGINX with zero CVEs currently.

Many of the containers are only for paying customers. But the NGINX container and also for example the Go container are free.
Some containers have a FIPS version, that isn't free.

Base Image

docker pull cgr.dev/chainguard/wolfi-base


NGINX container

https://images.chainguard.dev/directory/image/nginx/versions

docker pull cgr.dev/chainguard/nginx

Go container

https://images.chainguard.dev/directory/image/go/versions

docker pull cgr.dev/chainguard/go

• Comments [0]

Join us at Engage conference in May

Engage conference in The Hague is is around the corner

If you are not registered yet, here is the link -> https://engage.ug/pages/attend2025

At the time of Engage the final Domino 14.5 EA3 code drop will be available and all the features of the almost ready version are available.
This includes Domino IQ with external LLM support as you can read in the EAP forum.

There is a lot of new functionality not just in Domino out of the box. But there is also news on the OpenSource side and OpenNTF.

I am officially involved in those three sessions. But there are more topics I am currently focusing on.
Specially Open Source projects but also other Domino 14.5 features.

Some of the projects work hand in hand and some of it will be featured in the Domino Linux round table.
And there is new stuff that nobody has seen yet.

My blog was silent for a while because of too many interesting community projects and the last sprints of feature work for Domino 14.5.

• Comments [0]

How to update Domino when running in a container?

Today I just updated my production environment to Domino 14.0 FP4.
Let me show you how it works if you have everything setup.

If you are using the Domino container project you can add two additional components from the start script repository (https://nashcom.github.io/domino-startscript/).

1. Domino Container Control

https://nashcom.github.io/domino-startscript/dominoctl/

2. Domino Download Script

https://nashcom.github.io/domino-startscript/domdownload/


The container project and the start script functionality works hand in hand.
The build script leverages the Domino Download script if present and automatically downloads software for you if you have an internet connection an a MHS download token.

The update drills down to those 4 commands:


cd /local/github/domino-container
git pull
./build.sh
dominoctl update

1. Switch to the right directory
2. Pull changes for the project (which brings software.txt and current_version.txt to the latest state so that the build script knows what the latest version is)
3. Run the build script to prompt you for software to install when building the image
4. Update a container with a new image by throwing away the old container and creating a new one with the new image

The build menu below has options to store the current selected components and will remember those settings, which you can just confirm when building.

Depending on your build performance this this can take from like 5 to 10 minutes on modern machines.
The update (down-time) is only the time it takes to shutdown and restart the server.


If you are running the same image on multiple servers, you could push the image to a private registry.
Then the update on a server would be just a "docker pull" and "dominoctl update"
In my environment I usually build on all machines, because all of them run different configurations and platforms.

-- Daniel

• Comments [0]

Notes & Domino 14.0 FP4 released -- You should update!

This is the first version after the required IFs from December last year with the DateTimeDifference critical fix.
A IF (Interim Fix) is just a hotfix or combo hotfix. A fixpack is always the better fit.
You should also update your Notes Clients if you have not installed the IF.

"Domino 14.0 Fix Pack 4 is a collection of low-risk, high-impact fixes to help customers safely avoid known issues." as the Product Info document explains.

If you are running Domino 14.0+ you should enable AutoNotify to get the latest information directly in your Domino Directory.
It also includes links to the most important information about each software product listed as seen below in my example.

In addition you can download directly and distribute releases, FPs and IFs directly to Domino servers.

With Domino 14.5 you will also be able to update Domino servers on Windows and Linux automatically.
If you are at Engage conference in May, you might want to join my session about the latest information for Domino 14.5 AutoUpdate & AutoInstall.
There is also a webinar the development team did in January:

Presentation: https://www.openntf.org/Public/presentations.nsf/0/FE272EA9ED0A921286258C140062B8AB/$FILE/openntf_2025_domino_autoupdate.pdf
Youtube: https://youtu.be/pE_5FzgMKds

Most of my servers are running in containers. But that's a different project and I will have a separate blog post ...






Here is an example of the links you get as part of AutoNotify.

• Comments [0]

CentOS Stream 10 (Coughlan) and Redhat Enterprise 10 Beta

CentOS Stream 10 has been released end of last year. Meanwhile it is available at Hetzner and other providers.
So let's have a quick look and get the Domino container project ready for CentOS 10 and Redhat Enterprise ready.

I am not a big fan of CentOS Stream any more since they dropped CentOS Stream 8 quite early.
But on the other side CentOS Stream is a the base for the Redhat Enterprise Linux major and minor releases.
This makes it a perfect sandbox to try out the next Redhat versions.

CentOS Stream 10 has been released and Redhat Enterprise Linux 10 is in beta.
The following diagram nicely shows how Fedora, CentOS and Redhat Enterprise are maintained.

For details check the video in the official blog post where I also borrowed the diagram https://blog.centos.org/2024/12/introducing-centos-stream-10/





WARNING: The operating system CentOS Stream 10 (Coughlan) has NOT been tested by HCL!


CentOS stream is brand new and not tested by HCL.
Actually HCL is focusing on enterprise versions and is expected to test Redhat Enterprise 10 as soon it is available.

I looked into CentOS Stream 10 and also the Redhat Enterprise Linux 10 Beta which are both available as container images.
The container image did not work because a packet was missing which includes "whereis" -- which was always installed and is needed for NSD to work correctly.


CentOS Stream 10 VM installation

I also looked at the native installation on a Proxmox server.

There are no surprises in the setup UI. Just some minor art work.
The file system used by default is XFS instead of the expected ext4 most systems use.


How to download CentOS Stream 10

You can find the ISO here. This is a download link you can directly paste into for example a Proxmox ISO download.

https://mirrors.xtom.de/centos-stream/10-stream/BaseOS/x86_64/iso/CentOS-Stream-10-latest-x86_64-dvd1.iso


Docker Pull

The container image is on Redhat Quay like the previous images

docker pull quay.io/centos/centos:stream10


Testing with Red Hat Enterprise Linux 10.0 Beta (Coughlan)

The beta for Redhat Enterprise 10 is also already available.

The easiest way to test is to use the container image. The container image brings everything beside the kernel.
But that's fine for most of the tests. There are distributions with later kernel versions.

Here is the official blog post:

https://www.redhat.com/en/blog/red-hat-enterprise-linux-10-beta-now-available


podman run -it --rm registry.redhat.io/ubi10-beta/ubi:10.0-beta bash


Conclusion

All in all there are no surprises and it just works if you add the missing package - which I have added to the container build process.
On a native machine I had to add openssl to the native Domino Linux installer, because it is not part of the standard minimum install.

But beside that it all worked out of the box.


NSD shows a repeated warning

I noticed a new message in NSD which I have seen in other distributions.

egrep: warning: egrep is obsolescent; using grep -E

NSD is a cross platform script which still uses "egrep" for Linux.
Now the "egrep" package starts warning about egrep. The better option is "grep -E".

This should be addressed on Domino side to avoid warnings.

The warning is coming from a script which replaces egrep.
For now you can just remove the warning.


#!/usr/bin/sh
cmd=${0##*/}
echo "$cmd: warning: $cmd is obsolescent; using grep -E" >&2
exec grep -E "$@"


Comments Disabled

How to add a trusted root to Linux

The Domino container project is a very flexible way to install Domino.
Many business partners and consultants are using containers for their production environments and for testing.

The container project https://opensource.hcltechsw.com/domino-container/ supports many different distributions because there is no one size fits all.
In addition there might be needs to test software on different Linux distributions.

The container project is a great option to run with different distributions.
In fact you can switch between distributions for the same server very quickly by just applying a different base image.

For a new customer project we are looking into Debian as the host platform for the host OS running Docker and also for the container.
One of the reasons is that in the corporate world you need proxies to access the internet for example to get packet sources to install and update Linux.

But some customers have the need to run their own mirrors to tighter control the software they are building with and to optimize resources.

I recently added custom repository support for Ubuntu which is also supported by providers like Hetzner.
Now I am extending the custom repository support to also Debian 12 for this customer.

But because this is a community project, this will be available for anyone of course.

The customer also has the need to add their trusted root to the container image.
Usually I would recommend to build an own customer standard container image to derive all images from.
This base image would be build on top of the standard base image from the vendor and build the base for all of your Domino or other application container builds.

But I am still adding custom trusted root support to the Domino container project.
You will be able to just specify a trusted root to add to the local Linux trust store.

Like other low level functionality this works differently on different Linux flavors.

Here is what I am adding for SUSE, for Debian/Ubuntu and basically all the other Redhat/RPM based systems (I rested Rocky, Alma & Co so far).


if [ -x /usr/bin/zypper ]; then
  cp -f root.pem /usr/share/pki/trust/anchors
  update-ca-certificates

elif [ -x /usr/bin/apt-get ]; then
  apt install ca-certificates -y
  cp -f root.pem /usr/local/share/ca-certificates
  # Certs must have the .crt extension
  mv /usr/local/share/ca-certificates/*.pem /usr/local/share/ca-certificates/*.crt
  update-ca-certificates

else
  cp -f root.pem /etc/pki/ca-trust/source/anchors
  update-ca-trust
fi


You can test if a certificate is trusted using OpenSSL or Curl

curl -v https://microca.nashcom.org

openssl verify root-pem
root.pem: OK

• Comments [1171]

Must watch: Unlocking the Power of Nomad for Browsers and Mobile Devices

If you never looked into Nomad Mobile and/or Nomad Web or if you use it already: This is a great video of one after another new feature of the Nomad platform.

https://www.youtube.com/watch?v=lsgpr_P12WE

Nomad is a great platform for mobile devices but also for Desktop experience running a basic client experience in the browsers.
Over the last years the Nomad team did an outstanding job implementing feature after feature.

Since Nomad Server is available  it is super easy to deploy Nomad Web. No extra components are needed.
The Nomad server gets it's TLS certificate from CertMgr and can meanwhile passthru ACME Challenges to the Domino server.

AdminCentral is the first HCL provided template which is working on the Notes Client, Nomad Web and Nomad Mobile.
It shows what you can do with standard Notes application functionality today.

Our DNUG Lab environment has all components enabled. I am using AdminCentral to register and manage all lab users.

Have a look into the latest features. Kudos to the HCL Nomad team! You rock!

-- Daniel

• Comments [1176]

Domino CertMgr and TLS detailed information on GitHub

The CertMgr GitHub repository was initially mainly intended to provide information to CertMgr ACME DNS-TXT integration information.
But I added GitHub pages and a lot of other material over time.

When you work with certificates in Domino this is a must read!
There is howto material. tutorials. FAQs and also integrations to use CertMgr certificates outside Domino.

This would be a recommended read from beginning to the end.
It's complementing standard HCL documentation, which is mainly reference documentation like most vendor documentation.

The howto section also provides OpenSSL command line information how to retrieve and convert certificates.

https://opensource.hcltechsw.com/domino-cert-manager/

• Comments [1171]

Domino 14.5 EA2 Community Image patch for fixing December 2024 issue

HCL published a new Notes & Domino Windows DLL and Linux Lib for the problem occurred December 13 in the Early Access Program forum (EAP) --> https://hclsw.co/domino-14-5-eap-forum
This isn't a IF. Just a replacement of the main Notes/Domino DLL/Lib to address the problem also for the early access release.

The fix was on hold because the focus was on production and customer environments first.
Daily builds don't have support for IF/HF. Therefore there is a simple "sidepack" with just the affected files to replace.

You just stop your server or client and replace the files.
For a container image patching a container image isn't the right way.

But the container project supports custom add-ons. Usually this is intended to add software from 3rd parties, like a server task or extension manager.
The same process can also be used to patch Domino in the container at container build time.

Here is the official documentation --> https://opensource.hcltechsw.com/domino-container/concept_custom_addons/.

Let me share the steps I just took to patch my server.
I can't make the tar file available for download to just include it in our build options referenced via a HTTPS link.
But this process should be pretty straightforward and you can share this tar file for all internal servers you are building an image for.

Building the add-on to patch Domino 14.5 EA2

The first steps it to create a new directory to hold your file structure for the software you and to distribute
In the next step you create  sub directory for the Domino binary directory.

The -p options allows you to create the whole directory path a once:

mkdir -p /tmp/patch/domino-bin
cd /tmp/patch/domino-bin

Then you add the two files into that directory and package them up into a tar file.

tar -cvf /local/software/domino145ea2patchx.tar *

The directory structure look like this:

tree
.
└── domino-bin
 ├── libnotes.so
 └── libnotes.so.sym

Because the data needs to be verified to be added to the container, you have to provide the SHA256 of the tar file when getting it installed.
Linux provides the a command-line tool to generate the SHA256 checksum like this:

sha256sum /local/software/domino145ea2patch.tar
d088983544651940f71b8b7d9d942aaad84ecbe9fdbd24464178c858b9daff3b  /mnt/storagebox/software/domino145ea2patch.tar


Once you have the file in place and the checksum, you just add the custom add-on to your container build command-line.
The build script will take care for patching the binaries for your during the build process.

Custom add-ons are installed at the end of the build processes and therefore allow to patch also Domino binaries if needed.
The same process works for all kind of business partner applications.


-custom-addon=domino145ea2patch.tar#d088983544651940f71b8b7d9d942aaad84ecbe9fdbd24464178c858b9daff3b

• Comments [1181]