Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

alt

Daniel Nashed

 Domino 

Fixed in 14.5 FP1 - ID Vault password authentication to a Domino 14.5.0 ID Vault server fails with error: Illegal Security function code

Daniel Nashed – 2 October 2025 22:17:05

Domino 14.5 introduced a new more secure ID Vault format. This format works only for Notes 14.0 and higher clients.
In mixed client environments with older versions this new format caused that the clients could not access their ID in ID Vault.


The format gets updated when the ID is updated in ID Vault.

Here is the updated technote which describes the details
https://support.hcl-software.com/csm?id=kb_article&sysparm_article=KB0122915

With Domino 14.5 FP1 the format has been reverted to the previous format.
But you might have Notes.IDs with the new format, which some user might not be able to read.


Here is how to recover those files:


1. You cloud reset the password which would write the Notes.ID in the previous format

2. You could remove the current Notes.ID from ID Vault and let the client sync it again to ID Vault.


To make sure you don't loose the ID, you could copy the existing document into another database as a backup.

If the user also has a newer client which understands the new format, any kind of vault operations which writes the ID should correct it.


If you did not update your ID Vault servers yet, make sure you update to Domino 14.5 FP1.

 macOS  Notes 

HCL Notes Mac Client 14.5 FP1 is still experiencing issues on macOS 26 with M2 and higher

Daniel Nashed – 1 October 2025 13:32:13

Adam Osborn blogged about this already and we had an offline discussion about it --> https://ab1osborne.blogspot.com/2025/09/the-hcl-notes-mac-client-145-sp1-is.html

The Technote in it's current state does not reflect the issue Adam and others including me as seeing even after updating to 14.5 FP1.
https://support.hcl-software.com/csm?id=kb_article&sysparm_article=KB0124211

The technote states that the first time the client is started does not come up.
What we are seeing on M2 and higher is that the client starts cleanly but when you try to shutdown the client, the client hangs in a way that you can't even shutdown macOS cleanly.

The technote is not yet updated, but HCL is aware of the situation. Adam and also Friedhelm Klein from TIMETOACT both have tickets open.
I updated my M4 mac mini to macOS 26.0.1 and I can reproduce the same hang.

Friedhelm checked with his whole team which CPUs are affected (they tested various combinations of M1-M4 including Max).
It turns out it starts with M2 and the M1 CPU is still working fine.

If you on a Mac with a M2 and higher, I would recommend to wait for the fix before updating to macOS Tahoe.

Thanks to Adam and Friedhelm for this great community work to narrow this down.

-- Daniel


 Domino  Tika 

AHA Idea DOMINO-I-3091 - Allow Domino to use an external Tika server instead of Tika started from Domino

Daniel Nashed – 28 September 2025 09:51:16
Given the current Apache Tika Vulnerability, I want to bring up my request to have support for an external Tika server as an option.

This isn't a request to change what is shipped with Domino nor a change for the default deployment out of the box deployment.
It is a request to add an option to allow companies to use their own Tika deployment (e.g. in a containerized Tika server using the official Apache Tika container image)

Today that's not possible because the Domino FT index back-end starts/stops Tika automatically as a process in a Domino JVM.

If you like the idea, here is the AHA link to vote --> https://domino-ideas.hcltechsw.com/ideas/DOMINO-I-3091


-- Daniel
 Domino  Windows  Linux 

My personal thoughts about keeping OS and Domino versions up-to-date

Daniel Nashed – 26 September 2025 09:55:39

Specially for Domino admin and mail servers upgrading to a later Domino version should not be complicated (unless you have backup or anti-virus dependencies specially on Windows).

I can understand that updates of application servers might take some time.


But staying on older Domino releases like 12.0.x always comes with a risk. No vendor can back port all add-on component new major versions to older code streams.

A dependencies like Java 11 for Tika 3.x where Domino 12.x still is running on Java 8 can't be simply solved if the vendor of the add-on software does not provide a fix of the add-on component of the code stream needed.

If like in the current case Apache Tika does not provide a patch for an older version (which is already out of support) and the new version depends on a newer Java version, the "upstream tool chain dependency" will hit us.


Running on older releases always comes with risk.HCL has quite a number of code streams in support. Plus offers extended support for older versions -- which gives you the chance to stay on an older version if you really have to.

But I would also see all of us admins being responsible to move to later code streams if possible.


GDPR


GDPR by the way implies that companies do what is feasible to protect customer data. This also means being up-to-date with OS and application releases.

I am not saying you have to be on the latest code stream. But today 14.0 with the latest fixpacks is a best practice form security point of view.



Domino Auto Update


Domino 14.0 introduces the first phase of Auto Update to automatically deploy software to install.

Domino 14.5 introduces true Auto Update functionality for Domino on Windows and Linux.

It has been introduced to allow customers to keep their environments patched to the latest version from security and functionality point of view.



Operating system choice and version


Domino is cross platform since the early days and there is a flexibility of choice also today (Windows/Linux/AIX/OS400).

I would personally like to get Domino on ARM supported -- but that's a different story.
But it is impressive how cross platform the Domino code base is on server side.


IMHO Domino on Linux is the better choice than running on Windows for multiple reasons:


  • Security
  • Performance
  • Resource usage
  • Operational costs
  • Maintenance (like patching)
  • Automation
  • Independent from one large operating system vendor

I have spent a lot of time making Domino on Linux easier to deploy manage than on Windows.

Today Domino on Linux is the best choice unless you have special requirements which requires Windows.


For admin and mail servers I would not see any reason to not move to Linux.


Still HCL Domino gives you the flexibility of platform choice and both Windows and Linux are valid platforms.

But if you have Linux available as a platform in your company, running Domino on Linux is really a recommended combination for the reasons stated above.


Domino on Linux will also be the base for the new Domino Workspace Sovereign Cloud offering for exactly those reasons.


HCL does also allow you to run Domino on the Linux Enterprise version of you choice ->
https://opensource.hcltechsw.com/domino-linux/
You are not limited to the two main enterprise vendors.

Some of the distributions like Ubuntu offers a free Linux distribution and commercial support if needed.
But also Redhat and SUSE have free options which are based on the same or very similar code bases if you prefer a free Linux distribution and get Linux level support from a local partner.


Ubuntu for example is a very interesting platform for two additional reasons


  • In place major release updates (which by the way also work on client/desktop side)
  • ZFS support (which would deserve a separate blog post)

---


I am not saying you have to migrate all your environments to Linux. But it would provide a lot of benefits and is even easier to manage today with the tooling the Domino Linux Start Script project offers -->
https://nashcom.github.io/domino-startscript/
This includes an installation and run-time menu driven environment which makes it easy to install, manage, operate and update Domino on Linux.


In case you are having a container strategy, the container project is providing an even easier path Domino on Linux -->
https://opensource.hcltechsw.com/domino-container/.


Keep your operating system update to the latest version


No matter which platform you choose, HCL always stays up to date with operating system support.

Each new enterprise version will be looked at as soon it is available.


One recent addition was Windows 2025 and Redhat Linux 10.
Support most times happens as soon the next fixpack of a code stream ships.


If you are still on Windows 2016 or earlier, it would be time to move up -- even you are on Windows extended support.


The next enterprise platform updating to a new major version will be SUSE updating to SUSE enterprise 16.0.

I took a quick look into the components and there are no surprises from Kernel nor glibc level.


In contrast to the just releases Debian 13, which comes with a very recent glibc which needs a newer OpenJDK version, which is planned to ship with Domino 14.5.1 next year.



My Conclusion


When it comes to security all components in your stack matter. Different application versions have different OS level version support and add-on requirements like Java.

Staying up to date is important for all components of your stack.



 Tika 

Security Bulletin: HCL Notes is affected by an XML External Entity (XXE) vulnerability in Apache Tika (CVE-2025-54988)

Daniel Nashed – 22 September 2025 08:52:18

That's the risk you take when adding external libs to your software: You can be hit by an upstream vulnerability.

In this case Tika has an issue with indexing PDF attachments.


https://support.hcl-software.com/csm?id=kb_article&sysparm_article=KB0124165

Notes and Domino both run Tika as an external stand-alone Java application where the client or server is talking to it over TCP/IP loopback.

The Tika server is started as the same user then the client/server. On server side this should be usually be a none privileged user.


So the risk for Notes/Domino might be not as high as the original CVE rating.

Still it makes sense to replace Tika if you are indexing databases with attachments in your environment.


There will be a fix provided by HCL. But you can also replace the Tika jar file manually today.

Note: Replacing the Tika server jar will only work with Notes/Domino 14.0+ because the current Tika release will only work with Java 11+.

Notes/Domino introduced Java 17 in version 14.0. Older versions are still running Java 8.


https://tika.apache.org/download.html

Container image


The Domino container project supports replacing Tika at build time.

I have removed previous Tika versions from the software list and added the latest 3.2.3 version this morning.

If you are running the container image, you can just use the -tika option
to rebuild your container image with the fixed version of Tika.


Update 26.09.2025:

I had a couple of discussions offline and there is some discussion in the commends of this blog post.

Christian Henserler raised an interesting fine tune option to only exclude PDF instead of adding a whitelist.
You can exclude certain type of attachments to avoid the risk.


notes.ini FT_INDEX_IGNORE_ATTACHMENT_TYPES=*.pdf

https://help.hcl-software.com/domino/14.5.0/admin/modifying_file_attachment_indexing.html

This is specially interested on clients where you can't update quickly and where usually no attachment indexing might be needed.
You can deploy the notes.ini via desktop policy to ensure if someone uses a local FT index with attachment filters, the component in Tika is not invoked.


On the server side my recommendation remains: I would updated to Domino 14.0 or better 14.5 and switch to the newer Tika binary.
Or wait for the upcoming 14.5 FP1 and 14.0 FP5 which both will contain the fixed Tika version.


-- Daniel

ID Vault password authentication to a Domino 14.5.0 ID Vault server fails with error: Illegal Security function code

Daniel Nashed – 27 August 2025 19:59:53

Domino 14.5 introduced an ID Vault incompatibility with older client versions.

The issue is described in the following technote.


https://support.hcl-software.com/csm?id=kb_article&sysparm_article=KB0122915

If you are looking into upgrading and have older clients, you should leave your ID Vault server on 14.0 for now.
Domino 14.5 FP1 is planned to be shipped soon  and will provide an option to keep the older algorithm.


Below is the current info from the technote. I will write up more information as soon we have 14.5 FP1 available.

But it would be expected to have a way to control if the new algorithm is used.


What is important: Once the ID is in the wrong format, an older client cannot access it any more.

The new format is written when the ID is updated.

If you have a current issue, you should open a support ticket.



Notes/Domino 14.5.0 released with a new, stronger, default algorithm setting used during the password authentication protocol to the ID Vault. This new default algorithm only has latent support in 14.0.x clients and servers.

Notes clients and Domino servers that are pre-14.0 do not have the latent support for the new 14.5.0 default algorithm setting used in the ID Vault password authentication transaction.

Therefore any ID Vault password authentication transaction consisting of one endpoint running pre-14.0 code and the other end 14.5.0 code could encounter the ID Vault password authentication failure "Illegal Security function code".


A LotusScript NotesHTTPRequest change in Domino 14.5 you should know

Daniel Nashed – 26 August 2025 21:08:49
There is a change in Domino 14.5, which is on by default.
In general this is a great move into the right direction. I would wish we would get this for clients too next.

But this would mean we need to push root certs to clients and they might need to have cross certs because users can update their local names.nsf documents.
The CA certs in Domino directory are the same by default. But if you have you own root certificates, you need to import it into the server's Domino directory.

https://help.hcl-software.com/domino/14.5.0/admin/wn_145_security_features.html#wn_145_security_features__section_dbj_sn4_cfc

Starting with Domino 14.5 server, the NotesHTTPRequest LotusScript class methods will, by default, load Trusted Roots from the Domino Directory when running on a Domino server. Previously, trusted root certificates were loaded from the file "cacerts.pem" in the Domino data directory. This new change doesn't impact NotesHTTPRequest LotusScript class behavior on the Notes client.


If you prefer to maintain the previous behavior and continue using "cacerts.pem", you can do so by setting the server-side Notes.ini parameter: NotesHTTPRequest_Use_CACerts=1


 Tools 

Very helpful Color Picker in Chrome

Daniel Nashed – 25 August 2025 22:05:44

Notes has a hidden color picker, which works on the whole screen with a trick -- when you are in a color field.
But it is hard to pick colors for example for small text which has a share.

There is a great tool, which works on Chrome without any add-ons.
It uses a magnifying function for picking colors and allows to disable he result in various formats.

https://pickcoloronline.com/


Image:Very helpful Color Picker in Chrome

A First Look at SUSE Leap/Enterprise 16 Shipping in Fall 2025

Daniel Nashed – 24 August 2025 09:07:19

SUSE is preparing to release Leap/Enterprise 16 in November 2025.

A release candidate is already available, although I haven’t seen a container image yet. Other vendors often release container images first to make testing easier.

SUSE skipped providing 15.7 for openSUSE Leap because version 16.0 is around the corner, although the enterprise container image of 15.7 is still available.


I gave Leap 16.0 a quick test in a Proxmox VM, and there were no major surprises in terms of compatibility—especially compared to Debian 13, which I tested recently.


SUSE Enterprise remains one of the two fully validated Linux distributions, along with Red Hat Enterprise, which makes this release candidate worth an early look.


Since SUSE Enterprise and openSUSE Leap share the same code base, I focused on Leap—it’s much easier to get hands-on access (Download:
https://get.opensuse.org/leap/16.0/)

Installation


The installation menu has been completely reworked.
It’s now cleaner, HTML-based, and even easier to navigate than in previous versions.


Package Versions


The included package versions look solid and align with what you’d expect from a modern distribution:

 
  • Kernel 6.12.0
  • glibc 2.40
  • OpenSSL 3.5.0 (8 Apr 2025)
  • curl 8.14.1

This should ensure compatibility with current workloads.


Docker Build/Run-Time Support


The official Docker convenience installer didn’t work.
However, SUSE 16.0 ships with an up-to-date Docker version: 28.3.2-ce.


My container setup script reported the failure of the convenience installer. Hopefully, Docker will update their scripts before SUSE 16.0 ships.
If not, I’ll add a work-around into my setup scripts.


Conclusion


From a Domino perspective, everything looks good so far.
The kernel and glibc versions are comparable to RHEL 10, so no surprises should be expected once SUSE 16 officially ships.


Personally, I’m moving toward Ubuntu for some use cases—mainly because of built-in ZFS support.
Still, openSUSE Leap/Enterprise 16 looks like it will be an excellent platform choice for an European based Enterprise Linux.


Domino 14.5 is not supported on brand new Debian 13 (Trixie)

Daniel Nashed – 23 August 2025 09:38:07

Debian 13 (Trixie) shipped
https://www.debian.org/News/2025/20250809

It comes with many packages updated to very recent versions.

One of the bigger changes is a very recent version of glibc 2.41.


There is a way to patch the affected binary. But it is not a good idea.

HCL will need to look into Debian 13 or glibc 2.41 support and you should not try to get this working on your own.


But because there people out there and post solutions and backgrounds anyhow.

Here are the details what is happening and why so that nobody needs to start speculating.


- Domino 14.5 ships JDK 21.0.6.

- Debian 13 ships JDK 21.0.8 which does not show the same error


But this glibc 2.41 might have other changes and this would need a complete new test in release.


Glibc is part of container images. This means running on a Debian 13 container host should still just work fine.


-- Daniel




Error message on Domino


Here is the error message you already see when invoking the java -version command and it makes the HTTP Task fail to load.


opt/hcl/domino/bin/java -version

JVMJ9VM011W Unable to load j9jit29: /opt/hcl/domino/notes/14050000/linux/jvm/lib/default/libj9jit29.so: cannot enable executable stack as shared object requires: Invalid argument

openjdk version "21.0.6" 2025-01-21 LTS

IBM Semeru Runtime Open Edition 21.0.6.0 (build 21.0.6+7-LTS)

Eclipse OpenJ9 VM 21.0.6.0 (build openj9-0.49.0, JRE 21 Linux amd64-64-Bit Compressed References 20250121_380 (JIT disabled, AOT disabled)

OpenJ9   - 3c3d179854

OMR      - e49875871

JCL      - e01368f00df based on jdk-21.0.6+7)



Debian 13 shipped Java 21 version


java -version

openjdk version "21.0.8" 2025-07-15

OpenJDK Runtime Environment (build 21.0.8+9-Debian-1)

OpenJDK 64-Bit Server VM (build 21.0.8+9-Debian-1, mixed mode, sharing)



Problem description and background


The problem turns out to be a problem with a change in glibc 2.41.


https://unix.stackexchange.com/questions/792460/dlopen-fails-after-debian-trixie-libc-transition-cannot-enable-executable-st

-- snip --

This is indeed a glibc change: in version 2.41, which migrated to Debian 13 on March 13 (replacing 2.40), dlopen no longer allows libraries requiring executable stacks to be loaded if the stack is not already executable:


dlopen and dlmopen no longer make the stack executable if a shared library requires it, either implicitly because of a missing GNU_STACK ELF header (and default ABI permission having the executable bit set) or explicitly because of the executable bit in GNU_STACK, and the stack is not already executable. Instead, loading such objects will fail.


-- snip --


There is a way to patch the affected binary. But this is not something I would advice you to use.



How to patch the affected binary


The shared lib is the only binary that needs to get patched -- from what it looks lilke.


I have just done a very basic test.


execstack -q /opt/hcl/domino/notes/14050000/linux/jvm/lib/default/libj9jit29.so



How to get the execstack on Debian 13


The execstack package has been removed and is not available on Debian 13

The following way works. But this isn't anything I would like to a Docker build pipeline nor would do in production without support.


apt-get update

apt-get install -y --no-install-recommends wget ca-certificates libelf1 libselinux1


wget -O /tmp/execstack.deb
https://archive.ubuntu.com/ubuntu/pool/universe/p/prelink/execstack_0.0.20130503-1.1_amd64.deb

dpkg -i /tmp/execstack.deb

execstack -V || true


Links

    Archives


    • [HCL Domino]
    • [Domino on Linux]
    • [Nash!Com]
    • [Daniel Nashed]