Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

alt

Daniel Nashed

 Domino 

Suppress Domino show commands from log.nsf

Daniel Nashed – 27 December 2025 19:10:18

By default "show commands" are logged into log.nsf.
Usually you either need it in console.log only.
In some cases you only need it as a response to a remote console command.


notes.ini Log_Console controls the log behavior of "show" commands.

By default everything is logged to log.nsf.


Setting Log_Console=1 ensures the result is only logged to console.log -- which is usually what you want.

Only in highly restricted environments you want to set the parameter to 2 to enforce every request to be logged into log.nsf.


This includes requests with a prefixed exclamation mark.

Without setting it to 2 the prefixed exclamation mark does not log any output.


That's specially helpful for automating commands like "show stat", "show trans" commands in troubleshooting scenarios with Lotus Script or C-API.


Log_Console


By default, all console command output (except output preceded by an exclamation point) is recorded in the server log.
With Log_Console, you can control console command logging by setting this variable to the following:


Value / Description


0    Console command logging disabled
1    Console command output logged, unless it's prefixed with an exclamation point (this is the default; it produces the same behavior as omitting Log_Console from Notes.ini)

2    Console command output logged, whether or not it's prefixed with an exclamation point

 RSPAMD 

RSPAMD integration for Domino?

Daniel Nashed – 16 December 2025 18:57:12
My mom just asked me about a e-mail she got because she did not order anything online.
I got those type of messages for a while because spammers got more clever the last years.


Even with DKIM and SPF enabled, gray listing and some other rules in my SpamGeek application I still get spam.

So here is the new before x-mas project. I looked what open source solutions would fit well for Domino. This might also be a good option to add Antivirus to my SMTP gateway.

https://rspamd.com/

RSPAMD looks great as a plug-in into my existing SpamGeek to provide additional score.

It also has interesting extensibility and comes with a couple of interesting data pools to build rules with.

There is also a new integration for OpenAI and Ollama - which I want to look into once I got it integrated.

It also has options it integrate ClamAV and other virus scanners. Which are also on my list to look into after it is integrated.


The first idea to integrate it was to use the milter interface which I have already implemented as a client lib in C earlier.
But RSPAMD has another interesting option. They have a REST API which provides more input and result options.


You can add metadata like IP, helo and sender etc via HTTP headers and get JSON with a detailed score back.

RSPAMD also speaks HTTP over UNIX socket in addition to TCP/IP.


So I implemented  C/C++ code to post the EML with the metadata, get the result back in JSON format and parse it.
Because the integration will later need to work with the MIME stream I read from a document step by step I also implemented HTTP chunked transfers to not have to read the full file into memory first to know the content length.


This looks very promising and I will do some testing with existing data first.

Porting it to Windows would make it more complicated because the API differences.

If I would, introducing OpenSSL as the network layer would make sense to avoid Windows vs. Linux TCP/IP operations.


Linux is just so much more straightforward and flexible. Using a UNIX socket makes it more elegant.


Is anyone using RSPAMD already?




Image:RSPAMD integration for Domino?


 LoadTest  k6 

Grafana k6 Load testing

Daniel Nashed – 15 December 2025 23:42:05

For simple load-testing I have been using Apache ab.
https://httpd.apache.org/docs/2.4/programs/ab.html

Apache ab quite OK for basic testing and very simple to use.
But for more advanced testing there are more modern frameworks like Grafana k6.

I gave it a quick try today and it looks like a great tool.
It can also write to Prometheus and use Grafana to display results.
Most of the Grafana Dashboards use InfuxDB but the more straightforward way would be to write to Prometheus.

But even without visualization the results are very useful.

I first tried to install it on Ubuntu. But it isn't included in the distribution and needs to add separate repositories.
With a container it is pretty simple to use. For the results you can just run Prometheus and Grafana in a local docker-compose stack.

I think this looks like a new interesting project for adding load testing to the container automation testing.
Is anyone using Grafana k6 already? Do you use Grafana Dashboard integration? Are you writing into Prometheus or InfluxDB?
What type of load testing are you running?


Image:Grafana k6 Load testing

Run via Docker

docker run --rm --network host -i -v $(pwd):/k6  -e "K6_PROMETHEUS_RW_SERVER_URL=http://127.0.0.1:9090/api/v1/write" grafana/k6 run /k6/login.js --summary-export=/k6/summary.json --out experimental-prometheus-rw

The command writes a log file, a summary and writes into Prometheus.


Simple load test example:

Here is a very simple script with a login and some requests dumping cookies and result headers.

import http from 'k6/http';

export const options = {
  vus: 10,
  iterations: 1000,
};

export default function () {
  if (__ITER === 0) {
    http.post(
      'https://domsetup.notes.lab/names.nsf?Login',
      {
        username: 'user',
        password: 'password',
      }
    );

    console.log(`Login Request (VU=${__VU}, ITER=${__ITER})`);

    const jar = http.cookieJar();
    const cookies = jar.cookiesForURL('https://domsetup.notes.lab');
    const res = http.get('https://domsetup.notes.lab/homepage.nsf');

    console.log(`VU=${__VU} ITER=${__ITER} Status=${res.status}`);
    console.log('Cookies: ' + JSON.stringify(cookies, null, 2));
    console.log('Headers: ' + JSON.stringify(res.headers, null, 2));
  }

  http.get('https://domsetup.notes.lab/homepage.nsf');
}


 ACME 

Actalis Free Italian ACME CA for single host certificates

Daniel Nashed – 14 December 2025 20:27:52

Today I was looking for an European based ACME provider since Buypass stopped their free services.

ChatGPT found exactly one provider: Actalis.

I took a quick look and registered an account.

They offer free and commercial services. The free trier supports the usual 90 days certificates.
In contrast to Let's Encrypt they have the following limitations for fee certificates. But that is perfectly OK for CertMgr when automating certificate requests.

  • 90 Days. for ACME certs (commercial certs are also available for 1 year)
  • Only single hosts (not just single domain)
  • No wild card certs (which would require DNS-01 challenges)
  • As usual ACME certs are only validating the DNS name. Not the organization
  • ACME requires registering a free account and to use EAB as part of the ACME configuration (see screen shot below, it's part of your free account)

Here is how it looks like. It's a bit more configuration work then for Let's Encrypt.
But I think it is good to have a second option for certificates -- specially when it is an European provider.

Take a look at the commercial options as well. They might be a good choice too.

You need to import the trusted root. ACME providers and other CAs don't send the root.
Only the certificate chain with the leaf and the intermediate certs. The root must be in your trust store (in this case certstore.nsf trusted roots).

How to find the root certificate?


Usually the root is referenced in the certificate:


AuthInfoURL:
http://cacert.actalis.it/certs/actalis-autroot

The certificate is in DER format. So you need to convert it to PEM.


curl -s http://cacert.actalis.it/certs/actalis-autroot | openssl x509 -inform DER


You find the certificate also in the Domino directory and cacert.pem in the Domino directory on client and server.


Image:Actalis Free Italian ACME CA for single host certificates

Website:


https://www.actalis.com

ACME Account URL:


https://acme-api.actalis.com/acme/directory

ACME error for multiple hosts
:


Here is the error you get when requesting multiple certs:

Error  Your account only grants single-domain 90-days DV certificates (400).

For single hosts this works perfectly fine.


Image:Actalis Free Italian ACME CA for single host certificates


Image:Actalis Free Italian ACME CA for single host certificates


 libssh 

SFTP with user and password with libssh2

Daniel Nashed – 13 December 2025 18:09:09

Here is the challenge


lftp
and curl compiled in the right way, support SFTP with user/password required for HCL ticket uploads.
As explained earlier UBI does have an easy path to support SFTP with user/password.

The mission is to have SFTP support without new dependencies


I took another look of what options we have without introducing a new dependency to the container image.

Here is the idea: libssh2 is easy to use and supports user/password (
https://github.com/libssh2/libssh2).

nshmailx is already a C/C++ application which statically links OpenSSL on Alpine. The same works also for libssh2.

Here is the new code I just added ->
https://github.com/nashcom/nshmailx/blob/main/sftp.cpp

I am still testing it. But here are the new options for SFTP Put ->
https://github.com/nashcom/nshmailx/
Now I am waiting for HCL support to explain how the flow works to get a user/password for upload along with the ticket number (
https://support.hcl-software.com/csm?id=kb_article&sysparm_article=KB0010064).
The idea is to integrate the upload into the Domino diagnostic script.

Update: Added a performance counter, checksum support for get/put. a larger buffer of 1 MB brings better performance. The output now looks like this:


nshmailx -sput 127.0.0.1 -user notes -password xxx -local Domino_14.5.1_Linux_English_EA1.tar -remote Domino_14.5.1_Linux_English_EA1.tar -sha
Connecting to 127.0.0.1 (127.0.0.1:22)
SSH host key [ssh-ed25519] SHA256:DLIJRAR37twu6SPWczkMUW/ZJuU1Wx4zRSEpb58rBTo
Upload successful: Domino_14.5.1_Linux_English_EA1.tar -> Domino_14.5.1_Linux_English_EA1.tar (size 1.1 GB, transfer: 271.7 MB/sec, SHA256: b950cdfecf0d87d89a6ca64c8e24b5c23e46724b364cb34e2a1d84a124a1742e)
 Ubuntu 

First look - Ubuntu LTS 26.04 (Resolute Raccoon)

Daniel Nashed – 12 December 2025 21:10:25

Ubuntu is the next major distribution shipping their next long term release.
It's scheduled be shipped end of April. An early release is already available and I took a quick look.


The most simple test is to use the container image, which is already available.

But there is also a full ISO.


Here is the basic data.
The kernel is bumped up and also glibc version is 2.42 like what we have seen at Debian 13.

Because of Java dependencies discussed already for Debian 13 the first Domino Java version that will work is currently in Domino 14.5.1 shipping in the same time frame.


Hostname      :      nsh-ubuntu26

Linux OS      :      Ubuntu Resolute Raccoon (development branch)

Linux Version :      26.04

Kernel        :      6.17.0-6-generic

GNU libc      :      2.42

Timezone      :      UTC +0000

Locale        :      en_US.UTF-8


OpenSSL and curl also has been update to a much never version.

What I see today is:


OpenSSL 3.5.3 16 Sep 2025 (Library: OpenSSL 3.5.3 16 Sep 2025)


curl 8.17.0 (x86_64-pc-linux-gnu) libcurl/8.17.0 OpenSSL/3.5.3 zlib/1.3.1 brotli/1.1.0 zstd/1.5.7 libidn2/2.3.8 libpsl/0.21.2 libssh2/1.11.1 nghttp2/1.64.0 librtmp/2.3 mit-krb5/1.21.3 OpenLDAP/2.6.10

Release-Date: 2025-11-05, security patched: 8.17.0-1ubuntu1

Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss

Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd


NGINX isn't the latest version. But on a good level.


nginx version: nginx/1.28.0 (Ubuntu)


The current version is 1.29.4.



This is a great update. And it aligns Ubuntu with the latest Debian 13 version.

Which is newer than the reference Redhat Enterprise 10.


There is one detail that needs to get fixed in the container image. "rev" used by the container scripts moved to a different packet.

A normal Ubuntu installation will not show a difference. But in a container we have to install the package. But this is already addressed in the container image build.


Here are the details. The mascot image isn't ready yet.



Download:

https://releases.ubuntu.com/26.04-snapshot1/

Draft Release Notes:

https://discourse.ubuntu.com/t/resolute-raccoon-release-notes/59221



Regression in 14.5 - SPR # ASHEDNU9C2 XPages doesn’t preserve sessionScope variables across operations

Daniel Nashed – 12 December 2025 15:20:08

If you are using XPages applications this issue might hit you. I was about to update the DNUG production server to 14.5 FP1 and checked dependencies with the team responsible for applications.
One of the applications uses XPages and the business partner came back wit this TN and SPR


https://support.hcl-software.com/csm?id=kb_article&sysparm_article=KB0127489

I opened a ticket and got the info that a fix is currently being tested. I will provide an update once a public fix is available.
If you have already updated and run into this issue, you should open a ticket to get the test fix.


If you are running XPages I would probably wait until this fix is confirmed before updating application servers.


-- Daniel




 SFTP  Linux 

Mission impossible: sftp client with user/password support on Redhat UBI

Daniel Nashed – 12 December 2025 12:07:23

Redhat UBI isn't really my favorite container image.
It is licence friendly for container images. But that's it with the benefits.


Here is my current challenge. And this really shows that Linux is not always the same.

I am trying to build a SFTP upload with user/password.

There are two easy to use ways. Plus one other way with sshpass.
But none those methods work on UBI.


In short:

  • curl is compiled without sftp support
  • lftp is not available in UBI
  • scp alone does only support sftp with keys not user/password
  • sshpass is not available on UBI

I don't want to switch the container default image to a different based image, but UBI isn't really what helps me to provide a solution to use SFTP to upload diagnostic data to HCL support automatically.

Here are two ways that would work with Ubuntu and other base images. But this isn't available on UBI.


curl -u
notes:notes -T /etc/hosts sftp://127.0.0.1/home/notes/hosts
lftp -u notes,notes s
ftp://127.0.0.1 -e "put /etc/hosts; bye"

I really don't want a solution that doesn't work for the standard base image used by HCL and the community project (UBI).
Probably the safest bet will be to write a small sftp client using libssh2 and statically link it on Alpine to make it available independent from the distribution used.
Similar to what I do with nshmailx.
This is less a technical challenge development a solution. It's more a bundling/tool chain issue.


Here is a quick summary created by ChatGTP to show the the challenge.
Probably I could write something in Java like what is included in Traveler. But this would also be an external program, which needs to be deployed.

For Ubuntu as the base image it would work with curl or better lftp out of the box.
Maybe someone has an additional idea.




Image:Mission impossible: sftp client with user/password support on Redhat UBI


First Look - Proxmox 9.1.2 OCI Container support

Daniel Nashed – 10 December 2025 20:22:22

This is brand new and before someone asks: This is not ready for prime time.
You are better of using a Docker host on with a VM or LXC container.


The OCI runtime is not ready for prime time. Specially not for a container like Domino.


First of all pulling from a registry does not support authentication from what it looks like.
So you can't pull from a public registry that needs authentication.


But it would work with a private registry.

I first pulled simple images like busybox to test -- which worked well.

For Domino I exported the image from a Docker host in the following way and copied it into /var/lib/vz/template/cache/


docker save hclcom/domino:latest -o domino-docker.tar

skopeo copy docker-archive:domino-docker.tar oci-archive:domino-oci.tar



Having it copied there let me select it via the LXC menu.

As a volume I selected a mount to /local. But the permissions have not been set right (the data directory was owned by root).

Also when you jump into the container you are always root first.


This isn't what we know from other container run-times and there have been a couple of other smaller issues I had to work-around.

The HCL out of the box image with Redhat UBI did not get an IP address via DHCP. But my Ubuntu image got an IP.


I was able to setup Domino and it is just running fine.


Still this isn't anything you want to use today for Domino.

It is a "don't try this at home" configuration until they improve it.


I just looked into it because someone asked and I wanted to give it a quick try.


On Proxmox the better way would be a LXC container or a VM with a Docker host.


If you want to take a look, use a simple container like busybox first.



Image:First Look - Proxmox 9.1.2 OCI Container support

Image:First Look - Proxmox 9.1.2 OCI Container support


Image:First Look - Proxmox 9.1.2 OCI Container support

Image:First Look - Proxmox 9.1.2 OCI Container support


Image:First Look - Proxmox 9.1.2 OCI Container support


Domino Diagnostics Collection on Linux

Daniel Nashed – 9 December 2025 22:52:08

Every customer situation can bring ideas for new functionality in the Domino Linux Start Script and container image.

I added a Domino Diagnostic menu earlier.

By default it did collect IBM_TECHNICAL_SUPPORT data from the last 24 hours.


This container almost all information. But could lead to collecting duplicate data collection.

Now the start script diagnostics will remember last diagnostics information which has been already sent.


The diagnostic compressed tar file will contain either data for the last 24 hours or since the last collection if this is a shorter time range.


In addition the tar file contains the translog control file. it will be copied with a time stamp in the name.
The idea is to have a very straightforward way to collect the right data for any kind of troubleshooting.
The diagnostic menu also got beefed up.


Image:Domino Diagnostics Collection on Linux

Links

    Archives


    • [HCL Domino]
    • [Domino on Linux]
    • [Nash!Com]
    • [Daniel Nashed]