Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

alt

Daniel Nashed

HCL Nomad server 1.0.12 IF1 shipped with same file name than 1.0.12

Daniel Nashed – 21 July 2024 09:32:20


Nomad 1.0.12 has been replaced with a 1.0.2 IF1 version.

MHS has only the new version. The old version can't be downloaded any more.


But they left the file names the same. So you can't distinct the files by name once you downloaded them.


So you have to delete the old file and re-download it.


The same file name with a different content (resulting in a different hash and size), breaks automation.

For example it broke the Domino container build automation.


I first did not see what is going wrong.


The download has a different fileID. The old one is not available any more for download.


I have replaced the software.txt information to make it work again.
The version listed is 1.0.12 IF1, but the file name stays 1.0.12.


--- Update 2024.07.21 18:00 CEDT --

When you want to continue to build with your current version of the container build script and Domino download script, you need to invoke "domdownload" manually for a download.

The MHS software list now contains the file twice.

To address this issue I added a new feature to both scripts working hand in hand.
The software.txt contains the file name and the hash and we had to search for the fileID.

Now I am starting to replace the old IBM PartNumber field with the FileID step by step.
I have implemented the logic and pushed it to the develop and main branch of both projects:
  • Domino Container build script 2.3.1
  • Domino Download 1.0.3
     
I am going to add the fileIDs for newer software step by step.

So in future the build script is not depending on searching the MHS portal data and can instead use a direct download by fileID.

--- Update --

Here are the new details about both current downloads.

WebKit   :  HCL Nomad server 1.0.12 IF1 for Domino 14.0 Linux (includes Nomad web browser client)

Name     :  
nomad-server-1.0.12-for-domino-14.0.0-linux.tgz
Version  :  1.0.12if1

Platform :  linux

Size     :  219141671

SHA256   :  0083ac39d1371fd83fa41d5372e161ca2e5d0bae27181d88ff95c3959a986ab2

ID       :  dZTS5fllw2dNECaHuhtR5

Modified :  2024-07-15T00:00:00.000Z



WebKit   :  HCL Nomad server 1.0.12 IF1 for Domino 12.0.2 Linux (includes Nomad web browser client)

Name     :  
nomad-server-1.0.12-for-domino-12.0.2-linux.tgz
Version  :  1.0.12if1

Platform :  linux

Size     :  215493903

SHA256   :  84df2f4ba57568be481358259dc2d18542e0ddabfc97d3b898c1f2b5b5c28246

ID       :  zt2gdp54ImzgDHCTX26S6

Modified :  2024-07-15T00:00:00.000Z



Running Domino Windows container image on Windows 2022

Daniel Nashed – 7 July 2024 12:51:33

Two years ago I have been looking into Domino in a Windows container already.
The main purpose was to understand the technology and if this makes sense to be used in general.

IMHO container technology is mainly helpful on Linux. Containers on Linux use core OS level functionality, which is part of the Linux kernel.
Only Linux makes sense for production use for me.
But a Windows container can be a great test environment for automation testing and other test use cases.

I revisited my container build on Windows this weekend and first updated it to Domino 14 and also updated all involved tooling like 7Zip.
In addition I looked into how I could leverage a Windows container image for testing.


Docker Desktop vs Windows 2024 container services

Windows containers can either run on Docker Desktop or on Windows 2024 server using the Container environment.

When running on a Windows server you have to choose one of the container environments.
There are basically two environments which are mainly used:
  • Docker back-end with docker command-line
  • Containerd with nerdctl command-line
I have been mainly focusing on the Docker based environment for now. But Containerd should work as well.

Helpful tooling for Windows containers

There is no Remote Desktop or other graphical interface available for a container.
None of the remote tools like VNC work. A Windows container is really headless.

So you need to use command-line functionality for all your work.

Editing files would work over a host mapping. Taskmgr can be replaced by "tasklist".
But there are a couple of commands which would be useful to have in a container.

Windows Git client brings a lot of useful tools including "vi" (vim)

In the Linux world most operations are performed on a command-line in a SSH session.
Most of the Linux tools are available on Windows.

The Git client comes with a Git bash and a lot of useful commands you can use in a Windows command prompt.
This includes openssl, vi, nano (if you really prefer it ..) and most other Linux tools.

The OpenSSL version is much more up to date than what is available for Window.
The curl command tool does come without the Windows schannel security integration

When I started looking into Domino in a Windows container I already built a small domino.cmd start script, which can be used to start and stop Domino.
Everyone knowing the Linux start script should recognize the following output ...

Is anyone looking into Windows containers in general?
What are you using it for? I would be interested to hear from the field.
How do you like the idea of using Windows containers for testing?

-- Daniel

docker exec -it domino cmd

Microsoft Windows [Version 10.0.20348.2529]
(c) Microsoft Corporation. All rights reserved.

C:\>domino stop

Stopping Domino for Windows
 ... waiting for shutdown to complete
 ... waiting 10 seconds
 ... waiting 20 seconds
Domino stopped (23 sec)
Domino for Windows shutdown completed

C:\>

New Nomad Server features -- ACME HTTP-01 challenge support & HTTP redirects via port 9080

Daniel Nashed – 6 July 2024 09:34:50

There are two new features in the latest Nomad Server versions, introduced to Nomad Server without big notice.
I just got the question from a partner  why Nomad Server now binds port 9080 in addition to port 9443 and the internal communication port (only loop back).


The port might be used by other applications like the IBM Spectrum Protect (TDP) -- which was the problem in this customer case.
It turns out the TDP Java based restore GUI and does not work in combination without changing or disabling the port.


Nomad Server listens on port 9080 by default for HTTP redirects


The port is intended to redirect HTTP requests to HTTPS and is an additional functionality not directly needed by Nomad Server.

I would have wished this configuration would be disabled by default and only enabled if needed.
It would need additional configuration, because it is mainly intended to be used behind a reverse proxy.

The port can be changed or completely differently setting the port to 0.


For details check this documentation link:

https://help.hcltechsw.com/nomad/1.0_admin/config_options.html


Nomad Server can respond to Domino CertMgr ACME HTTP-01 challenges



In a configuration, where CertMgr runs behind a Nomad Server and no HTTP or redirected HTTP to HTTPS requests are possible, the Nomad server can handle ACME HTTP-01 challenges directly.

This configuration does not even need the HTTP task running on the Domino server. The Nomad Server reads the challenge response directly from certstore.nsf.



Testing the configuration


There is an example using the CertMgr diagnostic challenge described here -->
http://opensource.hcltechsw.com/domino-cert-manager/troubleshooting_acme_challenges/
After adding the challenge to certstore.nsf manually the challenge can be checked with a curl command or any other tool (e.g. web-browser).



curl -L -v
http://127.0.0.1:9080/.well-known/acme-challenge/DOMINO-CertMgr-DiagChallenge-HTTP01

*   Trying 127.0.0.1:9080...

* Connected to 127.0.0.1 (127.0.0.1) port 9080

> GET /.well-known/acme-challenge/DOMINO-CertMgr-DiagChallenge-HTTP01 HTTP/1.1

> Host: 127.0.0.1:9080

> User-Agent: curl/8.7.1

> Accept: */*

>

* Request completely sent off

< HTTP/1.1 200 OK

< X-Powered-By: Express

< Date: Fri, 05 Jul 2024 12:14:54 GMT

< Connection: keep-alive

< Keep-Alive: timeout=5

< Transfer-Encoding: chunked

<


DOMINO-ACME-PROTOCOL-CHALLENGE-DATA-OK* Connection #0 to host 127.0.0.1 left intact



DACHNUG in Regensburg - Domino Track & Lab

Daniel Nashed – 16 June 2024 15:57:19


This time the Domino focus group has sessions on Wednesday.

My session is the last session before lunch time and my focus will be Domino 14 including what is new in 14.0 FP1.

Therefore I am trying something new. I am publishing most of the slides before the session (but I keep some surprises for the presentation, explain details and answer questions).

You can also look at the presentation and prepare questions ahead of time and also stop by in the lab on Monday already.


The on-line lab is running the latest versions of Domino and companion products.

We look at  anything of what's new in any of the Domino track sessions and beyond.


On top of the lab, I just prepared a lab machine running Windows 11 leveraging Docker Desktop, the Windows Sandbox and other tools.

There is just too much new functionality for a one day track and a lot to discover.

The lab and the booth we are having is a good opportunity to take a look, discuss and add

I am looking forward to meet many of you in Regensburg this week.


-- Daniel


dachnug2024_domino14.pdf


Machine details
  • Windows 11 Sandbox
  • WSL running Ubuntu LTS 24.04
  • Docker Desktop
  • Nomad Web

This base allows to demo a lot of stuff and is great base for Windows machines.

Beside that I bring a Raspberry Pi running Ubuntu 24.04 to look into Nomad Web on an ARM machine.



What to explore
  • The DNUG Lab with Domino 14.0 FP1 installed
  • Windows Sandbox -- a great tool for testing
  • Automatic Domino configuration on Windows with OTS (Domino + Nomad web single command install)
  • Building & running the Domino Community image
  • Running Domino in a container on Docker and Docker Desktop
  • Grafana for Domino
  • Domino 14.0 FP1 Auto Update
  • Domino on Linux one command native installer & configuration
  • Domino Download Script
  • Many HCL & Nash!Com Open Source projects



Image:DACHNUG in Regensburg - Domino Track & Lab

 Domino  AWS 

Domino 14.0 is available on AWS marketplace

Daniel Nashed – 17 May 2024 07:20:45

Domino 14.0 is available on AWS marketplace



The AMI comes wtih 100 licenses and is more intended for new customers.
But it is great news to have Domino available on the AWS market place.


https://aws.amazon.com/marketplace/pp/prodview-ptjcau2wktk2m

It is based on Redhat Enterprise Linux 9 and comes fully installed including the start menu we have built for the community image.
The installation is actually using the new HCL Domino container community project build files in combination with the Domino Start Script.

This means you could build your own image in the same way -- even with additional build options.
You also get the same menu driven setup and run-time functionality.

I still need to change the documentation for the One Touch Domino install on the start script documentation page.
It is now using logic from the container build and supports the Domino download script.


See details about the new One Touch Domino install script here ->
https://blog.nashcom.de/nashcomblog.nsf/dx/introducing-the-domino-one-touch-installer-v2.htm

-- Daniel



Image:Domino 14.0 is available on AWS marketplace


Image:Domino 14.0 is available on AWS marketplaceImage:Domino 14.0 is available on AWS marketplace

 Notes  Domino 

Notes/Domino 12.0.2 FP4 shipped on all portals

Daniel Nashed – 16 May 2024 21:16:06

Personally I am running most of my servers on 14.0 FP1. But it's important to also have 12.0.2 code stream updates.

12.0.2 FP4 is available on Flexnet (FNO), My HCLSoftware download portal (MHS).


The AutoUpdate software.jwt also has been updated including the AUT Catalog enabled client WebKits.


I also updated the develop branch of the container project with 12.0.2 FP4 fixpack data.

But I would still recommend using 14.0 FP1 for containers ;-)


Technote with release notes -->
https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0113218

-- Daniel



Image:Notes/Domino 12.0.2 FP4 shipped on all portals
 Linux  Systemd 

A closer look at Linux Systemd

Daniel Nashed – 14 May 2024 07:27:58

Long time ago Linux switched from init.d scripts to the new systemd service.

Today systemd is the standard for most Linux servers. But it is still a mystery for many admins.
Systemd is much more than a simple replacement. But this also makes it a bit more complex.


I had to look into systemd to stop, update and start a server with an application and ran into a couple of interesting details and useful commands.


The following information uses the Domino Start Script systemd service for reference.

For more details about systemd check the main man page of the project --> https://www.freedesktop.org/software/systemd/man/latest/systemctl.html

-- Daniel



Listing systemd scripts


The Domino Start script is a systemd script and uses a well defined service.

Service files can be listed with a single command:


systemctl cat domino.service


Description=HCL Domino Server (notes)

After=syslog.target network.target


[Service]

User=notes

RuntimeDirectory=notes

RuntimeDirectoryPreserve=yes

PIDFile=/run/notes/domino.pid

ExecStart=/opt/nashcom/startscript/rc_domino_script start

ExecStop=/opt/nashcom/startscript/rc_domino_script stop

Type=forking

LimitNOFILE€000

LimitNPROC€00

TasksMax€00


TimeoutSec=600

TimeoutStopSec=300

RemainAfterExit=no



Checking properties of a systemd service


The Domino Start script is a systemd script and uses a well defined service.
But only uses a fraction of the available configuration options.
To list the complete configuration including some status like the current main PID, there is a show command:


systemctl show domino.service


To query just one setting the --property option is useful


systemctl show domino.service --property PIDFile

systemctl show domino.service --property MainPID=24334




Systemd Services use Control Groups (cgroups)


When starting a service the main process and all child processes are assigned to the same cgroup as you can see from the following output.


Tip:
systemd-cgls -u domino.service shows the Domino service only

All child processes are always part of the same cgroup. On shutdown systemd takes care of cleaning up processed, if not terminated correctly.
A future start of the service requires all previous processes to be terminated.



systemd-cgls


Control group /docker/85b1e6835ef25a9e71ceec49c04849102436f23a9030f017e95a694dfa3647e4:

-.slice

├─20158 bash

├─25761 systemd-cgls

├─25762 (pager)

├─init.scope (#601)

│ └─1 /sbin/init

└─system.slice (#614)

 ├─dbus-broker.service (#835)

 │ → user.invocation_id: 88585e258446434cac7d034f53a92d94

 │ → trusted.invocation_id: 88585e258446434cac7d034f53a92d94

 │ ├─10648 /usr/bin/dbus-broker-launch --scope system --audit

 │ └─10649 dbus-broker --log 4 --controller 9 --machine-id 8b3fdf8329b14614896d034706fba6d4 --max-bytes 536870912 --max-fds 4096 --max-matches 16384 --audit

 ├─domino.service (#2252)

 │ → user.invocation_id: c7e98a3ae0fe4856a082f1e7afb9f99f

 │ → trusted.invocation_id: c7e98a3ae0fe4856a082f1e7afb9f99f

 │ ├─24334 /bin/bash /opt/nashcom/startscript/rc_domino_script start

 │ ├─24391 /opt/hcl/domino/notes/latest/linux/server

 │ ├─24399 /opt/hcl/domino/notes/latest/linux/logasio NOTESLOGGER reserved

 │ ├─24407 /opt/hcl/domino/notes/latest/linux/event

 │ ├─24435 /opt/hcl/domino/notes/latest/linux/amgr -s

 │ ├─24436 /opt/hcl/domino/notes/latest/linux/adminp

 │ ├─24438 /opt/hcl/domino/notes/latest/linux/certmgr

 │ ├─25296 /opt/hcl/domino/notes/latest/linux/amgr -e 1

 └─systemd-journald.service (#692)

   → user.invocation_id: 43b26fdee15c48938fcbce780d931c8e

   → trusted.invocation_id: 43b26fdee15c48938fcbce780d931c8e

   └─19 /usr/lib/systemd/systemd-journald



Static linking C/C++ applications with OpenSSL

Daniel Nashed – 12 May 2024 09:15:31

Usually applications are dynamically linked to OpenSSL.

There are pros and cons about dynamically linking OpenSSL.

Some applications dynamically link with OpenSSL and provide their own version of the OpenSSL run-time DLLs/shared libs.

This causes more issues than it solves, because you end up with multiple instances and depending on the lib path set, this can cause the wrong instance to be loaded.

Using the OS level OpenSSL share libs on the other side might leave you with an older OpenSSL version.

OpenSSL is an essential part of Linux. But not every distribution updates OpenSSL at the same time.


This means usually it is best providing OpenSSL with your application.
To avoid conflicts with other applications statically linking is a good idea.
This allows full control of the OpenSSL version and makes OpenSSL really glue into the application.

Note: Current Notes/Domino releases are statically link very current OpenSSL versions as well.



How to statically link


The two static link libs required for OpenSSL are

  • libcrypto.a
  • libssl.a

On Linux the is an easy way to build them for your platform and version (glibc dependency) is to start a container and build it on your own from the sources once.

The resulting static link libs can be then used for your applications.


In my build environment I have the sources of OpenSSL and switch to the right branch to build the latest release version once.

My build scripts for my applications automatically detect if static link libs are available and link OpenSSL statically (part of the makefile).



Run a new container based on Redhat UBI 9 to ensure a stable build environment (glibc 2.34).


docker run --rm -it registry.access.redhat.com/ubi9/ubi bash



Install git, compiler and make


dnf install git gcc perl make -y



Create a directory, switch to it and clone the OpenSSL GitHub project


mkdir -p /local/github

cd /local/github

git clone
https://github.com/openssl/openssl.git
cd openssl



Switch to the right branch to get the latest stable version (in my case 3.3.0).


git checkout openssl-3.3.0



Finally configure and make OpenSSL


./config

make



Once the build completes, libcrypto.a and libssl.a are available in your current directory.



Clone nsh-tools and make nshcipher


cd ..

git clone
https://github.com/nashcom/nsh-tools.git
cd /nsh-tools/nshmailx

make



Checkout the resulting binary


All newer NashCom tools have a --version command-line option to dump the version and OpenSSL information if the application uses OpenSSL.


/nshmailx --version


Nash!Com SMTP Mail Tool 0.9.8

Copyright 2024, Nash!Com, Daniel Nashed

OpenSSL 3.3.0 9 Apr 2024

(Build on: OpenSSL 3.3.0 9 Apr 2024)


You can see that the build and the run-time OpenSSL is the same.
Also the below dependencies don't show any OpenSSL dynamic lib.


ldd nshmailx

      linux-vdso.so.1 (0x00007fff771a0000)

      libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f26638f3000)

      libc.so.6 => /lib64/libc.so.6 (0x00007f26636ea000)

      /lib64/ld-linux-x86-64.so.2 (0x00007f266390c000)

Building applications in a build container

Daniel Nashed – 12 May 2024 08:17:00


Specially when developing for different target versions of an OS or an application a build container can be very helpful.
But build containers are also really helpful in larger teams when everyone should use the exact same build environment.


The Domino container project supports adding the Notes/Domino C-API SDK to the container image.
In case of Domino libnotes.so is required. Therefore compiling requires at least an installed Domino server with the same or newer version than the SDK version.


I built a Domino 14.0 FP1 image including C-API 14.0 and tagged it hclcom/domino:build.


For this blog post I am using the simple test program in the container projects automation test directory -->
https://github.com/HCL-TECH-SOFTWARE/domino-container/tree/main/testing
The directory is defined as a volume inside the container /build.

To ensure the binary can be written I am running the build using the root.
Alternative you can ensure that the 1000:1000 has write permissions in the directory to write the object and binary.


The following command starts a temporary build container and invokes make  (for better readability the line is split into separate lines).
make
finds the makefile and uses the compiler inside the container to compile
.

docker run -it --rm -w /build --entrypoint= -v $(pwd):/build -u 0
-e LOTUS=/opt/hcl/domino
-e Notes_ExecDirectory=/opt/hcl/domino/notes/latest/linux
-e LD_LIBRARY_PATH=/opt/hcl/domino/notes/latest/linux
-e INCLUDE=/opt/hcl/domino/notesapi/include
hclcom/domino:build make



Here is the break down of the command into it's components.


I have added the OpenSSL Development environment packages to the C-API development environment.
Most Domino add-on applications don't require OpenSSL. But adding this option I can use this container build environment also for native C/C++ applications which use OpenSSL.




run


Runs a new container based on the specified image


-it


The specified command is invoked interactively instead of detached (-d)


--rm


Once the command is executed and the container is stopped, remove it


-w /build


Switch to the build directory as the working directory to ensure the make command is executed in the right directory


--entrypoint=


The Domino container has an entrypoint script configured to start Domino automatically when the container is started.
Specifying an empty entrypoint, skips invoking the container image defined entrypoint.


-v $(pwd):/build


"Mount" the current directory into the container at /build.


-u 0


Run as root to ensure the output can be written. Can be omitted when user 1000:1000 has write permissions to this directory.


-e LOTUS=/opt/hcl/domino


Export the main LOTUS environment variable pointing to the server, which is used in makefiles.


-e Notes_ExecDirectory=/opt/hcl/domino/notes/latest/linux


Export the Domino binary directory location used in makefiles.


-e LD_LIBRARY_PATH=/opt/hcl/domino/notes/latest/linux


Set the LIB path to allow the linker and runtime to find the notes share objects


-e INCLUDE=/opt/hcl/domino/notesapi/include


Finally set the makefile INCLUDE directory to the C-API expected header file definition to find the header files.


hclcom/domino:build


Image name of the build container


make


finally the make command executed inside the container.


Using Containers for software testing

Daniel Nashed – 5 May 2024 15:27:50

Using Containers for software testing

Containers are great to deploy software and also to test container images before deploying them.
But sometimes you want to just install software manually inside a "sandbox".
On Windows the Sandbox is a great way to test. On Linux you can just run a container.

In my current test case I am using a Redhat UBI Init image, which provides systemd support.
Usually systemd isn't what would be the right choice for a container image.
Containers usually run a single application/process and a entrypoint script is the right choice.

But for testing a container with systemd is an interesting option.
It requires a privileged container to support systemd -- which should be avoided for production environments.
For testing this is perfectly OK.


Run a new container

docker run -d --name init --privileged registry.access.redhat.com/ubi9-init:latest



Jump into the container

docker exec -it init bash


In my case I amend to end testing Domino installations.
I have updated the Domino native install script to work in this environment.
But there are many different use cases.

curl -sL https://raw.githubusercontent.com/nashcom/domino-startscript/develop/install_domino.sh | bash -


Links

    Archives


    • [HCL Domino]
    • [Domino on Linux]
    • [Nash!Com]
    • [Daniel Nashed]