Pretty-Printing JSON in Notes Client and Domino

The Lotus Script class for reading and writing JSON is that easy. There are not many examples and some functionality is missing.

JSON can be either condensed without any new lines and indentation.
That's great when you use it for back-end processing or REST services.

Why is pretty printing important

But in some cases you need pretty formatted JSON. Specially when you want to maintain it manually and extend it. For example for Domino OTS JSON files :-)
When you use JSON based configuration pretty printed JSON is very helpful.

Condensed JSON is also difficult to check into Git.
Everything looks modified when it is a single line.

I am building multiple applications for different use cases and don't want external Libs like the famous JQ command line.
ChatGPT wasn't really helpful in this case, because, it always suggested code with Java classes we don't have in Notes/Domino.

Here is how to get it working out of the box with Java

So I asked the crowd in the OpenNTF Discord development channel.

After some discussions Jesse Gallagher came up with a solution based in Java libs Notes/Domino has included.
It's pretty simple once you know what to look for as you can see in this blog post --> https://frostillic.us/blog/posts/2024/7/26/pretty-printing-json-in-the-desktop-notes-client-and-domino.


AHA idea to enhance Lotus Script

This is still a work-around for me and it would be good to have JSON Pretty Printing in Lotus Script.
So I created an AHA idea you could vote for if you find it important --> https://domino-ideas.hcltechsw.com/ideas/DDXP-I-1287.

It would be small change in the back-end which would help to have it working on client platforms which have no Java support.

This is really a great example of the community working together and you will see an application soon leveraging this functionality.
I have already an agent to generate OTS JSON from a database and dump it to a file.

Thanks to Jesse again for jumping in and coming up with this solution.

-- Daniel

• Comments [0]

HCL Nomad server 1.0.12 IF1 shipped with same file name than 1.0.12

Nomad 1.0.12 has been replaced with a 1.0.2 IF1 version.
MHS has only the new version. The old version can't be downloaded any more.

But they left the file names the same. So you can't distinct the files by name once you downloaded them.

So you have to delete the old file and re-download it.

The same file name with a different content (resulting in a different hash and size), breaks automation.
For example it broke the Domino container build automation.

I first did not see what is going wrong.

The download has a different fileID. The old one is not available any more for download.

I have replaced the software.txt information to make it work again.
The version listed is 1.0.12 IF1, but the file name stays 1.0.12.

--- Update 2024.07.21 18:00 CEDT --

When you want to continue to build with your current version of the container build script and Domino download script, you need to invoke "domdownload" manually for a download.

The MHS software list now contains the file twice.

To address this issue I added a new feature to both scripts working hand in hand.
The software.txt contains the file name and the hash and we had to search for the fileID.

Now I am starting to replace the old IBM PartNumber field with the FileID step by step.
I have implemented the logic and pushed it to the develop and main branch of both projects:

• Domino Container build script 2.3.1
• Domino Download 1.0.3
   

I am going to add the fileIDs for newer software step by step.

So in future the build script is not depending on searching the MHS portal data and can instead use a direct download by fileID.

--- Update --

Here are the new details about both current downloads.

WebKit   :  HCL Nomad server 1.0.12 IF1 for Domino 14.0 Linux (includes Nomad web browser client)
Name     :  nomad-server-1.0.12-for-domino-14.0.0-linux.tgz
Version  :  1.0.12if1
Platform :  linux
Size     :  219141671
SHA256   :  0083ac39d1371fd83fa41d5372e161ca2e5d0bae27181d88ff95c3959a986ab2
ID       :  dZTS5fllw2dNECaHuhtR5
Modified :  2024-07-15T00:00:00.000Z


WebKit   :  HCL Nomad server 1.0.12 IF1 for Domino 12.0.2 Linux (includes Nomad web browser client)
Name     :  nomad-server-1.0.12-for-domino-12.0.2-linux.tgz
Version  :  1.0.12if1
Platform :  linux
Size     :  215493903
SHA256   :  84df2f4ba57568be481358259dc2d18542e0ddabfc97d3b898c1f2b5b5c28246
ID       :  zt2gdp54ImzgDHCTX26S6
Modified :  2024-07-15T00:00:00.000Z

• Comments [0]

Running Domino Windows container image on Windows 2022

Two years ago I have been looking into Domino in a Windows container already.
The main purpose was to understand the technology and if this makes sense to be used in general.

IMHO container technology is mainly helpful on Linux. Containers on Linux use core OS level functionality, which is part of the Linux kernel.
Only Linux makes sense for production use for me.
But a Windows container can be a great test environment for automation testing and other test use cases.
I revisited my container build on Windows this weekend and first updated it to Domino 14 and also updated all involved tooling like 7Zip.
In addition I looked into how I could leverage a Windows container image for testing.

Docker Desktop vs Windows 2024 container services

Windows containers can either run on Docker Desktop or on Windows 2024 server using the Container environment.

When running on a Windows server you have to choose one of the container environments.
There are basically two environments which are mainly used:

• Docker back-end with docker command-line
• Containerd with nerdctl command-line
  

I have been mainly focusing on the Docker based environment for now. But Containerd should work as well.

Helpful tooling for Windows containers

There is no Remote Desktop or other graphical interface available for a container.
None of the remote tools like VNC work. A Windows container is really headless.
So you need to use command-line functionality for all your work.

Editing files would work over a host mapping. Taskmgr can be replaced by "tasklist".
But there are a couple of commands which would be useful to have in a container.

Windows Git client brings a lot of useful tools including "vi" (vim)

In the Linux world most operations are performed on a command-line in a SSH session.
Most of the Linux tools are available on Windows.

The Git client comes with a Git bash and a lot of useful commands you can use in a Windows command prompt.
This includes openssl, vi, nano (if you really prefer it ..) and most other Linux tools.

The OpenSSL version is much more up to date than what is available for Window.
The curl command tool does come without the Windows schannel security integration

When I started looking into Domino in a Windows container I already built a small domino.cmd start script, which can be used to start and stop Domino.
Everyone knowing the Linux start script should recognize the following output ...

Is anyone looking into Windows containers in general?
What are you using it for? I would be interested to hear from the field.
How do you like the idea of using Windows containers for testing?
-- Daniel

docker exec -it domino cmd

Microsoft Windows [Version 10.0.20348.2529]
(c) Microsoft Corporation. All rights reserved.

C:\>domino stop

Stopping Domino for Windows
 ... waiting for shutdown to complete
 ... waiting 10 seconds
 ... waiting 20 seconds
Domino stopped (23 sec)
Domino for Windows shutdown completed

C:\>

• Comments [0]

New Nomad Server features -- ACME HTTP-01 challenge support & HTTP redirects via port 9080

There are two new features in the latest Nomad Server versions, introduced to Nomad Server without big notice.
I just got the question from a partner  why Nomad Server now binds port 9080 in addition to port 9443 and the internal communication port (only loop back).

The port might be used by other applications like the IBM Spectrum Protect (TDP) -- which was the problem in this customer case.
It turns out the TDP Java based restore GUI and does not work in combination without changing or disabling the port.

Nomad Server listens on port 9080 by default for HTTP redirects

The port is intended to redirect HTTP requests to HTTPS and is an additional functionality not directly needed by Nomad Server.
I would have wished this configuration would be disabled by default and only enabled if needed.
It would need additional configuration, because it is mainly intended to be used behind a reverse proxy.
The port can be changed or completely differently setting the port to 0.

For details check this documentation link:
https://help.hcltechsw.com/nomad/1.0_admin/config_options.html


Nomad Server can respond to Domino CertMgr ACME HTTP-01 challenges


In a configuration, where CertMgr runs behind a Nomad Server and no HTTP or redirected HTTP to HTTPS requests are possible, the Nomad server can handle ACME HTTP-01 challenges directly.
This configuration does not even need the HTTP task running on the Domino server. The Nomad Server reads the challenge response directly from certstore.nsf.


Testing the configuration

There is an example using the CertMgr diagnostic challenge described here --> http://opensource.hcltechsw.com/domino-cert-manager/troubleshooting_acme_challenges/
After adding the challenge to certstore.nsf manually the challenge can be checked with a curl command or any other tool (e.g. web-browser).


curl -L -v http://127.0.0.1:9080/.well-known/acme-challenge/DOMINO-CertMgr-DiagChallenge-HTTP01

*   Trying 127.0.0.1:9080...
* Connected to 127.0.0.1 (127.0.0.1) port 9080
> GET /.well-known/acme-challenge/DOMINO-CertMgr-DiagChallenge-HTTP01 HTTP/1.1
> Host: 127.0.0.1:9080
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Date: Fri, 05 Jul 2024 12:14:54 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5
< Transfer-Encoding: chunked
<

DOMINO-ACME-PROTOCOL-CHALLENGE-DATA-OK* Connection #0 to host 127.0.0.1 left intact

• Comments [0]

DACHNUG in Regensburg - Domino Track & Lab

This time the Domino focus group has sessions on Wednesday.
My session is the last session before lunch time and my focus will be Domino 14 including what is new in 14.0 FP1.

Therefore I am trying something new. I am publishing most of the slides before the session (but I keep some surprises for the presentation, explain details and answer questions).
You can also look at the presentation and prepare questions ahead of time and also stop by in the lab on Monday already.

The on-line lab is running the latest versions of Domino and companion products.
We look at  anything of what's new in any of the Domino track sessions and beyond.

On top of the lab, I just prepared a lab machine running Windows 11 leveraging Docker Desktop, the Windows Sandbox and other tools.
There is just too much new functionality for a one day track and a lot to discover.
The lab and the booth we are having is a good opportunity to take a look, discuss and add

I am looking forward to meet many of you in Regensburg this week.

-- Daniel

dachnug2024_domino14.pdf

Machine details

• Windows 11 Sandbox
• WSL running Ubuntu LTS 24.04
• Docker Desktop
• Nomad Web

This base allows to demo a lot of stuff and is great base for Windows machines.
Beside that I bring a Raspberry Pi running Ubuntu 24.04 to look into Nomad Web on an ARM machine.


What to explore

• The DNUG Lab with Domino 14.0 FP1 installed
• Windows Sandbox -- a great tool for testing
• Automatic Domino configuration on Windows with OTS (Domino + Nomad web single command install)
• Building & running the Domino Community image
• Running Domino in a container on Docker and Docker Desktop
• Grafana for Domino
• Domino 14.0 FP1 Auto Update
• Domino on Linux one command native installer & configuration
• Domino Download Script
• Many HCL & Nash!Com Open Source projects
  

• Comments [0]

Domino 14.0 is available on AWS marketplace

Domino 14.0 is available on AWS marketplace


The AMI comes wtih 100 licenses and is more intended for new customers.
But it is great news to have Domino available on the AWS market place.

https://aws.amazon.com/marketplace/pp/prodview-ptjcau2wktk2m

It is based on Redhat Enterprise Linux 9 and comes fully installed including the start menu we have built for the community image.
The installation is actually using the new HCL Domino container community project build files in combination with the Domino Start Script.

This means you could build your own image in the same way -- even with additional build options.
You also get the same menu driven setup and run-time functionality.
I still need to change the documentation for the One Touch Domino install on the start script documentation page.
It is now using logic from the container build and supports the Domino download script.

See details about the new One Touch Domino install script here -> https://blog.nashcom.de/nashcomblog.nsf/dx/introducing-the-domino-one-touch-installer-v2.htm

-- Daniel

• Comments [0]

Notes/Domino 12.0.2 FP4 shipped on all portals

Personally I am running most of my servers on 14.0 FP1. But it's important to also have 12.0.2 code stream updates.

12.0.2 FP4 is available on Flexnet (FNO), My HCLSoftware download portal (MHS).

The AutoUpdate software.jwt also has been updated including the AUT Catalog enabled client WebKits.

I also updated the develop branch of the container project with 12.0.2 FP4 fixpack data.
But I would still recommend using 14.0 FP1 for containers ;-)

Technote with release notes --> https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0113218

-- Daniel

• Comments [0]

A closer look at Linux Systemd

Long time ago Linux switched from init.d scripts to the new systemd service.
Today systemd is the standard for most Linux servers. But it is still a mystery for many admins.
Systemd is much more than a simple replacement. But this also makes it a bit more complex.

I had to look into systemd to stop, update and start a server with an application and ran into a couple of interesting details and useful commands.

The following information uses the Domino Start Script systemd service for reference.
For more details about systemd check the main man page of the project --> https://www.freedesktop.org/software/systemd/man/latest/systemctl.html

-- Daniel


Listing systemd scripts

The Domino Start script is a systemd script and uses a well defined service.

Service files can be listed with a single command:

systemctl cat domino.service

Description=HCL Domino Server (notes)
After=syslog.target network.target

[Service]
User=notes
RuntimeDirectory=notes
RuntimeDirectoryPreserve=yes
PIDFile=/run/notes/domino.pid
ExecStart=/opt/nashcom/startscript/rc_domino_script start
ExecStop=/opt/nashcom/startscript/rc_domino_script stop
Type=forking
LimitNOFILE€000
LimitNPROC€00
TasksMax€00

TimeoutSec=600
TimeoutStopSec=300
RemainAfterExit=no


Checking properties of a systemd service

The Domino Start script is a systemd script and uses a well defined service.
But only uses a fraction of the available configuration options.
To list the complete configuration including some status like the current main PID, there is a show command:

systemctl show domino.service

To query just one setting the --property option is useful

systemctl show domino.service --property PIDFile
systemctl show domino.service --property MainPID=24334



Systemd Services use Control Groups (cgroups)

When starting a service the main process and all child processes are assigned to the same cgroup as you can see from the following output.

Tip: systemd-cgls -u domino.service shows the Domino service only

All child processes are always part of the same cgroup. On shutdown systemd takes care of cleaning up processed, if not terminated correctly.
A future start of the service requires all previous processes to be terminated.


systemd-cgls

Control group /docker/85b1e6835ef25a9e71ceec49c04849102436f23a9030f017e95a694dfa3647e4:
-.slice
├─20158 bash
├─25761 systemd-cgls
├─25762 (pager)
├─init.scope (#601)
│ └─1 /sbin/init
└─system.slice (#614)
 ├─dbus-broker.service (#835)
 │ → user.invocation_id: 88585e258446434cac7d034f53a92d94
 │ → trusted.invocation_id: 88585e258446434cac7d034f53a92d94
 │ ├─10648 /usr/bin/dbus-broker-launch --scope system --audit
 │ └─10649 dbus-broker --log 4 --controller 9 --machine-id 8b3fdf8329b14614896d034706fba6d4 --max-bytes 536870912 --max-fds 4096 --max-matches 16384 --audit
 ├─domino.service (#2252)
 │ → user.invocation_id: c7e98a3ae0fe4856a082f1e7afb9f99f
 │ → trusted.invocation_id: c7e98a3ae0fe4856a082f1e7afb9f99f
 │ ├─24334 /bin/bash /opt/nashcom/startscript/rc_domino_script start
 │ ├─24391 /opt/hcl/domino/notes/latest/linux/server
 │ ├─24399 /opt/hcl/domino/notes/latest/linux/logasio NOTESLOGGER reserved
 │ ├─24407 /opt/hcl/domino/notes/latest/linux/event
 │ ├─24435 /opt/hcl/domino/notes/latest/linux/amgr -s
 │ ├─24436 /opt/hcl/domino/notes/latest/linux/adminp
 │ ├─24438 /opt/hcl/domino/notes/latest/linux/certmgr
 │ ├─25296 /opt/hcl/domino/notes/latest/linux/amgr -e 1
 └─systemd-journald.service (#692)
   → user.invocation_id: 43b26fdee15c48938fcbce780d931c8e
   → trusted.invocation_id: 43b26fdee15c48938fcbce780d931c8e
   └─19 /usr/lib/systemd/systemd-journald

• Comments [0]

Static linking C/C++ applications with OpenSSL

Usually applications are dynamically linked to OpenSSL.
There are pros and cons about dynamically linking OpenSSL.

Some applications dynamically link with OpenSSL and provide their own version of the OpenSSL run-time DLLs/shared libs.
This causes more issues than it solves, because you end up with multiple instances and depending on the lib path set, this can cause the wrong instance to be loaded.

Using the OS level OpenSSL share libs on the other side might leave you with an older OpenSSL version.
OpenSSL is an essential part of Linux. But not every distribution updates OpenSSL at the same time.

This means usually it is best providing OpenSSL with your application.
To avoid conflicts with other applications statically linking is a good idea.
This allows full control of the OpenSSL version and makes OpenSSL really glue into the application.

Note: Current Notes/Domino releases are statically link very current OpenSSL versions as well.


How to statically link

The two static link libs required for OpenSSL are

• libcrypto.a
• libssl.a

On Linux the is an easy way to build them for your platform and version (glibc dependency) is to start a container and build it on your own from the sources once.
The resulting static link libs can be then used for your applications.

In my build environment I have the sources of OpenSSL and switch to the right branch to build the latest release version once.
My build scripts for my applications automatically detect if static link libs are available and link OpenSSL statically (part of the makefile).


Run a new container based on Redhat UBI 9 to ensure a stable build environment (glibc 2.34).

docker run --rm -it registry.access.redhat.com/ubi9/ubi bash


Install git, compiler and make

dnf install git gcc perl make -y


Create a directory, switch to it and clone the OpenSSL GitHub project

mkdir -p /local/github
cd /local/github
git clone https://github.com/openssl/openssl.git
cd openssl


Switch to the right branch to get the latest stable version (in my case 3.3.0).

git checkout openssl-3.3.0


Finally configure and make OpenSSL

./config
make


Once the build completes, libcrypto.a and libssl.a are available in your current directory.


Clone nsh-tools and make nshcipher

cd ..
git clone https://github.com/nashcom/nsh-tools.git
cd /nsh-tools/nshmailx
make


Checkout the resulting binary

All newer NashCom tools have a --version command-line option to dump the version and OpenSSL information if the application uses OpenSSL.

/nshmailx --version

Nash!Com SMTP Mail Tool 0.9.8
Copyright 2024, Nash!Com, Daniel Nashed
OpenSSL 3.3.0 9 Apr 2024
(Build on: OpenSSL 3.3.0 9 Apr 2024)

You can see that the build and the run-time OpenSSL is the same.
Also the below dependencies don't show any OpenSSL dynamic lib.

ldd nshmailx
      linux-vdso.so.1 (0x00007fff771a0000)
      libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f26638f3000)
      libc.so.6 => /lib64/libc.so.6 (0x00007f26636ea000)
      /lib64/ld-linux-x86-64.so.2 (0x00007f266390c000)

• Comments [1]

Building applications in a build container

Specially when developing for different target versions of an OS or an application a build container can be very helpful.
But build containers are also really helpful in larger teams when everyone should use the exact same build environment.

The Domino container project supports adding the Notes/Domino C-API SDK to the container image.
In case of Domino libnotes.so is required. Therefore compiling requires at least an installed Domino server with the same or newer version than the SDK version.

I built a Domino 14.0 FP1 image including C-API 14.0 and tagged it hclcom/domino:build.

For this blog post I am using the simple test program in the container projects automation test directory --> https://github.com/HCL-TECH-SOFTWARE/domino-container/tree/main/testing
The directory is defined as a volume inside the container /build.

To ensure the binary can be written I am running the build using the root.
Alternative you can ensure that the 1000:1000 has write permissions in the directory to write the object and binary.

The following command starts a temporary build container and invokes make  (for better readability the line is split into separate lines).
make finds the makefile and uses the compiler inside the container to compile.

docker run -it --rm -w /build --entrypoint= -v $(pwd):/build -u 0
-e LOTUS=/opt/hcl/domino
-e Notes_ExecDirectory=/opt/hcl/domino/notes/latest/linux
-e LD_LIBRARY_PATH=/opt/hcl/domino/notes/latest/linux
-e INCLUDE=/opt/hcl/domino/notesapi/include
hclcom/domino:build make


Here is the break down of the command into it's components.

I have added the OpenSSL Development environment packages to the C-API development environment.
Most Domino add-on applications don't require OpenSSL. But adding this option I can use this container build environment also for native C/C++ applications which use OpenSSL.



run

Runs a new container based on the specified image

-it

The specified command is invoked interactively instead of detached (-d)

--rm

Once the command is executed and the container is stopped, remove it

-w /build

Switch to the build directory as the working directory to ensure the make command is executed in the right directory

--entrypoint=

The Domino container has an entrypoint script configured to start Domino automatically when the container is started.
Specifying an empty entrypoint, skips invoking the container image defined entrypoint.

-v $(pwd):/build

"Mount" the current directory into the container at /build.

-u 0

Run as root to ensure the output can be written. Can be omitted when user 1000:1000 has write permissions to this directory.

-e LOTUS=/opt/hcl/domino

Export the main LOTUS environment variable pointing to the server, which is used in makefiles.

-e Notes_ExecDirectory=/opt/hcl/domino/notes/latest/linux

Export the Domino binary directory location used in makefiles.

-e LD_LIBRARY_PATH=/opt/hcl/domino/notes/latest/linux

Set the LIB path to allow the linker and runtime to find the notes share objects

-e INCLUDE=/opt/hcl/domino/notesapi/include

Finally set the makefile INCLUDE directory to the C-API expected header file definition to find the header files.

hclcom/domino:build

Image name of the build container

make

finally the make command executed inside the container.

• Comments [0]