Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

Daniel Nashed

Tags

IterimFix - What is it and how to install

Daniel Nashed – 20 December 2024 19:15:27

The following write up was on my todo list for a while.
But because of the current Interim Fix you should apply, I want to provide some details which might help to give a deeper understanding and to install fixes.

The new Domino 14 AutoUpdate feature knows about all the details of Releass/FP/IF/HF installers.
Join the January OpenNTF webinar to learn about what AutoUpdate can do to help you with automatically updates starting with Domino 14.5.

Most important take away

The most important take away from this write-up is that a previous hotfix (HF) must be uninstalled with the same HF installer it has been installed.

An uninstall is always required before installing a new FP or IF/HF.
Therefore you should leave the hotfix installer on the server for uninstalling it later.

When you run the HF installer for a second time, the HF is uninstalled.
There is no explicit switch for install or uninstall.

Domino 14.5 AutoInstall distributes the HF installer for uninstall automatically.

Domino Release

A Domino release is usually shipped once per year (like in the last releases) or 18 month from what the current schedule looks like for Domino 14.5.
It contains new major features and new versions of software components like a new Java version which needs significant work and additional testing.

A release contains new template versions for system and application databases. This also includes translated templates.
Releases are numbers according to a semantic versioning schema.

For example: 12.0.1, 12.0.2, 14.0, 14.5

Each of release is a separate code stream as you can see for builds available in early access code drops.

For example: V1450_12042024

Releases use the InstallAnywhere installer and can be installed on any existing release.
The installer takes care of uninstalling previous versions independent of which FP/IF/HF is installed..

Domino Fixpack (FP)

"A fixpack is collection of low-risk, high-impact fixes to help customers safely avoid known issues".

It might also contain smaller feature enhancements which did not make it into the release.
But a fixpack is mainly a "maintenance pack".

The current schedule based on what we have seen for 14.0 is around 4 month.

Fixpacks contain also fixes for example for iNotes Forms9.nsf. (Note Domino 14.5 will replace iNotes with Verse which is included in Domino already since Domino 14).
No translated templates are part of a fixpack. But for critical errors templates are distributed separately. For example in a technote article.

The naming of a Fixpack is "FP" with with a consecutive number.

For example: 14.0FP3

The next Fixpack for a code stream contains all fixes of the previous Fixpack.
Fixes in a Fixpack are also expected to be in the next major release.

Fixpacks use an HCL own installer which can detect Releases, Fixpacks and Hotfixes.
The Fixpack installer can be installed on the matching Release or a previous Fixpack.

The installer is able to uninstall an existing Fixpack before installing the new Fixpack.
But the installer cannot uninstall a IF or HF.

Domino Hotfix (HF)

A hotfix is a critical fix usually for a single SPR for a single customer.

In some special cases a customer needs multiple SPRs in a "combo-hotfix" distributed as a single HF.
This changes are mostly added to the next Releases and Fixpacks to be available for all customers.

Two different Hotfixes might contain completely different fixes.
Those fixes are exclusively distributed by HCL support to customers.

Hotfixes should be only installed on the server which needs this specific fix and should not be shared between customers and partners.
As soon the fixes are available on a Fixpack it is recommended to update to the Fixpack.

The naming schema is "HF" and a consecutive hotfix number per Release and Fixpack combination.

Example: 14.0FP3HF8

The number of the hotfix cannot be used to find out if the fix is related to another fix.
Information about the fix are provided by HCL when making the Hotfix available.

Hotfixes use a different HCL custom HCL installer which can detect Releases, Fixpacks and Hotfixes.
The installer can only update the version which exactly matches the Release/Fixpack combination.
It might be targeted to a Release or a Fixpack.

The same HF installer (web-kit) is required to uninstall an installed HF.
To replace a HF you have to use this installer or install a new release, which replaces all binaries.

Note: Fixes for Hotfix are not automatically added to the next Fixpack or Release.
Customers should check with HCL support if the fix is still required for the next Release or Fixpack.

Domino Interim Fix (IF)

A Domino Interim Fix can contain one or multiple fixes. An "IF" is a naming convention mapping to different HFs per platform.
IFs are created when important fixes have to be distributed to customers outside the normal Fixpack schedule available on the MHS portal (https://my.hcltechsw.com/).

The next IF contains all fixes from the previous IF for the same Release/Fixpack combination.
Fixes in an IF are expected to be in the next fixpack and release.

The naming schema is "IF" and a consecutive interim fix number per Release and Fixpack combination.

Example: 14.0FP3IF2

IFs are just a naming convention. The names are a bit different it is basically the same type of installer.
From the install web kit file and the installed software you cannot tell what the IF name is.

---

Domino 14 Autoupdate knows about IF and HF combinations.

- Software documents contain HF and IF labels
- Distributing and installing software is based on the IF labeling
- Autoupdate selects the right HF per platform when distributing or installing software
- The target versions and the server document contain HF and IF labels

Domino Container Image

- The Domino Container Image also knows about the dependencies.
- But because container images are always installed cleanly from scratch, there is no uninstall needed.
- The container image always provides the latest version. The installation menu is preset to the latest version automatically.

Comments [0]

Domino

Update on the critical Domino mail issue -- Fixes are available for the most recent releases

Daniel Nashed – 14 December 2024 18:25:08

The root cause of the issue has been found yesterday. The fix is build step by step for the different versions and made available as soon tested.
For up to date information please look into the following technote, which is updated

https://support.hcl-software.com/csm?id=kb_article&sysparm_article=KB0118192

The first software packages are available on MHS.
I have updated the Domino Community Container project with 14.0 FP3 IF1 already.
The autoupdate and auto notify date should arrive shortly as I heard.

My servers are already updated (mostly container based).-

Not every mail configuration is directly affected (for example if you just have one mail.box, not multiple servers in your mail routing transfer chain and use no server mail rules).
But in any case you should apply the fix, because there could be other side effects.

For an information about the technical background, refer to the support statement.

Update: 15.12.2024 0:42:

More and more packages are available for more platforms and older versions.
You should check https://my.hcltechsw.com/ and also the updated technote for updates.

There are going to me more platforms. OS400 will need some time and is the most complex platform.
If you are on Domino 9.01 or 10.0.1 you will need to be under extended support.

Update: 16.12.2024 0:35:

There have been a couple of questions in my blog.
I can't comment on anything that is not officially confirmed by HCL.
Please refer to the technote which is at least daily updated.

You might also find additional information on Thomas Hampel's blog (senior product manager Domino)

https://blog.thomashampel.com/

-- Daniel
Comments Disabled

Domino

CRITICAL ALERT: Don’t restart your Domino servers over the weekend, there is a critical issue which is been worked on with highest priority

Daniel Nashed – 13 December 2024 22:49:02

Below is the current content of the technote. The problem came up today and the root cause is identified.
Please do exactly what this technote say and don't try to fix it. A fix is on it's way.

https://support.hcl-software.com/csm?id=kb_article&sysparm_article=KB0118192

What is actually failing is for example that the MaxHops setting is set to 0 due to this problem. Also the number of mailboxes will be set to 1 if it was set higher.
If you have only one Domino server in the chain for mail-routine and just one mail.box, you might be lucky for the mail routing issue. But there might be other side effects of the underlaying problem.

Be prepared to update to the latest FP or the current stream you are working on.
If you are using the community container image, I will make sure the fix is added as soon it is available.

Domino 14.0 AutoUpdate supports distributing FPs and IFs already. If you on 14.0 you will be able to leverage AutoUpdate to distribute the IF.
If you need help talk to your business partner. This will be a mandatory fix you can't skip.

You should already prepare for the update.

---

CRITICAL ALERT: Mail not routing after Domino restarts beginning 13 December 24

Applies to

All versions of HCL Domino (v9.0.x, v10.0.x, 11.0.x, 12.0x, 14.0.x)

Issue

The HCL Domino Development team has identified an issue which will affect *ALL Domino server versions* as of TODAY, December 13, 2024.

If you restart your server, a router error will occur which results in delivery failures due to a routing loop. Mail rules will also start failing. This is a date/time issue in our code, and we will provide a hotfix as soon as possible.
Resolution

To avoid the problem **please do NOT restart your servers.** Please do not attempt to fix the issue by changing operating system time into the past. If you restarted Domino and are experiencing issues please contact support to open a case.
We will continue to provide updates on this page.

More Information

This has been reported to development via SPR BRISDBWRXJ.

Comments Disabled

Lotus Script

Lotus Script - Check if a file or folder exists

Daniel Nashed – 13 December 2024 11:23:34

There isn't a direct function in Lotus Script to check if files exist.
Most of us have written a function like this multiple times as a helper.

I am working on an application which will be available as open source soon and I needed one again...
Here is a version of a check function, which should cover all cases and might be useful for you too.
It doesn't handle hidden files -- I know. But I didn't want to add that logic.

-- Daniel

Function FileFolderExists (FileName As String) As Long

'Returns
' 2 = Directory exists
' 1 = File exists
' 0 = File/Directory does not exist
' -1 = Directory for file does not exist
' -2 = Other error

Dim result As String

FileFolderExists = -2
On Error GoTo ErrorHandler

result = Dir$(FileName, 16)

If (result = "") Then
FileFolderExists = 0
Exit Function
End If

result = Dir$(FileName, 0)

If (result = "") Then
FileFolderExists = 2
Else
FileFolderExists = 1
End If

Exit Function

ErrorHandler:

If (76 = Err) Then
FileFolderExists = -1
End If

Exit Function

End Function

Comments [1]

Notes

Notes Color fields, color view columns, color profiles and how they work

Daniel Nashed – 8 December 2024 22:01:24

Color support for views and folders in Notes is pretty cool. The functionality has been around since Notes 6.0 and is unchanged since then.
Someone really nailed it. But it it can be a bit tricky to implement.

The color profile support has been introduced for example for colorizing senders.
If you never used colorizing of senders, you should have a look into the calendar profile.
When trying to implement functionality it is always a good idea to look into what HCL or business partners have already developed and checkout how the functionality is implemented.

Image:Notes Color fields, color view columns, color profiles and how they work

Image:Notes Color fields, color view columns, color profiles and how they work

There are two different ways to use colors

The simple an straightforward way to use colors is to have a static color definition in a view or folder.
There is a setting in the column options to display the text in the column and all following columns with the specified color.

You can set foreground and background colors. The color settings are valid until the next color column. This means you can define multiple colors in one row Or just colorize some of the columns.
Usually the color columns are hidden. In the designer they still show up as listed below.
Colors are defined as three numbers (not text) in a list. You can combine foreground and background colors resulting in six number list elements.
You can convert text to numbers are shown below.

Image:Notes Color fields, color view columns, color profiles and how they work

Using color profiles

The more complex way to use colors is very flexible but a bit tricky.
For each color column you can specify a formula field in a profile.The name of the field is the name of the programmable name of the field.
The result of the formula is used as a color. But in this case the values are text -- very similar to how HTML color definitions look like. Just with two extra digits as a prefix. As you can see in the screen prints.

Here is how it looks like:

Image:Notes Color fields, color view columns, color profiles and how they work

Color definitions

You can use color fields which are stored as text. With eight digits.
Those values can be used in formulas. And those formulas are evaluated during view/folder index update.

Here is an example formula using it as a background color as shown in the screen shot above.

Formula Black:="00000000"; Red:="00FF0000"; Blue:="000000FF00"; Green:="000000FF"; @If( Subject ="Red";Red; Subject ="Blue";Blue; Subject ="Green";Green; Color) : Black;

The color field has some useful functionality

First of all it is notable that Notes has a color field. The color field can be used in multiple ways.
The main use case is to provide a color picker for an user -- like in the calendar profile.

Image:Notes Color fields, color view columns, color profiles and how they work

There are two ways to select colors. The first and most visible way is the "Notes" way.
But there is a second tab to select the RGB values separately.

Color Picker

The color picker is well hidden and shows only up when you hover over the lower part in the color scratch pad.
You can pick colors with a color picker.

Color Picker Trick

When keeping the color tool tip pressed you can pick colors from you whole screen :-)

Image:Notes Color fields, color view columns, color profiles and how they work

Notes Documentation "Allowing users to set colors in a view"

Here is the help topic describing the functionality.

https://help.hcl-software.com/dom_designer/14.0.0/basic/H_ALLOWING_USERS_TO_SET_COLORS_IN_A_VIEW_STEPS.html

Comments [1]

DKIM

Domino DKIM public export has a trailing NUL char and DNS TXT entries should be in quotes

Daniel Nashed – 7 December 2024 11:42:05

While ago I wrote a small helper application to automate DKIM creation.
It performs all the steps needed from key creation to export and can also use the formula based CertMgr DNS API integrations (I rewrote the logic in Lotus Script, CertMgr has a C implementation).

When looking into my DNS TXT records on Cloudflare I noticed the missing quotes.
Cloudflare DNS detects they are missing and does provide them in the right way.
But they also show in their UI that the entries are not quoted.

Here is an entry in the right format: "v=DKIM1; k=ed25519; p=8XEdKsnPk9HkHWKquLdfx5J7RqKh+4heMht3AjrmVU0=;" keymgmt export DKIM DNS csi-domino.com ed20241206 csi_domino_com_ed20241206.txt

When adding the quotes in my application for generating the right DNS TXT entries I ran into an issue where Lotus Script was not able to add a quote at the end of the string.
Looking at the file showed the NUL char, which I have seen before on Linux when working with a customer.

Now this all makes sense to me and I understand the ^@ I have seen earlier at a customer.

My application first removes the NUL char when reading the Notes Stream and adds the quotes.

Here is how it looks like in Notepad and vi with the NUL char

Notepad++

Image:Domino DKIM public export has a trailing NUL char and DNS TXT entries should be in quotes

Image:Domino DKIM public export has a trailing NUL char and DNS TXT entries should be in quotes

SPR #DNADDBPM94 DKIM exported TXT file contains NUL

There is a new SPR created yesterday, which will remove the trailing NUL.
In the meantime you should carefully look at the DNS TXT records. Having quotes around it in the TXT file is a convenience feature.

DNS TXT Records should be in quotes

The TXT file content is also not quoted. This is mainly because nobody really can say what certain DNS admin interfaces expect.
But the result should be always quoted.

Quoting gets more difficult when you have to manually split the entry because a DNS TXT record is limited to 255 bytes.

With a Ed25519 you are pretty safe and also a RSA 1024 key is OK. The fun begins with a RSA 2048 key.

I would anyhow recommend a Ed25519 and RSA 1024 in parallel. There isn't really a need for a RSA 2048 and you can risk software on the other side not getting the large DNS TXT record read.

My DKIM / DMARC setup database is still under test. Only a couple of people use it so far.
I just updated it to get the quotes properly read and written. And also the DNS TXT API I am borrowing from CertMgr now properly masks the quotes in JSON data.

Conclusion:

Usually customers have no automation for the process of creating those DNS TXT records and a DNS admin should know about adding the quotes.
It hit me because I took the file 1:1 and added the text to a DNS record using a provider API.

The NUL char mostly isn't a problem because when you copy & paste in an editor it should most times just go away.

Here are the key takeaways

- Be aware of the extra NUL char
- Add quotes around DNS TXT records
- You might need to split the data manually depending on your provider if the string exceeds 255 bytes

Comments [0]

NVIDIA

Monitoring NVIDIA GPUs

Daniel Nashed – 1 December 2024 09:23:02

NVIDIA tools

NIVIDA has great tools and toolkits. My focus right now is mostly Linux. But some of the tools are also available for Windows
Some useful tools are already come with Ubuntu. For example nvtop is a great and simple to use ad-hoc monitoring tool

NVIDA card under load

The NVIDIA RTX 4000 SFF Ada Generation is a small server grade card with 20 GB of RAM.

You can see the fan is already around 50% and the temperature on the card goes up. Power consumption is at 65 of 70 W.
My load test in this example is a simple multi threaded servertask performing LLAMA requests.

You can see all the relevant parameters monitored here. This includes the processes using the GPU. In my case the llama-server you can see at the very bottom of the screen print.

Image:Monitoring NVIDIA GPUs

nvptop is a great tool. But not really helpful in long term monitoring or to see those stats in combination with other stats.
There are some Grafana integrations which either use the NVIDIA toolkit to low level read the stats.

Some projects use the nvidia-smi to query information about the card. It also allows to print stats like other Linux tools (vmstat, iostat).
Beside the stats it can also print timedate and the card model.

Here is a sample command line and output:

nvidia-smi -l 1 --format=csv --query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used
2024/12/01 09:59:36.259, NVIDIA RTX 4000 SFF Ada Generation, 00000000:01:00.0, 565.57.01, P0, 4, 4, 57, 84 %, 67 %, 20475 MiB, 6514 MiB, 13532 MiB
2024/12/01 09:59:37.264, NVIDIA RTX 4000 SFF Ada Generation, 00000000:01:00.0, 565.57.01, P0, 4, 4, 57, 68 %, 39 %, 20475 MiB, 6514 MiB, 13532 MiB
2024/12/01 09:59:38.265, NVIDIA RTX 4000 SFF Ada Generation, 00000000:01:00.0, 565.57.01, P0, 4, 4, 57, 66 %, 39 %, 20475 MiB, 6514 MiB, 13532 MiB
2024/12/01 09:59:39.266, NVIDIA RTX 4000 SFF Ada Generation, 00000000:01:00.0, 565.57.01, P0, 4, 4, 58, 65 %, 41 %, 20475 MiB, 6514 MiB, 13532 MiB
2024/12/01 09:59:40.267, NVIDIA RTX 4000 SFF Ada Generation, 00000000:01:00.0, 565.57.01, P0, 4, 4, 58, 68 %, 40 %, 20475 MiB, 6514 MiB, 13532 MiB
2024/12/01 09:59:41.268, NVIDIA RTX 4000 SFF Ada Generation, 00000000:01:00.0, 565.57.01, P0, 4, 4, 58, 66 %, 43 %, 20475 MiB, 6514 MiB, 13532 MiB

The example above shows the values which can be printed.Not all those parameters make sense to combine.
For example the card model might need to be queried only once.

There are also options to leave out the headers and units: -format=csv,noheader,nounits. Those help to integrate the results into own applications.

Next Steps

Those tools the door for all kind of statistic integrations.
I could for example add those stats to Domino server stats and leverage existing integrations or the out of the box Domino statistics collection.

For now the simple graphic and the command-line is already very helpful to monitor the card.

The nvidia-smi command line is also available on Windows. nvtop is Linux only. But NVIDIA has also Windows based applications to show the status of your cards.

Comments [0]

llama NIVIA

Black Friday - Getting some new AI ready hardware

Daniel Nashed – 30 November 2024 00:24:19

AI functionality requires specific hardware resources. It's pretty clear that you can't avoid looking into NVIDIA GPUs.
But there is new Intel hardware with which helps with AI workloads as well. My test on my new 12th Gen Intel(R) Core(TM) i9-12900HK based machine has some good performance with it's 20 cores.
It's always a combination of GPUs and CPUs. So the new test notebook will be interesting to test.

My Thinkpad has a NVIDIA T1000 card, which is already quite OK for local tests.
The smaller Hetzner GPU server has a Nvidia RTX™ 4000 SFF Ada Generation with 20 GB RAM.

But I need some local test hardware to compile and run AI projects locally on Windows and Linux.
The main project which many other projects including Ollama are built on is llama.cpp https://github.com/ggerganov/llama.cpp.

But also the run-time aspect is interesting. On Docker and Kubernetes you need drivers to support GPUs inside the container.
Also hypervisors like Proxmox support GPU mapping into VMs.
A gaming notebook looks like the most reasonable hardware for a AI lab environment locally.
Black Friday is a great opportunity to get some cool new hardware.

Nvidia RTX™ 4060 comes with 8 GB RAM and decent performance.
The latest Intel CPUs support modern instruction sets
Ordered and looking forward to get my hands on it ..

https://www.asus.com/laptops/for-gaming/tuf-gaming/asus-tuf-gaming-f17-2023/

Image:Black Friday - Getting some new AI ready hardware

https://github.com/ollama/ollama/blob/main/docs/gpu.md

Image:Black Friday - Getting some new AI ready hardware

Comments [0]

LLM

Getting started with LLMs - Ollama is a good starting point

Daniel Nashed – 24 November 2024 19:49:40

llama.cpp is one of the most GitHub projects, which is used for many integrations https://github.com/ggerganov/llama.cpp.
But it is also quite complex to run, because you need to download LLMs.
Once you looked into other projects, you can really appreciate how much easier it can be to use Ollama.

https://ollama.com/

It's available for Linux, Mac and Windows.Ollama takes away all the complexity of compiling optimized for the target CPU and GPU.
And even more important: It automatically downloads and runs LLMs for you, without any conversion.

Ollama provides a registry for downloading LLMs.
For Windows and Mac there is a simple download. For Linux there is a simple convenience script to download.

Image:Getting started with LLMs - Ollama is a good starting point

Image:Getting started with LLMs - Ollama is a good starting point

Once installed you can download and start the LLM via ollama run llama3.2

That's really all you need to get started. Anyone who downloaded a LLM manually and you can start querying the LLM.
The server also listens to 127.0.0.1:11434 with an OpenAI compatible endpoint.
>>>what is the meaning of 42? The number 42 has become a cultural phenomenon due to its appearance in Douglas Adams' science fiction series "The Hitchhiker's Guide to the Galaxy." In the book, a supercomputer named Deep Thought is asked to find the "Answer to the Ultimate Question of Life, the Universe, and Everything." After 7.5 million years of computation, Deep Thought finally reveals that the answer is... 42. However, the book also notes that the characters don't actually know what the ultimate question is, making the answer essentially meaningless without context. This paradoxical situation has led to various interpretations and memes surrounding the number 42. Some possible meanings or interpretations of 42 include: 1. **The Answer**: In a humorous sense, 42 can be seen as the ultimate answer to life's mysteries. 2. **Absurdity**: The number 42 represents the absurdity and randomness of the universe, highlighting the limitations of human understanding. 3. **Symbolism**: Some see 42 as a symbolic representation of the impermanence and fleeting nature of existence. 4. **Mathematical curiosity**: For mathematicians, 42 is an interesting number that has no special properties or significance beyond its numerical value. Ultimately, the meaning of 42 is subjective and open to individual interpretation. It has become a cultural icon, symbolizing the search for answers to life's big questions, and often used humorously to represent "the answer" to any question.

List LLMs ollama list NAME ID SIZE MODIFIED llama3.2:latest a80c4f17acd5 2.0 GB 44 hours ago

Help command-line

Here are the command line options. It's pretty simple. ollama Usage: ollama [flags] ollama [command] Available Commands: serve Start ollama create Create a model from a Modelfile show Show information for a model run Run a model stop Stop a running model pull Pull a model from a registry push Push a model to a registry list List models ps List running models cp Copy a model rm Remove a model help Help about any command

Infos about models

To get details about a model, there is a useful command including the quantization.
That's specially important for lower performing GPUs.ollama show llama3.2 Model architecture llama parameters 3.2B context length 131072 embedding length 3072 quantization Q4_K_M Parameters stop "<|start_header_id|>" stop "<|end_header_id|>" stop "<|eot_id|>" License LLAMA 3.2 COMMUNITY LICENSE AGREEMENT Llama 3.2 Version Release Date: September 25, 2024

Comments [0]

Grafana Prometheus Domino Container

Grafana & Prometheus meets the Domino Container Image

Daniel Nashed – 24 November 2024 18:55:01

A while ago I started a new Grafana for Domino project -> https://github.com/nashcom/domino-grafana
It's a separate repository designed to work for Linux, containers and Windows in combination with the Prometheus Node Exporter.

A /metrics end-point was a missing component for the container image.

On Docker you usually install the Node Exporter native and read the domino.prom file from the container.

But in some scenarios the native /metrics exposed by the container makes sense.
To simplify installations the Nash!Com Domino Grafana project now contains a pre-compiled Linux binary added to software.txt.

Also the Node Exporter is added to software.txt.
At container image build time a single new option allows to install both components automatically.

-- Daniel
-node_exporter Installs Prometheus node_exporter into the container -domprom Installs Domino Prometheus statistics exporter -prometheus/prom Installs Domino Prometheus statistics exporter & Node Exporter

Image:Grafana & Prometheus meets the Domino Container Image

Image:Grafana & Prometheus meets the Domino Container Image

Comments [0]