Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

Daniel Nashed

Tags

DAOS Tune 2.0 available for download on MHS

Daniel Nashed – 22 October 2024 09:59:15

The DAOS Estimator was a separate download to analyze attachments to estimate the size and deduplication of attachments before you enable DAOS.
Starting with Domino 14.5 EA1 HCL provides a new completely rewritten tool which is now also available for Domino 12.0.1 and higher.

The documentation is available as a PDF on MHS download site as well.
You find the latest documentation in the EAP beta here ->https://help.hcl-software.com/domino/14.5.0/admin/admn_daostune_c.html.

The new DAOS Tune can be used to analyze servers before enabling DAOS and also to check if the DAOS settings could be optimized for an existing DAOS configuration.

The new tool provides more detailed information and recommendations.
You can also scan on one server and analyze the result file on another server.
But the most important point is that the new tool is dramatically faster.

If you have feedback or questions you should join the Domino Early Access program --> https://hclsw.co/domino-14-5-eap-forum

Note:
I also had to accept a new license agreement before I was able to download the software.
So if you use the download script, you have to visit the MHS site first.

-- Daniel

Image:DAOS Tune 2.0 available for download on MHS

Image:DAOS Tune 2.0 available for download on MHS

Here are the two new binaries for Windows and Linux -------------------------------------------------------------------------------- WebKit : DAOSTune Estimation Tool 2.0 for Linux Name : daostune Version : 2.0 Platform : linux Size : 808640 SHA256 : 4b9e7af34f3cbba9a699b4288e0e10088659c5b271b0abb58907630736107793 ID : vwWWuh6yyudOuRjutaHnW -------------------------------------------------------------------------------- WebKit : DAOSTune Estimation Tool 2.0 for Windows Name : daostune.exe Version : 2.0 Platform : windows Size : 121904 SHA256 : fd701f71fe9db9e6750ce9d209e12d1f81775b4c87721877e3e8f7a0b605eb73 ID : 0tKnEMt0jLK0LZhxuWve8 --------------------------------------------------------------------------------

Comments [0]

USB Linux

Running Linux from USB stick on a machine with secure boot

Daniel Nashed – 19 October 2024 14:24:05

Booting from USB isn't always easy. My Thinkpad has secure boot enabled.
Even Ubuntu on my USB stick was signed, the boot failed.
The root cause was that only Windows root certificates had been enabled.

Allowing 3rd party boot key signing CAs is tricky

The solution is to allow 3rd part signing certificates.
But you need to be very careful when disabling secure boot or enabling 3rd party signing certificates.

If you have Bitlocker enabled, you will need your recovery key to get Windows working again!

If you don't have your key, you are in real trouble. You should note your recovery key anyhow! Not just for this operation..
In my case it was a new Thinkpad and the setup had synced the recovery key to my Microsoft account already.
Not that I asked for it or allow they sync/backup, but in this case it was great to have it.

Once the secure boot has been updated, I could boot from my Ubuntu Desktop distribution.

Bootable USB sticks with persistent partition size

A bootable USB stick is good to have as a recovery option.
But you can also use it to setup a Linux with a ready to use configuration.

Rufus can add a persistent volume partition to setup a workstation or server which you could run on any kind of hardware without using the local disk.
Provided you get the secure boot working ..

Image:Running Linux from USB stick on a machine with secure boot

Image:Running Linux from USB stick on a machine with secure boot

Comments [0]

Domino

Domino 14.0FP2IF1 for Windows has been re-published

Daniel Nashed – 18 October 2024 17:08:25

A Domino Interimfix (IF) is actually a hotfix with one or multiple fixes built and published for all platforms. Only the IF number in the file really labels it as a IF.
Domino 14 AutoUpdate knows about hotfix and IF numbers per platform and maps them accordingly.

In case of this IF the hotfix build for Windows had Windows VS run-time environment dependencies.
because the hotfix installer is still 32bit this required the 32bit run-time VC libs to be installed.

HCL fixed the issue for future fixpacks. But now also re-published the IF1 for Windows.
The old file is removed and the new file has a new file name, new hash and fileID.

---

If you are using Domino 14.0 AutoUpdate and have not downloaded the file, the old IF will be removed from your autoupdate.nsf.
But if you downloaded it, the file stays in autoupdate.nsf and is marked as a legacy document.
That's because HCL would not want to delete any software you downloaded.

But the document is clearly marked as legacy and you can use the delete action to delete it.

The new IF has a new HF number as you see below. MHS is updated and also the software.jwt data which is the source for autoupdate.nsf has been updated.
WebKit : HCL Domino Server 14.0FP2IF1 for Windows 64bit Name : 140FP2HF18_W64.exe Version : 14.0fp2if1 Platform : windows Size : 32460848 SHA256 : 729fe5e006c0c46701e9e6098c24a573ab4a22f5352eeb8963d21ef088cc6f87 ID : dKN8b0rafN5ib3BEEx2Et Modified : 2024-10-17T00:00:00.000Z

Comments [0]

DNUG

DNUG local meet-up 17.10.24 (today) in Monheim - Focus Notes/Domino 14.5

Daniel Nashed – 17 October 2024 22:52:18

DNUG has local meet-ups in different regions. Today is the next meeting around the corner for me.
It's like the next small town close to me.

https://dnug.de/events/stammtische/rhein-ruhr/

Focus of this meet-up is Notes/Domino 14.5 Early Access Code Drop 1.

The main topics will be the new AutoInstall functionality and the OIDC server.
But we can also discuss other new features available in EA1.

https://help.hcl-software.com/domino/14.5.0/admin/whats_new_in_domino.html

Bring feedback and questions with you ... And if you are not around the corner, you should join the EAP Forum --> https://hclsw.co/domino-14-5-eap-forum

By the way .. the meet-up is organized by DNUG, but is open to anyone!

Comments [0]

Domino Container

Domino Container image custom add-on support enhancements

Daniel Nashed – 13 October 2024 19:28:20

There is a custom add-on functionality Matijn and Roberto just blogged about this week.

https://blog.martdj.nl/2024/10/10/building-custom-add-ons-for-your-domino-container-image/
https://www.robertoboccadoro.com/2024/10/10/upgrading-ontime-in-a-container/

This was the missing tigger for me to look into it again.
It's a quite new functionality which wasn't fully documented yet.

Documentation

I have added a new documentation mark down page-->https://opensource.hcltechsw.com/domino-container/concept_custom_addons/

Custom data dir add-on updating
The functionality wasn't complete. There was a small gap.
Templates added via domino-data dir functionality was only added to the full domino data zip, which is only updated for new servers and when the release changes.
Domino add-ons already had a functionality to update those template files.
But this wasn't yet the case for the custom add-ons.
I have added the functionality to create a custom data tar file per custom add-on, which is applied on server update when the add-on version changes.

New image label DominoContainer.custom-addons
ontime=11.5.0,nashcom-software=1.2.3

There is now also a new label for custom add-ons, which is using the file name of the custom add-on file.
For example nashcom.taz or ontime.zip.

The version can be added after another # like this:
ontime.zip#38159360f0fc2098f85fe4af2626e366ebdd02558c9423c5665cf9f702ae7b10#11.5.0

Support for custom-addons in ZIP format

Some business partner applications use ZIP files instead of tar files for installation.
This is probably because tar isn't a popular format on Windows even Windows has tar installed out of the box.

Now business partners could modify the structure of their ZIP file to automatically install their application.
I took the OnTime ZIP file and moved the templates and binary to the expected directory structure and tested it today.

It would be very cool if partners would use the same structure.

Comments [0]

NGINX

NGINX Authentication Based on Subrequest Result

Daniel Nashed – 12 October 2024 21:07:46

NGINX comes in two different versions. The free open source version and the commerical NGINX Plus.

The free version basically supports:

Basic authentication
Client certificate authentication
Allow access by IP address

NGINX also has a very flexible programming model in C or LUA.

But there is an interesting additional option:
https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-subrequest-authentication/

You can authenticate against another HTTP server.
When the user is not authenticated NGINX uses a sub request to check for authentication.

If the requests re turns a HTTP 200 OK, the user authentication was successful.
The result can be cached by authorization token (also in combination with the IP).

In my case I am sending the request to a local UNIX Socket instead of a web server.
The socket request is processed by a C++ program.
Using this approach I can implement any type of authentication on NGINX.
In my case I am verifying a JWT via OpenSSL code.

But this opens the door for any type of authentication integration.
The following example shows how it works in general.

Example configuration

Basic authentication
IP based authentication
Sub request integration
Cache the authentication even in tmpfs
Use a UNIX socket for the sub request

... http { proxy_cache_path /dev/shm/nginx keys_zone=auth_cache:1m; server { listen 443 ssl; listen [::]:443 ssl; server_name _; root /usr/share/nginx/html; ssl_protocols TLSv1.2 TLSv1.3; ssl_prefer_server_ciphers on; # Prefer the SSL ciphers for ECDSA: ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256'; # Use multiple curves. ssl_ecdh_curve secp521r1:secp384r1; ssl_certificate /etc/nginx/conf.d/cert.pem; ssl_certificate_key /etc/nginx/conf.d/key.pem; add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; http2 on; location = /auth { internal; proxy_pass http://unix:/tmp/auth.sock; proxy_pass_request_body off; proxy_set_header Content-Length ""; proxy_set_header X-Forwarded-Host $hostname; proxy_set_header X-Forwarded-For $remote_addr; proxy_set_header X-Orig-URI $request_uri; proxy_set_header X-Orig-Protocol $ssl_protocol; proxy_set_header X-Orig-Cipher $ssl_cipher; proxy_cache auth_cache; proxy_cache_valid any 1m; # Use the authorization header and IP for cache lookup proxy_cache_key "$http_authorization$remote_addr"; } location / { satisfy any; allow 127.0.0.1; allow 1.2.3.4; deny all; auth_basic "Software Download"; auth_basic_user_file /etc/nginx/conf.d/htpasswd; auth_request /auth; auth_request_set $auth_status $upstream_status; root /usr/share/nginx/html; index index.html; } } ...

Example request passed as a sub request

You can add all type of information to the sub request on top of the request itself.
IP address, hostname and TLS information can be helpful.
This opens the door for many different authentication options. ------ GET /auth HTTP/1.0 X-Forwarded-Host: volt.domino-lab.net X-Forwarded-For: 91.14.169.18 X-Orig-URI: /software.txt X-Orig-Protocol: TLSv1.3 X-Orig-Cipher: TLS_AES_256_GCM_SHA384 Host: localhost Connection: close user-agent: curl/7.81.0 accept: */* authorization: Basic ZG9ja2VyOmV...

Comments [0]

CertMgr ACME

Troubleshooting CertMgr ACME HTTP-01 challenges

Daniel Nashed – 9 October 2024 23:55:20

Let's Encrypt HTTP-01 challenges can be tricky sometimes.
The server needs inbound port 80 open.

There is a script to help troubleshooting https://letsdebug.net/

The Domino ACME HTTPS-01 challenge troubleshooting guide also references this project and provides additional information.
https://github.com/HCL-TECH-SOFTWARE/domino-cert-manager/blob/main/docs/troubleshooting_acme_challenges.md

I have updated the project to use the same troubleshooting challenge and updated the material.

But the Let's Debug project also provides a REST API for testing.
I looked at the project and came up with a Lotus Script Lib and added a form around it.

This is a true end to end test including a random challenge creation and a local pre-check via Curl.
The script is also a good tutorial for the HTTP Request class and JSON in Lotus Script.

The Script Lib and Class will be also the base for extending automatic setups.

Image:Troubleshooting CertMgr ACME HTTP-01 challenges

Image:Troubleshooting CertMgr ACME HTTP-01 challenges

Comments [0]

Domino

Ideas for a signing tool

Daniel Nashed – 2 October 2024 09:51:27

Template signing is an important part of your Notes/Domino security.
The only fully automated signing is leveraging the AdminP process.
But this would only allow you to sign with the current server.id.

Using the server.id isn't a great choice because anyone who can run unrestricted code can sign code with the server.id.

A best practice is to create a Template Signing ID, like HCL has a template signer which is used to sign all templates.

But how to you sign with a signing ID?

Currently out of the box you can only switch to the ID and sign the database manually.
There are business partner tools out that to fill this gap. But there is no out of the box solution.

If we want more developers to sign their template right, we should help them to sign.
I am working on a signing tool and I am still looking for hte right functionality.

The basic command-line tool is done. But I also want to support delegated signing using a request database.

But I already have command-line tool I am using and refining for my own needs.
Today I added a check to only sign design elements which have not been signed by the signer yet.
I can open a signing ID from local disk, a profile or from IDVault. But I can't get it's name using public APIs.

But maybe this is the more flexible way. You can specify a string to check when validating the signature.

Not updating the existing signature has a performance gain during signing. But even more important, it would skip design elements not modified and does not re-sign them.
So the template design element would not be modified.

I am planning to make the basic command line tool free.
And my intent is also to offer a complete solution including a request based model to OpenNTF and other none profit organizations for free.

What do you think? Is this helpful if I only provide the command-line for free as a first step?
It would be likely not be open source. But free.

-- Daniel

Nash!Com Database Design & Signing Tool 0.9.0 Copyright 2024, Nash!Com, Daniel Nashed
Usage: nshsign [Options]
-sign Sign database -refresh Refresh database design -changereplicaid Change replica ID -ntfserver Template server -id ID File to use for signing -password Password for NotesID -pwdcmd Password command to executed to get the NotesID password -user User name for looking up the user name in ID Vault -idfiledb Database name to detach the signing ID from -profile Profile name to detach the signing ID for (default: use Shimmer ID) -SkipSigner Signer to skip (case sensitive but can be part of a string) -title Title for database/template -template Template name to set -ntfversion Version of template -noteclass Note class (by default all design elements). For hex specify the prefix X or H (e.g x200 for agent notes) -publish Take a backup copy and store new template outside Notes/Domino

Examples with skipping existing signatures and without

nnshsign.exe mail12_signtest.nsf -SkipSigner Templatex -sign -id D:\notes\id\nashcom\nashcom_template_development.id -password xyz
[17C20:0002-1AA10] 02.10.2024 11:48:29 nshsign: WARNING: Database is locally encrypted: [mail12_signtest.nsf] [17C20:0002-1AA10] 02.10.2024 11:48:29 nshsign: Processing 1167 design elements [17C20:0002-1AA10] 02.10.2024 11:49:24 nshsign: Design elements: 1167 [Singed] [17C20:0002-1AA10] 02.10.2024 11:49:24 nshsign: Runtime - 00:54.969 [17C20:0002-1AA10] 02.10.2024 11:49:24 nshsign: Signing - 00:28.934 [17C20:0002-1AA10] 02.10.2024 11:49:24 nshsign: Done

nnshsign.exe mail12_signtest.nsf -SkipSigner Template -sign -id D:\notes\id\nashcom\nashcom_template_development.id -password xyz
[12624:0002-7E38] 02.10.2024 11:49:57 nshsign: WARNING: Database is locally encrypted: [mail12_signtest.nsf] [12624:0002-7E38] 02.10.2024 11:49:57 nshsign: Processing 1167 design elements [12624:0002-7E38] 02.10.2024 11:50:20 nshsign: Design elements: 1167 [already signed] [12624:0002-7E38] 02.10.2024 11:50:20 nshsign: Design elements: 0 [Singed] [12624:0002-7E38] 02.10.2024 11:50:20 nshsign: Runtime - 00:23.891 [12624:0002-7E38] 02.10.2024 11:50:20 nshsign: Done

Comments [2]

Lotus Script

Lotus Script NotesHTTPRequest Class returns double sized buffer for a GET request.

Daniel Nashed – 2 October 2024 08:42:00

I ran into this writing a new application published soon. It first looked like I got hit by the trailing CRLF.
But it turned out the buffer returned had the double len of what it should be.

This looks like a bug and there is already a SPR. HCL is investigating.
In my tests it is always double the size including the CRLF.

You could start searching for the end of the data. But how do you know where it ends?

My current work-around is to use the Content-Length header and truncate the string.

Here is a simple example of my work-around.

I would be interested if someone ran into similar issues. I tried all settings to prefer strings and UTF-8.
But this did not make a difference.

The returned JSON is not affected and returned differently.

I would like to hear from you if you ran into this too.
Or if you have different experiences and work-arounds.

-- Daniel Function HttpGetRequest (URL As String) As String Dim Session As New NotesSession Dim headers As Variant Dim req As NotesHTTPRequest Dim ret As Variant Dim HeaderName As String Dim HeaderValue As String Dim ResponseCode As Long Dim ResponseText As String Dim ContentLen As Long Dim ResponseStr As String HttpGetRequest = "" ContentLen = 0 Set req = session.CreateHttpRequest() req.maxredirects= 5 req.Preferstrings = True Call req.SetHeaderField("Content-Type", "text/plain") ret = req.Get (URL) ResponseStr = req.ResponseCode ResponseCode = Clng (Strtoken (ResponseStr, " ",2)) ResponseText = Strtoken (ResponseStr, " ",3) headers = req.GetResponseHeaders() Forall h In headers HeaderName = Strleft (h, ":") If ( HeaderName = "Content-Length") Then HeaderValue = Strright (h, HeaderName + ": ") ContentLen = Clng (HeaderValue) End If End Forall If (200 = ResponseCode) Then HttpGetRequest = Replace (Left (ret, ContentLen), Chr(13)+Chr(10), "") End If End Function

Comments [0]

Linux Domino Container

Linux LSOF is causing 100% CPU load inside a container in some configurations

Daniel Nashed – 2 October 2024 07:10:53

This hit me a couple of times and now I found the root cause.
I am still thinking about how to best solve it for everyone. But here are the current facts.

LSOF is the Linux tool to list open files and is used by NSD.
It runs into issues when the security limits allow too many open files.

Background

The Linux security limits prevent processes to use too much resources.
Traditionally the security limits for number of open files was set quite low to 1024.

But some applications like Domino need a lot of open files because each lib, each open file and each network connection is a file under Linux.
The recommended number of maximal open files for the Domino user "notes" is 80000.

Usually you set this in /etc/security/limits.conf.

The container image sets this limit correctly. But it turns out that Docker or better containerd sets a much higher limit in some cases.
If Domino would switch the user like on a normal server, the security limits would be enforced and set to 80000.
But because the user the container is started with is already the unprivileged user "notes", the user is not switched.

Therefore the security limits are not changed by the Linux container.
Usually security limits for number of files should look like this by default:

ulimit -n 1048576

In case of certain Docker / containerd configurations the container runs with a 1024 times higher limit:
ulimit -n 1073741816

This causes the high CPU usage with LSOF.

Solution

A first solution would be to explicitly set the number of open files when starting the container. docker run --rm -it--ulimit nofile=1048576:1048576 ubuntu bash

But probably it would be a better way to set the security limits before starting Domino inside the container to be on the safe side.
If the container launches with a too high limit an unprivileged user can still lower but not increase the limits.

I am currently testing if we should reduce the number of files to 1048576.
The only software currently affected is LSOF. But you never know and this would be a good defensive change.

Comments [0]