Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

alt

Daniel Nashed

Daniel Nashed's Blog

 DNUG 

DNUG local meet-up 17.10.24 (today) in Monheim - Focus Notes/Domino 14.5

Daniel Nashed – 17 October 2024 22:52:18

DNUG has local meet-ups in different regions. Today is the next meeting around the corner for me.
It's like the next small town close to me.

https://dnug.de/events/stammtische/rhein-ruhr/

Focus of this meet-up is Notes/Domino 14.5 Early Access Code Drop 1.

The main topics will be the new AutoInstall functionality and the OIDC server.
But we can also discuss other new features available in EA1.

https://help.hcl-software.com/domino/14.5.0/admin/whats_new_in_domino.html

Bring feedback and questions with you ... And if you are not around the corner, you should join the EAP Forum --> https://hclsw.co/domino-14-5-eap-forum

By the way .. the meet-up is organized by DNUG, but is open to anyone!


Image: DNUG local meet-up 17.10.24 (today) in Monheim - Focus Notes/Domino 14.5
 Domino  Container 

Domino Container image custom add-on support enhancements

Daniel Nashed – 13 October 2024 19:28:20

There is a custom add-on functionality Matijn and Roberto just blogged about this week.

https://blog.martdj.nl/2024/10/10/building-custom-add-ons-for-your-domino-container-image/
https://www.robertoboccadoro.com/2024/10/10/upgrading-ontime-in-a-container/

This was the missing tigger for me to look into it again.
It's a quite new functionality which wasn't fully documented yet.

Documentation

I have added a new documentation mark down page-->https://opensource.hcltechsw.com/domino-container/concept_custom_addons/

Custom data dir add-on updating
The functionality wasn't complete. There was a small gap.
Templates added via domino-data dir functionality was only added to the full domino data zip, which is only updated for new servers and when the release changes.
Domino add-ons already had a functionality to update those template files.
But this wasn't yet the case for the custom add-ons.
I have added the functionality to create a custom data tar file per custom add-on, which is applied on server update when the add-on version changes.

New image label DominoContainer.custom-addons

ontime=11.5.0,nashcom-software=1.2.3

There is now also a new label for custom add-ons, which is using the file name of the custom add-on file.
For example nashcom.taz or ontime.zip.

The version can be added after another # like this:

ontime.zip#38159360f0fc2098f85fe4af2626e366ebdd02558c9423c5665cf9f702ae7b10#11.5.0


Support for custom-addons in ZIP format

Some business partner applications use ZIP files instead of tar files for installation.
This is probably because tar isn't a popular format on Windows even Windows has tar installed out of the box.

Now  business partners could modify the structure of their ZIP file to automatically install their application.
I took the OnTime ZIP file and moved the templates and binary to the expected directory structure and tested it today.

It would be very cool if partners would use the same structure.


 NGINX 

NGINX Authentication Based on Subrequest Result

Daniel Nashed – 12 October 2024 21:07:46

NGINX comes in two different versions. The free open source version and the commerical NGINX Plus.

The free version basically supports:
  • Basic authentication
  • Client certificate authentication
  • Allow access by IP address

NGINX also has a very flexible programming model in C or LUA.

But there is an interesting additional option:
https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-subrequest-authentication/

You can authenticate against another HTTP server.
When the user is not authenticated NGINX uses a sub request to check for authentication.

If the requests re turns a HTTP 200 OK, the user authentication was successful.
The result can be cached by authorization token (also in combination with the IP).

In my case I am sending the request to a local UNIX Socket instead of a web server.
The socket request is processed by a C++ program.
Using this approach I can implement any type of authentication on NGINX.
In my case I am verifying a JWT via OpenSSL code.

But this opens the door for any type of authentication integration.
The following example shows how it works in general.

Example configuration
  • Basic authentication
  • IP based authentication
  • Sub request integration
  • Cache the authentication even in tmpfs
  • Use a UNIX socket for the sub request

...
http {

  proxy_cache_path /dev/shm/nginx keys_zone=auth_cache:1m;

 server {
      listen            443 ssl;
      listen       [::]:443 ssl;
      server_name  _;
      root         /usr/share/nginx/html;

      ssl_protocols TLSv1.2 TLSv1.3;
      ssl_prefer_server_ciphers on;

      # Prefer the SSL ciphers for ECDSA:
      ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256';

      # Use multiple curves.
      ssl_ecdh_curve secp521r1:secp384r1;

      ssl_certificate     /etc/nginx/conf.d/cert.pem;
      ssl_certificate_key /etc/nginx/conf.d/key.pem;

      add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

      http2 on;

      location = /auth {
          internal;
          proxy_pass              http://unix:/tmp/auth.sock;
          proxy_pass_request_body off;
          proxy_set_header        Content-Length "";
          proxy_set_header        X-Forwarded-Host $hostname;
          proxy_set_header        X-Forwarded-For  $remote_addr;
          proxy_set_header        X-Orig-URI       $request_uri;
          proxy_set_header        X-Orig-Protocol  $ssl_protocol;
          proxy_set_header        X-Orig-Cipher    $ssl_cipher;

          proxy_cache             auth_cache;
          proxy_cache_valid any   1m;
          # Use the authorization header and IP for cache lookup
          proxy_cache_key         "$http_authorization$remote_addr";
      }

      location / {
          satisfy any;
          allow 127.0.0.1;
          allow 1.2.3.4;
          deny all;

          auth_basic           "Software Download";
          auth_basic_user_file /etc/nginx/conf.d/htpasswd;

          auth_request     /auth;
          auth_request_set $auth_status $upstream_status;

          root   /usr/share/nginx/html;
          index  index.html;
      }
  }
...


Example request passed as a sub request

You can add all type of information to the sub request on top of the request itself.
IP address, hostname and TLS information can be helpful.
This opens the door for many different authentication options.

------
GET /auth HTTP/1.0
X-Forwarded-Host: volt.domino-lab.net
X-Forwarded-For: 91.14.169.18
X-Orig-URI: /software.txt
X-Orig-Protocol: TLSv1.3
X-Orig-Cipher: TLS_AES_256_GCM_SHA384
Host: localhost
Connection: close
user-agent: curl/7.81.0
accept: */*
authorization: Basic ZG9ja2VyOmV...



 CertMgr  ACME 

Troubleshooting CertMgr ACME HTTP-01 challenges

Daniel Nashed – 9 October 2024 23:55:20

Let's Encrypt HTTP-01 challenges can be tricky sometimes.
The server needs inbound port 80 open.

There is a script to help troubleshooting https://letsdebug.net/

The Domino ACME HTTPS-01 challenge troubleshooting guide also references this project and provides additional information.
https://github.com/HCL-TECH-SOFTWARE/domino-cert-manager/blob/main/docs/troubleshooting_acme_challenges.md

I have updated the project to use the same troubleshooting challenge and updated the material.

But the Let's Debug project also provides a REST API for testing.
I looked at the project and came up with a Lotus Script Lib and added a form around it.

This is a true end to end test including a random challenge creation and a local pre-check via Curl.
The script is also a good tutorial for the HTTP Request class and JSON in Lotus Script.

The Script Lib and Class will be also the base for extending automatic setups.


Image:Troubleshooting CertMgr ACME HTTP-01 challenges
 Domino 

Ideas for a signing tool

Daniel Nashed – 2 October 2024 09:51:27

Template signing is an important part of your Notes/Domino security.
The only fully automated signing is leveraging the AdminP process.
But this would only allow you to sign with the current server.id.

Using the server.id isn't a great choice because anyone who can run unrestricted code can sign code with the server.id.

A best practice is to create a Template Signing ID, like HCL has a template signer which is used to sign all templates.


But how to you sign with a signing ID?

Currently out of the box you can only switch to the ID and sign the database manually.
There are business partner tools out that to fill this gap. But there is no out of the box solution.

If we want more developers to sign their template right, we should help them to sign.
I am working on a signing tool and I am still looking for hte right functionality.

The basic command-line tool is done. But I also want to support delegated signing using a request database.

But I already have command-line tool I am using and refining for my own needs.
Today I added a check to only sign design elements which have not been signed by the signer yet.
I can open a signing ID from local disk, a profile or from IDVault. But I can't get it's name using public APIs.

But maybe this is the more flexible way. You can specify a string to check when validating the signature.

Not updating the existing signature has a performance gain during signing. But even more important, it would skip design elements not modified and does not re-sign them.
So the template design element would not be modified.

I am planning to make the basic command line tool free.
And my intent is also to offer a complete solution including a request based model to OpenNTF and other none profit organizations for free.

What do you think? Is this helpful if I only provide the command-line for free as a first step?
It would be likely not be open source. But free.

-- Daniel



Nash!Com Database Design & Signing Tool 0.9.0
Copyright 2024, Nash!Com, Daniel Nashed

Usage: nshsign [Options]

-sign                   Sign database
-refresh                Refresh database design
-changereplicaid        Change replica ID
-ntfserver     Template server
-id        ID File to use for signing
-password    Password for NotesID
-pwdcmd           Password command to executed to get the NotesID password
-user        User name for looking up the user name in ID Vault
-idfiledb    Database name to detach the signing ID from
-profile         Profile name to detach the signing ID for (default: use Shimmer ID)
-SkipSigner       Signer to skip (case sensitive but can be part of a string)
-title           Title for database/template
-template        Template name to set
-ntfversion   Version of template
-noteclass    Note class (by default all design elements). For hex specify the prefix X or H (e.g x200 for agent notes)
-publish     Take a backup copy and store new template outside Notes/Domino


Examples with skipping existing signatures and without


nnshsign.exe mail12_signtest.nsf -SkipSigner Templatex -sign -id D:\notes\id\nashcom\nashcom_template_development.id -password xyz

[17C20:0002-1AA10] 02.10.2024 11:48:29   nshsign: WARNING: Database is locally encrypted: [mail12_signtest.nsf]
[17C20:0002-1AA10] 02.10.2024 11:48:29   nshsign: Processing 1167 design elements
[17C20:0002-1AA10] 02.10.2024 11:49:24   nshsign: Design elements: 1167 [Singed]
[17C20:0002-1AA10] 02.10.2024 11:49:24   nshsign: Runtime - 00:54.969
[17C20:0002-1AA10] 02.10.2024 11:49:24   nshsign: Signing - 00:28.934
[17C20:0002-1AA10] 02.10.2024 11:49:24   nshsign: Done


nnshsign.exe mail12_signtest.nsf -SkipSigner Template -sign -id D:\notes\id\nashcom\nashcom_template_development.id -password xyz

[12624:0002-7E38] 02.10.2024 11:49:57   nshsign: WARNING: Database is locally encrypted: [mail12_signtest.nsf]
[12624:0002-7E38] 02.10.2024 11:49:57   nshsign: Processing 1167 design elements
[12624:0002-7E38] 02.10.2024 11:50:20   nshsign: Design elements: 1167 [already signed]
[12624:0002-7E38] 02.10.2024 11:50:20   nshsign: Design elements: 0 [Singed]
[12624:0002-7E38] 02.10.2024 11:50:20   nshsign: Runtime - 00:23.891
[12624:0002-7E38] 02.10.2024 11:50:20   nshsign: Done


 Lotus Script 

Lotus Script NotesHTTPRequest Class returns double sized buffer for a GET request.

Daniel Nashed – 2 October 2024 08:42:00

I ran into this writing a new application published soon. It first looked like I got hit by the trailing CRLF.
But it turned out the buffer returned had the double len of what it should be.

This looks like a bug and there is already a SPR. HCL is investigating.
In my tests it is always double the size including the CRLF.

You could start searching for the end of the data. But how do you know where it ends?

My current work-around is to use the Content-Length header and truncate the string.

Here is a simple example of my work-around.

I would be interested if someone ran into similar issues. I tried all settings to prefer strings and UTF-8.
But this did not make a difference.

The returned JSON is not affected and returned differently.

I would like to hear from you if you ran into this too.
Or if you have different experiences and work-arounds.

-- Daniel


Function HttpGetRequest (URL As String) As String
     
     Dim Session As New NotesSession
     Dim headers As Variant
     Dim req As NotesHTTPRequest
     
     Dim ret As Variant
     Dim HeaderName As String
     Dim HeaderValue As String
     Dim ResponseCode As Long
     Dim ResponseText As String
     Dim ContentLen As Long
     Dim ResponseStr As String
     
     HttpGetRequest = ""
     ContentLen = 0
     
     Set req = session.CreateHttpRequest()
     
     req.maxredirects= 5
     req.Preferstrings = True
     
     Call req.SetHeaderField("Content-Type", "text/plain")
     
     ret = req.Get (URL)
     
     ResponseStr = req.ResponseCode
     ResponseCode = Clng (Strtoken (ResponseStr, " ",2))
     ResponseText = Strtoken (ResponseStr, " ",3)
     
     headers = req.GetResponseHeaders()
     
     Forall h In headers
             HeaderName = Strleft (h, ":")
             
             If (        HeaderName = "Content-Length") Then
                     HeaderValue = Strright  (h, HeaderName + ": ")
                     ContentLen = Clng (HeaderValue)
             End If
             
     End Forall
     
     If (200 = ResponseCode) Then
             HttpGetRequest = Replace (Left (ret, ContentLen), Chr(13)+Chr(10), "")
     End If
     
End Function



 Linux  Domino  Container 

Linux LSOF is causing 100% CPU load inside a container in some configurations

Daniel Nashed – 2 October 2024 07:10:53

This hit me a couple of times and now I found the root cause.
I am still thinking about how to best solve it for everyone. But here are the current facts.

LSOF is the Linux tool to list open files and is used by NSD.
It runs into issues when the security limits allow too many open files.

Background

The Linux security limits prevent processes to use too much resources.
Traditionally the security limits for number of open files was set quite low to 1024.


But some applications like Domino need a lot of open files because each lib, each open file and each network connection is a file under Linux.
The recommended number of maximal open files for the Domino user "notes" is 80000.

Usually you set this in /etc/security/limits.conf.

The container image sets this limit correctly. But it turns out that Docker or better containerd sets a much higher limit in some cases.
If Domino would switch the user like on a normal server, the security limits would be enforced and set to 80000.
But because the user the container is started with is already the unprivileged user "notes", the user is not switched.

Therefore the security limits are not changed by the Linux container.
Usually security limits for number of files should look like this by default:


ulimit -n
1048576

In case of certain Docker / containerd configurations the container runs with a 1024 times higher limit:

ulimit -n
1073741816

This causes the high CPU usage with LSOF.


Solution

A first solution would be to explicitly set the number of open files when starting the container.

docker run --rm -it --ulimit nofile=1048576:1048576 ubuntu bash

But probably it would be a better way to set the security limits before starting Domino inside the container to be on the safe side.
If the container launches with a too high limit an unprivileged user can still lower but not increase the limits.

I am currently testing if we should reduce the number of files to 1048576.
The only software currently affected is LSOF. But you never know and this would be a good defensive change.


SUPER.HUMAN.SOFTWARE

Daniel Nashed – 1 October 2024 20:20:20

You might want to call me a tree hugger. When I had to get a copy of a old Domino 5.0.5 pubnames.ntf copy for a friend I ran into this installer screen.
It has bee a while and it has been an interesting ride so far.

With Domino 12 and later releases we got a lot back from the old passion and energy back into Domino server development.
The last couple of releases have been a blast.

Notes & Domino always was and still is a remarkable software product. Some of the base concepts are still unmatched by other software.

I had never more fun working with Domino. I my different roles I have never created more code then in the last couple of years.
There is just not enough time to get all my favorite projects done. Specially the Open Source projects take a lot of time -- But it is worth every minute for me!

I have worked with quite a bunch of other software at customers in the last years.
None of them had the same functionality when it comes to flexible, integrated application development, long term support of functionality and cross platform support.
The grass always looks greener on the other side. But did you look back to what you had?

Seeing this install screen below I am recalling the old times and want to give kudos to all the friends around the globe who are on the same journey for all those years.

Domino release 5.x was a mile stone for Domino. So as the current releases are!
A while ago I updated a Domino 5.0.3 server on Linux in 30 seconds to Domino 12.0.2 inside a Docker container at the HCL factory tour in Milan.
Try that with any other enterprise messaging system..

I could not be more proud of being part of that journey.

-- Daniel


Now the R5 launch reminds me on something else ...  
I AM ... https://www.youtube.com/watch?v=rRAhCCGb7BY
And look who posted it at that time ..


Image:SUPER.HUMAN.SOFTWARE






 Domio 

Disabling XPages if not needed reduces open files and HTTP start/stop time

Daniel Nashed – 29 September 2024 20:50:03

While working on setup automation I often ran into HTTP startup challenges.
It can take up to 40-50 seconds until the HTTP task is started.

If you look at open files, you notice that each thread has more than 70 files open.
This sums up to up quite some files and the HTTP server start/stop time is much slower.

In case you don't use XPages there is a simple switch to disable the XPages run-time and only load the standard Java components.

notes.ini INotesDisableXPageCMD=1

I first had the impression Java in general would cause overhead on start. But my tests drilled down to XPages/OSGI.

Specially when you run a Traveler server, this can cause even more files open. Traveler auto configuration sets the HTTP threads to 400.
But Traveler uses an OSGI servlet. With Traveler you cannot disable the XPages run-time.

It is still a good idea to set the number of HTTP threads on your server to the number of sessions you need for a Traveler server.
If you are running a small server you might want to lower "Number active threads:" in server document.

-- Daniel


Here is the Traveler message when you still disable it. Traveler will not work without it.

Traveler: SEVERE  *system Configuration parameter INOTESDISABLEXPAGECMD=1 in the notes.ini file is preventing the Traveler Servlet from starting.
For Traveler to be functional, please remove this parameter.  See the Domino Server information center for more information on notes.ini parameters.
 Notes  Domino 

Notes/Domino Named Documents -- What are they and where/when to use them?

Daniel Nashed – 29 September 2024 07:22:23

Named documents are new in Notes/Domino 12.0.1. They are very similar to profile documents, but have less side effects.
Profile documents are cached and if used in a web application might end up being opened by may threads.

Also when updating profile documents, it can take a while until all parts of Notes/Domino see the document.
Like a Profile document a Named Document is not shown in any view. But beside that they behave like normal documents.

They are just searched by name -- similar to profile documents. Therefore they are really fast.
The only down side using Named Documents, they are only available in Notes/Domino 12.0.1 and higher.

Some system databases in Domino use them. For example they are leveraged in AdminCentral and AutoUpdate which are both introduced in version 14.0.
Here is a reference to designer help --> https://help.hcl-software.com/dom_designer/14.0.0/basic/H_GETNAMEDDOCUMENT_METHOD.html

Profile Documents will stay and they should still be used for static data.
But if you have data which is often changed, you should consider using Named Documents instead.

For example the Domino AutoUpdate Refresh token document is using a Named Document.

Tip: Add version check for 12.0.1+ clients

It would be probably a good idea adding a version check to your template when you start using newer functionality.

OTS support for profiles and named documents

I have just opened an AHA idea for profile document and named document support in Domino OTS.
If you find this useful, please vote. --> https://domino-ideas.hcltechsw.com/ideas/DOMINO-I-2880

-- Daniel

Recent Entries

Feeds

Links

    Archives

    • [HCL Domino]
    • [Domino on Linux]
    • [Nash!Com]
    • [Daniel Nashed]