Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

alt

Daniel Nashed

Daniel Nashed's Blog

 Domino 

Ideas for a signing tool

Daniel Nashed – 2 October 2024 09:51:27

Template signing is an important part of your Notes/Domino security.
The only fully automated signing is leveraging the AdminP process.
But this would only allow you to sign with the current server.id.

Using the server.id isn't a great choice because anyone who can run unrestricted code can sign code with the server.id.

A best practice is to create a Template Signing ID, like HCL has a template signer which is used to sign all templates.


But how to you sign with a signing ID?

Currently out of the box you can only switch to the ID and sign the database manually.
There are business partner tools out that to fill this gap. But there is no out of the box solution.

If we want more developers to sign their template right, we should help them to sign.
I am working on a signing tool and I am still looking for hte right functionality.

The basic command-line tool is done. But I also want to support delegated signing using a request database.

But I already have command-line tool I am using and refining for my own needs.
Today I added a check to only sign design elements which have not been signed by the signer yet.
I can open a signing ID from local disk, a profile or from IDVault. But I can't get it's name using public APIs.

But maybe this is the more flexible way. You can specify a string to check when validating the signature.

Not updating the existing signature has a performance gain during signing. But even more important, it would skip design elements not modified and does not re-sign them.
So the template design element would not be modified.

I am planning to make the basic command line tool free.
And my intent is also to offer a complete solution including a request based model to OpenNTF and other none profit organizations for free.

What do you think? Is this helpful if I only provide the command-line for free as a first step?
It would be likely not be open source. But free.

-- Daniel



Nash!Com Database Design & Signing Tool 0.9.0
Copyright 2024, Nash!Com, Daniel Nashed

Usage: nshsign [Options]

-sign                   Sign database
-refresh                Refresh database design
-changereplicaid        Change replica ID
-ntfserver     Template server
-id        ID File to use for signing
-password    Password for NotesID
-pwdcmd           Password command to executed to get the NotesID password
-user        User name for looking up the user name in ID Vault
-idfiledb    Database name to detach the signing ID from
-profile         Profile name to detach the signing ID for (default: use Shimmer ID)
-SkipSigner       Signer to skip (case sensitive but can be part of a string)
-title           Title for database/template
-template        Template name to set
-ntfversion   Version of template
-noteclass    Note class (by default all design elements). For hex specify the prefix X or H (e.g x200 for agent notes)
-publish     Take a backup copy and store new template outside Notes/Domino


Examples with skipping existing signatures and without


nnshsign.exe mail12_signtest.nsf -SkipSigner Templatex -sign -id D:\notes\id\nashcom\nashcom_template_development.id -password xyz

[17C20:0002-1AA10] 02.10.2024 11:48:29   nshsign: WARNING: Database is locally encrypted: [mail12_signtest.nsf]
[17C20:0002-1AA10] 02.10.2024 11:48:29   nshsign: Processing 1167 design elements
[17C20:0002-1AA10] 02.10.2024 11:49:24   nshsign: Design elements: 1167 [Singed]
[17C20:0002-1AA10] 02.10.2024 11:49:24   nshsign: Runtime - 00:54.969
[17C20:0002-1AA10] 02.10.2024 11:49:24   nshsign: Signing - 00:28.934
[17C20:0002-1AA10] 02.10.2024 11:49:24   nshsign: Done


nnshsign.exe mail12_signtest.nsf -SkipSigner Template -sign -id D:\notes\id\nashcom\nashcom_template_development.id -password xyz

[12624:0002-7E38] 02.10.2024 11:49:57   nshsign: WARNING: Database is locally encrypted: [mail12_signtest.nsf]
[12624:0002-7E38] 02.10.2024 11:49:57   nshsign: Processing 1167 design elements
[12624:0002-7E38] 02.10.2024 11:50:20   nshsign: Design elements: 1167 [already signed]
[12624:0002-7E38] 02.10.2024 11:50:20   nshsign: Design elements: 0 [Singed]
[12624:0002-7E38] 02.10.2024 11:50:20   nshsign: Runtime - 00:23.891
[12624:0002-7E38] 02.10.2024 11:50:20   nshsign: Done


 Lotus Script 

Lotus Script NotesHTTPRequest Class returns double sized buffer for a GET request.

Daniel Nashed – 2 October 2024 08:42:00

I ran into this writing a new application published soon. It first looked like I got hit by the trailing CRLF.
But it turned out the buffer returned had the double len of what it should be.

This looks like a bug and there is already a SPR. HCL is investigating.
In my tests it is always double the size including the CRLF.

You could start searching for the end of the data. But how do you know where it ends?

My current work-around is to use the Content-Length header and truncate the string.

Here is a simple example of my work-around.

I would be interested if someone ran into similar issues. I tried all settings to prefer strings and UTF-8.
But this did not make a difference.

The returned JSON is not affected and returned differently.

I would like to hear from you if you ran into this too.
Or if you have different experiences and work-arounds.

-- Daniel


Function HttpGetRequest (URL As String) As String
     
     Dim Session As New NotesSession
     Dim headers As Variant
     Dim req As NotesHTTPRequest
     
     Dim ret As Variant
     Dim HeaderName As String
     Dim HeaderValue As String
     Dim ResponseCode As Long
     Dim ResponseText As String
     Dim ContentLen As Long
     Dim ResponseStr As String
     
     HttpGetRequest = ""
     ContentLen = 0
     
     Set req = session.CreateHttpRequest()
     
     req.maxredirects= 5
     req.Preferstrings = True
     
     Call req.SetHeaderField("Content-Type", "text/plain")
     
     ret = req.Get (URL)
     
     ResponseStr = req.ResponseCode
     ResponseCode = Clng (Strtoken (ResponseStr, " ",2))
     ResponseText = Strtoken (ResponseStr, " ",3)
     
     headers = req.GetResponseHeaders()
     
     Forall h In headers
             HeaderName = Strleft (h, ":")
             
             If (        HeaderName = "Content-Length") Then
                     HeaderValue = Strright  (h, HeaderName + ": ")
                     ContentLen = Clng (HeaderValue)
             End If
             
     End Forall
     
     If (200 = ResponseCode) Then
             HttpGetRequest = Replace (Left (ret, ContentLen), Chr(13)+Chr(10), "")
     End If
     
End Function



 Linux  Domino  Container 

Linux LSOF is causing 100% CPU load inside a container in some configurations

Daniel Nashed – 2 October 2024 07:10:53

This hit me a couple of times and now I found the root cause.
I am still thinking about how to best solve it for everyone. But here are the current facts.

LSOF is the Linux tool to list open files and is used by NSD.
It runs into issues when the security limits allow too many open files.

Background

The Linux security limits prevent processes to use too much resources.
Traditionally the security limits for number of open files was set quite low to 1024.


But some applications like Domino need a lot of open files because each lib, each open file and each network connection is a file under Linux.
The recommended number of maximal open files for the Domino user "notes" is 80000.

Usually you set this in /etc/security/limits.conf.

The container image sets this limit correctly. But it turns out that Docker or better containerd sets a much higher limit in some cases.
If Domino would switch the user like on a normal server, the security limits would be enforced and set to 80000.
But because the user the container is started with is already the unprivileged user "notes", the user is not switched.

Therefore the security limits are not changed by the Linux container.
Usually security limits for number of files should look like this by default:


ulimit -n
1048576

In case of certain Docker / containerd configurations the container runs with a 1024 times higher limit:

ulimit -n
1073741816

This causes the high CPU usage with LSOF.


Solution

A first solution would be to explicitly set the number of open files when starting the container.

docker run --rm -it --ulimit nofile=1048576:1048576 ubuntu bash

But probably it would be a better way to set the security limits before starting Domino inside the container to be on the safe side.
If the container launches with a too high limit an unprivileged user can still lower but not increase the limits.

I am currently testing if we should reduce the number of files to 1048576.
The only software currently affected is LSOF. But you never know and this would be a good defensive change.


SUPER.HUMAN.SOFTWARE

Daniel Nashed – 1 October 2024 20:20:20

You might want to call me a tree hugger. When I had to get a copy of a old Domino 5.0.5 pubnames.ntf copy for a friend I ran into this installer screen.
It has bee a while and it has been an interesting ride so far.

With Domino 12 and later releases we got a lot back from the old passion and energy back into Domino server development.
The last couple of releases have been a blast.

Notes & Domino always was and still is a remarkable software product. Some of the base concepts are still unmatched by other software.

I had never more fun working with Domino. I my different roles I have never created more code then in the last couple of years.
There is just not enough time to get all my favorite projects done. Specially the Open Source projects take a lot of time -- But it is worth every minute for me!

I have worked with quite a bunch of other software at customers in the last years.
None of them had the same functionality when it comes to flexible, integrated application development, long term support of functionality and cross platform support.
The grass always looks greener on the other side. But did you look back to what you had?

Seeing this install screen below I am recalling the old times and want to give kudos to all the friends around the globe who are on the same journey for all those years.

Domino release 5.x was a mile stone for Domino. So as the current releases are!
A while ago I updated a Domino 5.0.3 server on Linux in 30 seconds to Domino 12.0.2 inside a Docker container at the HCL factory tour in Milan.
Try that with any other enterprise messaging system..

I could not be more proud of being part of that journey.

-- Daniel


Now the R5 launch reminds me on something else ...  
I AM ... https://www.youtube.com/watch?v=rRAhCCGb7BY
And look who posted it at that time ..


Image:SUPER.HUMAN.SOFTWARE






 Domio 

Disabling XPages if not needed reduces open files and HTTP start/stop time

Daniel Nashed – 29 September 2024 20:50:03

While working on setup automation I often ran into HTTP startup challenges.
It can take up to 40-50 seconds until the HTTP task is started.

If you look at open files, you notice that each thread has more than 70 files open.
This sums up to up quite some files and the HTTP server start/stop time is much slower.

In case you don't use XPages there is a simple switch to disable the XPages run-time and only load the standard Java components.

notes.ini INotesDisableXPageCMD=1

I first had the impression Java in general would cause overhead on start. But my tests drilled down to XPages/OSGI.

Specially when you run a Traveler server, this can cause even more files open. Traveler auto configuration sets the HTTP threads to 400.
But Traveler uses an OSGI servlet. With Traveler you cannot disable the XPages run-time.

It is still a good idea to set the number of HTTP threads on your server to the number of sessions you need for a Traveler server.
If you are running a small server you might want to lower "Number active threads:" in server document.

-- Daniel


Here is the Traveler message when you still disable it. Traveler will not work without it.

Traveler: SEVERE  *system Configuration parameter INOTESDISABLEXPAGECMD=1 in the notes.ini file is preventing the Traveler Servlet from starting.
For Traveler to be functional, please remove this parameter.  See the Domino Server information center for more information on notes.ini parameters.
 Notes  Domino 

Notes/Domino Named Documents -- What are they and where/when to use them?

Daniel Nashed – 29 September 2024 07:22:23

Named documents are new in Notes/Domino 12.0.1. They are very similar to profile documents, but have less side effects.
Profile documents are cached and if used in a web application might end up being opened by may threads.

Also when updating profile documents, it can take a while until all parts of Notes/Domino see the document.
Like a Profile document a Named Document is not shown in any view. But beside that they behave like normal documents.

They are just searched by name -- similar to profile documents. Therefore they are really fast.
The only down side using Named Documents, they are only available in Notes/Domino 12.0.1 and higher.

Some system databases in Domino use them. For example they are leveraged in AdminCentral and AutoUpdate which are both introduced in version 14.0.
Here is a reference to designer help --> https://help.hcl-software.com/dom_designer/14.0.0/basic/H_GETNAMEDDOCUMENT_METHOD.html

Profile Documents will stay and they should still be used for static data.
But if you have data which is often changed, you should consider using Named Documents instead.

For example the Domino AutoUpdate Refresh token document is using a Named Document.

Tip: Add version check for 12.0.1+ clients

It would be probably a good idea adding a version check to your template when you start using newer functionality.

OTS support for profiles and named documents

I have just opened an AHA idea for profile document and named document support in Domino OTS.
If you find this useful, please vote. --> https://domino-ideas.hcltechsw.com/ideas/DOMINO-I-2880

-- Daniel
 fail2ban  nshmailx 

Replacing sendmail in Fail2Ban configuration with nshmailx

Daniel Nashed – 29 September 2024 06:35:50

nshmailx is a simple open source C based application, which started as a simple test tool.
But I figured out it can help me to remove the requirement to use separate tools for mailing on Linux.
In addition it allows to specify relay hosts and supports STARTTLS.

Most Linux tools use mailx (BSD version) for sending mail. You can also use nshmailx to replace mailx -- which supports the BSD mailx command-line for sending mail as well.

By default Fail2Ban wants to use sendmail for sending mail. This might introduce also unwanted additional packages on your server.

Here is a simple change to the Fail2Ban action in /etc/fail2ban/action.d/sendmail-common.conf.
You need to customize this script also to change the sender and destination for notifications.

The -file - option is important. It instructs nshmailx to read the e-mail body from stdin.

mailcmd = /usr/bin/nshmailx -from "" -to "" -file - -subject "Fail2Ban Report"

See https://github.com/nashcom/nsh-tools/tree/main/nshmailx for details about nshmailx.



 Linux 

Kail Linux Docker Container Image

Daniel Nashed – 28 September 2024 10:09:21

Kali Linux is a great distribution. It comes with very current software versions for OpenSSL, Curl and NGINX.

The distribution is actually specially interesting for "Penetration Testing, Security Research, Computer Forensics".
Therefore brings a lot of helpful tools for troubleshooting.

It's a Debian based distribution currently using glibc 2.40!.
The interesting part is that you can run it on any machine as a Docker image, can boot it from USB.
There is also a WSL image available.

It's probably not the Linux you want to use to run servers in a corporate environment.
But it would be great choice for troubleshooting.

I took a quick look and compiled my OpenSSL based tools without any surprises.

OK I must admit I also tested it as a base image for the Domino container image.
Because Ubuntu and Debian are on the supported list, it wasn't a big surprise it works as well.

This isn't really something you want to run in production and maybe it is not even a good idea to download the image in a corporate environment.
But testing out the boundaries of where software runs often brings valuable information about to make it better.
So I was curious what I find out ...


Kali Linux on WSL
The WSL documentation page is the most detailed one I have seen from any WSL based distribution.
And it is available in the Microsoft Store.

https://www.kali.org/docs/wsl/wsl-preparations/

Kali Linux Docker image page

https://www.kali.org/docs/containers/official-kalilinux-docker-images/


How to run it in a container

root@arm:~/.ssh# docker run -it --rm kalilinux/kali-rolling bash
┌──(root@xyz)
└─# cat /etc/os-release

PRETTY_NAME="Kali GNU/Linux Rolling"
NAME="Kali GNU/Linux"
VERSION_ID="2024.3"
VERSION="2024.3"
VERSION_CODENAME=kali-rolling
ID=kali
ID_LIKE=debian
HOME_URL="https://www.kali.org/"
SUPPORT_URL="https://forums.kali.org/"
BUG_REPORT_URL="https://bugs.kali.org/"
ANSI_COLOR="1;31"


ldd --version

ldd (Debian GLIBC 2.40-2) 2.40


openssl version

OpenSSL 3.3.2 3 Sep 2024 (Library: OpenSSL 3.3.2 3 Sep 2024)

curl --version

curl 8.9.1 (aarch64-unknown-linux-gnu) libcurl/8.9.1 GnuTLS/3.8.6 zlib/1.3.1 brotli/1.1.0 zstd/1.5.6 libidn2/2.3.7 libpsl/0.21.2 libssh2/1.11.0 nghttp2/1.63.0 ngtcp2/1.6.0 nghttp3/1.4.0 librtmp/2.3 OpenLDAP/2.5.18
Release-Date: 2024-07-31, security patched: 8.9.1-2
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

nginx -version

nginx version: nginx/1.26.0


 TLS  SSL 

Troubleshooting TLS connection problems -- Sometimes the problem is not what you think it is

Daniel Nashed – 27 September 2024 20:13:31

I got a help request today about configuring a SMTP connection for a relay host.
The admin configured an outgoing relay host. The connection was supposed to be SMTP over TLS on port 587. But it didn't work.

He got the following error:

SMTP: SSL bad peer certificate. Connection refused.


Then the admin enabled SSL debugging and got the following error message, but had no idea what it could mean:

[032279:000013-00007FC3EDA63640] 09/26/2024 11:35:41.57 SSLReadRecord> Rejecting connection - record contentType not in range for SSLv3 or TLS
[032279:000013-00007FC3EDA63640] 09/26/2024 11:35:41.57 SSLReadRecord> First 4 bytes of SSL/TLS record: 0x34 0x32 0x31 0x20
[032279:000013-00007FC3EDA63640] 09/26/2024 11:35:41.57 SSL_Handshake> After handshake state = HandshakeServerHello (5); Status = -6974

The most common root cause for this problem is that the other side is not speaking SSL/TLS at all.

In this particular case the other side was speaking STARTTLS instead of SMTP over TLS.
The start of the session is unencrypted until the STARTTLS command is issued and the handshake is performed.

I had a previous customer issue with a LDAP connection, which was expecting secure LDAP but the remote side was unencrypted.

Solution

In this case the solution was to switch the Domino server to STARTTLS (Negotiated TLS) in the server document for outgoing SMTP.
The LDAP case was solved by choosing the right encrypted port.


Conclusion

On first glance this shows up like a SSL/TLS connectivity problem. But knowing details about the SSL handshake helps you to get an idea about the root cause.
It is very unlikely a TLS version out of range could occur. So it is very likely the other side is not a SSL/TLS connection at all.

Now this blog post might help others to google the solution and we can prevent some support calls.


OpenNTF Webinar Slides -Unleash the Power of OTS (One Touch Setup)

Daniel Nashed – 27 September 2024 18:54:00

Domino One Touch Setup (OTS) has been around since Domino 12.0.

This week's webinar is a good starting point to look into Domino OTS.
But clearly 90 minutes are not sufficient for all the details.

You find more details in the complete examples.
And you can also raise issues if you have questions.

https://github.com/HCL-TECH-SOFTWARE/domino-one-touch-setup

Here is a link to slides and the video.

Slides

https://www.openntf.org/Public/presentations.nsf/0/B89491D386C4039B86258BA500584079/$FILE/openntf2024_domino_one_touch_setup.pdf

Video

https://youtu.be/VD09FwfsyiM

Image:OpenNTF Webinar Slides -Unleash the Power of OTS (One Touch Setup)


Recent Entries

Feeds

Links

    Archives

    • [HCL Domino]
    • [Domino on Linux]
    • [Nash!Com]
    • [Daniel Nashed]