Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...


Daniel Nashed


Serious Linux security issue: pwnkit: Local Privilege Escalation in polkit’s pkexec (CVE-2021-4034)

Daniel Nashed  26 January 2022 00:19:55

This is pretty bad, but can be only exploited when the user already has local access.
Now looking into why the package is installed at all by default on many systems, which don't even run a graphical interface is not understandable.

And how can a bug like this happen at all?

Here is the source of the information -->

And here are some technical details:

The package is used to allow certain applications to gain root access -- similar to what sudo does.

The binary has the setuid set, which makes it execute as root.

The application itself has to control who should gain root access (setuid permission can be only set for binaries -- not for shell scripts).

In this case the binary has a really stupid bug, that allow a local attacker to get root access on the machine.

The package that was installed on my machine is the following:

polkit-0.115-12.el8.x86_64 : An authorization framework

Repo        : baseos

Filename    : /usr/bin/pkexec

And here is the binary affected:

ll /usr/bin/pkexec

sr-xr-x 1 root root 31032 May 26 2021 /usr/bin/pkexec*

It might be installed in many Linux machines of different distributions.

My CentOS Stream 9 had it not installed.

After quickly looking into it, I removed it from all my systems via

yum remove polkit -y

Here are details about the fix I found:

But clearly the fix for me is to remove the package.

RHEL 8 provided a new version dated December. But CentOS still does not have it.


Of course you have to check for yourself if you have software like "cockpit" using this software to have system tasks executed.
But for me this is the easiest way to get rid of software like cockpit -- a web based admin tool.
I wasn't even aware it is installed by default.

Usually software that is really needed will complain if you try to remove a package they have strong dependencies.

If you need to keep the binary and no fix is available yet, you can change the permissions as suggested in the security advisory instead.

This incident clearly lets me look into my machines again and question packages installed by default on the distributions I am using.

I blogged about VMware PhotonOS a while ago. They do a great job looking into dependencies they really need and stripping down their images.
Specially on the Docker side.

The Domino Docker image and other Domino base images don't have the software installed by the way.
It is already a more stripped down image, without admin tools like the networkmanager and others.

You really have to balance between convenience and security.
I would see that on a server those kind of admin tools are not needed and software that manages access to restricted resources that usually need "root" permissions.

There might be other software that needs it. So you have to carefully check what to remove.

Thanks Toni for your feedback. My statement from last night was about what I did and that was a decision I took based on the installed software packages that needed it on my machines.

It was a bonus to get those other tools removed in my case :-)

-- Daniel

Is anyone still using Domino 11 or earlier on Docker?

Daniel Nashed  24 January 2022 00:38:24
Domino 12 One-Touch setup has been a game changer for Domino on Docker deployments.

Moving Domino 11 and older to legacy support

The Community image supported Domino since version 9.0.1 and had an own setup routine based on modifying the legacy PDS file setup.
With Domino 12 this isn't needed any more and removing this logic along with other consolidation will make the Docker Community image a lot easier to maintain.

There are so many cool new features that I can't think of why many people want to run Domino 11 on a modern environment like Docker or K8s.
But there are still use cases for older images specially for developers who have to test with older versions.

For now I created a "next" branch in the repository and made most of the changes today.
The plan is to keep a "legacy" branch including all the existing documentation and move on making the next branch the new master branch.
I consolidated the build scripts first. Then moved to the internal deployment scripts in the container, the documentation and also the examples.

Rancher Desktop support

At the same time I changed the way the build and run-time environments are detected to add Rancher Desktop support.

Rancher Desktop 1.0 Beta is released and removed support for their own build environment "kim".
The only available build environment is now "nerdctl". I looked into it earlier, but it wasn't ready to go.
But with the latest changes a Domino build, push to registry and run works.

With the "next" repository the "nerdctl"  environment is auto detected in a WSL environment.

There is still work in progress for Rancher Desktop support and the other changes.

It's not ready for testing. But I would like to understand who is still on Domino 11 or older.
We will continue to support the older branch. But I would not expect many changes in the older versions.

-- Daniel

WSL - Select static subnet and pinned IP address

Daniel Nashed  22 January 2022 15:35:05

WSL is a very cool solution to run Linux applications.
But there is a limitation, which makes it difficult to use for Domino servers.

WSL assigns a random sub-net on every start.
I raised an issue at the WSL GitHub project
For now I wrote a small helper tool to work-around this limitation.

Here is the source code and the binary:

I only focused on re-creating the network and I am adding the IP as an alias.
There is also API to create a new end-point, but I am leaving this part to Microsoft to fix.

I hope this helps others..

-- Daniel

Which type of internet CA and certificates are you using in test and production?

Daniel Nashed  22 January 2022 12:44:50
As you might now I looked into many different CAs for testing and mostly use Let's Encrypt and other ACME based certs for production.
With ACME the type of certificates has dramatically changed.

For testing I built my own CA based on OpenSSL C code just to understand how it works.
There are other CAs like HashiCorp or the SmallStep CA.
Many of those CAs can be used as Sub-CAs in a corporate environments.

So for example you could use the SmallStep CA to distribute certs using the ACME protocol.

Is this something you are using today? Or are you using the more classical approach to use a CA from Microsoft as part of your AD?

e-mail CSR test lab offering

For test environments I have created my own lab CA to use e-mail to receive CSRs.
It's based on a pre-delivery agent requesting the certificate via HTTPS using the Lotus Script HTTP class to request a cert from my CA service using the native based OpenSSL code.
If anyone is interested to test manual flows for CertMgr, I can provide an e-mail CSR services for test lab environments with my own service.

It was also the perfect CA to use for the lab we setup in the DNUG CertMgr and certificates workshop.
I can't give away the CA itself. But there are many others to look into, which make more sense to use.

Your feedback

But I would also like to understand what you are using today in your environment for corporate certificates and also external trusted certificates.

Which internal CA do you use and how does the flow look like to get certificates?

Here are two references for CAs I am using in my lab and I have blogged about earlier.
And I am also adding an example certificate from my lab CA. I am using ECDSA with NIST-P521 and SHA512 signatures.
Also to test out if all of the software I am using fully supports ECDSA certs.


SmallStep CA

Example Certificate

The signature for the leaf itself is still SHA-256 for now.

Subject    : DE/NRW/Hilden/NashCom/IT/
SAN        : *.nashcom.loc
Issuer     : X2 Server MiniCA/DominoLab

Not Before : 2022.01.13 21:09:44
Not After  : 2023.01.14 21:09:44 (expires in 357.0 days)

Serial     : 08EA0B176C796E5B4671015625F96979
Sign Alg   : ecdsa-with-SHA256
KeyUsage   : DigitalSignature
Extensions : KeyUsage, ExtKeyUsage
ExtKeyUsage: TLS Web Client Authentication, TLS Web Server Authentication
Key        : ECDSA NIST P-521
ASN1 OID   : secp521r1

AuthKeyId  : 88:B3:3A:E1:0B:FA:A7:6A:47:C5:BB:BF:0C:F0:71:ED:E4:3D:9D:E7
SubjKeyId  : 9D:6E:92:5C:AD:88:8D:49:A7:C9:5B:09:89:B5:02:33:6E:7D:7C:86
MD5        : C0:6A:F3:88:63:81:10:C2:B2:01:5D:EF:A8:D6:C3:F5
SHA1       : 87:D3:27:F1:8F:D7:9A:5F:EE:D8:39:D0:20:10:8D:B6:07:72:EC:B3
SHA256     : AB:E7:8D:AD:A9:99:B0:C9:27:4F:18:82:69:33:E9:DF:B3:34:B3:66:5C:D6:29:3E:F1:B3:86:C9:27:FD:E0:E1


Subject    : X2 Server MiniCA/DominoLab
Issuer     : MiniCA-EC521/DominoLab

Not Before : 2021.07.26 05:31:28
Not After  : 2031.07.27 05:31:28 (expires in 9.5 years)

Serial     : 52C00B80CF7996CC5903A8CEB5B28A66
Sign Alg   : ecdsa-with-SHA512
KeyUsage   : CrlSign
Extensions : BasicConstraints, CA, KeyUsage
Key        : ECDSA NIST P-521
ASN1 OID   : secp521r1

AuthKeyId  : B3:C7:6D:6D:91:00:2A:EC:9D:3A:7A:06:46:6B:93:91:36:75:39:D0
SubjKeyId  : 88:B3:3A:E1:0B:FA:A7:6A:47:C5:BB:BF:0C:F0:71:ED:E4:3D:9D:E7
MD5        : 80:FF:91:E2:91:78:21:A4:D8:9F:4A:82:26:99:BF:CA
SHA1       : 3D:88:6E:18:03:F3:49:EC:51:BB:30:6D:06:A9:C1:7F:F6:60:CF:D7
SHA256     : B9:CD:37:F7:26:F8:8B:29:ED:7A:AD:84:A9:51:C0:D9:36:DE:17:CE:20:48:44:BB:CC:3F:82:F9:C9:E9:10:1D


Root       : MiniCA-EC521/DominoLab
Not Before : 2021.07.25 21:51:05
Not After  : 2031.07.26 21:51:05 (expires in 9.5 years)

Serial     : 50206CAFF845BDD08D4ABE6FBFCD2BAC
Sign Alg   : ecdsa-with-SHA512
KeyUsage   : CrlSign
Extensions : BasicConstraints, CA, SelfSigned, KeyUsage
Key        : ECDSA NIST P-521
ASN1 OID   : secp521r1

SubjKeyId  : B3:C7:6D:6D:91:00:2A:EC:9D:3A:7A:06:46:6B:93:91:36:75:39:D0
MD5        : F8:44:85:17:76:24:D1:E6:E9:BF:52:2D:86:86:00:66
SHA1       : 62:82:D0:D2:55:D0:36:36:6C:74:0D:10:BA:EC:02:D8:54:AF:2E:5E
SHA256     : 27:AB:C6:0C:CD:19:C1:5F:BA:8C:4F:FF:C4:28:8F:96:F8:D2:D7:62:FF:F7:D6:35:B1:EE:3F:EA:65:5E:D5:C0


Domino 12.0.1 CertStore Workshop final preparations for next week

Daniel Nashed  15 January 2022 09:37:24

Image:Domino 12.0.1 CertStore Workshop final preparations for next week

This is going to be the most interesting workshop I ever did.
I spent a lot of time preparing the workshop about awesome new Domino 12 functionality that deserves it to be features like this.

The presentation is done and the automated server setup is prepared.
I am working on some detailed fit and finish to make it an awesome experience as well :-)

Fully automated Domino lab setup

We will use One-Touch Domino setup for Linux with One-Touch setup in combination with my new templating to bring up lab servers.
This will be the first, but definitely not the last workshop using this technology to provide a lab server for every participant.

Domino 12 with One-Touch setup makes it easy to automate setups.

Domino 12.0.1 Certificate management

We will cover all details including the latest features and there will be no questions unanswered.
This includes the setup, Let's Encrypt, adding external DNS providers, integrations and many more details.
And it is going to be all hands-on on an own Linux machine with production style DNS setup.

But we also cover certificates in general.

I am looking forward to the workshop on Tuesday.

This is a commercial workshop in German for DNUG.

But we could repeat it in future in English...

-- Daniel

Important Domino 12.0.1 IF1 for customers using DAOS

Daniel Nashed  7 January 2022 21:25:12

This issue has been already in 12.0, but was discovered to late to be included into 12.0.1.
HCL worked hard to get IF1 out ASAP. There was a hotfix already available end of the year.
But IF1 take a bit longer than distributing a hofix -- there is more testing is involved.

If you are running on 12.x, and you are using DAOS, you should install IF1.
If you are on 12.0, I would recommend moving to 12.0.1 with IF1.
But if you have to stay with 12.0 there is a hotfix available from HCL support.

The problem occurs when databases are fixuped and many NLOs are involved.
It turned out that this also affects compact, which is using a fixup internally for most of the compact operations.

So you really should get IF1 installed if running DAOS.

Here is the official technote and info about IF1.

I just updated the Domino Community image and made 12.0.1 IF1 the new default.

-- Daniel

Domino 12.0 Fixup for databases with a large number of NLOs can lead to Domino Server performance degradation

Interim Fixes for 12.0.1x versions of HCL Notes/Domino

One Touch Setup meets automated lab setup

Daniel Nashed  2 January 2022 19:06:19

Domino One-Touch is the base for all the automation.
Once you started automation, it's only limited to your imagination.

Probably this will officially be available after our DNUG CertMgr and certificate hands-on workshop, were we will use the auto deployment of course.
Stay tuned for more functionality.. Looks like ideas are floating faster then  I write them.
But it's just the 2nd day of the year..

-- Daniel

Image:One Touch Setup and meets automated lab setup

Note: I had to short cut the fields for the configuration with a X_ instead of SERVERSETUP_ because I hit the limit for field len with some variables.

Image:One Touch Setup and meets automated lab setup

As some of you know I have my own lab registration database automating Hetzner server deployment including DNS leveraging their REST APIs.
I just added an agent, generating JSON on the fly based on the IP address of them machine.
And I have added an auto deployment and download routine to the "setup" of my start script.

In combination with JSON templating, I can download and customize JSON on-the-fly in any way.
I added a JSON based customizable menu, which can be also used to deploy other software.

The auto JSON generation is a Lotus Script agent, which dynamically replaced place holders in JSON files as shown above.
Probably I should make the agent available to show how simple this

For my lab environment setup database I also introduced a new well known URL to auto deploy configurations.
So for any host in an internet domain (like would look like this -->
Now that JSON configuration with templating makes deployment via JSON easier than editing a ENV file, JSON setup is the new default!

And there are many options I will document over time.
One example would be   " domino setup  "
The short cut  " domino setup auto " would provide the same functionality.
And if your host is part of the domain, you could even use  " domino setup auto " to hit the default configuration in

There is a lot new stuff coming. It's all prepared. But it's a lot of work to write up the documentation.

I just moved the Domino start script documentation in mark down. And this will be the new base to write up easier to read documentation.
The One-Touch functionality is also moved into a separate file, to keep rc_domino_script small and the functionality separated.
The updated scripts are in the develop branch of the Domino Docker Community image, but not added to the start script tar for now.

Agent Code

Surprisingly this isn't much code and sort of elegant.
It has some tricky parts. The replacement function needs exactly the right number of elements.
Having empty elements will cause the resulting text to be empty.. Took me a while to find - specially the Redim needs to be for n-1 if the index is zero based.

Sub Initialize
        Dim session As New NotesSession
        Dim db As NotesDatabase
        Dim WebDoc As NotesDocument
        Dim TemplateDoc As NotesDocument
        Dim doc As NotesDocument
        Dim TemplateItem As NotesItem
        Dim FormulaStr As String
        Dim item As NotesItem
        Dim x As Long
        Dim count As Long
        Dim OneTouchConfig As String
        Dim ret As Variant
        Print "Content-type: application/json"
        On Error GoTo error_handler
        Set db = session.Currentdatabase
        Set WebDoc = session.DocumentContext

        Set Doc = GetDocByFormula ( {(Form = "Server") & IP = "} + WebDoc.remote_addr(0) + {"} )

        ' If remote IP not found, get default document for testing
        If (Doc Is Nothing) Then
                Set Doc = GetDocByFormula ( {(Form = "Server") & (IP = "")} )
        End if

        If (Doc Is Nothing) Then
                LogError "OneTouchSetup - No counfiguration for " + WebDoc.remote_addr(0)
                Exit Sub
        End If

        OneTouchConfig = doc.OneTouchConfig(0)
        If ("" = OneTouchConfig) then
                OneTouchConfig = "Default"
        End If

        Set TemplateDoc = GetDocByFormula ( { (Form = "OneTouchSetup") & (TemplateName = "} + OneTouchConfig + {") })
        If (TemplateDoc Is Nothing) Then
                LogError "No Template Doc found"
                Exit Sub
        End If
        Set TemplateItem = TemplateDoc.Getfirstitem("Body")
        If (TemplateItem Is Nothing) Then
                LogError "No Template Item found"
                Exit Sub
        End If

        FormulaStr = TemplateDoc.Formula(0)
        If ( "" <> FormulaStr) Then
                ret = Evaluate (FormulaStr, doc)
        End If
        count = 0
        ForAll i In doc.Items
                x = InStr (1,, "X_", 1)
                If (1 = x) And ("" <> i.text ) Then
                        count = count + 1
                End If
        End ForAll

        If (0 = count) Then
                LogError "No Fields found"
                Exit Sub
        End If

        ReDim ReplaceFromArray (count-1) As String
        ReDim ReplaceToArray (count-1) As String

        count = 0
        ForAll i In doc.Items
                x = InStr (1,, "X_", 1)
                If (1 = x) And ("" <> i.text ) Then
                        ReplaceFromArray(count) = "{{ SERVERSETUP_" + StrRight (UCase (, "X_") + " }}"
                        ReplaceToArray(count) = i.text
                        count = count + 1
                End If
        End ForAll

        Print Replace (TemplateItem.text, ReplaceFromArray, ReplaceToArray, 1, 1000)

        Exit Sub

        LogError "OneTouchSetup - Error: " + Error()
        Exit Sub

Critical Exchange on Prem issue: Check your servers ASAP -- Malware agent fails to load in 2022 -- FIP-FS-"Microsoft" Scan Engine Failed to Load

Daniel Nashed  1 January 2022 11:56:20
Sadly not a good start into 2022 for some Exchange on Prem admins.
You should check your servers ASAP to avoid surprises Monday morning.
Depending on the configuration a failure of the "Microsoft Filtering Management Service" might stop your mail flow!

I have not seen anything on heise yet. But I got this tweed -->
And there is a forum entry -->
Plus a more detailed post -->

I am not speculating what could be wrong in detail with the date, since I am not an expert..

Sorry to hear and I hope this is is resolved soon!

Best of luck!


Image:Critical Exchange on Prem issue: Check your servers ASAP -- Malware agent fails to load in 2022 -- FIP-FS-"Microsoft" Scan Engine Failed to Load

Image:Critical Exchange on Prem issue: Check your servers ASAP -- Malware agent fails to load in 2022 -- FIP-FS-"Microsoft" Scan Engine Failed to Load

Using Domino CertMgr with NGINX & Co

Daniel Nashed  31 December 2021 16:03:16

Domino CertMgr is easy to use and shows all the certificates in a moder UI.  Most other ACME solutions are quite cryptic and mainly command line oriented.
Also you might want to operate CertMgr with ACME HTTP-01 challenges behind a load balancer, reverse proxy etc.

I took a look this morning and came up with some simple redirect rules first:

Redirect HTTP to HTTPS on NGINX or other load-balancers

The following config redirects all standard traffic from HTTP to HTTS.
And the ACME HTTP-01 challenge is redirected to your CertMgr server.
The server can be on the same or any other machine. And the target can be HTTPS with any type of certificate -- ACME validation does not check existing certs.
Only the challenge is validated -- as long the connection can be established.

You can add this to existing HTTPS configurations to allow ACME-01 challenges and also ensure users are conveniently redirected to HTTPS if HTTP is requested, without opening HTTP (port 80) on any Domino server.

  # Port 80 is redirected to 443. Only ACME challenges are redirected to CertMgr server.

    server {
        listen 80 default_server;
        listen [::]:80 default_server;
        server_name _;

        location /.well-known/acme-challenge/ {
            return 301$request_uri;

        location / {
            return 301 https://$host$request_uri;

TLS configuration with CertMgr certs

Now that we took care of redirecting challenges and HTTP traffic, what about using certificates created on CertMgr on your external host as well?
It could be a certificate from Let's Encrypt requested by the flow we just established -- redirecting ACME HTTP-01 challenges to another server.

Or it could be even a wildcard cert confirmed by ACME DNS-01 challenges.

Here is the documentation I came up with. If you have questions, feedback or ideas, let me know.

It contains links to a full TLS NGINX example configuation:

And also to an update script with the logic described in the referenced document.

This integration allows you to update certificates automatically once you have deployed an exportable key from CertMgr.

I blogged about similar flows before. But this is now a complete example with the widely used NGINX.

-- Daniel

Introducing Domino One-Touch JSON templating - Without manual JSON editing :-)

Daniel Nashed  30 December 2021 12:42:19

Domino One-Touch setup with JSON format is really great stuff -- I love it since day one.
But it might be a bit difficult to edit for many admins -- Even if you use an prepared template, you have to edit the JSON file in an editor to specify server name, etc.

Using variables ala Helm or Ansible makes a lot of sense and if you leverage existing JSON config templates, you might get away with not editing JSON at all.
Both using the {{ Variable }} syntax. But I am more used to the shell variable syntax: ${Variable}. So I implemented both.

I already uploaed the script and updated JSON configuration files to the develop branch of our GitHub repository as part of the start script.


The script itself can be also used for any kind of variable replacement. It checks template variables and prompts for a value -- see example below.

If a variable is already set in the environment, the value is used as default prompt.
On purpose all place-holders match the One-Touch environment variable names.
If you source in an existing configuration, those variables are automatically used.
And sticking with the existing standard variables makes it more consistent.

Example config extract with both syntax options:

 "serverSetup": {
   "server": {
     "type": "first",
     "name": "
     "domainName": "
     "title": "
     "serverTasks": "replica,router,update,amgr,adminp,http,certmgr"

Running the start script config command now automatically uses the new functionality when specifying JSON. The result is a ready to use JSON file.
This is just a simple way to replace template variables. In future it would be a good idea to have a configuration database to build ready to use special tailored configurations.
But I think this is already really helpful to have. Not only for workshops and lab environments..

The script itself is surprisingly short including all checks -->

-- Daniel

Example setup:

domino setup json 1
Using Domino config File [/etc/sysconfig/rc_domino_config]



SERVER_TITLE: Demo Server One Touch








One-Touch Domino Validation

[3909618:000002-00007F687C6B60C0] Success - /local/notesdata/DominoAutoConfig.json is valid with respect to schema dominoOneTouchSetup.schema.json



    • [IBM Lotus Domino]
    • [Domino on Linux]
    • [Nash!Com]
    • [Daniel Nashed]