Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

 
alt

Daniel Nashed

 

Taking a closer look at Linux distributions for NGINX, Curl and OpenSSL for Domino

Daniel Nashed  6 October 2022 10:06:32

Martijn wrote an interesting post about NGINX versions
(
https://blog.martdj.nl/2022/08/26/nginx-as-reverse-proxy-on-centos-9-stream-a-problematic-combination/)

This inspired me to take a closer look into important software included in current Linux distributions.

In an earlier post I already showed how to update curl on Windows 11
(
https://blog.nashcom.de/nashcomblog.nsf/dx/replace-curl-shipped-with-windows-with-a-recent-version-not-build-by-microsoft.htm)

So the next step would be to look into Linux distributions.


Linux distributions


I took the current Docker images for the main distributions and checked the NGINX, OpenSSL and LibCurl/Curl version.

There is a dependency between LibCurl and OpenSSL when installed out of the box. All Linux platforms compile and ship LibCurl matching to the OpenSSL version on the platform.
Only when downloading LibCurl or other applications separately, there is a specific OpenSSL version usually bundled with the application.

Usually applications add libs to their application directory. This is often needed to run applications on different versions of the OS.
But it also means security patching your OS does not update important packages like OpenSSL, that are bundled with your applications!!
Many vendors lag behind patching their libs and they are also staying on older major releases for a long time.

On the other side it is not wise for an application vendor to rely on the OpenSSL version shipped with the OS.
Specially when supporting multiple Linux distributions and versions.


LibCurl and OpenSSL


Domino 12.0.1 ships with a matching OpenSSL 1.1.1 version in a separate lib / DLL in the Domino program directory.


LibCurl is linked directly into core Notes/Domino since Domino 10.x.


Linking code directly into a core application reduces conflict with other applications and also reduces the number of open files (each process would need file handles to open a Lib).
In addition this is also more secure, because nobody could sneak in a different Lib pretending to be the binary Domino expects to load.

Domino 12.0.2 links with OpenSSL 3.0.x and is not loading it dynamically.


Beginning with Domino 12.0.2 the new OpenSSL 3.0.x major version is linked into core Notes/Domino as well.

This means two important components are directly glued into core Domino.



Software versions in current Linux distributions


Lets take a look into the different distributions ship today and see which might fit best from security point of view.
There is one special platform: openSUSE installs OpenSSL 1.1.1 out of the box and you can manually install OpenSSL 3.0.x (openssl-3). That's why I listed both versions.

You might be surprised that version included in Domino 12.0.2 is even newer then the most current Linux distribution.


I marked really outdated versions which are problematic in red.
And I marked older versions in the corresponding major versions in yellow.
So a fully patched OpenSSL 1.1.1 version can be green. And an older OpenSSL 3.0.x version can be yellow.

Product NGINX Version OpenSSL Version LibCurl/Curl Version
NGINX Docker Image 1.23.1 1.1.1n, -- 15 Mar 2022 7.74.0
VMware Photon OS/Linux 4.0 1.22.0 3.0.3 - 3 May 2022 7.83.1
openSUSE Leap 15.4 1.21.5 1.1.1l  -- 24 Aug 2021,  3.0.1 -- 14 Dec 2021 7.79.1
Red Hat Enterprise Linux 9 UBI 1.20.1 3.0.1 -- 14 Dec 2021 7.76.1
Ubuntu 22.04.1 LTS (Jammy Jellyfish) 1.18.0 3.0.2 -- 15 Mar 2022 7.81.0
Debian GNU/Linux 11 (bullseye) 1.18.0 1.1.1n  -- 15 Mar 2022 7.74.0
Oracle Linux 9 1.20.1 3.0.1 -- 14 Dec 2021 7.76.1
Domino 12.0.2 - 3.0.5 -- 5 Jul 2022 7.83.0


Updated: 01.09.2022 --> I will recheck this periodically. Maybe with Docker automation


When you look at the Docker base images for the different Linux distributions we have a clear winner today: VMware Photon OS 4.0!

  • VMware Photon OS 4.0
    is up to date for all the three software packages we looked into.
  • openSUSE Leap 15.4
    is also a good choice -- and I hope they update their Curl version soon.
  • Ubuntu
    A good choice if you don't care about NGINX

A good way to run NGINX in an up to date version is a dedicated Docker container with the official image -->
https://hub.docker.com/_/nginx/.
This would give you always the latest version of NGINX and an update is easy to deploy. The older version of Curl should not be an issue, because the NGINX binaries are not using it (checked with ldd).

But for OpenSSL and LibCurl you are really depending on the base OS for the most critical parts like OpenSSH.
This is even more important on a Linux machine than on a Docker container image.
The Docker image runs locally and is usually protected by the base image.



HCL Notes macOS after Monterey 12.6 critical fix available on Flexnet 12.0.1FP1IF2

Daniel Nashed  22 September 2022 17:53:46

The OSX Monterey 12.6 update very unexpected caused the Notes client to fail starting.

12.0.1FP1IF2 is live now on the HCL License & Download Portal and can be deployed to address this SPR.
There are plans to release a 1101FP6 based Interim Fix once the fix is ready.

A very similar problem also occurred on iOS 15.7 and is fixed in iOS 16.0.
So it is apparently a chance introduced unplanned in macOS/iOS.

I am not a Mac used but some fellow Ambassadors reported their Notes Mac client is working again.

Don't install Notes 12.0.2 EAP5, which does not solve the issue completely.

Thanks to the Nomad and the Notes client team for this fast fix and updating the technote with status updates regularly.

For updated information check the technote: https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0100507

-- Daniel




Domino 12.0.2 One Touch setup with Let’s Encrypt certificates

Daniel Nashed  14 September 2022 23:50:33

One Touch setup (OTS) is quite powerful tool. But sometimes you need to know exactly what happens and combine functionality to make best use of it.
With Domino 12.0.2 OTS creates certstore.nsf automatically and you can let it create a MicroCA for you.
That's a standard flow in Domino 12.0.2 and easy to configure. Below you find the minimum configuration, which generates a RSA 4096 key.
There are additional settings to use a ECDSA key instead for example.


   "security": {
      "ACL": {
        "prohibitAnonymousAccess": true,
        "addLocalDomainAdmins": true
      },
      "TLSSetup": {
        "method": "dominoMicroCA"
      }
    }
  },


When the server is configured OTS invokes CertMgr for you to create the MicroCA and the certificate with 1 year expiration.
This MicroCA cert is only intended for testing and to have a safe server configuration out of the box.

But what if you want to use a Let's Encrypt certificate instead?

There is a quite simple way to just find and update the existing document with a appConfiguration.
And if you specify notes.ini CertMgr_ACCEPT_TOU=1 the ACME account license agreement will be automatically accepted (already part of 12.0.0).
The following JSON takes the existing document and just updates it to use Let's Encrypt (in my case the Staging account) and submits the request.

CertMgr will handle the quest and the TLS Cache will be updated immediately when the certificate is stored in certstore.nsf.

In case the Let's Encrypt request fails, the TLS Cache still holds the MicroCA certificate. The request is in error state, but the certificate is still valid.
This is intended by design to prevent automatic certificate request updates to fail existing services. The status of the document is running into an error, but the existing certificate is still used.
Having Let's Encrypt a standard OTS configuration might not help in many customer environments. So having it configured this way, will help admins who really know that their environment is properly configured for inbound HTTP-01 ACME challenges.
Making it a general feature would probably cause a lot of support calls.
Even reading this living troubleshooting doc should already help to understand why HTTP-01 ACME challenges are sometimes failing -->
https://github.com/HCL-TECH-SOFTWARE/domino-cert-manager/blob/main/docs/troubleshooting_acme_challenges.md

I hope this gives you additional ideas for OTS in your deployments.
-- Daniel

 "notesINI": {
      "CertMgr_ACCEPT_TOU": "1",
...

  "appConfiguration": {

    "databases": [

      {
        "filePath": "certstore.nsf",
        "action": "update",
        "documents": [
          {
            "action": "update",
            "computeWithForm": true,
            "findDocument": {
              "Form": "KeyFile"
            },
            "items": {
              "Status"      : "O",
              "Provider"    : "A",
              "AcmeAccount" : "LetsEncryptStaging"
            }
          }
        ]
      },
...

Download certificate chain without OpenSSL

Daniel Nashed  11 September 2022 10:17:08

Usually OpenSSL is the tool of choice for all type of certificate operations.

But what if no OpenSSL command line is available? Like in a Domino container where you can't install software?

After some research, I came up with the keytool, which is part of the JVM Domino ships.


It turned even out to be an easier way to download the certificate chain with this command line:


/opt/hcl/domino/notes/latest/linux/jvm/bin/keytool -printcert -rfc -sslserver 127.0.0.1 > certs.pem


This example connects to the local server over port 443 (default) and loads the complete certificate chain.
The option -rfc is important to write the certificate in PEM format.


Without the -rfc option, you get all the details about the certificates in the chain instead:

/opt/hcl/domino/notes/latest/linux/jvm/bin/keytool -printcert -sslserver 127.0.0.1

Certificate #0

====================================

Owner: O=Automation MicroCA Certificate

Issuer: CN=DominoMicroCA, O=AutoTestLab

Serial number: 69e50f549cb8e0d294ad2c6a884778c0

Valid from: Sat Sep 10 10:03:17 CEST 2022 until: Tue Sep 12 10:03:17 CEST 2023

Certificate fingerprints:

      SHA1: 4B:77:49:FB:95:3E:02:F5:8C:F7:C7:76:C1:F5:7F:EA:7B:60:CD:9C

      SHA256: 9E:E7:61:FA:D1:DA:57:AA:89:3F:A8:F8:F8:CB:69:D6:7F:DB:8A:18:C4:BB:01:2F:85:FD:1F:39:9B:69:8D:42

Signature algorithm name: SHA256withECDSA

Subject Public Key Algorithm: 384-bit EC (secp384r1) key

Version: 3


Extensions:


#1: ObjectId: 2.5.29.37 Criticality=false

ExtendedKeyUsages [

serverAuth

]


#2: ObjectId: 2.5.29.17 Criticality=false

SubjectAlternativeName [

DNSName: automation.notes.lab

]



Certificate #1

====================================

Owner: CN=DominoMicroCA, O=AutoTestLab

Issuer: CN=DominoMicroCA, O=AutoTestLab

Serial number: 4afa2efe1b11b03f88fc34d76b3574a2

Valid from: Sat Sep 10 10:03:17 CEST 2022 until: Mon May 26 10:03:17 CEST 2025

Certificate fingerprints:

      SHA1: 9C:B0:FD:60:25:B9:6D:8D:2C:F5:F3:CB:CF:F6:B8:73:EB:E9:22:EC

      SHA256: 7C:DA:2D:31:7A:7A:C9:B4:7B:24:36:0E:4F:A5:A6:81:D5:27:DB:45:57:03:59:61:2F:20:01:C0:BA:52:71:1B

Signature algorithm name: SHA256withECDSA

Subject Public Key Algorithm: 384-bit EC (secp384r1) key

Version: 3


Extensions:


#1: ObjectId: 2.5.29.19 Criticality=true

BasicConstraints:[

CA:true

PathLen:2147483647

]


#2: ObjectId: 2.5.29.15 Criticality=true

KeyUsage [

Key_CertSign

]


#3: ObjectId: 2.5.29.17 Criticality=false

SubjectAlternativeName [

DNSName: automation.notes.lab

]

Trivy vulnerability scan results for container images in a Notes database

Daniel Nashed  10 September 2022 11:49:40

Trivy (https://aquasecurity.github.io/trivy/)  is a very interesting tool to scan container images, configurations, GitHub projects and more.
I have used it to improve some of the configuration options of the container image.
Trivy supports a JSON formatted output with much more information, than the summary returned on a scan --> https://aquasecurity.github.io/trivy/v0.17.0/examples/report/

Here is an example command-line:

trivy image -f json -o /tmp/trivy_hcl_ubi9.json domino-container:ubi9


I took the result file and imported it into a Notes database. It's pretty simple using the Lotus Script class and the JSON parser.
An agent parses the result and updates the database.
It turns out some CVEs are reported more than once depending on the packages you have installed.
The database consolidates the report and provides a full overview.


Different base images have different CVEs

It's pretty interesting to see how different the Docker base images from different distributions deal with security patching.

I looked into current versions for important libs earlier and found out that VMware Photon is the clear winner, Followed y SUSE Leap and SUSE Enterprise.
That's going to be a separate post soon.

But also from security patch level point of view, VMware Photon 4.0 and the current SUSE Enterprise (https://registry.suse.com/) and Leap (https://hub.docker.com/r/opensuse/leap) images are ahead of the curve and have no single vulnerability reported for the packages required for Domino.

The Domino container project supports different base images to build on. The default is currently CentOS Stream 8, which is a commonly used based image.
I also have just added support for SUSE Enterprise -- which surprisingly does not install awk by default and caused the Domino installer to fail.
Not sure not sure it would make sense to switch to another default image from security point of view, because the RedHat base images are a behind VMware Photon and SUSE Leap and Enterprise versions.

Feedback?

What are you experiences with container security? Which is your favorite Linux base image and why?
Would the analysis database be interesting for others? Should I make it available open source as well?

-- Daniel

Examples:

Image:Trivy vulnerability scan results for container images in a Notes database



Image:Trivy vulnerability scan results for container images in a Notes database




K3s, Podman and a registry

Daniel Nashed  4 September 2022 20:08:34


Rancher Desktop is a great all-in-one desktop environment.
When running it with the Docker back-end you have all in one environment for development and run-time.

For a server, K3s (https://k3s.io) is my platform of choice. It is production ready and easy to deploy.

For Kubernetes, you always need a registry to pull images.
As soon you need custom images, you will need a registry to upload and download your image.

K3s allows you to configure private registries.
You could use any registry. I am just running the registry Docker image on Podman in my environment.

What I am describing here is just for a simple server test environment.
In production I would usw a Harbor registry (https://goharbor.io).


1. Create a local registry


podman run -d -p 5000:5000 --restart=always --name registry registry:latest

---

2. Add the insecure registry to Podman


vi /etc/containers/registries.conf.d/localhost.conf

[[registry]]
location = "localhost:5000"
insecure = true

---

3. Tag image and push to registry


podman tag hclcom/domino:12.0.2EAP4 localhost:5000/hclcom/domino:latest
podman push localhost:5000/hclcom/domino:latest

---

4. K3s add custom registry


vi /etc/rancher/k3s/registries.yaml


mirrors:
  localhost:
    endpoint:
      - "http://localhost:5000"

systemctl restart k3s


You find details here:  https://rancher.com/docs/k3s/latest/en/installation/private-registry/


OpenZFS finally available for RHEL/CentOS Stream 9

Daniel Nashed  3 September 2022 08:48:26

ZFS is my favorite file-systems!
Very flexible, with compression, de-duplication, snapshots, encryption support.


It is included in Ubuntu and SUSE -- but not RHEL and CentOS.

This means it needs to be build for the distribution.


CentOS Stream 9 was a pending deliverable and I just checked, it is finally available.


https://openzfs.github.io/openzfs-docs/Getting%20Started/RHEL-based%20distro/index.html

This was the last component missing for CentOS Stream 9.


Protecting AWS CLI credentials

Daniel Nashed  20 August 2022 12:13:46

Hmmm ... I have been looking for a solution for a while and I was really surprised there is no good solution out there.


Docker for example has credentials helper to protect passwords.

I just found an approach to protect my AWS credentials and this brings up the secure key/value store I started to implement first in Domino and then on Linux level.


Usually your credentials would look like the following -  A simple to read file, which cannot be protected.
I mostly use AWS CL it in a root user context, which has high permissions on the machine anyway -- but there are also ways to protect data from the root account!

Example: .aws/credentials

[default]

aws_access_key_id = XKIAZOXNXRHW7IPXCBBB

aws_secret_access_key = SY7/bI68F9Notess3NGFE/O7G70TdvpuT7wwV9Xi



You can configure AWS CLI to get it's credentials from another program.
Obviously the bash example is not intended to be used in production.
But I will probably extend my secure key/value store project to return AWS credentials in JSON format.

If you have an application, which could safely return data, the configuration would look like this:


Example: .aws/credentials with credentials helper


[profile developer]

credential_process = /local/aws/credentials.sh myaccount


Reference:
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sourcing-external.html


Secure Key/Value store idea


My vault idea isn't new and I played with remote and local callers.

For remote callers a TLS enabled interface authenticated by IP and secret could make sense

Local callers could be controlled by process calling and other aspects.


Building a secure key/value store would involve storing the data encrypted on disk and the lookup key could be defined in the configuration. And a separate secret could be used to decrypt the blob.


I don't know how much interest would be out there for a generic tool, which might even help us to protect Domino server.ids.

The first version will be probably for my current needs instead of a generic tool.


I have played with OpenSSL encryption APIs some weeks ago.

Using AES 256 for encryption would be a good idea and isn't that difficult to implement -But it would result independency on OpenSSL.


Any feedback about this new idea is really appreciated.

- Would be a generic tool interesting?
- Would you have specific use cases?
- How do you protect your AWS credentials today?


-- Daniel


Domino Community Docker Compose and multi-container support

Daniel Nashed  13 August 2022 23:06:05

Docker is an interesting environment also for local development and testing.
In some use-cases you might need Domino in combination with additional containers.


But this still only allows dominoctl to maintain one container.
With the latest update you can run different type of configurations.

Each user can have it's own configuration, located in .dominoctl in the user's home directory.
Or there can be project configurations in the current directory.

Here is the full documentation -->
https://nashcom.github.io/domino-startscript/dominoctl/#docker-compose-support

Domino Community Image - New Nomad Server install option

Daniel Nashed  30 July 2022 17:07:15

The Nomad server is a new offering to directly add Nomad support to your Domino server instead of using a SafeLinx server.
Recently I added a SafeLinx container to the Domino community project.

Now I am adding the Nomad Server to the Domino image as a new build options.

The build script now has a new option, which works like other options like -verse and -capi.
It just adds the Nomad server to the image and exposes the standard Nomad port 9443 in the new image.

Installing the Nomad Server is basically just an extract of the tar into the Domino binary directory.

The result is an image, with out of the box Nomad web support.

Even for a new server Nomad Web could be available configured via OneTouch Setup.
The only challenge here is to get a trusted certificate.

This can be Let's Encrypt or any other certificate you import via PEM or P12.
Or you could export the trusted root from a Domino MicroCA and add it as a trusted root to your browser.
The last step is probably only a good idea for a quick test environment.



./build.sh domino 12.0.2EAP3 -nomad
...

12.0.2EAP3          [OK] Domino_12.0.2_Linux_English_EAP3.tar
1.0.5               [OK] nomad-server-105-beta-domino-linux-1202.tgz

Links

    Archives


    • [IBM Lotus Domino]
    • [Domino on Linux]
    • [Nash!Com]
    • [Daniel Nashed]