eMudhra CA Key Compromise – how to remove the affected certs

The certs might be in different trust stores. Not all of them are in Domino admin's responsibility.
And there are many more applications to look into.

Maybe someone wants to provide all the information for Java in a similar way?
I am not a big Java fan. But Java in Notes/Domino works exactly the same way than any other application, which contains Java.

If you have questions let me know. If you run into bigger problems, I can also help remotely if needed.

I am going to look into the Linux trust stores next. But I would assume the Linux community is good at patching root certificates.
On Linux an easy test would be using curl. Same for Windows if it uses the right trust store.


I hope this helps a bit.

-- Daniel



cacerts.pem

In Domino the CA cert is in the current cacerts.pem in notesdata.
The current Domino server does not use the cacerts.pem file by default, but checks the Domino directory.

It would be still a good idea to remove the affected certs.
The cacerts.pem file is just a simple file, which you can edit or on clients deploy a new version.
That might be tricky if you don't have Mavel client or other software.
You would need to trigger some Lotus Script code for example.

Java Trust Store

In case of Java applications you would need to also patch or replace the JVM trust store.
Meanwhile the trust store is in PKCS12 (.p12) format. Even it is still named cacert with no extension and looks like the old JVM key store.

Removing and adding certs has been documented in many places.

Domino Directory

The Domino directory could contain the root too. I have checked in my Domino Directory and I can't find it.
Since it is a quite new root, it might not be in Domino Directory.


Finding out which roots to remove

There are actually two root certs which need to be removed.

1. The Affected root cert
2. An older already retired CA from Comodo

• I found the Comodo root in my Domino Diectory.
• But it is not listed in my current cacerts.pem.
• You might find it in older files. I got one but I did not check which version it belongs to. It was still on disk
   

I am listing the roots below for your convenience.

How to test your client

I wrote a small test button to check the cert. In my case I hit exactly this mixed situation where in one cacerts.pem the Comodo AAA Root cert was listed.
Removing the root made it worse, because then the Comodo cross cert was used.


Root certificates you should remove


1. The mainly affected cert


emSign ECC Root CA - G3
=======================
-----BEGIN CERTIFICATE-----
MIICTjCCAdOgAwIBAgIKPPYHqWhwDtqLhDAKBggqhkjOPQQDAzBrMQswCQYDVQQGEwJJTjETMBEG
A1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNobm9sb2dpZXMgTGltaXRlZDEg
MB4GA1UEAxMXZW1TaWduIEVDQyBSb290IENBIC0gRzMwHhcNMTgwMjE4MTgzMDAwWhcNNDMwMjE4
MTgzMDAwWjBrMQswCQYDVQQGEwJJTjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11
ZGhyYSBUZWNobm9sb2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290IENBIC0g
RzMwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAAQjpQy4LRL1KPOxst3iAhKAnjlfSU2fySU0WXTsuwYc
58Byr+iuL+FBVIcUqEqy6HyC5ltqtdyzdc6LBtCGI79G1Y4PPwT01xySfvalY8L1X44uT6EYGQIr
MgqCZH0Wk9GjQjBAMB0GA1UdDgQWBBR8XQKEE9TMipuBzhccLikenEhjQjAOBgNVHQ8BAf8EBAMC
AQYwDwYDVR0TAQH/BAUwAwEB/zAKBggqhkjOPQQDAwNpADBmAjEAvvNhzwIQHWSVB7gYboiFBS+D
CBeQyh+KTOgNG3qxrdWBCUfvO6wIBHxcmbHtRwfSAjEAnbpV/KlK6O3t5nYBQnvI+GDZjVGLVTv7
jHvrZQnD+JbNR6iC8hZVdyR+EhCVBCyj
-----END CERTIFICATE-----

#0
Root       : /C=IN/OU=emSign PKI/O=eMudhra Technologies Limited/CN=emSign ECC Root CA - G3
Not Before : 2018.02.18 18:30:00
Not After  : 2043.02.18 18:30:00 (expires in 16.3 years)

Serial     : 3CF607A968700EDA8B84
Sign Alg   : ecdsa-with-SHA384
KeyUsage   : CrlSign
Extensions : BasicConstraints, CA, SelfSigned, KeyUsage
Key        : ECDSA NIST P-384
ASN1 OID   : secp384r1


SubjKeyId  : 7C:5D:02:84:13:D4:CC:8A:9B:81:CE:17:1C:2E:29:1E:9C:48:63:42
SKI-SHA1   : 7C:5D:02:84:13:D4:CC:8A:9B:81:CE:17:1C:2E:29:1E:9C:48:63:42
SKI-SHA256 : CF:8B:70:15:7D:F6:C6:25:1C:22:90:20:CD:9B:3D:87:6B:E0:A1:FA:73:11:CC:46:19:31:F9:1A:67:D1:C4:B8

MD5        : CE:0B:72:D1:9F:88:8E:D0:50:03:E8:E3:B8:8B:67:40
SHA1       : 30:43:FA:4F:F2:57:DC:A0:C3:80:EE:2E:58:EA:78:B2:3F:E6:BB:C1
SHA256     : 86:A1:EC:BA:08:9C:4A:8D:3B:BE:27:34:C6:12:BA:34:1D:81:3E:04:3C:F9:E8:A8:62:CD:5C:57:A3:6B:BE:6B




2. Cross sing CA

The root CA is used to cross sign the  "emSign ECC Root CA - G3"  CA for legacy clients.

Form the status of the cert you can see that it isn't a big loss to delete the root document as well.
And you have to remove it to be on the safe side! This is not optional.



Comodo AAA Services root
========================
-----BEGIN CERTIFICATE-----
MIIEMjCCAxqgAwIBAgIBATANBgkqhkiG9w0BAQUFADB7MQswCQYDVQQGEwJHQjEbMBkGA1UECAwS
R3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHDAdTYWxmb3JkMRowGAYDVQQKDBFDb21vZG8gQ0Eg
TGltaXRlZDEhMB8GA1UEAwwYQUFBIENlcnRpZmljYXRlIFNlcnZpY2VzMB4XDTA0MDEwMTAwMDAw
MFoXDTI4MTIzMTIzNTk1OVowezELMAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0ZXIgTWFuY2hl
c3RlcjEQMA4GA1UEBwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0ZWQxITAfBgNV
BAMMGEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBAL5AnfRu4ep2hxxNRUSOvkbIgwadwSr+GB+O5AL686tdUIoWMQuaBtDFcCLNSS1UY8y2bmhG
C1Pqy0wkwLxyTurxFa70VJoSCsN6sjNg4tqJVfMiWPPe3M/vg4aijJRPn2jymJBGhCfHdr/jzDUs
i14HZGWCwEiwqJH5YZ92IFCokcdmtet4YgNW8IoaE+oxox6gmf049vYnMlhvB/VruPsUK6+3qszW
Y19zjNoFmag4qMsXeDZRrOme9Hg6jc8P2ULimAyrL58OAd7vn5lJ8S3frHRNG5i1R8XlKdH5kBjH
Ypy+g8cmez6KJcfA3Z3mNWgQIJ2P2N7Sw4ScDV7oL8kCAwEAAaOBwDCBvTAdBgNVHQ4EFgQUoBEK
Iz6W8Qfs4q8p74Klf9AwpLQwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wewYDVR0f
BHQwcjA4oDagNIYyaHR0cDovL2NybC5jb21vZG9jYS5jb20vQUFBQ2VydGlmaWNhdGVTZXJ2aWNl
cy5jcmwwNqA0oDKGMGh0dHA6Ly9jcmwuY29tb2RvLm5ldC9BQUFDZXJ0aWZpY2F0ZVNlcnZpY2Vz
LmNybDANBgkqhkiG9w0BAQUFAAOCAQEACFb8AvCb6P+k+tZ7xkSAzk/ExfYAWMymtrwUSWgEdujm
7l3sAg9g1o1QGE8mTgHj5rCl7r+8dFRBv/38ErjHT1r0iWAFf2C3BUrz9vHCv8S5dIa2LX1rzNLz
Rt0vxuBqw8M0Ayx9lt1awg6nCpnBBYurDC/zXDrPbDdVCYfeU0BsWO/8tqtlbgT2G9w84FoVxp7Z
8VlIMCFlA2zs6SFz7JsDoeA3raAVGI/6ugLOpyypEBMs1OUIJqsil2D4kF501KKaU73yqWjgom7C
12yxow+ev+to51byrvLjKzg6CYG1a4XXvi3tPxq3smPi9WIsgtRqAEFQ8TmDn5XpNpaYbg==
-----END CERTIFICATE-----



#0
Root       : /C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CNªA Certificate Services
Not Before : 2004.01.01 00:00:00
Not After  : 2028.12.31 23:59:59 (expires in 2.2 years)

Serial     : 01
Sign Alg   : sha1WithRSAEncryption
KeyUsage   : CrlSign
Extensions : BasicConstraints, CA, SelfSigned, KeyUsage
Key        : RSA 2048 bit

CRL        : http://crl.comodoca.com/AAACertificateServices.crl, http://crl.comodo.net/AAACertificateServices.crl

SubjKeyId  : A0:11:0A:23:3E:96:F1:07:EC:E2:AF:29:EF:82:A5:7F:D0:30:A4:B4
SKI-SHA1   : A0:11:0A:23:3E:96:F1:07:EC:E2:AF:29:EF:82:A5:7F:D0:30:A4:B4
SKI-SHA256 : CA:34:12:34:2C:B2:FE:91:4A:E1:82:B3:2A:5C:49:4B:B9:82:E6:47:B5:69:C5:27:70:C4:84:51:5F:CC:5E:E2

MD5        : 49:79:04:B0:EB:87:19:AC:47:B0:BC:11:51:9B:74:D0
SHA1       : D1:EB:23:A4:6D:17:D6:8F:D9:25:64:C2:F1:F1:60:17:64:D8:E3:49
SHA256     : D7:A7:A0:FB:5D:7E:27:31:D7:71:E9:48:4E:BC:DE:F7:1D:5F:0C:3E:0A:29:48:78:2B:C8:3E:E0:EA:69:9E:F4




Domino Directory entry to delete




Example Lotus Script code


Sub Click(Source As Button)
       
       Dim session As New NotesSession
       Dim req As NotesHTTPRequest
       Dim url As String
       Dim response As Variant
       
       url = "https://dv-g3b-aaa-emudhra.we-have-a-problem.com/"
       
       On Error Goto RequestError
       
       Set req = session.CreateHTTPRequest()
       req.Timeoutsec = 15
       req.Maxredirects = 0
       req.Preferstrings = True
       
       LogMsg "Testing: " & url
       
       response = req.Get(url)
       
       LogMsg "TLS connection succeeded"
       LogMsg "HTTP status: " & Cstr(req.Responsecode) + " Data: " + response
       Exit Sub
       
RequestError:

       LogMsg "Request failed"
       LogMsg "Error: " & Cstr(Err)
       LogMsg "Description: " & Error$
       
       Exit Sub
       
End Sub

• Comments [0]

Domino Container - Replay Domino and JVM cacerts

The Domino container image had certificate management for a while already.
But it was intended to add certificates and not remove them.

Removing certificates could be tricky. There are commands to do it.
But you have to hit the right certs in an automated way.

The new option allows to provide your own curated certificate list which will be always set when you provide it.
The functionality to add certificates still remains. The file will be replaced first and then the optional addition to the trust stores is executed.

What I am not changing is the Linux trust store.
Usually customers should build their specific base image with their OS level changes which are usually company wide.
So I am staying away from it for now unless someone has a strong requirement for it.

Different distributions have their own way to manage certs.
I added a cert addition. So I could build the replacement.

The container Linux layer is usually not used for any directly for client connections.

You find the documentation here:

https://github.com/HCL-TECH-SOFTWARE/domino-container/tree/main/dockerfiles/install_dir_domino/custom


-- Daniel

• Comments [0]

eMudhra CA Key Compromise – Why This Is a Serious Security Issue

On October 17, 2026, a serious security issue involving eight eMudhra intermediate Certificate Authorities (CAs) was reported.
The private signing keys of these CAs were generated with extremely weak randomness and can be recovered from publicly available information.

This means anyone who knows these keys can issue certificates for virtually any domain, not just domains previously using eMudhra certificates.

An attacker able to intercept network traffic could present a forged certificate for a legitimate website or API endpoint.
A vulnerable TLS client might accept that certificate as trusted, allowing the attacker to impersonate the server and potentially read or modify encrypted traffic.


Why Revoking Existing Certificates Is Not Enough

The compromised intermediate CA certificates have been revoked. However, revocation only protects clients that actually check and enforce it.
Many TLS implementations do not check certificate revocation by default. They validate the certificate chain against trusted root certificates but may never discover that an intermediate CA has been revoked.

As long as a client continues to trust the affected certificate chains without effective revocation checks, newly forged certificates can still be accepted.


What Administrators Should Do

There are two important actions:

1. Update trust stores and remove trust in the affected CA chains. This includes operating systems, applications, runtimes and containers. Removing an entire root CA may be necessary where more targeted distrust mechanisms are unavailable.
2. Verify certificate revocation handling. Ensure applications enforce revocation or another reliable mechanism for rejecting compromised CA certificates. Do not assume that certificate chain validation alone is sufficient.

Where to Check

• Browsers: Chrome, Firefox, Edge and Safari have already deployed protections.
• Operating systems: Windows, Linux and macOS trust stores need to be considered.
• Mobile devices: Android and iOS have received protections, but application-specific TLS implementations may behave differently.
• Applications: Java, OpenSSL, Go, Python and other runtimes may use independent certificate stores or verification mechanisms.
• Containers: Updating the host does not automatically update certificate bundles inside container images.

What About HCL Domino?

Domino administrators need to consider multiple certificate stores:

• cacerts.pem in notesdata directory: Contains affected trust anchors and needs to be reviewed and updated.
• Java cacerts: The bundled JVM uses a separate certificate trust store. Its contents and revocation behavior need to be verified for the specific Domino release.
• Domino Directory: The standard Domino Directory does not contain these public CA roots by default.
• Operating system: Linux and Windows certificate stores may be used by other applications and components running alongside Domino.
• certstore.nsf: Only contains Let's Encrypt roots by default and would only need to be checked if you added those roots

Updating one trust store does not necessarily protect applications using another.

How Can You Test Your Systems?

Dennis Jackson has published an excellent resource explaining the incident, providing cryptographic proof of the compromised keys and offering live tests for TLS clients.

https://emudhra.we-have-a-problem.com/


The tests can help determine whether a particular TLS client still accepts certificates issued through the compromised CA chains.
A successful test with your browser or curl does not automatically mean that other applications on the same machine are protected.

The most important lesson: Trusting a root CA without reliable revocation handling means trusting every certificate that can be issued through its subordinate CAs — including certificates signed with compromised keys.

Maybe we should start looking into the CA bundles we trust more closely. That's probably something to look at after this is resolved.
A while ago I looked into the list of root certs in trust stores. Usually vendors use the trust stores of the security stack they use (libcurl, JVM, etc).

• Comments [2]

Running Nomad Web on Ubuntu Touch Phone in a Container

This is the next level working with an Ubuntu Touch phone.
I have crafted an Ubuntu X Window Docker container and added RDP support.
Now I can RDP into my Ubuntu phone and start Nomad web in Firefox.

All the components are running on the same phone and there is still room for mow applications.

What I have tested first are the most challenging parts.
Normal Docker containers will be much easier to deploy.

I have an Yubikey working inside a container. And there are many other ARM applications in a Docker container which will work out of the box.
The Desktop container was built on another ARM machine and pulled down to the phone.

• Comments [0]

Ubuntu Phone - Checking CPU state and power consumtion

It is getting more interesting. The phone has 8 CPU cores.

- 4 Efficiency cores
- 4 Performance cores with a prime core with a bit more power

You can turn off individual cores. But that does not change idle power consumption.

ChatGPT wrote a nice script for me to include also battery and USB status.
That's pretty cool for performance / efficiency spelunking.

In my case I hooked up the phone with a power bank:

Turning off CPUs does not make a big difference. It looks like the kernel is pretty effective.

I might later tune maximum CPU frequency.
And I might add scripts to a GitHub project later.

What I found so far is pretty interesting.
I have a second script which shows the CPU/power information per interval.

• Comments [0]

Run Domino on a phone

Run Domino on a phone

The idea of running Domino on a phone came up when I was looking for a new small environment, similar to a Raspberry Pi.
There is no Domino support for ARM yet, but I have been looking into the ARM platform for other types of applications for quite a while. For example, NGINX and containers in general work very well on ARM.

Most of my projects already work on ARM, and I have a couple of ARM-based production servers running today.
For smaller emergency environments, I was looking for an even smaller form factor. My first idea was a tablet. But then, during a discussion with ChatGPT, the Ubuntu Touch project came up.
Ubuntu Touch only supports a limited number of phone models because it needs to be specifically adapted to the hardware of each mobile device.

The phone we picked is particularly interesting and is currently available refurbished. I got mine this week:
https://devices.ubuntu-touch.io/device/spacewar/

Getting Ubuntu Touch and Docker running

Switching the phone to Ubuntu Touch was the first adventure.
Once stage one was complete, I started looking into adding Docker. Installing additional system software on Ubuntu Touch can be challenging because the operating system itself is mostly read-only.
But Docker can be installed using Snap.

Once Docker was running and I had some native ARM containers up and running, another idea came up:

Why not run the x64/amd64 version of Domino in a container using emulation on the ARM phone?

The phone has eight CPU cores and 8 GB of RAM, which leaves plenty of resources for experimenting with amd64 emulation on ARM.
And that's where things started to get really interesting...

I’d continue from here with the actual emulation setup, then the first successful Domino startup, and finally a short section on performance and what this means in practice.

That gives the post a nice progression: phone → Ubuntu Touch → Docker/ARM → amd64 emulation → Domino.


 

• Comments [2]

Reproducible container builds on Ubuntu - Include phased updates

Ubuntu is phasing out some updates and only send them to a fractions of the machines.
Security updates are always included. For normal servers using phased updates is perfectly OK.

But for container builds it makes sense to always update all packages to have a reproducible build.

The command can be also used manually:

/usr/bin/apt-get -o APT::Get::Always-Include-Phased-Updates=true upgrade -y

I am adding it to the container build just for Ubuntu.

And I am cleaning up older versions for known distributions which are not available any more. Like Debian11 and Debian10 along with soon out of support Ubuntu 22.04 LTS

• Comments [0]

Updating Domino and Ubuntu

It has never been easier to update Domino and the OS stack -- provided you use the right base OS.


Domino Autoupdate

Domino introduced Autoupdate in 14.5. It allows to update Domino with a couple of clicks.
Currently this only updates the Domino release, fixpack and interim fix.
Later it will potentially update other packages like Verse etc.

It works on Windows and Linux.

Domino Container Image

The container image supports all add-ons. And updates a Domino server in seconds.

• Built an image
• Eventually distribute it to a registry to be used domain wide (like a Harbor registry)
• Shutdown the container, remove it and start it with the new image -> Done
   

Using a container image makes every installation clean and reproducible.
You can build it once and test it in a QE environment before deploying it in production.

Updating Ubuntu

Every distribution is a bit different. My favorite distribution is Ubuntu.
It has a couple of benefits:

• Very current packages and a good coverage of packages without extra repositories
• Native ZFS support
• Easy to use firewall
• Optional commercial support
   

And there is also an easy release upgrade path from one LTS release to the next release.

I have just upgraded all my production servers from Ubuntu 24.04.x LTS to 26.04.1 LTS.

Even if you don't use Domino Autoupdate, it is still a good way to find out Domino version and also provides details about the Linux distribution, kernel and glibc version.

• Comments [1]

Free ACME CAs, BSI RSA Requirements and Having a Plan B

Disclaimer: The Blog post content is not well formatted, because I gave up after 30 minutes with the Domino Blog template.
The text copied from somewhere else and reformatted at least does not work.
I can't spend more time on this today. This is breaking my nerves.

----


A customer raised a concern about the RSA-2048 intermediate CA used by Let's Encrypt.

There is nothing wrong with RSA-2048 from a current public WebPKI/browser perspective.
However, the German BSI TR-02102-2 recommends a minimum RSA key length of 3000 bits for certificate-signing keys. In practice, this means using RSA-3072 or larger.

BSI TR-02102-2 – TLS recommendations:

https://www.bsi.bund.de/SharedDocs/Downloads/DE/BSI/Publikationen/TechnischeRich tlinien/TR02102/BSI-TR-02102-2.pdf?__blob=publicationFile

This prompted me to test several free ACME providers with RSA-4096 certificates and look at the complete certificate chain, not just the server certificate.

ACME CA results

Let's Encrypt

- Free ACME: Yes
- Wildcard: Yes
- RSA CA chain: includes RSA-2048
- BSI RSA >=3000: No

Google Trust Services

- Free ACME: Yes
- Wildcard: Yes
- RSA CA chain: includes RSA-2048
- BSI RSA >=3000: No

ZeroSSL

- Free ACME: Yes
- Wildcard: Yes
- RSA CA chain: RSA-3072 / 4096 throughout
- BSI RSA >=3000: Yes

Actalis

Free ACME: Yes
- Wildcard: No (Not free at least)
- RSA CA chain: RSA-4096 throughout*
- BSI RSA >=3000: Yes

*Based on the certificate chain returned in my test.

Let's Encrypt

Let's Encrypt's current Generation Y RSA hierarchy uses RSA-2048 issuing intermediates.

With an RSA-4096 server certificate, the chain looks like this:

• RSA-4096  Server certificate
• RSA-2048  Let's Encrypt YR1
• RSA-4096  Root YR
• RSA-4096  ISRG Root X1
   
  

The important part is YR1. Even though the server certificate itself uses RSA-4096, the certificate chain contains an RSA-2048 issuing CA.

Therefore, increasing the RSA key size of the server certificate doesn't solve the BSI requirement.

Let's Encrypt could address this by introducing new RSA-3072 or RSA-4096 issuing intermediates and moving RSA issuance to them.
The current YR1/YR2/YR3 intermediates use RSA-2048 keys, so this would require new intermediate CA keys and certificates.

Let's Encrypt certificate hierarchy:

https://letsencrypt.org/certificates/

Google Trust Services

Google Trust Services also provides free ACME certificates including wildcard certificates.

I tested an RSA-4096 wildcard certificate. The resulting chain was:

• RSA-4096  Server certificate
• RSA-2048  Google Trust Services WR1
• RSA-4096  GTS Root R1
• RSA-2048  GlobalSign Root CA

Again, the relevant issuing CA, WR1, uses RSA-2048.

Google Trust Services therefore doesn't solve this particular requirement when using its current RSA certificate hierarchy.

Google Trust Services certificate repository:

https://pki.goog/repository/

ZeroSSL – RSA with wildcard support

ZeroSSL is the interesting result when RSA certificates are required.

ZeroSSL supports free ACME wildcard certificates, and the complete RSA certification path returned in my test uses at least RSA-3072:

• RSA-4096  Server certificate
• RSA-3072  ZeroSSL RSA DV SSL CA 2
• RSA-4096  Sectigo Public Server Authentication Root R46
• RSA-4096  USERTrust RSA Certification Authority
   
  

There is therefore no RSA-2048 certificate anywhere in this certification path.

This makes ZeroSSL a good option when RSA, wildcard certificates and the BSI RSA key-length recommendation are all requirements.

ZeroSSL ACME documentation:

https://zerossl.com/documentation/acme/

Actalis – a European alternative

Actalis is another interesting ACME provider, particularly for European customers. Actalis is based in Italy and provides free ACME certificates.
In my test, the complete RSA chain used RSA-4096, so it also meets the BSI RSA key-length recommendation.

The important limitation is that Actalis currently doesn't provide wildcard certificates through its free ACME service.
It is nevertheless an interesting European alternative for environments where wildcard certificates aren't required.

Actalis ACME:

https://www.actalis.com/acme-certificates.aspx

What about ECDSA?

There is another straightforward option: use ECDSA instead of RSA.

Let's Encrypt's current ECDSA hierarchy avoids the RSA-2048 issue. If all clients and applications in your environment support ECDSA certificates, this is a good option.

This gives us a fairly simple practical result:

• ECDSA + wildcard → Let's Encrypt
• RSA ≥3072 throughout + wildcard → ZeroSSL
• European CA without wildcard → Actalis

Have a Plan B

There is another lesson from this exercise which might be even more important than the RSA key sizes:

Don't depend on a single ACME CA.

When using a free public ACME service, a customer typically doesn't have an individual contract or SLA with that CA. Availability, policies, certificate hierarchies, rate limits or service conditions can change.
ACME is a standard protocol, so there is little reason to make certificate infrastructure unnecessarily dependent on one provider.

For example, even if Let's Encrypt is your normal CA, having a ZeroSSL ACME account configured and tested gives you an alternative. The reverse applies when ZeroSSL is your primary provider.

The important point is to test the alternative before you actually need it.

Domino CertMgr makes this easy

HCL Domino CertMgr supports the ACME providers discussed here and makes it easy to configure multiple ACME accounts centrally in certstore.nsf.
Certificate requests aren't tied to Let's Encrypt. Different ACME providers can be configured and used depending on the requirements of a particular certificate.

That makes a multi-provider strategy straightforward:

• ECDSA → Let's Encrypt
• RSA → ZeroSSL
• Plan B → Configure and test both
   
  

Actalis can additionally be useful as a European alternative where wildcard certificates aren't required.
Having more than one ACME provider configured and tested avoids unnecessary dependency on a single free service with which the customer might not have any contractual relationship.

For production certificate automation, having a working Plan B is a good idea regardless of which CA is the primary provider.

• Comments [0]

Economically Right-Sizing Domino Server Memory on Linux

More RAM is generally good for a Domino server. But more RAM is not automatically a good investment.
This matters more today because RAM has become significantly more expensive again, particularly when building servers with large memory configurations.

A useful concept from economics applies surprisingly well to server sizing: diminishing marginal utility.
The first additional GB of RAM can provide significant performance benefits. The next GB still helps, but usually a little less. Eventually, adding another 32 GB might provide only a small additional benefit.

The interesting question therefore isn't:

How much RAM can Domino use?

It is:

How much RAM provides a meaningful benefit for a workload?

On Linux, unused RAM normally doesn't remain unused. Linux uses available memory as filesystem cache.
A 64 GB Domino server might therefore look approximately like this:


64 GB RAM

8–10 GB   Domino + other processes
~50 GB     filesystem cache
remainder kernel / other usage

It is easy to look at this and conclude that the machine benefits from all 64 GB.
But there is an important distinction:

Memory being used is not the same as memory being required.

Linux will happily use additional RAM to cache filesystem data whenever memory is available.

Diminishing Marginal Utility

The first GBs of filesystem cache are extremely valuable because they contain the most frequently accessed data.
As the cache grows, increasingly less frequently accessed data is retained. Additional RAM still provides a benefit, but the incremental benefit gets smaller.

Conceptually:

Filesystem cache        Marginal benefit

8 GB                   Very high
16 GB                   High
24 GB                   Significant
32 GB                   Useful
48 GB                   Lower
64 GB                   Lower still

The numbers are only illustrative. The actual curve depends on the workload.The important principle is:

Doubling the filesystem cache does not double its performance benefit.

In my tests on one busy Domino mail server, for example, Linux was using around 55 GB of filesystem cache.
Reducing the available cache substantially did not result in a corresponding dramatic increase in physical disk I/O.
Even around 24 GB of cache, the server continued to behave very well.

That suggests that much of the additional cache was useful because it was available—not because the workload actually required it.

Different Domino Workloads Are Different

A mail server, Traveler server, directory server and SMTP server do not have the same memory requirements.
A busy mail server accessing many NSF databases can benefit substantially from filesystem cache.
Traveler is quite different. Its important memory consumers include the JVM, Domino HTTP and native process memory, while an HA configuration uses an external database for Traveler data.

Providing huge amounts of RAM to a Traveler VM simply to create a large filesystem cache may therefore provide little additional benefit.

RAM Isn't Cheap Anymore

For years, a common argument was: RAM is cheap. Just add more. That argument isn't nearly as convincing anymore.
Large server-memory configurations have become expensive again. And in virtualized environments, overprovisioning accumulates quickly.

Consider 20 Domino VMs with 32 GB more RAM each than their workloads meaningfully benefit from: 20 × 32 GB = 640 GB

That's 640 GB of physical infrastructure memory providing progressively smaller benefits.
The same memory could instead provide capacity for additional workloads, HA headroom, or simply reduce the cost of the infrastructure.

Right-Size Instead of Max-Size

The goal should not be to provide every Domino server with the largest possible filesystem cache.
The goal is to provide enough application memory, enough filesystem cache to capture the valuable part of the working set, and reasonable operational headroom.

Beyond that point, additional RAM still helps—but its marginal utility decreases.

That's the economic part of right-sizing:

Don't size memory simply because Linux knows how to use it. Size it according to the value that additional memory actually provides.

• Comments [0]