Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

 
alt

Daniel Nashed

 

Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates

Daniel Nashed  22 July 2021 05:41:53

Domino 12 introduces a new architecture for certificate management that provides improved flexibility, functionality, and security.

Due to those enhancements and details in the SNI handling, the older kyr cache can't handle lookups for DNS names.

The improved mapping can only work with the new TLS Cache which has been introduced in Domino V12 with many improvements -- the main features are listed below.


This means in consequence that SNI (Server Name Indication) and client certificate authentication will only work using the new functionality described below.

So for proper SNI handling and for client certificate authentication you have to reconfigure using the new functionality.


HCL strongly recommends to move to the new functionality in Domino V12 in general!
Once you updated to Domino V12 you will see the following log message when starting HTTP:


TLSCache-HTTP: The Certificate Store database (certstore.nsf) is not available on this server. Consider running the CertMgr task to create this database to enable enhanced TLS certificate management.
Cert Manager is not loaded or configured


It really makes a lot of sense to move to the new functionality also for many other reasons.

Let me outline again the main improvements and also show how to import kyr files
...
  • Securely storing TLS Credentials (key + leaf cert + intermediates + root cert) in certstore.nsf)
  • Domain wide easy to use, modern UI database
  • It's not just for ACME/Let's Encrypt operations! It also supports manual certificate import and relieves you from using command-line tools like OpenSSL and kyrtool.
  • Easier trusted root central management without the need to look into kyr files. There is a separate view and form for trusted roots in certstore.nsf
  • Support for ECDSA and RSA keys in parallel
  • Full support for SAN name lookups
  • Full support for wildcard certificate lookups
  • On the fly update of TLS Credentials when the database changes

Here are the steps needed to get started with cerstore.nsf and how to import your existing kyr files automatically.

The new task has been really designed to make certificate operations easier. This includes the migration to the new functionality.


For more details check the OpenNTF session and slides i blogged about earlier and which is available as a joint session from OpenNTF and the HCL Software Academy.


Here is a screen shot to show part of the new UI.
I am using CertMgr since Domino V12 was released on my production servers with ECDSA and RSA keys.
And also with Let's Encrypt integration with my DNS provider Hetzner.

If using Let's Encrypt the certificates auto renew after 60 days and are available immediately thanks to the new TLS Cache.


Image:Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates


1. Configure CertMgr


To configure CertMgr, choose one Domino 12 or higher server (Win64 or Linux64) as the CertMgr server. This server is often the domain administration server.

Run  the following command on that server to create the certstore.nsf database and initialize the CertMgr task:


load certmgr


Add CertMgr to the servertasks notes.ini parameter or schedule it to run in a program document to ensure that it always runs.



2. Configure certstore.nsf on Web servers


Run load certmgr on Domino 12 Web servers in the domain.

The certmgr task automatically connects to the CertMgr server and creates a replica of the certstore.nsf database on the Web servers.

The CertMgr on these servers operates as a CertMgr client and replicates certstore.nsf automatically with the CertMgr server.


After certstore.nsf is present on a server, the TLS cache is loaded automatically when you start any internet server task like HTTP.
Any update to the certstore.nsf database on a server dynamically reloads the TLS cache.


3. Import existing kyr files


Use CertMgr to import TLS Credentials for existing kyr files.

To import all kyr files for a server run:


load certmgr -importkyr all


This command creates a TLS credentials document for each configured kyr file (server doc and internet sites if configured).


You can also import individual kyr files:


load certmgr -importkyr my-server.kyr



Support for trusted roots


The import functionality is only intended for the TLS Credentials (key, leaf certificate, intermediate certificates and the matching trusted roots).


Client certificate authentication requires the trusted root of the issuing CA for all client certificates which are intended to be authenticated.

Importing selected trusted roots is intended as a manual one time operation to review trusted roots which are still required.


You can export trusted roots with the kyrtool command line tool in the following way:



Windows example:


cd /d d:/domino/data

c:/domino/bin/kyrtool.exe show roots -v -k keyfile.kyr


Linux/AIX example:


cd /local/notedata

/opt/hcl/domino/bin/kyrtool show roots -v -k keyfile.kyr


For details how to import trusted roots and assign them to TLS Credentials check the following help topic:


https://help.hcltechsw.com/domino/12.0.0/admin/secu_addingtrustedroots.html


Here is an example imported trusted root which you can just assign to a TLS Credentials document.

Image:Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates



Comments

1Uwe Brahm  23.07.2021 9:53:20  Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates

Hi Daniel,

Thanks for shining some more light onto the importance of using the all new certmgr from now on with Domino 12.

Three Questions:

1) I see warning or error messages after my importing all of our *.kyr files and now the system reports for Http e.g.::

TLSCache-HTTP: Cannot add host entry [somedomain.of-mine.de] to cache -- Maximum number of entries reached (13)

Similar messages can be seen as I have also IMAP and DIIOP enabled on my servers

TLSCache-imap: Error adding hostname for TLS Credential for server [somedomain.of-mine.de]

2) Is it save to remove the *.kyr files from the servers' data directory after importing them?

3) If the answer for 2) is yes, should/can I remove the filenames of my *.kyr from the site configuraton documents?

Regards,

Uwe

2Daniel Nashed  24.07.2021 14:20:41  Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates

Hi Uwe,

Oooh .. this is a bug!

The TLS cache calculates the number of host enrties needed.

There are two fields in the TLS Credentials document "Hostname" and "HostnameIDN".

The IDN field should be only filled when at least one IDN is used. But it turns out that the form in some cases writes a textlist with an empty entry into the document.

An empty string in a textlist still has one entry. If you have multiple SANs the number of host entries is too small and not all entries are loaded into the cache.

A work-around is to remove the empty HostNameIDN field from the document.

The 13 entries in your case are the 3 TLS Credentials documents + 10 which is always added on top.

So this will only occur if you have multiple TLS Credentials for one server with many SANs and this odd behavior of the form adding the empty string.

The fix should be to always use the "Hostname" field for calculating the entries.

In my TLS Credentials documents I have a mix of documents which are correct and some that have this empty field.

It looks like it is happening for manual operations or imported keys. All my ACME created documents look good.

There is already a SPR. So no support ticket needed.

-- Daniel

3Cora Kwok  13.07.2022 7:42:21  Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates

CertStore: HTTP: Cannot find TLS Credential for the keyring file created by make_keyring,cmd [server.kyr] Invalid cipher(s) seen for server CN=Serv/O=Org.

how to correct this error ?

Username : Admin/Org

servername: Serv/Org

hostname: serv.org.com

4Daniel Nashed  13.07.2022 8:22:43  Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates

@Cora Kwok, I can't help with error descriptions like this.

I sent you a mail with questions. But my blog isn't a proper support channel.

You should open a support ticket at HCL. In your case this looks like a usage problem.

-- Daniel

5Sandor Bessenyei  17.11.2022 13:43:38  Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates

Hello,

I have a comment to Uwe Brahm's 2nd and 3rd question. I just found https://help.hcltechsw.com/domino/12.0.0/admin/inst_settingupsecurityforinternetsitedocuments_t.html?hl=internet%2Csite%2Cdocuments , where HCL says following about the key file name in the Internet Site documents:

Key file name

Specify one of the following:

If a certstore.nsf configuration is used, specify the host name of the server or any other certificate present in certstore.nsf for the server.

If certstore.nsf configuration is not used, specify the kyr file.

I think this is the answer to the questions from Uwe. As soon as the "key file name" field is updated according to the certstore.nsf (and of course HTTP Task is restarted), the old .kyr files are not needed any more.

@Daniel Nashed: can you please confirm this?

Thank you.

Kind regards,

Sandor Bessenyei

6Daniel Nashed  28.11.2022 22:42:16  Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates

@Sandor,

the new cache can lookup kyr files -- if configured in the TLS Credentials document. And it native understands DNS names. Once switched to cerstore.nsf the best way is to use DNS names in keyfile fields. You should also remove the old keys from the disk. You can import them via command-line automatically into certstore documented and they are enabled directly for the right server. But Domino does not delete the files from disk. The old cache is not disabled, but the new TLS Cache using the cerstore.nsf documents, is the first cache to be looked up. Only if no hit is found, the SSL layer will perform a lookup in the old cache and eventually hit a kyr file, cached from disk. This is done by design to allow a smooth migration. But you should really import and change the configuration.

I hope that helps?

-- Daniel

7Sandor Bessenyei  12.12.2022 14:44:30  Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates

Hello Daniel,

thanks for the reply, it is more clear now.

Kind regards,

Sandor Bessenyei

8JF Vogelaar  13.12.2022 12:35:23   iNotes Init: CredStore Key Decrypt error: The operational key needed by the Credential Store is not found

Hi Daniel & all,

I'm trying to solve the following error message after having to setup a (new) win server 2019 and installing Domino 12.0.2:

iNotes Init: CredStore Key Decrypt error: The operational key needed by the Credential Store is not found

iNotes Init: Credential Store Configuration not enabled, less secure mode.

Does anyone have an idea how to solve this?

Many thanks,

JF Vogelaar

9Gilou  11.10.2023 10:33:34  Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates

Hi,

I have 4 smtp servers Release 12.0.2FP2 for Linux/64.

I have import the p12 certificate using Certificate Store.

The first server is the CertMgrServer, each TLS Credential use one unique SAN per server RSA 2048 and Manual.

Except for the first server, i have indicate the SAN in the TLS key file name (server doc, Field Name: HTTP_SSLKeyFile)

At first, i do the same thing for the first CertMgrServer (san in the HTTP_SSLKeyFile) but i this case TLS not working).

The only way i found is to let empty this HTTP_SSLKeyFile for the CertMgrServer, and every think work fine

do you know why ?

10Daniel Nashed  12.10.2023 21:55:29  Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates

@Gilou, I can't help with a description like this with "not working".

A server document needs a kyrfile name entry. it can and should be the SAN of the certificate, because the TLS Cache is DNS name aware.

The kyfile name indicates to the Domino server that TLS is available. Without that it should not work unless you have internet sites configured.

So you really need an entry in this field! I would need more details to help you.

11KhandakarAmir Faisal  29.10.2023 12:51:00  Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates

How to set a specific domino server as default CertMgr-Server, not CertMgr: Client mode. Any ways to do it?

12KhandakarAmir Faisal  29.10.2023 12:54:27  Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates

And thanks for the nice explanation. its help me to understand how to work with Certificate Manager now.

13Gilou  07.11.2023 15:45:16  Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates

Hi Daniel,

In fact, with Jerome Deniau we have seen that in 1201FP1, if this HTTP_SSLKeyFile was the san of the certificate for the master keymgr server (as for the 3 others servers) then we have the error :

028541:000002-00007F51C2E56740] 06/30/2023 12:12:39 PM CertStore: Error decrypting TLS encryption item [PrivateKey] : The encrypted data has been modified or the wrong key was used to decrypt it

[028541:000002-00007F51C2E56740] 06/30/2023 12:12:39 PM CertMgr: Cannot open TLS Credentials document 0x946 : The encrypted data has been modified or the wrong key was used to decrypt it

[028541:000002-00007F51C2E56740] 06/30/2023 12:12:39 PM CertMgr: 1 unexpected TLS Credentials errors found -- Please see previous logged messages for details.

By leaving this field empty for this first server TLS was OK.

Next, i have update in 1202 FP2 this 4 servers, and the TLS was not working with the error (only for the master server):

PROD [root@s1124 ~]# openssl s_client -starttls smtp -crlf -connect firstsmtpserver01a:25

CONNECTED(00000003)

write:errno=104

---

no peer certificate available

---

No client certificate CA names sent

---

SSL handshake has read 269 bytes and written 324 bytes

---

New, (NONE), Cipher is (NONE)

Secure Renegotiation IS NOT supported

Compression: NONE

Expansion: NONE

No ALPN negotiated

SSL-Session:

Protocol : TLSv1.2

Cipher : 0000

Session-ID:

Session-ID-ctx:

Master-Key:

Key-Arg : None

Krb5 Principal: None

PSK identity: None

PSK identity hint: None

Start Time: 1699362720

Timeout : 300 (sec)

Verify return code: 0 (ok)

Then at last i have put the san in the HTTP_SSLKeyFile and its ok

thanks you Daniel

--

Gilou

Links

    Archives


    • [HCL Domino]
    • [Domino on Linux]
    • [Nash!Com]
    • [Daniel Nashed]