Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates
Daniel Nashed – 22 July 2021 05:41:53
Domino 12 introduces a new architecture for certificate management that provides improved flexibility, functionality, and security.
Due to those enhancements and details in the SNI handling, the older kyr cache can't handle lookups for DNS names.
The improved mapping can only work with the new TLS Cache which has been introduced in Domino V12 with many improvements -- the main features are listed below.
This means in consequence that SNI (Server Name Indication) and client certificate authentication will only work using the new functionality described below.
So for proper SNI handling and for client certificate authentication you have to reconfigure using the new functionality.
HCL strongly recommends to move to the new functionality in Domino V12 in general!
Once you updated to Domino V12 you will see the following log message when starting HTTP:
TLSCache-HTTP: The Certificate Store database (certstore.nsf) is not available on this server. Consider running the CertMgr task to create this database to enable enhanced TLS certificate management.
Cert Manager is not loaded or configured
It really makes a lot of sense to move to the new functionality also for many other reasons.
Let me outline again the main improvements and also show how to import kyr files...
- Securely storing TLS Credentials (key + leaf cert + intermediates + root cert) in certstore.nsf)
- Domain wide easy to use, modern UI database
- It's not just for ACME/Let's Encrypt operations! It also supports manual certificate import and relieves you from using command-line tools like OpenSSL and kyrtool.
- Easier trusted root central management without the need to look into kyr files. There is a separate view and form for trusted roots in certstore.nsf
- Support for ECDSA and RSA keys in parallel
- Full support for SAN name lookups
- Full support for wildcard certificate lookups
- On the fly update of TLS Credentials when the database changes
Here are the steps needed to get started with cerstore.nsf and how to import your existing kyr files automatically.
The new task has been really designed to make certificate operations easier. This includes the migration to the new functionality.
For more details check the OpenNTF session and slides i blogged about earlier and which is available as a joint session from OpenNTF and the HCL Software Academy.
Here is a screen shot to show part of the new UI.
I am using CertMgr since Domino V12 was released on my production servers with ECDSA and RSA keys.
And also with Let's Encrypt integration with my DNS provider Hetzner.
If using Let's Encrypt the certificates auto renew after 60 days and are available immediately thanks to the new TLS Cache.
1. Configure CertMgr
To configure CertMgr, choose one Domino 12 or higher server (Win64 or Linux64) as the CertMgr server. This server is often the domain administration server.
Run the following command on that server to create the certstore.nsf database and initialize the CertMgr task:
load certmgr
Add CertMgr to the servertasks notes.ini parameter or schedule it to run in a program document to ensure that it always runs.
2. Configure certstore.nsf on Web servers
Run load certmgr on Domino 12 Web servers in the domain.
The certmgr task automatically connects to the CertMgr server and creates a replica of the certstore.nsf database on the Web servers.
The CertMgr on these servers operates as a CertMgr client and replicates certstore.nsf automatically with the CertMgr server.
After certstore.nsf is present on a server, the TLS cache is loaded automatically when you start any internet server task like HTTP.
Any update to the certstore.nsf database on a server dynamically reloads the TLS cache.
3. Import existing kyr files
Use CertMgr to import TLS Credentials for existing kyr files.
To import all kyr files for a server run:
load certmgr -importkyr all
This command creates a TLS credentials document for each configured kyr file (server doc and internet sites if configured).
You can also import individual kyr files:
load certmgr -importkyr my-server.kyr
Support for trusted roots
The import functionality is only intended for the TLS Credentials (key, leaf certificate, intermediate certificates and the matching trusted roots).
Client certificate authentication requires the trusted root of the issuing CA for all client certificates which are intended to be authenticated.
Importing selected trusted roots is intended as a manual one time operation to review trusted roots which are still required.
You can export trusted roots with the kyrtool command line tool in the following way:
Windows example:
cd /d d:/domino/data
c:/domino/bin/kyrtool.exe show roots -v -k keyfile.kyr
Linux/AIX example:
cd /local/notedata
/opt/hcl/domino/bin/kyrtool show roots -v -k keyfile.kyr
For details how to import trusted roots and assign them to TLS Credentials check the following help topic:
https://help.hcltechsw.com/domino/12.0.0/admin/secu_addingtrustedroots.html
Here is an example imported trusted root which you can just assign to a TLS Credentials document.
- Comments [14]
1Uwe Brahm 23.07.2021 9:53:20 Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates
Hi Daniel,
Thanks for shining some more light onto the importance of using the all new certmgr from now on with Domino 12.
Three Questions:
1) I see warning or error messages after my importing all of our *.kyr files and now the system reports for Http e.g.::
TLSCache-HTTP: Cannot add host entry [somedomain.of-mine.de] to cache -- Maximum number of entries reached (13)
Similar messages can be seen as I have also IMAP and DIIOP enabled on my servers
TLSCache-imap: Error adding hostname for TLS Credential for server [somedomain.of-mine.de]
2) Is it save to remove the *.kyr files from the servers' data directory after importing them?
3) If the answer for 2) is yes, should/can I remove the filenames of my *.kyr from the site configuraton documents?
Regards,
Uwe
2Daniel Nashed 24.07.2021 14:20:41 Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates
Hi Uwe,
Oooh .. this is a bug!
The TLS cache calculates the number of host enrties needed.
There are two fields in the TLS Credentials document "Hostname" and "HostnameIDN".
The IDN field should be only filled when at least one IDN is used. But it turns out that the form in some cases writes a textlist with an empty entry into the document.
An empty string in a textlist still has one entry. If you have multiple SANs the number of host entries is too small and not all entries are loaded into the cache.
A work-around is to remove the empty HostNameIDN field from the document.
The 13 entries in your case are the 3 TLS Credentials documents + 10 which is always added on top.
So this will only occur if you have multiple TLS Credentials for one server with many SANs and this odd behavior of the form adding the empty string.
The fix should be to always use the "Hostname" field for calculating the entries.
In my TLS Credentials documents I have a mix of documents which are correct and some that have this empty field.
It looks like it is happening for manual operations or imported keys. All my ACME created documents look good.
There is already a SPR. So no support ticket needed.
-- Daniel
3Cora Kwok 13.07.2022 7:42:21 Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates
CertStore: HTTP: Cannot find TLS Credential for the keyring file created by make_keyring,cmd [server.kyr] Invalid cipher(s) seen for server CN=Serv/O=Org.
how to correct this error ?
Username : Admin/Org
servername: Serv/Org
hostname: serv.org.com
4Daniel Nashed 13.07.2022 8:22:43 Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates
@Cora Kwok, I can't help with error descriptions like this.
I sent you a mail with questions. But my blog isn't a proper support channel.
You should open a support ticket at HCL. In your case this looks like a usage problem.
-- Daniel
5Sandor Bessenyei 17.11.2022 13:43:38 Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates
Hello,
I have a comment to Uwe Brahm's 2nd and 3rd question. I just found https://help.hcltechsw.com/domino/12.0.0/admin/inst_settingupsecurityforinternetsitedocuments_t.html?hl=internet%2Csite%2Cdocuments , where HCL says following about the key file name in the Internet Site documents:
Key file name
Specify one of the following:
If a certstore.nsf configuration is used, specify the host name of the server or any other certificate present in certstore.nsf for the server.
If certstore.nsf configuration is not used, specify the kyr file.
I think this is the answer to the questions from Uwe. As soon as the "key file name" field is updated according to the certstore.nsf (and of course HTTP Task is restarted), the old .kyr files are not needed any more.
@Daniel Nashed: can you please confirm this?
Thank you.
Kind regards,
Sandor Bessenyei
6Daniel Nashed 28.11.2022 22:42:16 Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates
@Sandor,
the new cache can lookup kyr files -- if configured in the TLS Credentials document. And it native understands DNS names. Once switched to cerstore.nsf the best way is to use DNS names in keyfile fields. You should also remove the old keys from the disk. You can import them via command-line automatically into certstore documented and they are enabled directly for the right server. But Domino does not delete the files from disk. The old cache is not disabled, but the new TLS Cache using the cerstore.nsf documents, is the first cache to be looked up. Only if no hit is found, the SSL layer will perform a lookup in the old cache and eventually hit a kyr file, cached from disk. This is done by design to allow a smooth migration. But you should really import and change the configuration.
I hope that helps?
-- Daniel
7Sandor Bessenyei 12.12.2022 14:44:30 Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates
Hello Daniel,
thanks for the reply, it is more clear now.
Kind regards,
Sandor Bessenyei
8JF Vogelaar 13.12.2022 12:35:23 iNotes Init: CredStore Key Decrypt error: The operational key needed by the Credential Store is not found
Hi Daniel & all,
I'm trying to solve the following error message after having to setup a (new) win server 2019 and installing Domino 12.0.2:
iNotes Init: CredStore Key Decrypt error: The operational key needed by the Credential Store is not found
iNotes Init: Credential Store Configuration not enabled, less secure mode.
Does anyone have an idea how to solve this?
Many thanks,
JF Vogelaar
9Gilou 11.10.2023 10:33:34 Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates
Hi,
I have 4 smtp servers Release 12.0.2FP2 for Linux/64.
I have import the p12 certificate using Certificate Store.
The first server is the CertMgrServer, each TLS Credential use one unique SAN per server RSA 2048 and Manual.
Except for the first server, i have indicate the SAN in the TLS key file name (server doc, Field Name: HTTP_SSLKeyFile)
At first, i do the same thing for the first CertMgrServer (san in the HTTP_SSLKeyFile) but i this case TLS not working).
The only way i found is to let empty this HTTP_SSLKeyFile for the CertMgrServer, and every think work fine
do you know why ?
10Daniel Nashed 12.10.2023 21:55:29 Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates
@Gilou, I can't help with a description like this with "not working".
A server document needs a kyrfile name entry. it can and should be the SAN of the certificate, because the TLS Cache is DNS name aware.
The kyfile name indicates to the Domino server that TLS is available. Without that it should not work unless you have internet sites configured.
So you really need an entry in this field! I would need more details to help you.
11KhandakarAmir Faisal 29.10.2023 12:51:00 Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates
How to set a specific domino server as default CertMgr-Server, not CertMgr: Client mode. Any ways to do it?
12KhandakarAmir Faisal 29.10.2023 12:54:27 Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates
And thanks for the nice explanation. its help me to understand how to work with Certificate Manager now.
13Gilou 07.11.2023 15:45:16 Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates
Hi Daniel,
In fact, with Jerome Deniau we have seen that in 1201FP1, if this HTTP_SSLKeyFile was the san of the certificate for the master keymgr server (as for the 3 others servers) then we have the error :
028541:000002-00007F51C2E56740] 06/30/2023 12:12:39 PM CertStore: Error decrypting TLS encryption item [PrivateKey] : The encrypted data has been modified or the wrong key was used to decrypt it
[028541:000002-00007F51C2E56740] 06/30/2023 12:12:39 PM CertMgr: Cannot open TLS Credentials document 0x946 : The encrypted data has been modified or the wrong key was used to decrypt it
[028541:000002-00007F51C2E56740] 06/30/2023 12:12:39 PM CertMgr: 1 unexpected TLS Credentials errors found -- Please see previous logged messages for details.
By leaving this field empty for this first server TLS was OK.
Next, i have update in 1202 FP2 this 4 servers, and the TLS was not working with the error (only for the master server):
PROD [root@s1124 ~]# openssl s_client -starttls smtp -crlf -connect firstsmtpserver01a:25
CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 269 bytes and written 324 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1699362720
Timeout : 300 (sec)
Verify return code: 0 (ok)
Then at last i have put the san in the HTTP_SSLKeyFile and its ok
thanks you Daniel
--
Gilou
14bonnou 23.10.2024 14:27:02 Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates
Hello,
I want to renew a certificate in the certstore.nsf
The old certificate expires soon and I would like to replace it with the new one
I import the pfx that has the same name as the old one in "manual"
the hostnames are the same
the keyring file is different
I ran the commands "tell http quit" & "load http"
and I have the following messages when I run the "load http" command. Do you know why?
23/10/2024 15:50:24 TLSCache-http: Error decrypting TLS Certificate cache entry for [0x956]: The encrypted data has been modified or the wrong key was used to decrypt it
23/10/2024 15:50:24 CertStore: http: Cannot find TLS Credential for [travelersrv2026hcl.kyr] (RSA: 1, CDSA: 1): Entry not found in index
23/10/2024 15:50:24 HTTP Server: SSL Error: Keyring file not found, key ring file [travelersrv2026hcl.kyr], [Default Server]
Best regards,
Bonnou
15Daniel Nashed 23.10.2024 22:13:56 Important: In Domino V12 certstore.nsf is the recommended way for TLS/SSL server certificates
@bonnou,
my blog isn't the right place to ask for help, because I need more information.
first of all you don't need to restart HTTP with CertMgr and the new TLS Cache.
The problem is that your are uploading the TLS credentials from a client and the client needs to encrypt the private key for the server.
You might have imported first and then added the server, which in some template versions don't catch that the key needs to be re-encrypted.
What you can try is a manual re-encryption by setting the status of the document to "Update Server List".
Save the document and do not "Submit" it!. The server will re-encrypt the private key for the listed servers.
You can also remove the server and save the document, which will re-encrypt it for the new server list.
Then do the same again with the added server.
Without more information about your version and template version this is the best guess I could make of what happened.
For sure the private key is not encrypted with your server's public key where you are trying to use the TLS Credentials doc.
You can also use a "load certmgr -showcerts" on the target server where you want to use it to see if it is properly loaded.
Very likely you missed the order of operations when importing it. The import is a client flow which encrypts the private key with the server's public key.
After that your user has no chance to read the private key any more and cannot re-encrypt it for the server.
So you have to use a server based flow to re-encrypt it as described. I hope this helps?
-- Daniel