Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...


Daniel Nashed


    Domino V12 Beta 3 CertMgr updates and a workaround

    Daniel Nashed  14 April 2021 05:59:17

    There have been a couple of important changes and additions in Beta 3 for CertMgr.
    Let me share the highlights from my point of view.  And also give some tips to avoid current smaller pitfalls I heard in the community from others testing.

    "Servers with access" selection is important

    In Domino V12 Beta3 is very important to select a "Server with access" for a TLS Credential that you create or update.
    Without a server name, the new TLS Cache will not use it.  In a one server scenario in Beta 2 and earlier the CertMgr server was always the same server as the server using the TLS Credential.

    Starting with Beta3 the Domain wide deployment is fully implemented and cerstore.nsf can be replicated in the domain (load certmgr on another server will automatically create the database).
    Even technically a CertMgr server can always decrypt the private key of the TLS Credential, the server will not use TLS Credentials without selecting them.

    You have to specify the server in "Servers with access" and let CertMgr re-encrypt the document (once you changed the field your are prompted).

    Work around for server select button

    Sadly the new server select button, which should have made the selection of servers easier, isn't working in the Beta3 template.
    You could either a.) past in the server names manually , b.) change the code of the button or c.) add a smart icon with the code listed below.

    This button can also help in own applications to select servers. There isn't a dialog only showing servers. The normal name dialog cannot be customized.
    It would be a great addition to be able to filter names field address dialogs! Like only show users, only groups only servers or a maybe even a custom formula?

    TLS Credentials cache

    There is a new TLS Credentials cache used instead of the older KYR file cache as soon certstore.nsf is available on a server.
    The new cache is fully taking benefit of the certstore.nsf database with new logic also covering wild card certificates and it can differ between RSA and ECDSA keys.

    And most important you don't need to restart any internet tasks like HTTP or SMTP to add or update a certificate :-)

    Trusted roots

    Beginning with Beta 3 also the trusted roots are part of the certstore.nsf.
    You can import your own trusted roots and make them available to TLS Credentials documents.

    Trusted roots are important for client certificate authentication and had to be manually imported to kyr files in earlier versions. Now it is an easy to use dialog in cerstore.nsf .
    You find the new dialog on the Security/Keys tab. The the example below -- of course I don't want to use Let's Encrypt for client certs .. This is just an example how to select those trusted roots.

    Image:Domino V12 Beta 3 CertMgr updates and a workaround

    Let's Encrypt and manual flow also uses new "Trusted Roots"

    Now that trusted roots are available in cerstore.nsf, having separate trusted root information in ACME account documents have been removed.

    The Let's Encrypt flow and also the "manual certificate" operations flow are automatically using the trusted roots in certstore.nsf.
    That means the trusted root is automatically added on those operations if found among trusted roots in certstore.nsf .

    Beginning with beta 3 certstore.ntf also has been updated with trusted roots for Let's Encrypt Staging and Production.

    But you might still see untrusted Staging certificates, because Let's Encrypt currently keeps changing the trusted root used for RSA keys.

    The new root for ECDSA keys and the Let's Encrypt production roots are working.

    I would ignore the missing trusted root warning for the Staging certificates. They cannot be used in production anyway.
    The Staging environment is intended for setup and testing. You can just switch to production any time if you feel like your configuration is OK.

    Pasting certificate PEM data with "automagical" processing

    Not everyone is eating certificates for breakfast.  It was always difficult to find the right trusted root and intermediate certificates.
    And the most difficult part was to bring the certificate chain into the right order.
    They had to be sorted from certificate to root in the right order with all the intermediate certificates.

    Beginning with Beta 3 you can past PEM data in any order and the server will try to make sense of what has been pasted.

    - Any order can be used
    - Duplicate certificates or intermediate certs are ignored
    - Not matching certs are ignored
    - Missing intermediates and trusted roots are added from the "Trusted Roots" documents in certstore.nsf

    So that means you can not only add your own CA trusted roots but also intermediate certificates into the new "Trusted Roots" documents.
    Intermediates will show up with a warning, that they are not a root certificate. But they are used for chain completion.

    The logic first finds the leaf certificate in the PEM data and will than auto complete the chain from the pasted PEM data and the Trusted roots in certstore.nsf if not found in the pasted PEM data.

    Finally the "Certificate Health" check is run to verify if the certificate chain is OK.
    So hopefully in all your certificate operations you will see a green "Valid"  for all your TLS Credentials in future as shown below.

    Image:Domino V12 Beta 3 CertMgr updates and a workaround

    Conclusion / Next Steps

    If you have not tried Domino V12 Beta and you are interested in certificate management for web server certificates, you should really have a look into this final beta -- now that all functionality is in place.

    I have been helping many customers creating kyr files for their environments and for most customers even after years this was still a mystery to them.
    In future you don't need cryptic tools OpenSSL to create keys, nor to create CSRs via OpenSSL. And the import process for certificates will be very straightforward.
    So beginning with Domino V12 you will not need any *.kyr files any more! But they are still supported!

    Oh in case you missed it ... This was already in beta 2 .. You can also use CertMgr to import into certstore.nsf your existing *.kyr files with a simple server console command.
    Either for a single .kyr file or for all .kyr files specified in server.doc and all internet sites ..

    I am looking forward to interesting presentations about CertMgr and certificate management with Domino V12.

    There is a lot to discover beside the simple and standard use cases. And there are also additional integration stories coming up.
    For example if you want to use certificates created by CertMgr for Sametime Meeting servers or a NGINX server etc.

    The integration options for REST based DNS API providers to create DNS TXT records for DNS-01 challenges will be an interesting topic for future blog posts and presentations.

    HCL provides two out of the box examples posted in the beta forum as ready to go DXL files to import for Cloudflare and Hetzner (German provider).
    And I have implemented and tested a couple of other provider interfaces like ACME DNS, Digital Ocean and some crazy variants with CNAME validations.

    There are plans to have an open source platform for the community to share those integrations -- Stay tuned.

    There is a lot to discover beside the standard documentation.. And there is a lot of room for your own integrations based on the provider interfaces.
    I have been using CertMgr already to automatically submit requests and pick up certificates via OpenSSL command-line and my own custom tools.

    -- Daniel

    Work around code for server select dialog issue

    Server:=@Subset(@DbName; 1);

    @If (Server=""; @Return (""); "");

    choice:=@PickList( [Custom]; Server : DB ; "Servers" ; "Servers with access:" ;"" ; 3);

    @SetField ("Servers"; @Unique(@Name([Canonicalize];choice:Servers)))


    1Brigitte Leitner  05.05.2023 16:02:56  Domino V12 Beta 3 CertMgr updates and a workaround

    Hello Daniel,

    We have a few questions about the Cert Manager:

    How can we change the admin server in the global settings? After a move, the original admin server disappears. Deleting the database and restarting the CertMgr unfortunately does not help, as it will search for the original admin server again and report an encryption error.

    Have you already conducted DNS API tests with OVH? If so, we would appreciate any suggestions.



    2Daniel Nashed  05.05.2023 20:08:07  Domino V12 Beta 3 CertMgr updates and a workaround

    Hi Brigitte,

    the server manager server is domain wide and you need a special command to re-encrypt all private keys to move it.

    the CertMgr server is stored in the directory profile to have all servers in the domain aware of the CertMgr server.

    When you look at the documentation of 12.0.0 you see that this was in the original design of the first version:

    -MIGRATETOSERVER servername

    Migrates the CertMgr process to a specified new server by using the new server to re-encyrpt all private keys in certstore.nsf. The new server must be a valid Domino server in the Domino domain with a replica of certstore.nsf.

    Run the command on the current CertMgr server. Before running the command, ensure all CertMgr processes are complete and then issue tell certmgr shutdown to shut down CertMgr.

    You find all currently supported providers here -->

    Your provider is not implemented. It's not complicated to implement one, if the provider has a good DNS API. There is a session from Engage which goes into very detail about how to write a provider interface. I can help as a partner to implement interfaces and I would do it for free if you publish, document it and support it in the OpenSource GitHub repository.



      • [HCL Domino]
      • [Domino on Linux]
      • [Nash!Com]
      • [Daniel Nashed]