Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

    OCSP stapling and why it matters -- Improve your HTTPS performance

    Daniel Nashed  January 30 2021 09:16:04 AM

    In one of my last posts I checked the Domino server HTTPS security rating.

    One of the features I configured was the OCSP stapling, which leads to a better rating.

    But why is OCSP stapling important and how can you check your server on your own?

    There is a great post in the Cloudflare blog explaining in detail why it is important and what they are doing to make their service more reliable for OSCP stapling.

    In very short

    OCSP in general is used by web-browsers to check the certificate status.

    With OCSP stapling the web server already provides a signed OCSP status of the certificate used.

    This avoids the browser to query the OCSP responder directly.
    Also in case the OCSP provider runs into performance issues or is even not available, your browser can still verify the certificate status.

    So this makes your web-server a bit more independent from your CA's OCSP responders.

    And also in case of free CA's like Let's Encrypt, we should be nice to them and reduce the load on their servers!

    For more details check the following great article:

    How to enable OCSP Stapling in Domino

    My tests have shown that just enabling the setting isn't sufficient SSL_ENABLE_OCSP_STAPLING=1.
    I have not been able to get it working without specifying the OCSP responder URL.

    You also have to provide the right OCSP responder URL.

    The address is part of your certificate and can be found using openssl like the following example shows.

    openssl x509 -in pluto.pem -noout -ocsp_uri



    set config OCSP_RESPONDER=

    After restarting the http task you can query the OCSP stapling status via openssl (see below).

    I also added the command to directly query the responder URL. This can be helpful for troubleshooting.

    The notes.ini parameter OCSP_RESPONDER needs to be set to enable the functionality.
    But this only defines the default responder, if no responder URL is found in the certificate.
    Today most certificates -- like Let's Encrypt contain the OCSP responder information.
    Domino reads that information and only uses the default if no information is found

    -- Daniel

    openssl s_client -connect -tlsextdebug -status

    OCSP Response Data:

       OCSP Response Status: successful (0x0)

       Response Type: Basic OCSP Response

       Version: 1 (0x0)

       Responder Id: C = US, O = Let's Encrypt, CN = R3

       Produced At: Jan 27 05:26:00 2021 GMT


       Certificate ID:

         Hash Algorithm: sha1

         Issuer Name Hash: 48DAC9A0FB2BD32D4FF0DE68D2F567B735F9B3C4

         Issuer Key Hash: 142EB317B75856CBAE500940E61FAF9D8B14C2C6

         Serial Number: 048635D3302F808EBEFDC34DF8244BDAEDD5

       Cert Status: good

       This Update: Jan 27 05:00:00 2021 GMT

       Next Update: Feb  3 05:00:00 2021 GMT

       Signature Algorithm: sha256WithRSAEncryption


    openssl ocsp -issuer ca.pem -cert pluto.pem -text -url
    OCSP Request Data:

       Version: 1 (0x0)

       Requestor List:

           Certificate ID:

             Hash Algorithm: sha1

             Issuer Name Hash: 48DAC9A0FB2BD32D4FF0DE68D2F567B735F9B3C4

             Issuer Key Hash: 142EB317B75856CBAE500940E61FAF9D8B14C2C6

             Serial Number: 048635D3302F808EBEFDC34DF8244BDAEDD5

       Request Extensions:

           OCSP Nonce:


    OCSP Response Data:

       OCSP Response Status: successful (0x0)

       Response Type: Basic OCSP Response

       Version: 1 (0x0)

       Responder Id: C = US, O = Let's Encrypt, CN = R3

       Produced At: Jan 27 05:26:00 2021 GMT


       Certificate ID:

         Hash Algorithm: sha1

         Issuer Name Hash: 48DAC9A0FB2BD32D4FF0DE68D2F567B735F9B3C4

         Issuer Key Hash: 142EB317B75856CBAE500940E61FAF9D8B14C2C6

         Serial Number: 048635D3302F808EBEFDC34DF8244BDAEDD5

       Cert Status: good

       This Update: Jan 27 05:00:00 2021 GMT

       Next Update: Feb  3 05:00:00 2021 GMT

       Signature Algorithm: sha256WithRSAEncryption


    WARNING: no nonce in response

    Response verify OK

    pluto.pem: good

           This Update: Jan 27 05:00:00 2021 GMT

           Next Update: Feb  3 05:00:00 2021 GMT

    Domino V12 Let’s Enrypt DNS-01 Challenges delegating a sub-domain to Digital Ocean

    Daniel Nashed  January 24 2021 05:45:43 PM

    Domino V12 Beta 1 supports DNS-01 challenge validation for Let's Encrypt and other ACME providers.

    The beta ships with two providers available in DXL file ready to import (the DXL file can be found in the Beta form).

    The beta forum contains a DXL file with 1. Cloudflare Inc and 2. Hetzner GmbH (a German provider) API as reference implementations.

    I took the configuration as a reference and implemented a Digital Ocean configuration.
    This option turns out a free solution which is very convenient for testing.

    Delegate a sub domain via NS records

    For my tests I delegated a sub-domain of an existing domain at another provider to Digital Ocean.

    To delegate a sub-domain I just created Name Server (NS) records pointing to Digital Ocean.



    In the next step I just added the sub-domain to my Digital Ocean account.

    Having this in place, I can now use the Digital Ocean DNS API to write DNS TXT records to validate DNS-01 challenges for Let's Encrypt/ACME.

    The DNS sub-domain delegation ensures the DNS servers at Digital Ocean are responsible for DNS requests for the sub-domain.

    CertMgr DNS Provider Configuration

    I have created a new configuration using the HTTP Request with @Formulas listed below.

    Now I can define a registered domain leveraging my configuration.

    The DNS Provider Configuration is pretty straightforward. You just need to configure a ADD and a DELETE operation in this case.

    The authorization token you generate at the Digital Ocean account is the only parameter you have to set, beside the registered domain.

    Having this configuration in place, you can immediately start registering certificates using any type of ACME provider.

    TLS Credentials
    Status: Issued  
    Domino Server Names:
    Status: Valid
    Certificate Expiration: Sat 24.04.2021 17:41:26
    Certificate Renew Date: Thu 25.03.2021 16:41:26
    Certificate Provider: ACME
    ACME Account: LetsEncryptStaging
    Certificate Authority:
    Key Type: ECDSA
    Curve Name: NIST P-384


    While developing a new provider configuration, there are 3 options, which are helpful.

    1. The "Test Formula" action helps you to simulate the results

    2. The "Insert Field" action allows to insert any of the standard fields without looking them up or typing them

    3. The DNS Provider Trace option shows all input data and output results to help figuring out the right parameters.

    For example in this case the "" field contains the DNS record ID returned by the ADD operation.

    This field is used in the DELETE operation to identify the created DNS TXT record.

    Here is my working Digital Ocean configuration at a glance showing both documents (config and account for my sub-domain).

    -- Daniel

    DNS Provider Account
    Registered Domain:
    Account Name: @ DigitalOcean
    Status: Enabled  
    DNS Provider Configuration: DigitalOcean  
    Configuration Values
    DNS Zone:
    email Address:
    Authorization Key:
    Authorization Token: 77931d012b385ed4def25412981d33a33be65a553d373c65a61196e5c170xxx *)

    *) No this is not my real token.


    DNS Provider Configuration
    Type: HTTP Request
    Status Formula: @if ( != ""; 200; 400)
    Request URL:
    DnsProviderDelay 42
    HTTP Request Tracing: Enabled  
    HTTP Add Request
    Request Type: POST
    URL Formula: cfg_URL + "/" + param_RegisteredDomain +"/records"
    Header Formula: ( "Content-Type: application/json" ) : ("Authorization: Bearer " + cfg_AuthToken )
    Post Data Formula: '{"type":"TXT","name":"' + param_DnsTxtName+ '.","data":"' + param_DnsTxtValue + '","ttl":30}'
    HTTP Delete Request
    Request Type: DELETE
    URL Formula: ID_TXT:@Text(; @if (ID_TXT= ""; ""; cfg_URL + "/" + param_RegisteredDomain + "/records/" + ID_TXT)
    Header Formula: ( "Content-Type: application/json" ) : ("Authorization: Bearer " + cfg_AuthToken )
    Post Data Formula:

    Domino 12 Beta 1 - HTTPS Review & Ratings

    Daniel Nashed  January 24 2021 07:03:46 AM

    Now that Domino V12 Beta1 is available, I took the opportunity to look into the current state of TLS and ciphers in Domino.

    Rating of the results

    You can see from the rating that we are at a very high score.
    Even without TLS 1.3 support, this already looks great!

    And looking into details you see, that you could switch to ECDSA keys already for most web servers.

    If are are interested in the details and want to test your own server -- even if not exposed to the internet .. continue reading and see the details from the commented results.

    -- Daniel

    Image:Domino 12 Beta 1 - HTTPS Review & Ratings

    Test Setup

    - Requested a new TLS certificate from Let's Encrypt using the new CertMgr integrated into Domino V12.

    - After receiving the certificate, I redirected HTTP to HTTPS (if you have HTTP enabled HSTS isn't working).

    - I set the following notes.ini variables to enable HSTS and OCSP for Let's Encrypt production:

    set config OCSP_RESPONDER=


    set config HTTP_HSTS_MAX_AGE=17280000

    Test Tool ""

    Even my server is reachable over the internet, I have been using a very cool shell script available on GitHub -- which will also work for internal hosts.

    Beside that I have looked into a SSL Labs test result, confirming my findings.

    yum install -y git bind-utils

    mkdir -p /local/github

    cd /local/github/

    git clone

    Details looking into output

    Let's have a closer look of the current state.

    Domino V12 introduces support for ECDSA crypto. So I requested a ECDSA certificate from Let's Encrypt using CertMgr, creating a ECDSA NIST-P 256 key.


     A  ECDSA NIST-P 256 is comparable to a RSA 3072 bit key. This should be fully sufficient and provides good security with best performance.

     The ratings for NIST-P 256 and NIST-P 384 are the same in my tests.

     A  NIST-P 384 key would be comparable to a RSA key size of 7680 bit!

    So my recommendation would be to stay with NIST-P 256!

    Ciphers for ECDSA

    When using a ECDSA key the standard cipher list is ignored and the following two ciphers are enabled automatically -- and all other ciphers are disabled.
    So there are no additional settings you have to configure. And by the way TLS 1.0 is disabled by default in Domino V12 Beta1 already.



    You can see from the client simulation in the detailed report below, that those ciphers are available in most platforms.

    Only very old clients don't support them.

    New curves supported

    Domino V12 supports two new curves X25519 and X448 which are leveraged with ECDSA

    Specially Curve X25519 is an important addition and you can see in the test report, that most client simulations negotiated a connection leveraging X25519.

    Curve X25519 is not just faster, it is also an independent implementation, not influenced by the NSA -- in contrast to the NIST-P curves.

    It has become "the de facto alternative to P-256" also because it's an independent implementation and is the preferred Curve used for Domino.

    (See details here:

    Detailed look into the test results

    You can see that the script delivers comparable results to the SSL Labs website.

    The big benefit is that it works also for intranet hosts, without any external connections.

    And you can get the output different formats.

    Running the scan for HTTPS on the standard is simple. You just specify the hostname:


    The scan is also faster and uses your locally installed OpenSSL and DNS lookup tools.

    Let's have a look into the details.

    ###########################################################       3.1dev from
       (124f6f5 2021-01-22 12:21:01 -- )

         This program is free software. Distribution and

                modification under GPLv2 permitted.


          Please file bugs @


    Using "OpenSSL 1.0.2-chacha (1.0.2k-dev)" [~183 ciphers]

    on localhost:./bin/openssl.Linux.x86_64

    (built: "Jan 18 17:12:17 2019", platform: "linux-x86_64")

    Start 2021-01-24 06:27:14        -->> ( <<--

    rDNS (

    Service detected:       HTTP

    Testing protocols via sockets except NPN+ALPN

    SSLv2      not offered (OK)

    SSLv3      not offered (OK)

    TLS 1      not offered

    TLS 1.1    not offered

    TLS 1.2    offered (OK)
    -- Only TLS 1.2 is offered
    TLS 1.3    not offered and downgraded to a weaker protocol

    NPN/SPDY   not offered

    ALPN/HTTP2 not offered

    Testing cipher categories

    NULL ciphers (no encryption)                      not offered (OK)

    Anonymous NULL Ciphers (no authentication)        not offered (OK)

    Export ciphers (w/o ADH+NULL)                     not offered (OK)

    LOW: 64 Bit + DES, RC[2,4], MD5 (w/o export)      not offered (OK)

    Triple DES Ciphers / IDEA                         not offered

    Obsoleted CBC ciphers (AES, ARIA etc.)            not offered

    Strong encryption (AEAD ciphers) with no FS       not offered

    Forward Secrecy strong encryption (AEAD ciphers)  offered (OK)
    -- Forward Secrecy is offered

    Testing server's cipher preferences

    Has server cipher order?     yes (OK)
    -- Server provides the cipher order
    Negotiated protocol          TLSv1.2

    Negotiated cipher            ECDHE-ECDSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
    -- openssl negotiated the right cipher.
    Cipher per protocol

    Hexcode  Cipher Suite Name (OpenSSL)       KeyExch.   Encryption  Bits     Cipher Suite Name (IANA/RFC)










    TLSv1.2 (server order)

    xc02c   ECDHE-ECDSA-AES256-GCM-SHA384     ECDH 256   AESGCM      256      TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384

    xc02b   ECDHE-ECDSA-AES128-GCM-SHA256     ECDH 256   AESGCM      128      TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

    The server cipher order lists the two ciphers mentioned above, with a favor to the more secure cipher



    Testing robust forward secrecy (FS) -- omitting Null Authentication/Encryption, 3DES, RC4

    FS is offered (OK)           ECDHE-ECDSA-AES256-GCM-SHA384 ECDHE-ECDSA-AES128-GCM-SHA256

    Elliptic curves offered:     prime256v1
    -- Curve offered is what we expect

    Testing server defaults (Server Hello)

    TLS extensions (standard)    "renegotiation info/#65281" "status request/#5"

    Session Ticket RFC 5077 hint no -- no lifetime advertised

    SSL Session ID support       yes

    Session Resumption           Tickets no, ID: yes

    TLS clock skew               Random values, no fingerprinting possible

    Signature Algorithm          SHA256 with RSA

    Server key size              EC 256 bits (curve P-256)
    -- yes we got a ECDSA NIST-P 256 key
    Server key usage             Digital Signature

    Server extended key usage    TLS Web Server Authentication, TLS Web Client Authentication

    Serial / Fingerprints        048635D3302F808EBEFDC34DF8244BDAEDD5 / SHA1 C264144A2B2FAE743441A4B51BC46722D8A1CDEC

                                 SHA256 C69718BF1A8BFD1436F4173659C21295E8232293D3918017280C2A4EF30BCA4D

    Common Name (CN)   

    subjectAltName (SAN)

    Trust (hostname)             Ok via SAN and CN (same w/o SNI)

    Chain of trust               Ok

    EV cert (experimental)       no

    Certificate Validity (UTC)   89 >= 30 days (2021-01-24 05:26 --> 2021-04-24 06:26)
    -- You can see that I just got the new certificate requested, because I had to compare P-256 and P-384 ;-)
    ETS/"eTLS", visibility info  not present

    Certificate Revocation List  --

    OCSP URI                -- our OCSP configuration worked
    OCSP stapling                offered, not revoked
    -- certificate looks good
    OCSP must staple extension   --

    DNS CAA RR (experimental)    not offered

    Certificate Transparency     yes (certificate extension)
    -- Let's Encrypt offers certificate transparency, which is the newer standard replacing "HTTP Public Key Pinning (HPKP)"
    Certificates provided        3

    Issuer                       R3 (Let's Encrypt from US)

    Intermediate cert validity   #1: ok > 40 days (2021-09-29 21:21). R3 <-- DST Root CA X3

                                 #2: ok > 40 days (2021-09-30 16:01). DST Root CA X3 <-- DST Root CA X3

    Intermediate Bad OCSP (exp.) Ok

    Testing HTTP header response @ "/"

    HTTP Status Code             401 Unauthorized  WWW-Authenticate: Basic realm="/"

    HTTP clock skew              +1 sec from localtime

    Strict Transport Security    200 days=17280000 s, includeSubDomains

    Public Key Pinning           --

    Server banner                Lotus-Domino

    Application banner           --

    Cookie(s)                    (none issued at "/") -- maybe better try target URL of 30x

    Security headers             X-Content-Type-Options: nosniff

                                 Cache-Control: no-cache

    Reverse Proxy banner         --

    Testing vulnerabilities

    All vulnerability test passed  -- But this is not new in Domino V12:

    Heartbleed (CVE-2014-0160)                not vulnerable (OK), no heartbeat extension

    CCS (CVE-2014-0224)                       not vulnerable (OK)

    Ticketbleed (CVE-2016-9244), experiment.  not vulnerable (OK), no session ticket extension

    ROBOT                                     Server does not support any cipher suites that use RSA key transport

    Secure Renegotiation (RFC 5746)           supported (OK)

    Secure Client-Initiated Renegotiation     not vulnerable (OK)

    CRIME, TLS (CVE-2012-4929)                not vulnerable (OK)

    BREACH (CVE-2013-3587)                    no gzip/deflate/compress/br HTTP compression (OK)  - only supplied "/" tested

    POODLE, SSL (CVE-2014-3566)               not vulnerable (OK), no SSLv3 support

    TLS_FALLBACK_SCSV (RFC 7507)              No fallback possible (OK), no protocol below TLS 1.2 offered

    SWEET32 (CVE-2016-2183, CVE-2016-6329)    not vulnerable (OK)

    FREAK (CVE-2015-0204)                     not vulnerable (OK)

    DROWN (CVE-2016-0800, CVE-2016-0703)      not vulnerable on this host and port (OK)

                                              no RSA certificate, thus certificate can't be used with SSLv2 elsewhere

    LOGJAM (CVE-2015-4000), experimental      not vulnerable (OK): no DH EXPORT ciphers, no DH key detected with <= TLS 1.2

    BEAST (CVE-2011-3389)                     not vulnerable (OK), no SSL3 or TLS1

    LUCKY13 (CVE-2013-0169), experimental     not vulnerable (OK)

    Winshock (CVE-2014-6321), experimental    not vulnerable (OK) - doesn't seem to be IIS 8.x

    RC4 (CVE-2013-2566, CVE-2015-2808)        no RC4 ciphers detected (OK)

    Running client simulations (HTTP) via sockets

    Browser                      Protocol  Cipher Suite Name (OpenSSL)       Forward Secrecy


    Android 4.4.2                TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)

    Android 5.0.0                TLSv1.2   ECDHE-ECDSA-AES128-GCM-SHA256     256 bit ECDH (P-256)

    Android 6.0                  TLSv1.2   ECDHE-ECDSA-AES128-GCM-SHA256     256 bit ECDH (P-256)

    Android 7.0 (native)         TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)

    Android 8.1 (native)         TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)

    Android 9.0 (native)         TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)

    Android 10.0 (native)        TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)

    Chrome 74 (Win 10)           TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)

    Chrome 79 (Win 10)           TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)

    Firefox 66 (Win 8.1/10)      TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)

    Firefox 71 (Win 10)          TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)

    IE 6 XP                      No connection

    IE 8 Win 7                   No connection

    IE 8 XP                      No connection

    IE 11 Win 7                  TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)

    IE 11 Win 8.1                TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)

    IE 11 Win Phone 8.1          TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)

    IE 11 Win 10                 TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)

    Edge 15 Win 10               TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)

    Edge 17 (Win 10)             TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)

    Opera 66 (Win 10)            TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)

    Safari 9 iOS 9               TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)

    Safari 9 OS X 10.11          TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)

    Safari 10 OS X 10.12         TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)

    Safari 12.1 (iOS 12.2)       TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)

    Safari 13.0 (macOS 10.14.6)  TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)

    Apple ATS 9 iOS 9            TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)

    Java 6u45                    No connection

    Java 7u25                    No connection

    Java 8u161                   TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)

    Java 11.0.2 (OpenJDK)        TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)

    Java 12.0.1 (OpenJDK)        TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)

    OpenSSL 1.0.2e               TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)
    --> OpenSSL 1.0.2 is not using the newer X25519 curve we prefer
    OpenSSL 1.1.0l (Debian)      TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)

    OpenSSL 1.1.1d (Debian)      TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)

    Thunderbird (68.3)           TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)

    You can see that only very very old OS/Browser/Software combination doesn't connect with those settings.

    And I would see it more as a feature that really old software, which isn't patched any more cannot connect to a server.

    Many clients already support the X25519 curve. All others use NIST-P 256 which is perfectly OK

    Rating (experimental)

    Rating specs (not complete)  SSL Labs's 'SSL Server Rating Guide' (version 2009q from 2020-01-30)

    Specification documentation
    Protocol Support (weighted)  100 (30)

    Key Exchange     (weighted)  100 (30)

    Cipher Strength  (weighted)  90 (36)

    Final Score                  96

    Overall Grade                A+

    The rate we get is in line what SSL Labs returned. Not only from the overall rate, but also from the detailed ratings.

    So a final score of 96 from 100 points looks very good to me!

    Done 2021-01-24 06:27:56 [  43s] -->> ( <<--

    The whole test took 43 seconds. You can run this against all your internal servers and see what you get back.

    This can be run also against SMTP servers and other services!

    Here is the command line for SMTP/STARTTLS:

    ./ --starttls=smtp

    Notes, Domino, Traveler V12 Beta1 released

    Daniel Nashed  January 22 2021 12:17:54 AM

    Notes, Domino and Traveler V12 Beta 1 is available

    Here is the official blog post..


    The software is available on Flexnet like we had the early access code drops. But now we got the clients and server installers in addition to the HCL Domino Docker image.


    In contrast to earlier years, the documentation isn't in a PDF.
    The documentation has been added to the standard documentation and you see the new features in "what's new sections" as part of the standard documentation.
    I spoke with customers and partners who have been confused about finding the trial image documented next to the new functionality for example.

    One of my favorite features: Automated Certificate Management

    Certificate Management has been already part of the early access code drops.
    With Beta1 Let's Encrypt / ACME DNS-01 challenges have been added.

    So you can request certificates from and for servers, which have no incoming HTTP connection.
    Instead a DNS-TXT record is used for validation (see details here -->

    You can get free certificates from Let's Encrypt and other providers supporting the ACME standard (e.g. ZeroSSL:,  BuyPass:

    Currently there are two reference implementations vor DNS Automation for Cloudflare, Inc. and Hetzner GmbH here in Germany.
    Beta 1 already includes a framework for implementing your own integration for your favorite provider.

    If you are interested Certificate Management, you should really look into the features in this first beta and provide feedback.

    And of course also test the other new functionality Thomas has mentioned.

    Domino Docker Project Update

    I have been already testing Domino V12 Beta1 with our Docker project and updated the images for Domino and Traveler.

    The new version names are already included in the develop branch -->

    So you could also use the community image already, when you download the Domino Linux web-kit to get your image build locally.

    Enjoy the beta and hopefully we see us in the Beta forum.

    -- Daniel

    Introducing Domino Container Script for Podman an Docker

    Daniel Nashed  January 17 2021 05:31:08 PM

    This is a brand new script, which is the logical extension to the Nash!Com Domino Start script.
    A while ago I introduced Docker support for the start script inside the Domino container.
    It comes with an entry-point script we are also leveraging in out Domino Docker Github community script.

    This new script is intended to manage and run your Docker and Podman containers.
    It includes also a systemd script to run a Domino server on Podman in production using the new systemd options introduced in Podman 1.7+.
    Beside that it includes a simple framework to build and manage your own add-on images

    Components & functionality

    - Predefined systemd script for running a Domino server on Podman 1.7+
    - Support for Docker 18+
    - sysconfig script to configure your container
    - Environment file to configure the server inside the container
    - domino_container script as your single point of service for all type of operations
    - Build environment for custom images on top of the HCL Domino image or 3rd party images like your Domino on Docker project

    It will allow to manage your server from the domino_container script including jumping into the container from command-line.
    And you can also leverage it to update your container even running with systemd on Podman.

    I moved one of my production servers to a new provider and decided to use Podman with this systemd scripts.
    But also when using Docker this script allows you to completely manage your server.

    This will replace the management script in the Docker project over time.
    The script is more flexible and also provides a simplified way to build add-on image.

    build_image directory
    With the new approach each image has it's own directory including a Container file (Podman uses this name instead of Dockerfile) and a build.cfg.
    You can clone this directory and define your own add-on images which you can build from the directory using the domino_container script.
    The build logic is in the script and the install logic for the image is in the install_dir.

    The out of the box example is already prepared to install server-tasks and extension managers. It can be used as a template and be customized and extended for any type of customer or partner image.

    Like the start script, the container_script has it's own install script (install_domino_container), which writes the files to the right directories.
    Once you installed it, you can run "domino_container" with all the options specified below.
    I hope this provides a quick overview and is what you need to run your Domino servers on Docker & Podman.

    Any feedback is welcome. It's a first version...
    -- Daniel

    Current commands

    domino_container help
    (Using config file /etc/sysconfig/domino_container)

    Nash!Com Container Script Version 1.0.0
    (Running on Podman Version 2.0.5)

    Usage: domino_container

           start | stop | restart | status | statusd | inspect | console | config | env
           domino | bash | update | attach | remove | removeimage | cleanup | port
           install | load | build | version | help

    start [live]    start an existing container (the 'live' option shows start script output)
    stop  [live]    stops container (the 'live' option shows start script output)
    restart         restarts or starts the server
    status          shows container status (running, exited, notexisting)
    statusd         shows the systemd status
    info            shows status and basic information about container and image
    inspect         shows detailed information about container and image
    console         open Domino console inside container
    logs            shows container logs (output from entry point script/start script)
    attach          attach to entry point script
    domino          pass a command to start script (e.g. domino nsd)
    bash [root]     invokes a bash in the running container. optionally run as root instead of notes user
    remove|rm       removes the container (if not running)
    removeimage|rmi removes the current container (you have to remove the container first)
    config|cfg      edit configuration
    update          updates the container if referenced image has changed (stops Domino, stops the container, runs a new image)
    install         install Podman
    load            load HCL Domino Docker image
    build           builds a current image -- even image tags might not have changed to ensure OS patches are installed
    port            show used tcp/ip ports for container
    enable|on       enable systemd service
    disable|off     disable systemd service
    clean           cleanup container and systemd if configured
    env             edit environment file
    version         shows script version information
    help            you already figured out what help does ;-)

    Domino on Linux Start Script Sametime 11 support

    Daniel Nashed  January 17 2021 11:28:42 AM
    Now that we have the Sametime Premium Meeting Server, there are more customers looking into Domino on Linux.

    In earlier days I had some issues getting the start script in combination with SLES working.
    So ST wasn't never on my support list.  The SLES issue was going way when switching from init.d scripts to systemd and replacing the SUSE specific "rc" logic -- which caused the issues earlier. I removed that logic completely, because systemd doesn't need it on SUSE platforms.

    With this change I started to look into Sametime on Linux again and I have now added Sametime Community server support to the start script.
    The new version 3.5.0 will be available on our Domino Docker github repo in the develop branch first.

    The missing part for ST 11 was adding the lib path for two specify ST directories.


    Those directories are now added for all installations and should not cause an issue on servers where no ST is installed.

    I also added a "resetstlogs" to include the ststart script "resetlogs" functionality.

    So the Domino service will leverage the start script and you can take benefit on a central entry point for all operations like start/stop/console, .. including the "resetstlogs" command.

    -- Daniel

      Y2K21 Notes property box - Created date is empty

      Daniel Nashed  January 4 2021 09:11:42 PM

      The following has been already reported to HCL development and there is a SPR# NNAIBWX3MD.

      This came up quite early in the HCL Ambassador discussion channel over the holidays.

      The property box is not showing a created data for documents created in 2021 and later unless there is a $Created date item like in incoming SMTP mails.

      This is just the properties box not showing the date. The functionality is completely working fine on all currently shipping versions and client platforms.

      You can still see it in for example formulas with @created, in NotesPeek or any other tool you might have installed.

      All functionality like replication etc. is working as expected. This is really just the properties box not showing the date.

      Image:Y2K21 Notes property box - Created date is empty

      Thomas Hampel just confirmed it in his blog and I would expect a technote to be published soon.

      -- Daniel

      Docker 20.10 and Docker Desktop 3.0 released

      Daniel Nashed  December 31 2020 03:38:04 PM

      I never said Docker is completely dead, even I see today better integration and more features in Podman.
      This new release of Docker CE is helping specially on CentOS 8 and you don't need to work-around compatibility issues with an older version of containerd shipped with CentOS 8.

      At the same time Docker updated Docker desktop to a new major version -- which also includes Docker version 20.10.

      The Docker Community project will continue to use Docker as the build platform, but also supports Podmand.

      I have just installed both today. Docker Desktop 3.0 did hang on installing the new version in my Win10 WSL2 environment.
      But re Windows reboot helped to get it installed -- actually this was a tip in a forum so I might not be the only one running into this issue.

      On CentOS 8, I had to uninstall Docker completely as the Docker official instructions reference -->

      In addition to their recommendations I also had to remove the docker0 network like this:

      firewall-cmd --permanent --zone=trusted --remove-interface=docker0

      So my installation steps have been:

      1. Uninstall docker-ce via "yum remove ..." as described on the CentOS installation page
      2. Remove the interface from the configuration -- if you added it earlier to get it working like me
      3. Reboot
      4. Use the new script to install Docker 20.10.

      It's working fine now on my CentOS 8 machine without any "hacks".

      And in my case I removed all existing VMs and images. The container data was on /local/.. and the container description was in my scripts.
      So I was able to uninstall Docker and remove all container data as suggested on the Docker homepage.
      Just uninstalling the software and removing the docker0 network should already work fine.

      There are enhancements like rootless Docker containers (not inside the container which is already a best practice for a long time but running the container without root permissions).

      I am not a big fan of that, because it still might cause issues. So having systemd start with root permissions isn't a big issue.

      There are a couple of other improvement, which might be interesting. I got the following link with a good summary

      Still Docker 20.10 is only adding mainly functionality already available in other environments like Podman.

      Docker is a great platform to start with. And specially Docker desktop integrated with Win10+WSL2 has some benefits.
      Also the documentation from Docker is more detailed.
      So as a starting point Docker is still a good platform to work with.

      Our last "Docker" workshop has been using CentOS with Docker 19.x. Sounds like I should now prepare the lab with CentOS 8 and Docker 20.10 ..

      -- Daniel

      SPAM from servers with IN-ARPA *

      Daniel Nashed  December 31 2020 10:55:27 AM
      Today I got a lot of SPAM today gray-listed from hosts in those domains.
      The worrying part is that this is split on different data centers apparently.
      So I got mails from the following sub-domain which have been listed in my SpamGeek with a high gray-list value and now got an explicit SPAM rating.
      Not expecting to have any customers sending me mails from there... But if they do, they need some really positive other rating based on content ..

      Moving applications into the cloud at providers with shared services can cause that your outgoing mail traffic is impacted by others misbehaving on the same shared service.

      I looked up one of the sub domains in SenderBase. None of those hosts have been rated with a "Good" rating -- currently 7 hosts are rated "Poor".
      And I know companies setting their CISCO ESA (which is using the SenderBase) to even not accept "Neutral" reputation (which is something I would never recommend).

      CISCO ESA (aka. IronPort) are one of the leading solutions used by many larger customers.
      And if you are having a "Poor" rating you are in serious outgoing e-mail trouble!

      -- Daniel


      Image:SPAM from servers with IN-ARPA *

      Domino on Podman in production

      Daniel Nashed  December 24 2020 09:01:28 AM

      This week and I am migrating my Domino server to a new provider updating to CentOS 8.

      Because I am working a lot with Domino in the Docker/Podman/Kubernetes space, I am looking into it also for my own server.

      The big benefit is that I can apply my private "NashCom" image derived from the standard image to all servers needed without installation. And switching versions is means just a restart.

      My own image will be based on the Docker Project version and use the add-on functionality to build your own image on top of it.

      In my case to add all my tools starting with the "nsh.." prefix.. And also the Domino V12 DSAPI filter for Let's Encrypt/ACME integration.


      I am looking into the latest Podman ( version that comes with CentOS 8.
      CentOS 8 has better Podman support than Docker support

      And in contrast to Docker, Podman is not using a daemon. For Docker I added my management script to our Docker project -->
      With Podman I will extend this concept and make it a full start script, which can be used in combination with systemd.

      Podman has added systemd functionality and you can generate your own systemd services with Podman -->
      I have looked into it and will bring this concept into a Domino start script that is Podman aware.

      Domino Start Script support

      The Domino start script itself runs inside the container  already. So you have most start script options available inside the container on all container platforms.

      This new functionality is intended for the outside host part in combination with Podman and systemd.

      Probably I will end up with a separate script based on the current management script and just integrate it with the classical start script.

      So that you could for example pass start script configuration variables from the host to the container running in Podman.

      The management script is already a kind of Docker/Podman start script. But it wasn't design for systemd integration yet.

      Not sure if I will keep it in the Docker project or add it to the start script to make it automatically installable etc.

      My start script is also part of the Docker project. So it will be probably just moved at some point.

      Not sure yet where it will be located. But it will also be able to work with the HCL out of the box Domino Docker image, not just with the Domino Docker project image.

      HCL Domino image support

      Both images are moving into the same direction anyway. We are adding new Domino V12 functionality into our Domino Docker project while the HCL image is taking ideas from our script and bring them into code.

      For example the "one touch scriptable setup" is now part of Domino V12 Early Access and works cross platform, not just for Linux or containers alone!

      This isn't based on scripts or Java attached to Domino. It's part of the server code.

      So as soon new functionality is added to the official HCL image, we adopt those new features and replace existing features in our script for newer Domino versions.

      On the other side we will still continue to complement what is there is additional functionality needed in the community.

      For now in the early access phase we don't have access to the Linux web kit as the base. But this will change as soon the betas are available early next year.

      So you will see our Domino Docker project to support Domino V12 right from the start of the first beta.

      Yes this isn't intended for production at that stage. But you can bet on me still upgrading at least a cluster mate to V12 early on.

      In fact I am running Domino V12 already in my hosted lab on a Docker host using the HCL V12 Early Access December drop.

      Podman limitations

      Podman isn't designed for large scale Domino deployments. You could run multiple "partitions" on the same machine and I will support creating more than one Domino systemd service.

      But you will need a separate IP address for each Domino instance if you want to expose NRPC on the same standard port 1352 on each instance.

      For all internet protocols like HTTP, SMTP and others you could have reverse proxy in front of it -- but today NRPC still needs it's a dedicated IP address.

      Also in many cases using the "host network" on Podman is the right choice. I will for example need it to get the real IP addresses for incoming SMTP traffic.

      And also I want to see the original IP addresses for incoming HTTP traffic.

      There are ways to pass the original IP address via X-FORWARD-FOR headers on HTTP.

      And for SMTP there are also ways to pass the original IP address. But this is a complexity you don't need for a single Domino instance running on Podman.

      The benefit running Podman and other container solutions is the easier management and deployment of the application in a container.

      IMHO if you are looking into larger scale deployment neither Podman nor Docker is the right choice.

      In that case you should look into Kubernetes (K8s) or other platforms based on it like OpenShift, Rancher or VMware Tanzu just to name some of the big players in that space.

      Docker vs Container

      When starting to look into the code I noticed that there are many variables prefixed "DOCKER_". And the default is Dockerfile to build a new image.
      Podman changed the name from Dockerfile to Containerfile which is the better term.
      The Domino Docker project will stay with the term "Docker" but the new Podman start script project will neither use Docker nor Podman for the naming unless something is platform specific.
      So the new naming convention will be CONTAINER_ instead of DOCKER_
      I think this new project is a good starting point to move to the new terminology. This really reflects better what it is today.


      If you are looking into Podman or other container technologies in combination with Domino I really want your feedback.

      What are using today and why? And what do you would like to use in future? Do you have any specific requirements or challenges?

      Our focus in the Domino Docker GitHub project is to make the image itself work nicely in all environments.

      This start script initiative is to help to bring containers into smaller environments without running a full blown Kubernetes.

      Of course I am also looking into that and I have a zoo of different K8s implementations running.

      Still it looks like Podman can be a good starting point for a smaller environment with one server per host.


      -- Daniel



        • [IBM Lotus Domino]
        • [Domino on Linux]
        • [Nash!Com]
        • [Daniel Nashed]