Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

    ConnectED Session Slides posted BP102: Practical IBM Notes and Domino Internet Security

    Daniel Nashed  27 January 2015 23:12:19
    Today I had the pleasure to present with Dave Kern about Domino internet security.
    Now that the presentation is public, I can speak about all the details that we presented. See the slides for all details.

    We covered what is already available in 9.0.1 FP3 and what is coming after FP3 quite soon.
    In the session demo we had a the SSL Test website showing a A- up to A+ rating depending on the configuration.

    There is a lot good stuff coming up in a scheduled interims fix. This includes TLS 1.2 and some new ciphers.

    Huge thanks to Dave, the security team and a other developers making it happen so fast. Very impressive work!

    You will find a final version of the slides on my "IBM Connected/Lotusphere" presentation page.

    If you have questions let me know by mail.

    Notes/Domino 9.0.1 FP3 has shipped

    Daniel Nashed  21 January 2015 18:06:14
    Today Notes/Domino 9.0.1 FP3 has been shipped.
    Already installed it on my production server.

    There are new new "SSL/TLS" releated fixes in FP3. But there are updates planned after FP3.
    So updating to FP3 is the base and you should consider an update soon.
    It's always better to install a FP than a IF which is technically a combo hotfix.

    There are also a couple of other important fixes in FP3.
    When you look into the Fixlist you see a couple of database/DAOS releated fixes.
    The FP also contains an updated JVM (SPR# KLYH9MKHPD - Updated the embedded JVM in Notes/Domino to Oracle July 2014 Critical Patch )

    Just to mention some of the fix areas. Check the full fixlist for details.

    Domino TLS POODLE Fix released

    Daniel Nashed  21 December 2014 00:30:09
    As reported before the IF that introduced TLS 1.0 is vulnerable to the new PODDLE issue.
    IBM released a new IF for all supported versions that fixes this issue.
    After installing the IF you can re-enable the CBC ciphers which are now reported as not vulnerable by the SSL Labs Test site.

    In addition to this fix IBM officially introduces a new notes.ini variable to disable SSL V3.

    DISABLE_SSLV3=1 will disable SSL V3 completely. But as mentioned before you should be completely sure if you want to completely disable SSL V3.

    SPR #KLYH9QXMQE: Disable SSL ini:
    SPR #KLYH9RMJGL: CVE-2014-8730 TLS 1.x Padding Vulnerability

    There is a reference technote and a list of IFs for all supported releases.

    Security Bulletin: TLS Padding Vulnerability affects IBM Domino (CVE-2014-8730)

    Fixes for this issue are currently available

        9.0.1 Fix Pack 2 Interim Fix 3
        9.0 Interim Fix 7
        8.5.3 Fix Pack 6 Interim Fix 6
        8.5.2 Fix Pack 4 Interim Fix 3
        8.5.1 Fix Pack 5 Interim Fix 3


    Daniel Nashed  10 December 2014 01:16:10
    There is a new exploit that affects TLS! Not all implementations of TLS are affected.
    But Domino and also some other solutions like the F5 load-balancer are on the list.

    For more details read -->

    The problem effects all CBC ciphers. IBM is working on a solution.
    Meanwhile you  can disable the CBC ciphers. Currently there are only two ciphers left.
    Not really completely what we want but it sounds like IBM is working on supporting more ciphers and later TLS versions (some ciphers only work with TLS 1.2).

    But the first priority is to fix the currently supported ciphers.

    As a work around for now change the cipher list in the Server doc / Internet Site doc. But this would not be uses for all protocols.

    The better way would be to specify the cipher list via notes.ini and have it set for all internet protocols.

    There is a notes.ini setting that can be used to specify the ciphers quite low-level with a single hex-byte list.


    Here is the corresponding list for the ciphers. We currently only want the RC4 ciphers -- even they are not the latest ciphers. But they are not affected by the new POODLE issue.

    04 - SSL_RSA_WITH_RC4_128_MD5

    05 - SSL_RSA_WITH_RC4_128_SHA

    By the way: Also the currently unsupported notes.ini parameter that completely disables SSL 3.0 has leaked in a open mic session.

    IBM is planning to change it to a more proper name and will document it.

    Surprisingly the parameter cannot be found on Google yet... It is going to change soon, so I will not post it in a blog.

    iNotes Redirect without Anonymous Access

    Daniel Nashed  5 December 2014 16:24:50
    When running iNotes you might only want to allow authenticated connections to your Domino Server over HTTP.
    But on the other side you want to use the iNotes Redirect database which contains some images and other design that should load even the user is not yet authenticated.

    There is a Wiki article that describes in detail what to do. Thanks to IBM pointing out that parameter!

    Starting with Domino 8.5.2 you can allow additional URLs that an anonyous users can access even the server allows only authenticated access.

    You can either allow the full database:


    Or in a more paranoid configuration only allow the specifc design elements which should be available -- example for "redir.nsf"

    If using Domino 8.5.3 or Lower version of the IBM Redirector ini should be set like this:


    If using Domino 9 or Later version of the IBM Redirector  ini should be set like this:


    Note: You have to restart the HTTP task (e.g. restart task http)

    -- Daniel

    Short Description Creating a Domino Keyring File with the new Keyring Tool and a Windows CA using Binary Formats

    Daniel Nashed  2 December 2014 09:52:36
    Now that more and more customers are using the new keyring tool we run into interesting constellations.
    Microsoft uses binary formats instead of the ascii based PEM format that the keyring tool requires.

    Openssl does not only help you to create the key and the certficates. You can also use it to convert the certificate formats.

    I have written a short step by step short documentation for my customer including some troubleshooting steps and tricks.

    To keep it short I have left out the results from the commands. But you find this information in the official IBM documentation (

    -- Daniel

    Short Description Creating a Domino Keyring File with the new Keyring Tool and a Windows CA using Binary Formats

    First of all you have to find a machine with openssl installed. The most easy way would be to log into any type of Linux machine with a current openssl version installed.
    But there are also Windows implementations.

    Create Private/Public Key

    In the first step you create a private public key using the standard openssl command-line.

    openssl genrsa -out server.key 4096

    Create Certificate Request

    In the next step you create a certificate request and have it processed by your CA.

    The command-line will prompt you for country, organisation name etc and the name of your server.

    That would be the DNS name of the server for example "" or in case you want to generate a wildcard certficate "*"

    openssl req -new -sha256 -key server.key -out server.csr

    Convert Binary Certificate Files to PEM Format

    The Domino kyrtool requires the text based PEM format but the Microsoft CA does generate binary files.

    The following commands can be used to convert the formats.

    First of all you convert the server certificate to from binary DER format to text based PEM format:

    openssl x509 -inform der -in server.cer -outform pem -out server.pem

    In the next step you convert the certificate chain from p7b binary format to PEM format as well

    openssl pkcs7 -print_certs -inform der -in certificate_chain.p7b -outform pem -out chain.pem

    Combine Key, Cert and Chain to a single file

    The most easy way to import all certificates you combine all parts into a single PEM formatted file.

    The order in the file must match leave to root order. In most cases you just copy all party into a single file.

    copy server.key+server.pem+chain.pem all.pem

    Now you have all parts in a single file in the right order.

    Note: In some cases the server certificate is already in the chain file. In that case you have to ensure that the certificate is only listed once

    Create Keyring File

    Download and install the new kyrtool -- just copy it into the Notes program directory.

    Afterward just create a new keying file and set a password.

    Note: Password only has to be entered the first time you access the keyring afterwards it is automatically read from the sth file.

    c:\Notes\kyrtool.exe create -k c:\cert\mykeyring.kyr -p SecurePassword

    Import Key, Cert and Chain

    In the next step you just import all parts we added to the single file into your keyring.

    c:\Notes\kyrtool.exe import all -k c:\cert\mykeyring.kyr -i c:\cert\all.pem

    Troubleshooting / Verification

    There are two interesting options that might be helpful.

    First of all before you import the certificates you can verify the file is complete and all certificates in the chain are present and matching.

    C:\Notes\kyrtool.exe verify c:\cert\all.pem

    The result be as follows:

    - A private key should be present

    - The should be no missing certs in the chain or mismatches

    - The last certificate in the chain is self-singed (root certificate)

    Another way to show the certs and dump them might be useful as well.

    This command line shows you all the certs with detailed information about each part.

    C:\Notes\kyrtool.exe show certs -i c:\cert\all.pem

    Verify existing keyring files

    If you want to verify an exiting keyfile you can combine "show certs" output into a file with a "verify" command on that output file.

    The only error you should have with that verify is that the private key is missing.
    That helps to verify that a keyring file has a complete chain.

    Traveler 9.0.1 IF7

    Daniel Nashed  7 November 2014 08:48:22
    Finally Traveler 9.0.1 IF7 is available.

    I don't see a fixlist yet but I got a fixlist from a customer from the latest hotfix he got.

    The IF should fix all attachment issues which came up with IF6, includes the latest Android client and should also have an updated certificate for APNS.

    So now you can install 9.0.1 IF7 in combination with Domino 9.0.1 FP2 IF1 which introduces TLS 1.0 in one go with just one downtime.

    FixCentral Download Link:

    -- Daniel

      Some Additonal TLS 1.0 Information

      Daniel Nashed  6 November 2014 16:28:26
      TLS 1.0 and the removal of SSL 3.0 from browsers that triggered the whole discussion is not just something that needs to be addresses on a Domino server.
      IBM has done a lot of work in quite a short time and now that customers are implementing the fix it shows that also other software is effected.

      Introducing TLS 1.0 for Domino was the first step from IBM to ensure that clients that only support TLS 1.0 and higher can still connect to the Domino server.
      For now IBM still has SSL 3.0 enabled to allow communication with software that does not yet support TLS 1.0 and they are preventing clients from the downgrade attacks as mentioned in the IBM technotes.

      Notes Client Software

      But Domino is not the only server for most customer environments. Many companies completely disable SSL 3.0 and cause issues with other client software.
      And also Notes Clients are affected for example when connecting to other HTTP servers or using secure IMAP, POP3, LDAP or SMTP.

      For example here in Germany GMX one of the larger, well known email-providers disabled SSL 3.0.
      In that case you need a fix for the Notes Client side. IBM did not yet ship the full set of clients because they are waiting for some I think unrelated Java patches.

      But because there is also an enhancement for SHA256 for the cert request database, IBM shipped already the Win32 Standard Client.
      The download is a bit more difficult to find on Fix Central but you should find it using the following link.

      If you need a client connection for one of the internet protocols and the server does only support TLS 1.0, you will need to install this IF.

      Other Client Software -- Other Issues

      Notes/Domino is not the only application having an issue with servers that don't support SSL 3.0 any more or servers that changes the way they negotiate SSL versions.
      Domino for example has SSL 2.0 including the SSL 2.0 handshake disabled with 9.0.1 FP2 IF1. Other servers might have done the same or similar.

      This leads to interesting interoperability challenges. For example older OpenSSL versions do not nicely negotiate their SSL level with Domino servers without explicitly specifying TLS 1.0 as Andrew Pollack found out.
      In case of wget in combination with an older OpenSSL version according to IBM the negotiation failed because that OpenSSL version used an V2 handshake, which failed and stopped the negotiation.

      And there might be other application issues where a server does not work nicely with your Java 1.6 application (which supports TLS 1.0 but maybe not the ciphers the server is expecting).
      Java 1.7 does also support TLS 1.1 and 1.2 but not in all cases you can switch to the later Java version.

      I have tested Java 1.6 in Notes with an unpatched Notes client against a server that does have SSL 3.0 completely disabled and the Java agent worked unmodified.
      But there are other parts in Notes that use the native SSL stack. And from what I heard from IBM some parts in Java also seem to use the native Notes/Domino stack instead of the Java stack.

      So when the browser vendors decided to stop supporting SSL 3.0 any more they did not just challenge the server vendors but because of the impact to client software all applications using SSL connections might be affected.
      When introducing new versions of software that support TLS and might not support SSL 3.0 at all or have a changed way to negotiate the session, you really have to test all your applications and see which SSL level they support and which types of ciphers.

      The SSL Test website ( tries to test what happens when you access your server with various client software and you should have a look if your server does support a cipher for all of you client access types.

      As I said, this is not just a challenge for Domino but also for other applications -- even if they are totally unrelated -- because many vendors are working on their SSL stack (or administrators disabling SSL 3.0 and below).

      Sometimes you have to specify the right SSL level (currently TLS 1.0 for Domino) to establish a connection and that could be even good from security point of view.

      On the others side you might have to think about updating the software on the client machine itself. For example older versions of OpenSSL should be updated to solve SSL handshake issues.

      There are many parts you should test. And this post should just give you some more background and a heads up what could break now or in the near future when more and more servers are patched/reconfigured. In many cases the solution is to update your software.

      -- Daniel

      Domino TLS 1.0 SHA-2 Support to prevent POODLE has been shipped today

      Daniel Nashed  4 November 2014 01:14:16

      As blogged before IBM was already working on addressing the POODLE attack by finally implementing TLS 1.0 for all internet protocols.

      Today IBM shipped an Interims Fix to introduce TLS 1.0 which is very important because many browsers and other software vendors are about to drop SSL 3.0 support.

      So you need those fixes to continue to use secure protocols like HTTS, secure SMTP, LDAP, IMAP, POP3, DIIOP..

      There are a couple of changes which are described in the following Wiki documents. And there are a couple of additional Wiki documents providing additional information.

      Basically this fix will allow TLS 1.0 and also allows you to use SHA-2 based certificates with a new introduced command-line key-ring tool called "kyrtool".

      The tool is a command line application that can manage your keyring files with SHA-2 support and you don't need the old ikeyman tool that many of us used before with all those limitations.

      I have been testing the tool on Windows and Linux and it is working like a charm. The Wiki contains step by step instructions how to use it in combination with openssl to generate a private key, signing requests and import trusted roots and certficates.

      You find very detailed step by step documentation in the referenced links.

      And you can start downloading the fix and the kyrtool today!
      I have it already running on my production Traveler server on Linux 64.

      Here are the details including download links and detailed descriptions.

      For TLS 1.0 support you just need to install the hotfix and all the defaults should just work fine. You need no additional settings.

      Note: IBM did not disable SSL 3.0 for compatibility reasons in this fist step. The first IF is intended to introduce TLS 1.0 to allow all applications to continue to work with Domino.

      Domino with this fix prevents a downgrade attacks if the client requested TLS 1.0. Some applications will still report that your server is vulnerable to POODLE because Domino still supports SSL 3.0 but this is not completely true. That's just a basic check for SSL 3.0.

      IMHO introducing TLS 1.0 in combination with preventing downgrade protocol attacks is the right first move.

      The fixes are available for  all supported platforms and releases (9.0.1 FP2, 9.0, 8.5.3 FP6, 8.5.2 FP4, 8.5.1 FP5).

      But you should be aware that SHA-2 is only available in Domino 9.0.x because 8.5.x releases "lack the cryptographic infrastructure for SHA-2. "

      Thanks to IBM and specially the security team who did a great job in a very short time!

      They have been already working on TLS and SHA-2 support before but had to change their plans because of the short term move to diable SSL 3.0 in browsers and other software.

      Here is the official quite detailed IBM documentation for TLS, SHA-2, the new key-ring tool "kyrtool" and information about how IBM addressed the "POODLE attack" with this fix.

      -- Daniel

      IBM Domino Interim Fixes to support TLS 1.0 which can be used to prevent the POODLE attack

      Generating a SHA-2 Keyring file

      IBM will add more articles in these categories around troubleshooting, tracing, and so on.

      TLS and SHA-2 Support and the POODLE Attack

      Daniel Nashed  21 October 2014 19:20:27

      IBM has officially responded to the POODLE attack and also officially responded to newer crypto standards.

      Very good news for Domino! IBM will introduce TLS 1.0 and SHA-2 support for all protocols soon!

      The current technotes mention a very short timeframe and it looks like we are going to get fixes at least for the current Domino 9.0.1 code stream.

      Some fixes will be also in the 8.5.x code-stream but some of the improvements like SHA-2 support cannot be back ported.

      So you should be prepared with all your internet facing servers to deploy 9.0.1 with current fixpack 2!

      IBM will introduce support for the current standards soon which will also address the POODLE attack.

      IMHO the risk right now is not very high and and most of the HTTPS internet facing servers for larger companies already use Secure Reverse Proxies.
      And you might need to have a closer look into what crypto levels those server currently support! You should disable SSL 3.0 on all servers as far as it is currently possible. This is not just true for Domino.

      IBM is working on improving the crypto "stack" (part of the lower network layer -- "NTI" which is the base for all Internet protocols and this includes in consequence also the keyring file used to store your internet certificates) in Domino in a short timeframe.

      Enclosed you find links for the two new technotes which provide details about what IBM is working on...

      We have been asking for years specially for TLS 1.0 and higher support for SMTP. Now it looks like we are getting it also for all other internet protocols!

      That's really great news!!

      -- Daniel

      How is IBM Domino impacted by the POODLE attack?

      Planned SHA-2 deliveries for IBM Domino 9.x