Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

    Planned Domino 9 SLES 12 Support

    Daniel Nashed  30 January 2015 00:25:38
    The question for SLES 12 has been raised during IIBM ConnectED. There is an issue with Domino on SLES 12 and SLES 12 is not currently supported (in contrast with RHEL 7).

    There is a SPR # YXYX9RA56Z "Error - Unable to Bind port 443 or 80" on SUSE12.

    I have checked in the Lab and got a similar info than what has been posted before on the web:
    "There is a known issue with SLES 12 where bindsock has issues. Before we can support SLES 12 and any other newer kernel with this issue, we will have to identify the issue and get it fixed - bindsock and it's code has never changed in this area so the issue is a change in behavior in the kernel".

    So for now you should not try to run Domino on SLES 12. It's not just not supported yet but it also it doesn't work yet.

    There is a chance in the kernel that SLES 12 that causes the bindsock operation to fail.

    Some  background: Ports below 1024 are restricted and need root permission. Bindsock has the sticky bit set, runs with root permission and is used to allow Domino to use ports like HTTPS, SMTP and all other restricted ports.
    The way that Domino implements this does not work on the current kernel any more.

    IBM is working on it closely with Novell and the following is the official statement from IBM about it:

    "We intend to support SLES12, just as we have supported every other SLES support. Our current goal is to add support with 9.0.1 FP4. We are working with the vendor to identify root cause of an intermittent bindsock issue we found during testing that is currently preventing us from supporting this version."



      ConnectED Session Slides posted BP102: Practical IBM Notes and Domino Internet Security

      Daniel Nashed  27 January 2015 23:12:19
      Today I had the pleasure to present with Dave Kern about Domino internet security.
      Now that the presentation is public, I can speak about all the details that we presented. See the slides for all details.

      We covered what is already available in 9.0.1 FP3 and what is coming after FP3 quite soon.
      In the session demo we had a the SSL Test website showing a A- up to A+ rating depending on the configuration.

      There is a lot good stuff coming up in a scheduled interims fix. This includes TLS 1.2 and some new ciphers.

      Huge thanks to Dave, the security team and a other developers making it happen so fast. Very impressive work!

      You will find a final version of the slides on my "IBM Connected/Lotusphere" presentation page.

      http://www.nashcom.de/lotusphere

      If you have questions let me know by mail.




      Notes/Domino 9.0.1 FP3 has shipped

      Daniel Nashed  21 January 2015 18:06:14
      Today Notes/Domino 9.0.1 FP3 has been shipped.
      Already installed it on my production server.

      There are new new "SSL/TLS" releated fixes in FP3. But there are updates planned after FP3.
      So updating to FP3 is the base and you should consider an update soon.
      It's always better to install a FP than a IF which is technically a combo hotfix.

      There are also a couple of other important fixes in FP3.
      When you look into the Fixlist you see a couple of database/DAOS releated fixes.
      The FP also contains an updated JVM (SPR# KLYH9MKHPD - Updated the embedded JVM in Notes/Domino to Oracle July 2014 Critical Patch )

      Just to mention some of the fix areas. Check the full fixlist for details.




      Domino TLS POODLE Fix released

      Daniel Nashed  21 December 2014 00:30:09
      As reported before the IF that introduced TLS 1.0 is vulnerable to the new PODDLE issue.
      IBM released a new IF for all supported versions that fixes this issue.
      After installing the IF you can re-enable the CBC ciphers which are now reported as not vulnerable by the SSL Labs Test site.

      In addition to this fix IBM officially introduces a new notes.ini variable to disable SSL V3.

      DISABLE_SSLV3=1 will disable SSL V3 completely. But as mentioned before you should be completely sure if you want to completely disable SSL V3.

      SPR #KLYH9QXMQE: Disable SSL ini:
      SPR #KLYH9RMJGL: CVE-2014-8730 TLS 1.x Padding Vulnerability

      There is a reference technote and a list of IFs for all supported releases.

      Security Bulletin: TLS Padding Vulnerability affects IBM Domino (CVE-2014-8730)
      http://www.ibm.com/support/docview.wss?uid=swg21693142


      Fixes for this issue are currently available

          9.0.1 Fix Pack 2 Interim Fix 3
          9.0 Interim Fix 7
          8.5.3 Fix Pack 6 Interim Fix 6
          8.5.2 Fix Pack 4 Interim Fix 3
          8.5.1 Fix Pack 5 Interim Fix 3

      New-Domino-POOLE-Iussue-now-with-TLS

      Daniel Nashed  10 December 2014 01:16:10
      There is a new exploit that affects TLS! Not all implementations of TLS are affected.
      But Domino and also some other solutions like the F5 load-balancer are on the list.


      For more details read -->
      https://community.qualys.com/blogs/securitylabs/2014/12/08/poodle-bites-tls

      The problem effects all CBC ciphers. IBM is working on a solution.
      Meanwhile you  can disable the CBC ciphers. Currently there are only two ciphers left.
      Not really completely what we want but it sounds like IBM is working on supporting more ciphers and later TLS versions (some ciphers only work with TLS 1.2).

      But the first priority is to fix the currently supported ciphers.

      As a work around for now change the cipher list in the Server doc / Internet Site doc. But this would not be uses for all protocols.

      The better way would be to specify the cipher list via notes.ini and have it set for all internet protocols.


      There is a notes.ini setting that can be used to specify the ciphers quite low-level with a single hex-byte list.


      SSLCipherSpec=0405


      Here is the corresponding list for the ciphers. We currently only want the RC4 ciphers -- even they are not the latest ciphers. But they are not affected by the new POODLE issue.


      04 - SSL_RSA_WITH_RC4_128_MD5

      05 - SSL_RSA_WITH_RC4_128_SHA


      By the way: Also the currently unsupported notes.ini parameter that completely disables SSL 3.0 has leaked in a open mic session.

      IBM is planning to change it to a more proper name and will document it.

      Surprisingly the parameter cannot be found on Google yet... It is going to change soon, so I will not post it in a blog.

      iNotes Redirect without Anonymous Access

      Daniel Nashed  5 December 2014 16:24:50
      When running iNotes you might only want to allow authenticated connections to your Domino Server over HTTP.
      But on the other side you want to use the iNotes Redirect database which contains some images and other design that should load even the user is not yet authenticated.

      There is a Wiki article that describes in detail what to do. Thanks to IBM pointing out that parameter!
      http://www.lotus.com/ldd/dominowiki.nsf/dx/Allowing_Anonymous_Access_to_iNotes_Redirect_images__while_preventing_Anonymous_Access_to_Domino_HTTP_
       

      Starting with Domino 8.5.2 you can allow additional URLs that an anonyous users can access even the server allows only authenticated access.

      You can either allow the full database:

      HTTPPublicUrls=/redir.nsf/*

      Or in a more paranoid configuration only allow the specifc design elements which should be available -- example for "redir.nsf"

      If using Domino 8.5.3 or Lower version of the IBM Redirector ini should be set like this:

      HTTPPublicUrls=/redir.nsf/iNotes-LoginBanner65short.gif:/redir.nsf/StylesheetLogin:/redir.nsf/Login.js


      If using Domino 9 or Later version of the IBM Redirector  ini should be set like this:

      HTTPPublicUrls=/redir.nsf/IBMLogo.gif:/redir.nsf/StylesheetLogin:/redir.nsf/Login.js


      Note: You have to restart the HTTP task (e.g. restart task http)

      -- Daniel
       

      Short Description Creating a Domino Keyring File with the new Keyring Tool and a Windows CA using Binary Formats

      Daniel Nashed  2 December 2014 09:52:36
      Now that more and more customers are using the new keyring tool we run into interesting constellations.
      Microsoft uses binary formats instead of the ascii based PEM format that the keyring tool requires.

      Openssl does not only help you to create the key and the certficates. You can also use it to convert the certificate formats.


      I have written a short step by step short documentation for my customer including some troubleshooting steps and tricks.

      To keep it short I have left out the results from the commands. But you find this information in the official IBM documentation (http://www.lotus.com/ldd/dominowiki.nsf/dx/Domino_keyring)


      -- Daniel



      Short Description Creating a Domino Keyring File with the new Keyring Tool and a Windows CA using Binary Formats


      First of all you have to find a machine with openssl installed. The most easy way would be to log into any type of Linux machine with a current openssl version installed.
      But there are also Windows implementations.

      Create Private/Public Key


      In the first step you create a private public key using the standard openssl command-line.


      openssl genrsa -out server.key 4096



      Create Certificate Request


      In the next step you create a certificate request and have it processed by your CA.

      The command-line will prompt you for country, organisation name etc and the name of your server.

      That would be the DNS name of the server for example "
      www.nashcom.de" or in case you want to generate a wildcard certficate "*.nashcom.de"


      openssl req -new -sha256 -key server.key -out server.csr



      Convert Binary Certificate Files to PEM Format


      The Domino kyrtool requires the text based PEM format but the Microsoft CA does generate binary files.

      The following commands can be used to convert the formats.


      First of all you convert the server certificate to from binary DER format to text based PEM format:


      openssl x509 -inform der -in server.cer -outform pem -out server.pem


      In the next step you convert the certificate chain from p7b binary format to PEM format as well


      openssl pkcs7 -print_certs -inform der -in certificate_chain.p7b -outform pem -out chain.pem



      Combine Key, Cert and Chain to a single file


      The most easy way to import all certificates you combine all parts into a single PEM formatted file.

      The order in the file must match leave to root order. In most cases you just copy all party into a single file.


      copy server.key+server.pem+chain.pem all.pem


      Now you have all parts in a single file in the right order.

      Note: In some cases the server certificate is already in the chain file. In that case you have to ensure that the certificate is only listed once



      Create Keyring File


      Download and install the new kyrtool -- just copy it into the Notes program directory.


      Afterward just create a new keying file and set a password.

      Note: Password only has to be entered the first time you access the keyring afterwards it is automatically read from the sth file.


      c:\Notes\kyrtool.exe create -k c:\cert\mykeyring.kyr -p SecurePassword



      Import Key, Cert and Chain


      In the next step you just import all parts we added to the single file into your keyring.


      c:\Notes\kyrtool.exe import all -k c:\cert\mykeyring.kyr -i c:\cert\all.pem



      Troubleshooting / Verification


      There are two interesting options that might be helpful.


      First of all before you import the certificates you can verify the file is complete and all certificates in the chain are present and matching.


      C:\Notes\kyrtool.exe verify c:\cert\all.pem


      The result be as follows:

      - A private key should be present

      - The should be no missing certs in the chain or mismatches

      - The last certificate in the chain is self-singed (root certificate)


      Another way to show the certs and dump them might be useful as well.

      This command line shows you all the certs with detailed information about each part.


      C:\Notes\kyrtool.exe show certs -i c:\cert\all.pem


      Verify existing keyring files


      If you want to verify an exiting keyfile you can combine "show certs" output into a file with a "verify" command on that output file.

      The only error you should have with that verify is that the private key is missing.
      That helps to verify that a keyring file has a complete chain.

      Traveler 9.0.1 IF7

      Daniel Nashed  7 November 2014 08:48:22
      Finally Traveler 9.0.1 IF7 is available.

      I don't see a fixlist yet but I got a fixlist from a customer from the latest hotfix he got.


      The IF should fix all attachment issues which came up with IF6, includes the latest Android client and should also have an updated certificate for APNS.


      So now you can install 9.0.1 IF7 in combination with Domino 9.0.1 FP2 IF1 which introduces TLS 1.0 in one go with just one downtime.

      FixCentral Download Link:


      http://www.ibm.com/support/fixcentral/swg/selectFixes?parent=ibm~Lotus&product=ibm/Lotus/Lotus+Notes+Traveler&release=9.0.1&platform=All&function=all


      -- Daniel

        Some Additonal TLS 1.0 Information

        Daniel Nashed  6 November 2014 16:28:26
        TLS 1.0 and the removal of SSL 3.0 from browsers that triggered the whole discussion is not just something that needs to be addresses on a Domino server.
        IBM has done a lot of work in quite a short time and now that customers are implementing the fix it shows that also other software is effected.

        Introducing TLS 1.0 for Domino was the first step from IBM to ensure that clients that only support TLS 1.0 and higher can still connect to the Domino server.
        For now IBM still has SSL 3.0 enabled to allow communication with software that does not yet support TLS 1.0 and they are preventing clients from the downgrade attacks as mentioned in the IBM technotes.

        Notes Client Software

        But Domino is not the only server for most customer environments. Many companies completely disable SSL 3.0 and cause issues with other client software.
        And also Notes Clients are affected for example when connecting to other HTTP servers or using secure IMAP, POP3, LDAP or SMTP.

        For example here in Germany GMX one of the larger, well known email-providers disabled SSL 3.0.
        In that case you need a fix for the Notes Client side. IBM did not yet ship the full set of clients because they are waiting for some I think unrelated Java patches.

        But because there is also an enhancement for SHA256 for the cert request database, IBM shipped already the Win32 Standard Client.
        The download is a bit more difficult to find on Fix Central but you should find it using the following link.

        http://www.ibm.com/support/fixcentral/swg/quickorder?parent=ibm%7ELotus&product=ibm/Lotus/Lotus+Notes&release=9.0.1.2&platform=Windows&function=fixId&fixids=Notes_901FP2IF2_W32_Standard&includeSupersedes=0&source=fc

        If you need a client connection for one of the internet protocols and the server does only support TLS 1.0, you will need to install this IF.

        Other Client Software -- Other Issues


        Notes/Domino is not the only application having an issue with servers that don't support SSL 3.0 any more or servers that changes the way they negotiate SSL versions.
        Domino for example has SSL 2.0 including the SSL 2.0 handshake disabled with 9.0.1 FP2 IF1. Other servers might have done the same or similar.

        This leads to interesting interoperability challenges. For example older OpenSSL versions do not nicely negotiate their SSL level with Domino servers without explicitly specifying TLS 1.0 as Andrew Pollack found out.
        In case of wget in combination with an older OpenSSL version according to IBM the negotiation failed because that OpenSSL version used an V2 handshake, which failed and stopped the negotiation.

        And there might be other application issues where a server does not work nicely with your Java 1.6 application (which supports TLS 1.0 but maybe not the ciphers the server is expecting).
        Java 1.7 does also support TLS 1.1 and 1.2 but not in all cases you can switch to the later Java version.

        I have tested Java 1.6 in Notes with an unpatched Notes client against a server that does have SSL 3.0 completely disabled and the Java agent worked unmodified.
        But there are other parts in Notes that use the native SSL stack. And from what I heard from IBM some parts in Java also seem to use the native Notes/Domino stack instead of the Java stack.


        So when the browser vendors decided to stop supporting SSL 3.0 any more they did not just challenge the server vendors but because of the impact to client software all applications using SSL connections might be affected.
        When introducing new versions of software that support TLS and might not support SSL 3.0 at all or have a changed way to negotiate the session, you really have to test all your applications and see which SSL level they support and which types of ciphers.


        The SSL Test website (https://www.ssllabs.com/ssltest/) tries to test what happens when you access your server with various client software and you should have a look if your server does support a cipher for all of you client access types.

        As I said, this is not just a challenge for Domino but also for other applications -- even if they are totally unrelated -- because many vendors are working on their SSL stack (or administrators disabling SSL 3.0 and below).

        Sometimes you have to specify the right SSL level (currently TLS 1.0 for Domino) to establish a connection and that could be even good from security point of view.

        On the others side you might have to think about updating the software on the client machine itself. For example older versions of OpenSSL should be updated to solve SSL handshake issues.

        There are many parts you should test. And this post should just give you some more background and a heads up what could break now or in the near future when more and more servers are patched/reconfigured. In many cases the solution is to update your software.


        -- Daniel



        Domino TLS 1.0 SHA-2 Support to prevent POODLE has been shipped today

        Daniel Nashed  4 November 2014 01:14:16


        As blogged before IBM was already working on addressing the POODLE attack by finally implementing TLS 1.0 for all internet protocols.


        Today IBM shipped an Interims Fix to introduce TLS 1.0 which is very important because many browsers and other software vendors are about to drop SSL 3.0 support.

        So you need those fixes to continue to use secure protocols like HTTS, secure SMTP, LDAP, IMAP, POP3, DIIOP..


        There are a couple of changes which are described in the following Wiki documents. And there are a couple of additional Wiki documents providing additional information.


        Basically this fix will allow TLS 1.0 and also allows you to use SHA-2 based certificates with a new introduced command-line key-ring tool called "kyrtool".

        The tool is a command line application that can manage your keyring files with SHA-2 support and you don't need the old ikeyman tool that many of us used before with all those limitations.

        I have been testing the tool on Windows and Linux and it is working like a charm. The Wiki contains step by step instructions how to use it in combination with openssl to generate a private key, signing requests and import trusted roots and certficates.


        You find very detailed step by step documentation in the referenced links.


        And you can start downloading the fix and the kyrtool today!
        I have it already running on my production Traveler server on Linux 64.


        Here are the details including download links and detailed descriptions.


        For TLS 1.0 support you just need to install the hotfix and all the defaults should just work fine. You need no additional settings.

        Note: IBM did not disable SSL 3.0 for compatibility reasons in this fist step. The first IF is intended to introduce TLS 1.0 to allow all applications to continue to work with Domino.


        Domino with this fix prevents a downgrade attacks if the client requested TLS 1.0. Some applications will still report that your server is vulnerable to POODLE because Domino still supports SSL 3.0 but this is not completely true. That's just a basic check for SSL 3.0.


        IMHO introducing TLS 1.0 in combination with preventing downgrade protocol attacks is the right first move.

        The fixes are available for  all supported platforms and releases (9.0.1 FP2, 9.0, 8.5.3 FP6, 8.5.2 FP4, 8.5.1 FP5).


        But you should be aware that SHA-2 is only available in Domino 9.0.x because 8.5.x releases "lack the cryptographic infrastructure for SHA-2. "



        Thanks to IBM and specially the security team who did a great job in a very short time!

        They have been already working on TLS and SHA-2 support before but had to change their plans because of the short term move to diable SSL 3.0 in browsers and other software.


        Here is the official quite detailed IBM documentation for TLS, SHA-2, the new key-ring tool "kyrtool" and information about how IBM addressed the "POODLE attack" with this fix.


        -- Daniel



        IBM Domino Interim Fixes to support TLS 1.0 which can be used to prevent the POODLE attack

        http://www.lotus.com/ldd/dominowiki.nsf/dx/IBM_Domino_TLS_1.0

        Generating a SHA-2 Keyring file

        http://www.lotus.com/ldd/dominowiki.nsf/dx/Domino_keyring

        IBM will add more articles in these categories around troubleshooting, tracing, and so on.


        http://www.lotus.com/ldd/dominowiki.nsf/xpViewTags.xsp?categoryFilter=SHA-2
        http://www.lotus.com/ldd/dominowiki.nsf/xpViewTags.xsp?categoryFilter=TLS