Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

SSL V2 HELO can be re-enabled with 9.0.1 FP3 IF1

Daniel Nashed  25 February 2015 20:45:11
As discussed before the security fixes introduced with the additon of TLS 1.0 removed V2 SSL HELO support.

This caused issues with applications that still use the V2 SSL HELO for compatibility issues. Specially older OpenSSL Versions did use V2 SSL HELO unless explicitly specifying TLS 1.0.
For most applications you can work-around it with updating the OpenSSL version to a current level.

But specially when using the SMTP STARTTLS extension we don't control what the connecting server uses.

IBM now allows to re-enable V2 SSL HELO if you really need to.

The reference SPR is #LMES9QRUZY Problem with incoming SMTP TLS connections after update to Domino 9.0.1 FP2IF1

But it does not mention the notes.ini parameter you need to enable it: SSL_ENABLE_INSECURE_SSLV2_HELLO=1

I have tested it with an older version of wget and got the following type of debug output:


25.02.2015 18:57:18,07 SSLReadRecord> Reading an insecure SSLv2 record by administrator request
25.02.2015 18:57:18,07 SSL2ReadRecord> Reading an insecure SSLv2 record by administrator request
25.02.2015 18:57:18,07 SSLProcessProtocolMessage> Record Content: 0
25.02.2015 18:57:18,07 SSLProcessProtocolMessage> Received an insecure SSLv2 record; processing by administrator request
25.02.2015 18:57:18,07 SSL2ProcessMessage> Message: 1
25.02.2015 18:57:18,07 SSL2ProcessClientHello> Processing SSLv2 ClientHello message requesting TLS1.0 (version 0x0301)
25.02.2015 18:57:18,07 SSL2ProcessClientHello> Client requested SSL_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)

SLES 12 support added in 9.0.1 FP3 IF1

Daniel Nashed  24 February 2015 19:19:30
There is a new section that you should note and regularly check: http://www.lotus.com/ldd/fixlist.nsf/WhatsNew/
This section will provide important updates to the fixlist.

In this case the support for SLES 12 with 9.0.1 FP3 IF1! WOW! That was a fast response!  Normally new major OS versions have to wait at least for a dot release! THANKS!!!

As posted before there was a technical issue with restricted ports because bindsock did not work any more because of kernel changes in SLES 12.

IBM and Novell worked on this and found a solution.

Beginning with IBM Domino 9.0.1 FP3 IF1, the IBM Domino Server is supported on SLES12. This interim fix included a fix for the following issue:
YXYX9RA56Z HTTP server can't be started with "Error - Unable to Bind port 443 or 80" on SUSE12

With the fix that made it into IF1 IBM is now supporting SLES 12 in the same way they support REHL 7!

RHEL 7 and SLES 12 are both using "systemd" which replaces the Linux init (rc scripts).
I got already a couple of questions about systemd support and I am currently working on a version of my start script that supports systemd.

It looks like it will be an additional service file in combination with some changes to the rc_domino script and also rc_domino_script as well.
systemd works completely different but I think I am on a good way adding support for it.

-- Daniel


Notes/Domino 9.0.1 FP3 - Java Console/Controller Incompatibility

Daniel Nashed  18 February 2015 10:35:44
As discussed before, it's not a good idea to completely disable SSLv3 too soon.
Notes/Domino 9.0.1 FP3 ships with a newer JVM version that completely disables SSLv3.
The Oracle team disabled SSLV3 by default but the IBM JVM team completely removed SSLv3.

The Domino server controller and Server Console are based on Java and use the SSL/TLS stack for communication.
Domino before FP3 uses SSLv3 only -- I don't want to start any theories about why ...

The newer version with FP3 and higher use TLS 1.0 only.

That means once you updated your client you cannot communicate via server controller with an older server.
And also means that you cannot communicate from an older client once you updated your server.

There is no easy work-around beside running two different clients.
Just using a different exe does not help because the main change is in the IBM JVM.
You could keep the old client binaries and clone the data directory and run the jconsole from two different directories to avoid using two different workstations.

-- Daniel


References:

http://www.ibm.com/support/docview.wss?uid=swg21695943

And information from the release notes:

9.0.1 Fix Pack 3 updates the embedded Notes/Domino JVM to 1.6 SR16 FP2 to address security vulnerabilities. This release has all of the content from the recently released POODLE and POODLE on TLS vulnerabilities in one easy to install package that includes the content from Domino 9.0.1 Fix Pack 2 Interim Fix 3 and Notes 9.0.1 Fix Pack 2 Interim Fix 4.

JVM 1.6 SR16 FP2 disabled SSLv3 and instead communicates only over TLS. If the Domino server is upgraded to 9.0.1 Fix Pack 3 (which contains JVM 1.6 SR16 FP2), the Java Console attempts to connect over SSLv3 to the JVM layer on the Domino server, which will accept only TLS connections. Applying 9.0.1 Fix Pack 3 on both the Domino server and the Java Console client will remedy the situation. For additional information, see technote 1695943 - Domino Console fails to connect to remote server after upgrading Notes or Domino to 9.0.1 Fix Pack 3


    Planned Domino 9 SLES 12 Support

    Daniel Nashed  30 January 2015 00:25:38
    The question for SLES 12 has been raised during IIBM ConnectED. There is an issue with Domino on SLES 12 and SLES 12 is not currently supported (in contrast with RHEL 7).

    There is a SPR # YXYX9RA56Z "Error - Unable to Bind port 443 or 80" on SUSE12.

    I have checked in the Lab and got a similar info than what has been posted before on the web:
    "There is a known issue with SLES 12 where bindsock has issues. Before we can support SLES 12 and any other newer kernel with this issue, we will have to identify the issue and get it fixed - bindsock and it's code has never changed in this area so the issue is a change in behavior in the kernel".

    So for now you should not try to run Domino on SLES 12. It's not just not supported yet but it also it doesn't work yet.

    There is a chance in the kernel that SLES 12 that causes the bindsock operation to fail.

    Some  background: Ports below 1024 are restricted and need root permission. Bindsock has the sticky bit set, runs with root permission and is used to allow Domino to use ports like HTTPS, SMTP and all other restricted ports.
    The way that Domino implements this does not work on the current kernel any more.

    IBM is working on it closely with Novell and the following is the official statement from IBM about it:

    "We intend to support SLES12, just as we have supported every other SLES support. Our current goal is to add support with 9.0.1 FP4. We are working with the vendor to identify root cause of an intermittent bindsock issue we found during testing that is currently preventing us from supporting this version."



      ConnectED Session Slides posted BP102: Practical IBM Notes and Domino Internet Security

      Daniel Nashed  27 January 2015 23:12:19
      Today I had the pleasure to present with Dave Kern about Domino internet security.
      Now that the presentation is public, I can speak about all the details that we presented. See the slides for all details.

      We covered what is already available in 9.0.1 FP3 and what is coming after FP3 quite soon.
      In the session demo we had a the SSL Test website showing a A- up to A+ rating depending on the configuration.

      There is a lot good stuff coming up in a scheduled interims fix. This includes TLS 1.2 and some new ciphers.

      Huge thanks to Dave, the security team and a other developers making it happen so fast. Very impressive work!

      You will find a final version of the slides on my "IBM Connected/Lotusphere" presentation page.

      http://www.nashcom.de/lotusphere

      If you have questions let me know by mail.




      Notes/Domino 9.0.1 FP3 has shipped

      Daniel Nashed  21 January 2015 18:06:14
      Today Notes/Domino 9.0.1 FP3 has been shipped.
      Already installed it on my production server.

      There are new new "SSL/TLS" releated fixes in FP3. But there are updates planned after FP3.
      So updating to FP3 is the base and you should consider an update soon.
      It's always better to install a FP than a IF which is technically a combo hotfix.

      There are also a couple of other important fixes in FP3.
      When you look into the Fixlist you see a couple of database/DAOS releated fixes.
      The FP also contains an updated JVM (SPR# KLYH9MKHPD - Updated the embedded JVM in Notes/Domino to Oracle July 2014 Critical Patch )

      Just to mention some of the fix areas. Check the full fixlist for details.




      Domino TLS POODLE Fix released

      Daniel Nashed  21 December 2014 00:30:09
      As reported before the IF that introduced TLS 1.0 is vulnerable to the new PODDLE issue.
      IBM released a new IF for all supported versions that fixes this issue.
      After installing the IF you can re-enable the CBC ciphers which are now reported as not vulnerable by the SSL Labs Test site.

      In addition to this fix IBM officially introduces a new notes.ini variable to disable SSL V3.

      DISABLE_SSLV3=1 will disable SSL V3 completely. But as mentioned before you should be completely sure if you want to completely disable SSL V3.

      SPR #KLYH9QXMQE: Disable SSL ini:
      SPR #KLYH9RMJGL: CVE-2014-8730 TLS 1.x Padding Vulnerability

      There is a reference technote and a list of IFs for all supported releases.

      Security Bulletin: TLS Padding Vulnerability affects IBM Domino (CVE-2014-8730)
      http://www.ibm.com/support/docview.wss?uid=swg21693142


      Fixes for this issue are currently available

          9.0.1 Fix Pack 2 Interim Fix 3
          9.0 Interim Fix 7
          8.5.3 Fix Pack 6 Interim Fix 6
          8.5.2 Fix Pack 4 Interim Fix 3
          8.5.1 Fix Pack 5 Interim Fix 3

      New-Domino-POOLE-Iussue-now-with-TLS

      Daniel Nashed  10 December 2014 01:16:10
      There is a new exploit that affects TLS! Not all implementations of TLS are affected.
      But Domino and also some other solutions like the F5 load-balancer are on the list.


      For more details read -->
      https://community.qualys.com/blogs/securitylabs/2014/12/08/poodle-bites-tls

      The problem effects all CBC ciphers. IBM is working on a solution.
      Meanwhile you  can disable the CBC ciphers. Currently there are only two ciphers left.
      Not really completely what we want but it sounds like IBM is working on supporting more ciphers and later TLS versions (some ciphers only work with TLS 1.2).

      But the first priority is to fix the currently supported ciphers.

      As a work around for now change the cipher list in the Server doc / Internet Site doc. But this would not be uses for all protocols.

      The better way would be to specify the cipher list via notes.ini and have it set for all internet protocols.


      There is a notes.ini setting that can be used to specify the ciphers quite low-level with a single hex-byte list.


      SSLCipherSpec=0405


      Here is the corresponding list for the ciphers. We currently only want the RC4 ciphers -- even they are not the latest ciphers. But they are not affected by the new POODLE issue.


      04 - SSL_RSA_WITH_RC4_128_MD5

      05 - SSL_RSA_WITH_RC4_128_SHA


      By the way: Also the currently unsupported notes.ini parameter that completely disables SSL 3.0 has leaked in a open mic session.

      IBM is planning to change it to a more proper name and will document it.

      Surprisingly the parameter cannot be found on Google yet... It is going to change soon, so I will not post it in a blog.

      iNotes Redirect without Anonymous Access

      Daniel Nashed  5 December 2014 16:24:50
      When running iNotes you might only want to allow authenticated connections to your Domino Server over HTTP.
      But on the other side you want to use the iNotes Redirect database which contains some images and other design that should load even the user is not yet authenticated.

      There is a Wiki article that describes in detail what to do. Thanks to IBM pointing out that parameter!
      http://www.lotus.com/ldd/dominowiki.nsf/dx/Allowing_Anonymous_Access_to_iNotes_Redirect_images__while_preventing_Anonymous_Access_to_Domino_HTTP_
       

      Starting with Domino 8.5.2 you can allow additional URLs that an anonyous users can access even the server allows only authenticated access.

      You can either allow the full database:

      HTTPPublicUrls=/redir.nsf/*

      Or in a more paranoid configuration only allow the specifc design elements which should be available -- example for "redir.nsf"

      If using Domino 8.5.3 or Lower version of the IBM Redirector ini should be set like this:

      HTTPPublicUrls=/redir.nsf/iNotes-LoginBanner65short.gif:/redir.nsf/StylesheetLogin:/redir.nsf/Login.js


      If using Domino 9 or Later version of the IBM Redirector  ini should be set like this:

      HTTPPublicUrls=/redir.nsf/IBMLogo.gif:/redir.nsf/StylesheetLogin:/redir.nsf/Login.js


      Note: You have to restart the HTTP task (e.g. restart task http)

      -- Daniel
       

      Short Description Creating a Domino Keyring File with the new Keyring Tool and a Windows CA using Binary Formats

      Daniel Nashed  2 December 2014 09:52:36
      Now that more and more customers are using the new keyring tool we run into interesting constellations.
      Microsoft uses binary formats instead of the ascii based PEM format that the keyring tool requires.

      Openssl does not only help you to create the key and the certficates. You can also use it to convert the certificate formats.


      I have written a short step by step short documentation for my customer including some troubleshooting steps and tricks.

      To keep it short I have left out the results from the commands. But you find this information in the official IBM documentation (http://www.lotus.com/ldd/dominowiki.nsf/dx/Domino_keyring)


      -- Daniel



      Short Description Creating a Domino Keyring File with the new Keyring Tool and a Windows CA using Binary Formats


      First of all you have to find a machine with openssl installed. The most easy way would be to log into any type of Linux machine with a current openssl version installed.
      But there are also Windows implementations.

      Create Private/Public Key


      In the first step you create a private public key using the standard openssl command-line.


      openssl genrsa -out server.key 4096



      Create Certificate Request


      In the next step you create a certificate request and have it processed by your CA.

      The command-line will prompt you for country, organisation name etc and the name of your server.

      That would be the DNS name of the server for example "
      www.nashcom.de" or in case you want to generate a wildcard certficate "*.nashcom.de"


      openssl req -new -sha256 -key server.key -out server.csr



      Convert Binary Certificate Files to PEM Format


      The Domino kyrtool requires the text based PEM format but the Microsoft CA does generate binary files.

      The following commands can be used to convert the formats.


      First of all you convert the server certificate to from binary DER format to text based PEM format:


      openssl x509 -inform der -in server.cer -outform pem -out server.pem


      In the next step you convert the certificate chain from p7b binary format to PEM format as well


      openssl pkcs7 -print_certs -inform der -in certificate_chain.p7b -outform pem -out chain.pem



      Combine Key, Cert and Chain to a single file


      The most easy way to import all certificates you combine all parts into a single PEM formatted file.

      The order in the file must match leave to root order. In most cases you just copy all party into a single file.


      copy server.key+server.pem+chain.pem all.pem


      Now you have all parts in a single file in the right order.

      Note: In some cases the server certificate is already in the chain file. In that case you have to ensure that the certificate is only listed once



      Create Keyring File


      Download and install the new kyrtool -- just copy it into the Notes program directory.


      Afterward just create a new keying file and set a password.

      Note: Password only has to be entered the first time you access the keyring afterwards it is automatically read from the sth file.


      c:\Notes\kyrtool.exe create -k c:\cert\mykeyring.kyr -p SecurePassword



      Import Key, Cert and Chain


      In the next step you just import all parts we added to the single file into your keyring.


      c:\Notes\kyrtool.exe import all -k c:\cert\mykeyring.kyr -i c:\cert\all.pem



      Troubleshooting / Verification


      There are two interesting options that might be helpful.


      First of all before you import the certificates you can verify the file is complete and all certificates in the chain are present and matching.


      C:\Notes\kyrtool.exe verify c:\cert\all.pem


      The result be as follows:

      - A private key should be present

      - The should be no missing certs in the chain or mismatches

      - The last certificate in the chain is self-singed (root certificate)


      Another way to show the certs and dump them might be useful as well.

      This command line shows you all the certs with detailed information about each part.


      C:\Notes\kyrtool.exe show certs -i c:\cert\all.pem


      Verify existing keyring files


      If you want to verify an exiting keyfile you can combine "show certs" output into a file with a "verify" command on that output file.

      The only error you should have with that verify is that the private key is missing.
      That helps to verify that a keyring file has a complete chain.