Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

New-Domino-POOLE-Iussue-now-with-TLS

Daniel Nashed  10 December 2014 01:16:10
There is a new exploit that affects TLS! Not all implementations of TLS are affected.
But Domino and also some other solutions like the F5 load-balancer are on the list.


For more details read -->
https://community.qualys.com/blogs/securitylabs/2014/12/08/poodle-bites-tls

The problem effects all CBC ciphers. IBM is working on a solution.
Meanwhile you  can disable the CBC ciphers. Currently there are only two ciphers left.
Not really completely what we want but it sounds like IBM is working on supporting more ciphers and later TLS versions (some ciphers only work with TLS 1.2).

But the first priority is to fix the currently supported ciphers.

As a work around for now change the cipher list in the Server doc / Internet Site doc. But this would not be uses for all protocols.

The better way would be to specify the cipher list via notes.ini and have it set for all internet protocols.


There is a notes.ini setting that can be used to specify the ciphers quite low-level with a single hex-byte list.


SSLCipherSpec=0405


Here is the corresponding list for the ciphers. We currently only want the RC4 ciphers -- even they are not the latest ciphers. But they are not affected by the new POODLE issue.


04 - SSL_RSA_WITH_RC4_128_MD5

05 - SSL_RSA_WITH_RC4_128_SHA


By the way: Also the currently unsupported notes.ini parameter that completely disables SSL 3.0 has leaked in a open mic session.

IBM is planning to change it to a more proper name and will document it.

Surprisingly the parameter cannot be found on Google yet... It is going to change soon, so I will not post it in a blog.

iNotes Redirect without Anonymous Access

Daniel Nashed  5 December 2014 16:24:50
When running iNotes you might only want to allow authenticated connections to your Domino Server over HTTP.
But on the other side you want to use the iNotes Redirect database which contains some images and other design that should load even the user is not yet authenticated.

There is a Wiki article that describes in detail what to do. Thanks to IBM pointing out that parameter!
http://www.lotus.com/ldd/dominowiki.nsf/dx/Allowing_Anonymous_Access_to_iNotes_Redirect_images__while_preventing_Anonymous_Access_to_Domino_HTTP_
 

Starting with Domino 8.5.2 you can allow additional URLs that an anonyous users can access even the server allows only authenticated access.

You can either allow the full database:

HTTPPublicUrls=/redir.nsf/*

Or in a more paranoid configuration only allow the specifc design elements which should be available -- example for "redir.nsf"

If using Domino 8.5.3 or Lower version of the IBM Redirector ini should be set like this:

HTTPPublicUrls=/redir.nsf/iNotes-LoginBanner65short.gif:/redir.nsf/StylesheetLogin:/redir.nsf/Login.js


If using Domino 9 or Later version of the IBM Redirector  ini should be set like this:

HTTPPublicUrls=/redir.nsf/IBMLogo.gif:/redir.nsf/StylesheetLogin:/redir.nsf/Login.js


Note: You have to restart the HTTP task (e.g. restart task http)

-- Daniel
 

Short Description Creating a Domino Keyring File with the new Keyring Tool and a Windows CA using Binary Formats

Daniel Nashed  2 December 2014 09:52:36
Now that more and more customers are using the new keyring tool we run into interesting constellations.
Microsoft uses binary formats instead of the ascii based PEM format that the keyring tool requires.

Openssl does not only help you to create the key and the certficates. You can also use it to convert the certificate formats.


I have written a short step by step short documentation for my customer including some troubleshooting steps and tricks.

To keep it short I have left out the results from the commands. But you find this information in the official IBM documentation (http://www.lotus.com/ldd/dominowiki.nsf/dx/Domino_keyring)


-- Daniel



Short Description Creating a Domino Keyring File with the new Keyring Tool and a Windows CA using Binary Formats


First of all you have to find a machine with openssl installed. The most easy way would be to log into any type of Linux machine with a current openssl version installed.
But there are also Windows implementations.

Create Private/Public Key


In the first step you create a private public key using the standard openssl command-line.


openssl genrsa -out server.key 4096



Create Certificate Request


In the next step you create a certificate request and have it processed by your CA.

The command-line will prompt you for country, organisation name etc and the name of your server.

That would be the DNS name of the server for example "
www.nashcom.de" or in case you want to generate a wildcard certficate "*.nashcom.de"


openssl req -new -sha256 -key server.key -out server.csr



Convert Binary Certificate Files to PEM Format


The Domino kyrtool requires the text based PEM format but the Microsoft CA does generate binary files.

The following commands can be used to convert the formats.


First of all you convert the server certificate to from binary DER format to text based PEM format:


openssl x509 -inform der -in server.cer -outform pem -out server.pem


In the next step you convert the certificate chain from p7b binary format to PEM format as well


openssl pkcs7 -print_certs -inform der -in certificate_chain.p7b -outform pem -out chain.pem



Combine Key, Cert and Chain to a single file


The most easy way to import all certificates you combine all parts into a single PEM formatted file.

The order in the file must match leave to root order. In most cases you just copy all party into a single file.


copy server.key+server.pem+chain.pem all.pem


Now you have all parts in a single file in the right order.

Note: In some cases the server certificate is already in the chain file. In that case you have to ensure that the certificate is only listed once



Create Keyring File


Download and install the new kyrtool -- just copy it into the Notes program directory.


Afterward just create a new keying file and set a password.

Note: Password only has to be entered the first time you access the keyring afterwards it is automatically read from the sth file.


c:\Notes\kyrtool.exe create -k c:\cert\mykeyring.kyr -p SecurePassword



Import Key, Cert and Chain


In the next step you just import all parts we added to the single file into your keyring.


c:\Notes\kyrtool.exe import all -k c:\cert\mykeyring.kyr -i c:\cert\all.pem



Troubleshooting / Verification


There are two interesting options that might be helpful.


First of all before you import the certificates you can verify the file is complete and all certificates in the chain are present and matching.


C:\Notes\kyrtool.exe verify c:\cert\all.pem


The result be as follows:

- A private key should be present

- The should be no missing certs in the chain or mismatches

- The last certificate in the chain is self-singed (root certificate)


Another way to show the certs and dump them might be useful as well.

This command line shows you all the certs with detailed information about each part.


C:\Notes\kyrtool.exe show certs -i c:\cert\all.pem


Verify existing keyring files


If you want to verify an exiting keyfile you can combine "show certs" output into a file with a "verify" command on that output file.

The only error you should have with that verify is that the private key is missing.
That helps to verify that a keyring file has a complete chain.

Traveler 9.0.1 IF7

Daniel Nashed  7 November 2014 08:48:22
Finally Traveler 9.0.1 IF7 is available.

I don't see a fixlist yet but I got a fixlist from a customer from the latest hotfix he got.


The IF should fix all attachment issues which came up with IF6, includes the latest Android client and should also have an updated certificate for APNS.


So now you can install 9.0.1 IF7 in combination with Domino 9.0.1 FP2 IF1 which introduces TLS 1.0 in one go with just one downtime.

FixCentral Download Link:


http://www.ibm.com/support/fixcentral/swg/selectFixes?parent=ibm~Lotus&product=ibm/Lotus/Lotus+Notes+Traveler&release=9.0.1&platform=All&function=all


-- Daniel

    Some Additonal TLS 1.0 Information

    Daniel Nashed  6 November 2014 16:28:26
    TLS 1.0 and the removal of SSL 3.0 from browsers that triggered the whole discussion is not just something that needs to be addresses on a Domino server.
    IBM has done a lot of work in quite a short time and now that customers are implementing the fix it shows that also other software is effected.

    Introducing TLS 1.0 for Domino was the first step from IBM to ensure that clients that only support TLS 1.0 and higher can still connect to the Domino server.
    For now IBM still has SSL 3.0 enabled to allow communication with software that does not yet support TLS 1.0 and they are preventing clients from the downgrade attacks as mentioned in the IBM technotes.

    Notes Client Software

    But Domino is not the only server for most customer environments. Many companies completely disable SSL 3.0 and cause issues with other client software.
    And also Notes Clients are affected for example when connecting to other HTTP servers or using secure IMAP, POP3, LDAP or SMTP.

    For example here in Germany GMX one of the larger, well known email-providers disabled SSL 3.0.
    In that case you need a fix for the Notes Client side. IBM did not yet ship the full set of clients because they are waiting for some I think unrelated Java patches.

    But because there is also an enhancement for SHA256 for the cert request database, IBM shipped already the Win32 Standard Client.
    The download is a bit more difficult to find on Fix Central but you should find it using the following link.

    http://www.ibm.com/support/fixcentral/swg/quickorder?parent=ibm%7ELotus&product=ibm/Lotus/Lotus+Notes&release=9.0.1.2&platform=Windows&function=fixId&fixids=Notes_901FP2IF2_W32_Standard&includeSupersedes=0&source=fc

    If you need a client connection for one of the internet protocols and the server does only support TLS 1.0, you will need to install this IF.

    Other Client Software -- Other Issues


    Notes/Domino is not the only application having an issue with servers that don't support SSL 3.0 any more or servers that changes the way they negotiate SSL versions.
    Domino for example has SSL 2.0 including the SSL 2.0 handshake disabled with 9.0.1 FP2 IF1. Other servers might have done the same or similar.

    This leads to interesting interoperability challenges. For example older OpenSSL versions do not nicely negotiate their SSL level with Domino servers without explicitly specifying TLS 1.0 as Andrew Pollack found out.
    In case of wget in combination with an older OpenSSL version according to IBM the negotiation failed because that OpenSSL version used an V2 handshake, which failed and stopped the negotiation.

    And there might be other application issues where a server does not work nicely with your Java 1.6 application (which supports TLS 1.0 but maybe not the ciphers the server is expecting).
    Java 1.7 does also support TLS 1.1 and 1.2 but not in all cases you can switch to the later Java version.

    I have tested Java 1.6 in Notes with an unpatched Notes client against a server that does have SSL 3.0 completely disabled and the Java agent worked unmodified.
    But there are other parts in Notes that use the native SSL stack. And from what I heard from IBM some parts in Java also seem to use the native Notes/Domino stack instead of the Java stack.


    So when the browser vendors decided to stop supporting SSL 3.0 any more they did not just challenge the server vendors but because of the impact to client software all applications using SSL connections might be affected.
    When introducing new versions of software that support TLS and might not support SSL 3.0 at all or have a changed way to negotiate the session, you really have to test all your applications and see which SSL level they support and which types of ciphers.


    The SSL Test website (https://www.ssllabs.com/ssltest/) tries to test what happens when you access your server with various client software and you should have a look if your server does support a cipher for all of you client access types.

    As I said, this is not just a challenge for Domino but also for other applications -- even if they are totally unrelated -- because many vendors are working on their SSL stack (or administrators disabling SSL 3.0 and below).

    Sometimes you have to specify the right SSL level (currently TLS 1.0 for Domino) to establish a connection and that could be even good from security point of view.

    On the others side you might have to think about updating the software on the client machine itself. For example older versions of OpenSSL should be updated to solve SSL handshake issues.

    There are many parts you should test. And this post should just give you some more background and a heads up what could break now or in the near future when more and more servers are patched/reconfigured. In many cases the solution is to update your software.


    -- Daniel



    Domino TLS 1.0 SHA-2 Support to prevent POODLE has been shipped today

    Daniel Nashed  4 November 2014 01:14:16


    As blogged before IBM was already working on addressing the POODLE attack by finally implementing TLS 1.0 for all internet protocols.


    Today IBM shipped an Interims Fix to introduce TLS 1.0 which is very important because many browsers and other software vendors are about to drop SSL 3.0 support.

    So you need those fixes to continue to use secure protocols like HTTS, secure SMTP, LDAP, IMAP, POP3, DIIOP..


    There are a couple of changes which are described in the following Wiki documents. And there are a couple of additional Wiki documents providing additional information.


    Basically this fix will allow TLS 1.0 and also allows you to use SHA-2 based certificates with a new introduced command-line key-ring tool called "kyrtool".

    The tool is a command line application that can manage your keyring files with SHA-2 support and you don't need the old ikeyman tool that many of us used before with all those limitations.

    I have been testing the tool on Windows and Linux and it is working like a charm. The Wiki contains step by step instructions how to use it in combination with openssl to generate a private key, signing requests and import trusted roots and certficates.


    You find very detailed step by step documentation in the referenced links.


    And you can start downloading the fix and the kyrtool today!
    I have it already running on my production Traveler server on Linux 64.


    Here are the details including download links and detailed descriptions.


    For TLS 1.0 support you just need to install the hotfix and all the defaults should just work fine. You need no additional settings.

    Note: IBM did not disable SSL 3.0 for compatibility reasons in this fist step. The first IF is intended to introduce TLS 1.0 to allow all applications to continue to work with Domino.


    Domino with this fix prevents a downgrade attacks if the client requested TLS 1.0. Some applications will still report that your server is vulnerable to POODLE because Domino still supports SSL 3.0 but this is not completely true. That's just a basic check for SSL 3.0.


    IMHO introducing TLS 1.0 in combination with preventing downgrade protocol attacks is the right first move.

    The fixes are available for  all supported platforms and releases (9.0.1 FP2, 9.0, 8.5.3 FP6, 8.5.2 FP4, 8.5.1 FP5).


    But you should be aware that SHA-2 is only available in Domino 9.0.x because 8.5.x releases "lack the cryptographic infrastructure for SHA-2. "



    Thanks to IBM and specially the security team who did a great job in a very short time!

    They have been already working on TLS and SHA-2 support before but had to change their plans because of the short term move to diable SSL 3.0 in browsers and other software.


    Here is the official quite detailed IBM documentation for TLS, SHA-2, the new key-ring tool "kyrtool" and information about how IBM addressed the "POODLE attack" with this fix.


    -- Daniel



    IBM Domino Interim Fixes to support TLS 1.0 which can be used to prevent the POODLE attack

    http://www.lotus.com/ldd/dominowiki.nsf/dx/IBM_Domino_TLS_1.0

    Generating a SHA-2 Keyring file

    http://www.lotus.com/ldd/dominowiki.nsf/dx/Domino_keyring

    IBM will add more articles in these categories around troubleshooting, tracing, and so on.


    http://www.lotus.com/ldd/dominowiki.nsf/xpViewTags.xsp?categoryFilter=SHA-2
    http://www.lotus.com/ldd/dominowiki.nsf/xpViewTags.xsp?categoryFilter=TLS

    TLS and SHA-2 Support and the POODLE Attack

    Daniel Nashed  21 October 2014 19:20:27

    IBM has officially responded to the POODLE attack and also officially responded to newer crypto standards.

    Very good news for Domino! IBM will introduce TLS 1.0 and SHA-2 support for all protocols soon!

    The current technotes mention a very short timeframe and it looks like we are going to get fixes at least for the current Domino 9.0.1 code stream.

    Some fixes will be also in the 8.5.x code-stream but some of the improvements like SHA-2 support cannot be back ported.


    So you should be prepared with all your internet facing servers to deploy 9.0.1 with current fixpack 2!


    IBM will introduce support for the current standards soon which will also address the POODLE attack.

    IMHO the risk right now is not very high and and most of the HTTPS internet facing servers for larger companies already use Secure Reverse Proxies.
    And you might need to have a closer look into what crypto levels those server currently support! You should disable SSL 3.0 on all servers as far as it is currently possible. This is not just true for Domino.


    IBM is working on improving the crypto "stack" (part of the lower network layer -- "NTI" which is the base for all Internet protocols and this includes in consequence also the keyring file used to store your internet certificates) in Domino in a short timeframe.


    Enclosed you find links for the two new technotes which provide details about what IBM is working on...


    We have been asking for years specially for TLS 1.0 and higher support for SMTP. Now it looks like we are getting it also for all other internet protocols!


    That's really great news!!


    -- Daniel



    How is IBM Domino impacted by the POODLE attack?


    http://www.ibm.com/support/docview.wss?uid=swg21687167


    Planned SHA-2 deliveries for IBM Domino 9.x


    http://www.ibm.com/support/docview.wss?uid=swg21418982

    Traveler Issues with Attachments containing special chars after updating to 9.0.1 IF6

    Daniel Nashed  27 September 2014 12:57:17

    Before leaving for holidays last week the first customer contacted me about issues with attachments that have blanks, umlauts or other characters in the attachment name.

    I could not reproduce it on iOS but on Android but without the error message in the log that he got.


    Meanwhile it is clear that this issue affects all devices types and there is a fix that should hopefully address this problem.


    IBM is working on a new IF to address the issue and also possible other related issues but meanwhile if you need to install IF6 for full iOS 8 support (issues with calendar, companion app) before the new IF is released you should request a fix from IBM by opening a PMR.


    There is a ARPA LO82085 describing the problem but it is just mentioning the "+" sign. But the problem is more general.


    Reference for this ARPA is -->
    https://www.ibm.com/support/entdocview.wss?uid=swg1LO82085

    Here is a sample error message


    19.09.2014 07:14:09   Notes Traveler: WARNING Could not find the StreamedDataInfo: Key(86 bytes)=...√úbersetzungen.doc@547EF898341C988FC1257D58001C6445_48980, RefId=... √úbersetzungen.doc@547EF898341C988FC1257D58001C6445, DataSize=-1, EncodingºSE64, StreamingCompleted=false, so the data was not streamed



    In addition that you cannot open those attachments on mobile devices we noticed a higher CPU utilization which should be related to this problem and can be a lot higher.


    Hopefully we get a new IF soon. I will keep you posted...

    Update 28.9.2014: There are multiple hotfixes for different issues and not all problems are solved.
    If you don't use the companion or todo app you should stay on 9.0.1 IF5 and wait for IF7.

    The problem(s) seem to be more complicated to fix.


    -- Daniel



    My Top 3 Formula Commands for working in the Notes Client

    Daniel Nashed  19 September 2014 06:53:48
    All of those commands are not new at all. They are all round for a very long time. But they make my day easier.
    I am surprised that many still don't know at least the first two.
    The last one is more a convenience when working with replicas.

    @Command([AdminRemoteConsole])

    Before Release 5 there wasn't an admin client and the admin/designer was integrated into the normal client.
    The old live console is still in the client and you don't need an admin client -- just the right permission.
    You can launch it from a smart icon and have a simple admin console without starting the admin client.

    @Command([Execute];"notepad"; @ConfigFile)

    Actually that is a combination of two things.
    @ConfigFile returns the location of your notes.ini. This can be very helpful if you at another user's Notes Client and don't know where the notes.ini is located.
    You can create a new memo (CTRL-M) write the text in the subject and press Shift + F9 to evaluate the formula.
    By the way Shift + F9 works in every field and you could also use it as a calculator.

    In combination with the execute this opens the notes.ini in a notepad for editing.

    @Command( [ReplicatorSendReceiveMail] )

    If you are using local replicas and got a new mail notification but the mail is not yet replicated this command helps to just send and receive mail.

    All three commands are not really new but you might have forgotten about them...


    -- Daniel

      Important Update on Traveler iOS 8 Support -- You have to install an IF!

      Daniel Nashed  15 September 2014 22:23:53


      There are some last minute changes in iOS which are only in the final version.

      Apple changed the EAS Sync ID which used to match the Device ID. There has been planning for that change for a while but Apple should have introduce that change already in the Beta releases.
      However this change causes issues in device mapping for the companion/todo app.

      IBM released a IF for 9.0.1/9.0.0.1/8.5.3 UP2 today to address this issue and added some background logic to map the device ID.

      There is a ARPA describing the issue https://www.ibm.com/support/entdocview.wss?uid=swg1LO81842

      The problem mainly occurs when you register new devices and causes issues with todo and companion app.
      Existing devices with existing profiles should keep their ActiveSync Device ID. But you will run into issues with new registered users and companion/todo app.

      The IF does also address a couple of other issues. Some are also iOS 8 releated.

      You should update your Traveler servers ASAP.

      Detlev put together a nice details description of what happens in the backend.
      See his blog for additional details --> http://www.netzgoetter.net/internet/blogs/netzgoetter.nsf/dx/traveler-ios-8-why-you-should-update-your-servers.htm

      References for fixes for all supported versions.
      You should update even if you are not using the companion/todo app.

      IBM should have sticked with their own rule not announcing support for something that has not yet shipped.
      But we as partners and customers wanted to know in advance what version will support iOS 8.

      Thanks for this very fast response from the Traveler team!
      There are always changes in new software releases and I was surprised that we get a support statement before the release.

      -- Daniel


      9.0.1 IF6
      http://www.lotus.com/ldd/dominowiki.nsf/dx/Lotus_Notes_Traveler_APAR_listing#901IF6

      9.0.0.1 IF7
      http://www.lotus.com/ldd/dominowiki.nsf/dx/Lotus_Notes_Traveler_APAR_listing#9001IF7

      8.5.3 UP2 IF7
      http://www.lotus.com/ldd/dominowiki.nsf/dx/Lotus_Notes_Traveler_APAR_listing#853UP2IF7