Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

First Perfect Forward Secrecy Ciphers shipped with 9.0.1 FP2 IF2

Daniel Nashed  30 March 2015 13:14:58
As posted before IBM shipped a new IF that introduces TLS 1.2 Along with this new version a set of ciphers have been added.
Some of them are enabled by default and other can be enabled using notes.ini settings.
Other ciphers that are regarded as "weak" have been removed from the default cipher list.

So by default without any additional settings you get the ciphers that IBM currently recommends.
What has been added to the default are the AEAD (AES-GCM) ciphers -- see details below.

There are additional ciphers that will provide "Perfect Forward Secrecy" (PFS) for some platforms/browsers.

IBM implemented Ephemeral Diffie-Hellman (DHE) ciphers. Those ciphers are used by many but not all platforms.
That's why even if you enable them you the SSL Test Site will not give you a better rating because not all the reference browsers will use PFS.

In addition those ciphers have a higher overhead to your Domino Server. Therefore IBM left the decision which cipher to add to administrators.
You have to find the right balance between security and performance.
Probably on a smaller server it will not have that much overhead. But on a larger server you might want to take special care and watch the CPU load of your server before and after you enabled the DHE ciphers!

The current default setting is that the cipher order on the server takes preference.

As mentioned before all the fixes currently have no design change because that will have to wait until 9.0.2.
Therefore also the cipher spec has to be enabled using notes.ini settings as already described in our ConnectED presentation.

There is a notes.ini setting described in a recent Wiki entry. Each cipher has an internal reference number that is standard.
Domino uses the two digit hexadecimal number to specify the ciphers you want to have enabled on your server.
The order of entries does not matter. You just have to make sure that you always use a two digit value per cipher -- even the cipher itself might have just one hex digit.
There is no space between the cipher numbers.

Here is what you get by default without any changes:




In addition to that you have the folllowing new DHE ciphers available.


So as an example when you want to enable all DHE ciphers and keep the other ciphers you set the following notes.ini setting and restart the servertasks like http.


So you could add those ciphers to your cipher list using the notes.ini setting.
Once you are done you can use the SSL Labs Test Website to check if the ciphers are properly configured.
What is nice on the website is that the website will "simulate" which client type will probably use which type of cipher when connecting given the current settings of your server.

Now you should have all the default ciphers and the DHE ciphers enabled.

You should take special care which ciphers to disable because you could block out certain devices types.

When testing with the SSL Tabs Test and also using Java applications I noticed that they will pick the DHE ciphers.
But Java 1.6/1.7 does currently not support more that 1024 bits. By default Domino uses higher key-length.

So Java sees that DHE ciphers are enabled and will try to use them. And it does not check before using it that it cannot handle larger key sizes than 1024.

That means if you enable DHE ciphers you might have to consider to lower the key-length used.
If you change the key-length to 1024 the SSL Labs Test site will report that your key is "weak".

So you have to balance lower security with compatibility at this point.

There is a notes.ini setting to specify the key-length for DHE ciphers.

You could set notes.ini SSL_DH_KEYSIZE=1024 to resolve this incompatibility.

There have been also discussions about other PFS ciphers that are used by other applications like older IE versions.

"Elliptic Curves ciphers" (ECDHE..) are supported by older IE versions and by Windows mobile.
But they are currently not implemented on the Domino side.

All the development work in this area based by priorities and demand. And IBM is releasing it step by step with IF fixes.
It's not confirmed IBM is working on those type of ciphers. I just wanted to mention it to explain why not all platforms will use PFS ciphers when you enable the DHE ciphers.
Also the ECDHE ciphers have better performance than the DHE ciphers. But the first priority was to implement the DHE ciphers because most platforms support it.
This was for sure not the last functionality update we get via a IF. I am looking forward to see that is next on the list.

Not all of the notes.ini settings are documented yet. I expect that IBM will publish another Wiki article soon.
I might update this blog entry or have a more complete article with more details as soon more information is available.

-- Daniel

Domino 9.0.1 FP3 IF3 is about to ship

Daniel Nashed  29 March 2015 12:33:52

Updated post:

IF2/IF3 already shipped. There is also a Wiki articile describing the changes.

The Fixlist for IF2/IF3 is confusing but it looks like the Wiki article explains it.


The fixes have the release date of 27.3. the client fixes are labled "IF3", the server fixes are labled "IF2".

Here is what the fixlist says and see my comments in-line.
You should also read the Wiki entry which will hopefully also have the settings for the PFS ciphers soon.

I have installed 9.0.1 FP2 IF2 on my production Linux Server.

And I can confirm that TLS 1.2 is implemented in this version and it looks like just the fixlist is confusing.

The fixes listed in the fixlist section "IF3" are included in server fixes labled "IF2". The right client release is "IF3" in contrast.

Without any additional settings this brings you to TLS 1.2 support with the following ciphers which brings Domino to a "A-" rating.

TLS_RSA_WITH_AES_256_GCM_SHA384 (0x9d)
TLS_RSA_WITH_AES_128_GCM_SHA256 (0x9c)
TLS_RSA_WITH_AES_256_CBC_SHA256 (0x3d)
TLS_RSA_WITH_AES_256_CBC_SHA    (0x35)
TLS_RSA_WITH_AES_128_CBC_SHA256 (0x3c)

TLS_RSA_WITH_AES_128_CBC_SHA    (0x2f)

The "A-" is because of missing PFS support for reference browsers.

As mentioned in the wiki article and also in the fixlist IBM also implemented some PFS ciphers.

"Perfect Forward Secrecy (PFS) via Ephemeral Diffie-Hellman (DHE)"

But those ciphers are disabled by default because they have higher overhead on the server and client side.
I will have a separate post for the PFS cipher support as soon official information is available.

Here is the commented SPR list

9.0.1 Fix Pack 3 Interim Fix 2 SPR #PSIH9SSAHC

-- PNG Vulnerability --

libpng is vulnerable to a heap-based buffer overflow, caused by improper bounds checking by the png_combine_row function when decompressing the IDAT_data.
A remote attacker could exploit this vulnerability using a "very wide interlaced" PNG image to overflow a buffer and execute arbitrary code on the system or cause a denial of service.

You should wait for IF3 planned to be released very soon. The SPR list for the fix is already public and the IF will contain a couple of important fixes and new TLS 1.2 support including new ciphers.

Enclosed you find the current list. The information about how to enable those new ciphers are not yet released. I post information about those new settings and will comment on them as soon they are released.

IF3 will contain a couple of pending fixes for other issues. For example the fix for the Google Calendar Feed in the Standard Notes Client which broke because of the change on the Goodle side.

Here is a commented fix list for IF2/IF3:

-- 9.0.1 Fix Pack 3 Interim Fix 3 --


 Add pinning to SHA-256 for TLS 1.2        


 TLS 1.2 Notes / Domino as a TLS client rejects handshake with server if no common signature algorithm available        


 TLS 1.2 Client handshake request rejected by Server if server certificate chain signature type not supported by the client        

--> There have been issues specially with TLS SMTP Connections. Those two fixes help to connect even in those cases.


 Remove RC4-SHA from the default cipher list for TLS 1.2        

--> RC4-MD5 have already been removed before. Now also the SHA based version is rated as weak on the Domino side and disabled by default


 Implement HSTS (Http Strict Transport Security).This header informs supported browsers that the site should only be accessed over an SSL-protected connection (HTTPS)        

--> On a server that only allows authenticated connections I would only enable the SSL port and disable port 80 in general.
We have to wait for the full documentation to see in which condition the header is automatically set.
It should be automatically send when only HTTPS is enabled.


 Add IP Information to HTTP Thread logs for SSL Handshake connections        


 Passing a directory to kyrtool will crash the tool        


 kyrtool import all sometimes reports "SECIssUpdateKeyringPrivateKey returned error 0x0720", "AVA separator not found" or "Syntax error in OID" when a '/' is in a certificate name part        

--> there have been a couple cases where certificates could not be parsed correctly. This fix should solve those issues.


 Add more detailed logging for SSL/TLS connections to help diagnose failed connections        

--> More detailed information is important for figuring out what is going wrong in some cases.


 New notes.ini SSL_DISABLE_TLS_10 to support Disabling TLS1.0 for compliance reasons. Used in conjunction with existing DISABLE_SSLV3=1 allows you to limit communication to TLS 1.2 only for protocols: HTTP, SMTP, LDAP, POP3 & IMAP        

--> For now I would disable SSLv3 only and keep TLS 1.0 enabled unless you are working in an controlled environment like an intranet and you know exactly that all clients support TLS 1.2.


 Added SHA-256 cipher specs for increased security with TLS 1.2        


 Added Advanced Encrption Standard (AES) Galois/Counter Mode for increased security with TLS 1.2        

--> New AES GCM ciphers. I will post details how to enable them as soon the exact implemented ciphers have been documented.

There will be documentation which ciphers are enabled by default and how to enable other ciphers.


 Added Perfect Forward Secrecy (PFS) via Ephemeral Diffie-Hellman (DHE) cipher specs for SSL/TLS        

--> New DHE ciphers which introduce PFS -- Perfect Forward Secrecy. I will post details how to enable them as soon the exact implemented ciphers have been documented.

There will be documentation which ciphers are enabled by default and how to enable other ciphers.

PFS is an important addition to allow more secure connections. This ensures that traffic cannot be recorded and decrypted later when the private key of one side gets compromised.


 Notes / Domino Support for TLS 1.2 (Transport Layer Security 1.2) with protocols: HTTP, SMTP, LDAP, POP3 & IMAP        

--> Support for TLS 1.2!!! That was announced at ConnectED to be available in Q1. Thanks to IBM and the team working on it.


 Administrator Client Shows Wrong File Sizes of database with DAOS size>0 After Server Restart        


 Getting Error When Using Google calendar Feeds         - Standard Client Only

--> Important client side fix for Google Calender integration which broke because of changes on the Google side


 [WINDOWS ONLY] - Additional Time Zone For Salvador & Buenos Aires Shows Incorrect Time - Standard Client Only

Find us at Engage Conference next Week

Daniel Nashed  26 March 2015 10:32:33
Next week many of us are travelling to Engage conference in Ghent.
I am already looking forward to an interesting conference and hopefully will see many of you there.

My presentation will be an updated version of the IBM Security Best Practices session Dave Kern and me presented at ConnectED conference in Orlando.
I will speak about the current status and the new stuff coming in end of Q1 in the area of TLS, SHA-256 and related security topics.

And as mentioned before I am working on RHEL 7 and SLES 12 systemd support for my Domino start script.
If you have questions or feedback please find me at the conference.

Also if you are interested in Domino on Linux you should attend Bill Malchisky's session "Title: Adm03.The BASHing Admins: The ICS Shell Scripting Class".
He will also will use the current version of my Domino start script in his session and will show some interesting stuff that is usefully for your daily Domino on Linux administration work

Here is a link to the track that includes our presentations.

-- Daniel

    Solution for Notes/Domino related process is still running when applying a Fixpack or Hotfix

    Daniel Nashed  25 March 2015 07:53:44
    The problem came up a couple of times and the solution seems still hard to find even it is listed in Kbase.

    When you try to install a fixpack or hotfix the installer reports that "Notes/Domino related process is still running" even Domino and NSD Service is stopped.
    It looks like that when the Notes statistics are registered on OS level the  "Windows Management Instrumentation Service" (short WMI Service) keeps Notes DLLs blocked.

    The workaround is to stop the "Windows Management Instrumentation Service" Service before starting the installation.
    This should solve the problem in most of the cases.

    If that does not help for other reasons my approach is to rename the Domino program directory, restart the server and rename it back before starting the installation.

    -- Daniel

    Fritzbox phone number lookup pre-delivery agent

    Daniel Nashed  9 March 2015 11:03:44
    There is a e-mail notification option in the Fritzbox which I am using for a while.
    But I did not find a nice way to sync my IBM Notes contacts to my Fritzbox yet.
    They offer just a connection to certain German e-mail providers.

    But since my mailfile contains all contacts, having a pre-delivery agent to do the lookup for an incoming call-notification was my "plan B".

    I build a view that ensures that the lookup can work against an international number format with +country code + area code + number without any special characters like dashes, blanks etc.
    And I did the same conversion for the number in the Fritzbox notification email.
    The hardest part was really the phone number conversion for the lookup. Everything else was very straightforward.

    I ran into one interesting special character issue when looking into converting the number.
    Apple used  hex xA0 (decimal 160) chars when you add a number with your iOS device instead of a bank (hex 0x20 - decimal 32).
    When looking up the char I was surprised because it was a "non breaking space" that I never have heard of before ->

    It works like a charm! I I just added a view to my mailfile and added the code to my pre-delivery agent that already contains my SpamGeek anti-spam folder move code.
    I someone is interested in that little solution or has an idea to sync contacts with a Fritzbox directly drop me a mail ...

    -- Daniel

    Domino Start Script systemd Support

    Daniel Nashed  6 March 2015 12:54:03
    Domino 9.0.1 FP3 IF1 also supports SLES12.
    So it is time to finish my work on systemd support which is the new service model used in RHEL7 and SLES12.

    Enclosed you find the current description of the changes in the start script for systemd support.
    Some parts really need to change to support the new model.
    But I am keeping the concept that rc_domino is the main entry point for all your operations.

    The following is a short description. I am currently writing the documentation for the new version of the script but it is not yet final.
    I have a working version for systemd support but I am still testing and making minor changes.
    If you have feedback or want to test it, let me know by mail or write a reply including your e-mail address.

    -- Daniel

    Domino Start Script systemd Support

    Beginning with RHEL7 and SLES12 Linux is using the new "systemd" ( for starting services daemons.

    All other platforms are also moving to systemd. rc scripts are still working to some extent. but it makes sense to switch to the systemd service model.

    Most of the functionality in the Domino start script will remain the same and you will also continue to use the same files and configuration. But the start/stop operations are done by a "domino.service".

    The start script will continue to have a central entry point per partition "rc_domino" but that script is not used by the "rc environment" any more. You can place the file in any location and it is not leveraged by systemd.

    systemd will use a new "domino.service" per Domino partition. the service file will directly invoke the main script logic "rc_domino_script" after switching to the right user and setting the right resources like number of open files (before this was done with "su - notes" and the limits configuration of the corresponding pam module).

    Starting and Stopping the Domino server can be done either by the rc_domino script which will invoke the right service calls in the background. Or directly using the systemd commands.

    Starting, Stopping and getting the Status

    systemctl start domino.service

    systemctl stop domino.service

    systemctl status domino.service

    Enabling and Disabling the Service

    systemctl enable domino.service

    systemctl disable domino.service

    The service file itself will be located in /etc/systemd/system.

    You have to install a service file per Domino partition.
    When you copy the file you have to make sure to have the right settings

    a.) ExecStart/ExecStop needs the right location for the rc_domino_script (still usually the Domino program directory)

    b.) Set the right user account name for your Domino server (usually "notes").

    The following example is what will ship with the start script and which needs to be copied to "/etc/systemd/system" before it can be enabled or started.

    Systemd service file shipped with the the start script


    Description=IBM Domino Server






    ExecStart=/opt/ibm/domino/rc_domino_script start
    ExecStop=/opt/ibm/domino/rc_domino_script stop





    The rc_domino script can be still used for all commands -- including starting and stopping Domino as a service (only "restart live" option is not implemented).

    You can continue to have rc_domino with the same or different names in the /etc/init.d directory or put it into any other location. It remains the central entry point for all operations.

    But the domino.service can also be started and stopped using "systemctl".

    rc_domino uses the configured name of the domino.service (in the header section of rc_domino script).

    systemd operations need root permissions. So it would be best to either start rc_domino for start/stop operations with root.

    One way to accomplish using root permissions is to allow sudo for the rc_domino script.

    The configuration in /etc/sysconfig/rc_domino_config_notes (or whatever your user name is) will remain the same and will still be read by rc_domino_script.

    The only difference is that the rc_domino_script is invoked by the systemd service instead of the rc_domino script for start/stop operations.

    When invoking start/stop live operations a combination of systemd commands and the existing rc_domino_script logic is used.

    New systemd status command

    The output from the systemd status command provides much more information than just if the service is started.

    Therefore when using systemd the rc_domino script has a new command to show the systemd status output.

    The new command is "statusd"

    How do you install the script with systemd?

    Copy rc_domino, rc_domino_script and rc_domino_config_notes to the right locations

    - Copy domino.service to etc/systemd/system.

    - Make the changes according to your environment

    - Enable the service via systemctl enable domino.service and have it started/stopped automatically or start/stop it either via systemd command or via rc_domino script commands.

    - rc_domino script contains the name of the systemd service. If you change the name or have multiple partitions you need to change the names accordingly

    How does it work?

    a.) Machine startup

    When the machine is started systemd will automatically start the domino.service.

    The domino.service will invoke the rc_domino_script (main script logic).

    rc_domino_script will open rc_domino_config_notes for configuration.

    b.) Start/Stop via rc_domino

    when rc_domino start is invoked the script will invoke the service via systemctl start/stop domino.service

    c.) Other script operations

    Other operations like "monitor" will continue unchanged and invoke the rc_domino_script.

    SSL V2 HELO can be re-enabled with 9.0.1 FP3 IF1

    Daniel Nashed  25 February 2015 20:45:11
    As discussed before the security fixes introduced with the additon of TLS 1.0 removed V2 SSL HELO support.

    This caused issues with applications that still use the V2 SSL HELO for compatibility issues. Specially older OpenSSL Versions did use V2 SSL HELO unless explicitly specifying TLS 1.0.
    For most applications you can work-around it with updating the OpenSSL version to a current level.

    But specially when using the SMTP STARTTLS extension we don't control what the connecting server uses.

    IBM now allows to re-enable V2 SSL HELO if you really need to.

    The reference SPR is #LMES9QRUZY Problem with incoming SMTP TLS connections after update to Domino 9.0.1 FP2IF1

    But it does not mention the notes.ini parameter you need to enable it: SSL_ENABLE_INSECURE_SSLV2_HELLO=1

    I have tested it with an older version of wget and got the following type of debug output:

    25.02.2015 18:57:18,07 SSLReadRecord> Reading an insecure SSLv2 record by administrator request
    25.02.2015 18:57:18,07 SSL2ReadRecord> Reading an insecure SSLv2 record by administrator request
    25.02.2015 18:57:18,07 SSLProcessProtocolMessage> Record Content: 0
    25.02.2015 18:57:18,07 SSLProcessProtocolMessage> Received an insecure SSLv2 record; processing by administrator request
    25.02.2015 18:57:18,07 SSL2ProcessMessage> Message: 1
    25.02.2015 18:57:18,07 SSL2ProcessClientHello> Processing SSLv2 ClientHello message requesting TLS1.0 (version 0x0301)
    25.02.2015 18:57:18,07 SSL2ProcessClientHello> Client requested SSL_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)

    SLES 12 support added in 9.0.1 FP3 IF1

    Daniel Nashed  24 February 2015 19:19:30
    There is a new section that you should note and regularly check:
    This section will provide important updates to the fixlist.

    In this case the support for SLES 12 with 9.0.1 FP3 IF1! WOW! That was a fast response!  Normally new major OS versions have to wait at least for a dot release! THANKS!!!

    As posted before there was a technical issue with restricted ports because bindsock did not work any more because of kernel changes in SLES 12.

    IBM and Novell worked on this and found a solution.

    Beginning with IBM Domino 9.0.1 FP3 IF1, the IBM Domino Server is supported on SLES12. This interim fix included a fix for the following issue:
    YXYX9RA56Z HTTP server can't be started with "Error - Unable to Bind port 443 or 80" on SUSE12

    With the fix that made it into IF1 IBM is now supporting SLES 12 in the same way they support REHL 7!

    RHEL 7 and SLES 12 are both using "systemd" which replaces the Linux init (rc scripts).
    I got already a couple of questions about systemd support and I am currently working on a version of my start script that supports systemd.

    It looks like it will be an additional service file in combination with some changes to the rc_domino script and also rc_domino_script as well.
    systemd works completely different but I think I am on a good way adding support for it.

    -- Daniel

    Notes/Domino 9.0.1 FP3 - Java Console/Controller Incompatibility

    Daniel Nashed  18 February 2015 10:35:44
    As discussed before, it's not a good idea to completely disable SSLv3 too soon.
    Notes/Domino 9.0.1 FP3 ships with a newer JVM version that completely disables SSLv3.
    The Oracle team disabled SSLV3 by default but the IBM JVM team completely removed SSLv3.

    The Domino server controller and Server Console are based on Java and use the SSL/TLS stack for communication.
    Domino before FP3 uses SSLv3 only -- I don't want to start any theories about why ...

    The newer version with FP3 and higher use TLS 1.0 only.

    That means once you updated your client you cannot communicate via server controller with an older server.
    And also means that you cannot communicate from an older client once you updated your server.

    There is no easy work-around beside running two different clients.
    Just using a different exe does not help because the main change is in the IBM JVM.
    You could keep the old client binaries and clone the data directory and run the jconsole from two different directories to avoid using two different workstations.

    -- Daniel


    And information from the release notes:

    9.0.1 Fix Pack 3 updates the embedded Notes/Domino JVM to 1.6 SR16 FP2 to address security vulnerabilities. This release has all of the content from the recently released POODLE and POODLE on TLS vulnerabilities in one easy to install package that includes the content from Domino 9.0.1 Fix Pack 2 Interim Fix 3 and Notes 9.0.1 Fix Pack 2 Interim Fix 4.

    JVM 1.6 SR16 FP2 disabled SSLv3 and instead communicates only over TLS. If the Domino server is upgraded to 9.0.1 Fix Pack 3 (which contains JVM 1.6 SR16 FP2), the Java Console attempts to connect over SSLv3 to the JVM layer on the Domino server, which will accept only TLS connections. Applying 9.0.1 Fix Pack 3 on both the Domino server and the Java Console client will remedy the situation. For additional information, see technote 1695943 - Domino Console fails to connect to remote server after upgrading Notes or Domino to 9.0.1 Fix Pack 3

      Planned Domino 9 SLES 12 Support

      Daniel Nashed  30 January 2015 00:25:38
      The question for SLES 12 has been raised during IIBM ConnectED. There is an issue with Domino on SLES 12 and SLES 12 is not currently supported (in contrast with RHEL 7).

      There is a SPR # YXYX9RA56Z "Error - Unable to Bind port 443 or 80" on SUSE12.

      I have checked in the Lab and got a similar info than what has been posted before on the web:
      "There is a known issue with SLES 12 where bindsock has issues. Before we can support SLES 12 and any other newer kernel with this issue, we will have to identify the issue and get it fixed - bindsock and it's code has never changed in this area so the issue is a change in behavior in the kernel".

      So for now you should not try to run Domino on SLES 12. It's not just not supported yet but it also it doesn't work yet.

      There is a chance in the kernel that SLES 12 that causes the bindsock operation to fail.

      Some  background: Ports below 1024 are restricted and need root permission. Bindsock has the sticky bit set, runs with root permission and is used to allow Domino to use ports like HTTPS, SMTP and all other restricted ports.
      The way that Domino implements this does not work on the current kernel any more.

      IBM is working on it closely with Novell and the following is the official statement from IBM about it:

      "We intend to support SLES12, just as we have supported every other SLES support. Our current goal is to add support with 9.0.1 FP4. We are working with the vendor to identify root cause of an intermittent bindsock issue we found during testing that is currently preventing us from supporting this version."