Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

Domino Federarted Web Login / SAML with F5 and ADFS 3.0

Daniel Nashed  25 April 2016 17:14:43

In the last couple of weeks I spent a lot of time with customer Web Federated Login workshops and implementations.
Not sure what happened but suddenly everyone is interested in SAML. It looks like more and more customers are looking into that because they have already implemented SSO for other applications like O365.

In one case a customer had an existing F5 configuration. In one other case we had a customer with Windows 2012 R2 and ADFS 3.0.

Both configurations are not officially supported yet but we got it to work! Specially the F5 configuration was tricky. But in general both are just another SAML 2.0 implementation.
We officially asked if those two configurations can be officially supported. It looks like it is more testing and documentation effort than any code change that is needed.
But it is not yet an officially supported configuration. Implementing ADFS 3.0 is quite similar than ADFS 2.0.

Win2012 R2 ADFS 3.0

ADFS 3.0 in fact is a nicer implementation and does not need any IIS components. Also the SSO portal application is now implemented in a way that the UI can be completely customized. You can add you logo, change the CSS or could even build your complete own page.

Also the installation is easier. ADFS 3.0 comes with Win2012 R2 and just needs to be enabled as a separate role. In contrast earlier versions shipped with SAML 1.1 support and you and to separately download and install SAML 2.0.

The configuration is very similar but you cannot use the cookbooks 1:1. Some configuration details are now set via PowerShell commands.
For example if you need to disable the extended protection when working with Chrome.

Domino SAML Implementation

In two customer situations we ran into an odd issue. When initiating the SAML login from the SSO portal like ADFS (with ADFS 3.0 the portal looks prettier and more customers might directly use the portal) redirecting to the Domino HTTP server caused a strange behavior.
The URL invoked should have been the default server URL but there have been som random chars at the end of the URL. Tracing turned out that this was a day one issue with Domino Web Federated Login (WFL) and it was never thought of that the first request is the login request with a redirect from the Identity provider (IdP) in our case ADFS 3.0 or the F5 applicance.

Even Domino uses a IdP Initiated model the first request was always initiated by Domino.

Here is the flow that Domino uses.

- Browser hits the Domino Server for a resource that needs authentication

- User ist redirected to the IdP for authentication --> the URL contains ?loginToRp to tell the ADFS server where to return to after authentication. This is a ADFS specific parameter which is for example not understood by F5.
  At the same time the server sets an undocumented cookie "DOMRELAYSTATE" which contains some data in binary format (base64 encoded) which contains the location to redirect to after login.

- User is authenticated at the IdP. Either via AD name and password or with a Kerberos ticket (Integrated Windows Authentication -- IWA) if configured and the user and workstation are in the current AD.

- Browser redirects back to Domino with the SAML post request

- After verifying the SAML data, the user is authenticated and a LTPA cookie is generated

- At the same time the server reads the "DOMRELAYSTATE" cookie, removes it and redirects the user to his original location

Redirection Issue

In our problem case the user had no "DOMRELAYSTATE" cookie which caused the server to add some garbage to the URL.
We got a hotfix for the issue which will hopefully make it into one of the next IFs. The SPR for this defect is SPR # MKINA8XN74.


So in general if you want to implement SAML right now I would use ADFS 3.0 and Windows 2012 R2 if you have the choice - even it is not yet supported by Domino.
But ADFS 3.0 is the much better product which is better supported by Microsoft. And it is a much cleaner implementation. No dependency on IIS as well!

Server Controller Issue when applying 9.0.1 FP5 IF2

Daniel Nashed  31 March 2016 14:27:11
After applying 9.0.1 FP5 IF2 you cannot connect to the server controller -- again!
That's another issue that cannot be fixed allowing MD5 in the java security files.

What you need is an updated version of the JVM patch. The new patch has a release data of 25.3.2016 an can be downloaded from Fixcentral.

Here is the relevant information from the updated technote referenced in the SPR.

SPR RSSNA6UU79 is fixed in version 9.0.1FP5 Interim Fix 2 (IF2) via a server code fix and an updated JVM patch (SR16FP20). IMPORTANT NOTE: It is required to install both 9.0.1FP5IF2 and the new JVM patch to address the issue. Download links are available in the following technote:

If you already installed the JVM patch before you will run into an error:

Patching tree diff from ".\jvm" to ".\jvm_dst" using diff file ".\patch.diff"...
    You are attempting to patch:
      pwi3260sr16fp2ifix-20141203_01(SR16 FP2+IV66900))
    With a patch that is valid for:
      src1 pwi3260sr16fp15-20151106_01(SR16 FP15))
    Tree diff file patch failed.

My work-around was to re-install FP5 which includes the previous JVM patch.

From there I was able to run the new JVM patch installer and also upgrade to 9.0.1 FP5 IF2.

If that does not work in your case you have to go all the way back to 9.0.1 because that is the last release that contains a full JVM (9.0.1 -> FP5 -> New JVM Patch -> 9.0.1 FP5 IF2).

In my case reapplying FP5 was sufficient.

After the installation the again has MD5 disabled and the console works. So apparently they build in a fix into IF2 and also did changes in the JVM patch.

There is also a fix included in the Notes Client fix 9.0.1 FP5 IF3 and you also have to update your local JVM with the new patch available.
I have so far just tested if the local server console on the Domino server works again.

But since the SPR is also fixed on client side, I assume it works as well.

-- Udpated JVM Patch Information --

Mar 25, 2016

 interim fix: JVMPatch_SR16FP20_RSSNA6UU79_W32_901.5_ClientServer (40.53 MB)

interim fix: JVMPatch_SR16FP20_RSSNA6UU79_W64_901.5_Server (76.78 MB)

Security Issue - IBM Domino AES GCM weak nonce generation vulnerability

Daniel Nashed  29 March 2016 12:02:24
There is a new vulnerability affecting AES GCM ciphers which have been introduced in 9.01. FP3 (enabled by default).
For very large data sets, IBM Domino Web servers using TLS and AES GCM generate a weak nonce which could be potentially used for a man-in-the-middle-attack.

All Domino 9 versions supporting those ciphers are affected and there is  new IF (9.0.1 FP5 IF2) which addresses this issue.

The IBM Domino AES GCM weak nonce generation vulnerability is tracked as SPR #KLYHA6ZP4F.

If you cannot update your server you should change your cipher spec to exclude those ciphers.

The following cipher spec would only allow the CBC ciphers and leave out the 6 GCM ciphers currently supported.

notes.ini SSLCipherSpecÀ28006BC0140039C0270067C013003D0035003C002F000A

The better option would be to install IF2.

Also the new Interims Fix includes a couple of other fixes. Including a fix for the Domino Console introduced by disabling MD5 in the last JVM patch as posted before.
There is no detail how SPR #RSSNA6UU79 addressed the console issue. I had no time to test it in detail yet.

Update 31.3.2016: There is a new issue with the Server Controller if you have applied the JVM fix as well.
The solution is to re-install the latest JVM patch which has apparently a fix as well.
See this new blog post for details -->

SPR Description
KLYHA6ZP4F Security Bulletin: Vulnerability in IBM Domino Web Server TLS AES GCM Nonce Generation (technote 1979604) Image:Security Issue - IBM Domino AES GCM weak nonce generation vulnerability
EDOE9HZLXH Using the colon character in the Domino server title break the Java console. Image:Security Issue - IBM Domino AES GCM weak nonce generation vulnerability
MKINA86V2A The Java console applet needs to be updated for Oracle JVMs Image:Security Issue - IBM Domino AES GCM weak nonce generation vulnerability
MKINA85TJB The java console applet needs the same fix as SODY9FFEYE (technote 1662233) Image:Security Issue - IBM Domino AES GCM weak nonce generation vulnerability
MKINA85TEQ The java console applet needs the same fix as SODY9DDBD5 (technote 1662233) Image:Security Issue - IBM Domino AES GCM weak nonce generation vulnerability
PMGYA4CHDZ Fixes intermittent Domino Server and Notes Client crash when organization is doing a key rollover. Crash occurs on both client and server side when trying to connect. Image:Security Issue - IBM Domino AES GCM weak nonce generation vulnerability
RSSNA6UU79 Domino Console won't connect even when scontroller is running (technote 1977125)

Details and references:

CVEID: CVE-2016-0270 / DESCRIPTION: IBM Domino contains an unspecified vulnerability that could lead to session snooping using man-in-the-middle techniques.

Critical: glibc security and bug fix update

Daniel Nashed  17 February 2016 14:02:45

There is a critical issue with the glibc lib that Linux and other systems are using.

The best short description I found is the following:

"A stack-based buffer overflow was found in the way the libresolv library
performed dual A/AAAA DNS queries. A remote attacker could create a
specially crafted DNS response which could cause libresolv to crash or,
potentially, execute code with the permissions of the user running the
library. Note: this issue is only exposed when libresolv is called from the
nss_dns NSS service module. (CVE-2015-7547)"

Redhat already released patches:

And there is also a patch from SuSE

I have already updated my CentOS 6 Linux machines (via yum update).

Another interesting link is from Heise with some details in German:

Thanks to my friend Harvey Pope pointing me to this bug and sending me the Heise link!


Domino Server Controller does not connect after upgrade to Java6SR16FP20

Daniel Nashed  16 February 2016 18:33:18
The IBM Java Team disabled MD5 in there latest patch to tighten security. But the Server Console currently can only use MD5 right now.
So by this intentionally change by the IBM Java Team the Domino Console cannot connect any more.

For now to have the Server Controller local and remotely working again you have to re-enable MD5.

This is a similar issue than what we had when the IBM Java team disabled SSLV3 some time ago.

There are two lines that you have to change in the ..jvm/lib/security/ file.

You have to remove MD5 from the disabled algorithms for now:

MD5, RSA keySize < 1024
jdk.tls.disabledAlgorithms=SSLv3, RC4,
MD5withRSA, DH keySize < 768

There is currently no other work-around for Windows. On Linux you could use the "monitor" command when using my start script and disable the server controller.

-- Daniel

Domino Start Script New Version 3.1.0

Daniel Nashed  11 February 2016 15:26:10

As already mentioned at IBM ConnectED last week, I am working on a new version of my start script.

Most of the new functionality has been build in because I found it useful for the customer environments I am working in.

On top of the new functionality I added a new script "rc_all" that can start, stop, cleanup, diag ... all partitions a the same time.
The new rc_all script is a separate script that will search for your Domino partition rc-scripts and is mainly interesting when you run Linux machines with multiple partitions.

The new version of the start script has a couple of other additions which could be quite helpful.

You can edit the notes.ini with a simple"ini" command or show current notes.ini setting.
"lastlog" shows you the last log lines from the output log which could be input for a grep command.
By default the last 100 lines but you can specify an optional the number of lines.

The new version can also enable/disable/check the rc-service with the "service" command.

For diagnostics I added the "stacks" command which will quickly just dumps the call-stacks without other NSD functionality (less overhead).

The biggest change is the cleanup of the log files. You can now specify how long logs are stored and the logs are removed when you use the "clearlog" command
or when you enable the log clear functionality at startup in the configuration.

There are a couple of options and I also added some functionality for clearing temporary files at startup.

I have also added more functionality for the system database compact (you can leverage DBMT and there are separate other options) and also a separate log.nsf compact is available.

One of my customers also requested that the log.nsf can be renamed on startup.
I have added a new feature to move the log.nsf with a different name to a backup directory on startup after n days (this avoids creating a new database on every short term startup).

All functionality is completely customizable. There are settings for each of the commands.

I would be very interested to get your feedback about the new commands and options and before I release it, I would like to send it to some admins for beta testing.

If you are interested, feel free to drop me a short mail and I will send you the current state of the start script.
I am using it in customer environments and on my servers already in production.

But I still want to get some feedback before I release it. Documentation is already updated.

Here is the complete extract of what is new in this release.

I hope you like the new functionality


Change History

V3.1.0 20.01.2016

New Features

New command "clearlog"
Clears logs, custom logs and log backups as configured.
Optionally you can specify custom log cleanup days with two additional parameters.
First parameter defines log cut-off days for logs and second parameter defines cut-off days for backup logs.

New command "version" shows version of start script

New command "inivar" displays notes.ini setting specified

New command "ini" to edit the notes.ini of the server

New command "lastlog" shows last log lines. by default 100 lines are displayed.
Optionally you can specify the number of log lines

New command "service" for Linux enables/disables the Domino server "service".
Works for rc-systems and also systemd. Allows to check, enable, disable the service.

New command "stacks" runs NSD stacks only

New option for command "archivelog"

- additional parameter to specify an additional string to add to the archive log file name

New Parameters to enable new features

Number of days until logs are cleared

Number of days until backup logs are cleared

Custom log clear script will be used instead of the standard log clear operations and replaces all other clear operations!

Compact task can now be specified. By default "compact" is used.
Another option would be to use "dbmt" in Domino 9.

Log compact options

Start log compact options

Rename log database on startup after n days

Target directory for rename log database on startup / default "log_backup" in data dir

Moving the log.nsf will be executed before starting the server and after startup compact/fixup operations
You can specify a directory inside or outside the Domino data directory


Option for the new "ini" command for changing the editor.

New option to specidy a different command for removing old tempfiles on startup (default: "rm -f")


New option to specify a different command for removing expired log files (default: "rm -f")


The script only deletes *TMP files in data directory which are at least 1 day old to ensure no important files are deleted.

The "nsd" command by default will generate a NSD with memcheck -- full nsd
"fullnsd" command is removed from documentation but can still be used.
"nsdnomem" is now used to generate a NSD without memcheck.

Problems Solved
When checking resources (shared mem, MQs, Semaphores) the ipcs command in combination with grep is used to check the resources for a certain partition/user.
The ipcs command does not allow to specify a user-name. The was a potential issue whe the login user-names where sub-strngs of each other.
Exampple "notes" and "notes1".  Even this is fixed it is still recommended to ensure that login names are not sub-strings of reach other.

Domino 9.0.1 FP5 IF1 with Security Fixes

Daniel Nashed  30 January 2016 14:47:59
There is a new IF1 for Domino 9.0.1 that includes two fixes we have waited for in the TLS area specially when communicating with STARTTLS and web-services as posted before on my blog.

SPR #KLYHA57S37 - Disable TLS Session Resumption on outbound connections by default        

This fix addresses and issue for outgoing STARTLS sessions on SMTP.

See some more details in my other blog post -->

SPR #MKENA4SQ7R - Domino TLS 1.2 Client Hello does not offer a Signature Algorithm extension causing some handshakes to fail        

The second issue is a problem with a missing security algorithm extension that causes connection issues which happened in many customer environments -- and it looks like this happened depending on the certificate used in some cases.
And also what the remote server supported. The fix implements the missing extensions and improves compatibility.

SPR #KLYHA5YRVP - Recommended security fix for IBM Domino (technote 1974958)        

The Domino SLOTH vulnerability is about collision attack with the MD5 hash function that is used in the TLS handshake.
The fix addresses this issue.

Here are the main details from the TN describing the SPR.

CVEID: CVE-2015-7575
DESCRIPTION: The TLS protocol could allow weaker than expected security caused by a collision attack when using the MD5 hash function for signing a ServerKeyExchange message during a TLS handshake. An attacker could exploit this vulnerability using man-in-the-middle techniques to impersonate a TLS server and obtain credentials.

See more details here ->

SPR #DKENA32JMP - Add support for Extended Master Secret (RFC 7627) to TLS 1.2        

This is a quite new RFC which has been implemented by Microsoft and Google for their browsers recently. Both sides need to support this extension!
Domino does now support this extension which eliminates a risk of a man-in- the-middle attack in some situations described in the RFC below.

The Transport Layer Security (TLS) master secret is not
cryptographically bound to important session parameters such as the
server certificate.  Consequently, it is possible for an active
attacker to set up two sessions, one with a client and another with a
server, such that the master secrets on the two sessions are the
same.  Thereafter, any mechanism that relies on the master secret for
authentication, including session resumption, becomes vulnerable to a
man-in-the-middle attack, where the attacker can simply forward
messages back and forth between the client and server.  This
specification defines a TLS extension that contextually binds the
master secret to a log of the full handshake that computes it, thus
preventing such attacks.

Linuxfest VII Gets a Slot at IBM Connect 2016

Daniel Nashed  30 January 2016 14:17:19
If you are attending IBM ConnectED in Orlando and you are interested in Linux you should attend the Linuxfest Session.

Thanks to Bill Malchisky we made it again into the agenda!  I am looking forward to this session and will bring the brand new Start Script Version 3.1.0 with many enhancements.

Here is a copy of Bills' original post. Looking forward to this session.

-- Daniel

Linuxfest VII Gets a Slot at IBM Connect 2016
Bill Malchisky  January 28 2016 02:00:00 AM
Linuxfest VII - The Penguin Awakens

After many months of planning and working with the events team, we are pleased to announce that Linuxfest is back for our seventh year. This is the only session at IBM Connect dedicated exclusively to Linux and IBM software. As we moved back to the last day lunch break, be certain to bring your box lunch and join us for an informative session Linux and IBM. New this year, we are in Event Connect and Session Preview Tools.

Date: Wednesday, 3 February
Time: 11:45 - 12:45 PM
Place: Orange G
Session ID: TI-1118
Audience: Admins, Developers, Architects
Speakers: Bill Malchisky, Wes Morgan, and Daniel Nashed

Ask questions and get informative answers from three passionate leading IBM on Linux SMEs.

Whether you've already deployed IBM technology on Linux or are "just interested," join us for the seventh installment of what has become an IBM Connect tradition. In this open discussion of the latest on Linux from IBM, you'll hear Business Partners, IBMers and IBM Champions talk about the most recent developments around our favorite operating system, share tips and tricks, and open the floor to your questions, successes and commentary as well. This is not a roadmap/strategy session; instead, it's a chance for you to learn what's out there for Linux, pick up technical know-how to ease your deployments, and connect with other IBM customers using Linux.

Image:Linuxfest VII Gets a Slot at IBM Connect 2016

Traveler shipped

Daniel Nashed  16 January 2016 15:23:21
Traveler is the first update shipped this year.

It comes with a number of fixes. See details here -->

And it solves an important issue for Traveler HA Servers. There is a technote describing the issue in detail and you should have a look into the new command introduced in this version as soon you have updated your servers.

The following TN #1974741 "Two scenarios where multiple accounts for users could be created on an IBM Traveler server HA pool" explains the new command and problem situation that might occur.

Have a look into the TN if you are running Traveler HA -->

There are two new "features" introduced with Traveler

- Calendar Ghosting (which was added in is enabled by default for IBM Verse clients starting in release  

- And the new "DbAccountsCheck" which can be used to diag and fix the problem described in the TN mentioned above.

TLS 1.2 Connection Issues with

Daniel Nashed  7 January 2016 11:57:08
Two of my customers had issues connecting to the Microsoft hosted environment over TLS 1.2 once we got the session resumption working (see previous blog posts).

My environment had the same configuration and could connect just fine.
It looks like the servers are behaving different with different certificates.
That's the only difference we saw in configuration.

After a couple of tests and working with IBM support we got a hotfix that we successfully tested yesterday.
I know of 3 customers who solved their connection issues that way.

The error you see in the logs is the following:

TLS/SSL connection -> failed with client certificates NOT supported by server signature algorithms
SMTPClient: SSL handshake error: 1C7Ah
Router: No messages transferred to ACME.COM (host via SMTP: SSL IO error. Remote session no longer responding.

SPR # MKENA4SQ7R Domino TLS 1.2 Client Hello does not offer a Signature Algorithm extension causing some handshakes to fail

This is one of the SPRs planned for the next IF.
There are other open issues that should be also fixed as well like the outgoing session resumption issues.

Short description what happens.

TLS 1.2 defines an extension to the Client Hello (signature algorithms) and this is officially required for TLS1.2 in contrast to earlier TLS versions.
Some servers implement the RFC quite strict and that could cause connection issues over TLS 1.2

The fix ensures that the signature algorithms are send which includes all the currently supported algorithms:

06 01 - SHA512/RSA
05 01 - SHA384/RSA
04 01 - SHA256/RSA
03 01 - SHA224/RSA
02 01 - SHA1/RSA
01 01 - MD5/RSA"

  • [IBM Lotus Domino]
  • [Domino on Linux]
  • [Nash!Com]
  • [Daniel Nashed]