Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

Traveler released with some fixes

Daniel Nashed  19 August 2016 00:12:42
There is a new traveler release that just shipped.
Some of the issues might affect you.

APAR # Abstract
LO82881 Domino server may crash if $NTTrack field is corrupted.
LO89471 Traveler invitee status may be incorrect if using mixed case internet addresses.
LO89606 Number of recipients limited to 100 when sending mail from a mobile device.
LO89745 Traveler server enters constrained state when load balancing a large number of users.
LO89772 Meeting chair may receive multiple notices from attendee who processes notice on an Apple Native Calendar application.
LO89840 IBM Verse mobile application fails to download entire mail for very large mail documents.
LO89952 Deleted device still present in the Web Administration UI after the 30 day reap interval.
LO89954 E-mail not in sent folder on mobile device when user sends and files an e-mail in Notes client.

Release documentation:


    Extended Master Secret Extension issue affects all Internet Protocols including STARTTLS

    Daniel Nashed  27 July 2016 10:23:28
    There is a an issue described in a technote which describes an issue with Win 2008 R2 and LDAP.
    This issue also occurs for other internet protocols!!

    It is specially important for servers using STARTTLS because you don't control which version and settings the receiving/sending host is using.

    So the issue I blogged about today does also affect other protocols. That's why I decided to have two blog posts to ensure it is better found on the web.

    Hiere is the info from the other blog post which also is relevant for your SMTP Servers.

    -- Daniel

    Domino  9.0.1 FP5 IF1 adds support for the Extended Master Secret Extension with TLS 1.2.

    Windows 2008 R2 does only supports TLS 1.0 but still sends the Extended Master Secret Extension in the server helo.

    Domino fails to connect because once this is offered Domino wants to use it.

    There is a work-around to disable this new functionality globally on the server via notes.ini


    This is just a work-around and the real fix would be that Microsoft provides  a fix for Win 2008 R2 to not send the extension with the helo.
    Later versions do support TLS 1.2 and do not have the issue.

    See the following technote for details ->

      Secure LDAP to Active Directory fails with Domino 9.0.1 FP5 IF1 and higher

      Daniel Nashed  27 July 2016 08:21:25

      Domino  9.0.1 FP5 IF1 adds support for the Extended Master Secret Extension with TLS 1.2.

      Windows 2008 R2 does only supports TLS 1.0 but still sends the Extended Master Secret Extension in the server helo.
      Domino fails to connect because once this is offered Domino wants to use it.

      There is a work-around to disable this new functionality globally on the server via notes.ini


      This is just a work-around and the real fix would be that Microsoft provides  a fix for Win 2008 R2 to not send the extension with the helo.
      Later versions do support TLS 1.2 and do not have the issue.

      See the following technote for details ->

      -- Daniel

      IBM Traveler released including a security fix

      Daniel Nashed  14 July 2016 09:45:20
      IBM Traveler shipped with some important changes.

      The first change is a security fix which is described below.

      But there is another security fix in the installer on Windows as well and some other fixes that could be affecting you.

      Upgraded my server already.

      -- Daniel

      Security Bulletin: XML External Entities Injection Vulnerability in IBM Traveler (CVE-2016-3039)
      IBM Traveler is vulnerable to a denial of service caused by an XML External Entity Injection (XXE) error when processing XML data.
      APAR # Abstract
      LO87689 Invitee status not updated on Mobile device when external invitee responds.
      LO88807 Add the immediately remove invitee from invite on mobile device may not remove the invitee.
      LO88916 Invitee status not updated on Outlook client when external invitee responds.
      LO88950 Event still appears ghosted on mobile device after process an info update from ghosted entry.
      LO89057 Upgrade install technology to prevent MS Windows DLL Loading vulnerability.
      LO89097 Traveler device may display EnterSendTo field if SendTo empty for non-draft message.
      LO89287 Warning message for NumberFormatException for empty string should be Info log message and not a warning.
      LO89357 Update to prevent XML External Entities Injection vulnerability.
      LO89358 Same full name contact could sync wrong contact photo.
      LO89421 Ghosted entry for non-repeating event Cancel notice may show additional options on mobile device.
      LO89499 APNS notifications for IBM Verse for iOS may be in English instead of device preferred language.
      LO89501 Attachments and in-line images missing content header may not sync to mobile device.
      LO89540 Traveler Utility application should warn if attempting to change the DB2 user name as this may change the schema name as well.
      LO89543 Prevent device from renaming folder to null string.
      LO89544 Accept reschedule of non-repeating event from ghosted entry on Apple iOS Calendar application may not take effect on server.

        IBM Mail Support for Microsoft Outlook officially released

        Daniel Nashed  29 June 2016 15:49:35
        Finally IBM has released IMSMO 2.0. It has been around already under controlled distribution and is finally available for all customers and partners.

        It enables you to connect a Microsoft Outlook 2013 client to a Domino V9.0.1 Server.

        The software is an add-on to your Domino server similar to what a Traveler does (in fact they share some code base but they are not the same!).
        Also the gateway need to resist on the mail-server of the user. We asked to have a way to use a gateway server approach but IBM implemented it this way.

        If you are running in a production environment and you are running with more than a handful of users it is strongly recommended to use DB2 for the state database instead of the derby database.
        So for every server that has IMSMO enabled you need a connection to a DB2 server for the state database.
        Another way would be to have a local replica of the mailfile on the IMSMO server for all users running an Outlook Client.

        This solution is not intended for a whole, large organization. It's more one of the flexible options to support Outlook Clients in your environment for users who want or need the client for some special reason.

        For the Outlook Client you have to deploy an add-on which handles the connection.
        The first version with the project name "Hawthorn" was using the Active Sync protocol for sync and rest services with a plug-in for additional services like OOO and busytime lookup. T
        hat's why only Outlook 2013 and higher was supported because earlier releases did not support ActiveSync.

        The new version uses SyncML which gives IBM more control over the connection and functionality. The plugin does now handle the whole connection from client to server over SyncML.
        Older clients can still connect over ActiveSync but the new model makes more sense to only use the new plug-in.
        IBM also plans to support other Outlook Clients leveraging the plug-in approach.

        If you are entitled to download Domino V9.0.1, you can download IBM mail support for Microsoft Outlook at no charge from Passport Advantage®.


        IBM Mail Support for Microsoft Outlook 2.0 Release Notes Multiplatform English (CNCZ9EN )


        IBM mail Add-on 2.0.0 (for IMSMO) Windows 64 bit Multilingual (CNBK6ML )
        IBM mail Add-on 2.0.0 (for IMSMO) AIX 64 bit Multilingual (CNBK7ML )
        IBM mail Add-on 2.0.0 (for IMSMO) Linux 64 Multilingual (CNBK8ML )


        IBM mail Add-in 2.0.0 (for IMSMO) Windows 32/64 bit Multilingual (CND43ML )

        But you should wait to download and install the client component. There is an issue where I have an open PMR since yesterday afternoon.
        It looks like there is an issue with the client package. Development reported back that they are working on it with high priority.
        I will blog about it once I have more details.

        Here is a quick link to the admin guide:

          Domino Catalog vs Domain Catalog

          Daniel Nashed  11 June 2016 16:50:33
          I while ago I ran into this and I did analyse how it works in detail. But I never posted this information.

          Today we ran into this again and I looked for my old documentation.

          -- Daniel

          Here is how it is intended to work and how most admins are currently using it.

          I was very surprised when I figured out how the catalog and domain catalog really work.

          My impression was always that the catalog.nsf is a replica that is replicated everywhere.

          But there are two different types of catalogs:

          Normal Catalog

          - Content resist only on one server and is not replicated. Local server will update the data.

          - Other servers have reader access and nobody beside the own server can update it.

          Domain Catalog

          - Contains database information for many or all servers

          - Other servers have editor access and database information is pulled from other replicats to the central database

          - If a server has no replica of the catalog.nsf the catalog on the Domain catalog server will spider the server and collect databases, else the catalog task will replicate the database!

          So in any case the catalog.nsf should be a replica on each server -- actually it is derived from the replica of the server's names.nsf (it's one of the special replicas).

          Each replica maintains it separate ACL and each server should only have manager access on his own replica.
          The catalog.nsf is designed to have different ACLs so that only the Domain catalog servers will contain all data.

          When the catalog task starts it will check if the config document document has the "domain catalog" enabled.

          In that case LocalDomainServers, LocalDomainCatalogServers will be editor access in catalog.nsf on the local server Else both groups are set to reader access instead!

          That would mean if you add an admin server or have a catalog.nsf created on a server in a way that other servers have manager access to the database the ACL it might replicate all data and the concept of the catalog distribution will not work in most cases!

          Normally an Administrator would expect that LocalDomainServers are manager on catalog.nsf and the HUB server is admin server of the database with manager access and the ACL should be the same on all servers.

          But the catalog.nsf is intended to have a different ACL on each server instead. That was complete news for me. I was often wondering what is happening with the LocalDomainServers and LocalDomainCatalogServers all the time in some cases.

          Because LocalDomainServers, LocalDomainCatalogServers ACL entries are always checked and updated when needed by the catalog servertask, you should not change those ACL entries!
          And in case you want your own group to control replication of catalog.nsf in your own way, you should not use those two ACL entries but create your own group.

          In case a server has manager rights in the remote database it would push his ACL to the other server or if the other server has manager access, receive the ACL changes -- depending also on the time-stamps in the ACL.

          So what could happen is that you end up with a permanently changing ACL and potentially all catalog entries on all servers in your Domino domain -- depending which server is domain catalog server.

          This is not the intended way the catalog should work! It is intended to have separate ACLs settings on each replica and every server manages it's own ACL settings.

          Also the mentioned two ACL entries should not be removed from the ACL because that causes the catalog task to fail updating the ACL and does stop the catalog task before updating the catalog content.

          What might happen in many cases is that you create a "normal" replica of your catalog.nsf. In that case you could already run into trouble because the ACL might contain an admin server entry that is copied.

          That means the server you replicated the catalog.nsf from will be able to update documents on that new server -- which is not the intended way it should work.

          The best way would be just to start the catalog servertask and have the server create an empty replica based on the derived replica id from the names.nsf.

          Catalog.nsf will than only be replicated to a Domain catalog replica instead to all servers.

          Again I was very surprised when I figured out the logic behind it in detail and how it is intended to work.

          If you ran into this situation the best way would be to start from scratch.

          You remove the catalog.nsf databases on all servers, verify the right settings for the "Domain catalog" in server document and plan your catalog.nsf distribution that way.

          Once you are done you can just run the catalog task on each catalog server to populate it.

          When that is done, you can start the catalog task again on all the domain catalog servers.

          Take care: If you start it before a server has a new replica the Domain catalog server would spider the server manually and you could end up with duplicate documents in the catalog once the catalog task has been started on the local server later on.

          As a trick you could set the "Limit domain cataloging to the following servers:" field next to the Domain Catalog field in the Servertasks / Domain Catalog tab. This field limits which servers should be queried.

          Also take care that an administrator with manager access should never replicate the catalog.nsf because that would also potentially break the ACL and the replication scheme.

            DE-Mail Mail-Template with Command Line DNS Lookup

            Daniel Nashed  8 June 2016 07:43:38
            We ran into a limitation with the DE-Mail Template that T-System implemented in their Notes Mail Template.

            It turned out that they are invoking a cmd.exe because this is the only way to return data directly from nslookup to the application with a redirect on Windows.
            The function is used to check if the recipient's domain is a DE-Mail domain and queries SRV records defined in RFC RFC 2782 (check for details).

            SRV Records can not be queried with  simple DNS lookup but need a more complex syntax as shown in the example below.

            Here is the actual code in the DE-Mail Template:

            'Call executeAndWait("CMD.EXE /C nslookup -type=SRV _ldaps._tcp."+domain+ipPart+" 2> "+tmpFilePath)

            Example nslookup:

            nslookup -type=SRV

            Non-authoritative answer:
        SRV service location:
                      priority       = 0
                      weight         = 5
                      port           = 636
                      svr hostname   =

            We have asked T-Systems to enhance the code a quite while ago because usually cmd.exe is not allowed in Citrix environments and it will also not work on Linux and Mac.
            It does not look like we are going to get a solution soon., so we implemented your own work-around.

            Luckily there is a Java class that implements functionality to query SRV records. Here is the first version of the code I wrote.
            I have added a Java agent and a function to invoke the Java agent to replace the current implementation of the lookup.
            The agent is invoked with the document in context from Lotus Script without the need to save the document first.

            Feel free to use this code, modify/enhance it and send feedback.

            -- Daniel

            -- Script Lib "DeMailFunctions" --

            - New Function CheckRecipient

            Function CheckRecipient (Doc As Notesdocument, Domain As String) As Integer

                   Dim theAgent As NotesAgent

                   Dim AgentString As String

                   Dim NoteID As String

                   Dim ret As Integer

                   Dim demail_recipient As Integer

                   Dim db As NotesDatabase

                   '1 = no DE-Mail recipient

                   '0 = valid DE-Mail domain

                   demail_recipient = 1

                   On Error Goto end_function

                   Set db = doc.ParentDatabase

                   doc.nslookup_domain = domain

                   Set theAgent = db.GetAgent("nslookup_srv")

                   If Not(theAgent Is Nothing) Then

                           ret = theAgent.RunWithDocumentContext(doc, "")

                           If (ret = 0) Then

                                   If (doc.nslookup_result(0) <> "") Then

                                           ' Print "nslookup.srv result -> " + doc.nslookup_result(0)

                                           demail_recipient = 0

                                   End If

                           End If


                   End If


                   Call doc.RemoveItem ("nslookup_domain")

                    Call doc.RemoveItem ("nslookup_result")
                    CheckRecipient = demail_recipient        
                   Exit Function

            End Function

            -- Change Function "checkRecipients" --

            Comment out the following two red lines and add the green line

            'Call executeAndWait("CMD.EXE /C nslookup -type=SRV _ldaps._tcp."+domain+ipPart+" 2> "+tmpFilePath)

            'If Not checkLookUpResult(tmpFilePath) Then                


            If (CheckRecipient (doc, domain)) Then        

            -- New Agent "(nslookup_srv)" --

            Add the following Java Agent code

            ' Written by Daniel Nashed (
            import lotus.domino.*;

            class JavaAgent extends AgentBase {

            public void NotesMain() {

            try {
                     Session session = getSession();

                     AgentContext agentContext = session.getAgentContext();

                     Document doc = agentContext.getDocumentContext();
            if (doc != null)

            "nslookup_result", "");
                             String nslookup_domain = doc.getItemValueString(
                             String nslookup_dnsserver = doc.getItemValueString(
            out.println("nslookup_domain -->" + nslookup_domain + "< ServerIP --> " + nslookup_dnsserver + "<");
                             Hashtable env =
            new Hashtable();
            "java.naming.factory.initial", "com.sun.jndi.dns.DnsContextFactory");
            if (nslookup_dnsserver == "")
            "java.naming.provider.url", "dns:");
            "java.naming.provider.url", "dns://" + nslookup_dnsserver);
                             DirContext ctx =
            new InitialDirContext(env);
            try {
                                     Attributes attrs = ctx.getAttributes(
            "_ldaps._tcp." + nslookup_domain, new String[] { "SRV" });
            if(attrs != null && attrs.size() > 0)

            // System.out.println ("---found something---");
                                             NamingEnumeration e = attrs.getAll();  
                                             String lookup_result;
            "nslookup_result", lookup_result);
            // System.out.println ("nothing returned");

            catch (NamingException e) {
            out.println ("--- Namelookup Result Catch ---");




            catch(Exception e) {
            out.println ("--- Namelookup - General Catch ---");





            43. DNUG 1. + 2. June in Hamburg

            Daniel Nashed  26 May 2016 09:58:42
            Image:43. DNUG 1. + 2. June in Hamburg

            Hi and good morning!

            This will be my first blog entry in Germany and it's about German about our German "Notes" User Group "DNUG"...

            Bei der DNUG hat sich einiges getan in den letzten Monaten! Seitdem die DNUG einen neuen Vorstand hat, werden einige Dinge anders angegangen und auch die Art und Weise, wie die DNUG Konferenz geplant wird, hat sich geändert.

            Die Geschäfts-Stelle ist jetzt virtuell und auch die Server der DNUG sind entsprechend virtualisiert. Aber auch an anderen Stellen sieht man, daß sich einiges verändert.

            Für die Organisation der Konferenz-Tracks sind die verschiedenen Fachgruppen verantwortlich.

            Zur Anmeldung für die Konferenz und für den Schedule werden Tools verwendet, die es günstig aus der Cloud auf dem Markt gibt. Die eigene EULUC Connections Platform der DNUG wandert in die IBM Cloud.

            Als Mitglied in der Fachgruppe Verse/Notes/Domino zusammen mit Anett Hammerschmidt und Manfred Lenz als Ansprechpartner und Mitglied in unserer Fachgruppe, haben wir uns um unseren "Track" gekümmert.

            Und ich denke sowohl in unserem als auch in allen anderen Tracks findet Ihr spannende Vorträge.

            Besonders ans Herz legen möchte ich Euch die Session zum Thema Verse on Premise
            ( und die Session zum Thema Microsoft und IBM (Link
            Aber auch die Lizenz-Session
            (, die wir in ähnlicher Form schon auf dem Domino Day hatten, wird bestimmt wieder interessant und wichtig mit vielen Diskussionen (im Bereich Lizenzierung, gibt es viele Details, die oft nicht bei Kunden und auch Business Partnern bekannt sind).

            Relativ neu ist auch bei der DNUG, daß IBM alle Fachgruppen mit einem Ansprechpartner unterstützt. Und für unseren Track und auch für andere Tracks stehen die Sprecher die übrige Zeit auf der Konferenz für Fragen zur Verfügung.

            Nutzt diese Gelegenheit um die Kollegen von IBM mit Fragen zu löchern und Feedback zu geben über das Produkt, Support und andere Dinge, die Ihr auf dem Herzen habt.

            Und auch preislich hat sich bei der DNUG in Bezug auf die Konferenz einiges getan. Die DNUG Konferenz ist für Mitglieder ein ganzes Stück günstiger geworden und auch die Referenten erhalten als Dank für ihren inhaltlichen Beitrag wieder eine kostenlose Konferenzteilnahme.

            Ich kann leider selbst dieses Jahr nicht mit auf der Konferenz dabei sein. Mein Urlaub war nicht mehr anders zu planen (das ist eine längere Geschichte).
            Aber ich wünsche allen eine tolle und erfolgreiche DNUG Konferenz in neuer Gestalt!!
            Ich wäre sehr gerne in Hamburg dabei. Dafür werde ich aber auf jeden Fall beim nächsten Domino Day der Fachgruppe Verse/Notes/Domino im Herbst wieder mit dabei sein.

            Für Themen und Vortragsideen und andere Dinge, um die wir uns als Fachgruppe beschäftigen sollen/wollen sind wir immer an Feedback und Mitarbeit interessiert.

            Und wer noch nicht auf der DNUG Konferenz angemeldet ist hier nochmal der Link zur Konferenz-Seite

            Ciao und bis bald!


            Domino 9.0.1 FP6

            Daniel Nashed  22 May 2016 21:55:43
            Domino 9.0.1 FP6 has been released a while ago. I have installed it and I got positive feedback from customers already.

            FP6 contains all the fixes from previous IFs and also the updated JVM Java60SR16FP20 which addresses a couple of security fixes.
            Also the server controller interoperability issue is fixed. But for a client based connection you also need to update your admin client!

            All the TLS fixes are also included and there is an additional fix for an issue in a TLS handshake.

            SPR# MKINA3SPYN - Fixes an intermittent Domino Server crash in crypto code when VerifyMAC() is passed a bogus length.

            There are many other potential crash issues and security issues that have been also fixed.
            So it makes a lot of sense to install it on Server side.

            In additional there are fixes that need notes.ini parameters.

            The first fix is to bring back the pre Domino 9 behaviour for removing documents from the trash folder after the threshold expiered.
            In Domino 8.5.x was checked on database open. With Domino 9 it is only checked with updall.

            There is a new notes.ini parameter CHECK_EXPIRED_SOFT_DELETES_ON_DBOPEN=1 which brings back the previous behaviour.

            SPR# HYYH9DF5GR - Fixes situation where emails in trash are not  removed even if "Permanently delete documents after X hours" is set. This fix introduced a new Notes.ini CHECK_EXPIRED_SOFT_DELETES_ON_DBOPEN=1. This is off be default.

            There is also an ID-Vault fix which needs a notes.ini parameter. The notes.ini parameter is missing in the SPR description.

            The fix introduced needs the this notes.ini parameter IDV_RefreshCerts=1
            By default the parameter is disabled for performance reasons. You should only enable it when you are in a key-rollover project.

            SPR# KLYH9ZDQNC -- IDVault Key Rollover State Can Be Incorrect Due to Timing Issue


            Domino Federarted Web Login / SAML with F5 and ADFS 3.0

            Daniel Nashed  25 April 2016 17:14:43

            In the last couple of weeks I spent a lot of time with customer Web Federated Login workshops and implementations.
            Not sure what happened but suddenly everyone is interested in SAML. It looks like more and more customers are looking into that because they have already implemented SSO for other applications like O365.

            In one case a customer had an existing F5 configuration. In one other case we had a customer with Windows 2012 R2 and ADFS 3.0.

            Both configurations are not officially supported yet but we got it to work! Specially the F5 configuration was tricky. But in general both are just another SAML 2.0 implementation.
            We officially asked if those two configurations can be officially supported. It looks like it is more testing and documentation effort than any code change that is needed.
            But it is not yet an officially supported configuration. Implementing ADFS 3.0 is quite similar than ADFS 2.0.

            Win2012 R2 ADFS 3.0

            ADFS 3.0 in fact is a nicer implementation and does not need any IIS components. Also the SSO portal application is now implemented in a way that the UI can be completely customized. You can add you logo, change the CSS or could even build your complete own page.

            Also the installation is easier. ADFS 3.0 comes with Win2012 R2 and just needs to be enabled as a separate role. In contrast earlier versions shipped with SAML 1.1 support and you and to separately download and install SAML 2.0.

            The configuration is very similar but you cannot use the cookbooks 1:1. Some configuration details are now set via PowerShell commands.
            For example if you need to disable the extended protection when working with Chrome.

            Domino SAML Implementation

            In two customer situations we ran into an odd issue. When initiating the SAML login from the SSO portal like ADFS (with ADFS 3.0 the portal looks prettier and more customers might directly use the portal) redirecting to the Domino HTTP server caused a strange behavior.
            The URL invoked should have been the default server URL but there have been som random chars at the end of the URL. Tracing turned out that this was a day one issue with Domino Web Federated Login (WFL) and it was never thought of that the first request is the login request with a redirect from the Identity provider (IdP) in our case ADFS 3.0 or the F5 applicance.

            Even Domino uses a IdP Initiated model the first request was always initiated by Domino.

            Here is the flow that Domino uses.

            - Browser hits the Domino Server for a resource that needs authentication

            - User ist redirected to the IdP for authentication --> the URL contains ?loginToRp to tell the ADFS server where to return to after authentication. This is a ADFS specific parameter which is for example not understood by F5.
              At the same time the server sets an undocumented cookie "DOMRELAYSTATE" which contains some data in binary format (base64 encoded) which contains the location to redirect to after login.

            - User is authenticated at the IdP. Either via AD name and password or with a Kerberos ticket (Integrated Windows Authentication -- IWA) if configured and the user and workstation are in the current AD.

            - Browser redirects back to Domino with the SAML post request

            - After verifying the SAML data, the user is authenticated and a LTPA cookie is generated

            - At the same time the server reads the "DOMRELAYSTATE" cookie, removes it and redirects the user to his original location

            Redirection Issue

            In our problem case the user had no "DOMRELAYSTATE" cookie which caused the server to add some garbage to the URL.
            We got a hotfix for the issue which will hopefully make it into one of the next IFs. The SPR for this defect is SPR # MKINA8XN74.


            So in general if you want to implement SAML right now I would use ADFS 3.0 and Windows 2012 R2 if you have the choice - even it is not yet supported by Domino.
            But ADFS 3.0 is the much better product which is better supported by Microsoft. And it is a much cleaner implementation. No dependency on IIS as well!

            • [IBM Lotus Domino]
            • [Domino on Linux]
            • [Nash!Com]
            • [Daniel Nashed]