Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

    Notes and Domino Future

    Daniel Nashed  23 September 2016 11:49:52

    There have been a lot of rumors and IBM is not very good in communicating road-maps since a couple of years.
    I hope we will see a clear statement about future functionality soon.

    There is already a public statement in the IBM blog that gives some answers and I have been at a couple of events where IBM explained part of the strategy.
    https://www.ibm.com/blogs/social-business/2016/09/12/ibm-notes-domino-v9-extends-support/

    Now that the current strategy is more clear and IBM decided to continue with Notes and Domino with incremental releases this support statement was needed.
    This support statement has been discussed on some customers and partners draw some maybe not completely correct conclusions from the statement.

    Here are some details and thoughts from my side.

    I hope this draws some light into the current discussion. And I am looking forward to your comments.

    -- Daniel

    In the last fixpacks and even in interims fixes IBM started to add functionality beside just shipping fixes. The security area is a good example.
    We now have full TLS 1.2 support and since 9.0.1 FP7 we have state of the art port encryption -- see previous blog posting for details.
    Also IBM shipped a Mac 64bit native client. All this shows that IBM is still investing in Notes and Domino on prem.

    There are still many features and requirements that are pending -- like Java 8 support which is really something that should IBM have shipped earlier.
    Also NIFNSF the ability to store view/folder index outside the NSF file for I/O optimization, reduced backup and larger database size is an important pending feature.

    I saw some draft roadmap slides that IBM is hopefully releasing in public soon with some additional details.

    Those features will be delivered via "Feature Packs" instead of "Fixpacks". Some details are not yet sorted out but we can expect 3 feature every year and templates changes are shipped separately in some way (there is no detailed information yet).
    This is a smoother way to develop and deploy new functionality. But this also means that customers have to deploy those fixes incrementally.
    Usually this is easier to bring into production instead of a major version -- at least on server side.

    Having said that IBM clearly states that the Domino server is an important asset in the IBM portfolio and you can see that from the fact that IBM Verse on prem leverages Domino 9.0.1 as the foundation.
    The Notes Client gets less and less attention. But it will be still supported and maintained in the Notes 9.0.1 code stream and we will also see some improvements here -- 64bit Mac client was a good example (no I don't think we need a W64 client nor that I think we would get it).

    IBM sets the focus on Web clients like IBM Verse and the current strategy is "mobile first" for all their current and future offerings.

    IMHO putting more money and energy into modern web and mobile applications is the right direction. A modern and more intuitive Web UI like IBM Verse makes a lot of sense not only for the younger work-force (Design Thinking plays an important role here).

    In contrast to other companies IBM still also has an "on prem strategy" in addition to their Cloud first/Cognitive strategy.


    The current extended support statement states that 8.5.3 will be still supported for another 2 years from now -- which gives customers another two years to migrate to 9.0.1.

    9.0.1 is at least supported until September 2021. "At least" is an important detail here because that does not mean that this will be the maximum time 9.0.1 will be supported with Feature Packs.

    (And we don't know yet if IBM does not decide to change the deployment model again and at some point ship a different Notes/Domino version like 9.0.2 or 10.0)

    What is clear from other IBM statements now that IBM uses Feature Packs to deliver new functionality you have to be on maintenance to be entitled to download and use those "FPs" in future.
    This is also not new. IBM did just not enforce for Notes/Domino and other collaboration software that you have to be on maintenance for fixpack entitlements.
    But this is a change for Notes/Domino customers from the current way fixpacks are handled!

      Traveler 9.0.1.14 -- Calendar Issue Tentative Accept

      Daniel Nashed  23 September 2016 11:03:33


      One of my customer ran into this issue with 9.0.1.14.
      When you tentative accept a meeting on your mobile device and accept it completely afterwards that change is not updated on the senders invitation.
      The tentative accept remains as the status.

      We have two PMRs and there is a APAR -> LO9030.

      If you did not update your Traveler Servers yet I would stay on 9.0.1.13 which is the first iOS 10 supported release and wait for the fix.

      If you already updated to 9.0.1.14 you also have to wait for the fix. Just to be sure, downgrading a Traveler server is not a good idea because this will lead to a resync of all devices!

      -- Daniel




      Setting up the first server and Certifier with 4096 bit keys instead of 1024 bit

      Daniel Nashed  21 September 2016 09:33:44
      Today at AdminCamp I got the question how to register a first server and the organisational certifier with larger key size.
      By detault the setup process is still using 1024bit -- I guess for compatibility.

      There is a notes.ini setting that increases the key length for the organisation, server and first admin.id.

      SETUP_FIRST_SERVER_PUBLIC_KEY_WIDTH=4096

      You have to set this parameters in your first servers notes.ini before you start the server for the first tile to do the server setup.

      -- Daniel




      iNotes borken with German locale with 9.0.1 FP7

      Daniel Nashed  15 September 2016 19:46:53
      Today I got a customer question about iNotes not working for him after updating to FP7.
      I can reproduce the issue on my Linux machine. Apparently something went wrong with the German locale. Reports show that the English locale should work fine.


      Also one of my customers forwarded me a IBM L1 support response for a different question which stated that if iNotes does not work with FP7 they should restore the FP6 Forms9.nsf databases.

      They got that info without any further information.


      I have shutdown my server and replaced the files with the backup copies on my server and started the server again.


      cd /local/notesdata/iNotes
      cp /opt/ibm/domino/notes/latest/linux/data1_bck/901FP6/localnotesdataiNotes/Forms9.nsf .

      cp /opt/ibm/domino/notes/latest/linux/data1_bck/901FP6/localnotesdataiNotes/Forms9s.nsf .



      The issue already occurs when you first launch iNotes. See the formula error screen below.


      I keep you posted what I hear.


      -- Daniel



      Image:iNotes borken with German locale with 9.0.1 FP7



      Notes & Domino 9.0.1 FP7 shipped

      Daniel Nashed  14 September 2016 12:07:02
      Notes and Domino 9.0.1 FP7 has shipped with quite a number of important fixes.

      - The JVM was updated to the current quarterly release replacing the JVM patches that came out since FP6.

      - There are stability fixes which include many areas including Compact, Archiving API, iNotes, DXL and also some important security fixes.


      In one client SPR even ADFS 3.0 is mentioned so maybe we can hope that we get full ADFS 3.0 at some point in one of the next FPs - which is high on my priority list since most new ADFS customer installations require ADFS 3.0.


      Oh I almost missed an important platform update. Citrix XenApp 7.7 is now supported since FP7 which was missing for many customers!



      Beside all those fixes which are a good reason to deploy FP7 there are two SPRs that I want to highlight.

      -- Important Linux 64bit Fix --


      The first SPR deals with a really bad issue that made IBM ship a separate new build of 9.0.1 to customers who ran into the issue.

      The fix needed a complete rebuild all Domino binaries/core components (because a central structure was affected) and could not be shipped in a normal FP. IBM found a way to address this issue in a FP!

      It is listed in the Fixlist under "Sametime" but the issue occurred in most cases in high load HTTP environments.

      In case you are running the special downloaded new 64bit compile you can now switch back to the standard builds (see more detailed information below).

      SPR# KBRN9Q7EZW - Fixed a Domino Linux 64-bit server crash or instability caused by duplicate thread ids.
      This is described in technote #1976013 and previously required a special Domino Linux 64-bit build to be provided.
      Now applying this Fixpack on Domino 9.0.1 will address the issue. Customers who previously received the special Domino Linux 64-bit build should uninstall it, re-install 9.0.1 Gold, followed by 9.0.1 FP7 or higher.



      -- AES and SHA-2 Support for Network Port Encryption --


      Dave Kern presented in Orlando already plans to update NRPC port encryption which have been planned for at that point 9.0.2.


      The new port encryption made it into FP7. If your client and server are both running FP7 or higher.

      Update 14.9.2016 19:00

      There is a new Technote describing all the details including two new settings plus one new debug setting.

      TN -> http://www.ibm.com/support/docview.wss?uid=swg21990283

      PORT_ENC_ADV
      controls the level of port encryption and enables the use of AES tickets.

      TICKET_ALG_SHA
      controls which cryptographic algorithm to use when constructing tickets. HMAC-SHA 256 is enabled by default.

      There is also one new debug setting DEBUG_PORT_ENC_ADV=1 which will enable debug for the new port encryption.


      I have upgraded my client and server and got the following with PORT_ENC_ADV on server side.
      In my previous test I wasn't aware that I had this parameter already in my notes.ini.
      But the parameter is required for the new encryption. The SH256 based signature algorithms are enabled by default.


      SPR# DKEN9N5PVK
      - Network port encryption now supports AES and SHA-2


      FP 6


      Authenticate {1B3F0009}: CN=xyz/OU=Srv/O=NashCom-Net

      T:
      RC2:128 E:1:  P:c:e S:RC4:128 A:4:1 L:N:N:N FS:

      FP 7


      Authenticate {1B3F0002}: CN=xzy/OU=Srv/O=NashCom-Net

      T:
      AES:128 E:1:  P:c:e S:AES-GCM:256 A:2:1 L:N:N:N FS:DHE-2048

      So it looks like the cipher implemented is: DHE-RSA-AES128-GCM-SHA256 with a DHE size of 2048.


      (You see the output with notes.ini log_authentication=1)


      -- Daniel

        Traveler 9.0.1.14 shipped with few but important fixes preparing for upcoming 9.0.1 FP7

        Daniel Nashed  8 September 2016 00:01:54
        The fixlist for 9.0.1.14 is quite short but it fixes a crash situation and issue in the upcoming Domino 9.0.1 FP7 fixpack which is scheduled for this month.

        APAR # Abstract
        LO89934 Meeting chair may get multiple response notices from invitee who is using Apple native calendar application.
        LO90090 Verbose flag missing from HADR command help display.
        LO90109 Domino API change in 9.0.1 FP7 could cause a server crash on Linux x64 if running IBM Traveler server 9.0.1.13 or earlier release.
        LO90110 Timing issue could cause server crash when checking platform disk status.


        Traveler 9.0.1.13 released with some fixes

        Daniel Nashed  19 August 2016 00:12:42
        There is a new traveler release that just shipped.
        Some of the issues might affect you.


        APAR # Abstract
        LO82881 Domino server may crash if $NTTrack field is corrupted.
        LO89471 Traveler invitee status may be incorrect if using mixed case internet addresses.
        LO89606 Number of recipients limited to 100 when sending mail from a mobile device.
        LO89745 Traveler server enters constrained state when load balancing a large number of users.
        LO89772 Meeting chair may receive multiple notices from attendee who processes notice on an Apple Native Calendar application.
        LO89840 IBM Verse mobile application fails to download entire mail for very large mail documents.
        LO89952 Deleted device still present in the Web Administration UI after the 30 day reap interval.
        LO89954 E-mail not in sent folder on mobile device when user sends and files an e-mail in Notes client.




        Release documentation:

        http://www.ibm.com/support/docview.wss?uid=swg21988973


        Fixlist:

        http://www.ibm.com/support/docview.wss?uid=swg21700212#90113

          Extended Master Secret Extension issue affects all Internet Protocols including STARTTLS

          Daniel Nashed  27 July 2016 10:23:28
          There is a an issue described in a technote which describes an issue with Win 2008 R2 and LDAP.
          This issue also occurs for other internet protocols!!

          It is specially important for servers using STARTTLS because you don't control which version and settings the receiving/sending host is using.

          So the issue I blogged about today does also affect other protocols. That's why I decided to have two blog posts to ensure it is better found on the web.

          Hiere is the info from the other blog post which also is relevant for your SMTP Servers.


          -- Daniel



          Domino  9.0.1 FP5 IF1 adds support for the Extended Master Secret Extension with TLS 1.2.

          Windows 2008 R2 does only supports TLS 1.0 but still sends the Extended Master Secret Extension in the server helo.

          Domino fails to connect because once this is offered Domino wants to use it.


          There is a work-around to disable this new functionality globally on the server via notes.ini


          SSL_DISABLE_EXTENDED_MASTER_SECRET=1


          This is just a work-around and the real fix would be that Microsoft provides  a fix for Win 2008 R2 to not send the extension with the helo.
          Later versions do support TLS 1.2 and do not have the issue.



          See the following technote for details ->
          http://www.ibm.com/support/docview.wss?uid=swg21987608

            Secure LDAP to Active Directory fails with Domino 9.0.1 FP5 IF1 and higher

            Daniel Nashed  27 July 2016 08:21:25

            Domino  9.0.1 FP5 IF1 adds support for the Extended Master Secret Extension with TLS 1.2.

            Windows 2008 R2 does only supports TLS 1.0 but still sends the Extended Master Secret Extension in the server helo.
            Domino fails to connect because once this is offered Domino wants to use it.

            There is a work-around to disable this new functionality globally on the server via notes.ini

            SSL_DISABLE_EXTENDED_MASTER_SECRET=1

            This is just a work-around and the real fix would be that Microsoft provides  a fix for Win 2008 R2 to not send the extension with the helo.
            Later versions do support TLS 1.2 and do not have the issue.


            See the following technote for details -> http://www.ibm.com/support/docview.wss?uid=swg21987608

            -- Daniel

            IBM Traveler 9.0.1.12 released including a security fix

            Daniel Nashed  14 July 2016 09:45:20
            IBM Traveler 9.0.1.12 shipped with some important changes.

            The first change is a security fix which is described below.


            But there is another security fix in the installer on Windows as well and some other fixes that could be affecting you.


            Upgraded my server already.

            -- Daniel



            Security Bulletin: XML External Entities Injection Vulnerability in IBM Traveler (CVE-2016-3039)
            IBM Traveler is vulnerable to a denial of service caused by an XML External Entity Injection (XXE) error when processing XML data.


            http://www.ibm.com/support/docview.wss?uid=swg21985858&myns=swglotus&mynp=OCSSYRPW&mync=E&cm_sp=swglotus-_-OCSSYRPW-_-E


            http://www.ibm.com/support/docview.wss?uid=swg21700212#90112
            APAR # Abstract
            LO87689 Invitee status not updated on Mobile device when external invitee responds.
            LO88807 Add the immediately remove invitee from invite on mobile device may not remove the invitee.
            LO88916 Invitee status not updated on Outlook client when external invitee responds.
            LO88950 Event still appears ghosted on mobile device after process an info update from ghosted entry.
            LO89057 Upgrade install technology to prevent MS Windows DLL Loading vulnerability.
            LO89097 Traveler device may display EnterSendTo field if SendTo empty for non-draft message.
            LO89287 Warning message for NumberFormatException for empty string should be Info log message and not a warning.
            LO89357 Update to prevent XML External Entities Injection vulnerability.
            LO89358 Same full name contact could sync wrong contact photo.
            LO89421 Ghosted entry for non-repeating event Cancel notice may show additional options on mobile device.
            LO89499 APNS notifications for IBM Verse for iOS may be in English instead of device preferred language.
            LO89501 Attachments and in-line images missing content header may not sync to mobile device.
            LO89540 Traveler Utility application should warn if attempting to change the DB2 user name as this may change the schema name as well.
            LO89543 Prevent device from renaming folder to null string.
            LO89544 Accept reschedule of non-repeating event from ghosted entry on Apple iOS Calendar application may not take effect on server.





            • [IBM Lotus Domino]
            • [Domino on Linux]
            • [Nash!Com]
            • [Daniel Nashed]