Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

First impressions from Engage Conference in Rotterdam this week

Daniel Nashed  23 May 2018 22:39:40
Jut back from #engageug. The conference was the best conference I have been for a long time!
Beside the great venue on the cruise ship the HCL sessions technical session and specially the round tables have been a big positive surprise for me.

As a already posted this week my impression is that the move from IBM to HCL gave a big positive push to the development team!
The feedback they took and there reactions to our ideas what the details we got about Notes/Domino/Verse/Sametime 10 was more than positive.
I am really looking forward to get my hands on the first beta arriving for a closed group in June and a beta 2 is planned in July for a larger audience.

HCL is not only investing in the Domino server and all the new stuff like support the client on iPad and on Android tables that is coming but they also invest in the Notes client!
Beside the new client functionality we have seen in the sessions there way more in the round table discussions that HCL and IBM did tell us. And they have been very open for additional feedback!
Not all of it will make it into Notes/Domino 10 but they are already planning for a Notes/Domino 11 release (planned for 2019).

Beside of of the features there is great news for AIX, iSeries/System i, SLES customers.
Heard yesterday at our Linux round table Domino 10 is planned to be supported on all of those platforms!!
The only platform that will be dropped is Win32. The plan is introduce Win64 support for Sametime 10 and remove the remaining road blocker for dropping Win32.

In addition to that there is great news for customers who want to run Domino on CentOS.
CentOS is a long term release/free community edition which is source code compatible with REHL.

So with the feedback we gave at the conference and in our roundtable IBM/HCL is planning to introduce "best effort" support for CentOS 7 in Domino 10!
That means that you will be able to submit PMRs. You will not longer have to reproduce it on one of the other supported platforms!
One of the other reasons is that the base image for Docker in Domino 10 will be very likely CentOS as well.

There are already many smaller customers using CentOS for their Domino environment. And I am also supporting CentOS for my Domino start script.

So I am very happy about this move! This will be specially interested for smaller customers and maybe new developers who want start to look into the new stack that is planned for Domino 10 with Node JS.

HCL showed live code in their sessions and had iPads with the current build of the HCL Nomad to play with!
I am really looking forward to get my hands on the betas!

Thanks again to Theo and the team this marvelous conference!


Domino 9.0.1 FP10 IF2 - one important fix is missing

Daniel Nashed  21 May 2018 09:06:21
Domino 9.0.1 FP10 IF2 has been released with some important fixes.

If you are on FP10 you should apply IF2. If you did not install FP10 you should wait for IF3.

There is one missing fix for a Domino server hang that you should be aware of.  I personally know at least about 3 customers who ran into this issue.
It can happen if two processes are updating the same database. For example update, replica, router and server!

It wasn't clear when IF2 was released and I had to double check with IBM before posting. That took some time.

So if you are not on FP10 you should wait for IF3 which should have those two fixes.

SWAS96ZP5B: Server hang due to semaphore deadlock between a doc update & nsf search (cc# 323200)
JMANAS8HZP - Crash on router after upgrade to 901FP9 on nIMAP with PANIC: OSBBlockAddr: Bad BBlock handle (FFFFFFFB) (cc# 322553)

-- Daniel

Interim Fix 2
RMASAT7KEX         Fixed an issue where LDAP does not properly escape # character in results        
YYYYAQK8QM         Fixed an HTTP PANIC on LookupHandle        
VNEN8J2MUZ         Fixed a router PANIC on LookupHandle        
RDUK5E6U8Z         Fixed an issue where sending a message from iNotes a second instance of "MIME-Version: 1.0" occurs in MIME header        
SWASAVKR2A         Fixed a hang w/ extension mgr search calls        
ARUIAXQW38         Fixed a panic at Domino startup if statically enable inline view indexes for Domino 9.0.1 Feature Pack 10        
MKINAXNM6V         Fixed an issue where check markers crash in WriteJavaScriptMethods()        

Destination Domino / Notes/Domino 10

Daniel Nashed  19 May 2018 11:26:16

Destination Domino / Notes/Domino 10

I haven't blogged much in the last weeks and I haven't been at IBM Think in Las Vegas for personal reasons.

The feedback I got from partners and customers after IBM Think and at the Swiss user Group (SNoUG) event in Z├╝rich was very positive.

I would have wished we would have got that energy back some years before.

Personally I wasn't a big fan of feature packs. It has been a great idea to ship incremental feature releases in an agile style as soon they are ready.

But renaming "FP" form Fixpack to Feature Pack also introduced some technical deployment challenges and the market did not really understand it.

There are improvements in every fixpack and feature packs. For example some long planned features like NIFNSF and large summary data.

I have spoken at many conferences about best practices and features introduced in the recent feature packs.

Those new backend database features and other new functionality showed long term commitment even there was some not completely clear communication about support extension for the 9.0.1 release.

Having a Domino 10 release and already speaking about future plans like "Domino 11" is great news!
Now with HCL taking over the development including all the developers a Domino 10 release makes a lot of sense.

Development moved to a HCL location near to the IBM offices in Littleton, MA.Now they can finish implementation of ideas they started to work on years ago.
What I hear there is a lot of new energy and they are looking for 70 new developers.

IBM was reducing investment and staff not only for Collaboration software but also in other areas.
Having HCL as a company who believes in the product and sees the potential behind it as a platform is great news!

HCL is a large international company with locations world-wide and they wouldn't invest in it, if they would not see future in it!

The Notes client on iOS is a good example. The first versions of it existed already for a while. It's an own port which will be also available on Android!
And by the way the Notes 10 client will share the code base with the Mac client. The current plan is to have the Windows client available in 64bit with Notes 11 in 2019.

Looking into the presentations from IBM Think in Las Vegas I did not see all features and new development capabilities planned for Domino 10 that I did hear from partners and customers after the conference and from other sources.

But IBM and HCL are very clear about the long term support for Notes/Domino!

At SNoUG the picture was more clear and I am looking forward to the Engage conference in Rotterdam to see more details  -->
We also finalized the agenda for the German DNUG event in Darmstadt mid of June  -->

I will be presenting at both conferences and I am looking forward to meet many of you at those two events!

You should also keep an eye on the Destination Domino website -->

And I will be more active on my blog after the Engage conference.

-- Daniel

Notes Client FP Installs fail starting 20.4.2018

Daniel Nashed  24 April 2018 09:38:04
Since 20.4.2018 Notes Client Feature Pack Installations are failing.
I got the first report on Friday and another customer pinged me on Monday.

And we are discussing among partners about this issue.

One partner reported that updates from FP9 to FP10 still work for him. But all other updates are failing.

IBM state that this isn't an issue with FP7 or FP8. But I have not verified that.

The root cause is that signatures for some plug-ins cannot be verified any more because the certificate validation expired.

It's not just one plug-in that fails and you might have a different plug-in that is reported to cause the error.

Here is the error message that you see n the Eclipse logs: NotAfter: Fri Apr 20 01:59:59 CEST 2018

If you turn back the time to an earlier date it's still works. But this isn't a work-around I would suggest.

There is another work-around to temporary allow expired certificates of signatures during install and enable it afterwards.

You would need to add the following setting to plugin_customization.ini and revert the setting back after installation.


But again this is also just a work-around.

IBM is aware of the issue and posted a technote ->

They are working on a solution. The technote says that they don't know exactly what is causing the issue. We have to wait for their update.

And I would recommend to wait before updating your clients until a fix is available instead using the work-arounds.

Update April 26, 2018:

IBM/HCL posted an update today in the TN. They found the issue, fixed it and hopefully if QE testing is successful we will have a fix soon.

@Marc, see they are posting an update for 9.0.1 FP9, FP10, Current MAC 64bit Client and also 8.5.3 FP6!
So the update for FP9 will be first and the other versions follow soon!

Here are the details from the update:

We have found a solution for the certificate issue for all products and versions. Please see the eGA for each product and version in the table below. Please note this is subject to change depending on the outcome of our QE Testing.
Product Version                                  Projected eGA
Notes Client 853FP6IF16                    Tuesday May 1, 2018
Notes Client 901FP9IF2                 Friday April 27, 2018
Notes Client 901FP10IF3                 Beginning of May 2018
MAC 64-bit IF15                         Tuesday May 1, 2018
Sametime Standalone 9.0.1 FP1         Beginning of May 2018

-- Daniel

Traveler Released

Daniel Nashed  7 March 2018 22:57:53
There is a new Traveler Release posted today. I don't see many critical issues but some might affect you.
I have just updated my server right now.Looks good so far.

There is a change in the database schema, when you run Traveler HA with a state database on SQL server:

"Note for customers that manage their database schema: IBM Traveler includes a database schema update for MS SQL Server deployments. It is only necessary to run verifyIndexes.sql to update the schema to latest level.
Otherwise no action is required unless upgrading from a version prior to If you use auto schema updates (default behavior) there is no action required."

-- Daniel

APAR # Abstract
LO93281 Modify an encrypted event from mobile device may corrupt event body.
LO93380 Support 32 bit Domino 9.0.1 Server.
LO93412 One index may cause performance problems on MS SQL Server.
LO93440 Incorrect default ACL for R6MemoMap.nsf
LO93455 Incorrect error code used for network error.
LO93466 Set $RFSaveInfo field on Reply/Forward from mobile device.
LO93491 Name used for time zone on mobile device does not match value used by Notes Client.
LO93522 Improve handling of very small in-line mime images.
LO93529 Web Administrator interface may show Verse for iOS device as not supporting data wipe.
LO93547 Not authorized message logged during network outage.
LO93596 Device may be missing e-mail if user has another device with a smaller filter window.
LO93599 Handle unexpected list format in notes.ini file.
LO93645 Event may not show on user's device when user was removed then re-invited to the event.
LO93660 Yellow status message displayed for Replicas table missing a Primary Key.
LO93663 Mail in sent folder may be missing content when configured to save with no attachments.
LO93706 Add NTS_JAVA_PARMS_EXT notes.ini parameter to allow for values larger than 256 characters.
LO93709 Attachment with DBCS characters in the file name may not display on mobile device.
LO93720 Update APNS Certificates, new expiration data March 30,2019.

    Notes and Domino 9.0.1 FP10 IF1 has been released

    Daniel Nashed  9 February 2018 08:39:40
    Notes and Domino 9.0.1 FP10 IF1 has been released.
    I already posted that the subject line encoding was already fixed with a HF I got thru support.
    And also that the version numbering issue has been fixed.

    The subject line issue also affects the Notes client when you run POP3/IMAP in the client because the same code is used.
    A fix for the client is planned. So in case you are using POP3/IMAP you should wait for the next IF.

    There is also a security vulnerability which has been pushed out to customers via FLASH: Security Bulletin.
    So in case you are running on Windows you are affected when any of the services like NSD or smart-upgrade! All releases are affected according to the technote.

    The LDAP authentication issue which was a regression in FP9 causing some type of authentications to fail.
    That regression is still pending to be fixed. So if you use your Domino as an authentication source via LDAP (for example for Sametime) you should still wait for the fix before upgrading that server.

    -- Daniel

    Domino Server 9.0.1 IF1

    PPUEASMHAW        Fixed a potential security vulnerability with the IBM Domino Diagnostics service Security Bulletin: IBM Notes NSD Privilege Escalation (technote 2010777)        
    PPUEASUDSF        Fixed a potential security vulnerability with the IBM Domino Diagnostics service Security Bulletin: IBM Notes Privilege Escalation in IBM Notes Diagnostics service (technote 2010767)        
    MKINAUWTG4        Fixed a potential Server memory leak        
    DVDI9UNH38                 Fixed a potential Server crash        
    JBAMAVKUPX        Fixed an issue where the subject line encoded (UTF-8) after upgrading to Domino 9.0.1FP10        
    KBRNAVLMA3        Fixed an issue where the version string returned was incorrect with the API NSFDbGetMajMinVersion after upgrading to Domino 9.0.1FP10        

    Notes Client 9.0.1 IF1

    PPUEASMHAW        Fixed a potential security vulnerability with the IBM Domino Diagnostics service Security Bulletin: IBM Notes NSD Privilege Escalation (technote 2010777)        
    PPUEASUDSF        Fixed a potential security vulnerability with the IBM Domino Diagnostics service Security Bulletin: IBM Notes Privilege Escalation in IBM Notes Diagnostics service (technote 2010767)        
    PPUEASNC5D        Fixed a potential security vulnerability with the IBM Notes Smart Update Service Security Bulletin: IBM Notes Privilege escalation in IBM Notes Smart Update Service  (technote 2010775)        
    KBRNAVLMA3        Fixed an issue where the version string returned was incorrect with the API NSFDbGetMajMinVersion after upgrading to Domino 9.0.1FP10

      Notes/Domino 9.0.1 FP10 Issues -- IBM is working on IF1 and is listening for more feedback

      Daniel Nashed  2 February 2018 23:05:01
      As posted before there is an issue with header encoding with umlauts which causes for example the subject to look scapbled.
      This issue is already SPRed and I got a mail that a hotfix is on the way (I have a PMR open).

      There are a couple of other issues development is working on. One is an issue I reported today about the Notes and Domino release version.

      The major Domino version is reported like this:

      9.0.1 FP10

      Release 900.0 QMR:11 QMU:0 Hotfix: 0 Fixpack: 0 (0)

      In contrast to


      Release 9.0 QMR:1 QMU:9 Hotfix: 0 Fixpack: 0 (0)

      This is causing issues with some add-on applications where licenses are issued based on the major Domino version like iQSuite and others.

      Beside those issues that are already posted, there are some others which are investigated.

      Stay tuned and check the following

      Update 03.02.2018 12:00

      I got a hotfix  9.0.1FP10HF47 for the subject conversion issue which I already installed.

      The first tests look good. The HF also contains a fix for the version number issue.

      Those fixes will be part of IF1. Stay tuned for more information.

      Release Information before the fix:

      Release 900.0 QMR:11 QMU:0 Hotfix: 0 Fixpack: 0 (0)

      After the fix:

      Release 9.0 QMR:1 QMU:9 Hotfix: 0 Fixpack: 0 (0)

      Update 05.02.2018 16:30

      The fix I got was just intended to fix the subject line issue. Not yet the build version issue.

      The build version should look like this with IF1 which is intended to be shipped this week.

      Local Notes/Domino Release 9.0 QMR:1 QMU:10 Hotfix: 0 Fixpack: 0 (0)

      Remote Notes/Domino Release 9.0 QMR:1 QMU:10 Hotfix: 0 Fixpack: 0 (0)

      Looks good for me now.

      Header/Subject Encoding issues after upgrading to 9.0.1 FP10

      Daniel Nashed  1 February 2018 13:09:04
      As Christian Henseler reported in the comments there is an issue with SMTP header encoding.
      Subjects that are UTF8 encoded are not converted.

      I can reproduce that on my server with different external mail accounts and also when using putty against a FP10 server. With FP9 it works well.

      They look like this:   =?utf-8?Q?Ã=84ü_Test_?=

      Until this problem is fixed you should not uprade to FP10!

      I have opened a PMR and also posted in the DP forum. L2 confirmed it looks like a regression.

      Stay tuned for updates.

      -- Daniel

      Notes/Domino 9.0.1 Feature Pack 10 shipped

      Daniel Nashed  31 January 2018 23:20:56

      Notes/Domino 9.0.1 Feature Pack 10 shipped.
      I updated my client and one of my production servers this morning. My upgrade went well. But as always I am interested in your feedback.

      When you install the admin/design client the result will be a single 1.8 JVM.
      Before we had a mix of JVM 1.8 for run-time and JVM 1.6 which made development more complicated.

      As you can read in the release notes beside the JVM version also the Eclipse version has been updated to work with JVM 1.8 and also to introduce new functionality.

      The compile time JVM is the biggest change in FP 10.

      The fixlist database is not updated yet but the readme already contains fixes.

      I have reverted my cipher list to the default and tested if the ROBOT SSL/TLS Attack is fixed.
      SSL Labs does not show any issues any more and the fix just comes in time before SSL Labs will give server not fixed against the  OBOT SSL/TLS Attack and using those vulnerable ciphers get a  "F" rating beginning of February.

      The browser stays removed. And there are still discussions among business partners who used embedded browser functionality in their solutions.

      ROBOT SSL/TLS Attack

      Daniel Nashed  17 January 2018 04:41:02
      This has not been widely discussed yet. But since SSL Labs will start reporting it with a rating of F beginning of February let me explain some background and what you could do.

      The issue has been there in a similar way before and is back. You can read the details here -->

      Affected are the older ciphers that are not widely used by current browsers/client. You could disable those ciphers until the issue is fixed.

      But on the other side most browsers/clients do support higher secure ciphers. And because by default the server cipher order is used, a client should not choose a weaker cipher.

      In addition because of Secure Renegotiation which is supported by Domino and most browsers/clients support it, no weaker cipher will be used than the best common cipher between client and server.

      That means that only a very small fraction of connections might use those affected ciphers and if you disable those the client cannot connect at all.

      A fix for the ROBOT Attack is planned for FP10.

      So IMHO there is no need right now to disable those affected older RSA ciphers unless you have very high security requirements or if you are concerned about your SSL labs rating ..

      If you disable those affected ciphers the warning on the SSL Labs test side goes away.

      Here is a more paranoid configuration of TLS ciphers that you could use:

      set config SSLCipherSpecÀ30009FC02F009EC028006BC0140039C02700670033C013

      restart task http

      If you look into the compatibility report, there is no current client that could not connect any more (even older IE versions would connect).
      The other positive effect would be that you would only support DHE and ECDHE ciphers which is a good idea in general..

      UPDATE 17.01.2018

      Andy Brunner had an interesting comment. In my cipher list I am still having 0033 which is rated as a weak cipher which is not enabled by default.

      I have a cipher configuration database where I still had that cipher listed.

      TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x33)

      If you still have this cipher listed and did not allow weak ciphers the server will give you a hint:

      SSLDisableExportCiphers> Disabling weak cipher
      DHE_RSA_WITH_AES_128_CBC_SHA. Set notes.ini "USE_WEAK_SSL_CIPHERS=1" to re-enable.

      So the better fitting cipher liste in this case would be

      set config SSLCipherSpecÀ30009FC02F009EC028006BC0140039C0270067C013
      restart task http

      The mentioned cipher is rated as weak by Domino because it is a cipher that internally uses "SHA"

      Update: I almost forgot and got reminded about this Java 1.6 issue.
      The cipher is rated as weak for another reason. Older Java can only support this DHE cipher with 1024 bit.

      That's a longer story which you can find details here -->
      and another blog post here with some more details and ideas -->

      -- Daniel

      Image:ROBOT SSL/TLS Attack


      • [IBM Lotus Domino]
      • [Domino on Linux]
      • [Nash!Com]
      • [Daniel Nashed]