Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

Traveler Released

Daniel Nashed  7 March 2018 22:57:53
There is a new Traveler Release posted today. I don't see many critical issues but some might affect you.
I have just updated my server right now.Looks good so far.

There is a change in the database schema, when you run Traveler HA with a state database on SQL server:

"Note for customers that manage their database schema: IBM Traveler includes a database schema update for MS SQL Server deployments. It is only necessary to run verifyIndexes.sql to update the schema to latest level.
Otherwise no action is required unless upgrading from a version prior to If you use auto schema updates (default behavior) there is no action required."

-- Daniel

APAR # Abstract
LO93281 Modify an encrypted event from mobile device may corrupt event body.
LO93380 Support 32 bit Domino 9.0.1 Server.
LO93412 One index may cause performance problems on MS SQL Server.
LO93440 Incorrect default ACL for R6MemoMap.nsf
LO93455 Incorrect error code used for network error.
LO93466 Set $RFSaveInfo field on Reply/Forward from mobile device.
LO93491 Name used for time zone on mobile device does not match value used by Notes Client.
LO93522 Improve handling of very small in-line mime images.
LO93529 Web Administrator interface may show Verse for iOS device as not supporting data wipe.
LO93547 Not authorized message logged during network outage.
LO93596 Device may be missing e-mail if user has another device with a smaller filter window.
LO93599 Handle unexpected list format in notes.ini file.
LO93645 Event may not show on user's device when user was removed then re-invited to the event.
LO93660 Yellow status message displayed for Replicas table missing a Primary Key.
LO93663 Mail in sent folder may be missing content when configured to save with no attachments.
LO93706 Add NTS_JAVA_PARMS_EXT notes.ini parameter to allow for values larger than 256 characters.
LO93709 Attachment with DBCS characters in the file name may not display on mobile device.
LO93720 Update APNS Certificates, new expiration data March 30,2019.

    Notes and Domino 9.0.1 IF1 has been released

    Daniel Nashed  9 February 2018 08:39:40
    Notes and Domino 9.0.1 FP10 IF1 has been released.
    I already posted that the subject line encoding was already fixed with a HF I got thru support.
    And also that the version numbering issue has been fixed.

    The subject line issue also affects the Notes client when you run POP3/IMAP in the client because the same code is used.
    A fix for the client is planned. So in case you are using POP3/IMAP you should wait for the next IF.

    There is also a security vulnerability which has been pushed out to customers via FLASH: Security Bulletin.
    So in case you are running on Windows you are affected when any of the services like NSD or smart-upgrade! All releases are affected according to the technote.

    The LDAP authentication issue which was a regression in FP9 causing some type of authentications to fail.
    That regression is still pending to be fixed. So if you use your Domino as an authentication source via LDAP (for example for Sametime) you should still wait for the fix before upgrading that server.

    -- Daniel

    Domino Server 9.0.1 IF1

    PPUEASMHAW        Fixed a potential security vulnerability with the IBM Domino Diagnostics service Security Bulletin: IBM Notes NSD Privilege Escalation (technote 2010777)        
    PPUEASUDSF        Fixed a potential security vulnerability with the IBM Domino Diagnostics service Security Bulletin: IBM Notes Privilege Escalation in IBM Notes Diagnostics service (technote 2010767)        
    MKINAUWTG4        Fixed a potential Server memory leak        
    DVDI9UNH38                 Fixed a potential Server crash        
    JBAMAVKUPX        Fixed an issue where the subject line encoded (UTF-8) after upgrading to Domino 9.0.1FP10        
    KBRNAVLMA3        Fixed an issue where the version string returned was incorrect with the API NSFDbGetMajMinVersion after upgrading to Domino 9.0.1FP10        

    Notes Client 9.0.1 IF1

    PPUEASMHAW        Fixed a potential security vulnerability with the IBM Domino Diagnostics service Security Bulletin: IBM Notes NSD Privilege Escalation (technote 2010777)        
    PPUEASUDSF        Fixed a potential security vulnerability with the IBM Domino Diagnostics service Security Bulletin: IBM Notes Privilege Escalation in IBM Notes Diagnostics service (technote 2010767)        
    PPUEASNC5D        Fixed a potential security vulnerability with the IBM Notes Smart Update Service Security Bulletin: IBM Notes Privilege escalation in IBM Notes Smart Update Service  (technote 2010775)        
    KBRNAVLMA3        Fixed an issue where the version string returned was incorrect with the API NSFDbGetMajMinVersion after upgrading to Domino 9.0.1FP10

      Notes/Domino 9.0.1 FP10 Issues -- IBM is working on IF1 and is listening for more feedback

      Daniel Nashed  2 February 2018 23:05:01
      As posted before there is an issue with header encoding with umlauts which causes for example the subject to look scapbled.
      This issue is already SPRed and I got a mail that a hotfix is on the way (I have a PMR open).

      There are a couple of other issues development is working on. One is an issue I reported today about the Notes and Domino release version.

      The major Domino version is reported like this:

      9.0.1 FP10

      Release 900.0 QMR:11 QMU:0 Hotfix: 0 Fixpack: 0 (0)

      In contrast to


      Release 9.0 QMR:1 QMU:9 Hotfix: 0 Fixpack: 0 (0)

      This is causing issues with some add-on applications where licenses are issued based on the major Domino version like iQSuite and others.

      Beside those issues that are already posted, there are some others which are investigated.

      Stay tuned and check the following

      Update 03.02.2018 12:00

      I got a hotfix  9.0.1FP10HF47 for the subject conversion issue which I already installed.

      The first tests look good. The HF also contains a fix for the version number issue.

      Those fixes will be part of IF1. Stay tuned for more information.

      Release Information before the fix:

      Release 900.0 QMR:11 QMU:0 Hotfix: 0 Fixpack: 0 (0)

      After the fix:

      Release 9.0 QMR:1 QMU:9 Hotfix: 0 Fixpack: 0 (0)

      Update 05.02.2018 16:30

      The fix I got was just intended to fix the subject line issue. Not yet the build version issue.

      The build version should look like this with IF1 which is intended to be shipped this week.

      Local Notes/Domino Release 9.0 QMR:1 QMU:10 Hotfix: 0 Fixpack: 0 (0)

      Remote Notes/Domino Release 9.0 QMR:1 QMU:10 Hotfix: 0 Fixpack: 0 (0)

      Looks good for me now.

      Header/Subject Encoding issues after upgrading to 9.0.1 FP10

      Daniel Nashed  1 February 2018 13:09:04
      As Christian Henseler reported in the comments there is an issue with SMTP header encoding.
      Subjects that are UTF8 encoded are not converted.

      I can reproduce that on my server with different external mail accounts and also when using putty against a FP10 server. With FP9 it works well.

      They look like this:   =?utf-8?Q?Ã=84ü_Test_?=

      Until this problem is fixed you should not uprade to FP10!

      I have opened a PMR and also posted in the DP forum. L2 confirmed it looks like a regression.

      Stay tuned for updates.

      -- Daniel

      Notes/Domino 9.0.1 Feature Pack 10 shipped

      Daniel Nashed  31 January 2018 23:20:56

      Notes/Domino 9.0.1 Feature Pack 10 shipped.
      I updated my client and one of my production servers this morning. My upgrade went well. But as always I am interested in your feedback.

      When you install the admin/design client the result will be a single 1.8 JVM.
      Before we had a mix of JVM 1.8 for run-time and JVM 1.6 which made development more complicated.

      As you can read in the release notes beside the JVM version also the Eclipse version has been updated to work with JVM 1.8 and also to introduce new functionality.

      The compile time JVM is the biggest change in FP 10.

      The fixlist database is not updated yet but the readme already contains fixes.

      I have reverted my cipher list to the default and tested if the ROBOT SSL/TLS Attack is fixed.
      SSL Labs does not show any issues any more and the fix just comes in time before SSL Labs will give server not fixed against the  OBOT SSL/TLS Attack and using those vulnerable ciphers get a  "F" rating beginning of February.

      The browser stays removed. And there are still discussions among business partners who used embedded browser functionality in their solutions.

      ROBOT SSL/TLS Attack

      Daniel Nashed  17 January 2018 04:41:02
      This has not been widely discussed yet. But since SSL Labs will start reporting it with a rating of F beginning of February let me explain some background and what you could do.

      The issue has been there in a similar way before and is back. You can read the details here -->

      Affected are the older ciphers that are not widely used by current browsers/client. You could disable those ciphers until the issue is fixed.

      But on the other side most browsers/clients do support higher secure ciphers. And because by default the server cipher order is used, a client should not choose a weaker cipher.

      In addition because of Secure Renegotiation which is supported by Domino and most browsers/clients support it, no weaker cipher will be used than the best common cipher between client and server.

      That means that only a very small fraction of connections might use those affected ciphers and if you disable those the client cannot connect at all.

      A fix for the ROBOT Attack is planned for FP10.

      So IMHO there is no need right now to disable those affected older RSA ciphers unless you have very high security requirements or if you are concerned about your SSL labs rating ..

      If you disable those affected ciphers the warning on the SSL Labs test side goes away.

      Here is a more paranoid configuration of TLS ciphers that you could use:

      set config SSLCipherSpecÀ30009FC02F009EC028006BC0140039C02700670033C013

      restart task http

      If you look into the compatibility report, there is no current client that could not connect any more (even older IE versions would connect).
      The other positive effect would be that you would only support DHE and ECDHE ciphers which is a good idea in general..

      UPDATE 17.01.2018

      Andy Brunner had an interesting comment. In my cipher list I am still having 0033 which is rated as a weak cipher which is not enabled by default.

      I have a cipher configuration database where I still had that cipher listed.

      TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x33)

      If you still have this cipher listed and did not allow weak ciphers the server will give you a hint:

      SSLDisableExportCiphers> Disabling weak cipher
      DHE_RSA_WITH_AES_128_CBC_SHA. Set notes.ini "USE_WEAK_SSL_CIPHERS=1" to re-enable.

      So the better fitting cipher liste in this case would be

      set config SSLCipherSpecÀ30009FC02F009EC028006BC0140039C0270067C013
      restart task http

      The mentioned cipher is rated as weak by Domino because it is a cipher that internally uses "SHA"

      Update: I almost forgot and got reminded about this Java 1.6 issue.
      The cipher is rated as weak for another reason. Older Java can only support this DHE cipher with 1024 bit.

      That's a longer story which you can find details here -->
      and another blog post here with some more details and ideas -->

      -- Daniel

      Image:ROBOT SSL/TLS Attack

      Meltdown and Spectre Exploit

      Daniel Nashed  5 January 2018 00:44:30

      There is a new security issue for most modern CPUs. Intel and AMD is affected in different ways.
      It's not something that is application specific. It's a CPU and OS level issue. Which affects also virtualization hosts.

      Here is the best website to get details -->

      And there are already some patches for some platforms.

      I have just installed the current kernel patches for CentOS (kernel 2.6.32-696.18.7).

      Here is the info from RHEL about the first patches

      We will probably see patches for other platforms including virtualization platforms like ESX.

      Those are the first fixes. And we will probably see more followup fixes.

      Update 06.01.2018:

      There is an interresting article describing some of the background and what hardware and software vendors are doing against it with different approaches.

      From what I see the applications with the biggest exposure to those bugs are web-browsers because they execute active code from remote (e.g. JavaScript).

      Here is also a current statement from Mozilla:

      And there is the status page for Chrome and other Google technologis:

      -- Daniel


      Daniel Nashed  31 December 2017 10:19:26
      There is a new APAR which describes a performance issue on Windows 2008 and earlier.
      The APAR is based on a PMR which I had open with IBM. So I want give you the full detail about what we found out instead of the summary described in  


      We had a situation where we needed to replicate databases from an existing Domino 8.5.3 FP6 Windows 2008 environment to a new Domino 9.0.1 FP9 Linux 64 environment.  
      The replication was quite slow and we tried all kind of optimization on Domino, Windows and Linux.
      Beside increasing the sending/receiving buffer and memory we have been looking into ways to optimize the the Domino configuration.

      It turned out that Domino NRPC network compression was not always helpful depending on the configuration. So we ended up to disable network compression in our particular case. But this might not help in your configuration. It's something that needs testing.
      Between the Domino application sending the data and the IP-Stack sending the actual data there is a layer called the "NTI" layer which is responsible for actually coordinating sending the data.  

      The buffer size can not be modified and depending on the transaction higher latency networks take some time for the round-trip between sending and receiving side.
      But the main issue we have seen was that sending attachments which have a bigger chunk size that is send over the network was also slow.  

      Windows 2008 TCP/IP Issue

      The issue we found on the Windows IP stack only had impact in network environments which have higher latency than a local network where the latency is around 1 ms.  
      Our environment had 1 GBit and around 6 ms latency which is already great for a wide area connection . If you are having higher latency the performance might be even lower!
      We also reproduced the slow performance also with a faster connection with similar latency (10 GBit network with 5-6 ms latency). So it is the latency that has impact!  

      For transferring attachments with my own written C-API test application we have seen 2,5 MB/sec transferring data from Win2008.
      In contrast Windows 2012 did not have the same issue and that was very strange for us.  

      After discussion with the network team and a lot of tests we found the following tuning parameters.
      Both parameter do not exist by default on Win2008 R2 but the DefaultSendWindow exists for example on Win7 (which hare  comparable network stack) with a smaller value.  


      Those settings ensures that the much more chunk sizes are send over the network before the IP-stack waits for the ACK from the other side. By default it was around 12 KB of data which was quite small!
      The first tests on our internal environment after the change showed 35 MB/sec!

      But that does not mean that normal replication will have the same performance because it is a mix of different transactions! We only tested object write transactions which had the biggest impact in our case.

      Object Write Chunk-Size is 256 KB

      In discussions with IBM we also found out that the documentation for changing the chunk size for sending attachment data was wrong.  

      The WIKI documentation say that the chunk size ist 64 KB and can be increased with a Notes.ini parameter up to 1 MB.  
      But it turns out that the parameter was only implemented as a test for a customer and the fix had never been added to the code.  

      Here is the technote describing Notes.ini SERVER_SEND_OBJECT_CHUNK_SIZE.  

      This is the only documentation for the parameter that should be corrected. The parameter does currently not exist and the default is 256 KBinstead of 64 KB. 
      I have been testing different chunk sizes between 64KB and 1 MB with a low level C-API application which writes attachments.
      And I found out that 256KB is a good balanced value. So there would be no need to change this parameter.

      Conclusion and some additional tips for AdminP

      When you are running on Win20012 or higher you don't need to change anything.  
      For Windows 2008 you should really set the registry parameter, because this will be a big boost for your replication performance.  

      On the other side the nature of replication is that document level replication will always take some time even in a local network.
      That's why Domino provides accelerated replica which uses a different transaction type.
      It's a kind of backup restore over the network. But that does only work if the database is not DAOS enabled.
      For DAOS enabled databases the replicator is used and it takes benefit of storage optimization.
      It will only send the attachment if it isn't yet on the remote side. But this might be still slower compared to an accelerated replica.

      To better utilize the bandwidth of your 1 GBit line we ended up having multiple AdminP threads leveraging the replicator code to push databases in parallel.
      There is ab enhancement in the 9.0.1 codestream (we got it backported to 8.5.3 FP6) which allows one process with multiple threads to replicate in parallel.

      And if you want AdminP to create the replica immediately instead of just creating a replica stub you need the following notes.ini parameter: ADMINP_EXCHANGE_ALL_UNREAD_MARKS=1.

      When you set this parameter Adminp actually pushes the database instead of creating a replica stub and also syncs all unread marks for the database.
      Note:  The admin4.nsf the request type will look like accelerated replica copy even DAOS is enabled on the database and the status of the request also looks a bit different.
      You should not be worried about that. It will use the normal replicator code including unread mark sync.


      Domino 9.0.1 FT Index Hang and potential crash

      Daniel Nashed  10 December 2017 06:08:37
      We ran into a hang situation multiple times during FT indexing. It turned out that this is a regression introduced in FP9 due to changes in the FT index area.

      In certain situations the FTIndex update does hang getting document data and will cause one CPU core to be maxed out for this thread.
      The description of the SPR says it is a "spike" but it more looks like the thread permanently uses CPU.

      This can happen with updall, DBMT and also other tasks updating the FT index.
      The process cannot be stopped and this will also cause that the server cannot be shutdown cleanly.

      We got a hotfix which will will be included in IF3. After applying the hotfix we had no new server hangs.

      I am including the call-stack for the hang to this blog post to have it searchable for others who might run into the same issue.
      If you have not installed FP9 you should wait for IF3. If you are on FP9 and run into this issue, take a full NSD, open a PMR and reference the mentioned SPR numbers to get the fix.

      -- Daniel

      -- Fixed SPRs --

      J3 server crashed on DBMT task, while full text indexing the database

      CPU spike when running dbmt (or updall/update task) and creating full text index.

      -- Call Stack --

      Thread 3 (Thread 0x7f2c5da71700 (LWP 17594)):
      #0  ODSToOrFromHost (toHost=32769, type=0, vbuffer=0x7f2c5da6e8e0, iterations=1) at ods.c:824
      #1  0x00007f2cba7ef8fe in ODSReadItem (src=0x7f2c54466d96, type=, dest=0x7f2c5da6e8e0) at ods.c:1420
      #2  0x00007f2cbab635e2 in GetChar(STREAM_CTX*, STREAM_DATA*) () from /opt/ibm/domino/notes/latest/linux/
      #3  0x00007f2cbab64932 in FTGetDocStream () from /opt/ibm/domino/notes/latest/linux/
      #4  0x00007f2c5d390919 in NotesStreamReadChar (arg=) at ftg_dstr.cpp:1412
      #5  0x00007f2cbab5ca7c in FTLexMatch () from /opt/ibm/domino/notes/latest/linux/
      #6  0x00007f2c5d39296c in FTGCreateIndex (pFTGCtx=0x7f2c4c00abf8) at ftg_dstr.cpp:1839
      #7  0x00007f2c5d38bac0 in CFTNoteIndexer::ProcessDoc(FTG_CTX *, struct {...} &) (this=, pFTGCtx=0x7f2c4c00abf8, docIndexerInfo=...) at ftgindex.cpp:2074
      #8  0x00007f2c5d38c5d1 in FTGIndexIDProc (Parameter=, NoteID=207326) at ftgindex.cpp:1685
      #9  0x00007f2cb999285d in IDEnumerate (hTable=536872571, Routine=0x7f2c5d38c343 , Parameter=0x7f2c4c00abf8) at idtable.c:2216
      #10 0x00007f2c5d38e252 in FTGIndex(FT_THREAD *, struct {...} *, WORD, char *) (pftt=0x7f2cb41004d0, pFTStreamCtx=0x7f2c4c00abf8, Options=392, StopFile=) at ftgindex.cpp:1146
      #11 0x00007f2cbab5adce in FTCallIndex () from /opt/ibm/domino/notes/latest/linux/
      #12 0x00007f2cbab5c3a3 in FTIndexExt2 () from /opt/ibm/domino/notes/latest/linux/
      #13 0x00007f2cb93e8485 in UpdateFullTextIndex (hDB=1154, Pathname=0x7f2cb4101648 "mail/c1/xn06451.nsf", Flags=201342976, fullTextStatus=8) at update.c:1239
      #14 0x00007f2cb93ea78f in UpdateCollectionsExt (_hModule=, Pathname=0x7f2cb4101648 "mail/c1/xn06451.nsf", Type=2, Flags=201342976, Flags2=0, mSecs=0, ViewNoteID=0, ContainerObjectID=0, ViewTitle=0x40a360 "", retDbTitle=0x0, fSrchSite=0, QueuedRequest=0, retbLater=0x0, fullTextStatus=8, wantsFulltext=0x0) at update.c:660
      #15 0x00007f2cb93ea957 in UpdateCollections (_hModule=32769, Pathname=0x0, Flags=, ViewNoteID=, ContainerObjectID=, ViewTitle=, retDbTitle=0x0, fSrchSite=0, QueuedRequest=0, retbLater=0x0, fullTextStatus=8, wantsFulltext=0x0) at update.c:106
      #16 0x0000000000405238 in UpdallThread (threadparam=) at dbmt.c:2108
      #17 0x00007f2cb98e7be3 in ThreadWrapper (Parameter=) at thread.c:1183
      #18 0x0000003aae007aa1 in start_thread () from /lib64/
      #19 0x0000003aadce8bcd in clone () from /lib64/

      End of Service for JVM 1.6

      Daniel Nashed  25 November 2017 13:34:30

      IBM uses the Oracle JVM as their base for their IBM JVM platform which is used in IBM products like Notes, Domin and Traveler.

      JVM 6.0 has been around for almost 10 years and is now discontinued since Sep 2017.
      Oracle discontinued their support for JVM 1.6 so IBM cannot support JVM 1.6 on their side.

      That also means for IBM platforms that there is no patch support for JVM 1.6!

      For Notes and Domino means you have to update to 9.0.1 FP8/FP10 for JVM 1.8 and hopefully FP10 will bring compile time for JVM 1.8 as well (current planning).

      If you are running on Notes/Domino 8.5.3 or an earlier 9.0.1 FP don't panic. Most Java applications on Domino are not directly accessible over the network. There is at least the Domino HTTP stack between the client and the Java application.

      On the client side you might have direct connection from the client to the internet. And for encrypted connections there have been limitations before in the SSL/TLS area as posted before.
      For example there is just very limited TLS 1.2 support in JVM 1.6 with just one chiper.

      I personally would still wait for Feature Pack 10 and have the full JVM 1.8 support also at compile time. But you should be aware that it is time to move to a current release.

      If you are on 9.0.1 you are just a "FP" install away. If you are on 8.5.3 there are another good reasons to move to a current 9.0.1 release from security point of view. For example missing SHA-256 support and no TLS 1.2 support - not just for JVM.

      Here is a link to the support cycle for the IBM JVM



      • [IBM Lotus Domino]
      • [Domino on Linux]
      • [Nash!Com]
      • [Daniel Nashed]