Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

    Notes and Domino 9.0.1 IF1 has been released

    Daniel Nashed  9 February 2018 08:39:40
    Notes and Domino 9.0.1 FP10 IF1 has been released.
    I already posted that the subject line encoding was already fixed with a HF I got thru support.
    And also that the version numbering issue has been fixed.

    The subject line issue also affects the Notes client when you run POP3/IMAP in the client because the same code is used.
    A fix for the client is planned. So in case you are using POP3/IMAP you should wait for the next IF.

    There is also a security vulnerability which has been pushed out to customers via FLASH: Security Bulletin.
    So in case you are running on Windows you are affected when any of the services like NSD or smart-upgrade! All releases are affected according to the technote.

    The LDAP authentication issue which was a regression in FP9 causing some type of authentications to fail.
    That regression is still pending to be fixed. So if you use your Domino as an authentication source via LDAP (for example for Sametime) you should still wait for the fix before upgrading that server.

    -- Daniel


    Domino Server 9.0.1 IF1

    PPUEASMHAW        Fixed a potential security vulnerability with the IBM Domino Diagnostics service Security Bulletin: IBM Notes NSD Privilege Escalation (technote 2010777)        
    PPUEASUDSF        Fixed a potential security vulnerability with the IBM Domino Diagnostics service Security Bulletin: IBM Notes Privilege Escalation in IBM Notes Diagnostics service (technote 2010767)        
    MKINAUWTG4        Fixed a potential Server memory leak        
    DVDI9UNH38                 Fixed a potential Server crash        
    JBAMAVKUPX        Fixed an issue where the subject line encoded (UTF-8) after upgrading to Domino 9.0.1FP10        
    KBRNAVLMA3        Fixed an issue where the version string returned was incorrect with the API NSFDbGetMajMinVersion after upgrading to Domino 9.0.1FP10        

    Notes Client 9.0.1 IF1

    PPUEASMHAW        Fixed a potential security vulnerability with the IBM Domino Diagnostics service Security Bulletin: IBM Notes NSD Privilege Escalation (technote 2010777)        
    PPUEASUDSF        Fixed a potential security vulnerability with the IBM Domino Diagnostics service Security Bulletin: IBM Notes Privilege Escalation in IBM Notes Diagnostics service (technote 2010767)        
    PPUEASNC5D        Fixed a potential security vulnerability with the IBM Notes Smart Update Service Security Bulletin: IBM Notes Privilege escalation in IBM Notes Smart Update Service  (technote 2010775)        
    KBRNAVLMA3        Fixed an issue where the version string returned was incorrect with the API NSFDbGetMajMinVersion after upgrading to Domino 9.0.1FP10


      Notes/Domino 9.0.1 FP10 Issues -- IBM is working on IF1 and is listening for more feedback

      Daniel Nashed  2 February 2018 23:05:01
      As posted before there is an issue with header encoding with umlauts which causes for example the subject to look scapbled.
      This issue is already SPRed and I got a mail that a hotfix is on the way (I have a PMR open).


      There are a couple of other issues development is working on. One is an issue I reported today about the Notes and Domino release version.

      The major Domino version is reported like this:


      9.0.1 FP10


      Release 900.0 QMR:11 QMU:0 Hotfix: 0 Fixpack: 0 (0)

      In contrast to

      9.0.1.FP9


      Release 9.0 QMR:1 QMU:9 Hotfix: 0 Fixpack: 0 (0)

      This is causing issues with some add-on applications where licenses are issued based on the major Domino version like iQSuite and others.


      Beside those issues that are already posted, there are some others which are investigated.


      Stay tuned and check the following
      post

      https://www.ibm.com/developerworks/community/blogs/LotusSupport/entry/Listening_to_your_feedback_on_Notes_Domino_9_0_1_FP10



      Update 03.02.2018 12:00


      I got a hotfix  9.0.1FP10HF47 for the subject conversion issue which I already installed.

      The first tests look good. The HF also contains a fix for the version number issue.


      Those fixes will be part of IF1. Stay tuned for more information.


      Release Information before the fix:

      Release 900.0 QMR:11 QMU:0 Hotfix: 0 Fixpack: 0 (0)


      After the fix:

      Release 9.0 QMR:1 QMU:9 Hotfix: 0 Fixpack: 0 (0)



      Update 05.02.2018 16:30

      The fix I got was just intended to fix the subject line issue. Not yet the build version issue.

      The build version should look like this with IF1 which is intended to be shipped this week.



      Local Notes/Domino Release 9.0 QMR:1 QMU:10 Hotfix: 0 Fixpack: 0 (0)

      Remote Notes/Domino Release 9.0 QMR:1 QMU:10 Hotfix: 0 Fixpack: 0 (0)

      Looks good for me now.





      Header/Subject Encoding issues after upgrading to 9.0.1 FP10

      Daniel Nashed  1 February 2018 13:09:04
      As Christian Henseler reported in the comments there is an issue with SMTP header encoding.
      Subjects that are UTF8 encoded are not converted.

      I can reproduce that on my server with different external mail accounts and also when using putty against a FP10 server. With FP9 it works well.

      They look like this:   =?utf-8?Q?Ã=84ü_Test_?=

      Until this problem is fixed you should not uprade to FP10!

      I have opened a PMR and also posted in the DP forum. L2 confirmed it looks like a regression.

      Stay tuned for updates.

      -- Daniel

      Notes/Domino 9.0.1 Feature Pack 10 shipped

      Daniel Nashed  31 January 2018 23:20:56

      Notes/Domino 9.0.1 Feature Pack 10 shipped.
      I updated my client and one of my production servers this morning. My upgrade went well. But as always I am interested in your feedback.

      When you install the admin/design client the result will be a single 1.8 JVM.
      Before we had a mix of JVM 1.8 for run-time and JVM 1.6 which made development more complicated.

      As you can read in the release notes beside the JVM version also the Eclipse version has been updated to work with JVM 1.8 and also to introduce new functionality.

      The compile time JVM is the biggest change in FP 10.

      The fixlist database is not updated yet but the readme already contains fixes.

      I have reverted my cipher list to the default and tested if the ROBOT SSL/TLS Attack is fixed.
      SSL Labs does not show any issues any more and the fix just comes in time before SSL Labs will give server not fixed against the  OBOT SSL/TLS Attack and using those vulnerable ciphers get a  "F" rating beginning of February.

      The browser stays removed. And there are still discussions among business partners who used embedded browser functionality in their solutions.


      http://www.lotus.com/ldd/fixlist.nsf/WhatsNew/86a6c4ba892f0218852581fc0067b4f4?OpenDocument

      ROBOT SSL/TLS Attack

      Daniel Nashed  17 January 2018 04:41:02
      This has not been widely discussed yet. But since SSL Labs will start reporting it with a rating of F beginning of February let me explain some background and what you could do.

      The issue has been there in a similar way before and is back. You can read the details here --> https://robotattack.org/


      Affected are the older ciphers that are not widely used by current browsers/client. You could disable those ciphers until the issue is fixed.

      But on the other side most browsers/clients do support higher secure ciphers. And because by default the server cipher order is used, a client should not choose a weaker cipher.

      In addition because of Secure Renegotiation which is supported by Domino and most browsers/clients support it, no weaker cipher will be used than the best common cipher between client and server.


      That means that only a very small fraction of connections might use those affected ciphers and if you disable those the client cannot connect at all.


      A fix for the ROBOT Attack is planned for FP10.

      So IMHO there is no need right now to disable those affected older RSA ciphers unless you have very high security requirements or if you are concerned about your SSL labs rating ..


      If you disable those affected ciphers the warning on the SSL Labs test side goes away.


      Here is a more paranoid configuration of TLS ciphers that you could use:


      set config SSLCipherSpecÀ30009FC02F009EC028006BC0140039C02700670033C013

      restart task http


      If you look into the compatibility report, there is no current client that could not connect any more (even older IE versions would connect).
      The other positive effect would be that you would only support DHE and ECDHE ciphers which is a good idea in general..


      UPDATE 17.01.2018

      Andy Brunner had an interesting comment. In my cipher list I am still having 0033 which is rated as a weak cipher which is not enabled by default.

      I have a cipher configuration database where I still had that cipher listed.


      TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x33)


      If you still have this cipher listed and did not allow weak ciphers the server will give you a hint:


      SSLDisableExportCiphers> Disabling weak cipher
      DHE_RSA_WITH_AES_128_CBC_SHA. Set notes.ini "USE_WEAK_SSL_CIPHERS=1" to re-enable.

      So the better fitting cipher liste in this case would be


      set config SSLCipherSpecÀ30009FC02F009EC028006BC0140039C0270067C013
      restart task http


      The mentioned cipher is rated as weak by Domino because it is a cipher that internally uses "SHA"


      Update: I almost forgot and got reminded about this Java 1.6 issue.
      The cipher is rated as weak for another reason. Older Java can only support this DHE cipher with 1024 bit.

      That's a longer story which you can find details here --> http://blog.nashcom.de/nashcomblog.nsf/dx/dha-with-more-than-1024-key-size-and-java-still-works.htm
      and another blog post here with some more details and ideas --> http://blog.nashcom.de/nashcomblog.nsf/dx/higher-crypt-standards-with-notesdomino-and-jvm-1.6.htm


      -- Daniel



      Image:ROBOT SSL/TLS Attack

      Meltdown and Spectre Exploit

      Daniel Nashed  5 January 2018 00:44:30

      There is a new security issue for most modern CPUs. Intel and AMD is affected in different ways.
      It's not something that is application specific. It's a CPU and OS level issue. Which affects also virtualization hosts.


      Here is the best website to get details -->
      https://meltdownattack.com

      And there are already some patches for some platforms.

      I have just installed the current kernel patches for CentOS (kernel 2.6.32-696.18.7).


      Here is the info from RHEL about the first patches
      https://access.redhat.com/errata/RHSA-2018:0008.

      We will probably see patches for other platforms including virtualization platforms like ESX.


      Those are the first fixes. And we will probably see more followup fixes.

      Update 06.01.2018:


      There is an interresting article describing some of the background and what hardware and software vendors are doing against it with different approaches.


      https://arstechnica.com/gadgets/2018/01/meltdown-and-spectre-heres-what-intel-apple-microsoft-others-are-doing-about-it/

      From what I see the applications with the biggest exposure to those bugs are web-browsers because they execute active code from remote (e.g. JavaScript).


      Here is also a current statement from Mozilla:


      https://blog.mozilla.org/security/2018/01/03/mitigations-landing-new-class-timing-attack/

      And there is the status page for Chrome and other Google technologis:



      https://support.google.com/faqs/answer/7622138#chrome


      -- Daniel

      DOMINO NETWORK PERFORMANCE OPTIMIZATION WINDOWS 2008 R2 VERSUS 2012

      Daniel Nashed  31 December 2017 10:19:26
      There is a new APAR which describes a performance issue on Windows 2008 and earlier.
      The APAR is based on a PMR which I had open with IBM. So I want give you the full detail about what we found out instead of the summary described in  

      LO93355: DOMINO NETWORK PERFORMANCE OPTIMIZATION WINDOWS 2008 R2 VERSUS 2012 -->
      https://www-01.ibm.com/support/entdocview.wss?uid=swg1LO93355

      We had a situation where we needed to replicate databases from an existing Domino 8.5.3 FP6 Windows 2008 environment to a new Domino 9.0.1 FP9 Linux 64 environment.  
      The replication was quite slow and we tried all kind of optimization on Domino, Windows and Linux.
      Beside increasing the sending/receiving buffer and memory we have been looking into ways to optimize the the Domino configuration.

      It turned out that Domino NRPC network compression was not always helpful depending on the configuration. So we ended up to disable network compression in our particular case. But this might not help in your configuration. It's something that needs testing.
      Between the Domino application sending the data and the IP-Stack sending the actual data there is a layer called the "NTI" layer which is responsible for actually coordinating sending the data.  

      The buffer size can not be modified and depending on the transaction higher latency networks take some time for the round-trip between sending and receiving side.
      But the main issue we have seen was that sending attachments which have a bigger chunk size that is send over the network was also slow.  


      Windows 2008 TCP/IP Issue
       


      The issue we found on the Windows IP stack only had impact in network environments which have higher latency than a local network where the latency is around 1 ms.  
      Our environment had 1 GBit and around 6 ms latency which is already great for a wide area connection . If you are having higher latency the performance might be even lower!
      We also reproduced the slow performance also with a faster connection with similar latency (10 GBit network with 5-6 ms latency). So it is the latency that has impact!  

      For transferring attachments with my own written C-API test application we have seen 2,5 MB/sec transferring data from Win2008.
      In contrast Windows 2012 did not have the same issue and that was very strange for us.  

      After discussion with the network team and a lot of tests we found the following tuning parameters.
      Both parameter do not exist by default on Win2008 R2 but the DefaultSendWindow exists for example on Win7 (which hare  comparable network stack) with a smaller value.  


      [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\AFD\Parameters]  
      "DefaultSendWindow"=dword:00080000  
      "DefaultReceiveWindow"=dword:00080000  


      Those settings ensures that the much more chunk sizes are send over the network before the IP-stack waits for the ACK from the other side. By default it was around 12 KB of data which was quite small!
      The first tests on our internal environment after the change showed 35 MB/sec!


      But that does not mean that normal replication will have the same performance because it is a mix of different transactions! We only tested object write transactions which had the biggest impact in our case.


      Object Write Chunk-Size is 256 KB
       


      In discussions with IBM we also found out that the documentation for changing the chunk size for sending attachment data was wrong.  

      The WIKI documentation say that the chunk size ist 64 KB and can be increased with a Notes.ini parameter up to 1 MB.  
      But it turns out that the parameter was only implemented as a test for a customer and the fix had never been added to the code.  


      Here is the technote describing Notes.ini SERVER_SEND_OBJECT_CHUNK_SIZE.  

      This is the only documentation for the parameter that should be corrected. The parameter does currently not exist and the default is 256 KBinstead of 64 KB.  


      https://www-10.lotus.com/ldd/dominowiki.nsf/dx/Optimising_NRPC_Bandwidth_Consumption_for_attachment
      I have been testing different chunk sizes between 64KB and 1 MB with a low level C-API application which writes attachments.
      And I found out that 256KB is a good balanced value. So there would be no need to change this parameter.

      Conclusion and some additional tips for AdminP


      When you are running on Win20012 or higher you don't need to change anything.  
      For Windows 2008 you should really set the registry parameter, because this will be a big boost for your replication performance.  

      On the other side the nature of replication is that document level replication will always take some time even in a local network.
      That's why Domino provides accelerated replica which uses a different transaction type.
      It's a kind of backup restore over the network. But that does only work if the database is not DAOS enabled.
      For DAOS enabled databases the replicator is used and it takes benefit of storage optimization.
      It will only send the attachment if it isn't yet on the remote side. But this might be still slower compared to an accelerated replica.

      To better utilize the bandwidth of your 1 GBit line we ended up having multiple AdminP threads leveraging the replicator code to push databases in parallel.
      There is ab enhancement in the 9.0.1 codestream (we got it backported to 8.5.3 FP6) which allows one process with multiple threads to replicate in parallel.


      And if you want AdminP to create the replica immediately instead of just creating a replica stub you need the following notes.ini parameter: ADMINP_EXCHANGE_ALL_UNREAD_MARKS=1.

      When you set this parameter Adminp actually pushes the database instead of creating a replica stub and also syncs all unread marks for the database.
      Note:  The admin4.nsf the request type will look like accelerated replica copy even DAOS is enabled on the database and the status of the request also looks a bit different.
      You should not be worried about that. It will use the normal replicator code including unread mark sync.

       

      Domino 9.0.1 FT Index Hang and potential crash

      Daniel Nashed  10 December 2017 06:08:37
      We ran into a hang situation multiple times during FT indexing. It turned out that this is a regression introduced in FP9 due to changes in the FT index area.

      In certain situations the FTIndex update does hang getting document data and will cause one CPU core to be maxed out for this thread.
      The description of the SPR says it is a "spike" but it more looks like the thread permanently uses CPU.

      This can happen with updall, DBMT and also other tasks updating the FT index.
      The process cannot be stopped and this will also cause that the server cannot be shutdown cleanly.

      We got a hotfix which will will be included in IF3. After applying the hotfix we had no new server hangs.

      I am including the call-stack for the hang to this blog post to have it searchable for others who might run into the same issue.
      If you have not installed FP9 you should wait for IF3. If you are on FP9 and run into this issue, take a full NSD, open a PMR and reference the mentioned SPR numbers to get the fix.

      -- Daniel


      -- Fixed SPRs --

      SPR #SVEM9SLCL7
      J3 server crashed on DBMT task, while full text indexing the database

      SPR #TDOOAT6LK9
      CPU spike when running dbmt (or updall/update task) and creating full text index.

      -- Call Stack --

      Thread 3 (Thread 0x7f2c5da71700 (LWP 17594)):
      #0  ODSToOrFromHost (toHost=32769, type=0, vbuffer=0x7f2c5da6e8e0, iterations=1) at ods.c:824
      #1  0x00007f2cba7ef8fe in ODSReadItem (src=0x7f2c54466d96, type=, dest=0x7f2c5da6e8e0) at ods.c:1420
      #2  0x00007f2cbab635e2 in GetChar(STREAM_CTX*, STREAM_DATA*) () from /opt/ibm/domino/notes/latest/linux/libnotes.so
      #3  0x00007f2cbab64932 in FTGetDocStream () from /opt/ibm/domino/notes/latest/linux/libnotes.so
      #4  0x00007f2c5d390919 in NotesStreamReadChar (arg=) at ftg_dstr.cpp:1412
      #5  0x00007f2cbab5ca7c in FTLexMatch () from /opt/ibm/domino/notes/latest/linux/libnotes.so
      #6  0x00007f2c5d39296c in FTGCreateIndex (pFTGCtx=0x7f2c4c00abf8) at ftg_dstr.cpp:1839
      #7  0x00007f2c5d38bac0 in CFTNoteIndexer::ProcessDoc(FTG_CTX *, struct {...} &) (this=, pFTGCtx=0x7f2c4c00abf8, docIndexerInfo=...) at ftgindex.cpp:2074
      #8  0x00007f2c5d38c5d1 in FTGIndexIDProc (Parameter=, NoteID=207326) at ftgindex.cpp:1685
      #9  0x00007f2cb999285d in IDEnumerate (hTable=536872571, Routine=0x7f2c5d38c343 , Parameter=0x7f2c4c00abf8) at idtable.c:2216
      #10 0x00007f2c5d38e252 in FTGIndex(FT_THREAD *, struct {...} *, WORD, char *) (pftt=0x7f2cb41004d0, pFTStreamCtx=0x7f2c4c00abf8, Options=392, StopFile=) at ftgindex.cpp:1146
      #11 0x00007f2cbab5adce in FTCallIndex () from /opt/ibm/domino/notes/latest/linux/libnotes.so
      #12 0x00007f2cbab5c3a3 in FTIndexExt2 () from /opt/ibm/domino/notes/latest/linux/libnotes.so
      #13 0x00007f2cb93e8485 in UpdateFullTextIndex (hDB=1154, Pathname=0x7f2cb4101648 "mail/c1/xn06451.nsf", Flags=201342976, fullTextStatus=8) at update.c:1239
      #14 0x00007f2cb93ea78f in UpdateCollectionsExt (_hModule=, Pathname=0x7f2cb4101648 "mail/c1/xn06451.nsf", Type=2, Flags=201342976, Flags2=0, mSecs=0, ViewNoteID=0, ContainerObjectID=0, ViewTitle=0x40a360 "", retDbTitle=0x0, fSrchSite=0, QueuedRequest=0, retbLater=0x0, fullTextStatus=8, wantsFulltext=0x0) at update.c:660
      #15 0x00007f2cb93ea957 in UpdateCollections (_hModule=32769, Pathname=0x0, Flags=, ViewNoteID=, ContainerObjectID=, ViewTitle=, retDbTitle=0x0, fSrchSite=0, QueuedRequest=0, retbLater=0x0, fullTextStatus=8, wantsFulltext=0x0) at update.c:106
      #16 0x0000000000405238 in UpdallThread (threadparam=) at dbmt.c:2108
      #17 0x00007f2cb98e7be3 in ThreadWrapper (Parameter=) at thread.c:1183
      #18 0x0000003aae007aa1 in start_thread () from /lib64/libpthread.so.0
      #19 0x0000003aadce8bcd in clone () from /lib64/libc.so.6

      End of Service for JVM 1.6

      Daniel Nashed  25 November 2017 13:34:30

      IBM uses the Oracle JVM as their base for their IBM JVM platform which is used in IBM products like Notes, Domin and Traveler.


      JVM 6.0 has been around for almost 10 years and is now discontinued since Sep 2017.
      Oracle discontinued their support for JVM 1.6 so IBM cannot support JVM 1.6 on their side.


      That also means for IBM platforms that there is no patch support for JVM 1.6!


      For Notes and Domino means you have to update to 9.0.1 FP8/FP10 for JVM 1.8 and hopefully FP10 will bring compile time for JVM 1.8 as well (current planning).


      If you are running on Notes/Domino 8.5.3 or an earlier 9.0.1 FP don't panic. Most Java applications on Domino are not directly accessible over the network. There is at least the Domino HTTP stack between the client and the Java application.

      On the client side you might have direct connection from the client to the internet. And for encrypted connections there have been limitations before in the SSL/TLS area as posted before.
      For example there is just very limited TLS 1.2 support in JVM 1.6 with just one chiper.


      I personally would still wait for Feature Pack 10 and have the full JVM 1.8 support also at compile time. But you should be aware that it is time to move to a current release.


      If you are on 9.0.1 you are just a "FP" install away. If you are on 8.5.3 there are another good reasons to move to a current 9.0.1 release from security point of view. For example missing SHA-256 support and no TLS 1.2 support - not just for JVM.



      Here is a link to the support cycle for the IBM JVM

      https://developer.ibm.com/javasdk/support/lifecycle/


      Daniel

      Traveler 9.0.1.20 Released

      Daniel Nashed  18 November 2017 14:44:58
      Traveler 9.0.1.20 has been released and I installed it already.
      As usual, if you are not waiting for an urgent open issue that is listed in the fix list, it might make sense to wait before installing a new version in production asap.
      I have installed it already befor the weekend and it looks good for my small environment.

      Beside the fixes listed below there is a new feature:
      • Support for invitee availability search from Calendar on Exchange ActiveSync clients.

      Still trying to test it. Not sure the iOS native calendar does support it.

      I did not have this on the radar and also never tested with the Verse app. Don't see it working here.
      Maybe someone has an idea?

      I did not testing but without luck on any of my clients.

      APAR # Abstract
      LO93044 Slow sync due to prime sync thread looping over large number of child documents.
      LO93067 Better handling of encrypted mail when syncing to mobile device.
      LO93070 Traveler cleanup bind command may fail when using MS SQL Server.
      LO93084 Better handling of Notes Doc Links when syncing to mobile devices.
      LO93196 Traveler "did not respond in time" messages on the console log.
      LO93217 Additional HTML to plain text conversion options to improve generated plain text content.
      LO93221 Do not include previous attachments on reply mails from MaaS Secure Mail client.
      LO93236 Improve crash prevention on Traveler server when processing documents.
      LO93238 Phone messages with HTML content may not display correctly on mobile device.
      LO93258 Traveler server may be unresponsive due to logging thread deadlock.
      LO93319 Support for Domino 9.0.1 FP10.


      Archives


      • [IBM Lotus Domino]
      • [Domino on Linux]
      • [Nash!Com]
      • [Daniel Nashed]