Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

ROBOT SSL/TLS Attack

Daniel Nashed  17 January 2018 04:41:02
This has not been widely discussed yet. But since SSL Labs will start reporting it with a rating of F beginning of February let me explain some background and what you could do.

The issue has been there in a similar way before and is back. You can read the details here --> https://robotattack.org/

Affected are the older ciphers that are not widely used by current browsers/client. You could disable those ciphers until the issue is fixed.
But on the other side most browsers/clients do support higher secure ciphers. And because by default the server cipher order is used, a client should not choose a weaker cipher.
In addition because of Secure Renegotiation which is supported by Domino and most browsers/clients support it, no weaker cipher will be used than the best common cipher between client and server.

That means that only a very small fraction of connections might use those affected ciphers and if you disable those the client cannot connect at all.

A fix for the ROBOT Attack is planned for FP10.

So IMHO there is no need right now to disable those affected older RSA ciphers unless you have very high security requirements or if you are concerned about your SSL labs rating ..

If you disable those affected ciphers the warning on the SSL Labs test side goes away.

Here is a more paranoid configuration of TLS ciphers that you could use:

set config SSLCipherSpecÀ30009FC02F009EC028006BC0140039C02700670033C013
restart task http

If you look into the compatibility report, there is no current client that could not connect any more (even older IE versions would connect).
The other positive effect would be that you would only support DHE and ECDHE ciphers which is a good idea in general..


-- Daniel




Image:ROBOT SSL/TLS Attack

Meltdown and Spectre Exploit

Daniel Nashed  5 January 2018 00:44:30

There is a new security issue for most modern CPUs. Intel and AMD is affected in different ways.
It's not something that is application specific. It's a CPU and OS level issue. Which affects also virtualization hosts.


Here is the best website to get details -->
https://meltdownattack.com

And there are already some patches for some platforms.

I have just installed the current kernel patches for CentOS (kernel 2.6.32-696.18.7).


Here is the info from RHEL about the first patches
https://access.redhat.com/errata/RHSA-2018:0008.

We will probably see patches for other platforms including virtualization platforms like ESX.


Those are the first fixes. And we will probably see more followup fixes.

Update 06.01.2018:


There is an interresting article describing some of the background and what hardware and software vendors are doing against it with different approaches.


https://arstechnica.com/gadgets/2018/01/meltdown-and-spectre-heres-what-intel-apple-microsoft-others-are-doing-about-it/

From what I see the applications with the biggest exposure to those bugs are web-browsers because they execute active code from remote (e.g. JavaScript).


Here is also a current statement from Mozilla:


https://blog.mozilla.org/security/2018/01/03/mitigations-landing-new-class-timing-attack/

And there is the status page for Chrome and other Google technologis:



https://support.google.com/faqs/answer/7622138#chrome


-- Daniel

DOMINO NETWORK PERFORMANCE OPTIMIZATION WINDOWS 2008 R2 VERSUS 2012

Daniel Nashed  31 December 2017 10:19:26
There is a new APAR which describes a performance issue on Windows 2008 and earlier.
The APAR is based on a PMR which I had open with IBM. So I want give you the full detail about what we found out instead of the summary described in  

LO93355: DOMINO NETWORK PERFORMANCE OPTIMIZATION WINDOWS 2008 R2 VERSUS 2012 -->
https://www-01.ibm.com/support/entdocview.wss?uid=swg1LO93355

We had a situation where we needed to replicate databases from an existing Domino 8.5.3 FP6 Windows 2008 environment to a new Domino 9.0.1 FP9 Linux 64 environment.  
The replication was quite slow and we tried all kind of optimization on Domino, Windows and Linux.
Beside increasing the sending/receiving buffer and memory we have been looking into ways to optimize the the Domino configuration.

It turned out that Domino NRPC network compression was not always helpful depending on the configuration. So we ended up to disable network compression in our particular case. But this might not help in your configuration. It's something that needs testing.
Between the Domino application sending the data and the IP-Stack sending the actual data there is a layer called the "NTI" layer which is responsible for actually coordinating sending the data.  

The buffer size can not be modified and depending on the transaction higher latency networks take some time for the round-trip between sending and receiving side.
But the main issue we have seen was that sending attachments which have a bigger chunk size that is send over the network was also slow.  


Windows 2008 TCP/IP Issue
 


The issue we found on the Windows IP stack only had impact in network environments which have higher latency than a local network where the latency is around 1 ms.  
Our environment had 1 GBit and around 6 ms latency which is already great for a wide area connection . If you are having higher latency the performance might be even lower!
We also reproduced the slow performance also with a faster connection with similar latency (10 GBit network with 5-6 ms latency). So it is the latency that has impact!  

For transferring attachments with my own written C-API test application we have seen 2,5 MB/sec transferring data from Win2008.
In contrast Windows 2012 did not have the same issue and that was very strange for us.  

After discussion with the network team and a lot of tests we found the following tuning parameters.
Both parameter do not exist by default on Win2008 R2 but the DefaultSendWindow exists for example on Win7 (which hare  comparable network stack) with a smaller value.  


[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\AFD\Parameters]  
"DefaultSendWindow"=dword:00080000  
"DefaultReceiveWindow"=dword:00080000  


Those settings ensures that the much more chunk sizes are send over the network before the IP-stack waits for the ACK from the other side. By default it was around 12 KB of data which was quite small!
The first tests on our internal environment after the change showed 35 MB/sec!


But that does not mean that normal replication will have the same performance because it is a mix of different transactions! We only tested object write transactions which had the biggest impact in our case.


Object Write Chunk-Size is 256 KB
 


In discussions with IBM we also found out that the documentation for changing the chunk size for sending attachment data was wrong.  

The WIKI documentation say that the chunk size ist 64 KB and can be increased with a Notes.ini parameter up to 1 MB.  
But it turns out that the parameter was only implemented as a test for a customer and the fix had never been added to the code.  


Here is the technote describing Notes.ini SERVER_SEND_OBJECT_CHUNK_SIZE.  

This is the only documentation for the parameter that should be corrected. The parameter does currently not exist and the default is 256 KBinstead of 64 KB.  


https://www-10.lotus.com/ldd/dominowiki.nsf/dx/Optimising_NRPC_Bandwidth_Consumption_for_attachment
I have been testing different chunk sizes between 64KB and 1 MB with a low level C-API application which writes attachments.
And I found out that 256KB is a good balanced value. So there would be no need to change this parameter.

Conclusion and some additional tips for AdminP


When you are running on Win20012 or higher you don't need to change anything.  
For Windows 2008 you should really set the registry parameter, because this will be a big boost for your replication performance.  

On the other side the nature of replication is that document level replication will always take some time even in a local network.
That's why Domino provides accelerated replica which uses a different transaction type.
It's a kind of backup restore over the network. But that does only work if the database is not DAOS enabled.
For DAOS enabled databases the replicator is used and it takes benefit of storage optimization.
It will only send the attachment if it isn't yet on the remote side. But this might be still slower compared to an accelerated replica.

To better utilize the bandwidth of your 1 GBit line we ended up having multiple AdminP threads leveraging the replicator code to push databases in parallel.
There is ab enhancement in the 9.0.1 codestream (we got it backported to 8.5.3 FP6) which allows one process with multiple threads to replicate in parallel.


And if you want AdminP to create the replica immediately instead of just creating a replica stub you need the following notes.ini parameter: ADMINP_EXCHANGE_ALL_UNREAD_MARKS=1.

When you set this parameter Adminp actually pushes the database instead of creating a replica stub and also syncs all unread marks for the database.
Note:  The admin4.nsf the request type will look like accelerated replica copy even DAOS is enabled on the database and the status of the request also looks a bit different.
You should not be worried about that. It will use the normal replicator code including unread mark sync.

 

Domino 9.0.1 FT Index Hang and potential crash

Daniel Nashed  10 December 2017 06:08:37
We ran into a hang situation multiple times during FT indexing. It turned out that this is a regression introduced in FP9 due to changes in the FT index area.

In certain situations the FTIndex update does hang getting document data and will cause one CPU core to be maxed out for this thread.
The description of the SPR says it is a "spike" but it more looks like the thread permanently uses CPU.

This can happen with updall, DBMT and also other tasks updating the FT index.
The process cannot be stopped and this will also cause that the server cannot be shutdown cleanly.

We got a hotfix which will will be included in IF3. After applying the hotfix we had no new server hangs.

I am including the call-stack for the hang to this blog post to have it searchable for others who might run into the same issue.
If you have not installed FP9 you should wait for IF3. If you are on FP9 and run into this issue, take a full NSD, open a PMR and reference the mentioned SPR numbers to get the fix.

-- Daniel


-- Fixed SPRs --

SPR #SVEM9SLCL7
J3 server crashed on DBMT task, while full text indexing the database

SPR #TDOOAT6LK9
CPU spike when running dbmt (or updall/update task) and creating full text index.

-- Call Stack --

Thread 3 (Thread 0x7f2c5da71700 (LWP 17594)):
#0  ODSToOrFromHost (toHost=32769, type=0, vbuffer=0x7f2c5da6e8e0, iterations=1) at ods.c:824
#1  0x00007f2cba7ef8fe in ODSReadItem (src=0x7f2c54466d96, type=, dest=0x7f2c5da6e8e0) at ods.c:1420
#2  0x00007f2cbab635e2 in GetChar(STREAM_CTX*, STREAM_DATA*) () from /opt/ibm/domino/notes/latest/linux/libnotes.so
#3  0x00007f2cbab64932 in FTGetDocStream () from /opt/ibm/domino/notes/latest/linux/libnotes.so
#4  0x00007f2c5d390919 in NotesStreamReadChar (arg=) at ftg_dstr.cpp:1412
#5  0x00007f2cbab5ca7c in FTLexMatch () from /opt/ibm/domino/notes/latest/linux/libnotes.so
#6  0x00007f2c5d39296c in FTGCreateIndex (pFTGCtx=0x7f2c4c00abf8) at ftg_dstr.cpp:1839
#7  0x00007f2c5d38bac0 in CFTNoteIndexer::ProcessDoc(FTG_CTX *, struct {...} &) (this=, pFTGCtx=0x7f2c4c00abf8, docIndexerInfo=...) at ftgindex.cpp:2074
#8  0x00007f2c5d38c5d1 in FTGIndexIDProc (Parameter=, NoteID=207326) at ftgindex.cpp:1685
#9  0x00007f2cb999285d in IDEnumerate (hTable=536872571, Routine=0x7f2c5d38c343 , Parameter=0x7f2c4c00abf8) at idtable.c:2216
#10 0x00007f2c5d38e252 in FTGIndex(FT_THREAD *, struct {...} *, WORD, char *) (pftt=0x7f2cb41004d0, pFTStreamCtx=0x7f2c4c00abf8, Options=392, StopFile=) at ftgindex.cpp:1146
#11 0x00007f2cbab5adce in FTCallIndex () from /opt/ibm/domino/notes/latest/linux/libnotes.so
#12 0x00007f2cbab5c3a3 in FTIndexExt2 () from /opt/ibm/domino/notes/latest/linux/libnotes.so
#13 0x00007f2cb93e8485 in UpdateFullTextIndex (hDB=1154, Pathname=0x7f2cb4101648 "mail/c1/xn06451.nsf", Flags=201342976, fullTextStatus=8) at update.c:1239
#14 0x00007f2cb93ea78f in UpdateCollectionsExt (_hModule=, Pathname=0x7f2cb4101648 "mail/c1/xn06451.nsf", Type=2, Flags=201342976, Flags2=0, mSecs=0, ViewNoteID=0, ContainerObjectID=0, ViewTitle=0x40a360 "", retDbTitle=0x0, fSrchSite=0, QueuedRequest=0, retbLater=0x0, fullTextStatus=8, wantsFulltext=0x0) at update.c:660
#15 0x00007f2cb93ea957 in UpdateCollections (_hModule=32769, Pathname=0x0, Flags=, ViewNoteID=, ContainerObjectID=, ViewTitle=, retDbTitle=0x0, fSrchSite=0, QueuedRequest=0, retbLater=0x0, fullTextStatus=8, wantsFulltext=0x0) at update.c:106
#16 0x0000000000405238 in UpdallThread (threadparam=) at dbmt.c:2108
#17 0x00007f2cb98e7be3 in ThreadWrapper (Parameter=) at thread.c:1183
#18 0x0000003aae007aa1 in start_thread () from /lib64/libpthread.so.0
#19 0x0000003aadce8bcd in clone () from /lib64/libc.so.6

End of Service for JVM 1.6

Daniel Nashed  25 November 2017 13:34:30

IBM uses the Oracle JVM as their base for their IBM JVM platform which is used in IBM products like Notes, Domin and Traveler.


JVM 6.0 has been around for almost 10 years and is now discontinued since Sep 2017.
Oracle discontinued their support for JVM 1.6 so IBM cannot support JVM 1.6 on their side.


That also means for IBM platforms that there is no patch support for JVM 1.6!


For Notes and Domino means you have to update to 9.0.1 FP8/FP10 for JVM 1.8 and hopefully FP10 will bring compile time for JVM 1.8 as well (current planning).


If you are running on Notes/Domino 8.5.3 or an earlier 9.0.1 FP don't panic. Most Java applications on Domino are not directly accessible over the network. There is at least the Domino HTTP stack between the client and the Java application.

On the client side you might have direct connection from the client to the internet. And for encrypted connections there have been limitations before in the SSL/TLS area as posted before.
For example there is just very limited TLS 1.2 support in JVM 1.6 with just one chiper.


I personally would still wait for Feature Pack 10 and have the full JVM 1.8 support also at compile time. But you should be aware that it is time to move to a current release.


If you are on 9.0.1 you are just a "FP" install away. If you are on 8.5.3 there are another good reasons to move to a current 9.0.1 release from security point of view. For example missing SHA-256 support and no TLS 1.2 support - not just for JVM.



Here is a link to the support cycle for the IBM JVM

https://developer.ibm.com/javasdk/support/lifecycle/


Daniel

Traveler 9.0.1.20 Released

Daniel Nashed  18 November 2017 14:44:58
Traveler 9.0.1.20 has been released and I installed it already.
As usual, if you are not waiting for an urgent open issue that is listed in the fix list, it might make sense to wait before installing a new version in production asap.
I have installed it already befor the weekend and it looks good for my small environment.

Beside the fixes listed below there is a new feature:
  • Support for invitee availability search from Calendar on Exchange ActiveSync clients.

Still trying to test it. Not sure the iOS native calendar does support it.

I did not have this on the radar and also never tested with the Verse app. Don't see it working here.
Maybe someone has an idea?

I did not testing but without luck on any of my clients.

APAR # Abstract
LO93044 Slow sync due to prime sync thread looping over large number of child documents.
LO93067 Better handling of encrypted mail when syncing to mobile device.
LO93070 Traveler cleanup bind command may fail when using MS SQL Server.
LO93084 Better handling of Notes Doc Links when syncing to mobile devices.
LO93196 Traveler "did not respond in time" messages on the console log.
LO93217 Additional HTML to plain text conversion options to improve generated plain text content.
LO93221 Do not include previous attachments on reply mails from MaaS Secure Mail client.
LO93236 Improve crash prevention on Traveler server when processing documents.
LO93238 Phone messages with HTML content may not display correctly on mobile device.
LO93258 Traveler server may be unresponsive due to logging thread deadlock.
LO93319 Support for Domino 9.0.1 FP10.


    VIEW_REBUILD_DIR changed to /dev/shm/view_rebuild

    Daniel Nashed  14 November 2017 14:35:22
    We just discovered an interesting configuration issue, which generates quite some logging and is a bit annoying.
    When you specify the view_rebuild_dir without the trailing slash / back-slash, the server will internally append the slash.

    But if you configured the view_rebuild_dir in the config document without the (back) slash the server will tell you every couple of minutes that the server changed the setting.

    This happens why the internal path is always stored with the trailing (back) slash and the notes.ini check to update the parameter compares against the config doc entry without the (back) slash.

    VIEW_REBUILD_DIR changed to /dev/shm/view_rebuild

    So you have a constant changing parameter -- even it looks the same in the notes.ini

    The correct notes.ini entry would be

    VIEW_REBUILD_DIR changed to /dev/shm/view_rebuild/

    including the trailing slash.

    This avoids the log messages.

    -- Daniel

    Erster DNUG "Domino Next" Event 23.11.2017

    Daniel Nashed  9 November 2017 13:44:18
    Nach den Ankündigungen zu Domino 10, der IBM & HCL Kooperation und Domino 2025 haben wir die Agenda für den DNUG Domino Day am 23.11.2017 in Düsseldorf umgestellt.

    Teil des Domino Days wird ein Domino Next Feedback Richtung Domino 10 und beyond.

    Neben Uffe Sorensen wir auch ein Kollege von HCL mit dabei sein.

    Es wird  im Vortrag von Uffe u.A. um die aktuellen Informationen zur Kooperation gehen.
    Ihr könnt alle eure Fragen mitbringen und im "Feedback" Teil geht es dann um Feedback für Wünsche für Domino 10 und beyond.

    Dieser Teil ist bewußt ans Ende gelegt, damit genügend Zeit für Fragen ist ...


    Ich bin sehr gespannt.


    PS: Der Event ist für DNUG Mitglieder kostenlos! Gegen eine Gebühr können auch nicht Mitglieder teilnehmen...

     

    https://www.eventbrite.de/e/dnug-fachgruppentag-domino-day-fachgruppe-verse-und-notesdomino-tickets-35785282744



    09:00 - 09:10
    Begrüßung
    Daniel Nashed - CEO (Nash!Com)
    Manfred Lenz - Technical Sales Professional, IBM Collaboration and Talent Solutions (IBM Software Sales)

    09:10-10:05
    UPDATE: IBM Notes/Domino Feature Packs
    Daniel Nashed - CEO (Nash!Com)

     
    10.05-11:05
    ApplicationInsights & IBM Domino Doublecheck - Der Weg um richtige Entscheidungen treffen zu können
    Christoph Adler - Senior Consultant (panagenda)

    11:05-12:00
    SSL Zertifikate unter Domino - Allgemeine Einführung und Vorstellung der kostenfreien CA Let's Encrypt
    Detlev Poettgen - Geschäftsführer (midpoints)

    12:00-13:00
    Mittagspause


    13:00-13:45
    Domino Application Cloud (DAC) & Domino on Docker
    Michael Finkenbrink - Certified Senior Architect, IBM Collaboration Solutions (IBM Software Services)

    13:45-14:15
    Kaffeepause

    14:15-15:15
    Keynote: IBM Notes/Domino and Verse On-Premises - News/Strategy/Roadmaps incl. Notes Domino 10
    Uffe Sorensen - Messaging & Collaboration Director (IBM Software Sales)
    n.n. - HCL Industries

    15:15 - 17:00
    Notes/Domino Next Feedback" - Workshop in Form eines Knowledge Cafes zum Thema "Anforderungen an Verse Notes Domino 10 und danach
    Uffe Sorensen - Messaging & Collaboration Director (IBM Software Sales)
    Peter Schütt - Leader IBM Collaboration Solutions Strategy D-A-CH (IBM Software Sales)
    Manfred Lenz - Technical Sales Professional, IBM Collaboration and Talent Solutions (IBM Software Sales)


    Domino on Linux Start Script 3.1.3 with changed way to request it

    Daniel Nashed  30 October 2017 05:43:05
    Just updated the start script to a new version with some minor changes.
    There was one issue with systemd on shutdown and I made a change in the way config files are used.

    Most of the new features are coming in either thru projects or when I want something for my own environment.
    I don't get much feedback or feature requests beside that.

    One change triggered by a project was how config files apply. We wanted to use the same configuration for all servers.
    But we wanted special settings for the Traveler servers. So I changed the way the config files apply.
    Now you can use a general config and additional or changed parameters for individual servers.
    That way you can have a general config that you deploy automatically and you keep a server specific file with changes.

    So in that case the general config would be /etc/sysconfig/rc_domino_config.
    And the specific config would be for example: /etc/sysconfig/rc_domino_config_notes.

    This would also work in partitioned environments where each server has a basic configuration and you want additional parameters for a partition.
    On the other side even on partitioned servers you could use variables which depend on variables like the DOMINO_USER.


    Changed way to request the start script

    I am also changing the way you can request the new version. Until now I had a request form.
    Now you just send a mail to dominostartscript  at nashcom.de with the subject "script".

    The old implementation was a servertask which read the data posted in database.
    I switched to a pre-delivery agent with some additional logic to check the message.
    So for example I am not triggering an automatic reply if the message is a reply or is an autosubmitted message.

    I am not yet updating the start script page and want to see first how this works with requests coming in thru the blog.
    And I hope you like the new way to request the start script? Any feedback is welcome.

    -- Daniel



    --------------
    Change History
    --------------

    V3.1.3 30.10.2017

    Problems Solved
    ---------------

    Fixed an issue with systemd in combination with server controller.
    Now the server controller correctly shutsdown when the service is stopped


    New Features
    ------------

    listini -- displays server's notes.ini

    Changes
    -------

    Changed sample rc_domino_config_notes setting DOMINO_PRE_SHUTDOWN_COMMAND to "tell traveler shutdown"


    V3.1.2 01.09.2017

    New Features
    ------------

    New check if Domino ".res" files exist and readable to generate warnings

    New short cut command "res" for "resources"

    Changes
    -------

    In previous version either the server specific config file was used or the default config file.

    The config files are now used in the following order to allow more flexible configurations:

    - First the default config-file is loaded if exists (by default: /etc/sysconfig/rc_domino_config)
    - In the next step the server specifc config-file (by default: /etc/sysconfig/rc_domino_config_notes) is included.
    - The server specific config file can add or overwrite configuration parameters.

    This allows very flexible configurations. You can specify global parameters in the default config file and have specific config files  per Domino partition.
    So you can now use both config files in combination or just one of them.

    Great news Notes Domino 10 and beyond

    Daniel Nashed  25 October 2017 20:03:05
    We got great news today. There will be a Notes & Domino 10 in 2018. And IBM also announced that with joined efforts with HCL Technologies they are working on a strategy for #Domino2025.

    This isn't a new partnership. IBM and HCL are already working together for Tivoli and Rational software. But is was still a big surprise today.

    Notes and Domino 9.0.1 FP10 is committed for 2017 and I am looking forward to hear more about the Notes & Domino Strategy planned for 2018 with Notes and Domino 10.

    And I am really looking forward actively giving feedback for future directions that IBM is asking customers and partners for.


    See details in this official links


    https://www.ibm.com/blogs/social-business/2017/10/25/ibm-announces-investment-notes-domino-version-10-beyond/

    https://www.ibm.com/social-business/us-en/announce/domino-jam2025/


    What a coincident. Tomorrow I am part of a Notes & Domino strategy workshop for a customer and I bet some of the slides of one of the co-speakers need some updates tonight :-)




    Image:Great news Notes Domino 10 and beyond

    Archives


    • [IBM Lotus Domino]
    • [Domino on Linux]
    • [Nash!Com]
    • [Daniel Nashed]