Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

Apple App Transport Security

Daniel Nashed  22 July 2015 09:50:30
Apple is introducing a new standard for their next OS versions.
App Transport Security (ATS) is planned for iOS 9 and OS X 10.11.


The current plan is to only support
  • TLS 1.2
  • >= 2048 bit RSA
  • SHA-256 signed web server certificates
  • ECDHE!!


TLS 1.2 is a good idea, 2048 RSA keys are a good idea and SHA-256 is also a good idea because SHA-1 is rated as insecure.


The general requirement for PFS ciphers  
(https://en.wikipedia.org/wiki/Forward_secrecy) is a good idea from security point of view.
But not everyone is supporting ECDHE (Elliptic curve Diffie–Hellman). The normal DHE Ciphers should be perfectly be OK from security point of view.


Maybe Apple is just allowing ECHDE because they have less overhead compared to the normal DHE Ciphers.

On the other side if ECDHE ciphers would be compromised in any way this would leave us with no supported cipher suite at all for communication.


Usually the server is responsible for the order in which ciphers are selected. There are server settings (like in current Domino 9.0.1 versions) to allow the client to select the cipher order.

So in general having a short cipher list with only secure ciphers is a good idea to really ensure that a strong cipher is selected!


But that will leave out many applications and will put a lot of pressure on many vendors and also on administrators implementing the latest software versions on server side.


As an app developer you can change you application to allow less secure TLS versions and ciphers.
But if you are running a server and the application is build against a newer API without those exceptions you will have to provide this strong security standard.


See this link for details --> https://developer.apple.com/library/prerelease/ios/technotes/App-Transport-Security-Technote

The Domino 8.5.x stack will not support TLS 1.2 and and SHA-256 because the code base does not include and SHA-256 support.


But even the current Domino 9.0.1 FP4 version does not completely comply with ATS. DHE is supported in the current Domino FPs and can be configured which would be a vaild and good PFS cipher. But that is not on the ATS list.
There is currently no support for ECDHE in native Domino.


So I am interested to see the feedback from software companies on this Apple move.


On the other side there are Apple servers not complying to those standards and we are still having issues with some Apple SMTP Servers using SSLV2Hello.

It's going to be interesting again to see what will happen when a vendor like Apple pushes standards so hard and in such a short time.


-- Daniel




Crash on iNotes when applying IF1 or the new 9.0.1 FP4 version

Daniel Nashed  20 July 2015 06:05:44
One of my customers and another partner reported a new crash when applying 9.0.1 FP4 IF1.
They both reported the exact same call-stack both running on Linux. I have no details yet but given the fact that there are two independent crash reports with the same call-stack this might be a more general issue.


I am waiting for more information and will update you ASAP once I hear anything new.
For now I would stay on the last IF of FP3 until we know what is happening.


Enclosed you find the call-stack for reference.

Update 13.08.2015:

IBM is working on a fix. One of my customers got a hotfix. It is not clear what is exactly broken.
But we also found some other details and a work-around.

It turns out that IBM moved from Dojo 1.5.2 to 1.5.4 in FP4.
The file owner is not set correctly in those new Dojo files. I have one Linux customer and one AIX customer where fixing the file-permissions did help to avoid the crash and also to get other functionality in XPages and other applications working again.

The permissions looked like this:

/local/notesdata/domino/js/dojo-1.5.4

[root@mail dojo-1.5.4]# ll
total 20
drwxr-xr-x. 12 10537 6001 4096 Jun  8 11:10 dijit
drwxr-xr-x. 13 10537 6001 4096 Jun  8 11:11 dojo
drwxr-xr-x. 53 10537 6001 4096 Jun  8 11:11 dojox
drwxr-xr-x.  5 10537 6001 4096 Jun  8 11:11 dwa
drwxr-xr-x.  5 10537 6001 4096 Jun  8 11:11 ibm

Changing them via chown -R notes:notes /local/notesdata/domino/js/dojo-1.5.4 did help in our case.
But the hotfix my Linux customer got, did fix it in some other way, because after applying the hotfix the file owners have been still wrong.


-- Daniel





Thread 44 (Thread 0x7fa2b8429700 (LWP 14306)):

#0  0x00007fa354c1c063 in select () from /lib64/libc.so.6

#1  0x00007fa355d6db47 in FRDoSleep (secs=, usecs=) at cleanup.c:986

#2  0x00007fa355d6e812 in OSRunExternalScript (

   passed_script=0x7fa2b841b340 "\"/export/opt/ibm/domino/notes/latest/linux/nsd.sh\" -batch -crashpid 12669 -crashtid 3091371776", flags=) at cleanup.c:4037

#3  0x00007fa355d6fba3 in OSFaultCleanupExt (action2take=0, CleanupScriptExecFlag=,
   iniFileName=0x0, szProcess=, Length=, CrashedPID=0x0) at cleanup.c:1574

#4  0x00007fa355d6ffaf in OSFaultCleanup (action2take=0, CleanupScriptExecFlag=0, iniFileName=0x0)

   at cleanup.c:1322

#5  0x00007fa355d3d9c0 in fatal_error (signl=11, info=, context=) at break.c:2519

#6  0x00007fa3006c4438 in jsig_handler ()

  from /export/opt/ibm/domino/notes/latest/linux/jvm/lib/amd64/default/libjsig.so

#7  0x00007fa30021132f in masterSynchSignalHandler ()

  from /export/opt/ibm/domino/notes/latest/linux/jvm/lib/amd64/default/libj9prt24.so

#8  

#9  0x00007fa354bafb32 in fgets () from /lib64/libc.so.6

#10 0x00007fa35433fa0f in Haiku::GetLastModified (this=, pNote=,
   argc=, argv=, argl=, rethResult=0x7fa2b841f22c,
   retResultLength=0x7fa2b841f228) at haiku/haiku.cpp:17170

#11 0x00007fa35430a864 in Haiku::AtFuncDispatch::ExecuteDbCommand (this=, pHaiku=,
   note=0x7fa2b8423480, index=, argc=1421540096, argv=0x7fa354ebbef8 ,
   argl=0x7fa2b8421340, bIsJsData=1, bIsHTML=0) at haiku/haiku.cpp:32883

#12 0x00007fa35430ab7e in Haiku::ExecuteDbCommand (this=, note=,
   nCmd=, argc=, argv=, argl=, bIsJsData=1,
   bIsHTML=0) at haiku/haiku.cpp:4731

#13 0x00007fa3543923e9 in HuDocNote::AddHaikuDbCommand (this=0x7fa2b8423480, iCmd=89, args=..., bIsJsData=1,
   bIsHTML=0) at haiku/HuDocNote.cpp:5549

#14 0x00007fa354481eb7 in ShBuiltInNameSpaceTag::Write (this=0x2c27148, formStream=0x7fa2b84235c0, layoutBody=...)

   at haiku/ShBodyParts.cpp:904

#15 0x00007fa3543e66b7 in HuLayout::WriteContents (this=0x7fa2b48ccdd8, formStream=0x7fa2b84235c0)

   at haiku/HuLayout.cpp:285

#16 0x00007fa354396727 in HuDocNote::GenerateHTML (this=0x7fa2b8423480, html=...) at haiku/HuDocNote.cpp:2589

#17 0x00007fa35433bad7 in Haiku::GenerateHtml (this=0x7fa2b8423470) at haiku/haiku.cpp:3964

#18 0x00007fa354373fd6 in Haiku::HandleDominoCmd (this=0x7fa2b8423470, cmd=...) at haiku/HandleOpenDoc.cpp:192

#19 0x00007fa35433ec30 in Haiku::HandleCmd (cmd=0x7fa2b48b6dd8, cmdHandler=...) at haiku/haiku.cpp:3441

#20 0x00007fa354144fbc in CmdHandlerBase::PrivHandle (this=0x7fa3026cc038, cmd=0x7fa2b48b6dd8, cachedCmd=0x0)

   at cmdhandb.cpp:129

#21 0x00007fa354144037 in CmdHandler::PrivHandle (this=0x7fa3026cc038, cmd=0x7fa2b48b6dd8) at cmdhand.cpp:102

#22 0x00007fa354143ca2 in CmdHandler::Handler (cmd=0x7fa2b48b6dd8, data=) at cmdhand.cpp:153

#23 0x00007fa354135cf5 in Cmd::Execute (this=0x7fa2b841eaa0) at cmd.cpp:1166

#24 0x00007fa3541afe68 in InotesHTTPProcessRequestImpl (ihReq=0x7fa2b54c0f88) at inotesif.cpp:2488

#25 0x00007fa3541b050e in InotesHTTPProcessRequest (ihReq=0x7fa2b841eaa0) at inotesif.cpp:2053

#26 0x00007fa3592114f3 in HTInotesRequest::ProcessRequest (this=0x7fa2b54c0f70) at htinotes.cpp:1254

#27 0x00007fa35920946b in HTRequestExtContainer::ProcessRequest (this=0x7fa2b54c0b08, appSpace=)

   at htextcon.cpp:1262

#28 0x00007fa35922487d in HTRequest::ProcessRequest (this=0x7fa2b54c0878) at htrequst.cpp:1880

#29 0x00007fa35922dc20 in HTSession::StartRequest (this=0x7fa2b54d61b0) at htsesson.cpp:620

#30 0x00007fa359239276 in HTWorkerThread::CheckForWork (this=0x7fa2b4ae7de8) at htwrkthr.cpp:226

#31 0x00007fa35923949b in HTWorkerThread::ThreadMain (this=0x7fa2b4ae7de8) at htwrkthr.cpp:90

#32 0x00007fa359233331 in HTThreadBeginProc (arg=0x7fa2b4ae7de8) at htthread.cpp:39

#33 0x00007fa355d65383 in ThreadWrapper (Parameter=) at thread.c:1155

#34 0x00007fa3558617b6 in start_thread () from /lib64/libpthread.so.0

#35 0x00007fa354c22d6d in clone () from /lib64/libc.so.6

#36 0x0000000000000000 in ?? ()

Crash after applying 9.0.1 FP4

Daniel Nashed  7 July 2015 06:33:52
I am working with IBM support since I installed FP4 directly after it shipped.
After installing FP4 I got a crash on startup. I first thought this is special to my environment and IBM support was blaming my unsupported CentOS 6.5 environment.


But it turned out that there was already a SPR # LKIM9UPQBL which has been already escalated to development. So it sounded like a more general issue that can happen in some configurations.


The bug has been reproduced on one of my customers with SLES 11 SP3 and I heard that other partners have been running into this also in their test environments.


I am waiting for more information from IBM. If you are planning to upgrade to FP4 you should wait until we get more details.

Here is an example call-stack that hopefully makes it into Google soon to have public information available for this call-stack.

Update: The problem is a regression. IBM did not add the server binary to the install kits.
Unix and Windows is affected even it might not cause a crash in every server configuration.
In my case the crash happened directly after server start. But to be sure you should install the IF or make sure you are installting the updated FP4 installer.

For more details check this technote -->
http://www.ibm.com/support/docview.wss?uid=swg21961701

-- Daniel



StaticHang = Virtual Thread [  server:12773: 147] (Native thread [  server:12773:3921537904]) (0x31e5/0x93/0xe9bdeb70)

Thread 60 (Thread 0xe9bdeb70 (LWP 13566)):
#0  0xf77b9430 in __kernel_vsyscall ()
#1  0x00555f81 in select () from /lib/libc.so.6
#2  0xf50aa2bf in FRDoSleep (secs=1, usecs=0) at cleanup.c:986
#3  0xf50ab0b0 in OSRunExternalScript (passed_script=0xe9bdc78c "\"/opt/ibm/domino/notes/latest/linux/nsd.sh\" -batch -crashpid 12773 -crashtid 3921537904", flags=1) at cleanup.c:4037
#4  0xf50ac6d0 in OSFaultCleanupExt (action2take=0, CleanupScriptExecFlag=4096, iniFileName=0x0, szProcess=0x0, Length=0, CrashedPID=0x0) at cleanup.c:1574
#5  0xf50acb9c in OSFaultCleanup (action2take=0, CleanupScriptExecFlag=4096, iniFileName=0x0) at cleanup.c:1322
#6  0xf50742b8 in fatal_error (signl=11, info=0xe9bdcb1c, context=0xe9bdcb9c) at break.c:2519
#7  
#8  0xf52ee99c in
SECFreeSSOInternetSitesConfig () from /opt/ibm/domino/notes/latest/linux/libnotes.so
#9  0x08119dc6 in
ServerFreeSortedSitesList (pmhSitesList=0xf2f85164, dwNumActiveHosts=2) at svsso.c:3031
#10 0x0811a022 in
SetStaticInternetSiteSSOConfig (dwNumAllocedEntries=100, dwNumActiveHosts=2, mhSitesList=2147615269) at svsso.c:2899
#11 0x0811b990 in UpdateStaticServerSSOConfigInfo (bUpdateSitesInfoOnly=1, bInternetSitesEnabled=1, bSSOServerEnabledFromServerDoc=0, dwSSOConfigLen=0, pSSOConfig=0x816db44 "", bIdpcatConfigExists=1) at svsso.c:1901
#12 0x0811bb33 in CheckServerSSOISitesConfigInfo () at svsso.c:2118
#13 0x08078b9e in PollTask (TaskId=..., VarBlock=...) at poll.c:1215
#14 0x08073cef in Scheduler (vArgumentPtr=0x0) at sched.c:339
#15 0xf50a07f1 in ThreadWrapper (Parameter=0x0) at thread.c:1155
#16 0x0061ab39 in start_thread () from /lib/libpthread.so.0
#17 0x0055dc2e in clone () from /lib/libc.so.6

IBM Notes Traveler 9.0.1.6 released with some important fixes

Daniel Nashed  2 July 2015 00:33:26
IBM Traveler 9.0.1.6 ships a couple of importan APAR fixes for the IBM Traveler

Some of the fixes solve problems in MIME & attachment handling which have been introduced in the last releases when the new MIME handling has been introduced.

Fixlist:

APAR #         Component         Abstract
LO84879         Server         Calendar notice may be sent multiple times or be sent by the server ID.
LO85144         Server         E-mail containing invalid zero character in WBXML encoding may not sync correctly to mobile device.
LO85222         Server         Attachment with an unknown content type may not download to device.
LO85237         Server         Proxy credentials may not be removed from notes.ini during startup.
LO85260         Server         When Trash sync first enabled, sync only today and later trash items to improve performance.
LO85283         Server         Mime format e-mail may sync to device without the body.
LO85357         Server         Attachment with forward slash in file name may not sync to mobile device.
LO85444         Server         Web Admin may not show data for a user and will recieve "Could not generated devicetype" error message.
LO85445         Server         Attachment with multiple dot characters in file name may not sync to mobile device.
LO85477         Server         On standalone server auto cleanup could impact security records then requiring re-approval if approval is enabled.

Here is the download link --> http://www.ibm.com/support/fixcentral/swg/selectFixes?parent=ibm~Lotus&product=ibm/Lotus/Lotus+Notes+Traveler&release=All&platform=All&function=all

You should really consider updating your server if you are on 9.0.1.4 or 9.0.1.5.

-- Daniel

IBM Verse Client for iOS shipped

Daniel Nashed  30 April 2015 09:24:40
Finally the IBM Verse App for iOS is released

https://itunes.apple.com/de/app/ibm-verse/id949952976

Image:IBM Verse for iOS app - Available in AppStore


You can either use it to access the IBM Connections Cloud or Traveler On-Premise environments.
Currently you can only use one account against either On-Premise or the cloud.

Take care that the first Traveler release supporting the client is 9.0.1.3 but you should install the latest 9.0.1.4 version.

The Verse client is a container app. You can still continue to use ActiveSync with the integrated apps.
It's not a replacement. Both ways to access the Traveler server are fully supported.

If you want a container app, IBM Verse is a good option for you but you should be aware that contacts and calendar cannot be accessed outside the IBM Verse app.
What I really like is the notifications that you get via Apple push notifications.

For testing I am currently using both in parallel and get the best of both worlds.
But in normal environments you should decide for one way to access your Traveler data.


IBM published an FAQ (part of the Traveler Documentation):
http://www.ibm.com/support/knowledgecenter/?lang=en#!/SSYRPW_9.0.1/iOSVerseIntro.html

Traveler 9.0.1.4 shipped

Daniel Nashed  29 April 2015 10:42:56


IBM has released the Traveler 9.0.1.4 which fixes the reported crash issue with MIME conversions mentioned earlier --> http://www.ibm.com/support/docview.wss?uid=swg1LO84505
If you are on 9.0.1.3 you should update asap.

There are a couple of other important fixes included -- see below.

Already installed, thanks Sebastian for the heads up!

-- Daniel
Release Date Component Build Level Documentation
April 29, 2015 Server 9.0.1.4 201504201605_20 IBM Traveler 9.0.1.4 Release Notes
IBM Traveler Product Documentation
Android Client 9.0.1.3 201504141229

APAR # Component Abstract
LO84142 Android Delay in displaying name lookup results from compose dialog.
LO84220 Server Change default for number of corporate lookup results from 30 to 120 results.
LO84239 Android Search e-mail on Android Tablet may display results from wrong e-mail.
LO84410 Server Incorrect language used when processing multiple calendar notices.
LO84334 Server Decline notice from device is not compatible with Exchange Server.
LO84316 Android Android client crash on old 2.x OS devices.
LO84411 Server Mime format calendar entries may not display special characters correctly.
LO84490 Android Send mail gets stuck in Outbox if the user is over quota.
LO84505 Server Server may crash processing a Mime document with invalid format.
LO84520 Android Imported calendars on Android device may not update unless there is Traveler Calendar update.
LO84555 Server Server busy message sent to the device may be misleading as to cause.
LO84568 Server Pre-approval and delete API may fail if orphan records encountered.
LO84569 Server Server performance issue related to HTTP getStatus request.
LO84597 Server E-mail using Delivery failure form may not sync full body to mobile device.
LO84660 Server Plain text conversion is adding extra space for div html tag.
LO84662 Server Mime format document with both plain and html text may not sync the plain text to the mobile device.
LO84663 Server Android may stop syncing mail after encountering a malformed Mime format document.
LO84665 Server Embedded images with name mime.jpg will not sync to mobile device.
LO84684 Server Change to device security settings may not sync immediately to BB and Windows devices.
LO84686 Server User stops receiving mail for couple hours if all mail replicas restarted in close proximity.
LO84723 Server No invitee status displayed for meetings created from Android client.


Traveler 9.0.1.3 server crashes when attempting to sync a MIME-formatted document missing a RFC822 header

Daniel Nashed  13 April 2015 09:05:51
You might want to wait updating your Traveler Server to 9.0.1.3 because of a MIME related bug that can cause crashes.
IBM now released a technote with official information about the issue --> Technote 21701590
If you already updated and have abnormal process terminations in the Traveler servertask you should not try to downgrade but instead request a fix from IBM (going back to an earlier version would cause a complete resync of all devices).


IBM is working on a 9.0.1.4 version with will -- according to the technote -- be released in April.


I am running 9.0.1.3 since it was released and did not yet run into a crash.
But if you did not update yet you should wait for 9.0.1.4.


-- Daniel

New Start Script Version 3.0 with systemd support released

Daniel Nashed  7 April 2015 10:12:21
There is a new version of the start script for Domino on Linux (also AIX and Solaris) that supports RHEL 7 and SLES 12 which a both now using systemd instead of the older init scripts.
When you are migrating to one of those platforms you have to switch to the new start script and also use systemd to start/stop your Domino server.

Also for the new versions of Linux the start script remains the main main entry point for all your operations with the server.
But for start and stop you will need root permissions or your Linux admin can allow you to use the start script with root permissions via "sudo".
The start script an invoke all the needed systemd commands to start and stop the Domino server. But you can also use the systemd commands instead.

I have updated and rewrote part of the documentation. If you are familiar with the start script already you should be aware that there are some changes.
There is a new "domino.service" file which represents the systemd service. You need one of those files for each partition along with the rc_domino file.
In the domino.service file there are references to the rc_domino_script which need to match the path where you have installed the script.
And also rc_domino needs information which service file should be used. By default the service name is commended out to work with previous versions.
If you are running with systemd you have to set the "DOMINO_SYSTEMD_NAME" variable to your domino.service.

The documentation contains information about all changes and there is a "systemd" section in the readme as well.

In addition I added an additional status command. "statusd" gives you the systemd status for your service.

And I have also added another not related command which I wanted for my own environments.

The "resources" command shows you all resources the server currently uses (processes, shared memory, semaphores, MQs ..).

Here is a link to the script page --> http://www.nashcom.de/nshweb/pages/startscript.htm
You can request the new version with the form that page.

There are also some other minor changes all documented in the version history.

If you have any questions let me know by mail.

Enjoy the new version

Daniel


DHA with more than 1024 key size and Java still works

Daniel Nashed  6 April 2015 22:58:19
As posted before Java 6 and 7 cannot handle DHE key sizes above 1024 bit.
The work-around was to limit the DHE key size via notes.ini parameter SSL_DH_KEYSIZE=1024.
But this reduced the key size for all other clients that used DHE as well.

There is another idea who to work-around this limitation.
Java does only support the following DHE cipher:

33 - DHE_RSA_WITH_AES_128_CBC_SHA

This is the weakest DHE cipher supported by Domino. If we disable this cipher, Java will not use DHE any more and we are not limited by the DHE 1024 bit key-size that is the maximum size that Java supports.

Disabling this cipher results in the following ciphers to be used. for Java For Java 8 a different DHE cipher is implemented and the 1024 does not apply for Java 8.

Java 6u45          TLS 1.0         TLS_RSA_WITH_AES_128_CBC_SHA (0x2f)   No FS         128
Java 7u25         TLS 1.0         TLS_RSA_WITH_AES_128_CBC_SHA (0x2f)   No FS         128

Java 8u31         TLS 1.2         TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x9e)   FS         128


This sounds like a good work-around for the Java DHE key-size limitation.

The resulting cipher spec for DHE with all other recommended ciphers enabled is the following:

SSLCIPHERSPEC=9D9C3D3C352F0A39676B9E9F

For more details check my previous blog posts.

-- Daniel


New Version of KyrTool released

Daniel Nashed  3 April 2015 08:38:12
There is a newer version of the key ring tool that has been released on fix-central.

Here is the list of fixes for the newer version.
You should also update your client and server to the latest available IF because there are also fixes in the back-end for some issues parsing certificates.

By the way ... I really like the command line kyrtool. A couple of days ago a customer asked me for some maintenance of their existing key ring files.
Their CA expired and we had to remove the root CA from over 150 key-ring files.
Using a shell script in combination with the kyrtool allowed me to export the private key and certificates, use "sed" to modify the file, create a new key-ring file, re-import and verify the key-ring file.
We even dumped information about the keys, certs etc and validation of the key-ring files into a CSV file to have an overview :-)

-- Daniel
DKEN9U5UEX Fix crash if pem file provided as input file has embedded nulls
KLYH9UBNGW Add Sha 256 Pinning to the kyrtool - displaying the digest on show commands
MKIN9QHT5W Fix kyrtool crashing when attempting the create command and giving an existing directory for the keyfile name
DKEN9RVQGD Fix kyrtool sometimes erroring on import all command



http://www.ibm.com/support/fixcentral/swg/selectFixes?parent=ibm~Lotus&product=ibm/Lotus/Lotus+Domino&release=9.0.1.2&platform=All&function=fixId&fixids=KYRTool_9x_ClientServer&includeSupersedes=0