Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

TLS and SHA-2 Support and the POODLE Attack

Daniel Nashed  21 October 2014 19:20:27

IBM has officially responded to the POODLE attack and also officially responded to newer crypto standards.

Very good news for Domino! IBM will introduce TLS 1.0 and SHA-2 support for all protocols soon!

The current technotes mention a very short timeframe and it looks like we are going to get fixes at least for the current Domino 9.0.1 code stream.

Some fixes will be also in the 8.5.x code-stream but some of the improvements like SHA-2 support cannot be back ported.

So you should be prepared with all your internet facing servers to deploy 9.0.1 with current fixpack 2!

IBM will introduce support for the current standards soon which will also address the POODLE attack.

IMHO the risk right now is not very high and and most of the HTTPS internet facing servers for larger companies already use Secure Reverse Proxies.
And you might need to have a closer look into what crypto levels those server currently support! You should disable SSL 3.0 on all servers as far as it is currently possible. This is not just true for Domino.

IBM is working on improving the crypto "stack" (part of the lower network layer -- "NTI" which is the base for all Internet protocols and this includes in consequence also the keyring file used to store your internet certificates) in Domino in a short timeframe.

Enclosed you find links for the two new technotes which provide details about what IBM is working on...

We have been asking for years specially for TLS 1.0 and higher support for SMTP. Now it looks like we are getting it also for all other internet protocols!

That's really great news!!

-- Daniel

How is IBM Domino impacted by the POODLE attack?

Planned SHA-2 deliveries for IBM Domino 9.x

Traveler Issues with Attachments containing special chars after updating to 9.0.1 IF6

Daniel Nashed  27 September 2014 12:57:17

Before leaving for holidays last week the first customer contacted me about issues with attachments that have blanks, umlauts or other characters in the attachment name.

I could not reproduce it on iOS but on Android but without the error message in the log that he got.

Meanwhile it is clear that this issue affects all devices types and there is a fix that should hopefully address this problem.

IBM is working on a new IF to address the issue and also possible other related issues but meanwhile if you need to install IF6 for full iOS 8 support (issues with calendar, companion app) before the new IF is released you should request a fix from IBM by opening a PMR.

There is a ARPA LO82085 describing the problem but it is just mentioning the "+" sign. But the problem is more general.

Reference for this ARPA is -->

Here is a sample error message

19.09.2014 07:14:09   Notes Traveler: WARNING Could not find the StreamedDataInfo: Key(86 bytes)=...Übersetzungen.doc@547EF898341C988FC1257D58001C6445_48980, RefId=... Übersetzungen.doc@547EF898341C988FC1257D58001C6445, DataSize=-1, EncodingºSE64, StreamingCompleted=false, so the data was not streamed

In addition that you cannot open those attachments on mobile devices we noticed a higher CPU utilization which should be related to this problem and can be a lot higher.

Hopefully we get a new IF soon. I will keep you posted...

Update 28.9.2014: There are multiple hotfixes for different issues and not all problems are solved.
If you don't use the companion or todo app you should stay on 9.0.1 IF5 and wait for IF7.

The problem(s) seem to be more complicated to fix.

-- Daniel

My Top 3 Formula Commands for working in the Notes Client

Daniel Nashed  19 September 2014 06:53:48
All of those commands are not new at all. They are all round for a very long time. But they make my day easier.
I am surprised that many still don't know at least the first two.
The last one is more a convenience when working with replicas.


Before Release 5 there wasn't an admin client and the admin/designer was integrated into the normal client.
The old live console is still in the client and you don't need an admin client -- just the right permission.
You can launch it from a smart icon and have a simple admin console without starting the admin client.

@Command([Execute];"notepad"; @ConfigFile)

Actually that is a combination of two things.
@ConfigFile returns the location of your notes.ini. This can be very helpful if you at another user's Notes Client and don't know where the notes.ini is located.
You can create a new memo (CTRL-M) write the text in the subject and press Shift + F9 to evaluate the formula.
By the way Shift + F9 works in every field and you could also use it as a calculator.

In combination with the execute this opens the notes.ini in a notepad for editing.

@Command( [ReplicatorSendReceiveMail] )

If you are using local replicas and got a new mail notification but the mail is not yet replicated this command helps to just send and receive mail.

All three commands are not really new but you might have forgotten about them...

-- Daniel

    Important Update on Traveler iOS 8 Support -- You have to install an IF!

    Daniel Nashed  15 September 2014 22:23:53

    There are some last minute changes in iOS which are only in the final version.

    Apple changed the EAS Sync ID which used to match the Device ID. There has been planning for that change for a while but Apple should have introduce that change already in the Beta releases.
    However this change causes issues in device mapping for the companion/todo app.

    IBM released a IF for 9.0.1/ UP2 today to address this issue and added some background logic to map the device ID.

    There is a ARPA describing the issue

    The problem mainly occurs when you register new devices and causes issues with todo and companion app.
    Existing devices with existing profiles should keep their ActiveSync Device ID. But you will run into issues with new registered users and companion/todo app.

    The IF does also address a couple of other issues. Some are also iOS 8 releated.

    You should update your Traveler servers ASAP.

    Detlev put together a nice details description of what happens in the backend.
    See his blog for additional details -->

    References for fixes for all supported versions.
    You should update even if you are not using the companion/todo app.

    IBM should have sticked with their own rule not announcing support for something that has not yet shipped.
    But we as partners and customers wanted to know in advance what version will support iOS 8.

    Thanks for this very fast response from the Traveler team!
    There are always changes in new software releases and I was surprised that we get a support statement before the release.

    -- Daniel

    9.0.1 IF6 IF7

    8.5.3 UP2 IF7

    Traveler iOS 8 Support

    Daniel Nashed  10 September 2014 17:23:05
    Update: IBM released and IF to address some last minute fixes required for iOS 8!!

    See this blog post for details


    iOS is released soon (hopefully 17.9 for existing devices) and I already got some customer questions about it.

    There is a technote describing the Traveler support for iOS 8.

    The good news everything should work fine and new app versions for iOS are on their way.

    Traveler supports iOS 8 with 8.5.3 Upgrade Pack 2 and higher but I would highly recommend that you update to the latest and greates release 9.0.1 IF5 anyway.

    Only the latest IFs will recognize iOS 8 correctly because they have the build-in codes for the new OS release.

    See all details in the official support technote

    -- Daniel

    Important Platform Support Additions in Notes/Domino 9.0.1 FP2

    Daniel Nashed  21 August 2014 17:59:38
    The new fixpack adds the following platform support:

    9.0.1 FP2 adds support for the following:

        Citrix XenApp 7.5 for Client
        Internet Explorer 11 for xPages
        RHEL7 for Server

    I got the question for RHEL7 already a couple of weeks ago and I think it is great news to have RHEL7 support introduced with a fixpack! That does not always happen!

    The release notes have been updated today and tests are completed.

    A big thanks to IBM also for the other two important platform version updates!!

    -- Daniel

    Traveler 9.0.1 IF5 shipped

    Daniel Nashed  30 July 2014 08:06:19
    Traveler 9.0.1 IF5 shipped just in time for updating a customer yesterday -- after we planned the downtime for more than a month -- funny.
    First updated my Linux box before updating the customer server on Windows.

    The Linux silent install on Linux was a lot quicker than the one on Windows.

    There are a couple of important fixes for all devices types and a new version of the Android client.

    IBM Notes Traveler 9.0.1 Interim Fix 5

    Release Date Component Build Levels Release Documentation
    July 28, 2014 Server
    Android Client
    9.0.1 IF5 Release Documentation

    APAR # Component Abstract
    LO78514external link Server Accepting meeting reschedule or update on iOS 7 device may not update the server copy.
    LO79236external link Server Exception thrown processing repeating event with empty date time stamp.
    LO79453external link Server Reschedule from BlackBerry device may not show correctly for attendees.
    LO79507external link Server Some calendar entries may be missing on device after issues Traveler reset.
    LO79517external link Server Extra reply notice may be generated for non-repeating event.
    LO79665external link Server Import of Notes Calendar may generate duplicate events.
    LO79714external link Android LED notification not working on some Android devices.
    LO79747external link Android Unable to reply to or forward e-mail to user name that contains an ampersand.
    LO79754external link Android Uncommon file extension may not launch when selected from Notes Traveler client on Android.
    LO79796external link Android Field used to edit Out of Office message doesn't scroll in Notes Traveler client for Android devices.
    LO79811external link Server ActiveSync provision loop may cause resync of all data.
    LO79824external link Server Traveler tracking field may grow to large in mail document.
    LO79933external link Server Device security view may not display all devices in Traveler HA Pool.
    LO79952external link Server BlackBerry device may send incorrect date for event instance.
    LO79960external link Server Third e-mail address for contact created on mobile device may get replaced by first e-mail address when edited by Notes Web.
    LO79975external link Server Plain test mail from Android should use UTF-8 encoding.
    LO79999external link Server Return receipt message not consistent with Notes Client.
    LO80087external link Server Create contact on Apple device and IM Address field may appear unexpectedly.
    LO80092external link Server Event summary data may be too large for document processing.
    LO80163external link Android German translation for ToDo not correct on Android client.
    LO80183external link Server Send mail from Apple device with no text and an image may loose the image.
    LO80296external link Android Tablet view instead of phone view displayed on Sony Xperia T2 Ultra Phone.
    LO80340external link Server Apple devices running iOS 7.1.x may periodically resync folders and other content.
    LO80343external link Server Out of Office formatting error being logged by Traveler Server.
    LO80373external link Server BB 10 device may get stuck in Calendar event sync loop.
    LO80415external link Server Double incompatible with String error message on Traveler server.
    LO80422external link Server Too many unsupported start date warning messages on Traveler server.
    LO80423external link Server Cleanup does not always cleanup all users.
    LO80425external link Server May see field too large to save document error due to presence of BlackBerry fields.
    LO80552external link Android Samsung Galaxy S5 fingerprint scanner is not recognized as valid option when password type is unrestricted.
    LO80595external link Server Deadlock on mail server table.
    LO80777external link Server Change read statice for Calendar notices when processing from mobile device.
    LO80925external link Server Support confirmation notice on Apple devices when changes are included.
    LO81006external link Android Calendar alarm may not dismiss on some Android devices.
    LO81091external link Server Improve event fixup Traveler command.
    LO81158external link Server Handle NTS_BODY_THRESHOLD like normal truncation scenario

    Force Traveler to use IPv4 instead of IPv6

    Daniel Nashed  28 July 2014 15:41:26
    We ran into this in a customer situation.
    The code used in Traveler is Java based. For the Servlet and also for the Travler servertask.
    Even if you specify notes.ini NTS_HOST_IP_ADDR with an IPv4 address Traveler might use IPv6.
    If you are in stand-alone mode this should not cause any issues.
    But if you are in HA mode connecting to a remote machine might cause trouble in some situations.

    My recommendation would be to completely disable IPv6 on the machine unless you really need it.
    At some hosted environments like a hosted virtual server (not a root server) you cannot disable IPv6 on Linux completely -- because it needs to be changed on kernel level and you don't control that on those machines.

    On my machine hosted at a provider I went thru the hotline to find out that I cannot disable it.

    But I still wanted to use a IPv4 address for my Traveler server.

    There are two steps that you have to follow.

    First of all convince the Traveler servertask to use IPv4 addresses.
    That can be done with the following options that you pass to Java.

    Second you have to convince the Traveler servlet to use IPv4 only.

    You can pass Java parameters via the HTTP configuration.

    Create a read-only file that is owned by root like this -- Can be located in the data directory if you set the right permissions.

    -rw-r--r-- 1 root root 31 Jul 21 16:27

    Add the following parameter:

    And specify the file in the notes.ini of your Traveler server:

    notes.ini JavaUserOptionsFile=/local/notesdata/

    After restarting the Traveler servertask and HTTP you should see via netstat, that 50125 (Traveler Servertask) and 50126 (Traveler OSGI Servelet) are binding to IPv4.

    -- Daniel

    DAOS NLO Encryption and Decryption

    Daniel Nashed  28 May 2014 10:18:57
    We have been asking for this functionality since DAOS was releases and now there is finally a solution.

    In some cases customers have to either switch of DAOS NLO encryption for a server or enable it later on. Or even want to move from one to another

    There are two SPRs (#PMAO9C6R9G / #GFAL9AKKJZ) described in the following technote -->
    The TN also describes how to use this new functionality.

    There are a couple of details that you should be aware of. First of all the two SPRs are not included in shipping code and are also not yet listed in the fixlist database.
    But they have been submitted to the 9.0.1 code-stream as far I understood.

    The output of the commands are printed to the console (using xprintf which is the equivalent of the internal console write call).
    I have asked if the output can be written to a file via -o opton in future. But for now you have to use the redirect invoking the daosmgr command.

    The TN also mentioned this fix-numbers. So if you need this functionality urgently you can try to request a hotfix from IBM.

    And as described in the TN you should use the migration to either encrypted or unencrypted offline. The move is a major migration. All NLOs will be rewritten most cases. This should be planned for a weekend and should be a one time action only.

    What are the szenarios and reasons to change the encryption of the NLOs?

    In many cases NLOs are encrypted because when DAOS was introduced to an environment someone forgot to set the notes.ini parameter to disable DAOS_ENCRYPT_NLO=0.

    But most customers don't require encryption of NLOs.

    If the NSF files on your Domino server are not encrypted and the is not protected by a password, it does not make much sense to have the NLOs encrypted.
    It is even harder to find the right information in a NLO than in a NSF file. And if you copy the NLOs to a different machine including the if it has no password, you can read the NLO anyway.
    So in most cases not having NLO encryption enabled is a best practice for a couple of reasons and the encryption only adds security when the is protected as well.

    Encryption adds not that much overhead at runtime but there are a couple of other reasons.

    First of all if you want to use another cluster member to copy missing NLOs as more simple restore scenario when a NLO is missing this is only possible if NLOs are not encrypted.

    Second if you have storage like a NetApp where you have enabled block-level deduplication and point multiple DAOS stores to the same NetApp volume you can save a lot of disk storage because the same NLOs will have the same blocks. This does only work if the NLO is not encrypted because the same NLO on different servers will be encrypted with a different key (actually even on the same server when encrypted later the file could be different because of a different "session key").

    On top of that some backup solutions support block-level deduplication. And that could save space on the backup side as well if encryption is disabled.
    With encryption enabled there is amost no block-level deduplication.

    In addition moving DAOS stored among servers when you switch the is much more simple without encryption.
    But if you have the new options in the daosmgr you could now re-encrypt NLO files with a new
    I would only do this if you really really need it. In normal cases in such a migration scenario I would use the new functionality to disable NLO encryption for the above reasons.

    IMHO it is still good to have NLO encryption enabled by default to avoid discussions about DAOS security.
    But in reality in at least 80% customer environments NLO encryption is not required overhead and complexity.

    I know others think differently about it and that's just my humble opinion...

    On the other side we also have customers who started without encryption and now need to encrypt all databases, NLOs and also protect the with a password (including the need for a solution to apply the password on server start in a secure way).

    Thanks to IBM to make this change and have it implemented in a flexible way to do it both ways including a verification options for encryption status of all NLOs.

    -- Daniel

    Details About ODS 52 shipped with Notes/Domino 9.0.1

    Daniel Nashed  29 April 2014 07:09:14
    I got a couple of questions from multiple customer about ODS 52 which has been introduced in 9.0.1.
    There is a bit of confusion about the new ODS and there is not much public available information.

    First of all the new ODS 52 is optional and you only need it in some special cases.

    It is not enabled by default and in the same way that you needed to set the new ODS it will also be implemented in 9.0.1

    How to migrate to the new ODS?

    You will need to set notes.ini CREATE_R9_DATABASES=1.

    And the new ODS is available and important for clients and servers.

    There are different ways to move databases to the new ODS on servers and clients.

    For clients you will need to set NSF_UpdateODS=1 in combination with CREATE_R9_DATABASES=1 which lets the client convert to the new ODS.

    On the server side you will need to set CREATE_R9_DATABASES=1 and use a copy-style compact.

    You can either leverage the compact or the preferred method would be to leverage DBMT which would also generate an unfragmented new NSF file by default.

    e.g. DBMT –compactThreads 6 –updallThreads 0

    Why to migrate to the new ODS?

    There are multiple reasons to migrate to the new ODS.

    a.) Issue with encrypted databases

    The best public available information about it is from John Paganetti's IBM Connect 2014 presentation. Thanks John for sharing those details!
    Everthing else I found is either not detailed or not public..

    Issue 1: Medium and Strong Encrypted Databases

    - Problem – Rare note corruption when updating a note, only occurs with Medium or Strong encrypted databases

    - Has existed since Notes/Domino began using Medium and Strong encryption

    - Not noticed because vast majority of databases have replicas and fixup would discard the corrupted note and next replication the note would come back in just fine

    - Resolution – Best way to maintain backward compatibility and interoperability was to address with a change to the on-disk-structure (ODS)

    Issue 2: Medium Encrypted Databases

    - Problem – Rare note corruption when updating a note, only occurs with Medium encrypted databases

    – Has existed since Notes/Domino began using Medium encryption

    – Not noticed because vast majority of databases have replicas and fixup would discard the corrupted note and next replication the note would come back in just fine

    Resolution – The fix for this issue would affect the vast majority of the data and hence there were security concerns it could potentially weaken the current Medium encryption strength.
    As a work around, Security team recommends customers go to ODS52 and upgrade existing Medium Encrypted databases to Strong

    If you are using encrypted databases either on Notes client or on Domino server you should update to the new ODS!
    But this requires to be on 9.0.1 code -- also on the client.

    You will have more likely encrypted databases on a client than on a server.

    IMHO On the server -- unless you have a password on your (and a tool to manage that on server startup) -- you should disable encryption.

    Without a password on the there is not much sense encrypting databases (and NLOs).

    But in case you need encryption you should update to ODS52 and switch to strong encryption.

    There is also another detail that John shows in his presentation.

    I have not seen any public information for the overhead that encryption has on CPU utilization. And this information is quite useful.

    NRPC run of Win2008 R2 Server 64-Bit @ 4000 Users, mail9 template
     Not Encrypted  35% CPU
     Medium Encrypted  39% CPU
     Strong Encrypted  48% CPU

    On a client this is not really much overhead -- unless you are on a Citrix server.

    But for a server this can be quite some overhead.

    If you don't want that additional overhead there is a fix that helps also with medium encrypted databases.

    But you will need to compact the database to the "new" medium encryption with ODS52 as well.

    This is clearly more a work-around and the security team recommends to upgrade to strong encryption if you can.

    Here is the way to enable the fix:


    - Next copy style compact of existing Medium Encrypted databases will be ODS52 with new Medium Encryption which has fix applied

    You can update all your medium encrypted databases to strong encryption leveraging copy style compact.

    The notes.ini setting you need for that is COMPACT_UPGRADE_MEDIUM_ENCRYPTION_TO_STRONG=1.

    This parameter can be quite helpful because it would be a manual step to migrate to strong encryption without it.

    And you should disable the parameter when you are done with upgrading all databases to strong encryption.

    On Notes clients databases are usually encrypted by default. The notes.ini setting LOCAL_DB_ENCRYPT_DEFAULT determines which encryption strength to use
    (0 = No Encryption, 1 = Simple Encryption, 2 = Medium Encryption, 3 = Strong Encryption)

    So you should have enabled the following for new databases that should be encrypted with strong encryption.



    Note: In case your workstation uses local disk encryption and/or you are using shared login there is also not much sense in encrypting databases.

    a.) Issue with large attachments

    There is an issue with attachments larger than 2 GB which is fixed in ODS52 in 9.0.1

    Fix for ZXZG85KJRK: Large attachments above 2 GB fail

    You need Notes 9.0.1 clients and Domino 9.0.1 servers in combination with ODS 52 to get this completely addressed.

    Details are available in the following technote:

    This issue is another reason to upgrade to the new ODS even this is an issue that might only hit you in very rare conditions.

    Additonal Note:

    There are also settings to log the database encryption used. They will report the current encrpytion level based on the settings the first time a database is opened.

    Administrators may now easily identify which databases are currently encrypted and the encryption level, by setting the following notes.ini variable


    Utilizes a Bit Mask

    1 is “Show Simple”

    2 is “Show Medium”

    4 is “Show Strong”

    To see all Encrypted Databases

    Simple, Medium and Strong (1+2+4 = 7)

    Set SHOW_ENCRYPTED_DATABASES = 7 in notes.ini

    When encrypted databases are opened for the first time - 0 to 1 transition, one of the following messages will be logged

    “Current encryption strength: SIMPLE - < absolute file path >”
    “Current encryption strength: STRONG - < absolute file path >”

    Legacy Medium encrypted database

    “Current encryption strength: MEDIUM - < absolute file path >”

    New Medium encrypted database with fix (+)

    “Current encryption strength: MEDIUM+ - < absolute file path >”

    As long as running Release 9.0.1, SHOW_ENCRYPTED_DATABASES works for all database ODS levels


    It makes sense to switch to the new ODS in some cases but you don't need to necessarily put it directly into your upgrade path -- at least on server side.

    This can be done afterwards with a copy-style compact that you should run once in a while on any database.

    DBMT in 9.0.1 helps you to keep databases defragmented -- check one of my recent blog entries for details.

    And in the same step you can upgrade the ODS if needed.

    On the server side there is most of the times really no reason to use encrypted databases in the first place.

    So as not mentioned in other postings about the new ODS52 the most important step is to migrate to the new ODS on client side.

    Unless you have users storing 2 GB attachments in their mailfiles...