Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

    Domino on Linux Platforms

    Daniel Nashed  4 March 2019 15:11:21
    Now that we have official Domino support for CentOS I would expect everyone who needs a free server OS to look into CentOS.
    Still -- I got the second person in two weeks looking into Domino on Ubuntu.

    Just to be clear -- everything that is not RHEL, SLES or CentOS LTS with a matching release is completely unsupported by IBM/HCL and also by my Start Script and other solutions.
    I will not answer any questions for anyone using Ubuntu or other distributions!

    My start script is completely free of charge and I am trying to answer every question.
    But I cannot spend additional time to look into other distributions.

    And I also don't see a need for looking into other distribution since CentOS is supported.
    CentOS is also the base for the official Domino on Docker script. And is also used as the current development platform at HCL.

    There are huge differences between Linux versions and distributions and it doesn't make sense to use anything untested and unsupported!

    CentOS administration isn't that difficult. But maybe I should post a quickstart HOWTO for Domino on CentOS?
    A setup from a minimal image just takes a couple of minutes and there are not many command line steps to follow.
    I have added a sample rule for the firewall configuration xml to my start script which can be used to open the NRPC port.
    Firewall and network configuration might be the only two a bit more complicated steps beside systemd (but systemd is something that you will have to deal with in any case).

    -- Daniel


    Traveler Optimization for Slow Backend Mail Server Connections

    Daniel Nashed  3 March 2019 09:30:57
    In the last couple of month we have been working on performance bottlenecks for customers with higher latency network connections between Traveler and the back-end mail-servers.
    It took a while until we got all the fixes implemented after very detailed analysis (for example I wrote an extension manager to track object reads).

    The good news is that those fixed are included in the current release and most of the settings are now even enabled by default in the latest releases.


    [Side note about Traveler accepted latency]


    IBM/HCL recommend that the connection between your mail-servers and Traveler servers should have less than 50 ms latency!
    But you don't have always a choice. On the other hand I have seen corporate network connections with latency around 5/6 ms today!
    Even internet connections between two different provides I use are around 6 ms!

    See technote for recommendations and troubleshooting steps:


    https://www.ibm.com/support/docview.wss?uid=swg21961707

    My first observation was that the attachments for richtext messages are sent multiple times over the network during sync, which lead to the first fix already implemented in the Traveler 9.0.1 code stream.

    After we got the fix, I figured out that also MIME messages have been effected in similar way -- It was just harder to track.


    Specially on WAN networks transferring attachments multiple times causes additional network utilization and in combination with higher latency also causes slower sync.

    Not just when the attachment is syned, because attachments might be pre-streamed in some cases.

    The changes are very low-level in the back-end how Traveler uses the Domino APIs. So the overhead was only trackable below the Traveler interface to Domino (C-API calls).


    The two many changes have been implemented in Traveler 10.0.1 / 10.0.0 and one fix needed a changed notes.ini to not pre-stream the attachment.

    In our first hotfixes the parameter needed to be disabled NTS_ATTACHMENT_PRESTREAM=false but since 10.0.1 the parameter is disabled by default.


    The pre-stream of attachments was needed for blackberry devices which need the exact size before syncing a document. Unless you have blackberry devices the new default should work for you.


    The two main fixes are the following:


    Traveler 10.0.1.1
    TRAV-3279         MIME message processing reads attachments multiple times


    Traveler 10.0

    TRAV-3004         Avoid streaming attachments just to calculate size.



    In addition Traveler 10.0 introduces two other optimization fixes for slow network connections:


    TRAV-3165         Reduce Dispatch logging to reduce network utilization.

    TRAV-2952         Master Monitor queue bottlenecked by slow response from mail servers.



    Network Session Optimization


    There is one additional notes.ini Parameter which is helpful to optimize back-end connections between Traveler and the Domino mail-servers.

    I have worked in two larger environments with a high number of Domino mail-servers in the same Traveler HA pool.


    Usually you should use separate Traveler pools for servers in different locations and best practices would be to have a Traveler pool in the same data center than your mail-servers. But this isn't possible in all customer environments.

    In combination with a high number of users on different mail-servers and a single Traveler HA pool, we have seen many open network connections per mail-server.

    You can see up to 40 ESTABLISHED network sessions with mail-servers for a longer time.


    The following (finally officially documented) NTS parameter helps to optimize and properly recycle those Domino NRPC network sessions between Traveler and your mail-servers.

    If you are experiencing a high number of open NRPC sessions per Domino back-end mail-servers, you should have a look into this parameter.


    NTS_DOMINO_THREADS_OPTIMIZE_RECYCLE=false


    Controls whether IBM Traveler threads that use Domino API calls are Domino initialized and terminated when IBM Traveler is done with the thread and
    the thread is destroyed (true) or when each usage of the thread for a user's device is done but the thread is not destroyed (false).
    True saves the overhead of doing the initialization and termination for each user's device but NRPC connections are cached per thread and only released upon the termination.
    If your IBM Traveler server is talking over NRPC to a large number of mail servers (for example, more than 100) and the IBM Traveler server is running out of TCP/IP network ports,
    you may want to change this value to False to force more frequent thread terminations which release NRPC connections more frequently.

    Traveler causing replication/save conflict for invitations accepted in Notes client

    Daniel Nashed  19 February 2019 15:29:54

    Traveler causing replication/save conflict for invitations accepted in Notes client

    This issue can occur for normal invitations and also with *.ICS files that you add to your calendar.

    We found out that this is a timing issue between the user interaction and the background processing of the Traveler server.


    There are two different scenarios:


    a.) Accept an invitation (technically a "notice" document which when accepted is saved as an "appointment".


    b.) An *.ICS file which is launched and turned into a "notice" which is than when accepted saved as an "appointment"


    In both cases it can happen that the so called meeting ghosting adds a field "ILNT_GhostUNID" into the "notice" document causing a replication/save conflict when the user interacts at the same time.

    Calendar ghosting is used to display the invitation as a not yet accepted meeting in a calendar.


    Right now the only way to safely avoid this issue is to disable Calendar Ghosting functionality on the Traveler side.

    The functionality on the Notes side will continue to work. This just disables the Traveler functionality.


    We got the following notes.ini settings for disabling the Ghosting Calendar functionality. You need to disable both!


    My support ticket is still open and I will update this blog entry when I get a SPR.


    -- Daniel


    NTS_IOS_CALENDAR_INITIAL_GHOST=false


    Traveler will ghost meeting updates to the Apple iOS Calendar application's calendar even when the initial invitation has not been accepted yet. Ghosting enables you to see the original invitation and schedule change correctly. Applies to iOS 9 or later

    The default is true.



    NTS_CALENDAR_GHOSTING_SYNCML=false


    Allows meeting invitations to be "ghosted" on the Verse mobile client calendar (iOS and Android). Ghosting enables you to see the original invitation and schedule change correctly.

    The default is true.



    If you are interested in what happens see the details below:


    a.)


    The user opens the invitation ("notice" document) before the Traveler processes the "notice".

    While the user has the invitation open Traveler updates the document and adds the "ILNT_GhostUNID" to it.

    The user accepts the meeting and saves turns the "notice" into an "appointment", which causes the replication/save conflict


    --> The conflict occurs when the user opens the invitation before the Traveler has processed the invitation


    b.)


    When an user opens an *.ICS attachment a new "notice" document will be created in the user mail-file an the document will be opened directly in the client.

    Meanwhile the Traveler server will see the document and add the "ILNT_GhostUNID".

    Afterwards the user accepts the meeting and updates the "notice" to turn it into an "appointment"


    --> The conflict occurs when the user is taking "too long" to accept the meeting.


    But in this scenario the delay needed is quite short and you have to be lucky!
    It will happen most of the time because the Traveler server has a subscription into the mail-files with the notification API and by default checks for new data every 3 seconds.

    The only work-around from user side that would be save is:


    - Launch the *.ICS File to turn it into a notice

    - close the document and wait a while until Traveler processed the document

    - re-open it to accept it


    We first ran into scenario a.) but it turned out scenario b.) is the one that causes more issues because you have to be either very fast to accept the invitation before Traveler processes the document or use the described work-around.


    Replication/save conflicts in calendar documents can cause that appointments are not shown in the calendar even you accepted them and you cannot accept them again because it already exists with the same ApptUNID (internal ID used for identifying an appointment).


    IBM Think 2019 Session - Domino on Docker Boot Camp

    Daniel Nashed  10 February 2019 12:15:20
    Sadly I cannot make it to IBM Think this year.
    But I would like to highlight a session where I have some contribution.

    Thomas Hampel invited me to work with him on the IBM Docker image and beside contributing my start script I also worked with him on some parts of the Docker script.

    We finally ended up adding a kind of "start & management-script" for your Docker containers which is helpful in a stand alone Docker environment if you don't have Docker management tools in place.
    It will be part of my start script and added to the IBM Docker script. It will help you to get your Docker images build, created, updated, started, stopped, ..  and you can even directly attach to the Domino server console from the host command-line!

    Thomas will show the management part of the Docker project the for the first time in his session.

    Everyone who is attending IBM Think 2019 next week in San Francisco, have an interesting and fun conference!

    -- Daniel


    IBM Think 2019 Session - Domino on Docker Boot Camp

    https://myibm.ibm.com/events/think/all-sessions/session/7557A

    Wednesday, 10:30 AM - 11:10 AM | Session ID: 7557A
    Moscone South, Exhibit Level, Hall D | Data & AI Think Tank E

    Docker containerization in Domino V10 has become a powerful tool in the administrator arsenal. Join this session to learn the best practices for bringing Docker into your datacenter or hybrid cloud deployment.

    Speaker: Thomas Hampel, IBM


    Image:IBM Think 2019 Session - Domino on Docker Boot Camp


      Domino on Docker Requirements and Configuration

      Daniel Nashed  9 February 2019 11:17:06
      A while ago I started into look into Domino for Docker.
      I have contributed my start script under Apache 2.0 license to the official IBM Docker container.

      The first version is already available but we are still not 100% sure about versioning. And add-on product support.
      Currently the GitHub repository contains the Docker script and helper script to automatically build an image on your own environment.

      For license and legal reasons IBM cannot just put the ready to go image into the Docker registry to just pull it down.
      But there easy to follow scripts to get it working. I would still wish that at some point having a official Docker community image at least will be available.

      In addition I am preparing a Docker start script that will do all operations you need to get your Domino server up and running on Docker.
      It will build an own customizable image that will be based on the official IBM Domino Docker image.

      The script will -- very similar to my start script -- help you with all steps in an unmanaged Docker environment, From build, run, start, stop to updating, log collecting and interacting with the container.

      Thomas Hampel (IBM) will present the Docker image and also the Docker start script that we plan to include at IBM Think next week.

      Probably this will be included into the GitHub project either as part of my start script or an extra script.
      My start script already includes a Docker entry script and official Docker support for that reason.

      But beside that there is some other important information that I would like to share.

      There are some requirements that you will need to look into when you want to successfully run Domino on Docker.
      Those requirements did not make it yet into the IBM Docker technote but I want to share them for everyone who is running Domino on Docker.

      -- Daniel


      Domino on Docker Requirements and Configuration

      We have tested creating and running the image with the following environment.
      There are some special settings that you need to successful run Domino on Docker.
      And there have been recent changes to Docker. So you should make sure that you are running a current Docker version.

      Environment

      - CentOS Linux release 7.5.1804 (Core)
      - Docker CE 18.09.0

      - Linux and OSX, Windows is not supported!


      Storage and Driver Requirements

      Overlay2 Driver

      The overlay driver is an important component in your Docker infrastructure.
      There is a newer overlay driver "overlay2" which is required for the overlay file-system used by your container to run properly.

      Here is the official requirement on the Docker side

       https://docs.docker.com/storage/storagedriver/overlayfs-driver/


      But before you can start to change your overlay driver, your file-system needs to support d_type / ftype=1.

      When your file-system is not formatted in the right way, you will see the following warning message:

      WARNING: overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior.

              Reformat the filesystem with ftype=1 to enable d_type support.

              Running without d_type support will not be supported in future releases



      You can check if you already have ftype=1 check the following:

      xfs_info /dev/sdb | grep ftype


      naming   =version 2              bsize=4096   ascii-ci=0 ftype=1


      If this doesn't return anything you have to reformat the file-system with the right options.
      The easiest and most convenient way is to create new disk and format it in the right way.
      A best practice is to use this file-system for /var/lib/docker.

      Create a new disk for /var/lib/docker

      To create a new file-system on an additional disk you can use the following command.

      mkfs -t xfs -n ftype=1 /dev/sdb


      Afterwards you can mount the disk via fsstat. Before you mount it you should move the existing data to a different location and and move it back to the new file-system afterwards.

      vi /etc/fsstab
      dev/sdb         /var/lib/docker   xfs     defaults    1 1


      Changing the overlay driver

      Now that you ensured the disk has the right configuration you can change the overlay driver.

      You have to create a new configuration file and add the following configuration.

      vi /etc/docker/daemon.json


      {
       "storage-driver": "overlay2",
       "storage-opts": [
         "overlay2.override_kernel_check=true"
       ]
      }


      Requirements for NSD

      When looking into NSD inside the Docker container you first have to ensure that you have the right packages install.

      You will need the the GNU Debugger (gdb) and also the lsof tools to show open files and handles.

      yum install lsof gdb


      But this isn't all you need. There are missing permissions when your run NSD/GDB inside Docker.

      You have to start it with the following additional settings:

       docker run -p 1352:1352 -p 80:80 -p 443:443 --name docker-name --cap-add=SYS_PTRACE -v notesdata:/local/notesdata -v /etc/localtime:/etc/localtime:ro ibmcom/domino:10.0.0



      Conclusion and Result

      After this change all my file move and remove operations did work!
      The older overlay driver is not really supported any more but it is still the default driver at least in my installation.

      It's not just the driver! The driver depends on features in the XFS.
      You can query the overlay driver type via:

      docker inspect --format '{{ .Driver }}' container_name

      This will show if you have the right overlay2 driver.

      There have been some very odd issues when running NSD if you don't have the overlay2 driver.
      And also without GDB an LSOF is really required. And last but not  least you need to provide the right permissions to have GDB attach to your running processes.


      Be aware of Soundex

      Daniel Nashed  7 February 2019 14:54:49
      We just ran into an issue where I user was removed from the Domino directory and the server had setup Fullname than local part for Mail lookup configured.
      What happened is that the wrong user got the e-mail!

      @Soundex

      Returns the Soundex (the Lotus Notes phonetic speller) code for the specified string.

      The phonetic name can result in the same short "hash" value for multiple names. There are a lot of collisions even for names that are not the same!.

      I can only recommend to stay away from Fullname than Local Part for mail lookups. But if you can't you should be aware of @Soundex.

      @soundex ("nsh@nashcom.de") for example returns N200.

      I opened a support ticket and they pointed me to the following AHA entry. https://domino.ideas.aha.io/ideas/DOMINO-I-548

      IMHO this feature request to allow Soundex to be completely disabled makes a lot of sense!

      -- Daniel

      Domino HTTP Basic Authentication still uses ISO-8859-1

      Daniel Nashed  7 February 2019 04:44:32

      Specially for Mobile devices HTTP Basic Authentication is needed.
      Those devices don't understand login forms and forms based authentication.
      For forms based authentication you can configure which charset to use and most environments should be already setup to use UTF-8.

      For basic authentication there wasn't really a standard and the first implementations used ISO-8859-1.

      I just had a support ticket with IBM double checking about a way to change the charset to UTF-8.

      It's currently not possible and there is an enhancement request:

      SPR # DKENAJTT9G :Enhancement: Non-ASCII UTF-8 passwords don't work over basicAuth


      There is a newer RFC superseding the previous SPR.

      I have looked a Domino idea to have this enhancement request on the radar --> https://domino.ideas.aha.io/ideas/DOMINO-I-570

      -- Daniel

      See https://tools.ietf.org/html/rfc7617 for details.

      Since 2015 there is RFC 7617, which obsoletes RFC 2617. In contrast to the old RFC, the new RFC explicitly defines the character encoding to be used for username and password.
      • The default encoding is still undefined. Is is only required to be compatible with US-ASCII (meaning it maps ASCII bytes to ASCII bytes, like UTF-8 does).
      • The server can optionally send an additional authentication parameter charset="UTF-8" in its challenge, like this:
        WWW-Authenticate: Basic realm="myChosenRealm", charset="UTF-8"
        This announces that the server will accept non-ASCII characters in username / password, and that it expects them to be encoded in UTF-8 (specifically Normalization Form C). Note that only UTF-8 is allowed.

      Shortcut Key Issue Notes Client 10.0.1 G1 Languages - Download on hold

      Daniel Nashed  6 February 2019 18:34:21
      There is a critical issue with the standard client and G1 languages that have just been released yesterday.
      According to this technote the issue is under investigation and the download as have been stopped for now.

      That's all info I have right now ..

      -- Daniel

      https://www.ibm.com/support/docview.wss?uid=ibm10870434


      Shortcut Key issue impacting Notes Client 10.0.1 Group 1 Language release

      Ask the IBM Support Agent Tool
      Flashes (Alerts)

      Abstract

      An defect is under investigation with the Notes 10.0.1 Standard client language releases impacting keyboard shortcut keys that are improperly mapped. While the issue is under investigation, these kits have been put on hold from All IBM Download sites to minimize the impact to customers. This document will be updated as we progress.
      Content

      IBM has received reports and has confirmed an issue in Notes Standard language kits where keyboard shortcuts are not working as expected / mapped to incorrect key combination for that language. Since keyboard shortcuts are a very basic operation, the below kits have been put on hold while we investigate the issue.

      Traveler Sync Issue with more than one device

      Daniel Nashed  5 February 2019 14:24:24

      We ran into a situation where secondary devices not used all the time had missing mails, contacts and events.
      This was a long going support ticket, because it was very difficult to provide data from when the problem initially occurred.

      It turned out that this is caused by a bug in the way the cache worked. The cache is removed after the device is inactive (by default 24 hours) and the next sync when coming back when the device came back was affected by this.

      The fix is in Traveler 10.0.0 and higher. Traveler 10 is the next version after 9.0.1.21 and works on a Domino 9.0.1 server with current FPs (I would recommend using the latest IF for FP10).
      In contrast to Domino 10, Traveler 10 is an incremental release -- even it has some new features. So installing the Traveler 10.0.1 release on your Domino 9.0.1 FP10 server is perfectly OK.

      For some internal reason the fix was not included in the fixlist but the fixlist has been updated end of last month.
      See description of the fix here --> https://www.ibm.com/support/docview.wss?uid=swg1LO93818

      From what we see this does not only happen if the Traveler server was shutdown but also when all devices for an user are offline.

      To figure out if you have the issue, there is a command "DbRecordsCheck"  that you can run on your Traveler server. This check takes a while and goes thru all sync state entries for all users and devices.
      It will tell you which users have missing device records by comparing the table of documents that should be synced with what actually is synced.

      You can also take a dump for an individual user and check the dumped data for missing "DB records".

      Example:
      tell traveler dump daniel nashed

      Check the dump for lines that look like this:

        100000000000181001: ApplDMPT12XYZABC DB record was not found for this device.  LGUID: 100000000035031204 Type: 100000000000000401 (Event)
        100000000035510212: 6978dbc6ffab4180a1e1c7f16d42f70e timeSyncInDevice: 1543308447 (11/27/2018 09:47:27) timeSent: 1543308447
      (11/27/2018 09:47:27) DeviceRecordId: 100000000035031204 tsTaggedForSlowSync: 0 mChangeData: 0 mChangeMove: 0

      But if you want check all your users the db records check command is the right way.

      It comes in two different modes

      1. just check the records and show affected users
      2. check the records and if missing records are identified reset the device


      We took the approach to first check for all users and from the list we took the VIP users and users we know have been on the road and reset them manually.

      Example:
      tell traveler reset ApplDMPT12XYZABC daniel nashed


      The command is either

      Example:
      tell traveler DbRecordsCheck show 2500

      Or if you directly want to repair by resetting the users:

      Example:
      tell traveler DbRecordsCheck repair 2500

      The number is the maximum number of users that should be checked/fixed.

      See https://www.ibm.com/support/docview.wss?uid=swg1LO87614 for reference.

      The result looks like this

      -- snip --

      10.12.2018 12:41:41   Traveler: IBM Traveler Database is checking the records for 2202 accounts...
      10.12.2018 15:02:38   Traveler: 316 out of 2202 accounts were missing records and may need to be reset.
      10.12.2018 15:02:38   Traveler: Command DbRecordsCheck Show complete.

      The error for a user looks like this
      :

      10.12.2018 12:44:08   Traveler: CN=xyz.../O=Acme with account ID 100000000001234567 is missing at least one Traveler database record for a device but not all devices.  The first encountered record to be missing has LGUID 200000000012345678 and was not found


      -- snip --

      The command runs a while (it could be 1 hour or more for 1000 users) and checks one user after another.
      So if you are concerned about resetting too many users at a time, the reset will be spread over time just by the time it takes to analyze.


      Conclusion/Recommendation:

      If you are concerned that you might have this issue, you should do a DbRecordsCheck show first.
      When you have users facing this issue, you should upgrade to Traveler 10.0.1 first and afterwards run the DbRecordsCheck repair command or reset users/devices individual.

      If an user is in a good network location, it will take a couple of seconds to resync a device.
      But you should take care when users are on the road with a slow network connection!



      Notes/Domino 10.0.1 G1 Language Versions available today

      Daniel Nashed  5 February 2019 10:03:24
      I got this question many times in the last weeks.
      Today the G1 Language Versions will be available.


      In my Passport Advantage Account they did not show up yet.
      But a customer reported that he can see and download some of them already.


      PartnerWorld download also shows some of the files.

      It might take a while until all files are available. But they should show up today!

      Update: Andreas posted a detailed list with part numbers and descriptions -->
      https://ponte.ch/blog/get-your-10-0-1-language-packs-with-part-numbers/

      Another Update 6.2.2019:

      Download of some G1 Versions are on hold because of a keyboard local language issue with the Standard client --  Oooops


      See details below

      -- Daniel

      https://www.ibm.com/support/docview.wss?uid=ibm10870434


      Shortcut Key issue impacting Notes Client 10.0.1 Group 1 Language release

      Ask the IBM Support Agent Tool
      Flashes (Alerts)

      Abstract

      An defect is under investigation with the Notes 10.0.1 Standard client language releases impacting keyboard shortcut keys that are improperly mapped. While the issue is under investigation, these kits have been put on hold from All IBM Download sites to minimize the impact to customers. This document will be updated as we progress.
      Content

      IBM has received reports and has confirmed an issue in Notes Standard language kits where keyboard shortcuts are not working as expected / mapped to incorrect key combination for that language. Since keyboard shortcuts are a very basic operation, the below kits have been put on hold while we investigate the issue.


      Archives


      • [IBM Lotus Domino]
      • [Domino on Linux]
      • [Nash!Com]
      • [Daniel Nashed]