Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

Notes Client 9.0.1 FP9 F1 released

Daniel Nashed  14 October 2017 19:31:24
There is also a client IF1 for 9.0.1 FP9 which fixes one part of the issue that I reported.
Depending on your configuration MIME messages sent did show up with different fonts on Notes clients.
It happened in edit mode or when the embedded MIME browser was disabled.

What has been fixed is that the IF1 client shows correct fronts. But earlier clients still shows different fonts (for example if you send a mail with sans serif it will show up in serif).

I don't know if that can be fixed at all but IBM is aware of it and is looking into it.

-- Daniel



AYAVAQF7WZ         Fix an issue where sent internet mail shows as "serif" font instead of "san serif"        
JVEKARBEP2         Fixed an issue where the contents are not displayed after editing if a Richtext field contains an image and "Store contents as HTML and MIME" is enabled
YNABANLSUB         Fix an error 4399 "Value Is Out Of Range" when running deleteuser lotusscript        

Domino 9.0.1 FP 9 IF2 available with important fixes

Daniel Nashed  13 October 2017 11:09:53
Two of the issues fixed in IF2 have been discussed before in my blog.
But there are also two other critical issues fixed.

Some of my customers reported DBMT and updall hangs which have been fixed with TDOOAREP8W.
And the Private on first use folder issue also has been reported before.

If you have installed 9.0.1 FP 9 you should update to IF2!


-- Daniel


JPAIAQ5SKW
        PANIC: DbMarkCorrupt! (d:\notefile\admin4.nsf Dbiid: 0x3D91E116 0x3C07FE17)        

JVEKAQSGCC
        Shared, Private on First Use Folder not working as expected in 9.0.1 FP9. It is not possible to view the folder in Designer.        

TDOOAREP8W
        Performances issue (deadlock) with long held lock after update to Domino 9.0.1 FP9.        

YNABANLSUB
        Error 4399 (Value Is Out Of Range) When Running Deleteuser Lotusscript.        

IBM Champion Program Nominations are open

Daniel Nashed  10 October 2017 20:05:55
The IBM champion nominations have just started today.

You can nominate your favorite persons in the community to appreciate what they are doing for the community (--> https://developer.ibm.com/dwblog/2017/ibm-champion-program-nominations/)

Libby just expressed it in short words what a champion makes stand out. Let me quote instead of just passing a link!

-- Daniel

"You may know an IBM Champion if…

The best way to understand the IBM Champions program is to know an IBM Champion. Do you think you know a Champion? Here are 7 questions to ask yourself about your technical community. Do these sound familiar?

  • Someone freely gives you advice in person, on Twitter, on forums…
  • You regularly read the blog of someone for help or shared code…
  • Someone in your organization knows everything there is to know about your IBM software or hardware and makes it all work together and is always the one answering questions…
  • There is someone whose presentations you enjoy and learn from…
  • Someone is putting their own time into organizing an event you value…
  • Someone has positively influenced your choices of what to deploy and when…
  • There is someone whose contribution you would miss if they were to stop…
If you put a name or a face to those 7 questions, you may know an IBM Champion… or someone who should become an IBM Champion!"

See this link for more details about IBM Champions --> https://developer.ibm.com/dwblog/2017/ibm-champion-program-nominations/


Installing C-API Applications on Linux

Daniel Nashed  4 October 2017 12:51:45
When installing binaries on Linux you have to be aware of the directory structure for the files installed in the opt directory.

For installing a servertask the recommended way is to copy it to the Domino binary directory and create a start link.

For myself I created a script that handles installation of servertasks and extension managers because I don't want to do those steps manually and my script comes with a wrapper script that benefits of sudo when installing binaries on my development environment.
Also you have to take care that binaries are not running when replacing them.

Linux does not prevent you from replacing the file but the Domino server might crash after your replaced the binary. This is true for extension managers and also servertasks.
In addition the script does check if the file changed at all before replacing the binary.

So I will add this new script into the "extras" directory of my start script. But let me quickly describe what you would do in general to install a binary correctly and post the first official version of my script.


In general Domino is usually installed by default today in

/opt/ibm/domino

Below that directory there is another directory  /opt/ibm/domino/notes.

Inside that directory we have the latest major version installed (linke 90010 in my case) and a link to the latest version


drwxr-xr-x 4 root root 4096 Apr 24  2014 90010
lrwxrwxrwx 1 root root    5 Sep 26  2015 latest -> 90010

Best practice is always to use the "latest" directory

So the actual directory the binaries located is:

/opt/ibm/domino/notes/latest/linux

This directory contains all the binaries and is the place where you should copy new binaries and make them executable:

cp mybinary  /opt/ibm/domino/notes/latest/linux
chmod 755 /opt/ibm/domino/notes/latest/linux/mybinary


For an extension manager there is nothing else to do. But for a servertask you have to create a startup-link in

/opt/ibm/domino/bin

The directory contains startup links pointing to a startup script for each servertask (OK not all IBM installed servetasks have a startup link so the installer is missing to create the links).
If you environment is setup correctly with the right path settings (for example when running my start-script) that's usually not an issue.


lrwxrwxrwx 1 root root 33 Sep 26  2015 server -> /opt/ibm/domino/bin/tools/startup


This symbolic link points to another symbolic link which finally points to the internal startup script in the binary directory:

lrwxrwxrwx 1 root root   42 Sep 26  2015 startup -> /opt/ibm/domino/notes/latest/linux/startup


Invoking the following commands generates your link for the servertask. Domino creates absolute links. But I prefer relative links.


cd /opt/ibm/domino/bin
ln -s tools/startup mybinary


So what finally happens when you invoke the file on your own or if you are using the load command on the server console, is that Linux starts the internal script leveraging the symbolic link which will set the right environment for your servertask etc and than start the actual binary.

I hope this helps a bit to understand the structure and how to install binaries on Linux.

-- Daniel





#!/bin/sh


check_binary_busy()
{
  if [ ! -e "$1" ]; then
    return 0
  fi

  TARGET_REAL_BIN=`readlink -f $1`
  FOUND_TARGETS=`lsof | awk '{print $9}' | grep "$TARGET_REAL_BIN"`

  if [ -n "$FOUND_TARGETS" ]; then
    return 1
  else
    return 0
  fi
}


install_binary()
{
  SOURCE_BIN=$1

  if [ -z "$SOURCE_BIN" ]; then
    echo "no file specified"
    return 0
  fi

  INSTALL_BIN_NAME=`basename $SOURCE_BIN`

  if [ -z "$INSTALL_BIN_NAME" ]; then
    echo "no file specified"
    return 0
  fi

  TARGET_BIN=$Notes_ExecDirectory/$INSTALL_BIN_NAME

  if [ -e "$TARGET_BIN" ]; then

    cmp -s "$SOURCE_BIN" "$TARGET_BIN"
    if [ $? -eq 0 ]; then
      echo "File did not change -- No update needed"
      return 0
    fi

    if [ ! -w "$TARGET_BIN" ]; then
      echo "Error - Can not update binary '$TARGET_BIN' -- No write permissions"
      return 1
    fi


    check_binary_busy "$TARGET_BIN"

    if [ $? -eq 1 ]; then
      echo "Error - Can not update binary '$TARGET_BIN' -- Binary in use"
      return 1
    fi
  fi
 
  cp -f "$SOURCE_BIN" "$TARGET_BIN"
  chmod 755 "$TARGET_BIN"

  case "$INSTALL_BIN_NAME" in
    *.so)
      echo "Installed '$INSTALL_BIN_NAME' Extension-Manager"
      ;;

    *)
      cd $LOTUS/bin
      ln -f -s tools/startup "$INSTALL_BIN_NAME"
      echo "Installed '$INSTALL_BIN_NAME' Servertask"
      ;;

  esac

  return 0
}


export LOTUS=/opt/ibm/domino
export Notes_ExecDirectory=$LOTUS/notes/latest/linux

install_binary "$1"

exit 0

Known issues with Domino 9.0.1 FP9

Daniel Nashed  27 September 2017 05:38:56

A couple of customers and partners asked me about current known issues with FP9 in my blog and offline.
Beside the issue with the garbage chars fixed in IF1 there are 3 other issues that could prevent you from upgrading to FP9.

There is an issue with private on first use views and folders on the server side which prevents those views and folders to be created.

IBM has a hotfix for this as Sascha already reported in my blog comments.

SPR# JVEKAQSGCC / LO92948: SHARED, PRIVATE ON FIRST USE FOLDER NOT WORKING AS EXPECTED IN 9.0.1 FP9. IT IS NOT POSSIBLE TO VIEW THE FOLDER IN DESIGNER

In addition some customers ran into a hang during view update with DBMT where IBM is working on a solution. One customer already got a test hotfix but I have no confirmed status yet if this finally solved the regression.

SPR TDOOAREP8W Performances issue with long held lock after update to Domino 9.0.1 FP9


IMHO both SPRs are good candidates for an IF2.


I keep you posted what I hear.

-- Daniel


Fix Available: SMTP regression issue in Domino 9.0.1 FP9 can cause malformed headers

Daniel Nashed  16 September 2017 00:43:31
Finally we got IF1 for 9.0.1 FP9 for the issue I reported in an earlier blog post
.
The regression was introduced by a fix that IBM has removed in IF1 (and I got a hotfix earlier as mentioned in an earlier blog post).

The root cause is an issue with malformed headers  -- specially the from header that are generated at message itemization.

Depending on your configuration this causes garbage chars in your headers. In any case some functionality like SMTPVerifyAuthenticatedSender=1 or capturing mail for certain recipients via SMTPSaveImportErrors=3 and SMTPSaveFileFrom=sender  did not work any more.

This is not the final fix. IBM is working on resolving the regression. So this fix along with another agent bug fix is really just a quick fix to allow you to deploy FP9.

I installed the fix on my Linux64 machine which shows up as 9.0.1 FP9 HF 63 and it resolves the regression.

See details here:

http://www.ibm.com/support/docview.wss?uid=swg22008327

IF1 contains the two fixes:

KBRNAQKKK9
        Domino agents crash in the backend in FP8 with a memory overwrite        

JCARAQSJB6
        SMTP regression issue in Domino 9.0.1FP9 can cause malformed headers & prevent Internet mail delivery with SMTPVerifyAuthenticatedSender=1 (technote 2008327)        

Domino Performance issue on some Linux Versions

Daniel Nashed  14 September 2017 12:13:17
When working on a larger Domino migration and consolidation project I ran into an new Linux specific performance issues that might hit some of you depending on your Linux version.
I have tested with current RHEL 7 servers which are not affected.

But on customer site we are using the latest patch level of RHEL 6.9 and I have also seen it with SLES 11 SP2/3. I did not yet test with SLES 12 (maybe someone volunteers to do some testing).


There has been an issue in the 8.5.3 code stream which has been fixed in 8.5.3 FP2.


SPR# PHEY8RJHXR - Fixed a performance issue where creating multiple documents with attachments led to high NETIO delays on Linux, Mac, and IBM i, resulting in slower transactions for other users accessing other databases.


The old issue has been a timing issue between the Domino network stack/listener and the scheduling of the kernel.

The change was to use native pthread semaphores in the Domino network layer.


But already at that time we saw some performance issues with the standard kernel tuning

(see -->
http://blog.nashcom.de/nashcomblog.nsf/dx/runfaster1-for-domino-on-linux.htm for details).

Over time some other changes in the kernel made the default settings used for the CFS process scheduler to not work nicely with Domino in some kernel versions.


I discovered this slow down specially for attachment write transactions when troubleshooting some Windows related issues on the sending side (working on another blog post for the Windows 2008 issue).


But at least on RHEL 6.x and some SLES versions the receiving side can and should be optimized.


For testing I wrote a simple servertask which  creates attachments in a remote database from memory to benchmark the performance.

It turned out that with standard kernel settings for server in a local network we have been able to write with 25 MB/sec.

With the kernel tuning changes we have been able to write with over 100 MB/sec.


Attachment write operations are just one part of the communication but specially when consolidating servers all attachments have to be transferred which will be the bigger part of the data that has to be transferred.


The setting I found is responsible for the CFS scheduler behavior for process scheduling. It specially hits larger transfer operations like attachments (I did not test other transactions types).


By default the setting is set to 12 ms on RHEL 6.9  (take care it is specified in nano seconds). This is causing some timing issues with the Domino network layer.


I have found a recommendation for SAP on Linux which suggested to reduce the value to 1 ms. But in my testing already reducing it to 6 ms did help.


My suggestion would be to set the value to 4 ms.


You can change the parameter via:


echo 4000000 >  /proc/sys/kernel/sched_latency_ns


OR you can permanently set it in /etc/sysctl.conf


kernel.sched_latency_ns = 4000000


Automatically set after boot or run once sysctl -p



Again this setting might not be needed for all Linux versions and should be Domino release independent (I have tested the lates 8.5.3 FP6 versions and 9.0.1 FP8/9).


So you could either set it as a best practice or use my test tool to check you current performance.
My tests with the latest RHEL 7 version did show that even setting the value much higher there, did not have any performance impact.


I am happy to send over the test tool for Windows or Linux. I cannot make it available for download because I don't want to spead the binaries uncontrolled.

But feel free to contact me by mail and I am also interested to see your results, when you test it.

The tool can create documents in any target database and you can specify the number of documents (default is 10).


See detailed test results below.


-- Daniel



-- Test with Default Settings --


cat /proc/sys/kernel/sched_latency_ns

12000000

/local/notesdata $ /opt/ibm/domino/bin/nshobj dsim012\!\!admin/nshobj.nsf

Local Notes/Domino Release 9.0 QMR:1 QMU:9 Hotfix: 0 Fixpack: 0 (0)

Remote Notes/Domino Release 9.0 QMR:1 QMU:9 Hotfix: 0 Fixpack: 0 (0)

Database:   'dsim012!!admin/nshobj.nsf'

Att-Size:   2097152

Chunk-Size: 262144

Count:      10

Total:      814

Minimum:    19

Maximum:    194

Average:    81

MB/Sec:     25,3



-- Test with Modified Settings --


echo 4000000 >  /proc/sys/kernel/sched_latency_ns


/local/notesdata $ /opt/ibm/domino/bin/nshobj dsim012\!\!admin/nshobj.nsf

Local Notes/Domino Release 9.0 QMR:1 QMU:9 Hotfix: 0 Fixpack: 0 (0)

Remote Notes/Domino Release 9.0 QMR:1 QMU:9 Hotfix: 0 Fixpack: 0 (0)

Database:   'dsim012!!admin/nshobj.nsf'

Att-Size:   2097152

Chunk-Size: 262144

Count:      10

Total:      192

Minimum:    16

Maximum:    35

Average:    19

MB/Sec:     107,8


How to resolve synchronization issues that start after upgrading to IBM Traveler 9.0.1.18 (or higher)

Daniel Nashed  9 September 2017 11:21:53
If you are running on Traveler 9.0.1.18 and higher you should read the following support flash technote in detail.

http://www.ibm.com/support/docview.wss?uid=swg22005703

You must read this technote if you are running on 9.0.1.18 and higher.
And with this new information it makes a lot of sense to move to this new version soon.

As mentioned before, IBM changed the default security mode for Traveler.
Traveler uses a run as user feature to ensure that all functionality is invoked in the name of the user.


Therefore Traveler server has to be listed in the trusted server on the security tab of the mail-server (which already caused a yellow status warning on your servers in earlier versions in preparation).

But there are additional requirements for each mail-database to correctly sync with this new security modeil.
Some of them have not been documented in detail before this technote was available. And Traveler 9.0.1.19 has more detailed checking/logging if capabilities in access for a database is missing.
Also there is a fallback per user to the old mode, if not all requirements are full filled for a mail-database.
For example Maximum Internet Access for a mail-database needs to be set to Editor or higher.

The technote describes the requirements and the new error logging in very detail. And also all options that you have to disable the new access mode for the server or per user.

-- Daniel


Traveler 9.0.1.19 with important fixes

Daniel Nashed  8 September 2017 09:15:12
We have been waiting for Traveler 9.0.1.19 for some important fixes and also updates SQL server support and push certificate update:
 
 
  • Support for MS SQL Server 2016 Enterprise Edition.
  • Updated APNS Certificates with expiration 8/1/2018.
  • Improvements for the Run as User Feature.

But the most important changes are for the "Run as User" Feature which has been introduced in 9.0.1.18.
Some of my customers and issues with Traveler profiles which could not be read correctly in some cases.

Beside this fix there are a couple of minor enhancements listed below.

-- Daniel

Fixlist:
APAR # Abstract
LO92524 Sync performance impacted if syncing a large repeating calendar event.
LO92525 Reply notice sent from an FYI recipient for a calendar event when processed on a mobile device.
LO92557 Cancelled event may appear ghosted after the cancel on the iOS Native Calendar application.
LO92638 Add invitee from native iOS Calendar application and the recipient may be added twice to the meeting.
LO92645 Error reading some policy documents when Run as User feature is enabled.
LO92713 User unable to sync when sync request is internally routed from an older server to a 9.0.1.18 or later server.
LO92728 DB Connection exception during migration from Derby to an Enterprise database.
LO92783 Android user incorrectly denied access if specific set of administration settings are enabled.
LO92829 Handle comma character in display name for mail sent from an Outlook client.
LO92881 High CPU may be seen on the database server for one particular SQL query.
LO92897 Update APNS Certificates, new expiration is 8/1/2017.



SSLV3 disabled by default since 9.0.1 FP9

Daniel Nashed  5 September 2017 16:18:57
This change has been discussed a while ago.
Now it was finally time to disable default SSLv3 in Domino.

The SPR did not make it into the fixlist. Thanks Thibaud Maes for your mail!

The change addressed by SPR # DKENAKNSEG will affect all connection types that utilise the native Domino security stack such as HTTPS and secure DIIOP.

If you still need SSLv3 you need this new notes.ini parameter ENABLE_SSLV3=1

There are not many applications left that need SSLV3 ...

Daniel


  • [IBM Lotus Domino]
  • [Domino on Linux]
  • [Nash!Com]
  • [Daniel Nashed]