Domino Performance issue on some Linux Versions
Daniel Nashed – 14 September 2017 10:13:17
When working on a larger Domino migration and consolidation project I ran into an new Linux specific performance issues that might hit some of you depending on your Linux version. I have tested with current RHEL 7 servers which are not affected.
But on customer site we are using the latest patch level of RHEL 6.9 and I have also seen it with SLES 11 SP2/3. I did not yet test with SLES 12 (maybe someone volunteers to do some testing).
There has been an issue in the 8.5.3 code stream which has been fixed in 8.5.3 FP2.
SPR# PHEY8RJHXR - Fixed a performance issue where creating multiple documents with attachments led to high NETIO delays on Linux, Mac, and IBM i, resulting in slower transactions for other users accessing other databases.
The old issue has been a timing issue between the Domino network stack/listener and the scheduling of the kernel.
The change was to use native pthread semaphores in the Domino network layer.
But already at that time we saw some performance issues with the standard kernel tuning
(see --> http://blog.nashcom.de/nashcomblog.nsf/dx/runfaster1-for-domino-on-linux.htm for details).
Over time some other changes in the kernel made the default settings used for the CFS process scheduler to not work nicely with Domino in some kernel versions.
I discovered this slow down specially for attachment write transactions when troubleshooting some Windows related issues on the sending side (working on another blog post for the Windows 2008 issue).
But at least on RHEL 6.x and some SLES versions the receiving side can and should be optimized.
For testing I wrote a simple servertask which creates attachments in a remote database from memory to benchmark the performance.
It turned out that with standard kernel settings for server in a local network we have been able to write with 25 MB/sec.
With the kernel tuning changes we have been able to write with over 100 MB/sec.
Attachment write operations are just one part of the communication but specially when consolidating servers all attachments have to be transferred which will be the bigger part of the data that has to be transferred.
The setting I found is responsible for the CFS scheduler behavior for process scheduling. It specially hits larger transfer operations like attachments (I did not test other transactions types).
By default the setting is set to 12 ms on RHEL 6.9 (take care it is specified in nano seconds). This is causing some timing issues with the Domino network layer.
I have found a recommendation for SAP on Linux which suggested to reduce the value to 1 ms. But in my testing already reducing it to 6 ms did help.
My suggestion would be to set the value to 4 ms.
You can change the parameter via:
echo 4000000 > /proc/sys/kernel/sched_latency_ns
OR you can permanently set it in /etc/sysctl.conf
kernel.sched_latency_ns = 4000000
Automatically set after boot or run once sysctl -p
Again this setting might not be needed for all Linux versions and should be Domino release independent (I have tested the lates 8.5.3 FP6 versions and 9.0.1 FP8/9).
So you could either set it as a best practice or use my test tool to check you current performance.
My tests with the latest RHEL 7 version did show that even setting the value much higher there, did not have any performance impact.
I am happy to send over the test tool for Windows or Linux. I cannot make it available for download because I don't want to spead the binaries uncontrolled.
But feel free to contact me by mail and I am also interested to see your results, when you test it.
The tool can create documents in any target database and you can specify the number of documents (default is 10).
See detailed test results below.
-- Daniel
-- Test with Default Settings --
cat /proc/sys/kernel/sched_latency_ns
12000000
/local/notesdata $ /opt/ibm/domino/bin/nshobj dsim012\!\!admin/nshobj.nsf
Local Notes/Domino Release 9.0 QMR:1 QMU:9 Hotfix: 0 Fixpack: 0 (0)
Remote Notes/Domino Release 9.0 QMR:1 QMU:9 Hotfix: 0 Fixpack: 0 (0)
Database: 'dsim012!!admin/nshobj.nsf'
Att-Size: 2097152
Chunk-Size: 262144
Count: 10
Total: 814
Minimum: 19
Maximum: 194
Average: 81
MB/Sec: 25,3
-- Test with Modified Settings --
echo 4000000 > /proc/sys/kernel/sched_latency_ns
/local/notesdata $ /opt/ibm/domino/bin/nshobj dsim012\!\!admin/nshobj.nsf
Local Notes/Domino Release 9.0 QMR:1 QMU:9 Hotfix: 0 Fixpack: 0 (0)
Remote Notes/Domino Release 9.0 QMR:1 QMU:9 Hotfix: 0 Fixpack: 0 (0)
Database: 'dsim012!!admin/nshobj.nsf'
Att-Size: 2097152
Chunk-Size: 262144
Count: 10
Total: 192
Minimum: 16
Maximum: 35
Average: 19
MB/Sec: 107,8
- Comments [3]
![[HCL Domino]](../lotus-domino.gif){kind=small}
![[Domino on Linux]](../domino-linux.gif){kind=small}
![[Nash!Com]](../nashcom.gif){kind=small}
![[Daniel Nashed]](../daniel-nsh.gif){kind=small}