Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

 
alt

Daniel Nashed

 

Domino Performance issue on some Linux Versions

Daniel Nashed  14 September 2017 10:13:17
When working on a larger Domino migration and consolidation project I ran into an new Linux specific performance issues that might hit some of you depending on your Linux version.
I have tested with current RHEL 7 servers which are not affected.

But on customer site we are using the latest patch level of RHEL 6.9 and I have also seen it with SLES 11 SP2/3. I did not yet test with SLES 12 (maybe someone volunteers to do some testing).


There has been an issue in the 8.5.3 code stream which has been fixed in 8.5.3 FP2.


SPR# PHEY8RJHXR - Fixed a performance issue where creating multiple documents with attachments led to high NETIO delays on Linux, Mac, and IBM i, resulting in slower transactions for other users accessing other databases.


The old issue has been a timing issue between the Domino network stack/listener and the scheduling of the kernel.

The change was to use native pthread semaphores in the Domino network layer.


But already at that time we saw some performance issues with the standard kernel tuning

(see -->
http://blog.nashcom.de/nashcomblog.nsf/dx/runfaster1-for-domino-on-linux.htm for details).

Over time some other changes in the kernel made the default settings used for the CFS process scheduler to not work nicely with Domino in some kernel versions.


I discovered this slow down specially for attachment write transactions when troubleshooting some Windows related issues on the sending side (working on another blog post for the Windows 2008 issue).


But at least on RHEL 6.x and some SLES versions the receiving side can and should be optimized.


For testing I wrote a simple servertask which  creates attachments in a remote database from memory to benchmark the performance.

It turned out that with standard kernel settings for server in a local network we have been able to write with 25 MB/sec.

With the kernel tuning changes we have been able to write with over 100 MB/sec.


Attachment write operations are just one part of the communication but specially when consolidating servers all attachments have to be transferred which will be the bigger part of the data that has to be transferred.


The setting I found is responsible for the CFS scheduler behavior for process scheduling. It specially hits larger transfer operations like attachments (I did not test other transactions types).


By default the setting is set to 12 ms on RHEL 6.9  (take care it is specified in nano seconds). This is causing some timing issues with the Domino network layer.


I have found a recommendation for SAP on Linux which suggested to reduce the value to 1 ms. But in my testing already reducing it to 6 ms did help.


My suggestion would be to set the value to 4 ms.


You can change the parameter via:


echo 4000000 >  /proc/sys/kernel/sched_latency_ns


OR you can permanently set it in /etc/sysctl.conf


kernel.sched_latency_ns = 4000000


Automatically set after boot or run once sysctl -p



Again this setting might not be needed for all Linux versions and should be Domino release independent (I have tested the lates 8.5.3 FP6 versions and 9.0.1 FP8/9).


So you could either set it as a best practice or use my test tool to check you current performance.
My tests with the latest RHEL 7 version did show that even setting the value much higher there, did not have any performance impact.


I am happy to send over the test tool for Windows or Linux. I cannot make it available for download because I don't want to spead the binaries uncontrolled.

But feel free to contact me by mail and I am also interested to see your results, when you test it.

The tool can create documents in any target database and you can specify the number of documents (default is 10).


See detailed test results below.


-- Daniel



-- Test with Default Settings --


cat /proc/sys/kernel/sched_latency_ns

12000000

/local/notesdata $ /opt/ibm/domino/bin/nshobj dsim012\!\!admin/nshobj.nsf

Local Notes/Domino Release 9.0 QMR:1 QMU:9 Hotfix: 0 Fixpack: 0 (0)

Remote Notes/Domino Release 9.0 QMR:1 QMU:9 Hotfix: 0 Fixpack: 0 (0)

Database:   'dsim012!!admin/nshobj.nsf'

Att-Size:   2097152

Chunk-Size: 262144

Count:      10

Total:      814

Minimum:    19

Maximum:    194

Average:    81

MB/Sec:     25,3



-- Test with Modified Settings --


echo 4000000 >  /proc/sys/kernel/sched_latency_ns


/local/notesdata $ /opt/ibm/domino/bin/nshobj dsim012\!\!admin/nshobj.nsf

Local Notes/Domino Release 9.0 QMR:1 QMU:9 Hotfix: 0 Fixpack: 0 (0)

Remote Notes/Domino Release 9.0 QMR:1 QMU:9 Hotfix: 0 Fixpack: 0 (0)

Database:   'dsim012!!admin/nshobj.nsf'

Att-Size:   2097152

Chunk-Size: 262144

Count:      10

Total:      192

Minimum:    16

Maximum:    35

Average:    19

MB/Sec:     107,8


Comments

1Jesper Kiaer  18.09.2017 12:22:56  Domino Performance issue on some Linux Versions

Thanks for the tip, have just implemented it and it does seem faster.

On a same note ...did you see my blogentry on the TCP notes.ini setting?

{ Link }

I made a huge change in performance on my server when downloading attachments.

2Daniel Nashed  19.09.2017 5:12:27  Domino Performance issue on some Linux Versions

@Jesper, yes I saw your blog posting and I just sent you an email to your mail address listed in your blog post reply to ask for details and give some background infos about the port line.

3Hubertus  26.09.2017 14:59:07  Domino Performance issue on some Linux Versions

Hi Daniel,

I want to test my servers.

Would you share your tool with me?

Thanks a lot.

Kind regards

Hubertus

Links

    Archives


    • [HCL Domino]
    • [Domino on Linux]
    • [Nash!Com]
    • [Daniel Nashed]