Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

Domino on Linux Performance Tweaks with current Kernels

Daniel Nashed  27 February 2012 09:20:40

We found a performance issue with SLES 11 and also RHEL 6. Due to a new process scheduler (CFS) introduced there seem to be timing issues with some new functionality.

The new scheduler CFS is the Completely Fair Scheduler which has some new functionality.
From what we figured out so far some functionality of CFS causes timing issues with the Domino server.
The Domino server has many active threads and processes which seem to not work well with the standard settings of CFS.

What happens is that under some load the response time of the server increases. There are some transactions that take 100 ms longer than usual.
Domino Development are still working with the Novell on identifying the root cause. We have an escalated PMR open with IBM and and a case open Novell and they are still trying to figure out what in detail is going wrong.

This recommended workaround has no negative affect on the system that we have seen, nor do we expect a negative affect. 

The work-around we found is to disable some functionality in CFS.

As root, run the command

echo "0" > /proc/sys/kernel/sched_features

This will set the CFS features to 0 until the system is rebooted. If the system is rebooted you will need to run the command again.

I have small test application "nshopen" that opens many documents and reports the response time.
Let me know if you want this test application for performance testing your Linux machines with and without the fix.

We are still trying to find out what exactly is going wrong with the timing and there is some other ongoing work.
I will post an update when we find more details. For now you should use the mentioned tuning.

-- Daniel
Comments

1JensK  28.02.2012 15:44:07  Domino on Linux Performance Tweaks with current Kernels

Hi Daniel.

Interesting. Do you have indications wether this issue relates to certain kernel versions or to all kernel versions of SLES 11?

We are not experiencing major performance issues at present althought some times the CPU util rises to 100%.

How do we detect if the cause of this rise in CPU is the scheduler?

We are currently running Domino 8.5.1 32 bit on SLES 11sp1 64bit intel architecture on VMware. Kernel version 2.6.32.49-0.3

Regards Jens

2Daniel Nashed  28.02.2012 16:14:40  Domino on Linux Performance Tweaks with current Kernels

@Jens, it's not a matter of high CPU usage. it's more a scheduling issue with the CFS scheduler. we have a delay in the server process when processing requests. this is more a timing issue than a high CPU issue.

you will see the delay when you look into the client_clock on your Notes Client or use my nshopen tool to measure the response time of "NoteOpen" transactions on cliend side.

All kernels with the CFS scheduler are affected.

Another note: You should update to a more recent Domino version! I would recommend that you move to 8.5.3 or maybe way when FP1 ships next month.

-- Daniel

3Wee Wah  14.03.2012 8:44:59  Domino on Linux Performance Tweaks with current Kernels

I would like to have your test program. We are already running R8.5.3. I'm noticing a lot of agent manager timeouts for no apparent reason. Thanks.

4Gregor  16.03.2012 10:48:44  Domino on Linux Performance Tweaks with current Kernels

Im working almost for three years to solve the issue why domino under linux guest (esxi) is much slower than windows guest (esxi). Something in the tcp configuration or data transfer handling is not working properly in domino. Other file transfers are working fast.

Do you have any sugesstions? I probably tried them almost all.

regards,

Gregor

5Pablo Jacobovici  24.06.2016 4:35:26  Domino on Linux Performance Tweaks with current Kernels

Hi Daniel, first of all congratulations for your blog, technically speaking it is really excellent.

I have encountered a similar issue in RHEL 6.4 64 bits with an application with a high user concurrency where every time users accessed the application run an agent on the WebQueryOpen event.

The performance of this application was really bad.

After analyzing the problem for several months with ibm, they request me to define FullTrimFix=0 in the notes.ini.

Although the documentation i found about this variable refers to arrays, they told me that also changes the way the semaphores are handled in memory when objects are created and destroyed very frequently.

This change lowered the number of context switches considerably.

They went from 200,000 to 10,000 approximately and greatly improved performance.

The improvement was approximately 50 to 1.

In tests I've done this works with version 8.5.3 and 9.0.1.

I have not tried other versions but surely work or at least it worth a try.

Again congratulations for your blog.

6Daniel Nashed  27.06.2016 11:27:57  Domino on Linux Performance Tweaks with current Kernels

@Pablo, Thanks for your comment! Interesting! I have sent you an email offline to discuss some details.


  • [IBM Lotus Domino]
  • [Domino on Linux]
  • [Nash!Com]
  • [Daniel Nashed]