Daniel Nashed's Blog

NSD hangs because of LSOF 100% CPU use in Domino container - because of too high open file limits

I ran into this on some of my systems. When invoking a NSD it never finishes.
On Linux NSD is a shell script leveragin the Linux GNU Debugger (gdb) and also lsof to list open files.

It looks like some Docker versions add too high values into security limits for open files. This causes lsof to hang.

Reducing the security limits via /etc/security/limits.conf does not help.
The limits are set when starting the container already.

But a simple solution I found is to set the security limits in the Docker configuration.

You really should take action and validate your settings.
If you have this misconfiguration a server will not be able to recover from a crash, which invokes a full NSD including lsof.

-- Daniel


/etc/docker/daemon.json

{
  "default-ulimits": {
    "nofile": {
      "Hard": 80000,
      "Name": "nofile",
      "Soft": 80000
    }
  }
}


How to query the values

The Domino start script writes a log which first dumps the envrionment including secuity limits.

the start script command "domino log" opens the log in an editor and you can examine the first part of the log to see the security limit configuration

But you can also check manually inside the container:

ulimit -aH
ulimit -a


In my case the limit was set to high by the service and the configuration above.
You should check your environment to ensure you are not running into this pitfall.

• Comments [0]