Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

    Taking full benefit of RAM for File-System Cache with Domino on W64

    Daniel Nashed  13 March 2014 12:48:59

    A long time ago I already blogged about the changes IBM introduced for the file-system cache.
    And I ran into this in customer situations many times. I have described it in my IBM Connect session but because I got questions about it again, I think it makes sense to mention it again.

    The default settings they implemented might impact you when you add a lot of RAM to your Domino server.

    We have seen dramatical reduction of read I/O when adding a lot of RAM to the Windows machine because Windows 64 can leverage the 64bit address space for taking all the remaining memory for file-system case.

    But by default there is a very high physical memory limit for the file-system cache.– It will try to use all memory which can cause Domino Memory to be swapped out

    On startup of the Domino server the W64 call “SetSystemFileCacheSize()” is used to limit the cache.
    Since Domino 8 and higher ships a 64bit helper binary “cacheset.exe” to set the cache size for Domino 32bit. Domino 64bit has this call integrated into the core code.

    When the code is executed the system privilege “SE_INCREASE_QUOTA_NAME” is needed (See TN #1391477 for details).

    By default the value is set to  30% of memory. That would only work well with a machine with around 8 GB of memory, but even there some tuning might make sense because Domino will usually allocate less than 4 GB of memory.

    So you can tune the percentage used via notes.ini MEM_FSCachePercentMem=n
    The settings depends on the RAM and the memory that Domino needs in your environment .

    Example: 16 GB RAM, 6 GB reserved for Domino/OS = MEM_FSCachePercentMem=65

    You can check the current settings with “cacheset.exe -g”
    Here is the output from a machine with 8 GB RAM without any settings after the Domino Server has been started once.

    cacheset.exe -g
    Existing file system cache values are minSizeRead 824488 kb, maxSizeRead 2473264 kb, flags 5

    This is really a parameter that you have to look at when you run Domino 32bit/64bit on Windows!

    -- Daniel


    Comments

    1Richard Moy  13.03.2014 13:56:35  Taking full benefit of RAM for File-System Cache with Domino on W64

    Sorry for being naive, but does this have any impact for a Linux box?

    2Daniel Nashed  14.03.2014 9:32:02  Taking full benefit of RAM for File-System Cache with Domino on W64

    This is only for Windows 64 with Domino 32bit or 64bit.

    Linux is not affected and in fact on Linux side the file-system caching works completely different -- and IMHO more effective.

    On Windows the file-system cache is implemented using normal virtual memory.

    That means the file-system memory competes with the memory used for applications and could cause that applications page out their data if the cache grows.

    That's why we need a parameter like this to be set. There is also no registry value. An application has to use this system call which will than set the memory limitations globally for the whole Windows machine.

    -- Daniel

    3Carsten Lührmann  25.03.2014 16:43:47  Taking full benefit of RAM for File-System Cache with Domino on W64

    So true, we also saw a tremendous reduction of disk IO on our mail servers.

    Maybe one tip for everyone who don't know if buying more memory will help: Domino leverages the so-called Copy-Interface for file system access (more on that here: { Link } There are performance counters that can give you quite some nice insight into how your cache performs, most importantly "Cache -> Copy Read Hits %". It basically tells you how many cachable filesystem accesses are served from the cache. Be sure to check the relevance by also looking at "Cache -> Copy Reads / sec" and "Logical Disk ->Disk Transfers / sec" counters. Domino seems to produce a pretty high ratio of cachable requests from my observations.

    By the way: cacheset.exe can be used while the server is running, so you can experiment with different settings and see the results (almost) instantly.

    4Daniel Nashed  26.03.2014 12:20:41  Taking full benefit of RAM for File-System Cache with Domino on W64

    right the cacheset is only used to run the api call. it's a simple tool to read and set the settings.

    in case you run a partitioned server both servers need to have the same settings. but you have to take into account the additonal memory needed for the second server.

    thanks for adding info for the counters. IMHO this is a runfaster=1 parameter specially when the machine has a lot of RAM. without it the cache would be too limited.

    -- daniel

    5Christian Henseler  26.03.2014 13:40:19  Taking full benefit of RAM for File-System Cache with Domino on W64

    working link for the TechNet article File Cache Performance and Tuning:

    { Link }

    It's an Windows 2000 article, dated back to 2002.

    Microsoft Performance Tuning Guides for Windows Server 2012 R2 can be found here:

    { Link }


    • [IBM Lotus Domino]
    • [Domino on Linux]
    • [Nash!Com]
    • [Daniel Nashed]