Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

alt

Daniel Nashed

Taking full benefit of RAM for File-System Cache with Domino on W64

Daniel Nashed – 13 March 2014 11:48:59

A long time ago I already blogged about the changes IBM introduced for the file-system cache.
And I ran into this in customer situations many times. I have described it in my IBM Connect session but because I got questions about it again, I think it makes sense to mention it again.

The default settings they implemented might impact you when you add a lot of RAM to your Domino server.

We have seen dramatical reduction of read I/O when adding a lot of RAM to the Windows machine because Windows 64 can leverage the 64bit address space for taking all the remaining memory for file-system case.

But by default there is a very high physical memory limit for the file-system cache.– It will try to use all memory which can cause Domino Memory to be swapped out

On startup of the Domino server the W64 call “SetSystemFileCacheSize()” is used to limit the cache.
Since Domino 8 and higher ships a 64bit helper binary “cacheset.exe” to set the cache size for Domino 32bit. Domino 64bit has this call integrated into the core code.

When the code is executed the system privilege “SE_INCREASE_QUOTA_NAME” is needed (See TN #1391477 for details).

By default the value is set to  30% of memory. That would only work well with a machine with around 8 GB of memory, but even there some tuning might make sense because Domino will usually allocate less than 4 GB of memory.

So you can tune the percentage used via notes.ini MEM_FSCachePercentMem=n
The settings depends on the RAM and the memory that Domino needs in your environment .

Example: 16 GB RAM, 6 GB reserved for Domino/OS = MEM_FSCachePercentMem=65

You can check the current settings with “cacheset.exe -g”
Here is the output from a machine with 8 GB RAM without any settings after the Domino Server has been started once.

cacheset.exe -g
Existing file system cache values are minSizeRead 824488 kb, maxSizeRead 2473264 kb, flags 5

This is really a parameter that you have to look at when you run Domino 32bit/64bit on Windows!

-- Daniel


Comments

1Richard Moy  13.03.2014 12:56:35  Taking full benefit of RAM for File-System Cache with Domino on W64

Sorry for being naive, but does this have any impact for a Linux box?

2Daniel Nashed  14.03.2014 8:32:02  Taking full benefit of RAM for File-System Cache with Domino on W64

This is only for Windows 64 with Domino 32bit or 64bit.

Linux is not affected and in fact on Linux side the file-system caching works completely different -- and IMHO more effective.

On Windows the file-system cache is implemented using normal virtual memory.

That means the file-system memory competes with the memory used for applications and could cause that applications page out their data if the cache grows.

That's why we need a parameter like this to be set. There is also no registry value. An application has to use this system call which will than set the memory limitations globally for the whole Windows machine.

-- Daniel

3Carsten Lührmann  25.03.2014 15:43:47  Taking full benefit of RAM for File-System Cache with Domino on W64

So true, we also saw a tremendous reduction of disk IO on our mail servers.

Maybe one tip for everyone who don't know if buying more memory will help: Domino leverages the so-called Copy-Interface for file system access (more on that here: { Link } There are performance counters that can give you quite some nice insight into how your cache performs, most importantly "Cache -> Copy Read Hits %". It basically tells you how many cachable filesystem accesses are served from the cache. Be sure to check the relevance by also looking at "Cache -> Copy Reads / sec" and "Logical Disk ->Disk Transfers / sec" counters. Domino seems to produce a pretty high ratio of cachable requests from my observations.

By the way: cacheset.exe can be used while the server is running, so you can experiment with different settings and see the results (almost) instantly.

4Daniel Nashed  26.03.2014 11:20:41  Taking full benefit of RAM for File-System Cache with Domino on W64

right the cacheset is only used to run the api call. it's a simple tool to read and set the settings.

in case you run a partitioned server both servers need to have the same settings. but you have to take into account the additonal memory needed for the second server.

thanks for adding info for the counters. IMHO this is a runfaster=1 parameter specially when the machine has a lot of RAM. without it the cache would be too limited.

-- daniel

5Christian Henseler  26.03.2014 12:40:19  Taking full benefit of RAM for File-System Cache with Domino on W64

working link for the TechNet article File Cache Performance and Tuning:

{ Link }

It's an Windows 2000 article, dated back to 2002.

Microsoft Performance Tuning Guides for Windows Server 2012 R2 can be found here:

{ Link }

6Michal Novacek  11.02.2020 13:35:10  Taking full benefit of RAM for File-System Cache with Domino on W64

Hi Daniel, I bumped into the problem with FS cache on Windows with 64GB RAM. I could not find exact meaning of settings of MEM_FSCachePercentMem option. Default setting of 30% means that 70% of physical RAM is allocated for Windows filecache and 30% remains for apps? So when I set MEM_FSCachePercentMem to for example 40% that means I restricted Windows filecache to 40% of physical memory only? Or vice versa? 40% means only 40% is allocated for apps? And Windows filecache is "good" or "bad" because I dont understand if Domino uses own cache or that Windows cache. Thx

7Daniel Nashed  11.02.2020 15:21:30  Taking full benefit of RAM for File-System Cache with Domino on W64

@Michal,

It's the other way round!! the percentage is the amount used for file-caching .. on a large server you want maximum of 8 GB for Domino or even less..

By default with 30% only 21GB is used for file-system cache... You can set the file-system cache to 85 or 90% !!

That would be MEM_FSCachePercentMem= 90 :-)

8Massimo Nadalin  11.08.2023 12:17:00  Taking full benefit of RAM for File-System Cache with Domino on W64

Hi Daniel

File System Cache on Windows Server 2022 may cause some issues:

https://help.hcltechsw.com/domino/12.0.0/admin/wn_notesini_for_ws2022.html

HCL recommends to entirely disable it with OS_DISABLE_CACHESET=1

9Daniel Nashed  16.08.2023 20:12:59  Taking full benefit of RAM for File-System Cache with Domino on W64

Hi Massimo,

yes this is a new problem Microsoft introducted. In 12.0.2 FP1 and higher the parameter is off by default for Win2019 and Win2022.

I have been part of the orignal problem analysis at the time when this change has been introduced to fix another Microsoft file-system issue.

Now they broke the workaround and it is not 100% clear to me if on very large size of open files, we don't run into the same problem again we had initially when the work-around to limit the file-system cache has been introduced.

I am not really a big Windows fan. If you are concerned about performance and security, Linux might be the better choice specially for larger servers.

-- Daniel

Links

    Archives


    • [HCL Domino]
    • [Domino on Linux]
    • [Nash!Com]
    • [Daniel Nashed]