Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

 
alt

Daniel Nashed

 

New Performance Problem with Domino on Windows 2008 64bit

Daniel Nashed  1 October 2010 16:05:16
Since Domino is supported on Windows 64bit there have been a couple of issues in the performance area. Most of them have been already technoted and IBM has build work-arounds.

For example the problem with the large amount of memory used by the cache manager which competed with memory allocations by Domino. To prevent the cache manager from using too much physical memory IBM uses the system call SetSystemFileCacheSize(..) (ref: http://msdn.microsoft.com/en-us/library/aa965240%28VS.85%29.aspx). But it turned out this only fixed part of the underlying issue. To understand what is going wrong you have to understand how the Windows Cache Manager works internally.

How the Windows Cache Manager Works

The Windows Cache Manager uses Virtual Memory to store the cache pages one by one for each open file in block sizes of 256 kb. Allocated pages stay in Virtual Memory for a while depending on multiple factors. The most important factor is if the file is still open by any running process. The Virtual Memory Manager manages the physical memory used for the cache. The amount of memory is the working set of the cache manager specified via the SetSystemFileCacheSize Windows API call.

Domino 8.5.x sets this value by default to 30% of the physical memory which is a good starting point. But on the other hand, if the Virtual Memory for the file-system cache becomes larger and larger the Virtual Memory manager will throw out many pages but keeps the open files still in the Virtual Memory of the File-System cache.

IMHO this is not how a cache manager should be implemented. But in normal conditions this is still not a big issue because the file-system cache should not grow too much. But this becomes an issue when a lot of data needs to be cached. The biggest limitation we ran into the 1 TB limit for the Virtual Memory used by the cache manager.

For example if you have a lot open files with a lot read activity (for example during a Domino backup) the pages read stay in Virtual Memory of the cache manager. The total limit is 1 TB even the physical memory might be only a couple of GBs.
So even the data stays in Virtual Memory of the cache manage it might be already removed by the Virtual memory manager because the working-set of the cache manager is far smaller than the Virtual Memory.

What can go wrong with the cache manager

There have been a couple of issues that caused high CPU load when Windows needed to scan thru all the Virtual Memory pages. Some of the general issues have been addressed in Windows 2008 R2. But even Win2008 R2 still needs hotfixes which you need to manually apply. Still with all the fixes the file-system cache can become very ineffective. Depending on what fixes you have installed the problem shows up in different ways.
Either you have a temporary high CPU load (without the fixes) or at some point the I/O rate drops dramatically.

What happens in case of Domino

It turned out that Microsoft uses different cache strategies for different ways files are opened. Domino 7 and higher (also the current 8.5.2 version) uses the following flag to open all NSF files:

FILE_FLAG_RANDOM_ACCESS
(0x10000000)
Access is intended to be random. The system can use this as a hint to optimize file caching.
(reference:
http://msdn.microsoft.com/en-us/library/aa363858%28VS.85%29.aspx)

IBM is using this flag to reduce the amount of read ahead caching because NSF files are more or less read in a random way. But it turned out that in Windows 2008 64bit this flag triggers that the file blocks that are read stay in the cache until the file is closed.
Because Domino keeps file open in the Database Cache (dbcache) for performance reasons it takes quite a long time until the cache is released. Especially during a backup of a larger Domino server the file-system cache will grow dramatically and causes performance issues.

I have done a lot of testing with different ways opening files. It turned out that the first file-handle opened with the
FILE_FLAG_RANDOM_ACCESS causes that all following file reads of all other processes are stay in the cache for a long time. This explains the high cache usage we have seen with Domino also in earlier versions.

The documentation about the flag is not really clear on the Microsoft side and the behaviour changed with Windows 2008 64bit. Even when reading the Windows Internal documentation this is not really clear.


But it turned out that even with all the fixes in Windows 2008 the file-system cache still works very ineffective when FILE_FLAG_RANDOM_ACCESS is used.

Solution

In our customer situation we got a hotfix to disable the attribute when opening Notes databases. All the performance tests have shown that this is not just the solution for our particular issue with backup performance but also helps to increase performance in other situations. IBM is currently checking if this flag can be disabled in a future fixpack of Domino. We got a SPR #KBRN899NF6 and a hotfix to provide a notes.ini variable to disable the FILE_FLAG_RANDOM_ACCESS for now.

Update: It is important that you enabled the following notes.ini parameter once you have installed the hotfix:
Disable_Random_RW_File_ATTR=1


If you are running on Win2008 with a data volume above 500 GB you request this fix. Hopefully we get this chance into the next Fixpack

Support is working on a technote but I wanted to give you a heads up and provide some details about the background of the problem.

Let me know if you have any questions or feedback.

-- Daniel





Comments

1Erik Brooks  01.10.2010 17:20:29  New Performance Problem with Domino on Windows 2008 64bit

Wow, great write up. This is very good to know.

Assuming backup software was off, would a "dbcache flush" on the console cause Domino to release the handles and free up the cache memory?

2Daniel Nashed  02.10.2010 10:46:58  New Performance Problem with Domino on Windows 2008 64bit

@Eric, yes a "dbache flush" will close all open cache handles and once a file is closed by the backup application the file-system cache would be released.

Actually reducing the number of cache entries and also a manual flush regularly during backup was our first work-around but is not really a good solution for the problem.

-- Daniel

3Lars Berntrop-Bos  11.10.2010 12:26:52  New Performance Problem with Domino on Windows 2008 64bit

Would this impact Windows 7 x64 running Notes client or Domino?

4Daniel Nashed  11.10.2010 19:55:39  New Performance Problem with Domino on Windows 2008 64bit

@Lars,

Win7 64bit is also affected. For a larger server I would not recommend to use Win7 anyway. You need a server OS to be supported.

The main issues occure when large databases are used. But the cache in general works different if the random flag is not used.

In general this could impact also a Notes client. But if you have smaller databases and a lot of RAM the current mode could also give a benefit -- because most of the NSF would be in cache.

-- Daniel

5Mathieu  13.10.2010 10:14:13  New Performance Problem with Domino on Windows 2008 64bit

I have applied this hotfix to one of our server (note that upgrading to 8.5.1 FP4 is a requirement) but IBM Support did not provide me with the NOTES.ini setting.

Could you give me more details about it?

Thanks.

6Daniel Nashed  13.10.2010 16:35:40  New Performance Problem with Domino on Windows 2008 64bit

@Mathieu,

you mean you requested SPR #KBRN899NF6 and they did not give you the notes.ini setting?

the fix does only work with the notes.ini setting. acutally the title of the SPR does include the notes.ini setting

SPR# KBRN899NF6: Disable_Random_RW_File_ATTR notes.ini variable does not work

So for the current fix in the context of this SPR you need to enable the following notes.ini setting

Disable_Random_RW_File_ATTR=1

The setting was implemented when the Random flag was introduced in Domino 7.

But the notes.ini parameter was undocumented and never worked for some reasons.

The SPR was raised when we needed to be able to disable the random file open flag.

We should check offline why support did not give you the notes.ini parameter.

-- Daniel

7Dirk  18.10.2010 8:46:39  New Performance Problem with Domino on Windows 2008 64bit

Hello Daniel,

now, after I've installed the Hotfix my server suht down normally. No high CPU usage any longer.

Thank you for this information!

(Win 2008 R2 with Domino 8.5.2 32Bit)

8Daniel Nashed  29.11.2010 14:26:46  New Performance Problem with Domino on Windows 2008 64bit

@Dave, as long you have not more than 200-300 GB of space it will not give you a huge overhead.

but if you are running a larger environment you should get the fix.

-- Daniel

9dave  29.11.2010 14:32:57  New Performance Problem with Domino on Windows 2008 64bit

I have Domino 8.5.2 on Windows 2008 x64 (non R2) and 16 GB of memory.

Will this problem affect me?

10Daniel L  09.12.2010 12:05:53  New Performance Problem with Domino on Windows 2008 64bit

We have still the same problem with:

Lotus Domino Server 64 Bit Release 8.5.2 HF252 for Windows/64

Our Monitoring system "Solarwinds" is detecting 100% RAM load on every Domino server.

As I know we have to wait for a solution at least until the release of Domino 8.5.3

Cheers

Daniel L

11Daniel Nashed  09.12.2010 21:44:57  New Performance Problem with Domino on Windows 2008 64bit

@Daniel, does your current hotfix contain the fix for the notes.ini variable and did you set the notes.ini parameter?

There will be no other fix in 8.5.3 for the random flag but it will be off by default.

Also you say that your machine is showing 100% use of memory by Domino. That sounds like it is not the cache filling up the memory but Domino.

Domino does usually not allocate more than around 4 GB RAM in total. In most cases even less.

So it would be good to know yout memory values from the machine itself (from performance monitor for example).

Feel free to ping me offline to send some trace data. I am not sure you have the same problem at least your description does not match.

-- Daniel

12Iustinn  24.01.2011 12:06:08  New Performance Problem with Domino on Windows 2008 64bit

Same problem, Domino 8.5.2 FP1 x64 on win 2008 r2 x64.

100%load on dbcache flush.

100% load on "quit/restart" and system hangs for about 45min! (during this time the system is inoperable)

After the notes.ini parameter was applied, the system hangs for only 30-40sec during quit.

Still NOT ok + since no more caching is done, performance should decrease, write? ...

13Max  22.08.2011 12:39:47  New Performance Problem with Domino on Windows 2008 64bit

Hello Daniel,

we have a similar issue, but it is happening on a file server where large NSF files are stored. In this case, the Domino servers run in a datacenter, and the users access a local file server where there NSF replicas are stored. Besides this, we see exactly the same behavior, the fileserver is running Windows 2008 R2 x64, with 4 GB or RAM. Installing the hotfix didn't help yet, we are considering about using the notes.ini additional parameter. Would it work/apply in this case?

Your help/advice is much appreciated.

Thanks,

Max

14liamsi  02.11.2011 19:04:16  New Performance Problem with Domino on Windows 2008 64bit

Hello Daniel,

I have gone though many of your blogs reallys great helpful tips just need some help. As of now we are running lotus domino 8.5.2FP3 HF119 on windows server 2008r2 64 bit VMWare. here the issue is after putting down the domino we are facing server performance issue not able to start domino and no other tasks on the server.

15liamsi  02.11.2011 19:05:54  New Performance Problem with Domino Traveler on Windows 2008 64bit

Our traveler server SAI is always zero(0) even the resource free what will be the issue.

16Nishanth  10.07.2012 10:43:40  New Performance Problem with Domino on Windows 2008 64bit

Domino 8.5.3, widnows server 2008 R2.

We have constant issue with our domino server. It stops responding and below are the error messages in log.

07/07/2012 10:59:19 Error full text indexing document NT000008F6 E:\Lotus\Domino\data\mtdata\mtstore.ft (rc=3892) Not enough memory for Full Text Indexing or Search

07/07/2012 10:59:20 Full Text message: Memory allocation error. errcode = 520

07/07/2012 10:59:20 Error full text indexing document NT000008F6 E:\Lotus\Domino\data\mtdata\mtstore.ft (rc=3892) Not enough memory for Full Text Indexing or Search

07/07/2012 11:05:34 Error full text indexing document NT00001166 E:\Lotus\Domino\data\RightFax\RFAXSYNC.ft (rc=3892) Not enough memory for Full Text Indexing or Search

07/07/2012 11:09:55 The TCP/IP protocol stack reported that it ran out of memory. If you are using the IBM Lotus Notes Client, you may need to close applications or restart your system. For additional help, refer to the IBM Technotes. If you are running an IBM Lotus Domino - Automatically reducing TCPIP port maximum sessions to 15. Currently, 1 sessions are in the process of being recycled.

windows system log gives the below error:

Windows successfully diagnosed a low virtual memory condition. Following programs consumed the most virtual memory. Nhttp.exe (4836), nserver.exe (2288),

RAM: 4GB, Page File 512 MB.

Increasing page file is not an option as the company standard is 512 MB.

Please advice

17Gray  18.09.2012 9:49:59  New Performance Problem with Domino on Windows 2008 64bit

hi Daniel, seriously good post. Referring to Nishanth's more recent question about lack of memory whilst indexing, we have this problem a lot. The DBs it throws up are all vast (18 - 24GB), so no surprises there.

Any suggestions apart from implementing a more sensible archive procedure?

18Stefan Moser  01.11.2012 10:36:37  New Performance Problem with Domino on Windows 2008 64bit

Thanks Daniel, helpful article.

19Alex Elliott  01.04.2013 22:05:21  New Performance Problem with Domino on Windows 2008 64bit

If you're running Domino on a Windows 2012 server which is also a Domain Controller you can force the server to use disk cacheing to greatly improve performance with the following registry tweak:

Path: HKLM\System\CurrentControlSet\Services\NTDS\Parameters

Value: "DSA Heuristics"

Type: REG_SZ

Setting this to "1" (Note: the type MUST be a string type) prevents AD service from disabling write cache.

After updating the registry go into Device Manager and enable write cacheing on the drive(s).

Reboot the server and you should see a big improvement in performance. Some locking / timeout issues in Domino will go away as well.

Alex

20Nicolas Hernandez  22.11.2013 12:25:05  New Performance Problem with Domino on Windows 2008 64bit

Thanks Daniel, this article describe what I am suffering right now, but my architecture is a little bit different. Windows 2008 R2 over Vmware vCenter 5 with Lotus Domino 9.

Does this fixpack apply to Domino 9 too?

"Solution

In our customer situation we got a hotfix to disable the attribute when opening Notes databases. All the performance tests have shown that this is not just the solution for our particular issue with backup performance but also helps to increase performance in other situations. IBM is currently checking if this flag can be disabled in a future fixpack of Domino. We got a SPR #KBRN899NF6 and a hotfix to provide a notes.ini variable to disable the FILE_FLAG_RANDOM_ACCESS for now.

Update: It is important that you enabled the following notes.ini parameter once you have installed the hotfix:

Disable_Random_RW_File_ATTR=1"

21Zankar  27.11.2013 5:48:04  New Performance Problem with Domino on Windows 2008 64bit

Thanks for this article and suggestions.

Disable_Random_RW_File_ATTR=1

After enabling this into windows 2008 r2 sp1

lotus domino 9, Windows memory utilization is decreased and domino performance also improved.

22WiSa  22.08.2016 14:19:46  New Performance Problem with Domino on Windows 2008 64bit

Hi,

Is the entry Disable_Random_RW_File_ATTR=1 still a valid option for Windows 2012 R2 servers ?

Our environment Domino 9.0.1FP6 & Windows 2012 R2

23Daniel Nashed  23.08.2016 8:00:59  New Performance Problem with Domino on Windows 2008 64bit

Disable_Random_RW_File_ATTR was a temporary work-around. you should not use it any more on Windows.

-- Daniel

24Vish  21.02.2017 15:35:43  New Performance Problem with Domino on Windows 2008 64bit

Hello Daniel,

we have a similar issue, but it is happening on a file server where large NSF files are stored. In this case, the Domino servers run in a datacenter, and the users access a local file server where there NSF replicas are stored. Besides this, we see exactly the same behavior, the fileserver is running Windows 2008 R2 x64, with 4 GB or RAM. Installing the hotfix didn't help yet, we are considering about using the notes.ini additional parameter. Would it work/apply in this case?

Your help/advice is much appreciated.

Thanks,

Max

Links

    Archives


    • [HCL Domino]
    • [Domino on Linux]
    • [Nash!Com]
    • [Daniel Nashed]