Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

 
alt

Daniel Nashed

 

Why I think we don’t need to switch to 64bit now

Daniel Nashed  23 April 2009 15:17:00

A couple of customers and partners are asking me very week about native Domino 64bit.
Here is a post I wrote in a non public dicussion database but to avoid answering the question over and over again here is my current take about native Domino 64ibt.

--- in short ---

Using a 64bit OS with Domino 32bit gives you already most of the benefits in current releases.
We will get hopefully a 64bit optimized Domino 64bit version but right now there is not much benefit for 90% Domino environments.

--- long version ---

It is important to start looking into 64bit native Domino for all ISVs to be prepared for the future.
IMHO right now specially on Win64 using the native does not give you much benefit in standard environments. 64bit native Domino only helps if you need a LOT of memory.

We have seen large Domino HTTP servers running out of memory that benefit from native 64bit.
Also LEI servers on native 64bit that gain performance. But for most other servers 32bit Domino has a quite small performance gain.

The future is in the 64bit native application space and Domino on AIX and Windows is really as a platform. I am currently working with multiple ISVs to help porting their applications to Win64 and I have ported some of my applications to 64bit already.

It sounds harder than it is for a plain Domino C-API application if you have applied best practices when you wrote it. The devil is in the detail when you run external libs or other software that is invoked.

The real benefit you gain is thru using a 64bit OS. Specially on Windows you move up from a total memory limit of 2GB for shared and local process memory to a 4GB address limit.

By default even 8.5 does only allow you to have 2GB of shared memory with Domino32 on Win64.
You have to set ConstrainedSHMSizeMB=3072 to allow more memory.

IMPORTANT: But all binaries need to be linked with the /LARGEADDRESSAWARE flag to support more than 2 GB of memory.
I blogged about this a couple of days ago (
http://blog.nashcom.de/nashcomblog.nsf/dx/largeaddressaware-is-needed-for-domino-32bit-on-win64.htm?opendocument&comments)

So once you are running a 64bit OS you have most benefits already because the OS base allows the application to use more memory.
There is a 64bit / 32bit test result http://www.ibm.com/developerworks/lotus/library/domino8-64bit/ which shows there is not much difference in memory usage, CPU and I/O at least on Windows.

The memory that Domino will utilize even for larger server will most of the time not exceed 3 GB.
So there is not much you currently gain from moving to native 64bit.

Compared to other applications that just throw in more memory to reduce the I/O load Domino 8.0.x and 8.5.x introduces dramatic I/O reduction thru optimization in many areas of the server without increasing the need for more memory. Most effort is currently put into this area which IMHO is currently the right way to go.

Just to name some of the features that reduce I/O
Design Note & Data compression, SCR, OOO integrated into the mail router, on demand collations, streamlined design access, translog optimization, mail router optimization, scan for changed folder optimization, ... and last but most important DAOS

This provides much more TCO reduction than just allowing to throw more memory into a 64bit environment.

I have a detailed Speed Geeking session about 64bit downloadable on my homepage (http://www.nashcom.de/lotusphere) for some more details.

IMHO it is important to be prepared for 64bit and start looking into it right now.
But most customers are moving to native 64bit right now to expect a benefit that is not yet there.

That's specially true for Win64 on AIX it looks a bit different.
There is performance improvement as you see in the performance test I referenced and also the memory on AIX is segmented into 256MB blocks which causes some limitations for a 32bit application.
That goes away with native 64bit. So IMHO it is more important to look into native 64bit onAIX than on Windows.

But yes there are customers asking for 64bit and most of this is driven because other vendors use 64bit as a selling argument.
In fact if you read the details they are just throwing more memory into their JetEngine to compensate poor application design -- ok now I am a bit off topic but let me past an interesting article from a Microsoft blog dealing with their approach to I/O to show the difference in the Domino approach.I marked the important sections (I did this a while go when looking into 64bit and put some resources together to be prepared when customers are asking).

Side Note: There is one more interesting 64bit platform. zLinux switched completely to 64bit and there are very promising and interesting performance gains on that platform thru this move.

I am waiting to hear what IBM is planning in the area "64bit native Domino application feature exploitation" when they are done with all the great optimization they have done so far in 8.0.x and 8.5.x.


-- Daniel




Here is what M$ does to improve I/O.
IMHO the Domino way is the better approach.
But at some point Domino should start benefiting from native 64bit.
The good news is that we get better I/O and scalability improvements without the need to push customers to 64bit.


Understanding Exchange Server 2007 I/O improvements from 64 bit
http://msexchangeteam.com/archive/2006/09/08/428860.aspx

-- extract --
Exchange 2007

A major motivation for Exchange to use 64-bit is not ability to crunch bigger numbers, but to get more memory. In fact, we can access a lot more. Most 64-bit computers on the market can address a few hundred GBs of RAM. As mentioned before, more RAM means we can keep data in memory longer and save repeated trips to disk. But doesn't RAM cost money? Yes it does, but it's much cheaper than disk up to about 32 GB. Based on this, to optimize for IO reduction we recommend about 5MB of Jet database buffer cache for each user plus 2GB. So for 4000 users, you'd want 20GB + 2GB or about 24GB. This would mean a 20GB of jet cache vs. 1GB in Exchange 2000/2003. For our lab tests, we started at 1.0 IOPS and went to .54, entirely in reduction of reads; a MAJOR savings.

Our next bit of magic was to increase the number of storage groups. Moving from having 1 storage group (logs) for 5 databases to having a 1:1 relationship means more transaction logs (but not more files). Overall, there's no net change in bytes (same number of users). In Exchange 2000/2003, large servers typically deployed with 1000 users per storage group and the checkpoint depth was 20MB. This corresponds to 20KB of checkpoint per user. This limited the number of pages that could be delayed. By deploying more storage groups, we can delay more pages and get more batching and optimization. Also, the parts of the database that store views can store more messages on a single page. In our lab test (as listed above) this moved our I/O from .54 IOPS to .43 IOPS, stemming from a drop in write I/Os.

We didn't stop there. Now that the cache was bigger, we also increased the page size from 4KB to 8KB. The page size is the size of 'packets' of data that Jet stores on disk. It is the minimum size Exchange will fetch from the disk. The problem with this is that in some cases we might need all 8K (a message body) and other times we might not (a simple message with no body). Overall each page has twice as much data, but we can only have 1/2 as many pages in the Jet cache. Because of this, 8K pages *could possibly* hurt instead of help. Having a larger cache decreases the chances of this significantly by helping keep useful pages longer (minimizing the risk that we don't have the useful page in memory). The huge positive of 8K pages is that our internal structures in the database (trees) can be shorter. Shorter trees mean less I/Os to get to the pages that store actual user data. We also get the added benefit of storing more in the same place. In Exchange 2000/2003, we stored messages and messages bodies in separate locations, meaning at least 2 disk I/Os. Now, if the message and the body is less than 8K (our data indicates around 75% of messages are less than 8K) we store them in 1 location. This means savings on writes and savings on reads. In our lab tests, this change took us from .43 to .27 IOPS!

Links

    Archives


    • [HCL Domino]
    • [Domino on Linux]
    • [Nash!Com]
    • [Daniel Nashed]