Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

Domino 9.0.1 DBMT Compact File Pre-Alloc

Daniel Nashed  4 November 2013 06:33:50
This was on my list for a long time. When you leverage a copy-style compact the resulting database is often more fragmented than it was before because of the way a copy style compact works.
A copy style compact generates a temp file and copies the database note by note into the database.
For each document there is a check if the target database still has sufficient space fee. Else the database is extended to make the new document and/or object fit into the database.

The space allocated is around the size rounded up by the allocation granularity (which is an internal value of a database that cannot be changed).


So depending on how the underlying files-system works and depending on the free space in the file-system etc it can happen that the new database file has a lot of fragments already when the compact finishes.


DBMT uses copy style compacts only and is intended to re-organize your databases.

Therefore IBM built the new pre-allocation functionality first into the DBMT in Domino 9.0.1.


We have asked if this could be added also to the core compact call for business partner applications and for the compact servertask to also leverage this new functionality.

Hopefully we get this new functionality exposed in other places as well.


The  new functionality is not yet documented but I got the OK to share the info I got from IBM in advance:


-- snip --


We prealloc the .TMP file to the used space size of the .NSF. On non-windows platforms we write to the file in large blocks so get the file less fragmented. On Windows, we do the write as well, but it appears that is not necessary (it seems just setting the file size (at least through our api we use) makes the files more contiguous).


There is an ini that can be used to change the prealloc size, DBMT_PREFORMAT_PERCENT. We use the current used space for the .nsf and multiply by the specified percent, so if you specify 200, we double the size of the .NSF, if you specify 80, we only allocate 80% of the used space.


-- snip --


So when you do a copy style compact leveraging DBMT the new database should be less fragemented. I normal cases you don't need to change the allocation size but if you want to have some contiguous space in the database left after the compact finishes, you could set the size a bit higher like DBMT_PREFORMAT_PERCENT=120 but this would increase the size for large and small databases at the same time. So probably it is best to keep the size at 100%.


On a server that is DAOS enabled and all databases are DAOS enabled the physical size of the database might not be too big and setting the pre-allocation to 110-120% might make more sense.
The size calculated is based on the new phyiscial size (excluding the DAOS objects).


-- Daniel

Comments

1Mike  05.11.2013 15:47:33  Domino 9.0.1 DBMT Compact File Pre-Alloc

Have IBM shared why there is no option for doing an in place compact -b or -B to avoid creating a new DBIID each time?

2Steve  05.11.2013 22:00:24  Domino 9.0.1 DBMT Compact File Pre-Alloc

a compact -B does assign a new dbiid. A compact -b is only good for recovering orphaned space, it does nothing to reorganize the data

3Grant  02.09.2015 12:49:33  Domino 9.0.1 DBMT Compact File Pre-Alloc

Don't use DBMT to compact DB's into DAOS!!!!

What happens is it makes the .tmp file the same size as the pre-existing databases making the space used on new .nsf databases very low.

I ran out of disk space doing this, so word of warning.

4Daniel Nashed  07.09.2015 10:43:57  Domino 9.0.1 DBMT Compact File Pre-Alloc

that's something that should be at least technoted or fixed. pre-alloc with the DAOS on switch does not make sense.

but on the other hand I would have never used DBMT for that. I wasn't even aware that DBMT has a DAOS on switch.

I am usually running my own tool "nshrun" because that gives me more flexibility but you could also run multi-threaded compact with -#


  • [IBM Lotus Domino]
  • [Domino on Linux]
  • [Nash!Com]
  • [Daniel Nashed]