Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

 
alt

Daniel Nashed

 

Daniel Nashed's Blog

    Domino Storage optimization -- why are there still customers not leveraging the full potential?

    Daniel Nashed  19 September 2021 10:08:37

    In the context of Domino Backup I am looking again into storage optimization.
    There are many different ways to backup Domino databases.
    One of the most advanced option is to leverage a snapshot vendor solution.

    For a snapshot reducing the NSF storage and moving all data out of the data directory, which doesn't need a snapshot backup should be clear to every admin.
    But storage optimization best practices are beneficial for any server and also for run-time performance.

    It's still not clear why so many customers are not taking all the benefits from storage optimization Domino offers.

    I looked back into a presentation I did in 2008 when Domino 8.5.x introduced DAOS and I copied one of the most essential slides.

    Here is a very quick summary of storage optimization benefits without going into too much detail.
    All the functionality is easy to implement and I really don't see why it's not used more.

    What is missing to get more admins implement all of it?


    -- Daniel


    Design compression
    • Reduces 50% of the design.
    • Enabled by default on mail templates but needs to be applied on older databases manually.
    Data compression
    • Reduces 50% of the notes data (basically everything that is not an attachment).
    • Enabled by default on mail templates but needs to be applied on older databases manually.
    DAOS
    • Usually more than 70% in databases are attachments.
    • Moving the attachments to DAOS allows incremental DAOS backup.
    • NSF is reduced by 70% and backup, compact and other operations greatly benefit in performance and backup storage costs.
    • Deduplication safes 30-35% of storage on attachments on average on top of that!
      DAOS T2 could be used to offload older attachments to cheaper storage if needed.

    NIFNSF
    • Moves NIF data from the database to a separate NDX file which can be also stored in a different file-system outside the data directory.
    • Up to 10-15% less storage needed in NSF.
    FT Index moved to a different directory -- specially important for snapshot backups
    • You can move the FT index to a different disk so a snapshot backup would not backup that storage.
    • Sometimes moving FT is also helpful to avoid increasing your NSF file-system.
    DBMT
    • The database maintenance tool is designed to perform all maintenance operations on databases.
    • I still see that most customers use compact -B to maintain databases.
    • DBMT has many advantages including pre-allocating required disk space and allowing the file-system to align the file on disk properly.
    • The compact is a copy style compact and should be performed at most once per week.
    • The servertask is the recommended way to maintain your databases!
    For snapshot backup for sure you want to split data from from index to avoid taking a backup of index data.
    But also for standard backup you greatly benefit form storage optimization outlined above.



    Image:Domino Storage optimization -- why are there still customers not leveraging the full potential?
    Comments

    1Florian Vogler  19.09.2021 14:15:48  Domino Storage optimization -- why are there still customers not leveraging the full potential?

    I think the true issue is that it has to be enabled, and that comes with two problems:

    1.) the question whether there are databases for which it better be left disabled?

    2.) it being a task that is easily overlooked - whether once for all (or only many?), or whenever new databases are created.

    Assuming it is just safe to use, why wouldn’t the default be enabled and also couldn’t there be a (one) simple feature to enable all the great options at once for all databases?

    2Don  20.09.2021 7:58:11  Domino Storage optimization -- why are there still customers not leveraging the full potential?

    I think this also depends a lot of the architecture of the Domino application. For example we have most of the code in Java (osgi-bundles) and the application has a lot of databases (thousands) so a lot of the features do not have much effect (Design compression, DAOS).

    So of course we moved the FT-index, do data compression, we look at NIFNSF (which I think also has a performance side-effect) but not all of these options are suited for our setup. That may apply to more customers.

    3Oliver Regelmann  22.09.2021 9:32:30  Domino Storage optimization -- why are there still customers not leveraging the full potential?

    I think storage (production and backup) is so cheap some just don't want to hassle with additional tasks configuring and managing it. Even so when there might be implications regarding the recovery (as with DAOS when you've got to restore those files manually).

    Most just backup the complete VM anyway for quick recovery so why bother with different disks?

    Similar in migration scenarios to Exchange. Nobody cares about the additional storage this might need.

    Maybe it would be a good idea if Domino did some of these features automatically without the need to set notes.ini parameters.

    4Daniel Nashed  22.09.2021 11:28:46  Domino Storage optimization -- why are there still customers not leveraging the full potential?

    Hi Oliver,

    this is a very indifferent view! And Microsoft tells you to get inexpensive disks without RAID and they just throw in more resources. And they want physical hardware instead of virtual environments which are standard for a long time in the enterprise world.

    Yes Domino could handle things more automatic. Compression is enabled by default for the templates. NIFNSF could be enabled automatically without moving NDX to a different volume. Other options like DAOS need some planning and coordination.

    A snapshot alone is not the solution to all problems! I am looking into this for years and I am working on backup integrations for major backup vendors like Veeam for Domino.

    DAOS is almost transparent for admins for normal backup and restore operations and dramatically reduce the need for backup space. Managed store is not cheap! And when someone moves to Microsoft costs for storage, licenses or functionality is never a problem!

    But investing some hours to optimize an existing environment that worked like a charm for years seems to be a problem.

    The blog post was also to raise awareness and get feedback. DBMT and other features are not rocket science and I am really wondering why it has not used more.

    -- Daniel

    5Peter  23.09.2021 7:35:44  Domino Storage optimization -- why are there still customers not leveraging the full potential?

    While I am a big fan of DAOS, i think you are a bit to optimistic with your numbers.

    Compression of attachments first made things worse for use, because we had double attachments in DAOS (lz1 and huffmann).

    Then I don't see how encrypted daos-files can lead to a meaningful deduplication ration. Or is disabling encryption the recommended way today?

    6Daniel Nashed  23.09.2021 15:49:28  Domino Storage optimization -- why are there still customers not leveraging the full potential?

    Hi Peter,

    I have never been a fan of encrypting DAOS objects and my best practice is to keep it unencrypted unless you are encrypting your NSFs and putting a password on your server.id!

    You have to distinct between multiple things.

    a.) de-duplication by the DAOS hash for database objects that are the same -> Not attachments that's why different compression causes multiple NLOs. Mixed compression is a challenge! But this works under the encryption layer and encryption is applied after de-dudup for each NLO

    b.) de-duplication on file-system level. This only works with no encryption is enabled. It works among different NLOs and especially if the same storage container has the same NLOs for different servers like on a de-duplicating NAS.

    -- Daniel

    Recent Entries

    Feeds

    Links

      Archives

      • [HCL Domino]
      • [Domino on Linux]
      • [Nash!Com]
      • [Daniel Nashed]