Daniel Nashed's Blog

Domino Backup Feedback, Requirements and my new project

Domino Backup

As a consultant I see many Domino environments every year and the topic "Domino Backup" is coming up regularly.
Today there are still Domino servers not being backuped with Domino supported backup applications!
And I even see servers without Domino transaction logging enabled!!

It's really a best practice and requirement to enable transaction logging for any type of Domino server for data consistency and concurrent access to databases
-- even you are not leveraging point in time recovery and incremental backup when transaction logging is configured in archive style mode!

And the only fully supported way to backup a database online is to have a backup agent which leverages the Domino Backup C-API!
Open file backup and Microsoft Volume Shadow Copy (see reference below) and snapshot backup that does not leverage the backup API  is not supported and can cause data loss!

Therefore it is essential to have a supported backup solution for your environment.
Usually you are not having a separate solution for Domino but have selected your strategic backup solution.

Most larger vendors like IBM, EMC and NetApp support Domino.
On the other side  for Backup Exec does not support Domino any more
(I have my personal opinion about Backup Exec and how they implemented their Domino and specially their DAOS Support. But that's another story).

Getting feedback about current solutions

There is a AHA idea to collect a list of support backup solutions for Domino (see AHA idea in references below).
And I would volunteer to collect them thru comments here and emails and check with IBM/HCL if they want to technote them or if I post them on my blog.
So feel free to send me the solution you are using and I am also interested in your feedback.

That might be a good starting point for others to check which backup solution might fit them.
I think we have to distinct between larger enterprise solutions and solutions for smaller environments.

Enterprise products I work with

I have personally worked mainly with IBM TSM/TDP (now called IBM Spectrum Protect) and the EMC Networke. Both work like a charm on command-line. I don't like their GUIs at all -- LOL.
And I helped customers with Linux deployment scripts and also backup/restore scripts including DAOS restore.

Beside those two, NetApp has a very interesting backup solution (see references below) which allows fully supported snapshots by bringing all databases into backup mode before initiating the snapshot.
They implemented their own snapshot agent. But it only works with their back-end storage.

Requirements

With storage optimization in Domino the disk size which needs to be backed up with a Domino aware backup agent is getting smaller.
DAOS Files with dedupliate attachment data will be stored outside the NSF in a separate file-system can be backed up separately with a file agent.
View indexes can be moved into separate .NDX files leveraging NIFNSF introduced in Domino 9.0.1 FP8. And FT Index Files can be moved to a separate disk already since Domino 8.5.3.

So independent from your backup approach (file backup or snapshot) this can reduce your on-line Domino backup server storage and time already dramatically.
We even have customers putting their DAOS files into a central storage system like a NetApp via NFS and gain additional optimization by block level deduplicating NLO files from multiple servers (see references below).
And HCL is looking into further optimization for Domino 11 by introducing a tiered DAOS storage model where older NLOs can be moved to any S3 compliant storage on prem or in the cloud.

So we are mainly having the requirement to backup NSF files on-line. All other parts can be backed up without a Domino aware solution or for index files, need no backup!

Current Requirements

Today most servers are virtualized and more and more customers are moving to snapshot type of backups. But not everyone might be using NetApp storage.
In addition we have smaller environments or hosted environments where those enterprise solutions are too expensive or too complicated to implement.

There are also requests on the AHA website (see AHA idea in references below) for an affordable backup solution for smaller environments.
Having a way to store backups into simple tar files or rsync them to a different server are common requests I hear also from partners for their own hosted servers.
And also for my own Domino servers I could benefit from a simple backup solution.

I looked into what is already available today on OpenNTF. But that wasn't fully fitting my requirements. The current project "just" generates a consistent file-copy of the NSF leveraging the backup C-API.
My intention would be to have flexible Domino backup application which should fit for any type of backup solution -- either file or snapshot based!

I spent some time over the last weekends and build a first version which integrates with Backup solutions invoking command-line scripts per database or snapshot begin and end scripts in case of snapshot backup.

My current idea is to make this application available for free for smaller environments (e.g. 10 users, 100 database per server) and for example the Domino Community server and have a commercial version available for larger environments.

This would allow any type of integration and one of the reference implementations would be a Linux & resync or tar backup with all required scripts.
But I also want to work with vendors (already spoke with one who support snapshot backup) to see if my solution could complement their existing applications and enable them for Domino.

Here is a very brief overview of the current implementation. I am really interested to get your feedback if this is an approach that could be helpful.



My Domino Backup Project "nshback"

• On-line Domino Backup leveraging the Domino Backup C-API
• Support for circular log with full backup
• Support for archive translog and incremental backup
• Point in time restore leveraging transaction logs
• Incremental backup (either in combination with archive-style translog or just backup databases that have changed since the last backup)
• Support for snapshot backup of file-system (bring all databases into backup mode, take snapshot and backup delta occurred during backup)
• Store backup logs in CSV files along with the backup (for disaster recovery) and also store it in a central NSF (for invoking restore requests directly from the NSF)
• Flexible restore options that allow to: bring database online, point in time recovery, disable replication, change replica ID, change title, disable all agents, etc.
• Support for Win64 and Linux64
• Command-Line integration for backup tools for file and snapshot backup

Those features are already implemented. The remaining challenging part is how to organize and maintain the backups and to have proper backup retention.
This does highly depend also on the back-end storage used for the backups.

I am looking into S3 compliant storage right now for the mentioned two tier DAOS storage leveraging S3 storage planned for Domino 11.
So I am also looking into S3 integration for this backup solution.
I have found an interesting project (https://hub.docker.com/r/minio/mc/) which offers a command-line interface for Windows and Linux to S3 storage which could be easily integrated leveraging scripting.

Open points

I am still working on the right way to restore all databases in disaster recovery mode.
But that will also depend on the back-end used for storing the backups. Also the round trip from backup to restore needs a manual step today.
That's also mainly because of the flexibility for backup solutions.
I might start with a full reference implementation for back-end storage leveraging one tar/zip file for each backup and continue from there with integrating with other infrastructure.

While implementing the incremental/delta backup functionality I figured out that there is a C-API call which is not completely helpful to figure out if a database needs a new backup.
I have worked around it by comparing the DBIID of the database (which is actually the datetime when this instance was created. But there are two internal fields in a database header that show when a database was last backed up (see AHA idea in references below).
But that's just an enhancement which would allow more granular control.


What do you think???

I am interested in all type of feedbacks, either via comments or per mail. I really want to understand what the community needs.
And I also want to understand what currently works good for your with your current backup solutions.
I am not saying that you should move from your current solution. It's more that I would like to introduce new options that might help in certain cases.

-- Daniel


Appendix - References and additional information


1.) AHA Idea - Requesting to provide a public and official list of third party tools for backup compliant with Domino

https://domino.ideas.aha.io/ideas/DOMINO-I-706

2.) AHA Idea - A simple backup client for the NSF

https://domino.ideas.aha.io/ideas/DOMINO-I-131

3.) AHA Idea - Need Backup C-API Call to get backup start and end date for a database

https://domino.ideas.aha.io/ideas/DOMINO-I-529


4.) Is Microsoft Volume Shadow Copy supported for Domino backups?

http://www.ibm.com/support/docview.wss?uid=swg21196479

5.) Blog Post - Domino with NetApp Storage

http://blog.nashcom.de/nashcomblog.nsf/dx/domino-with-netapp-storage.htm

6.) NetApp Snap Creator® Framework 4.3.1

https://mysupport.netapp.com/documentation/docweb/index.html?productID=62378

• Comments [13]