Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

Domino Backup Feedback, Requirements and my new project

Daniel Nashed  4 April 2019 13:01:08
Domino Backup

As a consultant I see many Domino environments every year and the topic "Domino Backup" is coming up regularly.

Today there are still Domino servers not being backuped with Domino supported backup applications!

And I even see servers without Domino transaction logging enabled!!


It's really a best practice and requirement to enable transaction logging for any type of Domino server for data consistency and concurrent access to databases

-- even you are not leveraging point in time recovery and incremental backup when transaction logging is configured in archive style mode!


And the only fully supported way to backup a database online is to have a backup agent which leverages the Domino Backup C-API!
Open file backup and Microsoft Volume Shadow Copy (see reference below) and snapshot backup that does not leverage the backup API  is not supported and can cause data loss!


Therefore it is essential to have a supported backup solution for your environment.

Usually you are not having a separate solution for Domino but have selected your strategic backup solution.


Most larger vendors like IBM, EMC and NetApp support Domino.
On the other side  for Backup Exec does not support Domino any more

(I have my personal opinion about Backup Exec and how they implemented their Domino and specially their DAOS Support. But that's another story).


Getting feedback about current solutions


There is a AHA idea to collect a list of support backup solutions for Domino (see AHA idea in references below).

And I would volunteer to collect them thru comments here and emails and check with IBM/HCL if they want to technote them or if I post them on my blog.
So feel free to send me the solution you are using and I am also interested in your feedback.

That might be a good starting point for others to check which backup solution might fit them.

I think we have to distinct between larger enterprise solutions and solutions for smaller environments.


Enterprise products I work with


I have personally worked mainly with IBM TSM/TDP (now called IBM Spectrum Protect) and the EMC Networke. Both work like a charm on command-line. I don't like their GUIs at all -- LOL.

And I helped customers with Linux deployment scripts and also backup/restore scripts including DAOS restore.

Beside those two, NetApp has a very interesting backup solution (see references below) which allows fully supported snapshots by bringing all databases into backup mode before initiating the snapshot.

They implemented their own snapshot agent. But it only works with their back-end storage.

Requirements


With storage optimization in Domino the disk size which needs to be backed up with a Domino aware backup agent is getting smaller.

DAOS Files with dedupliate attachment data will be stored outside the NSF in a separate file-system can be backed up separately with a file agent.

View indexes can be moved into separate .NDX files leveraging NIFNSF introduced in Domino 9.0.1 FP8. And FT Index Files can be moved to a separate disk already since Domino 8.5.3.


So independent from your backup approach (file backup or snapshot) this can reduce your on-line Domino backup server storage and time already dramatically.

We even have customers putting their DAOS files into a central storage system like a NetApp via NFS and gain additional optimization by block level deduplicating NLO files from multiple servers (see references below).

And HCL is looking into further optimization for Domino 11 by introducing a tiered DAOS storage model where older NLOs can be moved to any S3 compliant storage on prem or in the cloud.


So we are mainly having the requirement to backup NSF files on-line. All other parts can be backed up without a Domino aware solution or for index files, need no backup!


Current Requirements


Today most servers are virtualized and more and more customers are moving to snapshot type of backups. But not everyone might be using NetApp storage.
In addition we have smaller environments or hosted environments where those enterprise solutions are too expensive or too complicated to implement.


There are also requests on the AHA website (see AHA idea in references below) for an affordable backup solution for smaller environments.
Having a way to store backups into simple tar files or rsync them to a different server are common requests I hear also from partners for their own hosted servers.

And also for my own Domino servers I could benefit from a simple backup solution.

I looked into what is already available today on OpenNTF. But that wasn't fully fitting my requirements. The current project "just" generates a consistent file-copy of the NSF leveraging the backup C-API.

My intention would be to have flexible Domino backup application which should fit for any type of backup solution -- either file or snapshot based!


I spent some time over the last weekends and build a first version which integrates with Backup solutions invoking command-line scripts per database or snapshot begin and end scripts in case of snapshot backup.


My current idea is to make this application available for free for smaller environments (e.g. 10 users, 100 database per server) and for example the Domino Community server and have a commercial version available for larger environments.

This would allow any type of integration and one of the reference implementations would be a Linux & resync or tar backup with all required scripts.

But I also want to work with vendors (already spoke with one who support snapshot backup) to see if my solution could complement their existing applications and enable them for Domino.


Here is a very brief overview of the current implementation. I am really interested to get your feedback if this is an approach that could be helpful.




My Domino Backup Project "nshback"
  • On-line Domino Backup leveraging the Domino Backup C-API
  • Support for circular log with full backup
  • Support for archive translog and incremental backup
  • Point in time restore leveraging transaction logs
  • Incremental backup (either in combination with archive-style translog or just backup databases that have changed since the last backup)
  • Support for snapshot backup of file-system (bring all databases into backup mode, take snapshot and backup delta occurred during backup)
  • Store backup logs in CSV files along with the backup (for disaster recovery) and also store it in a central NSF (for invoking restore requests directly from the NSF)
  • Flexible restore options that allow to: bring database online, point in time recovery, disable replication, change replica ID, change title, disable all agents, etc.
  • Support for Win64 and Linux64
  • Command-Line integration for backup tools for file and snapshot backup

Those features are already implemented. The remaining challenging part is how to organize and maintain the backups and to have proper backup retention.

This does highly depend also on the back-end storage used for the backups.


I am looking into S3 compliant storage right now for the mentioned two tier DAOS storage leveraging S3 storage planned for Domino 11.

So I am also looking into S3 integration for this backup solution.
I have found an interesting project (
https://hub.docker.com/r/minio/mc/) which offers a command-line interface for Windows and Linux to S3 storage which could be easily integrated leveraging scripting.

Open points


I am still working on the right way to restore all databases in disaster recovery mode.
But that will also depend on the back-end used for storing the backups. Also the round trip from backup to restore needs a manual step today.

That's also mainly because of the flexibility for backup solutions.
I might start with a full reference implementation for back-end storage leveraging one tar/zip file for each backup and continue from there with integrating with other infrastructure.


While implementing the incremental/delta backup functionality I figured out that there is a C-API call which is not completely helpful to figure out if a database needs a new backup.

I have worked around it by comparing the DBIID of the database (which is actually the datetime when this instance was created. But there are two internal fields in a database header that show when a database was last backed up (see AHA idea in references below).

But that's just an enhancement which would allow more granular control.



What do you think???


I am interested in all type of feedbacks, either via comments or per mail. I really want to understand what the community needs.

And I also want to understand what currently works good for your with your current backup solutions.

I am not saying that you should move from your current solution. It's more that I would like to introduce new options that might help in certain cases.


-- Daniel


Appendix - References and additional information


1.) AHA Idea - Requesting to provide a public and official list of third party tools for backup compliant with Domino


https://domino.ideas.aha.io/ideas/DOMINO-I-706

2.) AHA Idea - A simple backup client for the NSF


https://domino.ideas.aha.io/ideas/DOMINO-I-131

3.) AHA Idea - Need Backup C-API Call to get backup start and end date for a database


https://domino.ideas.aha.io/ideas/DOMINO-I-529


4.) Is Microsoft Volume Shadow Copy supported for Domino backups?


http://www.ibm.com/support/docview.wss?uid=swg21196479

5.) Blog Post - Domino with NetApp Storage


http://blog.nashcom.de/nashcomblog.nsf/dx/domino-with-netapp-storage.htm

6.) NetApp Snap Creator® Framework 4.3.1


https://mysupport.netapp.com/documentation/docweb/index.html?productID=62378
Comments

1Rick  04.04.2019 14:09:01  Domino Backup Feedback, Requirements and my new project

We are also searching for a backup solution. Currently we're looking into Handy Backup.

2Christian Henseler  04.04.2019 16:09:32  Domino Backup Feedback, Requirements and my new project

Veritas NetBackup, 8.x supports Domino 10

https://download.veritas.com/resources/content/live/OSVC/100032000/100032809/en_US/nbu_80_db_scl.pdf?__gda__=1554529673_c0b1d1544b8713a18f39dbb27166adc4

3Simon  04.04.2019 18:45:07  Domino Backup Feedback, Requirements and my new project

All sounds great so far.

Some ideas:-

1. A self-service portal that allows users to submit specific types of restore request, i.e. restore this document, restore this folder (admins can decide whether to deploy it and what type of requests to allow)

2. I wonder if putting all databases into read-only/backup mode on a large Domino server could be dangerous if the backup/disk technology being used is slow to take a snapshot (could memory get exhausted and cause an outage?). Could there be an timeout option or monitor to check for memory usage that could start bringing databases back on-line if memory becomes too low? (not sure if there is a specific amount of memory available database changes during a backup?)

3. Would you sell the finished product to HCL so it could be included in the base product?!

4Ben  05.04.2019 7:40:52  Domino Backup Feedback, Requirements and my new project

We are currently using Commvault as a backup solution. It supports backups of Notes Databases as well as individul documents(via transaction log backups) - https://documentation.commvault.com/commvault/v11/article?p=16337.htm. The only (small) drawback is,that the backup agent keeps some notes .dll files locked,meaning that we have to manually stop the backup services whe updating the server.Commvault has a gui to manage the services though.

5Pierre Lundqvist  05.04.2019 8:26:35  Domino Backup Feedback, Requirements and my new project

We use BAckupExec, although very dissapointed they withdrew the support for Domino. We used it only to do daily backups without archive transaction logging, so I cant say how good they were implementing that. We still run the old BackupExec versions that support domino. Their EOS is 2nd april 2019, and I guess we will have to pay extra to get extened support on those 7 installations.

Otherwise we use Spectrum protect, and it works like a charm. We use the GUI only. And now our company has bout data domain from Dell/EMC, so I will have a look on that.

But I still search for a replacement for the small spoke sites that are using backupexec. A software that is cheaper than netbackup, which is awkwardly expensive. I do see a market here and I hope you can write a good software that has support for local tapedrives

6Harald Wolf  05.04.2019 13:26:03  Domino Backup Feedback, Requirements and my new project

We use BackupExec for our Domino V9.01 Servers, allthough only V.9.0.0 is supported (as far as i know). We have had issues with it and DAOS in the past, which we could manange with Daniels help. Thank you, again.

On our V10.0.1 Test-Server the BackupExec doesn't see any databases. So were looking for a replacement too.

Our List so far:

SEP Sesam

IBM Tivoli Dataprotect (now Spektrum ...)

Veritas netbackup

ArcServ (Until now Domino V10 is not tested and not officialy supported but should work)

We are very interested in Daniel's backup project, because it would make us independent from expensive backup-solutions which support Domino natively.

7Gurmaj Singh  10.04.2019 18:30:34  Domino Backup Feedback, Requirements and my new project

Hi Daniel, thanks for your ideas. We use Veeam here and that works OK but there is still a concern that it is not application aware so we are not fully confident it will work with open files or non-commited changes. Veeam recommend a brief pause of the server before the backup which can work but is not an enterprise solution. Hopefully HCL can get Veeam to work on this somehow to provide assurance. Thanks.

8John Curtis  20.04.2019 14:25:00  Domino Backup Feedback, Requirements and my new project

Sorry it took so long, but circulating this internally at HCL

9Balazs Viszlai  31.05.2019 11:57:39  Domino Backup Feedback, Requirements and my new project

We are using Commvault backup, what use Domino Transaction Logs to provide point-in-time restore. We are satisfied with it, works fine and seamless.

10Rob Kirkland  03.07.2019 8:01:44  Domino Backup Feedback, Requirements and my new project

I have a customer asking what's the best Domino TL-aware backup solution. I've been beating my head against Google trying to find useful information. Then I stumbled upon this post. Daniel, you're the Bomb! I love your ideas and I'm working my way through your AHA links.

Archives


  • [IBM Lotus Domino]
  • [Domino on Linux]
  • [Nash!Com]
  • [Daniel Nashed]