Daniel Nashed's Blog

Domino V12 Backup & Restore leveraging native Borg Backup Integration

Introduction

Domino V12 Beta 3 introduces a Backup & Restore which is designed for easy integration with 3rd party backup solutions.

The server tasks "backup" and "restore" are accompanied by a new dominobackup.ntf/nsf which holds all the configuration, logs and a database inventory.
There is an admin friendly Notes based UI which allows you to restore databases with many options including restoring deleted documents and folders into the original database.

The solution is intended to complement existing backup solutions and to make it easier for customers and partners to integrate with existing backup solutions which are not Domino aware today.
This includes simple solutions based on
 

• Integrated file backup operations
• Custom scripted integration
• and also Snapshot backups!

It's all a matter of providing the right integration scripts.
Domino Backup & Restore itself is not intended to be a full backup application in the classical way.
It is more a middle-ware and an integration point on the one side and it is providing very flexible restore operations on the other side.
The focus is on NSF backup and transaction log backup -- which are the two components a classical file backup cannot cover without providing an application specific backup agent.

That means you still have to backup additional files in the data directory like notes.ini server.id etc and also the DAOS repository with a standard file-backup or have them included in a snapshot.


Storage Back-End for the default file copy implementation

By default Domino Backup is configured to perform a file copy backup to a location you define in the main configuration tab.
But running a full backup on all your NSF files to a central location every night doesn't really scale from resource usage point of view.

If you have de-duplicating and compression storage, it can still be a valid option also for larger environments.

There are many solutions and even the free ZFS file-system does offer dramatic storage reduction for nightly backup on Domino databases.
Beside ZFS there is for example also the backup vendor Cohesity offering a file share (internally called: view) to backup data.
This share is de-duplicated and compressed and can be part of a backup job on Cohesity side.
You can have similar storage optimization also on storage appliances from NetApp and others.

One of the advantages is that delta files created during backup, can be automatically applied to the databases.
And the resulting backup is consistent on it's own -- without the need of a restore operation.

I have customers using a similar configuration with Cohesity backup with very good de-duplication results in production with a similar solution today.


Domino Storage Optimization first

In this customer case we looked into optimizing the compact strategy to further optimize and reduce physical backup storage usage.
Mainly benefiting of all storage optimization for NSF (data/design compression, off-loading attachments to DAOS, moving out view indexes via NIFNSF).

There are a lot of storage optimization options introduced over the years, that you should also take benefit for your backups -- not just for performance over the day.
The DBMT server task be leveraged to compact database with space pre-allocation once per week or even two weeks only!


Example Implementation: Borg Backup

Beside file backup operations there are more flexible options to integrate with existing file backup applications and make those applications Domino aware.

To show how to fully integrate Domino Backup with an existing backup solution I am using Borg Backup for the following reasons:

• Open source & free solution available in CentOS (epel-release repository)
• Backup de-duplication and compression support!
• Support for encrypted backups
• Local and remote repositories (securely communicating over SSH)
• Very easy to setup
• Command-line interface available for backup and restore

Just showing an example would be going half way. I moved one of my servers to Podman already and it is running
our Domino Docker Community image and I have been looking for an efficient, fully supported Domino backup solution.
Bringing Borg Backup into a container environment makes it also very easy to deploy.

So I took the following components and glued them together with all the software and options needed.
I can't go into all details in this blog article, but it is all open source and included in our Docker project. And you can have a look yourself (and ask questions) ...
Also I will write another short blog article describing all the steps involved if using all the components described here.


Borg Backup
https://borgbackup.readthedocs.io

Domino V12 Backup & Restore Beta 3
https://help.hcltechsw.com/domino/12.0.0/admin/admn_backupandrestore.html

Domino Docker Community Project
https://github.com/IBM/domino-docker

Nash!Com Domino Container Script (domino_container)
Used and included in the Docker project


Borg Backup Integration

Let's look into all the components involved step by step. There is a separate blog entry for installing Borg Backup inside Docker.
I added the Borg Backup software along with the OpenSSL client and the FUSE software to our Domino Docker container.

So when you build your image using the "-borg" option, the build script automagically includes all components required for Borg Backup.

This also includes integration shell scripts leveraged by Domino Backup& Restore later.
Those scripts are technically part of the NashCom start script and added to the /opt/hcl/domino/backup/borg directory during installation.
Of course this integration can be leveraged on any other Domino V12 on Linux installation without containers.

This directory also contains the default DXL configuration file config.dxl.
All the references and names in this configuration file are already pointing to the right script locations.
You can simply import them into your dominobackup.nsf via the action menu.

Important: The existing file-system copy backup configuration, is enabled by default and needs to be disabled.


New directories in the container

The Domino Docker project already defines a couple of directories, which could be used as volume mount points.

For general backup and restore operations there are two new directories:

/local/backup   --> directory which could hold file-system backup data for NSF/translog and also holds the backup log files by default (configurable in the config document).
/local/restore --> empty mount point with proper access permissions to allow backup solutions like Borg Backup to mount their restore for copying restored databases back.

When you choose the Borg Backup build option another default directory will be created

/local/borg  --> can be used with a separate volume for a local Borg Backup directory.

Those directories are all created automatically at image build time to have them available later as volume mount targets.


Borg Backup scripts

The software used is all located in the extras directory in my start script --> https://github.com/IBM/domino-docker/tree/develop/start_script/extra/backup/borg
So you can leverage those scripts also native on Linux without the Docker project.


Running a new container with Borg Backup support

Now that Borg Backup is added to your image, you would need to create a new container with Borg software installed.
In my previous blog post --> https://blog.nashcom.de/nashcomblog.nsf/dx/running-borgbackup-restore-on-docker.htm
I already explained that the restore operation needs to mount a Borg backup.

That's where the /local/restore mount point is used. But user space mounts only work if you have the /dev/fuse device available in your container.

To add this to your container add the following options to your docker or podman run statement:  --device /dev/fuse --cap-add SYS_ADMIN

In case you are running the Domino container script, there is a new option available which enables this option automatically for you.

# Domino V12 Borg Backup support (enables FUSE device)
BORG_BACKUP=yes

Important: If you already have the docker_container script installed, you have to update the script to have the new logic available.
You just need to execute the install_domino_container script again.


Final Steps to get your first backup running

Now you can import the DXL configuration into dominobackup.nsf and disabled the existing configuration.
The database is created when you run the backup task for the first time.

Once the database is created you create a new Borg backup repository in the default location (you should add a separate volume for /local/borg ).

Inside the container run the following command to create a new backup repository.
The configuration in the Domino backup side you imported via DXL already contains this target location.

borg init --encryption=repokey-blake2 /local/borg


Now you have all components in place you can start your first backup and restore using the standard Domino Backup & Restore operations.

load backup

I would start with something simple first. The following command-line takes a backup of your log.nsf

load backup log.nsf



Next steps

So this was a lot of material to cover. I will write up another blog entry showing all the steps again as a script.
And a fellow HCL Ambassador already tested it in his environment and will show it running with production data in a separate blog post on his blog.

There is a lot of potential using de-duplicating and compression technology with Domino NSF backup.
See this as a starting point for more information to come.
This might be also a good starting point for a white paper covering different scenarios and best practices.

All information in this blog posted is posted to be shared and re-used in other articles or presentations.
If you are looking into implementing scripts on your own for other backup scenarios, I would be very interested to hear from you.

From what I heard HCL is planning an open source repository to host integrations like this.

The scripts included in the GitHub repository are open source and intended for reuse, adopting and to use as an example implementation.

Here is a link --> https://github.com/IBM/domino-docker/tree/develop/start_script/extra/backup/borg

And there is also script from zero to hero from CentOS 8 plain using the Docker Community project to Domino V12 including Podman and Borg Backup installation referenced.


Let me know what you think ..

-- Daniel

• Comments [2]