Domino Community Docker Compose and multi-container support

Daniel Nashed  13 August 2022 23:06:05

Docker is an interesting environment also for local development and testing.
In some use-cases you might need Domino in combination with additional containers.

But this still only allows dominoctl to maintain one container.
With the latest update you can run different type of configurations.

Each user can have it's own configuration, located in .dominoctl in the user's home directory.
Or there can be project configurations in the current directory.
Here is the full documentation --> https://nashcom.github.io/domino-startscript/dominoctl/#docker-compose-support

• Comments [0]

Domino Community Image - New Nomad Server install option

Daniel Nashed  30 July 2022 17:07:15

The Nomad server is a new offering to directly add Nomad support to your Domino server instead of using a SafeLinx server.
Recently I added a SafeLinx container to the Domino community project.

Now I am adding the Nomad Server to the Domino image as a new build options.

The build script now has a new option, which works like other options like -verse and -capi.
It just adds the Nomad server to the image and exposes the standard Nomad port 9443 in the new image.

Installing the Nomad Server is basically just an extract of the tar into the Domino binary directory.

The result is an image, with out of the box Nomad web support.

Even for a new server Nomad Web could be available configured via OneTouch Setup.
The only challenge here is to get a trusted certificate.

This can be Let's Encrypt or any other certificate you import via PEM or P12.
Or you could export the trusted root from a Domino MicroCA and add it as a trusted root to your browser.
The last step is probably only a good idea for a quick test environment.


./build.sh domino 12.0.2EAP3 -nomad
...

12.0.2EAP3          [OK] Domino_12.0.2_Linux_English_EAP3.tar
1.0.5               [OK] nomad-server-105-beta-domino-linux-1202.tgz

• Comments [0]

SUSE Leap update to a new service pack 15.4

Daniel Nashed  25 July 2022 09:46:41

In contrast to Redhat based systems with yum/dnf, SUSE updates are only happening inside the current service pack.
A zypper update only brings you to the latest updates within a service pack.

But if you skim the update instructions it boils down to the following commands, which I performed today to bring my server from SP3 to SP4.


1. Bring your server to the latest patches for your current service pack and reboot:

zypper update
reboot

2. Swich to the new repositories

zypper --releasever=15.4 refresh

3. Update to service pack 4

zypper --releasever=15.4 dup

4. Final reboot

reboot

5. Check Linux version

cat /etc/os-release


Of course you should backup and check if you have any special conditions required for applications installed.
But for a standard server this should be the required steps needed.

Oh the hole process just takes a couple of minutes and the first reboot was needed, because the kernel changed.

• Comments [0]

Domino Container automation testing

Daniel Nashed  24 July 2022 07:32:06

Containers are not only a good way to run Domino. It is also the perfect environment for automation testing.
Domino 12 introduced OneTouch Setup to automate deployments, which also lets you create reproducible Domino test server scenarios.

As a starting point I am building an automation test for the Domino image itself.
So in future for every commit on the Git repository I can run automation to ensure the image works.

The test automation can be used in your own environment as well after an image built in your environment.
It can be also be extended for your own application testing.

Docker allows you to run commands inside a running container via "docker exec" commands.
So any kind of test operations could be performed. For example you could run a curl command inside the running container.

I will start with a base-line test to see the main operations of the container are tested.
And I would be interested which type of testing you would add to the container image.

• Bring a server up with OneTouch Setup
• Use OneTouch Setup to create a CertMgr Micro CA and create a TLS Credentials document for HTTPS
• Create an internet site via OneTouch Setup, start HTTP and check if the server responds via HTTPS

This should be already a good starting point and if those operations are successful, many components are indirectly tested.
But there will be more testing to also ensure the Domino start script and other components work well.

• Check Domino version
• Check JVM version
• Check if transaction log has been created
• Stop and start server with the embedded start script
• Test various start script operations like archiving logs
• Check if databases created by OneTouch Setup have been created
• Run the default backup configuration and backup a database.
  The backup application by design creates the dominobackup.nsf on first start-up and ships with a ready to use configuration.
  Starting the backup the second time will backup to /local/backup.

A lot of the new Domino features are designed to support automation testing.
And can help you to build your own test environments for QA testing and other scenarios.

The options are endless and I am going to provide a framework for adding your own tests.
The framework includes easy to use test results, with auto generated CSV and JSON output in a standardized way.

Here is a sneak peek of what type of output am working on.
Logging test results are as simple as calling a single script function:

test_result "domino.server.onetouch.microca" "Domino One Touch MicroCA" "SUCCESS"

--------------------------------------------------------------------------------
Test Results JSON
--------------------------------------------------------------------------------

{
"testResults": {
"harness": "DominoCommunityImage",
"suite": "Regression",
"testClient": "testing.notes.lab",
"testServer": "testing.notes.lab",
"platform": "Linux-Docker",
"testBuild": "12.0.1FP1",
"testcase": [
 {
   "name": "domino.server.onetouch.microca",
   "description": "Domino One Touch MicroCA",
   "executionResult": "SUCCESS",
   "errorText": ""
 },
 {
   "name": "domino.server.onetouch.database",
   "description": "Domino One Touch Create Database",
   "executionResult": "ERROR",
   "errorText": ""
 }
]
}
}

--------------------------------------------------------------------------------
Test Results
--------------------------------------------------------------------------------

[SUCCESS]    domino.server.onetouch.microca
[ERROR]      domino.server.onetouch.database

--------------------------------------------------------------------------------

Success :   1
Error   :   1
Total   :   2

• Comments [0]

Customizing Domino Backup mail notifications

Daniel Nashed  22 July 2022 12:22:08

Domino Backup offers to send e-mails depending on the status of your backup.
By default you are getting an e-mail in case of error or warning.

I am rarely getting error messages from my servers.
In this case here I updated my server to a new kernel and ZFS drivers failed to build.
I rei-compiled the ZFS kernel drivers to make ZFS available again.hBut I did not re-start the Podman container. So the ZFS volume mount was not available in the container.
It's great to see that it works and you are getting an error mail immediately.

The error reporting is build directly into the backup servertask.
The default report is on purpose designed like this.
It's intended to look good on mobile devices.


My failed backup as an example:


Status: Error
Server Name: notes.nashcom.de/Srv/NashCom-Net
Node: notes.nashcom.de
Mode: FULL
Backup Name: default
Backup Ref Date: 20220722100013
Backup Start: 22.07.2022 12:00:13 CEDT
Backup End: 22.07.2022 12:01:13 CEDT
Runtime (minutes): 1,0
Total Size (GB): 0

Databases: 281
Processed: 0
Errors: 281

Not Modified: 0
Excluded: 0
Skipped due to Compact: 0
Compact Retries: 0

-->Link


Most admins don't know that you can customize the report and have you own rules how and in which cases you want to be notified.

• The basic configuration is defining Error/Warning/OK.
• The target address on purpose is a formula and not a fixed address.
• The formula is executed on the backup log document
• So you can build any kind of special logic -- even time based to send messages to different addressed depending on status, time etc.
• You can also override the backup status formula, set the report sender and sender internet address (default the server name).
• In addition you could specify an agent, which should run on the log document for further customization.

On top of that the Advanced Tab allows you to specify an alternate form (Notification form:) if you want a different look & feel for for you backup report.
The report is rendered with the form. And there is also a hidden alternate form for reference and to be copied for customization.

It has been all built into Domino 12.0 backup to allow admins to fully customize notification.
I almost forgot about this flexibility and my failed backup, lead to this blog post.

Have  great weekend!

Daniel

• Comments [5]

Nomad Server 12.0.2 on Linux just works

Daniel Nashed  20 July 2022 18:42:39

The Nomad Server is a small component, you install on your Domino server.
And it is bundled with the Nomad Web files. So it is a all-in one server add-on solution.

The installation sounds more complicated then it is.
It's really simple to install. And I am thinking about making it an install option for the Domino community container image.

Here is the official documentation: https://help.hcltechsw.com/nomad/beta/nomadweb/index.html

But let me summarize the main points:

• Just expand the tar into your Domino server binary directory
• You need to have a proper cert! A self-signed will not work
• But the Nomad Server team added support to CertStore TLS Credentials
  When you specify a lookup hostname like NOMAD_WEB_HOST=myhost.mydomain.com, the Nomad Server will find the TLS Credentials document
• After that, you can just "load nomad" to get the server started
• By default, it will listen on port 9443
• There is a yml file where you could specify a different port and a couple of other settings

You can use Let's Encrypt certificates or the manual flow to create a CSR and get a certificate.
If you are behind a reverse proxy, you could also use a MicroCA cert.

Your certificate must be trusted in your browser.

Exporting the root certificate of your Domino MicroCA would be a valid approach for a local test server.
But usually, you should get a proper cert.

The lookup of the cert is based on the hostname. A wildcard certificate would work. You just need to specify the wildcard name.

Current limitation: The hostname can only exist once. So if you have the same name for two certificates, the lookup will fail in the first beta build.

Once the server is running, you can access it via:

https://myhost.mydomain.com:9443.

One requirement is ID Vault. But that's not new to the Nomad Server.

Of course, you could have a NGINX in front of it using SNI on port 443 for the same IP address.
I posted a sample configuration recently for the SafeLinx community container. You will need some specific config for web sockets.

Example reference: https://github.com/HCL-TECH-SOFTWARE/domino-container/blob/main/examples/safelinx/nginx.conf

But this is an extra option, which you will not need. It works well also with the separate port.

-- Daniel

• Comments [8]

Why run Domino in a container today

Daniel Nashed  17 July 2022 10:57:40

As my of you know, I am a big fan of running Domino and other applications in a container.
This can be a classical Docker/Podman deployment or K8s.

Containers might not be good for everyone. But a lot of software is available in a "Docker image", which can run in multiple environments.
Domino's main deployment model will not change to Docker.
Whereas some HCL applications like Sametime changed to a container deployment model.

Domino is one big monolithic application, running in one container.
But there can be still benefits to running Domino containerized.

Containerization does not mean it has to be in the cloud or that you can't use native resources like standard disk volumes or direct host network access.
It's more about running the Domino binaries from a standardized, automatically deployable image.


Most Domino container deployments are on Docker, not Kubernetes (K8s)

So far I see more demand on Docker than on real-world K8s deployments.
Running Domino on K8s usually makes sense if you have a K8s strategy in-house.

Domino isn't the platform where you would start if you look into K8s first.
Unless you are a large provider running many containers at scale for many customers.
The principles of containers are very much the same for Docker and K8s.
And also the container images are usually the same.


The benefit of running in a container

A container is built from a blueprint and the installation is always the same.
So you are getting always the same installation already tested in a QA environment or on a test server.
Also, many others are running the same installation from a consistent image.

Updating

The Domino community container image comes with updated templates and binaries and logic inside the image takes care of deploying and updating templates.
So an update drills down to

• Shutdown and remove the old container
• Create a container based on the new image with the data you already had

Bringing up new servers

Configuring a new server does not require any new software installation.
Instead, you just need some configuration steps:

• Define your container run-time environment (image to use, volumes to mount, network to use ..)
• Define environment variables to configure your container
• For Domino, you also need a OneTouch Setup JSON file to configure your first or additional servers in your Domain

This is a pretty easy and consistent way to bring up new servers.
And also updates are much easier. For setting up Domino I added a "setup" option to get a standard JSON-based Domino OneTouch Setup for the first servers and additional servers.
Which makes it easy to get your server up and running in minutes. Including a standardized, best practices configuration.


Additional software for a container

Additional software, which is not in your data directory has to be added to the image.
If this is an in-house application or you have to just copy some files and make them executable, this is a very simple add-on image, based on a standard Domino container image.
But for commercial add-on applications, which also need configuration and have their own update mechanism, this could be complicated.

Add-on software like virus scanners and backup agents for Domino are probably the best examples.


Domino 12 features helping in the container world

On a local single Docker instance, deploying native tools like backup could still make sense.
But moving to K8s is usually quite challenging.
One of the reasons Domino 12 offers native backup integration for different kinds of backup back-ends and also to support native anti-virus in Domino 12.0.2 is to support containerized environments and to make deployment easier and standardized.
For the new content scan anti-virus integration you would for example just need a configuration and point your Domino server to an ICAP server -- maybe even running in another container. Or using an appliance.


Applications composed of multiple containers

I created a new SafeLinx Nomad Web container image recently.
The image contains SafeLinx and also Nomad Web and optionally also drivers for MySQL.
In smaller environments, you don't need a relational state database. But in already deployments you need an external database.

The beauty of containerization is that databases like Microsoft SQL Server or MySQL are also available as a container.
So you only have to add a MySQL container and configure it to be used by your SafeLinx server.
There is no installation needed. The image is just "pulled" from Docker Hub.

Bringing up two applications together becomes just a configuration step in glueing the components together.
So in my example, I created a docker-compose setup and added a container for MySQL.


QA environments & Test automation

Bringing up a consistent, auto-deployed environment of multiple containers together, can be also very interesting for QA and test automation.
Let's say you are a software vendor and want to test your nightly builds to run a certain test, a Docker environment can be very helpful.
You could use a docker-compose.yml to define your environment and after bringing up the servers in a defined way, you could run commands inside the container for automation testing.

There are many ways to leverage this type of environment and I have built automation scripts for different use cases to test complex applications including injecting commands at different states of the test flow.
The key components here are a well-defined always consistent test environment, which can be recreated any time with updated application or Linux base level software.


Container environments are usually Linux based

There are also containers on Windows and you can run Docker images on Docker Desktop on Windows.
In the end, it is always Linux based in the back-end in one or another way.
Docker Desktop is leveraging WSL2 containers today.

Depending on how you run your Docker environment there is not much you see directly from WSL2.
But in general, in a production environment, the full infrastructure is a Linux stack.

So you need some base Linux to know how to get containers up and running. And also the base OS for containers is Linux and you should have some basic Linux skills.
On the other side my Domino container script (dominoctl) and my Domino start script running inside the container, make it very easy to support and maintain Docker/Podman-based Domino container environments.

I don't see Linux as a huge challenge. And any IT professional should build up some basic Linux skills today.
There are so many benefits to running Linux-based servers. And Docker or Podman makes running applications on Linux easier.


Testing with different Domino and base operating system versions

The Domino community container image allows you to build the container image on various Linux base images including RedHat, CentOS-based Linux, SUSE, Debian, Ubuntu and a couple more.
The images can be used to quickly switch between different base operating systems and Domino versions.
This helped me to narrow down issues between different versions many times.


My personal conclusion

I have many Domino servers running on Docker and Podman for various reasons. And I am using it not only for production but also for development purposes.
It helps me dramatically in my work for different customers and environments. And I am enabling customers and partners to leverage containers for their requirements.

Containers are not solving all your problems automatically.
But in today's IT infrastructure containers play an important role.
And even Domino isn't a microservice-based architecture, you can benefit from many container functionalities.

• Comments [0]

Nomad Web 1.0.4 released. SafeLinx Community container updated

Daniel Nashed  16 July 2022 09:59:40

Nomad Web 1.0.4 is released and I just updated the community container to the latest version.
Updating is as simple as downloading the new ZIP and pulling the GitHub repo.

https://opensource.hcltechsw.com/domino-container/safelinx/

The release notes https://help.hcltechsw.com/nomad/1.0_web/nomad_web_new.html show two new features.

• Offline Mode
• Replication
  

For more details check the links provided in the release notes linked above.

Now that I have the SafeLinx Nomad Web container, installation and updating is fun.

• Comments [0]

Domino 12 Restore point in time

Daniel Nashed  14 July 2022 09:57:02

Domino Backup can restore databases point in time!

For other backup applications this functionality is usually only available with archive transaction log.
But with circular translog or linear translog mode, the most current backup should have all the translogs available to recover point in time as well.

Domino Restore allows you to restore point in time in that case.
Even it is not guaranteed that the translogs are still there, this can be still a good configuration if you have servers without dramatic load.

Circular / Linear Translog

To make it fit for larger servers you could add like 20-40 GB of translog in linear mode (it is like circular translog just with more than 4 GB space).
Running this configuration would allow point in time recovery for the latest backups without the complexity of archive style transaction logging.
All my servers are configured this way and this is usually what really helps me.
For example if you work on a template and want to recover the state of some code from half an hour ago.
With point in time recovery you could go back to any state of the application.

Not a disaster recovery option, but there is also clustering...

Obviously you can't use circular/linear translog for a disaster recovery, because they are not in your backup.
But translogs are usually only in backuped every 2-4 hours anyway.
In case of important servers, you can address disaster recovery with Domino clustering.
And you can bet that I have backup on both of my cluster mates with different backup back-ends. OK maybe that's overkill for a one person Domino environment ...

How to restore point in time




1. First select the target restore time. the exact time the user requested.
This field on purpose has a timezone to select the point in time from user perspective. They might be in a different timezone then the server and the admin.

2. Now you can press the button to identify the right backup -- which will find the latest backup before the point in time you selected.
The backup time is in GMT winter time/UTC. The time will be automatically converted in the back-end for you to UTC.

3. For a point in time recovery you have to enable "Recover point in time"

4. Optionally, if you don't want to restore to the current time, you need to specify the point in time.

Another helper button next to the field helps you to put in the right time.
If you press the button, it will use the time in point specified at (1).

• Comments [0]

Linux shell scripts: Difference between "set" and "env" -- fixed the Domino start script

Daniel Nashed  9 July 2022 07:22:50

The Domino start script has always been using "set" to list the environment variables, before running the sever.
This information can be important to understand the environment passed to your Domino server at start-up.
It turns out that there had been a change over time, which causes much more information to be listed, then just the environment variables.

set command

"set" is an internal bash command, which has access to all local variables. not only the exported variables.
This includes the script itself up to the point it is already parsed.
That's why the start script is shown at the start of the notes.log of the server.
It also includes local variables, which are not exported and are not passed to the Domino server when started.

env command

There is an external command "env" also installed on every Linux server.
In contrast to "set" the "env" command only lists exported environment variables and does also not list the script itself.

Interesting details that I wasn't aware before I looked into it this morning after getting another question why the script is listed from an Australian admin.
It's not a critical change, so it will just make it into the next version.
If you want to change it on your own, just change "set" to "env".

• Comments [1]