Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

Weekend Project "Domino on Docker Update"

Daniel Nashed  13 April 2019 19:09:53
There are a couple of updates pending for the offical IBM Docker project --> https://github.com/IBM/domino-docker
  • Support for 10.0.1 FP1
  • Support for Domino Community Edition
  • Preparation for supporting add-on products like "Traveler", "VOP", "AppDevPack" and later maybe "Sametime"
  • Making software downloads more reliable and provide better error checking
  • Install Log checking for installed software
  • Official implementation of the Domino on Dock management script.

The main challenge is proper version support without having multiple dockerfiles.
So I am working on a way to have the version tags and the installation files outside the dockerfile.

Lables and variables can be passed to the docker build command and overrwrite default settings.

Every FP will be an own full image installed on top of centos/latest.
Add-on products or customized versions are always installed as an own layer .


We discussed this week about having multiple layers like "10.0.1 -> 10.0.1 FP1 --> 10.0.1 FP1 IF1 but that would make image management more difficult for admins.

The build script could take care of building a "10.0.1" image before building a 10.0.1 FP1 image that is based on the other image.

But there isn't much benefit, beside a bit space reduction.

The add-on products and customization will have a separate layer and will use the current Domino image. For example Domino --> Traveler.


Does this make sense for you?  I first also thought that having the different versions build on each other would make sense. But we don't see the benefits. What do you think?


There will be a new file "software.txt" containing version numbers, download-file names and hashes.


Filenames of the downloads are the biggest challenge. The community edition, for example, has complete different filenames...

This map and download file will allow specifying versions without adding the download filename in the dockerfile or in the build file.


The community edition will be installed as a different product "DOMINO_CE" instead of "DOMINO" because also the FPs have different names and hashes ..
And of course, the directories inside the extracted software tar have a different directory structure (e.g. linux64/DominoEval/.. ).
The guys building it, have no idea how we are using it.


But I think I figured out a good way to organize versioning :-)


The new version will be also prepared for upcoming Interims Fixed and Hotfixes.

And I will propably also add JVM patches in the next step.

There is also a management and customization script to configure, build, run, mange, update Domino Docker containers.


-- Daniel







Comments

1Daniele Vistalli  13.04.2019 20:06:44  Weekend Project Domino on Docker Update

Hello Daniel, great work as usual.

My two cents:

- Having FPs as layers on top of base versions: PRO: Save space / better layer caching / CONS: Complexity (maybe)

- Addons product: this is the hard part. I can't see a way to have a one size fits all image so I'd just provide scripts where you can flag what you want in and get the image out. I wouldn't go for layering of products. New version -> Rebuild

- The build process: I've looked at the scripts and there's a lot going on that makes it difficult for newbies. I'd suggest the entire build process gets some help from docker-compose and multi-stage builds. If you wanna see this / talk about this I'm available. We just begun working with docker-compose for all our projects. It would also be a dream for developers willing to have Domino + Proton as it would allow you to start all what you need in a shot (but actually this would be an helper using the images you build with your process).

My original (and current) concern is about how to pack and update templates/resources that are layered on top of existing filesystem (NTFs, dominodata, html resources) that the tipical user messes with. So part of the process would be to isolate NTFs and other stuff changed by the FP to re-apply on top of a persistent filesystem (or to remove NTFs from a container completely).

I also see there are not "formal" VOLUME definitions in the docker file. I'd spend some work defining that part. My personal list of volumes is:

- Notesdata as a VOLUME

- View indexes as a VOLUME

- View Rebuild directory as a VOLUME

- FTIndex as a VOLUME

- Transaction logs as a VOLUME

This will ideally allow to do storage tiering in case we bring it to Docker/K8S systems with multiple classes of storage.

2Lars Berntrop-Bos  14.04.2019 0:09:01  Weekend Project Domino on Docker Update

Hi Daniel and Daniele,

Another VOLUME for DAOS objects would probably make sense.

As would another VOLUME for the Notes_TempDir directory.

3Shillem Volpato  16.04.2019 10:48:35  Weekend Project Domino on Docker Update

You might want to take a look at my project: https://github.com/shillem/domino-docker

It isn't so refined in terms of verbosity but it has already all the things you are wrapping your head around (minus the community stuff, which I thought would work out of the box but now realize IBM messed up again, actually no surprise there).

Regarding volume mapping I came to the conclusion that it's really burdensome to define a number of volumes (that could be used or not). My approach isn't perfect as it relies on the fact that I am not using a docker volume per se but I'm mapping one host folder: the domino folder (that contains the data folder). Once you map the domino folder instead of the data folder you provide an entry point for any other folder to live on the same level (at that point it should be easy from the host standpoint to converge the other folders to that one point)

4Shillem Volpato  16.04.2019 10:54:15  Weekend Project Domino on Docker Update

@Daniele Vistalli

Upgrade in docker circumstance is kind of a pain. My take at the moment is to throw a temporary container and run a rsync from the new data folder to the existing data (see last paragraph README.md in my project https://github.com/shillem/domino-docker)

5Daniel Nashed  17.04.2019 17:22:33  Weekend Project "Domino on Docker Update"

@Shillem, there is a simple solution. We already have a basic implementation and actually that's the next part that we are going to enhance for updates.

For a normal Docker environment local data from the image is copied to a volume of the container on first start.

So for example when the image installs all the data to /local/notesdata the mount will contain the installed data directory.

That works well for installation. On update the container is removed and a new run command will start a new container.

In that case the /local/notesdata will be mounted unchanged. For FPs and IF/HFs this could be already an issue because the iNotes directory is not updated.

And of course also templates etc. are not updated.

For that reason the install script has an option to move the install data directory outside the /local/notesdata to have it available in the container when it is started.

On startup the domino_docker_entrypoint.sh (belongs to my start script) is called.

This entrypoint is special because it not only contains logic to start the server and to wait for a SIGTERM to stop the server via start script but it also invokes the docker_prestart.sh is called.

This script does configure the server on first startup and also is intended to copy templates and other updates that are added by the installer.

The current plan is to check for a version number file in the /local/notesdata directory and compare it to the file in the install_data.

Right now this is optional but we are thinking of making this the default behavior.

This would solve template and other updates like a patched iNotes directory.

I have just submitted an update to the develop branch which implements a lot of new backend preparation for new functionality.

The update was mainly to support the Domino Community Edition and FP1 and future updates in a flexible way.

If you want to give more feedback you should look into the develop branch.

I am planning to do more work over the long Easter weekend :-)

After the next changes for template updating etc. we have to look into documentation updates.

There is a lot of stuff in the project that is not really documented in detail and some of it should also be enabled by default.

We are planning to support Proton but we have to find a way to also configure it.

For the Community Edition the Proton part is really important.

In addition to that I wrote a management script that Thomas has briefly demoed at Think.

That script allows you to do all kind of operations including building custom images and is a kind of "Docker start script" with a lot of options.

We will update that part also in the next update. We are about to prepare an updated session for Engage and we will be also at Engage for a Docker and Linux roundtable.

Once that is done we really need more detailed documentation specially for admins who never looked into Domino on the one side and Domino admins who never looked into Docker.

-- Daniel

6Panu Haaramo  19.04.2019 16:11:40  Weekend Project Domino on Docker Update

Hi, not really related to FP1 but I'm trying to use Docker for Domino the first time:

./build_domino.sh { Link }

Downloading Domino Installation files ...

tar: This does not look like a tar archive

tar: Exiting with failure status due to previous errors

File [{ Link } not downloaded correctly - terminating installation

The command '/bin/sh -c yum update -y && yum install -y bc perl which wget nano unzip rsync lsof gdb sysstat file && yum clean all >/dev/null && rm -fr /var/cache/yum && $LocalInstallDir/install_domino.sh && rm -rf $LocalInstallDir' returned a non-zero code: 1

~/install/domino-docker/dockerfiles/domino

I can download the file with wget using exactly the URL shown above and I can execute "tar -tvf" for the file without problems. Any ideas? Is it mandatory to provide tar files via HTTP or can I just place them to a directory? I'm using python SimpleHTTPServer to serve the files from same server.

Thanks, Panu

Archives


  • [IBM Lotus Domino]
  • [Domino on Linux]
  • [Nash!Com]
  • [Daniel Nashed]