 Daniel Nashed | Daniel Nashed 24 November 2022 23:23:44 Personally I am not a big fan of the language packs. I keep the server in English and just add the templates I need on a single server if needed. But there are some customers who really want also German system databases. Each Language Pack for each Domino release will require separate properties, LP ini files and a lot of testing. Therefore additional language packs are added on request. I would still encourage everyone to stay with an English container image. This was still a good step to understand the LP silent installer. The installation is another install option for the Domino base image. You just specify the language "de" on command-line. The installer will take care replacing the files and the templates are packaged into the install data tar automatically. -- Daniel ./build.sh domino -domlp=de
12.0.2 [OK] Domino_12.0.2_Linux_English.tar
de-12.0.2 [OK] Domino_12.0.2_SLP_German.tar
--------------------------------------------------------------------------------
Installing Language Pack de-12.0.2
--------------------------------------------------------------------------------
Running Domino Language Pack Silent Install -- This takes a while ...
Language Pack installed successfully Daniel Nashed 23 November 2022 21:48:54
Now that Domino 12.0.2 is has a native VSS Writer, we can look into new interesting integrations.
In my session at SUTOL conference this week, I showed a first version of a Restic integration for Domino 12.0.2 via VSS.
Restic is a very interesting application (https://restic.net) - It's a single binary written in GO
- And uses a approach like Borg Backup uses. But in contrast to Borg Backup it has full Windows support.
- This includes VSS Writer + AutoRecovery support!
- It is Open Source, efficient, flexible & secure.
- And very simple to setup & use!
Here are the two commands to backup your Domino server with Restic:
restic.exe init -r c:\backup
restic.exe backup -r c:\backup e:\notesdata --use-fs-snapshot
The first command creates a backup repository. The backup repository can be a local disk, a SFTP/SSH target or a S3 drive.
And in combination with rclone there are more options available. But let's stay with this simple file storage location for now.
The second command takes a snapshot of your Domino data disk and stores it as a new backup snapshot in the newly created repository.
In my simple example I don't have any excludes and usually you store FT index, NIF outside the Domino directory. But you can also specify excludes.
The application is pretty fast and provides deduplication and compression.
When you run it again is even dramatically faster, because most of the data is only read and not written to the local or remote location.
Linux support
Sadly I have not found a way to use it properly with Linux, because I would need to take a snapshot for every NSF separately.
There isn't a way to send files step by step to Restic. It always takes a single run. Even feeding it with files via STDIN waits for all files read, before it starts processing.
I will check with the project on GitHub, if there is a way we could use or if there will be something in future.
Today it would only work in combination with ZFS which can provide a snapshot for backup.
But if ZFS is in place, we can directly leverage ZFS storage snapshots and send them encrypted to a remote location.
Still even in that combination, Restic could be a good solution for Linux, too.
Windows support
On the other side the Windows support with VSS Writer support and AutoRecovery is awesome!
You don't need any integration on the Domino 12.0.2 side and it will automagically work, when you start the backupvss servertask.
The task registers the VSS Writer and when you launch a Ristic backup, the Domino server is performing all the operations automatically.
Restore operations
I have a Windows batch file already to restore. It calls Restic to find the backup tag file to identify the right backup in the first step.
Once found via the "restic find" command, a "restic dump" command is used to write the mydb.nsf.DAD file into the right location.
The integration is not yet available on GitHub. But if someone wants to try it out, let me know.
I would be also interested to hear if someone is already using Restic for Domino or other applications.
Backup for a local desktop environment
It sounds like it will be at least one of my options to backup my environment.
And it can be also a great option for a local development server on a desktop.
Maybe I am bit crazy. I mentioned in my SUTOL session, it might be even possible to run it on a 64bit client.
The VSS Writer interface is only available as a 64bit API. So it could only run in the 64bit client.
Totally unsupported. But I copied nbackup.exe, nrestore.exe, nbackupvss.exe and dominobackup.ntf to my 64bit Notes client and the backup just works ;-)
Daniel Nashed 20 November 2022 10:52:06 During HCL Factory the Notes/Domino & related add-on products major release updates have been released. I already blogged about many of the new features during the Early Access phase. See some details and references below. Congrats specially to the server team & security team for an impressive list of features! -- Daniel Notes 12.0.2 New workspace UI The Notes 12.0.1 UI with the 64x64 icons was a good step to a modern UI. Personally I see the new workspace design as a step back. Specially when looking into the new twisty to select the replica. There is a notes.ini to keep the more modern looking 12.0.1 UI notes.ini EnableV1202WorkspaceLook=0
I am currently switching to 64x64 pixel icons on my workspace updating all my applications. IBM Carbon design offers over 960 pictograms https://carbondesignsystem.com/guidelines/pictograms/library/. They are free to use and HCL is using the carbon design icons already for view icons. I found an easy way to convert them into Notes database icons combining them with the HCL background icons. Stay tuned for another blog post with detailed information. New 64bit Client The new 64bit client is and important step into the future and you can use the new 64bit client to check out all your applications. I would wait while before deploying the 64bit client in production. Specially when you have add-on applications you need to test. There is a technote explaining some limitations including a last minute regression. Guidelines in updating applications to run on the 64-bit version of HCL Notes 12.0.2 --> https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0101520 Beside that all your add-ons need to support 64bit as well (extension managers, add-on menus, LotusScript to C-API native calls!). Specially native C-API calls can be tricky and you really need to test all your applications. Now with 12.0.2 you have a released 64bit client, you can prepare a 64bit transition. But I would wait for at least a first IF or FP to deploy it on larger scale in production! Domino Restyle The new restyle functionality to modernize the look & feel of your Notes application is implemented in core Notes and is available in Nomad Web and the new Notes client. With designer access and higher, you can run the restyle directly from your Notes client (File > Application > Restyle). For more details check about Notes client and designer features check: Notes: https://help.hcltechsw.com/notes/12.0.2/client/whatsnew_1202.html Designer: https://help.hcltechsw.com/dom_designer/12.0.2/basic/wn_12.0.2.html Domino 12.0.2 Many of the bullet points below would be worth a separate blog post. I have blogged about many of the features already during early access beta. And there is more to come, now that 12.0.2 is released. - Virus scanning leveraging the ICAP protocol (https://en.wikipedia.org/wiki/Internet_Content_Adaptation_Protocol)
- Inbound mail SPF support (https://en.wikipedia.org/wiki/Sender_Policy_Framework)
- Inbound DKIM support (https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail)
- Outgoing free time lookup for Microsoft 365 users
- DAOS Encryption Manager (daosencmgr) to manage DAOS object encryption
- Domino 12.0.2 on Windows is a true VSS Writer (new servertask backupvss) which allows to use any VSS compliant backup application to backup Domino!
- OpenID Connect /OIDC native authentication support natively integrated into the HTTP task
- CertMgr is now available on AIX
- CertMgr is used to create JConsole/Controller certificates with an own JConsole Micro CA
- Micro CA certs can be exported and newly created CAs are valid for 20 years
- Update of internet certificate roots in Domino directory including adding certificate details in certificate document (and also cerstore.nsf document)
- For ICAP and OIDC certstore.nsf contains a new option to centralize manage trusted roots. Mandatory for ICAP, optional for OIDC.
- CertMgr remote certificate check health check via LibCurl request for all standard protocols like HTTP, ICAP, LDAPS, IMAPS, POP3S (using trusted roots from cerstore.nsf or names.nsf).
(There is a new back-end API which is planned to be used in more areas in future) - Notes/Domino includes OpenSSL 3.0.5 including a fix for the recent OpenSSL security issue
- Update of LibCurl to 7.83.0 (a recent version, more current than in most Linux distributions)
- Tika Server (used for attachment indexing) update to a new major version 2.4.1
- New Domino container image, based on the HCL community image
- Add-on container image for Traveler 12.0.2 and Domino Leap 1.1 based on the new image (previously Domino Volt, also just released this week!)
For more details https://help.hcltechsw.com/domino/12.0.2/admin/wn_12.0.2.html Nomad Server 1.0.5 for Domino 12.0.2 One important side note, if you are running the new Nomad Server. To run the Nomad server on Domino 12.0.2 you need a separate package supporting 12.0.2 Daniel Nashed 13 November 2022 09:09:59
In Domino 12.0.2 Inbound SPF and DKIM is just a simple configuration option in the server configuration document.
The SPF check was already available in Domino 12.0.1 indicated by the Received_SPF item.
When Domino 12.0.2 ships this week, all existing SpamGeek versions can be configured to support a DKIM as well.
Beside the fields Received_SPF and DKIM_Signature there is a new item Authentication_Results.
The item is the standard header Authentication-Results to indicate SPF and DKIM results (see https://www.rfc-editor.org/rfc/rfc7001 for details).
To check the new header, SpamGeek just needs new rule documents.
Because those new fields are not known by SpamGeek out of the box, the rule needs to run on the mail document instead of the log document.
I configured my server to copy the fields into the log document (configuration profile/optional) instead to also have them stored in SpamGeek log.
Enclosed are the new rules already waiting on my servers to work with the Domino 12.0.2 release version.
I added scores also for special cases to see verify the different cases. You might want to add your own weight to those formulas. Field Name: Authentication_Results Authentication_Results: notes.lab 1; spf=pass smtp.mailfrom=nsh@notes.lab (sender IP 1.2.3.4); dkim=pass header.s=09302021 header.d=notes.lab; dkim=pass header.s=ed10122021 header.d=notes.lab In my example the sender uses a RSA and Ed25519 key. The authentication header will contain multiple results in this case.
-- Daniel
SpamGeek for Domino - Rule Document | | Name: | Domino-SPF-Check | | | | Description: | Domino 12.0.2 SPF Status Check | | | | Status: | Enable | | | | Category: | SPF | | | | Type: | BlackList | | | | Event: | Accept | | | | Score-Multiplier: | 1 | | | | Formula: | x:@Left (@Right (Authentication_Results; "spf=");" ");
@if (
x = ""; 21;
x = "pass"; -1;
x = "neutral";10;
x = "none"; 20;
x = "invalid"; 30;
x = "softfail"; 40;
x = "permerror"; 40;
x = "fail"; 90;
0) | |
|
SpamGeek for Domino - Rule Document | | Name: | Domino-DKIM-Check | | | | Description: | Domino 12.0.2 DKIM Status Check | | | | Status: | Enable | | | | Category: | DKIM | | | | Type: | BlackList | | | | Event: | Accept | | | | Score-Multiplier: | 1 | | | | Formula: | x:@Left (@Right (Authentication_Results; "dkim=");" ");
@if (
x = ""; 1;
x = "pass"; -2;
x = "neutral";-1;
x = "none"; 1;
x = "invalid"; 30;
x = "softfail"; 40;
x = "permerror"; 40;
x = "fail"; 90;
0) | |
|
Daniel Nashed 25 October 2022 11:13:05 The previous macOS Monterey 12.6, caused the Notes client to require an interims fix to work, because of an unexpected macOS change. I am not a Mac user, but I got feedback from the field, that the Notes 12.0.1 client works on Ventura. There is a smaller known issue: The first start after updating fails. But after restarting the client should work! The upcoming Notes 12.0.2 version (expected in November) will address this. Check the following technote about MacOS Ventura support: https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0101181 -- Daniel Daniel Nashed 11 October 2022 08:39:22 By the default the ownership of a file copied into a running container is the owner from the host. If you are copying a file owned by root, this might not be what you want and you have to be careful when copying files.
Depending on how you want to use the copied file, this ownership isn't what you want.
You can't pass user/group or permissions to the docker cp command.
Changing the owner or mode would need root permissions inside the container.
Containers usually run with an unprivileged application user. For Domino this is notes:notes with 1000:1000.
You would need to run another command via "docker exec -u 0 ..." to change the file ownership or permissions.
But there is a more convenient way with one command...
Docker copy with tar
There is one additional option for the docker cp command:
You can stream data in tar format into the container ( if the container contains the tar program).
Docker automatically extracts the contents of the file streamed.
This can be helpful specially in test environments where you want to automatically copy data into a just started container.
Below is a sample command line. The interesting part is the dash. The dash for tar means write into stdout.
The dash in the docker command means get the tar data from stdin.
This isn't just a convenient way to copy a single file. It's also helpful for more data you package up via tar.
This example copies a server.id into the server's data directory and sets the required permissions and owner:
tar -cf - server.id --mode u=rw,g=,o= --owner 1000 --group 1000 | docker cp - domino-container:/local/notesdata
The command also works with podman (tested with Podman 4.2.0 and Docker 20.10.18).
Daniel Nashed 8 October 2022 15:33:35 Preparing my session for DNUG Domino Day online next week, I noticed how much security related functionality the team added in Domino 12.0.2. This is again a security release!  All the new features are enabled in our DNUG Lab I am managing for DNUG for every member to look into. This now includes OpenID OIDC authentication with Google as an example. Many of the topics in my session could be an own session or even a 4 hour workshop.. Domino 12.0.2 is in it's final early access code drop EAP5 and is planned to be available end of this year. https://dnug.de/event/dnug-online-domino/ At the end of the event we are planning a slot for feedback & questions. I will have the lab prepared for any kind of questions and short demos in the open end last part... Daniel Nashed 7 October 2022 11:02:18 Domino 12.0.2 introduced SPF checking leveraging the same libspf2 project (https://www.libspf2.org/), I have been using for a while in SpamGeek on Linux. I had a special SpamGeek build, which I distributed only to some friends for testing and it had a hard dependency to libspf2. Now with Domino 12.0.2 SPF checking is implemented in Domino native. And you can just enable it in the configuration document as shown below. DKIM inbound is also listed in the Domino directory design. But it is not part of Domino 12.0.2 EAP5. I tried to enable it, but it only writes the DKIM_Signature item. There isn't a status item yet -- which is planned for Domino 12.0.2 GA from what I understand. So for now let's focus on SPF checking.  Once you enabled the settings in the optional SpamGeek options, you can add two fields to the log documents. In a future SpamGeek version, I will add the new fields by default. But for now you can just configure the additional fields. Another way to implement the rule would be to run the rule document on the original note instead of the log document. But I want the fields to be logged in SpamGeek log anyhow.  Once you introduced the new fields, you can just create a new rule. Here is a copy of my rule document. 
Here is the formula for pasting. You might adjust the score values to your own needs.
x:= @Left (Received_SPF; " "); @if ( x = "pass"; -10; x = "neutral"; 0; x = "none"; 20; x = "invalid"; 30; x = "softfail"; 40; x = "permerror"; 40; x = "fail"; 90; 0) Once DKIM inbound is officially available, I will have another post how to add a similar rule for inbound DKIM. For now we could just check the DKIM signature item. But I would wait for the official status item. Daniel Nashed 6 October 2022 10:06:32
Martijn wrote an interesting post about NGINX versions
(https://blog.martdj.nl/2022/08/26/nginx-as-reverse-proxy-on-centos-9-stream-a-problematic-combination/)
This inspired me to take a closer look into important software included in current Linux distributions.
In an earlier post I already showed how to update curl on Windows 11
(https://blog.nashcom.de/nashcomblog.nsf/dx/replace-curl-shipped-with-windows-with-a-recent-version-not-build-by-microsoft.htm)
So the next step would be to look into Linux distributions.
Linux distributions
I took the current Docker images for the main distributions and checked the NGINX, OpenSSL and LibCurl/Curl version.
There is a dependency between LibCurl and OpenSSL when installed out of the box. All Linux platforms compile and ship LibCurl matching to the OpenSSL version on the platform.
Only when downloading LibCurl or other applications separately, there is a specific OpenSSL version usually bundled with the application.
Usually applications add libs to their application directory. This is often needed to run applications on different versions of the OS.
But it also means security patching your OS does not update important packages like OpenSSL, that are bundled with your applications!!
Many vendors lag behind patching their libs and they are also staying on older major releases for a long time.
On the other side it is not wise for an application vendor to rely on the OpenSSL version shipped with the OS. Specially when supporting multiple Linux distributions and versions.
LibCurl and OpenSSL
Domino 12.0.1 ships with a matching OpenSSL 1.1.1 version in a separate lib / DLL in the Domino program directory.
LibCurl is linked directly into core Notes/Domino since Domino 10.x.
Linking code directly into a core application reduces conflict with other applications and also reduces the number of open files (each process would need file handles to open a Lib).
In addition this is also more secure, because nobody could sneak in a different Lib pretending to be the binary Domino expects to load.
Domino 12.0.2 links with OpenSSL 3.0.x and is not loading it dynamically.
Beginning with Domino 12.0.2 the new OpenSSL 3.0.x major version is linked into core Notes/Domino as well.
This means two important components are directly glued into core Domino.
Software versions in current Linux distributions
Lets take a look into the different distributions ship today and see which might fit best from security point of view.
There is one special platform: openSUSE installs OpenSSL 1.1.1 out of the box and you can manually install OpenSSL 3.0.x (openssl-3). That's why I listed both versions.
You might be surprised that version included in Domino 12.0.2 is even newer then the most current Linux distribution. I marked really outdated versions which are problematic in red. And I marked older versions in the corresponding major versions in yellow. So a fully patched OpenSSL 1.1.1 version can be green. And an older OpenSSL 3.0.x version can be yellow. | Product | NGINX Version | OpenSSL Version | LibCurl/Curl Version | | | NGINX Docker Image | 1.23.1 | 1.1.1n, -- 15 Mar 2022 | 7.74.0 | | | VMware Photon OS/Linux 4.0 | 1.22.0 | 3.0.3 - 3 May 2022 | 7.83.1 | | | openSUSE Leap 15.4 | 1.21.5 | 1.1.1l -- 24 Aug 2021, 3.0.1 -- 14 Dec 2021 | 7.79.1 | | | Red Hat Enterprise Linux 9 UBI | 1.20.1 | 3.0.1 -- 14 Dec 2021 | 7.76.1 | | | Ubuntu 22.04.1 LTS (Jammy Jellyfish) | 1.18.0 | 3.0.2 -- 15 Mar 2022 | 7.81.0 | | | Debian GNU/Linux 11 (bullseye) | 1.18.0 | 1.1.1n -- 15 Mar 2022 | 7.74.0 | | | Oracle Linux 9 | 1.20.1 | 3.0.1 -- 14 Dec 2021 | 7.76.1 | | | Domino 12.0.2 | - | 3.0.5 -- 5 Jul 2022 | 7.83.0 | | Updated: 01.09.2022 --> I will recheck this periodically. Maybe with Docker automation
When you look at the Docker base images for the different Linux distributions we have a clear winner today: VMware Photon OS 4.0! - VMware Photon OS 4.0
is up to date for all the three software packages we looked into.
- openSUSE Leap 15.4
is also a good choice -- and I hope they update their Curl version soon.
- Ubuntu
A good choice if you don't care about NGINX
A good way to run NGINX in an up to date version is a dedicated Docker container with the official image --> https://hub.docker.com/_/nginx/.
This would give you always the latest version of NGINX and an update is easy to deploy. The older version of Curl should not be an issue, because the NGINX binaries are not using it (checked with ldd).
But for OpenSSL and LibCurl you are really depending on the base OS for the most critical parts like OpenSSH.
This is even more important on a Linux machine than on a Docker container image.
The Docker image runs locally and is usually protected by the base image. Daniel Nashed 22 September 2022 17:53:46 The OSX Monterey 12.6 update very unexpected caused the Notes client to fail starting. 12.0.1FP1IF2 is live now on the HCL License & Download Portal and can be deployed to address this SPR. There are plans to release a 1101FP6 based Interim Fix once the fix is ready. A very similar problem also occurred on iOS 15.7 and is fixed in iOS 16.0. So it is apparently a chance introduced unplanned in macOS/iOS. I am not a Mac used but some fellow Ambassadors reported their Notes Mac client is working again. Don't install Notes 12.0.2 EAP5, which does not solve the issue completely. Thanks to the Nomad and the Notes client team for this fast fix and updating the technote with status updates regularly. For updated information check the technote: https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0100507 -- Daniel | |
![[IBM Lotus Domino]](../lotus-domino.gif){kind=small}
![[Domino on Linux]](../domino-linux.gif){kind=small}
![[Nash!Com]](../nashcom.gif){kind=small}
![[Daniel Nashed]](../daniel-nsh.gif){kind=small}