Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

 
alt

Daniel Nashed

 

Auth0 ODIC OpenID with Domino & Some other interesting findings

Daniel Nashed  22 February 2024 00:00:51

We are working on a ODIC setup with a German business partner for a larger German customer.
Auth0 is one of the major providers. We got it working but only with some tricks for now.

It turned out the Auth0 OIDC endpoint has a cache expiration for 15 seconds. This looks like a setting that can't be changed.
The Domino OIDC cache uses the expiration header to invalidate the cache. So our cache on the Domino side was constantly reloading and invalid in some cases.
You really have to have an expiration that is at least a couple of minutes. Better at least 1 hour.


Faking the cache expiration

This has been reported to HCL and the team is working on an enhancement.
Meanwhile I came up with a work-around setting up a Fake provider on a NGINX server to forward the requests.


1. Simple NGINX proxy config overwriting the expired header
2. Certificate created by an exportable MicroCA certificate which is trusted by the Domino OIDC config.

Samesite "Strict" can break the configuration
If the ODIC provider runs under the same domain than your Domino server, you are all set.

But if one domain involved is different, any redirect to a different domain -- even the final request goes to the original URL, will let the browser not send back the cookie.
So you have to change your cookie policy to Lax in this case. See details in Domino 14.0 help ->
https://help.hcltechsw.com/domino/14.0.0/admin/conf_samesite_cookie.html

Settings in general are straightforward

Auth0 sets an ID as the audience. Domino 14.0 allows you to specify the audience in the a OIDC configuration document.
The claim is by  default the e-mail address. So you are all set with the settings.


I hope this helps to configure it out. If you need help for this type of configuration or other OIDC providers or security related configurations, I am always trying to help as long it isn't exceeding my community limits.
But you can always involve me as a business partner as well.

-- Daniel



DNUG local meetup / Stammtisch Rhein-Ruhr 22.02.2024 in Monheim

Daniel Nashed  19 February 2024 12:05:09
Image:DNUG local meetup / Stammtisch Rhein-Ruhr 22.02.2024 in Monheim

Beside the larger DNUG events, there are regional meet-ups.
This time in Monheim a smaller town next to my home town.


DNUG local meet-ups are very informal and open to anyone. Not just DNUG members! Anyone is welcome !!

https://dnug.de/events/stammtische/rhein-ruhr/

Beside the short Domino AutoUpdate presentation the focus is everything that's new in Domino 12.0.x and 14.0. And any topic you want to discuss.


If you are close to this location, it is definitely worth joining us.
An informal event is also a great opportunity to ask off the record questions ..


Right now we are a small number of people. But we will meet in any case.


-- Daniel


Domino Backup/Restore with multiple configurations and targets

Daniel Nashed  18 February 2024 13:34:07

Domino Back/Restore is a flexible framework for native Domino backup.
The dominobackup.nsf plays an important role for backup and restore operation.

It contains the following type of content.
  • Backup/restore/prune configuration
  • Inventory documents for restore operations
  • Restore requests
  • Backup logs

You could run backup with different excludes defined on command-line.
Or just backup selected databases or incremental backups.
But there cannot be different active configurations nor different backup retention in one dominobackup.nsf



A server can only have one active backup configuration?


That's generally true. And this makes most sense for most customer environments and keeps it simple.
Specially when using VSS Writer initiated backup, it would be too complicated to run different type of backups.

Also when using archive style transaction logs and incremental backups, there should be only one backup.

But for full backups initiated by Domino Backup there could be situations where multiple backups might be desired.


Coexistence and migration


Also when switching from one backup integration to another backup integration you want to ensure you can still restore your backups.

For that reason Domino Backup & Restore servertasks support the -configdb parameter which allows to specify a different database on command line.

When migrating to another backup solution, the old database could be renamed and a new database for the new dominobackup.nsf implementation would be put in place.
The old database could be still used for restores.

An admin would select old backups from the old inventor and create a restore request in the old database.
The command-line would just need to reference the old Domino Backup database.

The same parameter works for taking a backup. The database determines the configuration and also contains logs and the inventory.

The -configdb parameter is in already the initial implementation from day one for Domino 12.0.


Limitations


Please use this parameter with care and only for the type of configurations where it makes sense.
The VSS Writer implementation always has to use the main configuration.
And only one backup configuration can be used for archive style backup.



Example command lines


load backup -configdb backup2.nsf

load restore -configdb backup2.nsf



Note


The restore action in the UI always performs are restore without additional parameters and would not work in the second database.

But it could be customized to include the current database in the console command invocation.



My production environment


I am going to introduce the new version of BorgBackup to my production servers soon.
Today I am taking a full backup to a ZFS deduplicating and compressing target twice a day.
 
But I probably I will move one of those backups to a BorgBackup target on a Hetzner Storage box.

-- Daniel





Domino autoupdate.nsf for fast internal software downloads

Daniel Nashed  18 February 2024 10:59:11

Domino Autoupdate has been introduced in Domino 14.0. It offers automatic downloads from My HCLSoftware download, which has been on early access in parallel and has been released at the same time.


My Engage session will go into detail about the functionality with tips and tricks and additional information round both features and the new Domino Download script
(https://nashcom.github.io/domino-startscript/domdownload/).
But I want already provide some details about options available today with simple integrations.

Now that we have and internal repository centrally managed which can be also replicate to remote locations, it would make sense to leverage it for company internal downloads.
Each software document contains one attachment, which we would just need to located and generate a download link.

A download command could look like the following:



curl -LO -u user:password '
https://software.acme.com/software.nsf/download?OpenAgent&filePath=Domino_14.0_Linux_English.tar'


Simplified URL with Domino redirects


To simplify and hide the URL, a simple redirect rule could be added:


Image:User Domino autoupdate.nsf for fast internal software downloads


This would allow to download with this simplified URL.



curl -LO -u user:password
https://software.acme.com/software/Domino_14.0_Linux_English.tar

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                          Dload  Upload   Total   Spent    Left  Speed

 0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100 1068M  100 1068M    0     0   140M      0  0:00:07  0:00:07 --:--:--  152M




Domino 14.0 HTTP download performance improved for uncompressed attachments


The download rate in this example is real world performance between two Hetzner servers.
Both sides are connected with a fast network and have fast disks. At this speed not only the Domino server components are important.

Yes this is a Domino NSF based download using standard Domino HTTPS!

The AutoUpdate team did performance tests with those type of downloads via HTTP and found a bottleneck, which is fixed in Domino 14.0.

Performance of uncompressed attachment download has been dramatically improved.

The download of compressed attachments (LZ1 or Huffman) also improved, but not at the same scale because the back-end requires uncompressing buffering, which has impact on the HTTP download performance.


Best practice for Domino HTTP download performance


A best practice is to store attachments which need to be downloaed always uncompressed. Usually the attachment itself is already compressed.
What we also found out is that DAOS further improves HTTP download performance and attachment performance in general.




Example agent to find documents and generate a redirect


The agent just needs to parse the request, find the document and generate a redirect link with some simple print statements.

I plan to provide a simple "software.nsf database, which complements the functionality of autoupdate.nsf with functionality like this.
It would also provide a simple download link for software.jwt, which contains all the information about available software including the SHA256 checksum.



Option
Public
Option
Declare

Sub
Initialize
         
 
Dim session As New NotesSession
 
Dim db As NotesDatabase
 
Dim WebDoc As NotesDocument
 
Dim doc As NotesDocument

 
On Error GoTo error_handler
 
 
Set db = session.Currentdatabase
 
Set WebDoc = session.DocumentContext

 
If ("GET" = WebDoc.REQUEST_METHOD(0)) Then

         
Call  DownloadRedirect (WebDoc)

         
Exit Sub
 
End If

 
' Other requests are ignored
 
Exit Sub
 
error_handler:

 
Print "Content-type: text/plain"
 
Print ""
 
Print "Error processing request"
 
 
Exit Sub
 

End
Sub


Sub
DownloadRedirect (WebDoc As NotesDocument)
 
Dim KeyName As String        
 
Dim db As New NotesDatabase ("", "autoupdate.nsf")
 
Dim doc As NotesDocument
 
Dim count As Integer

 
On Error GoTo error_handler
 
 KeyName =
StrToken (StrToken (WebDoc.QUERY_STRING_DECODED(0), "&", 2), "=", 2)
 
 
If ("" = KeyName) Then
         
Call SendError ("Not Software specified")
         
Exit Sub
 
End If

 
Set Doc = GetDocByFormula (db, {(Form = "Software") & fileName = "} + KeyName + {"})

 
If (Doc Is Nothing) Then
         
Call SendError ("Software not found: " + KeyName)
         
Exit Sub
 
End If

 
If (doc.Status(0) <> "A") Then
         
Call SendError ("Software not available")
         
Exit Sub
 
End If

 
Print "Location: /" + db.Filepath + "/0/" + doc.Universalid + "/$File/" + KeyName
 
Print {Content-Disposition: attachment; filename="} +  KeyName + {"}
 
Print ""

 
Exit Sub
 
error_handler:

 LogError
"Software - Error: " + Error()
 
 
Exit Sub
 

End
Sub

Easy to use container image providing ICAP support for ClamAV for Domino CScan

Daniel Nashed  18 February 2024 10:21:06

My friend and fellow Ambassador Roberto Boccadoro submitted an Engage session about Domino CScan with ICAP.
Sadly his session did not make it into the agenda. But he is part of two OpenNTF sessions.


This session idea lead to a new OpenSource project I initiated to help with ICAP support.

Thanks Roberto for pushing me to get this implemented! :-)


The new project provides a simple to build container image, which natively offers ICAP services over TLS with a ClamAV container in the back-end.

The container is ready to be consumed with Domino CScan/ICAP (
https://help.hcltechsw.com/domino/14.0.0/admin/conf_scanningattachmentsforviruses.html).

It comes with a docker-compose file which glues the official ClamAV container with this new image.


You can either supply your own certificates or let the container create it's own root CA and issue server certificates.


The project is mainly intended for testing. But I would be interested to get your feedback to see if this could be also work for production in future.


https://github.com/nashcom/nsh-c-icap


The project also offers an optional NGINX container to provide ClamAV over TLS, which I plan to use when I revisit my other project -->
https://blog.nashcom.de/nashcomblog.nsf/dx/domino-antivius-powered-by-calmav.htm


Image:Easy to use container image providing ICAP support for ClamAV for Domino CScan


Running Domino in LXC containers on Proxmox requires a trick

Daniel Nashed  18 February 2024 23:01:10

Now that VMware might not be everyone's darling any more because of it's new mother ship, I took another look at Proxmox.
I know them for quite a while and they are doing a great job. I rebuilt a Intel NUC with 2 TB NVMe disk with the current version of Proxmox.

Proxmox supports full VMs and also LXC -- which is an interesting option for testing in lab environments.
You can setup a new Linux test machine in minutes from a template. And there are ready to use templates for all major Linux distributions.
I had a post long time ago about Proxmox automation on command-line.

LXC needs special settings during install


My first attempts installing Domino native and in a Docker container crashed and burned.

It turned out the UID/GID mapping for LXC didn't play nice with the high UID/GID the Domino webkits have.

Here is an example of the owner with names and UID/GIDs for the Domino webkit:

-rwxr-xr-x nightly/scnotes
-rwxr-xr-x 51714544/252601622  



When tar is started as root user, the original owner will be restored by default.
Usually this isn't a big deal. But it breaks the installation in a LXC container, if the UID/GID cannot be mapped (the default range is 65535).

See error messages at the end of the post.

Work around for the installer


The work-around is to untar the files with
--no-same-owner option.

After the files are extracted with this trick, the InstallAnywhere installer still fails, because it extracts the installer run-time JVM in the same way.

But GNU tar provides a way to export a variable to add options to the tar command-line invocations.
This export does the trick to install Domino native and in a container image with Docker or Podman inside a LXC instance.


export TAR_OPTIONS=--no-same-owner


I have tested with Ubuntu LTS 22.04 and CentOS Stream LXC instances.
The change is already in the Domino Container GitHub repository in the develop branch (along with other functionality I am currently testing).


Proxmox might get more popular and is a great platform


I guess some of you will take a closer look into Proxmox soon. So this information might be helpful.
Proxmox offers a couple very nice details. Like you can specify a download URL for ISO image including checksum calculation.
The virtualization is KVM based using Linux kernel virtualization. And the Debian OS supports ZFS out of the box.

Proxmox also provides a backup server and also has native Ceph support.


Support Hint


You need to be a bit careful with LXC containers. Like with Docker containers the kernel is shared with the LXC instances.
The glibc is provided by the LXC instance. But the kernel is the host's 6.5 kernel, which is currently unsupported by Domino -- but works.

Specially for Linux based environments Proxmox is a great option!
If you want to play with it, you can also run it with embedded virtualization inside different hypervisor like VMware ESXi.

-- Daniel

Image:Running Domino in LXC containers on Proxmox requires a trick


For reference and searching. here are the error messages

Extracting the tar

tar -xvf Domino_14.0_Linux_English.tar
linux64/
linux64/tools/
linux64/tools/installLinux.bin
tar: linux64/tools/installLinux.bin: Cannot change ownership to uid 51714544, gid 252601622: Invalid argument
linux64/tools/removeFiles.txt
tar: linux64/tools/removeFiles.txt: Cannot change ownership to uid 51714544, gid 252601622: Invalid argument
linux64/tools/nui.cfg
tar: linux64/tools/nui.cfg: Cannot change ownership to uid 51714544, gid 252601622: Invalid argument


Running InstallAnywhere

/install

HCL Domino for Linux/Unix Install Program
 ----------------------------------------
Preparing to install
Extracting the JRE from the installer archive...
Unpacking the JRE...
The included VM could not be unarchived (TAR). Please try to download the installer again and make sure that you download using 'binary' mode. Please do not attempt to install this currently downloaded copy.






Certificate ASN.1 Decoding online

Daniel Nashed  14 February 2024 21:02:48

Now that I posted the TLS 1.2 interactive information side today, some of you might also want to get details out of certificates.

Certificates are usually public information. So it should be OK to paste them into the website https://asn1js.eu/.
But there is a GitHub project referenced and you could run it also locally.

The inner guts of certificates are presented in ASN.1. When you ever has looked at OpenSSL C code, you will recognize the structures.

The interactive parser can be quite helpful if you ever need to leave the normal path working with OpenSSL command line converting certs between PEM, DER, PKCS12 and other formats.

The ASN.1 form is basically what you get when you convert to DER. It's a binary format fun to read. But usually you don't have to look at it. The normal OpenSSL code and other security libs hide most of the complexity.

On the website you also find some examples of certificates you can load.

-- Daniel


Image:ASN.1 Decoding online


The Illustrated TLS 1.2 Connection -- Every byte explained

Daniel Nashed  14 February 2024 12:16:08

While debugging a TLS connection issue, I ran into this website -->https://tls12.xargs.org/

It provides more details then most admins ever want to know. But it is a great resource understanding a TLS connection.
In my case I was looking for the first bytes when sending the server certificate in TLS 1.2 certificate handshake message --> https://tls12.xargs.org/#server-certificate/annotated

The certificate is just a ANS.1 DER encoding. But it is prefixed with a header -- which I was looking for.
This website saved me some time reading thru the RFCs and is a more than valuable resource understanding details and even provide information about command-lines to get further information.
If you really want to know about certificates this page also dives into the ASN.1 format -->
https://tls12.xargs.org/certificate.html#server-certificate-detail/annotated

Even if you just want to get a basic understanding, this page is really cool. And you can drill down to the last byte of the data going over the wire.

Domino has debug settings for all of this information to troubleshoot connection issues etc.
There are SSL debug level settings for different parts of the TLS connection and even lower level NTI (Notes network abstraction layer) to get all those bytes captured.
But also OpenSSL provides those details running with -debug.


Something to try is the start of a SMTP STARTTLS connection:

openssl s_client -connect mail.acme.com:25 -starttls smtp -crlf -debug

-- Daniel


Image:The Illustrated TLS 1.2 Connection -- Every byte explained



Meet me at Engage 2024, April 22-24 in Antwerp

Daniel Nashed  11 February 2024 14:57:21

Engage is one of my two conference highlights every year along with DNUG conference.
The year went by so quickly again. But a lot happened in the past year.

I am looking forward to another great conference and meet many of you and also discuss about projects I am working on.


For all details about the conference and registration check the homepage ->
https://engage.ug/

Looking forward to see you at Engage!


Daniel


My Sessions


The main session will be about the new Domino 14.0 AutoUpdate feature.
I am also part of the Linux round table session joining Bill and Thomas.

Domino on Linux and specially also Domino on Linux container environments have been a big part of my work in 2023.


But there have been a couple of other projects I am involved with. Including some new OpenSource projects as listed below.
Some of the projects might play a role in other sessions. Like the container project will be part of Martjin's Domino container session.

The following are the main projects with some sub projects I am mainly working on.



HCL OpenSource GitHub projects


Domino Container

https://github.com/HCL-TECH-SOFTWARE/domino-container/

Domino Certificate Manager (CertMgr)

https://opensource.hcltechsw.com/domino-cert-manager/

Domino Backup

https://opensource.hcltechsw.com/domino-backup/

Domino One Touch Setup

https://github.com/HCL-TECH-SOFTWARE/domino-one-touch-setup/


Domino Linux

https://github.com/HCL-TECH-SOFTWARE/domino-linux/



Nash!Com GitHub projects



Domino Start Script

https://nashcom.github.io/domino-startscript/

Contains also


New Domino Download Script
        -->
https://nashcom.github.io/domino-startscript/domdownload/
Domino Container control (domctl)
-->
https://nashcom.github.io/domino-startscript/dominoctl/



Domino Borg Backup Integration V2.0

https://github.com/nashcom/domino-borg/

nsh-c-icap integration

https://github.com/nashcom/nsh-c-icap/


Domino Install Project

https://github.com/nashcom/domino-install/


Nash!Com Tools

https://github.com/nashcom/nsh-tools

Contains multiple tools. See project sub-readme files for details


nshmailx
  ->
https://github.com/nashcom/nsh-tools/tree/main/nshmailx/
nshchiper
->
https://github.com/nashcom/nsh-tools/tree/main/nshcipher/
nshjwt
     ->
https://github.com/nashcom/nsh-tools/tree/main/nshjwt/


Domino Backup customized and centralized logging

Daniel Nashed  11 February 2024 14:07:51

This question came up last week in a business partner workshop.
The partner wanted to centralize the logging of all Domino backup instances.


dominobackup.nsf intended per server. You could configure a global configuration database and local instances for the backup inventory.
In theory it could be one database for multiple could be also replicated in smaller environments.
This would not be recommended.

But there is an easier way for a centralized overview of all your Domino backups.


Customizing mail notification


The mail notification can be completely customized with formulas and you could even run your own agent.

The recipient of the message is formula based and could take into account any field of the log document.
And also the status check can be customized.

By default the sender address for the message is the server and can also be customized.
Usually it is a good idea to keep the server as the sender. But in some cases this might need to be customized as well.


The more interesting part is that the notification mail isn't just a mail with a summary -- which also be customizable changing the notification form (compute with form is used).
The resulting mail contains all the fields from the original log document. The mail could be send to a mail-in database. Which allows centralized logging.

In addition you could plug in any logic with the agent call-out hook also defined in this configuration.


You find a documentation for those fields here -->
https://opensource.hcltechsw.com/domino-backup/interface/#notification-tab

Image:Domino Backup customized and centralized logging


Image:Domino Backup customized and centralized logging


Links

    Archives


    • [HCL Domino]
    • [Domino on Linux]
    • [Nash!Com]
    • [Daniel Nashed]