 Daniel Nashed | Daniel Nashed 28 November 2023 22:35:35
After the official Domino V14 launch on December 7, the DNUG Domino focus group organizes another Deep Dive Domino event.
We will cover all new features and answer your questions about Notes/Domino V14 and latest version of Domino companion products. Time and date are not a coincident ;-) .. and there is a lot to discover. The DNUG Lab is already updated to Domino 14.0 EA3 and I will update it to the GA version as soon it is available. You will get first hands experience. Sorry that this is even is in German. But there will be other events hopefully soon to cover the new features. My favorite features: - Passkey authentication - Domino AutoUpdate - Admin Central Event page --> https://dnug.de/event/dnug-deep-dive-domino/ The registration link is on the page. Looking forward to may of you attending.
-- Daniel  Daniel Nashed 26 November 2023 10:55:36 Introduction The My HCL Software Portal is still an early access offering in parallel to Domino 14 early access. It is planned to replace the Flexnet download soon and way easier and much faster to navigate. The website just works and has awesome performance. Domino 14 AutoUpdate leverages a new software download API to automatically download software into autoupdate.nsf. The download just needs a download token, which can be requested if you are log into https://my.hcltechsw.com/. I have been looking for a way to automatically download software for a couple of years. Domino Download Now with the new portal and this new API it is possible to write a Bash script for full command-line operations including a simple to use menu. There are two different modes. By default the script uses My HCL Software navigation. But alternatively it can also leverage Domino 14 Auto Update software.jwt, which has more granular information and allows a more structured download package browsing experience. The script is a single file, which can install itself and all required packages -- Usually only JQ is missing. More detailed information about the script is part of the start script project --> https://nashcom.github.io/domino-startscript/domdownload/ The script has many useful options and validates the downloaded software using the SHA256 provided in the portal or software.jwt. Implementation details The script uses JQ to parse JSON data and uses Curl to download software. Meta data is cached to reduce the round trip time when navigating the software data. Also the access token is cached to avoid rotating the download refresh token on every request. The first version of the script even comes with it's own performance tracing tool, which helped me to track down some Ubuntu local DNS resolver issues. Have a look and provide feedback If you are running Linux or MacOS in your environment, you should have a look! I am using it every day in many different scenarios. For Windows I added GitBash support. But the better way would be to install WSL and run the script on Linux leveraging /mnt integration for Windows file systems. This is the first version of the script. The script was on preview in the HCL Ambassador community and the Domino 14 EAP feedback forum. I also added domdownload support into the HCL Community Container project. build.sh automatically detects it and if present, the software downloads are fully automated. Example: ./domdownload -autoupdate After selecting the software in a menu, information for the software package found is displayed and a curl based download is started. -------------------------------------------------------------------------------- Selected software -------------------------------------------------------------------------------- [1] HCL Domino Server 14.0 for Linux English - Early Access October 2023 -------------------------------------------------------------------------------- WebKit : HCL Domino Server 14.0 for Linux English - Early Access October 2023 Product : domino Type : server Platform : Linux/64 Version : 14.0 Name : Domino_14.0_Linux_English_EA3.tar Size : 1125591040 SHA256 : 8174c1a927571ba61b4c084c5863de6d7f48ed0d5ab9e293f7476fd41b9835c0 ID : LVZSBZ3rMMjQY4RF36hhb -------------------------------------------------------------------------------- Downloading /mnt/storagebox/software/Domino_14.0_Linux_English_EA3.tar ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1073M 100 1073M 0 0 147M 0 0:00:07 0:00:07 --:--:-- 112M Download completed [OK] Hash verified /mnt/storagebox/software/Domino_14.0_Linux_English_EA3.tar Daniel Nashed 31 October 2023 09:18:50 argv[0] on Linux and Windows
Looking into the APIs for creating new processes shows that and application can specify any string as arg[0] when calling your process. This leads to the conclusion that using argv[0] as a trustworthy information isn't a good idea. I would even recommend not using it at all even for display purposes and would just skip argv[0] completely. What would be a good way to determine the binary name?
On Linux the proc file system contains information about processes. It's by the way a very easy to use standardized way accessing process information. There is even a special link "/proc/self" which represents the running process binary. Using readlink ("/proc/self/exe"...) is a simple way to get the current binary name. On Windows using GetModuleFileName (NULL, ...) is a good way to get the current binary name. -- Daniel Refrences
Linux: https://man7.org/linux/man-pages/man3/exec.3.html Example: execl (szProgram, szProgram, szArg1, szArg2, NULL); Windows: https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessa Example: bSuccess = CreateProcessAsUser( NULL, // Application name pszCommandLine, NULL, NULL, FALSE, NORMAL_PRIORITY_CLASS | CREATE_NEW_CONSOLE | CREATE_NEW_PROCESS_GROUP, NULL, pszWorkDir, &StartupInfo, &ProcessInfo); Daniel Nashed 19 October 2023 08:35:53
Domino V14 is planned to ship end of this year. For Windows the system requirements don't really change, because of the universal run-time.
But for Linux a newer compiler brings new OS dependencies. Specially the glibc version, which brings the base run-time support for C and also the C++ standard libs are important.
An application build with a newer compiler on a newer Linux version does not run on older versions with lower glibc versions.
glibc is the The GNU C Library - https://www.gnu.org/software/libc/
The new version required was released in August 2021 and is part of most current long term release Linux distributions.
HCL Domino 14.0 System Requirements (Early Access Program Drop 3)
https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0105128
The system requirements state the following versions:
RHEL 9.0 Linux-equivalent OS
Equivalent OS with the following kernel/packages:
kernel-5.14 x86_64 or higher 5.14 kernel
glibc-2.34-28 x86_64 or higher
libstdc++-11.2.1 x86_64 or higher
What does that mean for your environment
You will have to bring servers to at least those versions to run.
glibc dependencies are not negotiable and glibc can't be updated. It's an essential part of each Linux OS.
This would mean an update to the following type of long term release versions: - Redhat/CentOS Stream 9.x based distributions (CentOS Stream 9, AlmaLinux 9, RockyLinux 9, Oracle Linux 9, Amazon Linux 2023, ...)
- Ubuntu 22.04 LTS
- VMware Photon OS 5
SUSE Linux Support
Currently the latest SUSE Leap and Enterprise versions ship with glibc 2.31 -- which is too old to run Domino.
You could run a SUSE 15.5 host, with a supported container image. The glibc is part of the image. The kernel is the the kernel of the base Linux running the container (e.g. Docker, Podman, K8s).
SUSE is expected to ship a compatible glibc mid of 2024.
Conclusion
The current Linux distributions all have their major versions updated for a while. For example Redhat/CentOS is at version 9.2.
It is important to keep the base operating system updated.
This is really a required and important change. You should move ahead and plan accordingly.
I have updated most of my servers already to CentOS 9.2 and Ubuntu 22.04 LTS.
Daniel Nashed 14 October 2023 08:12:06 When looking into Domino V14 EA3 autoupdate.nsf downloads I have been testing some proxy connections with different locally installed proxies (Squid HTTP proxy and Dante Socks proxy). On first look the performance seemed to be dramatically different. But on second look the difference was the way I tested the download. The second download via proxy was almost as fast as the performance I had without a proxy. When testing performance, caching can dramatically influence the results. Usually it is disk cache performance where you have to drop file caches on your machine. My HCL Software download performance depends on server side cache In this case it turned out that the My HCL Software download caches the files in the Content Delivery Network (CDN). If nobody from my region downloaded the file for a while, the file needs to be read from the back-end. Testing with different network connections Of course download performance also depends on the internet connection. My AWS machine was surprisingly a lot of slower than my Hetzner server... Looks like they only have giga bit connections for their normal servers. But then 114 MB/sec is as good as it can be! For a first download without the cache I got like 15 MB/sec on a AWS machine and 115 MB/sec for a cached file. On my Hetzner server I got more than double the performance. Once the file was cached I hit a download rate of 365 MB/sec! Another interesting surprise was that my ARM machine was 20 MB/sec faster than the Intel machine (Intel Xeon Processor, Skylake, IBRS, 3300 CPU MHz). And 50 MB/sec faster than a machine with just 1 Intel CPU core. All in all the performance for downloads from the new My HCL Software Download site rocks! To be fair I did the same test with Flexnet. I got some mixed results. But they also cache the download in some way. The first read was half the speed I got for a none cached file on the new portal. But I got comparable download speed once test. Conclusion OK those are not the number you can expect in normal corporate downloads at all. But it shows which factors influence download performance. If you have a proxy which inspects downloaded data, the performance could also look completely difference. But my test was mainly a check if proxy connections works well and to see where different download performance came from. It's still a pretty impressive download performance. -- Daniel Test result with curl command line (no proxy) on a Hetzner machine with ARM CPU (Data center located in Germany) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1052M 100 1052M 0 0 365M 0 0:00:02 0:00:02 --:--:-- 365M Daniel Nashed 8 October 2023 07:04:21 This discussion came up in an OpenNTF Discord channel. The question was if this might be a good idea to keep OS level anti-virus enabled for Notes/Domino files. There is a clear statement from HCL about exclusions. But the technote doesn't explain why those exclusions are important. The exclusion might be different for each anti-virus production in detail. It also depends on customer IT policies how to exclude data. This can be either by path, extension or process. There are also recommendations from some anti-virus vendors stating the same exclusions for their specific product: Guidelines for excluding Notes and Domino directory and files when running an operating system Antivirus https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0093046 Why are exclusions important Performance First of all on access or on write scanning can dramatically slow down performance of a server or client. On a client even loading all the small Eclipse program/binary files can slow down the client. But that's a different story. Stability of the server OS virus scanners run on very low OS level and can block files very low level. If a file gets blocked/removed while Domino is using the file, the process or at least the operation will fail. Depending on the file and the state, this could cause a partial or complete server failure (e.g. crash). Effectiveness of OS level scans Scanning on access often has a time lack as I found out in my tests. So the file might be written anyway and deleted afterwards when using it. Even for not encrypted data, The storage of file data inside a NSF or even DAOS NLO are not really consumable. Often data is also compressed on top of it. For no part of the server OS level anti-virus would add reliable security. How to scan Domino data for viruses and other threads To scan Domino data inside a NSF or insider NLOs. you would need a Domino aware solution. There are business partner solutions out there, which provide mail flow and on rest scanning. HCL implemented the ICAP standard for mail flow scanning in Domino 12.0.2 to leverage external anti-virus appliances supporting the ICAP standard. The interest in this new feature was quite low. And also ideas to improve the CScan solution was not really getting many votes. Even if you have existing gateway solutions to scan attachments, periodically scanning Notes databases would be a requirement high on my wish list. Surprisingly the interest seems to be very low. Also the idea to provide other protocols like ClamAV protocol support (free, open source based anti-virus) was not getting many votes. With CScan in place, adding more protocol and a command-line scan, would be incremental effort. But probably Domino product management would not take it into account with 7 votes. I would be very interested to understand why there isn't much demand in the field, because I had similar low feedback for my ClamAV integration project earlier (see below). -- Daniel AHA ideas to vote for AHA idea - ClamAV support for Domino CScan -> 9 votes https://domino-ideas.hcltechsw.com/ideas/DOMINO-I-2472 AHA idea - Periodic CScan on rest scan of databases -> 7 votes https://domino-ideas.hcltechsw.com/ideas/DOMINO-I-2473 ClamAV related blog posts https://blog.nashcom.de/nashcomblog.nsf/dx/domino-antivius-powered-by-calmav.htm https://blog.nashcom.de/nashcomblog.nsf/dx/revisiting-anti-virus-for-domino-do-you-have-feedback.htm Daniel Nashed 8 October 2023 04:45:45
Domino 12.0.2 added support for SELinux in enforced mode, which is enabled by default by newer installations.
SELinux is a lower level security feature, which can even limit processes running with root permissions.
But the application needs to have a SELinux profile.
I ran into this week on my own on a RHEL 9.2 machine and I got the same problem from a partner yesterday.
It turns out that systemd can't read from /tmp any more. But the Domino service from my Nash!Com start script writes the domino process id into the /tmp folder.
With SELinux enabled you get the following error message when looking into your service status (domino statusd).
The start and stop operations of your server will also hang, because systemctl will hang.
systemd[1]: domino.service: Can't convert PID files /tmp/domino.pid O_PATH file descriptor to proper file descriptor: Permission denied
Admins probably notice first that the service is handing on startup:
Starting systemd domino.service
Systemd seems to not handle this error gracefully and the specified timeout seems not to work.
How to find out what SELinux does not allow
Meanwhile SELinux comes with great tools to analyze issues.
You can see that SELinux prevented systemd to read /tmp/domino.pid.
audit2allow -w -a|grep systemd
type=AVC msg=audit(1696738388.170:348): avc: denied { open } for pid=1 comm="systemd" path="/tmp/domino.pid" dev="dm-0" ino=134293379 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=file permissive=0
How to address this issue
The right location for process pid files would be /run or /var/run (which is a symbolic link to /run) where pid files are usually stored.
Systemd is allowed to read the directory and all sub directories.
But Domino as an application without root permissions cannot write into this directory.
This looks like a catch-22, but can be solved by creating a sub directory for Domino.
A small complication is that the directory would be deleted on every boot and the start script running with the "notes" user can't create directories.
Systemd offers a way to create those sub-directories per application by specifying a RuntimeDirectory directory created below /run.
Once tested for a couple of days I will change the default configuration of the start script to use this new location and new logic.
The install_script bash script would not overwrite existing configurations without specifying the "upd" option.
But it would be in the right location by default for future installation.
To change the configuration manually, change the two following files as listed below.
I have not found any good documentation about this recent change. So this might also help other Linux admins for their own services.
/etc/systemd/system/domino.service
PIDFile=/run/notes/domino.pid
RuntimeDirectory=notes
RuntimeDirectoryPreserve=yes
ExecStart=/opt/nashcom/startscript/rc_domino_script start
/etc/sysconfig/rc_domino_config
DOMINO_PID_FILE=/run/notes/domino.pid
After you made those changes you have to update systemd systemctl daemon-reload
Update 18. October 2023: It tuned out that systemd has a build-in functionality to create a sub-directory for an application. Systemd offers a way to create sub-directories in /run for applications. This is a cleaner approach instead of using ExecStartPre commands. I am leaving those older changes here for reference, because they could be helpful for other use cases. Systemd offers ExecStartPre operations, which are executed before the ExecStart statement starts the Domino service.
Prefixing the command with a plus sign runs the operation with root permissions.
ExecStartPre=+/bin/mkdir -p /run/notes
ExecStartPre=+/bin/chown notes:notes /run/notes
Update 9. October 2023:
There was one more tweak needed. The command to create the directory needs to be mkdir -p -- else the 2nd run of the server fails because the directory is already there.
I pushed the changed start script to GitHub in the develop branch with a new version number 3.7.3.
No new features and I will work on documenting details later. It's more important that this change needs to go out.
You will need to update your configuration! If you did not change anything a "install_script upd" would update the configuration and the service. You will also need to reload the service!
If you changed the configuration you need to change the service and the config manually as outlined above. Daniel Nashed 4 October 2023 09:08:02
Start your downloads.. The final Version 14 early access versions is available.
Most of you already know about the new My HCL software portal to download.
The new portal is in Early Access in parallel to Domino V14 and you you should try this out instead of going to Flexnet ..
Here is the direct link to the EAP download already available https://my.hcltechsw.com/downloads/domino/eap
It also provides links to the most important resoures like what's new and also the EAP forum!
HCL Community Domino project support for 14 EA3
I have just updated the develop and main branch with EA3
And there is something new for your to try out. Traveler is usually an add-on image.
But we added support to add Traveler and Domino Leap as a build option directly to the Domino image.
With the following command-line you can build an image with Domino and Traveler at the same time.
./build.sh domino 14.0.0EA3 +traveler=14.0.0EA3
The Nomad server and also Verse is already included in Domino and automatically installed.
Domino Leap support for EA3.
Meanwhile also Domino Leap has been updated to a newer version. The latest version Domino Leap 1.1.2 now works on Domino V14.
There have been pending changes to work with Domino V14 and the new Java 17 version.
Features you have to try out in EA3
AdminCentral
AdminCentral has improved with more functionality and also a lot of UI enhancements specially on mobile devices.
You should try it out on an iPad and also a phone!
AutoUpdate
AutoUpdate has been enhanced to work now hand in hand with the My HCL Software portal to download and distribute software automatically directly via autoupdate.nsf.
Take look into the documentation and check out the EAP forum for details.
Passkeys for web authentication
Passkeys have been added in EA2. If you did not take a look, you should take a look with EA3, which introduces the UI components in the standard dialog and also additional information in the passkey.nsf.
This includes device type where possible. Last login date and much other functionality.
Passkeys is an awesome new technology which Domino adopted very early. It is more secure than user/password and provides much easier log-in experience for users on desktops and mobile devices.
I will blog separately about passkeys and other features. And you should look into the EAP forum for discussions around it.
DAOS Repair in a cluster! When repair was introduced in Domino 10, there was a small but important blind spot.
Whole database have been repaired. But individual NLOs could not be repaired.
So now in a cluster when NLOs are missing, they can be repaired from another cluster mate. The server which is missing NLOs can pull them from another cluster mate.
Update for important security components
Another impressive detail change.
Domino again updates OpenSSL to a very recent version OpenSSL 3.1.2. Which is outstanding for an enterprise applications.
And it makes Domino again more up to date then most Linux distributions!
Important install changes on Windows
Beginning with Domino 14.0.0 EA3 Domino on Windows does not support running with the System Account any more out of the box.
During install you have to provide an existing user without admin permissions, which will be used to run the Domino service.
All access permissions are set accordingly for the user during install.
This will be quite a change for many administrators and there is an interesting discussion in the EAP forum about this change.
For details you should join the EAP forum.
This is just a quick first info about EAP3. You should look into the documentation and join the forum.
-- Daniel Daniel Nashed 30 September 2023 11:20:11 Domino V14 EA2 added the Nomad Server and Verse to the Domino installer package and installs both automatically with Domino. This is a great move to simplify Domino installations and updates! The container project has support for both for quite a while along with other community only options. There is an add-on image for HCL Traveler and Domino Leap. Both need an additional build step to create a HCL Traveler or Domino Leap container on top of the Domino image. HCL Traveler and Domino Leap option Because HCL also has a separate image and the community image is now the base for the HCL image since Domino 12.0.2 we never changed the layering. But why not add a new option to build Traveler and Domino Leap directly into your Domino image. Here is a sample "all you can build" command-line, which installs all the latest components directly during Domino 12.0.2 container build. ./build.sh domino +traveler +verse +restapi +capi +nomad +leap +domlp=de 12.0.2 [OK] Domino_12.0.2_Linux_English.tar 12.0.2FP2 [OK] Domino_1202FP2_Linux.tar 3.1.0 [OK] HCL_Verse_3.1.0.zip de-12.0.2 [OK] Domino_12.0.2_SLP_German.tar 1.0.7 [OK] Domino_REST_API_V1.0.7_Installer.tar.gz 1.0.9-12.0.2 [OK] nomad-server-1.0.9-for-domino-12.0.2-linux.tgz 12.0.2FP1 [OK] Traveler_12.0.2FP1_Linux_ML.tar.gz 1.1.2 [OK] hcl.dleap-1.1.2.18.zip 12.0.2 [OK] CAPIToolkit_1202_MP.zip Support for the latest versions including RESTAPI 1.0.7 I have also updated the latest version support for all the components.
This is specially interesting for the RESTAPI, which now supports Domino V14. The installer is now split into one for Domino V14 and one for earlier versions. Background is that Domino V14 introduced Java 17. The RESTAPI installer leverages the Domino JVM, which switches to Version 17 in Domino 14. The container build script is now version aware and uses the right installer package for each version. Domino V14 EA2 support The community container project supports Domino V14 EA2 and will introduce EA3 support as soon it is available. But I am pushing the changes just explained already today. I need to double check if all components still work. Domino German Language support I added the LP a while ago and never got any feedback. Installing the LP isn't that straightforward. There is an installer properties file and a LP ini file which defines what to add and replace. I cloned the settings for Domino V14 and need to review it again once EA3 is available. There is still work in progress. But it should work as it is today. I hope you like the additions to the Domino Container Community project. The official HCL image stays the same. All of those options are community only options. -- Daniel Daniel Nashed 26 September 2023 07:00:00 Documentation is always a challenge. This is specially true when it comes to complex topics like SSL/TLS certificates.
Many admins still use their old cook books to get certificates created.
When HCL introduced CertMgr in Domino 12.0 the team asked for feedback in the early code drops.
And the team is keeping asking in public and private forums since then.
Beside "it is still too complicated" and other complains there wasn't much feedback.
Call for action
We really need your help to get it right. We need detailed feedback and questions.
My new plan is to turn questions into FAQs and Howto documents in this GitHub repository.
The repository was mainly intended for CertMgr integrations.
But already contained a living troubleshooting document for Let's Encrypt ACME HTTP-01 challenges where support got most questions.
New GitHub Pages CertMgr site
Now we are turning this project into the home for everything CertMgr related.
If you have questions and feedback, this should be your first place to check in future!
Ideas can be discussed in the new forum in the project. But they should finally turn into official AHA ideas.
I hope this new GitHub page helps you. Please provide feedback directly in the project.
New GitHub pages site with a link to the GitHub project page:
https://opensource.hcltechsw.com/domino-cert-manager/
This is just a starting point and will grow over time with your feedback and questions!
-- Daniel
 | |
![[HCL Domino]](../lotus-domino.gif){kind=small}
![[Domino on Linux]](../domino-linux.gif){kind=small}
![[Nash!Com]](../nashcom.gif){kind=small}
![[Daniel Nashed]](../daniel-nsh.gif){kind=small}