 Daniel Nashed | Daniel Nashed 31 May 2023 17:12:57 Domino backup is around since 12.0 and it got improvements in every release. There are not many current AHA ideas for Domino Backup & Restore. One smaller feature you can see in EAP1 is the backup of the notes.ini. The notes.ini gets the same type of date stamp you know from the log. And it is attached to the backup log document and/or written to the backup configuration directory. The backup log document is cleaned up with the purging of a backup along with the file on disk with the same name if configured. This closes a small gap in for backing up relevant information for a disaster recovery. Another important part would be the server.id, which should never be backed up to the same location for security reasons. My personal wish would be to have ID Vault support for server.ids. With this small change you have a notes.ini backup always available -- without any restore operation. Usually the backup log directory is backed up as well as the dominobackup.nsf. But in a disaster case for a restore a database location might not be the right way. Domino backup exports the configuration in DXL and can even read the configuration from the DXL instead of the NSF for a restore. And there is a way to rebuild the backup inventory for a backup from the log documents. -- Daniel  Daniel Nashed 31 May 2023 17:10:27 
Notes/Domino/Traveler EA1 has been released today.
If you want to know what you will get in EA1, there is a web cast tomorrow
Webinar: Grab a Sneak Peek of Our Upcoming HCL Domino v14 Release -- June 1, 2024, 10 AM EDT
There is webinar showing the highlights of the first code drop.
Link to register --> https://register.gotowebinar.com/register/7634955298071775582
New software download experience for 14.0.0 EA1 software first
Before you start download from Flexnet, HCL is starting an official preview for the new download experience.
Right now only with the new software which is part of the early access program.
I have used it already to download software to my lab, to my build environments and the DNUG Labg.
https://my.hcltechsw.com/
Domino V14 Early Access forum
For questions, feedback and additional information, you should really joint the EAP forum -> https://hclsw.co/domino14-eap-forum
HCL Domino Container Community Image
The build script has been already updated and the container images are tested.
https://opensource.hcltechsw.com/domino-container/
You can build a Domino container including a Nomad container if you like. Or just the Domino container image:
./build.sh domino 14.0.0EA1 -nomad=1.0.8-12.0.2
And on top you can add Traveler 14.0.0 EA1
./build.sh traveler 14.0.0EA1 -from=hclcom/domino:14.0.0EA1
You can expect separate blog posts over time about the new functionality.
I have tested out most of it already in the DNUG Lab and I am in preparation for the conference and workshop.
Of course the lab and the presentation will focus on Domino V14.0.0 EA1.
-- Daniel
Daniel Nashed 30 May 2023 08:06:42
The standard ingress controller for K8s is usually NGINX.
K3s uses Traefik as the out of the box ingress controller (because it needs to be available on more platforms etc.), which needs a bit different configuration compared to NGINX.
For my DNUG workshop I am looking into K3s explicitly, because it is a much smaller, easier to handle distribution for a lab environment.
RKE2 comes with NGINX. But K3s uses Traefik.
Traefik has a bit different concept and a very nice dashboard, which is disabled by default.
The dashboard itself is a good way to show how to configure a service and an ingress controller using Traefik.
I am using my own TLS secret, which I imported manually (see one of last weeks posts).
There isn't much documentation about how to define an Ingress with Traefik.
That's why I am adding this configuration below as an example.
-- Daniel
---
apiVersion: v1
kind: Service
metadata:
name: traefik-dashboard
namespace: kube-system
labels:
app.kubernetes.io/instance: traefik
app.kubernetes.io/name: traefik-dashboard
spec:
type: ClusterIP
ports:
- name: traefik
port: 9000
targetPort: traefik
protocol: TCP
selector:
app.kubernetes.io/instance: traefik-kube-system
app.kubernetes.io/name: traefik
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: traefik-dashboard
namespace: kube-system
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: websecure
traefik.ingress.kubernetes.io/router.tls: "true"
spec:
ingressClassName: traefik
tls:
- secretName: tls-secret
rules:
- host: traefik.acme.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: traefik-dashboard
port:
number: 9000
--- Daniel Nashed 29 May 2023 12:52:29 Depending on what you ask ChatGPT the answers are quite good. But ChatGPT also derives information and is not always right. For example for Lotus Script it puts together information from multiple different areas and also makes up new properties, it thinks it knows about. Sometimes it is also mixed up with different classes. I have tested it with various type of questions with real world more complicated questions for Lotus Script but also for LibCurl code and others. It even makes up notes.ini variables which are not existing. I got questions from another partner about certain parameters, which turned out to not exist. And the source of the info turned out to be ChatGPT. It's not wrong to use ChatGPT to get an idea and it can be very helpful pointing your to solutions. But please make sure you validate what you get back and do not take it 1:1 as ultimate truth. Specially make sure you don't talk to me, other consultants or HCL support about information from ChatGPT in a way that implies that those parameters exist and they are validated. One best practice is to take parameters you get and search them on your own to find them in another context. -- Daniel Daniel Nashed 29 May 2023 12:41:02
Domino One Touch Setup has been designed with flexibility in mind, with special focus on getting a server up in a secure way.
On Docker you can just mount PEM files into the container. On Kubernetes TLS Certificates and Keys are stored in secrets.
Personally I am not a big fan of storing PEM files on disk. But you could at least set a password on the PEM file you import.
Here is a basic example how to create a secret on K8s and reference it in OTS.
Even the simple environment variable setup supports the security settings for CertMgr.
Of course the same functionality is also available with the more flexible JSON based configuration.
For simplification I am configuring a Domino server without volume and other options to just focus on configuring TLS.
Applications like NGINX expect certificates and keys in separate files. This is also the style used by K8s for an Ingress and other components requiring TLS Certificates.
kubectl create secret tls nginx-ssl --cert=cert.pem --key=key.pem
CertMgr expects the key and certificate chain in one file.
To convert a PEM file to a secret, you can use a similar command. It will just wrap the PEM file into a secret.
kubectl create secret generic domino-tls --from-file=tls.pem
In your pod configuration you can just mount your secret and specify it in your OTS configuration.
Once the server is running, you can check out the TLS credentials doc directly on the console.
In my example this is a Let's Encrypt wild card certificate created on another CertMgr instance.
tell certmgr show certs
[000580:000002-00007F9CE86D1DC0] Subject key identifier Key info Expiration KeyFile/Tag Host names (SANs)
[000580:000002-00007F9CE86D1DC0] ------------------------------------------------------------------------------------------------------------------------------------------------------
[000580:000002-00007F9CE86D1DC0] 62AB F770 B3EF 1091 ... NIST P-256 68.6 days *.lab.dnug.eu lab.dnug.eu
[000580:000002-00007F9CE86D1DC0] ------------------------------------------------------------------------------------------------------------------------------------------------------
[000580:000002-00007F9CE86D1DC0] 1 TLS Credentials
Side Note:
You can even specify an export password, if you want to be able to export the TLS Credentials doc later.
But take care to specify a password with sufficient entropy, else your setup will fail. A good idea is to test the password in CertMgr UI creating an exportable key.
-- Daniel
Simple example pod definition
apiVersion: v1
kind: Pod
metadata:
name: domino-nashcom
namespace: default
spec:
imagePullSecrets:
- name: regcred
- env:
- name: LANG
value: "en_US.UTF-8"
- name: SetupAutoConfigure
value: "1"
- name: SERVERSETUP_SERVER_TYPE
value: first
- name: SERVERSETUP_ADMIN_FIRSTNAME
value: "Daniel"
- name: SERVERSETUP_ADMIN_LASTNAME
value: "Nashed"
- name: SERVERSETUP_ADMIN_PASSWORD
value: "domino12rocks"
- name: SERVERSETUP_ADMIN_IDFILEPATH
value: "admin.id"
- name: SERVERSETUP_NETWORK_HOSTNAME
value: "domino12.dnug.eu"
- name: SERVERSETUP_ORG_CERTIFIERPASSWORD
value: "domino4rocks"
- name: SERVERSETUP_SERVER_DOMAINNAME
value: "NashCom"
- name: SERVERSETUP_ORG_ORGNAME
value: "NashCom"
- name: SERVERSETUP_SERVER_NAME
value: "domino12.dnug.eu"
- name: SERVERSETUP_SERVER_SERVERTASKS
value: "replica,router,update,amgr,adminp,http"
- name: SERVERSETUP_SECURITY_ACL_PROHIBITANONYMOUSACCESS
value: "true"
- name: SERVERSETUP_SECURITY_TLSSETUP_METHOD
value: "import"
- name: SERVERSETUP_SECURITY_TLSSETUP_IMPORTFILEPATH
value: "/etc/domino/ssl/tls.pem"
- name: SERVERSETUP_SECURITY_TLSSETUP_EXPORTPASSWORD
value: "Domino14TLSExport42EXPORT"
name: domino
image: docker.io/nashcom/domino:latest
ports:
- containerPort: 443
volumeMounts:
- mountPath: "/etc/domino/ssl"
name: domino-tls
readOnly: true
restartPolicy: Always
volumes:
- name: domino-tls
secret:
secretName: domino-tls Daniel Nashed 27 May 2023 15:57:13
The Windows Sandbox is a quite helpful tool. I would wish there would be more documentation.
I had a issue with name resolution and it turned out to be quite weird.
Host name resolution comes from host computer
The host file of the sandbox isn't fully used. The sandbox is designed to work closely with the host network, which also includes host name resolution.
This means that host file entries are coming from the host computer!
The virtual DNS server uses the host computer host file and your host DNS to resolve names.
Edge browser also checks Windows sandbox host file
This wasn't quite clear to me, because Microsoft Edge browser somehow also uses host entries from the sandbox.
Use case running your Domino and Nomad server inside Windows sandbox
I my case I wanted Nomad web to work inside the sandbox and it needs the host name entry to dispatch traffic to the Domino server over it's FQDN.
For Nomad web I deployed a static mapping in the Nomad configuration as a work-around.
But because there are also other parts of Domino like the server controller and server console client application, I did some more research and found to the entry must be in the host computer's host file.
So you just need to add your test server's FQDN into your hosts file on your local computer before starting the Windows sandbox.
This is usually not a problem, since you can have multiple entries for an IP address like 127.0.0.1.
But still it needs admin permissions to update the host file in c:\windows\system32\drivers\etc\hosts
Example:
127.0.0.1 domino14.dominolab.loc
-- Daniel Daniel Nashed 26 May 2023 10:34:41 
You can get hands on experience with Notes/Domino V14 end of this month. Here are some tips to get prepared.
All of the software is only intended for non-production use! So you should prepare a VM to get started.
But you should really take a look and have a try. Specially for business partners this is a call for action to test their applications with the updated back-end components.
As announced earlier Notes/Domino moved to up to date compilers and a newer Java version. Also the client is 64bit only.
I hope to see many of you in the EAP forum or at DNUG conference face to face.
-- Daniel
Windows Sandbox
A very convenient way is the Windows Sandbox, which is basically a disposable Windows shipped with Windows 10/11.
I am using it all the time for testing. You can map persistent data if you want to keep the data if you shutdown your sandbox.
The sandbox could run a Notes client and Domino server at the same time with the local network.
Check out my earlier Windows Sandbox posts. It's an optional component, which is easier to install and manage than any VM environment.
Domino Community image support for V14 EAP1
Like in the previous years you can expect the community project to support EAP1.
The container image will support building it with Nomad server and other add-ons on top for testing.
(https://github.com/HCL-TECH-SOFTWARE/domino-container)
You can test the server on a Docker server or on Docker desktop, Rancher Desktop or any of your favorite container environments.
and DNUG Conference in June
The DNUG conference mid of June in Siegburg/Germany will be also an opportunity for a hands-on experience --> https://dnug.de/dachnug50/
As soon Domino V14 EAP1 is released, we will bring it to the DNUG lab -> https://dnug.de/dnug-lab/.
Each DNUG member can get access to the lab and I will use the lab in my Domino V14 EAP1 presentation at the conference.
We will also have a booth at the conference for hands on experience.
Domino V14 Early Access forum
There will be an early access forum again in the same way as in the years before. Your existing accounts should still work. I just tested my account and I can already access it.
The URL is already available -> https://hclsw.co/domino14-eap-forum
The forum is already live and contains the system requirements for Notes/Domino/Traveler.
Webinar: Grab a Sneak Peek of Our Upcoming HCL Domino v14 Release -- June 1, 2024, 10 AM EDT
There is webinar showing the highlights of the first code drop.
Link to register --> https://register.gotowebinar.com/register/7634955298071775582
Downloading the early access software
Every customer on maintenance has access to EAP software. Stay tuned for how to download the web-kit installers and container image.
Some Important changes in Notes/Domino V14
Notes
The Notes client will be only available in 64bit. The compiler has changed to Visual Studio 2022 and the Java version is updated to Java 17.
You should start testing all your applications with 64bit and report back in the forum what works and what might fail.
Domino on Windows
The server is also built using the Visual Studio 2022 compiler. But because of the Windows universal run-time it will continue to work in the same way it did in 12.0.2.
The JVM is also updated to Java 17. So you should also start to test all your server side applications.
Server notes.ini is now in data directory
On Linux/AIX and on Windows partitioned servers the notes.ini has been always in data directory.
Now also for standard servers the notes.ini will be created in the server's data directory.
Domino on Linux
The Linux server switched to the Redhat Enterprise 9.1 version. This brings the dependency for glibc 2.34 and higher.
If you are running native on Linux, your OS has to support at least glibc 2.34 !!
This means native Domino on Linux will only run on the latest versions of most of the Linux distributions.
Important: The upcoming SUSE 15.5 release will not have the required glibc support! But you can still run Domino inside a container on SUSE (see option below).
Here are some other distributions which will work:
- CentOS Stream 9
- Rocky Linux 9
- Alma Linux 9
- Ubuntu LTS 24.04
- VMware Photon OS 5.0.
- Current AmazonLinux
- Oracle Linux 9
How to find out your glibc version
The easiest way is to check the version via ldd command.
ldd --version
ldd (GNU libc) 2.34
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
Domino on Docker/Podman/K8s
In case you are running Docker the image provides glibc support. Only the kernel is coming from Linux host.
The HCL image is build on top of the RHEL UBI image version 9.1 corresponding to the RHEL 9.1 version.
The community image will switch to CentOS Stream 9 for the new container image. But it will also support other base images with glibc 2.34+ support.
This is just a starting point. You should really join the EAP forum for more details coming soon.
Daniel Nashed 25 May 2023 09:27:30 Yesterday I worked on a lab configuration based on Windows Sandbox, Domino and Nomad Web. The biggest challenge is to have a trusted certificate for Nomad Web.
Nomad Server running with the Micro CA A Nomad Server can use Domino CertMgr Micro CA Certs. But the root is not trusted in your browser.
I took a closer look and came up with a simple solution. which makes the import dramatically easier. No more searching for the right trust store and handling PEM files manually. Programmatically importing the Micro CA root The first command downloads the certificate chain for your Nomad server into a PEM file. It's using the keytool from Java, which you have already installed, when installing Domino. The Windows Sandbox does not have the OpenSSL command line installed by default. The loop below extracts the root certificate from the chain, because the tool to import the root can only handle a single certificate a a time. And finally the third command imports the MicroCA trusted root into the "root" key Store. Microsoft Edge and also Google Chrome can immediately use it. This will be a component in other scripts you see in future. And this might be also useful for your own deployments. -- Daniel c:\domino\bin\jvm\bin\keytool.exe -printcert -rfc -sslserver 127.0.0.1:9443 > cert.pem for /f "tokens=*" %%a in (cert.pem) do ( if "%%a" == "-----BEGIN CERTIFICATE-----" echo > root.pem echo %%a >> root.pem ) certutil.exe -addstore -f root root.pem Daniel Nashed 23 May 2023 09:27:19 My previous post was mainly about HTTP traffic and I mentioned TLS/SSL don't use the maximum number of connections settings, because they have a SSL/TLS session.
Establishing a new TLS session has significant overhead! And you have to make sure in any application, that those sessions are cached and resumed. I revisited a blog post from 2012 where I explained a fix, which went into 8.5.3. And was enabled in 8.5.4 by default (which turned into the 9.0 release when shipped as far I recall).
There was an issue with the session cache and a new cache had been implemented in 8.5.3. Today the new cache is the default and SSL_USE_ADDSESSION2=1 does not exist any more. SSL_RESUMABLE_SESSIONS=n Number of SSL/TLS sessions cached and can be resumed. Default is still 50. You might want to increase it to 400 or higher for a large server depending on your needs. SSL_SESSION_SIZE=n Default size of the buffer allocated for the entry. If the size is too small, the memory is re-allocated when the session is created. The SSL cert size has impact in the size required. The default is quite small and would not fit current needs in many cases.
At the time the original blog post was written, the size requirements have been different. I have this set to 8000 bytes since that time. So it is hard to tell for me, how often the default size does not fit. If your value is too low, you might see messages like this on your console periodically: "New SSL session data length of X bytes is larger than the current size of Y bytes." "You may want to set the Notes.ini variable SSL_SESSION_SIZE to at least Z bytes." My starting point for tuning I would probably set it to 8000 today. Which would be around 3 MB of memory for 400 session entries. Using ECDSA NIST P-256 keys instead of RSA keys Another important point about SSL/TLS performance is to switch from RSA to ECDSA keys for your web server certificate. This would reduce the computing effort on both ends and would also introduce two modern ciphers selected automatically for ECDSA keys. Domino fully supports ECDSA ciphers up to NIST P-512 since version 12.0.0. But you see from the list in the documentation, that you are absolutely fine today with ECDSA NIST P-256. The two ciphers selected are: - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xC02B)
- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xC02C)
Enable Cipher and session logging So see SSL session resumption and the TLS version and cipher selected, there is log setting which I have enabled on my servers. I also wrote a small log analysis database to evaluate the ciphers. But now with ECDSA keys, I am less interested in checking the cipher selected. set config SSL_LOG_SUCCESS=1 TLS1.2 resumed 79.194.8.70 (52459) -> 172.18.0.3 (443) - ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xC02C)
--- Reference for ECDSA: https://help.hcltechsw.com/domino/12.0.2/admin/wn_ECDSA_cryptography.html Reference for old SPR for the session cache: SPR# SFPN69ET56 (8.5.3) - Added the new Notes.ini variable (SSL_SESSION_SIZE) that can be used to set the default starting size of elements in the SSL session resumption cache. Increasing the size of this variable will result in a lower incidence of elements being reallocated on the fly in order to hold large X.509 client certificates, but will also consume more memory. Beginning in 8.5.2 Fix Pack 3 and 8.5.3, there is a new notes.ini SSL_USE_ADDSESSION2. If set SSL_USE_ADDSESSION2=1 implements a new version of the SSL_ADDSESSION routine that resolves some stability issues. Daniel Nashed 22 May 2023 09:49:47
When performing performance tests with a new tool I wrote based on LibCurl, I noticed many connections in network TIME_WAIT.
It turned out I had persistent connections enabled. But the value was set to low on the server.
Other servers with higher values, had much less TIME_WAIT sessions.
Performing some basic load test using a remote client, it turned out that for a larger number of HTTP requests in a short time, this setting can improve your Domino HTTP server performance.
This also reduces the number of TCP/IP sessions opened.
Most of my servers did already use the following values -- The default setting would be 5 requests, which is a bit too low.
HTTP persistent connection: Enabled
Maximum requests per persistent connection: 100
---
Description: Specify the maximum number of HTTP requests that can be handled on one persistent connection. The default is 5.
Reference: https://help.hcltechsw.com/domino/12.0.2/admin/tune_specifyingnetworktimeoutsonthewebserver_t.html
Simple Test Setup
I had a very simple setup for my quick test. The tool ran the following load:
| Threads | 10 | | Requests per thread | 1000 | | Total Transfer | 56 MB | | Data/Req | 5887 bytes |
Which resulted in the following performance values:
| Test | Average request time (ms) | Total run-time (sec) | | 5 requests per connection | 180 | 200 | | 10 requests per connection | 97 | 103 | | 100 requests per connection | 81 | 83 | | TLS connection | 187 | 207 |
TLS Sessions
One important side note is that TLS/SSL sessions work differently. A TLS/SSL session has way higher overhead than establishing a new TCP/IP session.
Therefore the TLS/SLL stack uses TLS session resumption and tries to reuse sessions as much as it can.
You can see the TLS performance with a ECDSA NIST P-256 key in the table as reference.
Log output from SSL_LOG_SUCCESS=1
TLS1.2 resumed 79.194.8.70 (52459) -> 172.18.0.3 (443) - ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xC02C)
Conclusion
You should really double check, if your Domino HTTP server has persistent connections enabled and in most cases you should really increase value of the request at least to 40. | |
![[HCL Domino]](../lotus-domino.gif){kind=small}
![[Domino on Linux]](../domino-linux.gif){kind=small}
![[Nash!Com]](../nashcom.gif){kind=small}
![[Daniel Nashed]](../daniel-nsh.gif){kind=small}