 Daniel Nashed | Daniel Nashed 22 September 2022 17:53:46 The OSX Monterey 12.6 update very unexpected caused the Notes client to fail starting. 12.0.1FP1IF2 is live now on the HCL License & Download Portal and can be deployed to address this SPR. There are plans to release a 1101FP6 based Interim Fix once the fix is ready. A very similar problem also occurred on iOS 15.7 and is fixed in iOS 16.0. So it is apparently a chance introduced unplanned in macOS/iOS. I am not a Mac used but some fellow Ambassadors reported their Notes Mac client is working again. Don't install Notes 12.0.2 EAP5, which does not solve the issue completely. Thanks to the Nomad and the Notes client team for this fast fix and updating the technote with status updates regularly. For updated information check the technote: https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0100507 -- Daniel Daniel Nashed 14 September 2022 23:50:33 One Touch setup (OTS) is quite powerful tool. But sometimes you need to know exactly what happens and combine functionality to make best use of it. With Domino 12.0.2 OTS creates certstore.nsf automatically and you can let it create a MicroCA for you. That's a standard flow in Domino 12.0.2 and easy to configure. Below you find the minimum configuration, which generates a RSA 4096 key.
There are additional settings to use a ECDSA key instead for example. "security": { "ACL": { "prohibitAnonymousAccess": true, "addLocalDomainAdmins": true }, "TLSSetup": { "method": "dominoMicroCA" } } }, When the server is configured OTS invokes CertMgr for you to create the MicroCA and the certificate with 1 year expiration.
This MicroCA cert is only intended for testing and to have a safe server configuration out of the box.
But what if you want to use a Let's Encrypt certificate instead?
There is a quite simple way to just find and update the existing document with a appConfiguration. And if you specify notes.ini CertMgr_ACCEPT_TOU=1 the ACME account license agreement will be automatically accepted (already part of 12.0.0).
The following JSON takes the existing document and just updates it to use Let's Encrypt (in my case the Staging account) and submits the request. CertMgr will handle the quest and the TLS Cache will be updated immediately when the certificate is stored in certstore.nsf. In case the Let's Encrypt request fails, the TLS Cache still holds the MicroCA certificate. The request is in error state, but the certificate is still valid. This is intended by design to prevent automatic certificate request updates to fail existing services. The status of the document is running into an error, but the existing certificate is still used. Having Let's Encrypt a standard OTS configuration might not help in many customer environments. So having it configured this way, will help admins who really know that their environment is properly configured for inbound HTTP-01 ACME challenges.
Making it a general feature would probably cause a lot of support calls.
Even reading this living troubleshooting doc should already help to understand why HTTP-01 ACME challenges are sometimes failing --> https://github.com/HCL-TECH-SOFTWARE/domino-cert-manager/blob/main/docs/troubleshooting_acme_challenges.md I hope this gives you additional ideas for OTS in your deployments. -- Daniel "notesINI": { "CertMgr_ACCEPT_TOU": "1", ... "appConfiguration": { "databases": [ { "filePath": "certstore.nsf", "action": "update", "documents": [ { "action": "update", "computeWithForm": true, "findDocument": { "Form": "KeyFile" }, "items": { "Status" : "O", "Provider" : "A", "AcmeAccount" : "LetsEncryptStaging" } } ] }, ... Daniel Nashed 11 September 2022 10:17:08
Usually OpenSSL is the tool of choice for all type of certificate operations.
But what if no OpenSSL command line is available? Like in a Domino container where you can't install software?
After some research, I came up with the keytool, which is part of the JVM Domino ships.
It turned even out to be an easier way to download the certificate chain with this command line:
/opt/hcl/domino/notes/latest/linux/jvm/bin/keytool -printcert -rfc -sslserver 127.0.0.1 > certs.pem
This example connects to the local server over port 443 (default) and loads the complete certificate chain.
The option -rfc is important to write the certificate in PEM format.
Without the -rfc option, you get all the details about the certificates in the chain instead:
/opt/hcl/domino/notes/latest/linux/jvm/bin/keytool -printcert -sslserver 127.0.0.1
Certificate #0
====================================
Owner: O=Automation MicroCA Certificate
Issuer: CN=DominoMicroCA, O=AutoTestLab
Serial number: 69e50f549cb8e0d294ad2c6a884778c0
Valid from: Sat Sep 10 10:03:17 CEST 2022 until: Tue Sep 12 10:03:17 CEST 2023
Certificate fingerprints:
SHA1: 4B:77:49:FB:95:3E:02:F5:8C:F7:C7:76:C1:F5:7F:EA:7B:60:CD:9C
SHA256: 9E:E7:61:FA:D1:DA:57:AA:89:3F:A8:F8:F8:CB:69:D6:7F:DB:8A:18:C4:BB:01:2F:85:FD:1F:39:9B:69:8D:42
Signature algorithm name: SHA256withECDSA
Subject Public Key Algorithm: 384-bit EC (secp384r1) key
Version: 3
Extensions:
#1: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
serverAuth
]
#2: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
DNSName: automation.notes.lab
]
Certificate #1
====================================
Owner: CN=DominoMicroCA, O=AutoTestLab
Issuer: CN=DominoMicroCA, O=AutoTestLab
Serial number: 4afa2efe1b11b03f88fc34d76b3574a2
Valid from: Sat Sep 10 10:03:17 CEST 2022 until: Mon May 26 10:03:17 CEST 2025
Certificate fingerprints:
SHA1: 9C:B0:FD:60:25:B9:6D:8D:2C:F5:F3:CB:CF:F6:B8:73:EB:E9:22:EC
SHA256: 7C:DA:2D:31:7A:7A:C9:B4:7B:24:36:0E:4F:A5:A6:81:D5:27:DB:45:57:03:59:61:2F:20:01:C0:BA:52:71:1B
Signature algorithm name: SHA256withECDSA
Subject Public Key Algorithm: 384-bit EC (secp384r1) key
Version: 3
Extensions:
#1: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
CA:true
PathLen:2147483647
]
#2: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
Key_CertSign
]
#3: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
DNSName: automation.notes.lab
] Daniel Nashed 10 September 2022 11:49:40 Trivy (https://aquasecurity.github.io/trivy/) is a very interesting tool to scan container images, configurations, GitHub projects and more. I have used it to improve some of the configuration options of the container image. Trivy supports a JSON formatted output with much more information, than the summary returned on a scan --> https://aquasecurity.github.io/trivy/v0.17.0/examples/report/ Here is an example command-line: trivy image -f json -o /tmp/trivy_hcl_ubi9.json domino-container:ubi9 I took the result file and imported it into a Notes database. It's pretty simple using the Lotus Script class and the JSON parser. An agent parses the result and updates the database.
It turns out some CVEs are reported more than once depending on the packages you have installed.
The database consolidates the report and provides a full overview. Different base images have different CVEs It's pretty interesting to see how different the Docker base images from different distributions deal with security patching. I looked into current versions for important libs earlier and found out that VMware Photon is the clear winner, Followed y SUSE Leap and SUSE Enterprise. That's going to be a separate post soon. But also from security patch level point of view, VMware Photon 4.0 and the current SUSE Enterprise (https://registry.suse.com/) and Leap (https://hub.docker.com/r/opensuse/leap) images are ahead of the curve and have no single vulnerability reported for the packages required for Domino. The Domino container project supports different base images to build on. The default is currently CentOS Stream 8, which is a commonly used based image. I also have just added support for SUSE Enterprise -- which surprisingly does not install awk by default and caused the Domino installer to fail. Not sure not sure it would make sense to switch to another default image from security point of view, because the RedHat base images are a behind VMware Photon and SUSE Leap and Enterprise versions. Feedback? What are you experiences with container security? Which is your favorite Linux base image and why? Would the analysis database be interesting for others? Should I make it available open source as well? -- Daniel Examples:   Daniel Nashed 4 September 2022 20:08:34 Rancher Desktop is a great all-in-one desktop environment. When running it with the Docker back-end you have all in one environment for development and run-time. For a server, K3s (https://k3s.io) is my platform of choice. It is production ready and easy to deploy. For Kubernetes, you always need a registry to pull images. As soon you need custom images, you will need a registry to upload and download your image. K3s allows you to configure private registries. You could use any registry. I am just running the registry Docker image on Podman in my environment. What I am describing here is just for a simple server test environment. In production I would usw a Harbor registry (https://goharbor.io). 1. Create a local registry
podman run -d -p 5000:5000 --restart=always --name registry registry:latest --- 2. Add the insecure registry to Podman
vi /etc/containers/registries.conf.d/localhost.conf [[registry]] location = "localhost:5000" insecure = true --- 3. Tag image and push to registry
podman tag hclcom/domino:12.0.2EAP4 localhost:5000/hclcom/domino:latest podman push localhost:5000/hclcom/domino:latest --- 4. K3s add custom registry
vi /etc/rancher/k3s/registries.yaml mirrors: localhost: endpoint: - "http://localhost:5000" systemctl restart k3s You find details here: https://rancher.com/docs/k3s/latest/en/installation/private-registry/ Daniel Nashed 3 September 2022 08:48:26 ZFS is my favorite file-systems!
Very flexible, with compression, de-duplication, snapshots, encryption support.
It is included in Ubuntu and SUSE -- but not RHEL and CentOS.
This means it needs to be build for the distribution.
CentOS Stream 9 was a pending deliverable and I just checked, it is finally available.
https://openzfs.github.io/openzfs-docs/Getting%20Started/RHEL-based%20distro/index.html
This was the last component missing for CentOS Stream 9.
Daniel Nashed 20 August 2022 12:13:46
Hmmm ... I have been looking for a solution for a while and I was really surprised there is no good solution out there.
Docker for example has credentials helper to protect passwords.
I just found an approach to protect my AWS credentials and this brings up the secure key/value store I started to implement first in Domino and then on Linux level.
Usually your credentials would look like the following - A simple to read file, which cannot be protected.
I mostly use AWS CL it in a root user context, which has high permissions on the machine anyway -- but there are also ways to protect data from the root account!
Example: .aws/credentials
[default]
aws_access_key_id = XKIAZOXNXRHW7IPXCBBB
aws_secret_access_key = SY7/bI68F9Notess3NGFE/O7G70TdvpuT7wwV9Xi
You can configure AWS CLI to get it's credentials from another program.
Obviously the bash example is not intended to be used in production.
But I will probably extend my secure key/value store project to return AWS credentials in JSON format.
If you have an application, which could safely return data, the configuration would look like this:
Example: .aws/credentials with credentials helper
[profile developer]
credential_process = /local/aws/credentials.sh myaccount
Reference: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sourcing-external.html
Secure Key/Value store idea
My vault idea isn't new and I played with remote and local callers.
For remote callers a TLS enabled interface authenticated by IP and secret could make sense
Local callers could be controlled by process calling and other aspects.
Building a secure key/value store would involve storing the data encrypted on disk and the lookup key could be defined in the configuration. And a separate secret could be used to decrypt the blob.
I don't know how much interest would be out there for a generic tool, which might even help us to protect Domino server.ids.
The first version will be probably for my current needs instead of a generic tool.
I have played with OpenSSL encryption APIs some weeks ago.
Using AES 256 for encryption would be a good idea and isn't that difficult to implement -But it would result independency on OpenSSL.
Any feedback about this new idea is really appreciated.
- Would be a generic tool interesting?
- Would you have specific use cases?
- How do you protect your AWS credentials today?
-- Daniel
Daniel Nashed 13 August 2022 23:06:05 Docker is an interesting environment also for local development and testing.
In some use-cases you might need Domino in combination with additional containers.
But this still only allows dominoctl to maintain one container.
With the latest update you can run different type of configurations.
Each user can have it's own configuration, located in .dominoctl in the user's home directory.
Or there can be project configurations in the current directory.
Here is the full documentation --> https://nashcom.github.io/domino-startscript/dominoctl/#docker-compose-support Daniel Nashed 30 July 2022 17:07:15 The Nomad server is a new offering to directly add Nomad support to your Domino server instead of using a SafeLinx server. Recently I added a SafeLinx container to the Domino community project. Now I am adding the Nomad Server to the Domino image as a new build options. The build script now has a new option, which works like other options like -verse and -capi. It just adds the Nomad server to the image and exposes the standard Nomad port 9443 in the new image. Installing the Nomad Server is basically just an extract of the tar into the Domino binary directory. The result is an image, with out of the box Nomad web support. Even for a new server Nomad Web could be available configured via OneTouch Setup. The only challenge here is to get a trusted certificate. This can be Let's Encrypt or any other certificate you import via PEM or P12. Or you could export the trusted root from a Domino MicroCA and add it as a trusted root to your browser.
The last step is probably only a good idea for a quick test environment. ./build.sh domino 12.0.2EAP3 -nomad ... 12.0.2EAP3 [OK] Domino_12.0.2_Linux_English_EAP3.tar 1.0.5 [OK] nomad-server-105-beta-domino-linux-1202.tgz Daniel Nashed 25 July 2022 09:46:41 In contrast to Redhat based systems with yum/dnf, SUSE updates are only happening inside the current service pack. A zypper update only brings you to the latest updates within a service pack. But if you skim the update instructions it boils down to the following commands, which I performed today to bring my server from SP3 to SP4. 1. Bring your server to the latest patches for your current service pack and reboot: zypper update reboot 2. Swich to the new repositories zypper --releasever=15.4 refresh 3. Update to service pack 4 zypper --releasever=15.4 dup 4. Final reboot reboot 5. Check Linux version cat /etc/os-release Of course you should backup and check if you have any special conditions required for applications installed. But for a standard server this should be the required steps needed. Oh the hole process just takes a couple of minutes and the first reboot was needed, because the kernel changed. | |
![[IBM Lotus Domino]](../lotus-domino.gif){kind=small}
![[Domino on Linux]](../domino-linux.gif){kind=small}
![[Nash!Com]](../nashcom.gif){kind=small}
![[Daniel Nashed]](../daniel-nsh.gif){kind=small}