Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...


Daniel Nashed


    Domino CertMgr GitHub Repository with additional material

    Daniel Nashed  26 September 2023 09:00:00

    Documentation is always a challenge. This is specially true when it comes to complex topics like SSL/TLS certificates.
    Many admins still use their old cook books to get certificates created.

    When HCL introduced CertMgr in Domino 12.0 the team asked for feedback in the early code drops.
    And the team is keeping asking in public and private forums since then.

    Beside "it is still too complicated" and other complains there wasn't much feedback.

    Call for action

    We really need your help to get it right. We need detailed feedback and questions.

    My new plan is to turn questions into FAQs and Howto documents in this GitHub repository.

    The repository was mainly intended for CertMgr integrations.

    But already contained a living troubleshooting document for Let's Encrypt ACME HTTP-01 challenges where support got most questions.

    New GitHub Pages CertMgr site

    Now we are turning this project into the home for everything CertMgr related.

    If you have questions and feedback, this should be your first place to check in future!

    Ideas can be discussed in the new forum in the project. But they should finally turn into official AHA ideas.

    I hope this new GitHub page helps you. Please provide feedback directly in the project.

    New GitHub pages site with a link to the GitHub project page:

    This is just a starting point and will grow over time with your feedback and questions!

    -- Daniel

    Image:Domino CertMgr GitHub Repository with additional material

    Who moved my Domino keyfile.kyr files?

    Daniel Nashed  24 September 2023 08:15:26

    Domino security in 2023

    Domino 12.0 introduced a new, standards based and open way to work with web server certificates.

    Instead of using command-line tools like OpenSSL and the Domino kyrtool you can now manage all web server certificates in a domain wide certstore.nsf.
    The new functionality based on the well known text based PEM standard for certificates provides simplified flows and automation options for all type of certificates.

    Domino 12 also introduces the more modern ECDSA (sometimes referred as ECC) keys/certificates which are based on elliptic crypto which has dramatically less overhead.
    Now that Domino 14 is around the corner, you should really switch to Domino V12.0.2 and take advantage of Domino 12 certificate management and many other new security features.

    Moving from keyfile.kyr to certstore.nsf

    The legacy kyr files can be automatically imported into certstore.nsf with a single command-line operation (load certmgr -importkyr all).
    Moving to Domino V12 CertMgr would also be a good opportunity to revisit your SSL/TLS configuration and look into the new Let's Encrypt / ACME integration for external facing servers.
    But in any case you should benefit of the new manual certificate flow, which automates full flow from key creation to CSR and finally importing the received leaf certificate and the whole certificate chain.

    Call for action

    If you are still using your old cook books for managing your web server certificates, you should really move on to the more modern way to renew your certificates once per year (actually 398 days).

    My 2023 OpenNTF session with many details about Domino 12.0.2 security and CertMgr is a good starting point:$FILE/openntf2023_domino_security_final.pdf

    GitHub project

    There is also a GitHub repository including a troubleshooting document for ACME challenges.

    Let's try something new... We have just enabled discussions in the project to provide a central place for all kind of question, ideas and to support each other.

    -- Daniel

    Domino on Apple Silicon Update

    Daniel Nashed  16 September 2023 18:51:22

    Domino isn't designed to run on ARM  emulations are getting better.
    As posted before Domino on Windows ARM works quite well.

    But some developers need a local Domino container for testing.

    The current Docker Desktop and Rancher Desktop versions with Rosetta emulation works quite well.
    The only problematic part is the JVM which doesn't work in an emulated Intel environment.

    This also includes the Domino Install Anywhere installer, which is based on Java as well.

    What you should not try to get working

    Building the Domino container doesn't work well because Java is involved.
    Running Java code doesn't work and the JVM needs to be disabled for Amgr and HTTP

    The Domino disable JVM

    Disabling the JVM needs two notes.ini parameters to be set:


    After you disabled the JVM, you can't run any Java code, which includes all OSGI and also Traveler.
    But beside that the Domino server works quite well.
    An environment like this is only suitable for a local development environment.
    You can't expect great performance because of the CPU emulation.

    Build the image remotely or import the HCL container image

    Building the image on Apple Silicon is very problematic. I got it running with some tweaks, but it is a moving target.
    The best option is to remotely build the image or import it.

    Recommended Setup

    If you have another machine running an Intel CPU it would be a much better choice to run the Domino container on a remote machine.
    This could be a Linux VM on another machine or a remotely hosted Linux machine.

    I would run the Domino server native on Linux with a Docker or Podman server.
    Usually you would operate it remotely by opening a SSH session.

    Avoid remote access for Docker daemon

    Docker uses a client server model. So you could in theory run the Docker daemon on the Linux box and access it remotely from a Docker client running on your Mac.
    I would personally avoid that unless you are on very tight control of the network.

    The connection would need to be secured by a VPN for example WireGuard or other technologies (

    The following configuration should be used with extreme care!

    Usually remote access to the Docker daemon is not needed and should be really avoided unless you know exactly what you do.

    Named Document instead of profile documents since Notes/Domino 12.0.1

    Daniel Nashed  24 June 2023 12:20:39
    Profile documents are a good way to store static information, which should be cached and ready fast without a view lookup.
    But sometimes profile documents are not the right feature to use.

    There is a new functionality in Notes/Domino 12.0.1 to have a named note, which isn't a profile doc.
    This functionality is used in Admin Central in Domino 14 to store some information that is updating and should not be cached like a profile doc.

    It requires a Notes 12.0.1 or higher client and is documented here -->
    Here is a simple example:
            Dim session As New NotesSession
            Dim db As NotesDatabase
            Dim doc As NotesDocument
            Set db = session.CurrentDatabase
            Set doc = db.GetNamedDocument ("MyNamedDoc")
            ForAll X In Doc.Items
                    MessageBox x.Name + ": "+ x.text
            End ForAll

      DNUG 50. Anniversary Conference - a long community journey

      Daniel Nashed  10 June 2023 09:53:24
      DNUG 50. Anniversary Conference

      preparations for #DACHNUG conference next week...
      This is our 50. anniversary conference. I still recall my first presentations long long time ago.

      It was my first public speaking, before I attended my first Orlando Lotusphere the year after.

      I have been a speak at every conference since then and I am so proud for DNUG to be a platform for us for such a long time.

      A community like this lives from the participating members. As for most organizations like this, there are a few people running the show (it's actually a normal distribution like many other things in life).

      It's all about the community and the key contributors

      This year shows again, that we have still a great crowd of people around in the community.

      I recall the time in the business partner forum. A world wide replicated database for business partners around the world Lotus at that time, to share ideas and collaborate.

      DNUG has a couple of focus groups. The two most active groups are the Domino focus group and the communications focus group.

      In my role as member of the Domino focus group I am running the DNUG Lab.

      But I want to specially highlight and say thank you to the DNUG communication focus group to implement a ST infrastructure just for the conference (we have one permanent implementation, too).

      DACHNUG Conference Chat

      Everyone registered got a mail with log in credentials including QR codes to download the software from the app stores and the login settings.

      I just had to add my mail-address and my registration number. And it just works.

      Special thank you to Erik Schwalb from HCL for his continuos engagement beyond the communications focus group and to make this happen!

      But also a special thanks for all the active focus group members and the DNUG board for their engagement over the last years! You are part of the heart and soul of our community!

      It has been a long and interesting journey for many of us so far

      Looking into the first messages in "Coffee Bar" chat in this ST community makes me realize again how many of us are around since the very early days in this community.

      I see screen prints from books for Lotus Notes 3.3 in the chat..

      There are many of us being around since Notes version 3.x or 4.x.

      I have started with Notes 3.3 and started in production with Notes on HP-UX shortly after.

      This is by the way when the first version of today's Domino Linux and AIX start script was born.

      What makes Notes and Domino so special that people spend their whole career working with this product?

      A while ago I tried to explain what Notes really is and what makes it special -->
      The key components of Notes are still unmatched by other software today.

      And the cross platform support is still very impressive.

      Count down to DACHNUG conference in Siegburg next week

      Daniel Nashed  7 June 2023 08:58:25

      Time flies so fast.. DACHNUG conference is already next week.
      I got questions how to book one of the workshops, when you already have a ticket for the conference.

      The FAQs for the conference says you just need to update your existing ticket on Eventbrite like this:

      If you want to join the Domino Kubernetes workshop, we have some spots left.

      I did send a mail to all participants about what they already know about Linux, Docker and K8s.
      Also about what they expect from the workshop. I am already looking forward to the discussion part of the workshop.

      I am almost done with the preparations. Still thinking about which platform to deploy (Ubuntu vs. CentOS and RKE2 vs. K8s).
      Probably it's going to be CentOS Stream 9, because Ubuntu is quite different from Linux admin point of view.

      If someone wants to attend the conference last minute. For some reasons the hotels are all booked in Siegburg and also Cologne and Bonn.
      We don't have an idea why. But hopefully there are some last minute rooms.

      Looking forward to meet many of you...

      -- Daniel

      Image:Count down to DACHNUG conference in Siegburg next week

      Domiono V14 backup for notes.ini

      Daniel Nashed  31 May 2023 17:12:57

      Domino backup is around since 12.0 and it got improvements in every release.
      There are not many current AHA ideas for Domino Backup & Restore.

      One smaller feature you can see in EAP1 is the backup of the notes.ini.

      The notes.ini gets the same type of date stamp you know from the log.
      And it is attached to the backup log document and/or written to the backup configuration directory.

      The backup log document is cleaned up with the purging of a backup along with the file on disk with the same name if configured.

      This closes a small gap in for backing up relevant information for a disaster recovery.
      Another important part would be the, which should never be backed up to the same location for security reasons.
      My personal wish would be to have ID Vault support for server.ids.

      With this small change you have a notes.ini backup always available -- without any restore operation.

      Usually the backup log directory is backed up as well as the dominobackup.nsf.

      But in a disaster case for a restore a database location might not be the right way.
      Domino backup exports the configuration in DXL and can even read the configuration from the DXL instead of the NSF for a restore.
      And there is a way to rebuild the backup inventory for a backup from the log documents.

      -- Daniel

      Image:Domiono V14 backup for notes.ini

      Notes/Domino/Traveler EA1 available

      Daniel Nashed  31 May 2023 17:10:27
      Image:Notes/Domino/Traveler EA1 availableImage:Notes/Domino/Traveler EA1 available

      Notes/Domino/Traveler EA1 has been released today.

      If you want to know what you will get in EA1, there is a web cast tomorrow

      Webinar: Grab a Sneak Peek of Our Upcoming HCL Domino v14 Release -- June 1, 2024, 10 AM EDT

      There is webinar showing the highlights of the first code drop.

      Link to register -->

      New software download experience for 14.0.0 EA1 software first

      Before you start download from Flexnet, HCL is starting an official preview for the new download experience.

      Right now only with the new software which is part of the early access program.

      I have used it already to download software to my lab, to my build environments and the DNUG Labg.

      Domino V14 Early Access forum

      For questions, feedback and additional information, you should really joint the EAP forum  

      HCL Domino Container Community Image

      The build script has been already updated and the container images are tested.

      You can build a Domino container including a Nomad container if you like. Or just the Domino container image:

      ./ domino 14.0.0EA1 -nomad=1.0.8-12.0.2

      And on top you can add Traveler 14.0.0 EA1

      ./ traveler 14.0.0EA1 -from=hclcom/domino:14.0.0EA1

      You can expect separate blog posts over time about the new functionality.
      I have tested out most of it already in the DNUG Lab and I am in preparation for the conference and workshop.

      Of course the lab and the presentation will focus on Domino V14.0.0 EA1.

      -- Daniel

      K3s Traefik Ingress Controller including Dashboard

      Daniel Nashed  30 May 2023 08:06:42

      Image:K3s Traefik Ingress Controller including Dashboard

      The standard ingress controller for K8s is usually NGINX.

      K3s uses Traefik as the out of the box ingress controller (because it needs to be available on more platforms etc.), which needs a bit different configuration compared to NGINX.

      For my DNUG workshop I am looking into K3s explicitly, because it is a much smaller, easier to handle distribution for a lab environment.

      RKE2 comes with NGINX. But K3s uses Traefik.

      Traefik has a bit different concept and a very nice dashboard, which is disabled by default.

      The dashboard itself is a good way to show how to configure a service and an ingress controller using Traefik.
      I am using my own TLS secret, which I imported manually (see one of last weeks posts).

      There isn't much documentation about how to define an Ingress with Traefik.

      That's why I am adding this configuration below as an example.

      -- Daniel


      apiVersion: v1

      kind: Service


      name: traefik-dashboard

      namespace: kube-system

      labels: traefik traefik-dashboard


      type: ClusterIP


      - name: traefik

      port: 9000

      targetPort: traefik

      protocol: TCP

      selector: traefik-kube-system traefik



      kind: Ingress


      name: traefik-dashboard

      namespace: kube-system

      annotations: websecure "true"


      ingressClassName: traefik


      - secretName: tls-secret


      - host:



            - path: /

              pathType: Prefix



                  name: traefik-dashboard


                    number: 9000


      ChatGPT presents derived answers as facts

      Daniel Nashed  29 May 2023 12:52:29

      Depending on what you ask ChatGPT the answers are quite good.
      But ChatGPT also derives information and is not always right.
      For example for Lotus Script it puts together information from multiple different areas and also makes up new properties, it thinks it knows about.
      Sometimes it is also mixed up with different classes.

      I have tested it with various type of questions with real world more complicated questions for Lotus Script but also for LibCurl code and others.

      It even makes up notes.ini variables which are not existing.
      I got questions from another partner about certain parameters, which turned out to not exist. And the source of the info turned out to be ChatGPT.

      It's not wrong to use ChatGPT to get an idea and it can be very helpful pointing your to solutions.
      But please make sure you validate what you get back and do not take it 1:1 as ultimate truth.
      Specially make sure you don't talk to me, other consultants or HCL support about information from ChatGPT in a way that implies that those parameters exist and they are validated.

      One best practice is to take parameters you get and search them on your own to find them in another context.

      -- Daniel



        • [HCL Domino]
        • [Domino on Linux]
        • [Nash!Com]
        • [Daniel Nashed]