Introducing DomProbe – The Missing Piece for Prometheus NRPC Monitoring

The Prometheus ecosystem already provides excellent monitoring components.
The standard Blackbox Exporter is perfect for probing HTTP, HTTPS, TCP, ICMP and many other network protocols, while exporters such as node_exporter and DomProm provide detailed operating system and Domino server metrics.
But there has always been one missing piece: Native NRPC monitoring for HCL Domino.
That's exactly why I created DomProbe.

Why NRPC Monitoring Matters

Many critical Domino functions depend on NRPC: Notes client connectivity, replication, clustering, administration, mail routing, and server-to-server communication.
A Domino server may still answer HTTPS requests while NRPC is no longer functioning correctly. Likewise, a simple TCP connection to port 1352 only tells you that something is listening—it does not confirm that Domino is actually able to process NRPC requests.
DomProbe performs a real NRPC transaction using the native Domino C API. It connects to the target server, measures request and response latency, reports the Domino server state, and can optionally verify that a specific database can be opened.
This provides a much more meaningful availability check than simply testing whether port 1352 is open.


A Natural Extension to the Prometheus Ecosystem

DomProbe follows the same architecture as the Prometheus Blackbox Exporter.
Prometheus sends an HTTP request to DomProbe, which performs the NRPC probe and returns the results in standard Prometheus metrics format.
Existing Prometheus and Grafana dashboards, alert rules, and scrape configurations work exactly as expected—the only difference is that the probe itself understands Domino.
Typical metrics include:

• NRPC availability
• Domino server state (Available, Restricted, Busy, Not Reachable)
• Client-to-server latency
• Server-to-client latency
• Total probe duration
• Optional database open verification and timing

Combined with DomProm and node_exporter, this finally completes the monitoring stack for Domino environments.

Monitor Your Entire Domino Infrastructure

Although DomProbe runs as a native Domino server task on Linux, it is not limited to probing the local server.
A single DomProbe instance can probe any Domino server reachable via NRPC.
For example, you might deploy one DomProbe instance in each data center:

• A Domino server in Data Center A probes all production servers in its own site as well as selected servers in Data Center B.
• Another Domino server in Data Center B performs the same checks in the opposite direction.

This verifies much more than whether the target servers are running. It also confirms that NRPC communication between Domino servers is actually working.
Routing problems, firewall issues, VPN failures, or replication connectivity problems become immediately visible from the perspective of another Domino server.
Since Prometheus simply calls the /probe endpoint with different target parameters, a single DomProbe instance can monitor dozens or even hundreds of Domino servers without requiring additional software on every monitored system.

Lightweight by Design

DomProbe intentionally focuses on availability monitoring.
It does not try to replace detailed monitoring solutions such as DomProm or node_exporter.

Instead, the components complement each other:

• Prometheus Blackbox Exporter — HTTP, HTTPS, TCP, ICMP, DNS and other standard protocols
• DomProbe — Native Domino NRPC availability and optional database access
• DomProm — Domino server statistics, replication, mail routing, performance and health metrics
• node_exporter — Operating system metrics

Together they provide a complete monitoring solution for HCL Domino.

Native Domino Integration

DomProbe runs as a native Domino server task on Linux and integrates directly with the Domino runtime using the Domino C API.
The task exposes familiar Prometheus endpoints such as /probe, /metrics, and /health, making integration with existing Prometheus environments straightforward.


Open Source


DomProbe closes one of the last remaining gaps in the Prometheus monitoring ecosystem for HCL Domino by bringing native NRPC probing to Prometheus.
If you're already using Prometheus and Grafana to monitor your Domino infrastructure, DomProbe adds the missing visibility into the protocol that powers Domino itself.

https://github.com/nashcom/domino-blackbox-exporter

• Comments [0]

Meet nshini – A Notes.ini Editing and Conversion Tool

If you've administered HCL Domino long enough, you've probably edited notes.ini more times than you can count. It looks like a simple text file—but it isn't.

notes.ini is stored in LMBCS, a Lotus-specific character encoding.
Modern editors expect UTF-8. While ASCII-only settings are usually safe, any non-ASCII value—passwords, international characters, or localized paths—can be silently corrupted simply by opening and saving the file in a standard editor.

That's exactly why I created nshini after working with another HCL partner on one of his support tickets.
In his case support asked him to make changes to the notes.ini which broke umlauts.

nshini transparently converts notes.ini between LMBCS and UTF-8 using the official Domino C API. You edit a normal UTF-8 file in your preferred editor, and the tool safely converts everything back when you're done.

This makes working with notes.ini much more comfortable on both clients and servers.
When changing settings on a server you can always use "set config".

On Windows, nshini integrates seamlessly with Notepad++, allowing you to use one of the best text editors available, complete with syntax highlighting, search, and visual diff support.
On Linux, it works equally well with your favorite editor while providing clean UTF-8 output for scripts and command-line tools.

Beyond editing, converting notes.ini to UTF-8 opens the door to modern tooling. Configuration can be compared with standard diff tools, processed by scripts, indexed, or even analyzed by AI workflows without worrying about legacy character encoding.
nshini also provides commands to read and update notes.ini variables through the Domino API, making it useful for automation as well as interactive administration.

The project is open source and available on GitHub:

GitHub: https://github.com/nashcom/nshini

• Comments [0]

Domino NRPC Proxy – A Modern Proxy Platform for Domino

The Domino NRPC Proxy project started with a simple but ambitious goal: provide a modern, production-ready proxy for Domino NRPC connections that integrates naturally with Docker and Kubernetes.

While traditional TCP proxies can forward NRPC traffic, they have no understanding of the Domino protocol itself.
At the heart of the project is a custom NGINX Stream module that understands the initial Domino NRPC handshake and enables intelligent routing based on the requested Domino server.
Since then, the project has grown well beyond its original scope.
Today it provides a complete proxy platform for Domino, combining NRPC routing, HTTPS reverse proxy support, generic stream proxying.


Key Features

• Custom NGINX Stream module for Domino NRPC
• Intelligent NRPC routing based on the requested Domino server
• HTTPS reverse proxy support
• Generic TCP stream proxy support
• Optional ACME certificate management using LEGO
• Dynamic configuration generation from templates
• Zero-downtime configuration updates
• Automatic certificate reload
• Native Docker and Kubernetes support
• Docker Secrets and Kubernetes Secrets integration
• Environment variable driven configuration
• Own Prometheus metrics endpoint including details about configuration status

The container build compiles NGINX and the stream module. The Image can be build with the included build script. The image is also available on the GitHub registry as part of the GitHub project.


Intelligent NRPC Routing


The Domino NRPC Proxy is much more than a TCP forwarder.
The project includes a custom NGINX Stream module developed specifically for Domino NRPC. The module understands the initial NRPC protocol exchange and extracts the Domino server name requested by the Notes client.
This information is then used to make an intelligent routing decision before forwarding the connection to the appropriate back-end Domino server.
The entire process is completely transparent to the client and requires no changes to existing Notes or Domino installations.

This makes it possible to:

• Publish a single NRPC endpoint for multiple Domino servers.
• Route Notes clients automatically to the requested server.
• Hide backend Domino servers inside private Docker or Kubernetes networks.
• Dynamically route connections as infrastructure changes.
• Simplify load balancer configurations.
• Scale from a single Domino server to large Domino deployments.

This capability is particularly valuable in Kubernetes environments, where Domino servers are represented by services and pods whose network topology may change over time.


HTTPS Reverse Proxy


Although NRPC remains the primary focus of the project, most Domino environments also expose HTTP and HTTPS services.
The Domino NRPC Proxy now includes a fully integrated HTTPS reverse proxy based on NGINX.

Features include:

• HTTPS reverse proxy
• HTTP to HTTPS redirection
• Multiple virtual hosts
• Configurable upstream servers
• Dynamic configuration generation

Running both NRPC and HTTPS within the same proxy simplifies deployments while providing a consistent operational model.


Generic Stream Proxy Support


The custom NRPC module is one part of the project.
The surrounding infrastructure has been designed to support generic TCP stream proxying as well.
This allows the same container to proxy additional TCP services using the same configuration framework, making the project useful beyond Domino-specific scenarios.


Designed for Docker and Kubernetes


The project was designed from day one with containers in mind.
Whether you're deploying a single Domino server with Docker Compose or operating a large Kubernetes cluster, the same container image can be used without modification.
Configuration is generated automatically during startup, allowing deployments to be driven entirely by environment variables and secrets.
The result is a container that fits naturally into Infrastructure-as-Code workflows and automated deployments.


Dynamic Configuration Templating


Maintaining multiple static NGINX configuration files quickly becomes difficult as deployments grow.
Instead, the Domino NRPC Proxy gene7rates its configuration dynamically from templates.

The templating system supports:

• HTTP configuration
• Stream configuration
• Environment variable substitution
• Optional feature enablement
• User supplied templates
• Deployment specific customization

This approach keeps the container image generic while allowing extensive customization without rebuilding images.


Zero-Downtime Configuration Updates


Container environments are dynamic by nature.
Services may be added, removed or updated while the proxy continues running.
Whenever configuration changes are detected, the proxy automatically:

• Generates the new configuration
• Validates it using nginx -t
• Atomically replaces the active configuration
• Reloads NGINX without interrupting existing client sessions

Existing NRPC and HTTPS connections continue uninterrupted while new connections immediately use the updated configuration.


Automatic Certificate Management


The latest versions add optional integrated ACME support using the excellent LEGO client.
The proxy can automatically request and renew certificates from Let's Encrypt or any RFC 8555 compatible ACME provider.

Supported features include:

• Let's Encrypt Production and Staging
• Alternative ACME providers
• HTTP-01 challenge support
• Automatic renewals
• Environment variable based configuration
• Automatic NGINX reload after successful renewal
• Support LEGO environment configuration

The implementation also handles the initial bootstrap process, allowing completely unattended deployments.


Automatic Certificate Reload


Certificates may also be provided by external certificate management solutions.
The Domino NRPC Proxy automatically detects updated certificates and reloads NGINX without requiring container restarts.
This enables continuous certificate management without downtime or maintenance windows.


Secret Integration


Security is an important design goal.
Sensitive configuration data should never be embedded in container images.

The Domino NRPC Proxy provides native support for:

• Docker Secrets
• Kubernetes Secrets

Certificates, private keys, bearer tokens, API credentials and other sensitive configuration can be securely mounted while remaining fully compatible with the templating framework.


Built Around Automation


One of the guiding principles throughout the project has been automation.
A modern infrastructure component should require as little manual configuration as possible.
The Domino NRPC Proxy therefore focuses on:

• Convention over configuration
• Environment variable driven deployments
• Dynamic configuration generation
• Automated certificate management
• Automatic configuration validation
• Zero-downtime updates

The result is a proxy that is equally at home in a simple Docker installation or a fully automated Kubernetes platform.


Part of the Domino Infrastructure Ecosystem


The Domino NRPC Proxy has become one of the core building blocks of a larger Domino infrastructure ecosystem.
It integrates naturally with projects such as:

• Domino Fleet Manager
• Cube Control
• CertMgr
• Domino monitoring components

Each project is designed to work independently, while together they provide a modern platform for deploying, managing and operating HCL Domino environments.


Open Source


The Domino NRPC Proxy is available as an open source project on GitHub:
https://github.com/nashcom/domino-nrpc-proxy
The project has evolved considerably since its first release and continues to grow with new features and deployment options.
If you're running Domino in Docker, Kubernetes, or simply looking for a modern proxy platform for Domino services, I hope you'll find it useful.

• Comments [0]

Behind the Scenes: Linux Install Fest Lab at Engage 2026

Bill, Matijn and I did three hands on sessions at Engage.
If you wondered how we have been doing it. Here is how this worked and it could be leveraged for own workshops.
Using cloud hosted servers can make a lot of sense. And it is very cost efficient.

Hetzner is a provider with a very modern and easy to use GUI. But they also have a great API to create and manage servers.
I am using them for years for DNUG and other events. And I would make it available for friends & family.

Recently Hetzner moved their separate DNS API into the central cloud API -->https://docs.hetzner.cloud/reference/cloud
I took this opportunity to implement most of this API in an existing Lotus Script class (Script Lib) and read the full inventory for a project to manage it from a Notes database.
Each database would match 1:1 a Hetzner project. We created a private network for the project and offered services like NFS based file mounts to provide workshop data.

The database is mainly designed for lab setups with the same server configuration used for all servers.
A profile document is used to define the type, size, location etc of the server.
What I also added is support for cloud.init configurations.

For our servers I just needed a host name. The e-mail address is used for an integrated mail workflow to notify users -- including sending them their server name and IP.
When creating a new server also a DNS record is created and also the IN-ARPA entry is set properly.

Creating a new server just takes a couple of seconds. In our case you have to wait a moment because we installed additional software, configured a NFS mount and rebooted the machine.

Because we did not expect everyone to register ahead of time thru the official Engage registration processing a mail flow from the Leap App to our database, I added a registration form to feed in new server requests directly.
The slides had a registration QR code to fill in the registration code.

This gave us a very flexible way to generate new servers on-the-fly.

The whole lab setup was done in a reusable way. So all the time spent on it wasn't just for this one conference.
And I will extend it over time for other use cases. The Lotus Script Lib is the core of the application, but with my additions also reading the inventory could be re-used in other areas.

This is a good example use case for a Notes database. It would probably have taking me much longer with most other approaches.

See some details below. I also added label support and to specify firewalls directly or via labels.
The could.init replaced my earlier Ansible scripting I used to customize servers after they are created.
But I also added a Ansible host list export and we used the same SSH key for all servers beside the management server which hosted the file shares.

-- Daniel

         

• Comments [6]

Domino On Proxmox LXC Containers Part II

As mentioned at Engage, I am working on Domino container deployments on Proxmox.
The first deliverable was a new build option for the container image to create two components:

• A LXC template which can be cloned into a new LXC container using pct command line
• A /opt volume which is read-only mounted into each container

You can see from the configuration dump below how this would look like.
The second deliverable is a management script "dompct" which allows to create and manage LXC containers.
I added some screen shots below.

The new script is already part of the develop branch of the Domino Start script, but I am still enhancing and testing it.
It comes with configuration profiles which can be managed and assigned to servers when created.

This includes ZFS volumes assigned to LXC containers -- as you can also see from the example below.
The new option build.sh -pct and the new dompct tool standardize and simplify Proxmox LXC deployments.

There isn't full documentation yet in the GitHub project. But dompct has command line help already.
You can either use command-line parameters or the menu.
The menu prompts to select a container if not specified. Or a profile if not specified.
It also prompts for the hostname when creating a new container and the hostname is not specified at command-line.

The tool can be used interactively or for automated provisioning. That's why I also added -json for output.

The Proxmox LXC is a first implementation and I mainly wrote it for one customer who is actively looking into Domino on Proxmox deployments.
But I wanted to make it available already for feedback.


--- Configuration output ---

--------------------------------------------------------------------------------
Config LXC 800
--------------------------------------------------------------------------------

arch: amd64
cores: 4
features: nesting=1
hostname: domino-01
memory: 8192
mp0: /rpool/data/domino-opt-20260419-1053,mp=/opt,ro=1
mp1: /rpool/data/subvol-800-domino-local,mp=/local
mp2: /rpool/data/subvol-800-domino-nsf,mp=/local/notesdata
mp3: /rpool/data/subvol-800-domino-translog,mp=/local/translog
mp4: /rpool/data/subvol-800-domino-daos,mp=/local/daos
mp5: /rpool/data/subvol-800-domino-backup,mp=/local/backup
net0: name=eth0,bridge=vmbr0,hwaddr¼:24:11:75:9A:30,ip=dhcp,type=veth
ostype: ubuntu
rootfs: local-zfs:subvol-800-disk-0,size=20G
swap: 0
tags: domino
unprivileged: 1

description:
HCL Domino server 800









--- Configuration profile example ---


# Domino LXC configuration - default

PCT_TAGS=domino
PCT_DATA_POOL=rpool/data
PCT_DAOS_POOL=rpool/data
PCT_TRANSLOG_POOL=rpool/data

PCT_TRANSLOG_SIZE_GB=5
PCT_NSF_SIZE_GB=100
PCT_BACKUP_SIZE_GB=100
PCT_DAOS_SIZE_GB=100

PCT_RAM_GB=8
PCT_SWAP_GB=0
PCT_CPU=4

env_DOMSETUP_ENABLED=1
env_NODE_EXPORTER_OPTIONS=default


PCT_NET0_TEMPLATE=name=eth0,bridge=vmbr0,ip=%PCT_IP%/24,gw=192.168.96.98



--- Command Line Parameters ---


dompct - LXC Container Control Utility

Usage:

./dompct.sh <command> [VMID] [options]

Commands:

create              Create new container
start               Start container
stop                Stop container
status              Show container status
enter | bash        Enter container shell
config              Show container configuration
update              Update container
destroy             Destroy container
about               Show container information
profile             Select or apply profile
list                List containers
KILL                Force kill container
KILL-WITH-DISKS     Kill container and remove disks

Global Options:

-profile=<name>     Use profile (from ~/.dompct/*.cfg)
-host=<name>        Set hostname
-hostname=<name>    Same as -host
-tags=<tags>        Set Proxmox tags (comma-separated)
-ip=<ip>            Assign IP address
-description=<txt>  Set container description
-opt-vol=<opts>     Volume options (advanced)

Output Options:

-json               Output in JSON format (where supported)

Arguments:

VMID                Numeric container ID (required for most commands)

Examples:

./dompct.sh list
./dompct.sh start 800
./dompct.sh create -profile=mail
./dompct.sh profile
./dompct.sh destroy 800

Notes:

- If VMID is omitted, interactive menu may be used
- Profiles are stored in: ~/.dompct/

• Comments [238]

Ubuntu 26.04 LTS released with a last minute surprise -- Kernel 7.0

Today finally the release shipped and they just updated the website and also provided all ISO images.
I have been using the release candidate for a while natively and for Docker containers.

• Desktop
• Server
• WSL
• Container Image default for ubuntu is also version 26.04.

--- Kernel ---


The RC kernel was: 6.19.0-9-generic
The released kernel : 7.0.0-14-generic

--- GLIBC ---

GLIBC is also very up to date: (Ubuntu GLIBC 2.43-2ubuntu2) 2.43


The main challenge here is a major new kernel release.
The change isn't as big as it sounds on top of the latest 6.19.
Ubuntu might have waiting for the kernel to be released and updated it because of newer CPU support in kernel 7.0 which they wanted to be prepared for.

But this raises questions for Domino on Ubuntu 26.04 LTS.
I would not update any production environments yet.
This is way too new. But it works well in my personal testing.

If you give it a try in a test environment I would be interested in your findings.
But given the major kernel update, I would really wait for any production deployment.


Official release statement for 14.5.1

https://support.hcl-software.com/csm?id=kb_article&sysparm_article=KB0128491

Equivalent OS with the following kernel/packages:

kernel-6.12.0-55.9.1.el10_0.x86_64 or higher 6.12 kernel
glibc-2.39-37.el10.x86_64 or higher
libstdc++-14.2.1-7.el10.x86_64 or higher

You can see that from Kernel level and also GLIBC level Ubuntu 26.04 is quite more current then what is listed as officially tested and supported for Domino.

Note about Ubuntu system requirements

There has been a change in system requirements for the deskop. The hardware resource requirements have been bumped up a bit.
On the server side the hardware resource requirements did not change. 1,5 GB is still a pretty low requirement.
With Domino I would at least have 4 GB for a small server anyhow. But the 1,5 GB show the low resource requirements a Linux server has -- on top of the application requirements.

• Comments [6]

Engage 2026 Presentation Slides Leveraging CertMgr and Resources

As promised at Engage this week, here is my presentation including additional material as promised in the session.
The Script Lib is a first version which I wrote for one of my projects to integrate certificate management.
I am happy to share it and I am looking forward to feedback.

What I also added is the code I used in my demo to integrate with HashiCorp.
For the presentation I also setup a new HashiCorp development project which might help to get started with HashiCorp.

The HashiCorp instance I was using in my demo has been automatically setup using this project.
This includes the provisioners for ACME and API level integration.

I hope the resources help you to get started with your own integrations.

HashiCorp is an interesting option for certifcate management inside a company.
This includes ACME support and also the secrets engine to securely distribute secrets like TLS private keys.

-- Daniel


Engage Presentation slides

https://github.com/HCL-TECH-SOFTWARE/domino-cert-manager/blob/main/presentations/engage_2026_certmgr.pdf

Example Integration: CertMgr HashiCorp

https://opensource.hcltechsw.com/domino-cert-manager/hashicorp/

Script Lib Source code

https://github.com/HCL-TECH-SOFTWARE/domino-cert-manager/tree/main/resources

HashiCorp Deployment Project to get started with HashiCorp

https://github.com/nashcom/nsh-vault-deploy

• Comments [6]

Performance Challenges on modern Intel Hybrid CPUs with Notes and other applications

After moving to a new notebook, my Notes client got very slow when I was in a Sametime meeting or ran other a bit more CPU intensive tasks.
It turns out that Windows is giving the Notes client the slower E-Core CPU cores, which results in a very slow UI responsiveness.

HCL is aware of this and we had a call last week.
If you are running a modern Intel CPU on your workstation/notebook you should read this link for detailed information and a tool

https://github.com/nashcom/nshcpuset

I wrote up all the technical details and a small troubleshooting and work-around tool.

My take away from this is that we have to watch more for the CPU P-Cores then the total number of cores a modern CPU has.

The notebook I got has 2 physical P-Cores with Hyper-Threading + 8 E-Cores.
This doesn't hit Notes alone! We had interesting experiences with VMware workstation before they looked into it.

Darren blogged about the VMware issues he ran into  https://blog.darrenduke.net/darren/ddbz.nsf/dx/type-2-hypervisors-and-the-evils-of-e-cores.htm

I am not a fan of Processor Lasso and looked into the APIs on my own to get a clear picture.
Processor Lasso is a too big tool with a lot of options.
You can also use it, but the easier path is to use my small program.

-- Daniel

• Comments [6]

Domino on Proxmox deployment models

With the current cost increases for VMware licenses and also hardware some of my customers start to look very seriously for alternate solutions.
One deployment method would be Proxmox with local ZFS disks.

ZFS is a very interesting file-system and volume manager combined. It offers a lot of flexibility and choices.
Compression, de-duplication, optimized record size, snapshots, encryption and more benefits.

I am a big fan for a while and I did blog about Domino on Proxmox before.
Last weekend I looked into different deployment methods for Domino on Proxmox for a customer.

A VM with Linux + Docker + Domino might be additional overhead which could be optimized.

LXC containers use native ZFS volumes and are very efficient:

• Lightweight Linux instance with shared kernel like Docker
• Leverage ZFS host subvolumes for a true end to end storage management and less overhead
• Standardized OS images

What is missing is the automated installation we know from Docker.
I am looking into that right now and if you are interested in Domino on Proxmox, I want to hear from you.

At Engage I will show case Domino on Proxmox. I will bring a Proxmox server running on my notebook with an automated Domino installation including management scripts with me.
I would like to hear from you what type of deployments you are looking into.

We are working on multiple deployment options optimized for Proxmox in combination with Domino clustering to optimize the deployment footprint.
This includes DAOS storage deduplication cross servers and backup.

-- Daniel

My Notebook deployment on VMware workstation with Proxmox 4 CPU cores + 8 RAM as a test/demo installation for on the road.
For LXCs it does not even need the hardware support.

• Comments [423]

Join Us at Engage 2026: Domino on Linux, Installfest and CertMgr

Engage Conference 2026 is just around the corner. A quick look at the agenda reveals several Linux-focused sessions—and that’s no coincidence. Linux continues to play a key role in sovereign and future-proof solutions. As in previous years, Bill is hosting the popular Domino on Linux round table. In addition, the conference will feature a Linux desktop session along with a series of Domino on Linux Installfest sessions. These sessions can be attended independently, but they are also designed to complement each other and build progressively. Bill, Martijn and I have teamed up to present a full range of sessions—from beginner through to expert level. This includes hands-on labs using on-demand virtual machines at Hetzner, along with forward and reverse DNS within our domino-lab.net environment. Even if you can’t attend the Installfest sessions, the Linux round table is highly recommended. It’s a great opportunity to ask questions, share feedback, and connect with others—especially if you’re already running Domino on Linux or planning to explore it. For example on Proxmox. We’ll also share updates on what has been developed over the past year, along with a new initiative aimed at making Domino on Linux more accessible for administrators. As part of this effort, a new repository has been launched as a central entry point into the Domino on Linux ecosystem: https://nashcom.github.io/nsh-domino-linux/ Beyond the Linux sessions, I will present another session on Domino CertMgr. If certificate management and automation are topics of interest, this session should definitely be on your list. You can explore the full agenda here: https://engage.ug/pages/session2026 We are looking forward to seeing you at Engage 2026. Bill | Martijn | Daniel

• Comments [6]