Economically Right-Sizing Domino Server Memory on Linux

More RAM is generally good for a Domino server. But more RAM is not automatically a good investment.
This matters more today because RAM has become significantly more expensive again, particularly when building servers with large memory configurations.

A useful concept from economics applies surprisingly well to server sizing: diminishing marginal utility.
The first additional GB of RAM can provide significant performance benefits. The next GB still helps, but usually a little less. Eventually, adding another 32 GB might provide only a small additional benefit.

The interesting question therefore isn't:

How much RAM can Domino use?

It is:

How much RAM provides a meaningful benefit for a workload?

On Linux, unused RAM normally doesn't remain unused. Linux uses available memory as filesystem cache.
A 64 GB Domino server might therefore look approximately like this:


64 GB RAM

8–10 GB   Domino + other processes
~50 GB     filesystem cache
remainder kernel / other usage

It is easy to look at this and conclude that the machine benefits from all 64 GB.
But there is an important distinction:

Memory being used is not the same as memory being required.

Linux will happily use additional RAM to cache filesystem data whenever memory is available.

Diminishing Marginal Utility

The first GBs of filesystem cache are extremely valuable because they contain the most frequently accessed data.
As the cache grows, increasingly less frequently accessed data is retained. Additional RAM still provides a benefit, but the incremental benefit gets smaller.

Conceptually:

Filesystem cache        Marginal benefit

8 GB                   Very high
16 GB                   High
24 GB                   Significant
32 GB                   Useful
48 GB                   Lower
64 GB                   Lower still

The numbers are only illustrative. The actual curve depends on the workload.The important principle is:

Doubling the filesystem cache does not double its performance benefit.

In my tests on one busy Domino mail server, for example, Linux was using around 55 GB of filesystem cache.
Reducing the available cache substantially did not result in a corresponding dramatic increase in physical disk I/O.
Even around 24 GB of cache, the server continued to behave very well.

That suggests that much of the additional cache was useful because it was available—not because the workload actually required it.

Different Domino Workloads Are Different

A mail server, Traveler server, directory server and SMTP server do not have the same memory requirements.
A busy mail server accessing many NSF databases can benefit substantially from filesystem cache.
Traveler is quite different. Its important memory consumers include the JVM, Domino HTTP and native process memory, while an HA configuration uses an external database for Traveler data.

Providing huge amounts of RAM to a Traveler VM simply to create a large filesystem cache may therefore provide little additional benefit.

RAM Isn't Cheap Anymore

For years, a common argument was: RAM is cheap. Just add more. That argument isn't nearly as convincing anymore.
Large server-memory configurations have become expensive again. And in virtualized environments, overprovisioning accumulates quickly.

Consider 20 Domino VMs with 32 GB more RAM each than their workloads meaningfully benefit from: 20 × 32 GB = 640 GB

That's 640 GB of physical infrastructure memory providing progressively smaller benefits.
The same memory could instead provide capacity for additional workloads, HA headroom, or simply reduce the cost of the infrastructure.

Right-Size Instead of Max-Size

The goal should not be to provide every Domino server with the largest possible filesystem cache.
The goal is to provide enough application memory, enough filesystem cache to capture the valuable part of the working set, and reasonable operational headroom.

Beyond that point, additional RAM still helps—but its marginal utility decreases.

That's the economic part of right-sizing:

Don't size memory simply because Linux knows how to use it. Size it according to the value that additional memory actually provides.

• Comments [0]

VictoriaLogs for Domino

Grafana Loki is my current favorit and fits well to the the remaining Grafana stack we have for Domino metrics.

When looking into OpenTelemetry formats VictoriaLogs hit my radar.
It's pretty simple to setup in a Docker container and works pretty well even on a local setup.

https://docs.victoriametrics.com/victorialogs/

The entry point for OpenTelemetry is different than for Loki and the format is "protobuf":

http://localhost:9428/insert/opentelemetry/v1/logs



Setting up a container via Docker Compose


services:

  victorialogs:
    image: victoriametrics/victoria-logs:latest
    container_name: victorialogs
    hostname: victorialogs
    restart: always

    ports:
      - "9428:9428"

    command:
      - "-storageDataPath=/victoria-logs-data"
      - "-retentionPeriod=30d"

    volumes:
      - victoria-logs-data:/victoria-logs-data

volumes:
  victoria-logs-data:


Here is how it looks like

• Comments [0]

Domino Log Forwarder goes OpenTelemetry

I have been working quite a bit on my Domino Log Forwarder project over the last week.
What started as a small utility to forward Domino console output to Grafana Loki has evolved into a more generic OpenTelemetry log forwarder.

One of the most important changes is that there is no Loki-specific output anymore.

otelfwd now sends OpenTelemetry logs using OTLP/HTTP JSON. Grafana Loki is one of the receivers I use and test with, but the forwarder itself doesn't depend on Loki.

One forwarder, multiple input methods

There are several ways to feed data into otelfwd.

The Domino console is the simplest one. Domino STDOUT is piped into otelfwd, just like with the original version of the project.

The new domfwd Domino server add-in goes a step further. It reads structured events directly from the Domino Event Monitoring queue and sends them to otelfwd through a local UNIX socket.

Other applications can use the same structured interface through a UNIX socket or a TCP connection bound to 127.0.0.1.


There is a UNIX datagram syslog interface. I use this, for example, to send NGINX access and error logs directly into otelfwd.

All those inputs use the same forwarding infrastructure from that point on.

Reliable OTLP delivery

otelfwd converts and groups the incoming records into OpenTelemetry resources and scopes and sends them using OTLP/HTTP JSON.
Before sending, records are batched for efficient transmission.
More importantly, otelfwd has its own write-ahead log (WAL). If the configured OTLP receiver isn't reachable, records are kept locally and automatically replayed when the receiver becomes available again.

The backend is no longer Loki only

This is probably the biggest architectural change compared with the original Domino Log Forwarder.
There is no longer a Loki Push API implementation in the forwarder -> The output is OpenTelemetry only.

For my own environment I currently use Grafana Loki, which provides an OTLP endpoint. But from the perspective of otelfwd, Loki is simply an OTLP receiver.

This also makes it much easier to use the same forwarder with other observability platforms. While testing other backends, VictoriaLogs turned out to be an interesting case.
VictoriaLogs also supports OpenTelemetry logs, but its native OpenTelemetry JSON ingestion format is different from the standard OTLP/HTTP JSON representation used by otelfwd.

Native Domino Events forwarder

The project now contains also a Domino Events Forwarder. It's a native integration into the Domino Event Monitoring Stack and allows you to push enhanced events to any OpenTelemetry backend.

See the updated project on GitHub: https://github.com/nashcom/domino-log-forwarder

• Comments [0]

Veeam Backup 13.1 Application Repositories perfect fit for Domino Backup

Veeam Backup Application Repositories are an interesting new option for application backups — and they are a particularly good fit for HCL Domino Backup & Restore.

The repository is based on ZFS and provides NFS storage that application servers can use directly.
This matches the Domino backup architecture very well: Domino Backup & Restore can simply copy databases, transaction logs, DAOS data and other backup data to an NFS target using its standard file-copy flow.
There is no need for a special backup agent or configuration on Domino Backup side. The standard file copy mode works for backup and restore.





The repository supports NFSv4 and benefits from ZFS fast deduplication, which can be particularly useful for Domino backup data.
Deduplication isn't configurable in the GUI today. But there is a way to get root access to the repository to enable Fast Dedup on the ZFS data sets.

Linux Domino servers can mount the repository directly via NFS. On Windows, NFS should currently also be used for this scenario. In my testing, NFSv3 from Windows provided good performance.
The important difference compared with using an arbitrary NFS server is the integration with Veeam.
The Application Repository becomes part of the normal Veeam Backup workflows, while Domino remains responsible for creating an application-consistent backup using its native Backup & Restore functionality.

Each server will get it's own ZFS data set in ZFS pool with permissions per server.
This creates a very clean separation of responsibilities:

Domino creates the application-consistent backup. Veeam provides and manages the backup repository and integrates the resulting data into the overall backup infrastructure.

For Domino environments, that is a remarkably natural combination.

• Comments [0]

Codeberg and Forgejo A European GitHub Alternative

Codeberg and Forgejo — A European GitHub Alternative

I have been looking at alternatives and additions to GitHub, particularly solutions that are European, open source and can also be self-hosted.


1. Codeberg — Community Git Hosting from Germany


Codeberg is a non-profit Git hosting platform operated by Codeberg e.V., a registered non-profit association based in Berlin, Germany.
It provides much of what we expect from a modern development platform: Git repositories, organizations, issues, pull requests, releases, packages and CI/CD.
And it isn't limited to source code. Codeberg also provides an OCI/Docker container registry.

As a first test I have migrated (that's the term they use to mirror repositories) a project and publish a container image:

docker pull codeberg.org/nashcom/nshgeoip:latest


2. Forgejo — Run Your Own Git Platform


The software behind Codeberg is Forgejo, a free and open-source software forge. Which is actually based on Gitea.
Forgejo provides Git hosting, organizations, issues, pull requests, releases, packages, webhooks, a container registry and Forgejo Actions for CI/CD.
Even more interesting: Forgejo can be self-hosted. Binaries and container images are available, making it relatively easy to operate your own complete Git development platform.


A European Open-Source Ecosystem


This combination is what makes it particularly interesting:

• Codeberg gives us a public, community-operated Git platform from Germany.
• Forgejo gives us the open-source software to operate the same infrastructure ourselves.
  
  

And both go considerably beyond Git repositories — including CI/CD and an OCI container registry.
For my existing projects, GitHub will remain the primary platform for now.

But Codeberg is an excellent additional European home for the projects, while Forgejo is particularly interesting for building a completely self-hosted development infrastructure.

We could also create mirrors as a backup for GitHub hosted projects.
Here is an example result of an Action I wrote to compile a project and publish the container image as a package.
The main difference is that in contrast to GitHub you have to create your own runners to run your compile and image builds.
I ran my build on a Hetzner Linux box with Docker installed. It just needs "node" and a small application + a token.

• Comments [0]

Moving from Fail2Ban to CrowdSec

For many years I have been using Fail2Ban to protect HCL Domino servers against brute-force attacks.
Fail2Ban is small, reliable, and does its job very well:


Log -> Fail2Ban -> Firewall


I also built additional tooling around Fail2Ban to make deployment and integration with Domino easier.
But our requirements have evolved. We don't just want to detect an attack on a single server and add an IP address to its local firewall.
We want to share decisions between systems and, just as importantly, have much better visibility into what is happening.
That's why I have started moving our environment from Fail2Ban to CrowdSec.


A different architecture


CrowdSec separates log processing, attack detection, decisions, and enforcement.
Our implementation looks roughly like this:
Domino ----\
SSH --------> CrowdSec -> Decisions -> Firewall
NGINX -----/      |
                +-> OpenTelemetry -> Loki -> Grafana

CrowdSec parsers turn log messages into structured events. Scenarios detect suspicious behavior and create decisions. Remediation components then enforce those decisions — in our case using nftables.
For Domino, I implemented our own parser and brute-force scenario. It currently detects repeated Internet password authentication failures and creates a ban after the configured threshold is exceeded. (GitHub)


Standalone or with a central hub


Each protected server can run completely standalone.
For larger environments, I also implemented an optional self-hosted CrowdSec hub. It provides the central CrowdSec Local API (LAPI), allowing multiple servers to report alerts and retrieve decisions.


Server 1 ----\
Server 2 -----\
Server 3 ------> CrowdSec Hub -> Decisions
Server 4 -----/         |
                     Bouncers


This means an attack detected by one system can become a decision available to the other connected systems.
The hub is optional. A server only starts using it when it is explicitly registered. (GitHub)
This is also important for environments where everything needs to remain inside the company infrastructure.


OpenTelemetry provides the missing visibility


One of my favorite parts of the implementation is the OpenTelemetry integration.
CrowdSec decisions are forwarded as structured OpenTelemetry events:


CrowdSec -> OpenTelemetry -> Loki -> Grafana


This gives us a completely different view compared with inspecting Fail2Ban logs or firewall rules.
The Grafana dashboard shows decisions from all connected systems, including the host, service, source IP, ban duration, event count, country, ASN/network organization, and geographic location.
We can see decisions over time and display their origins on a map.
So CrowdSec becomes more than an automated blocking mechanism. It becomes another source of structured security telemetry in our observability infrastructure.


Fail2Ban is still a great tool


This move isn't because there is anything fundamentally wrong with Fail2Ban.
For the classic requirement


watch log -> detect pattern -> block IP


Fail2Ban remains a very good solution.
Our requirements have simply grown beyond that model.
CrowdSec gives us multiple event sources, centralized decisions, independent remediation, GeoIP enrichment, and integration with our OpenTelemetry observability infrastructure.
The most important change therefore isn't really Fail2Ban versus CrowdSec.
It is this:


Before:

Domino -> Fail2Ban -> local firewall


Now:

Domino ----\
SSH --------> CrowdSec -> Decisions -> Remediation -> nftables
NGINX -----/      |
                +-> OpenTelemetry -> Loki -> Grafana

We now get both sides of the equation:


automated protection and visibility into what is happening.


CrowdSec Tools


I have published the tooling used for this setup as open source.
It contains crdsectl for installing and managing CrowdSec on protected servers and crdsec-hub for the optional self-hosted central CrowdSec infrastructure.
The Domino integration is the first service-specific implementation, with the architecture designed to support additional services as well. (GitHub)

See the GitHub project for more information

nashcom/crowdsec-tools on GitHub

• Comments [0]

nshgeoip — Small Local GeoIP Lookup Service

While working on CrowdSec and NGINX integration, I needed a simple and efficient way to perform local GeoIP lookups.
There are several ways to integrate GeoIP information directly into individual products, but I wanted something more generic: a small local service that can be shared by NGINX, CrowdSec integrations, scripts, containers, and other applications.

The result is nshgeoip.

What is nshgeoip?

nshgeoip is a small C++ service using MaxMind MMDB databases through libmaxminddb.
It provides GeoIP information such as:

• Country and continent
• ASN and organization
• City and postal code
• Latitude and longitude

The normal interface is a UNIX domain socket:

IP address -> nshgeoip -> local MMDB lookup -> GeoIP information

An optional TCP listener is available for cases where the consumer runs in another container or network namespace.

A lookup is simply:

GET /lookup?ip=8.8.8.8

and returns information as JSON as well as useful X-GeoIP-* response headers.

For example, NGINX can use nshgeoip through an auth_request subrequest to obtain GeoIP information and then decide itself how that information should be used.


Alpine static binary without any external dependencies


For the container build, nshgeoip is compiled as a fully static binary using musl on Alpine Linux, including a statically linked libmaxminddb.
Alpine is only used as the build stage. The final runtime image is built FROM scratch, so Alpine itself isn't part of the resulting image.
There is no Linux distribution, shell, package manager, or other runtime dependency — essentially just the nshgeoip binary.
The service provides GeoIP lookups, health checks, Prometheus metrics, IPv4/IPv6 support, UNIX socket and optional TCP access, and concurrent request processing.
The GeoIP databases themselves are mounted separately and can therefore be updated independently from the container image.


Fast local lookups


The MaxMind MMDB databases are opened once at startup using the normal memory-mapped access provided by libmaxminddb.
There is no need for an additional application-level cache or to reopen a database for every request.

A small fixed-size worker pool handles concurrent requests.
During load testing, nshgeoip easily handled tens of thousands of lookups per second.
That's far beyond what is required for the intended use case, where a local service such as NGINX performs a GeoIP lookup while processing a request.


Health and metrics included


Because nshgeoip is also intended to run as a small infrastructure service, health and monitoring support are built in:


/lookup
/health
/metrics


/health can be used for Docker and Kubernetes health checks.
The FROM scratch container doesn't contain curl, wget, or even a shell, so the binary also provides its own health-check command.
This allows Docker to check the service without adding another tool to the image just for health monitoring.
/metrics provides Prometheus metrics including request counters, lookup results, HTTP response codes, and the age of the loaded GeoIP databases.
Metrics can optionally also be written periodically to a Prometheus textfile collector directory.


Getting information about GeoIP databases


There are separate command line tools to check the database. But they are not included in Ubuntu and other distributions.
So I added the functionality also into the binary.

For example:

nshgeoip --check-db /var/lib/GeoIP/GeoLite2-City.mmdb

shows information including the database type, build date, and age.
The service also reports database age through its health and Prometheus interfaces, making it possible to detect outdated GeoIP data automatically.


Local GeoIP mirror for internal environments


For environments where servers should not download GeoIP databases directly from the Internet, I also added a small update and synchronization architecture.
One designated system downloads the databases from MaxMind and provides them through an internal NGINX mirror. Other systems synchronize the databases from this internal source with checksum verification.

This means only one system needs MaxMind credentials and Internet access.
It also fits well with isolated or tightly controlled server environments where application servers shouldn't make external connections just to keep GeoIP information current.


Very small footprint


With a container image below 1 MB, no runtime operating system, no outbound network dependency, and built-in health and Prometheus support, it turned into a useful little infrastructure building block.


Source code


nshgeoip is open source and available on GitHub:

nashcom/nshgeoip

https://github.com/nashcom/nshgeoip

• Comments [0]

Ubuntu 26.04.1 LTS Is About to Be Released – Upgrade Test Successful

Ubuntu 26.04.1 LTS is about to be released. The first point release of Ubuntu 26.04 LTS is scheduled for today, August 27, 2026.
At the time of writing, the Ubuntu 26.04.1 ISO images are not yet available and the regular LTS upgrade path from Ubuntu 24.04 LTS has not yet been enabled.
However, the Ubuntu 26.04.1 updates are already available in the package repositories. Existing Ubuntu 26.04 installations can already be updated to the 26.04.1 package level.
I tested this first with an Ubuntu Docker container and then went one step further and successfully upgraded an Ubuntu 24.04 LTS Proxmox LXC container to Ubuntu 26.04.1 LTS.


Ubuntu 26.04.1 Updates Are Already Available


Existing Ubuntu 26.04 systems can already be updated normally.
A simple way to verify this is with the official Ubuntu 26.04 Docker image:

docker run --rm -it ubuntu:latest


Inside the container:


apt update
apt full-upgrade -y
cat /etc/os-release

After installing the available updates, the container reports:
PRETTY_NAME="Ubuntu 26.04.1 LTS"

Testing an Upgrade from Ubuntu 24.04 LTS


The next question was more interesting: Can we already upgrade an existing Ubuntu 24.04 LTS installation to Ubuntu 26.04.1?
For this test, I used an Ubuntu 24.04 LTS LXC container running on Proxmox.
This is a perfect environment for an early upgrade test because the complete container can easily be snapshotted and rolled back.
My test container has CT ID 142.


First, I created a snapshot on the Proxmox host:

pct snapshot 142 before-ubuntu-26

The available snapshots can be displayed with:


pct listsnapshot 142

And if required, the complete test can be rolled back:


pct rollback 142 before-ubuntu-26


Regular LTS Upgrade Is Not Enabled Yet


After bringing the Ubuntu 24.04 installation up to date, I checked for the new LTS release:


do-release-upgrade -c

At the time of testing, the result was:
Checking for a new Ubuntu release
There is no development version of an LTS available.
To upgrade to the latest non-LTS development release
set Prompt=normal in /etc/update-manager/release-upgrades.

This means the regular Ubuntu 24.04 LTS → Ubuntu 26.04 LTS upgrade path has not yet been enabled.
That is not unexpected while the 26.04.1 release process is still in progress.


Testing the Upgrade Anyway


Because this was a lab container with a Proxmox snapshot, I decided to test the development upgrade option:

do-release-upgrade -d

The -d option tells do-release-upgrade to also consider the development release.
The upgrader found Resolute, the Ubuntu 26.04 release, and started the upgrade.
The upgrade from Ubuntu 24.04 completed successfully.
After rebooting the container:

cat /etc/os-release

shows:

PRETTY_NAME="Ubuntu 26.04.1 LTS"

So we successfully upgraded an Ubuntu 24.04 LTS installation to the upcoming Ubuntu 26.04.1 LTS package level.


Current Status


At the time of writing, the situation looks like this:

Ubuntu 26.04 -> 26.04.1 update       Available
24.04 -> 26.04.1 with -d             Tested successfully
Ubuntu 26.04.1 ISO                   Not yet available
Regular 24.04 LTS -> 26.04 upgrade   Not yet enabled

Ubuntu 26.04.1 is clearly very close to final publication. Existing Ubuntu 26.04 installations can already update to the 26.04.1 package level, and the release upgrade from Ubuntu 24.04 LTS also worked without issues in my Proxmox LXC test when using do-release-upgrade -d.
For production systems running Ubuntu 24.04 LTS, I would still wait until Canonical officially enables the normal LTS upgrade path.

There is little reason to use do-release-upgrade -d on a production server just to get the new release a little earlier.
But for a snapshot-backed lab system, it was a good opportunity to test the Ubuntu 24.04 LTS → 26.04.1 LTS upgrade ahead of the official rollout.
So far, both the regular 26.04 update and the 24.04 release upgrade look good.
Ubuntu is one of the few distributions supporting major release updates inline.
This is pretty awsome and I used it for earlier releases already.

The upgrade option is available as soon the .1 release ships as this pre-check test showed.
Once available the -c option should work.

• Comments [0]

Push Proxmox statistics into Grafana Prometheus

Proxmox supported Graphite and InfluxDB already for a while.
Now they added OpenTelemetry as another option.



OpenTelemetry is the vendor independent standard, which helps integrating.
I already added OpenTelemetry support for the Grafana Loki instance in the Domino Grafana compose stack.

Now I am adding an OpenTelemetry enpoint to Prometheus.
It's only one option to enable the endpoint and another NGINX listener with authentication to expose it in the compose stack.

With this in place you can just configure OpenTelemetry on Proxmox.
Personally I will not need those extra stats for my Proxmox host. But this is all about adding options and looking into new open standards.






There is a ready to go first dashboard you want to look into:

https://grafana.com/grafana/dashboards/23855-proxmox-ve-dashboard


But this would be only a starting point. I found out an issue with I/O Wait.
The original isn't calculating the value correctly.

proxmox_node_cpustat_iowait_seconds_total is a very specific stat, which is a total over time. But for all CPUs.

The original puts it into relation with up time. We need a rate and need to put it in context to the number of CPUs from what it looks like.
I did some testing and it now matches Linux sar output.


rate(proxmox_node_cpustat_iowait_seconds_total{node="$server"}[30s])/proxmox_node_cpustat_cpus_ratio{node="$server"}

• Comments [0]

Build faster with parallel make operations

When compiling complex projects parallel compile operations can dramatically speed up the build pipeline.
In my case I was compiling NGINX and also building LibCurl and OpenSSL from scratch.

Both type of build pipelines are running much faster with 4 threads instead of one.

The following lines use up to 4 cores. Usually going beyond the 4 cores is not helpful and could block too many resources on a larger machine.
Specially because the parallel processes would also use more memory in combination.

It turned out that the max of 4 are a good balance.
Another interresting optimization was to let the build container use tmpfs instead of a disk.

In my case I am setting build flags globally. But it would also work with just: make -j4.

-- Daniel


# Use up to 4 CPU cores for make
if [[ -z "${MAKEFLAGS:-}" ]]; then
BUILD_JOBS=$(nproc 2>/dev/null || echo 1)
(( BUILD_JOBS > 4 )) && BUILD_JOBS=4
export MAKEFLAGS="-j${BUILD_JOBS}"
echo "Parallel Build Jobs: ${BUILD_JOBS}"
fi

• Comments [0]