Moving from Fail2Ban to CrowdSec

For many years I have been using Fail2Ban to protect HCL Domino servers against brute-force attacks.
Fail2Ban is small, reliable, and does its job very well:


Log -> Fail2Ban -> Firewall


I also built additional tooling around Fail2Ban to make deployment and integration with Domino easier.
But our requirements have evolved. We don't just want to detect an attack on a single server and add an IP address to its local firewall.
We want to share decisions between systems and, just as importantly, have much better visibility into what is happening.
That's why I have started moving our environment from Fail2Ban to CrowdSec.


A different architecture


CrowdSec separates log processing, attack detection, decisions, and enforcement.
Our implementation looks roughly like this:
Domino ----\
SSH --------> CrowdSec -> Decisions -> Firewall
NGINX -----/      |
                +-> OpenTelemetry -> Loki -> Grafana

CrowdSec parsers turn log messages into structured events. Scenarios detect suspicious behavior and create decisions. Remediation components then enforce those decisions — in our case using nftables.
For Domino, I implemented our own parser and brute-force scenario. It currently detects repeated Internet password authentication failures and creates a ban after the configured threshold is exceeded. (GitHub)


Standalone or with a central hub


Each protected server can run completely standalone.
For larger environments, I also implemented an optional self-hosted CrowdSec hub. It provides the central CrowdSec Local API (LAPI), allowing multiple servers to report alerts and retrieve decisions.


Server 1 ----\
Server 2 -----\
Server 3 ------> CrowdSec Hub -> Decisions
Server 4 -----/         |
                     Bouncers


This means an attack detected by one system can become a decision available to the other connected systems.
The hub is optional. A server only starts using it when it is explicitly registered. (GitHub)
This is also important for environments where everything needs to remain inside the company infrastructure.


OpenTelemetry provides the missing visibility


One of my favorite parts of the implementation is the OpenTelemetry integration.
CrowdSec decisions are forwarded as structured OpenTelemetry events:


CrowdSec -> OpenTelemetry -> Loki -> Grafana


This gives us a completely different view compared with inspecting Fail2Ban logs or firewall rules.
The Grafana dashboard shows decisions from all connected systems, including the host, service, source IP, ban duration, event count, country, ASN/network organization, and geographic location.
We can see decisions over time and display their origins on a map.
So CrowdSec becomes more than an automated blocking mechanism. It becomes another source of structured security telemetry in our observability infrastructure.


Fail2Ban is still a great tool


This move isn't because there is anything fundamentally wrong with Fail2Ban.
For the classic requirement


watch log -> detect pattern -> block IP


Fail2Ban remains a very good solution.
Our requirements have simply grown beyond that model.
CrowdSec gives us multiple event sources, centralized decisions, independent remediation, GeoIP enrichment, and integration with our OpenTelemetry observability infrastructure.
The most important change therefore isn't really Fail2Ban versus CrowdSec.
It is this:


Before:

Domino -> Fail2Ban -> local firewall


Now:

Domino ----\
SSH --------> CrowdSec -> Decisions -> Remediation -> nftables
NGINX -----/      |
                +-> OpenTelemetry -> Loki -> Grafana

We now get both sides of the equation:


automated protection and visibility into what is happening.


CrowdSec Tools


I have published the tooling used for this setup as open source.
It contains crdsectl for installing and managing CrowdSec on protected servers and crdsec-hub for the optional self-hosted central CrowdSec infrastructure.
The Domino integration is the first service-specific implementation, with the architecture designed to support additional services as well. (GitHub)

See the GitHub project for more information

nashcom/crowdsec-tools on GitHub

• Comments [0]

nshgeoip — Small Local GeoIP Lookup Service

While working on CrowdSec and NGINX integration, I needed a simple and efficient way to perform local GeoIP lookups.
There are several ways to integrate GeoIP information directly into individual products, but I wanted something more generic: a small local service that can be shared by NGINX, CrowdSec integrations, scripts, containers, and other applications.

The result is nshgeoip.

What is nshgeoip?

nshgeoip is a small C++ service using MaxMind MMDB databases through libmaxminddb.
It provides GeoIP information such as:

• Country and continent
• ASN and organization
• City and postal code
• Latitude and longitude

The normal interface is a UNIX domain socket:

IP address -> nshgeoip -> local MMDB lookup -> GeoIP information

An optional TCP listener is available for cases where the consumer runs in another container or network namespace.

A lookup is simply:

GET /lookup?ip=8.8.8.8

and returns information as JSON as well as useful X-GeoIP-* response headers.

For example, NGINX can use nshgeoip through an auth_request subrequest to obtain GeoIP information and then decide itself how that information should be used.


Alpine static binary without any external dependencies


For the container build, nshgeoip is compiled as a fully static binary using musl on Alpine Linux, including a statically linked libmaxminddb.
Alpine is only used as the build stage. The final runtime image is built FROM scratch, so Alpine itself isn't part of the resulting image.
There is no Linux distribution, shell, package manager, or other runtime dependency — essentially just the nshgeoip binary.
The service provides GeoIP lookups, health checks, Prometheus metrics, IPv4/IPv6 support, UNIX socket and optional TCP access, and concurrent request processing.
The GeoIP databases themselves are mounted separately and can therefore be updated independently from the container image.


Fast local lookups


The MaxMind MMDB databases are opened once at startup using the normal memory-mapped access provided by libmaxminddb.
There is no need for an additional application-level cache or to reopen a database for every request.

A small fixed-size worker pool handles concurrent requests.
During load testing, nshgeoip easily handled tens of thousands of lookups per second.
That's far beyond what is required for the intended use case, where a local service such as NGINX performs a GeoIP lookup while processing a request.


Health and metrics included


Because nshgeoip is also intended to run as a small infrastructure service, health and monitoring support are built in:


/lookup
/health
/metrics


/health can be used for Docker and Kubernetes health checks.
The FROM scratch container doesn't contain curl, wget, or even a shell, so the binary also provides its own health-check command.
This allows Docker to check the service without adding another tool to the image just for health monitoring.
/metrics provides Prometheus metrics including request counters, lookup results, HTTP response codes, and the age of the loaded GeoIP databases.
Metrics can optionally also be written periodically to a Prometheus textfile collector directory.


Getting information about GeoIP databases


There are separate command line tools to check the database. But they are not included in Ubuntu and other distributions.
So I added the functionality also into the binary.

For example:

nshgeoip --check-db /var/lib/GeoIP/GeoLite2-City.mmdb

shows information including the database type, build date, and age.
The service also reports database age through its health and Prometheus interfaces, making it possible to detect outdated GeoIP data automatically.


Local GeoIP mirror for internal environments


For environments where servers should not download GeoIP databases directly from the Internet, I also added a small update and synchronization architecture.
One designated system downloads the databases from MaxMind and provides them through an internal NGINX mirror. Other systems synchronize the databases from this internal source with checksum verification.

This means only one system needs MaxMind credentials and Internet access.
It also fits well with isolated or tightly controlled server environments where application servers shouldn't make external connections just to keep GeoIP information current.


Very small footprint


With a container image below 1 MB, no runtime operating system, no outbound network dependency, and built-in health and Prometheus support, it turned into a useful little infrastructure building block.


Source code


nshgeoip is open source and available on GitHub:

nashcom/nshgeoip

https://github.com/nashcom/nshgeoip

• Comments [0]

Ubuntu 26.04.1 LTS Is About to Be Released – Upgrade Test Successful

Ubuntu 26.04.1 LTS is about to be released. The first point release of Ubuntu 26.04 LTS is scheduled for today, August 27, 2026.
At the time of writing, the Ubuntu 26.04.1 ISO images are not yet available and the regular LTS upgrade path from Ubuntu 24.04 LTS has not yet been enabled.
However, the Ubuntu 26.04.1 updates are already available in the package repositories. Existing Ubuntu 26.04 installations can already be updated to the 26.04.1 package level.
I tested this first with an Ubuntu Docker container and then went one step further and successfully upgraded an Ubuntu 24.04 LTS Proxmox LXC container to Ubuntu 26.04.1 LTS.


Ubuntu 26.04.1 Updates Are Already Available


Existing Ubuntu 26.04 systems can already be updated normally.
A simple way to verify this is with the official Ubuntu 26.04 Docker image:

docker run --rm -it ubuntu:latest


Inside the container:


apt update
apt full-upgrade -y
cat /etc/os-release

After installing the available updates, the container reports:
PRETTY_NAME="Ubuntu 26.04.1 LTS"

Testing an Upgrade from Ubuntu 24.04 LTS


The next question was more interesting: Can we already upgrade an existing Ubuntu 24.04 LTS installation to Ubuntu 26.04.1?
For this test, I used an Ubuntu 24.04 LTS LXC container running on Proxmox.
This is a perfect environment for an early upgrade test because the complete container can easily be snapshotted and rolled back.
My test container has CT ID 142.


First, I created a snapshot on the Proxmox host:

pct snapshot 142 before-ubuntu-26

The available snapshots can be displayed with:


pct listsnapshot 142

And if required, the complete test can be rolled back:


pct rollback 142 before-ubuntu-26


Regular LTS Upgrade Is Not Enabled Yet


After bringing the Ubuntu 24.04 installation up to date, I checked for the new LTS release:


do-release-upgrade -c

At the time of testing, the result was:
Checking for a new Ubuntu release
There is no development version of an LTS available.
To upgrade to the latest non-LTS development release
set Prompt=normal in /etc/update-manager/release-upgrades.

This means the regular Ubuntu 24.04 LTS → Ubuntu 26.04 LTS upgrade path has not yet been enabled.
That is not unexpected while the 26.04.1 release process is still in progress.


Testing the Upgrade Anyway


Because this was a lab container with a Proxmox snapshot, I decided to test the development upgrade option:

do-release-upgrade -d

The -d option tells do-release-upgrade to also consider the development release.
The upgrader found Resolute, the Ubuntu 26.04 release, and started the upgrade.
The upgrade from Ubuntu 24.04 completed successfully.
After rebooting the container:

cat /etc/os-release

shows:

PRETTY_NAME="Ubuntu 26.04.1 LTS"

So we successfully upgraded an Ubuntu 24.04 LTS installation to the upcoming Ubuntu 26.04.1 LTS package level.


Current Status


At the time of writing, the situation looks like this:

Ubuntu 26.04 -> 26.04.1 update       Available
24.04 -> 26.04.1 with -d             Tested successfully
Ubuntu 26.04.1 ISO                   Not yet available
Regular 24.04 LTS -> 26.04 upgrade   Not yet enabled

Ubuntu 26.04.1 is clearly very close to final publication. Existing Ubuntu 26.04 installations can already update to the 26.04.1 package level, and the release upgrade from Ubuntu 24.04 LTS also worked without issues in my Proxmox LXC test when using do-release-upgrade -d.
For production systems running Ubuntu 24.04 LTS, I would still wait until Canonical officially enables the normal LTS upgrade path.

There is little reason to use do-release-upgrade -d on a production server just to get the new release a little earlier.
But for a snapshot-backed lab system, it was a good opportunity to test the Ubuntu 24.04 LTS → 26.04.1 LTS upgrade ahead of the official rollout.
So far, both the regular 26.04 update and the 24.04 release upgrade look good.
Ubuntu is one of the few distributions supporting major release updates inline.
This is pretty awsome and I used it for earlier releases already.

The upgrade option is available as soon the .1 release ships as this pre-check test showed.
Once available the -c option should work.

• Comments [0]

Push Proxmox statistics into Grafana Prometheus

Proxmox supported Graphite and InfluxDB already for a while.
Now they added OpenTelemetry as another option.



OpenTelemetry is the vendor independent standard, which helps integrating.
I already added OpenTelemetry support for the Grafana Loki instance in the Domino Grafana compose stack.

Now I am adding an OpenTelemetry enpoint to Prometheus.
It's only one option to enable the endpoint and another NGINX listener with authentication to expose it in the compose stack.

With this in place you can just configure OpenTelemetry on Proxmox.
Personally I will not need those extra stats for my Proxmox host. But this is all about adding options and looking into new open standards.






There is a ready to go first dashboard you want to look into:

https://grafana.com/grafana/dashboards/23855-proxmox-ve-dashboard


But this would be only a starting point. I found out an issue with I/O Wait.
The original isn't calculating the value correctly.

proxmox_node_cpustat_iowait_seconds_total is a very specific stat, which is a total over time. But for all CPUs.

The original puts it into relation with up time. We need a rate and need to put it in context to the number of CPUs from what it looks like.
I did some testing and it now matches Linux sar output.


rate(proxmox_node_cpustat_iowait_seconds_total{node="$server"}[30s])/proxmox_node_cpustat_cpus_ratio{node="$server"}

• Comments [0]

Build faster with parallel make operations

When compiling complex projects parallel compile operations can dramatically speed up the build pipeline.
In my case I was compiling NGINX and also building LibCurl and OpenSSL from scratch.

Both type of build pipelines are running much faster with 4 threads instead of one.

The following lines use up to 4 cores. Usually going beyond the 4 cores is not helpful and could block too many resources on a larger machine.
Specially because the parallel processes would also use more memory in combination.

It turned out that the max of 4 are a good balance.
Another interresting optimization was to let the build container use tmpfs instead of a disk.

In my case I am setting build flags globally. But it would also work with just: make -j4.

-- Daniel


# Use up to 4 CPU cores for make
if [[ -z "${MAKEFLAGS:-}" ]]; then
BUILD_JOBS=$(nproc 2>/dev/null || echo 1)
(( BUILD_JOBS > 4 )) && BUILD_JOBS=4
export MAKEFLAGS="-j${BUILD_JOBS}"
echo "Parallel Build Jobs: ${BUILD_JOBS}"
fi

• Comments [0]

Using Server Groups for Domino Replication

One of the lesser-known features in Domino server-to-server replication is that the Destination Server field in a Connection document does not have to contain a single server name. Instead, it can reference a Server Group.
According to the Domino documentation:

"You can also specify a Group name that contains server names so that the Source server replicates with each server listed in the group you specify.
To do this, you create a group that contains servers only, and specify 'Servers only' as the group type. The group cannot contain the names of other groups of servers."
https://help.hcl-software.com/domino/14.5.1/admin/conf_schedulingservertoserverreplication_t.html


At first glance this looks like an elegant way to simplify replication topologies.

A Simple Example

Instead of creating individual Connection documents like this:
ServerA --> ServerB
ServerA --> ServerC
ServerA --> ServerD

you can create a group:

---

Source Server: ServerA
Destination Server: ReplicationTargets

ServerB
ServerC
ServerD

---

and configure a single Connection document:

Domino will iterate over every server in the group and establish replication entry for each member.
This can significantly reduce the number of Connection documents that need to be maintained, especially in hub-and-spoke environments.

Requirements

There are a few rules that are easy to overlook:

• The group type must be Servers only.
• Nested groups are not supported.

The Hidden Catch

During testing we ran into an important limitation that is not immediately obvious from the documentation.
The group approach works well only if Domino can determine how to reach every destination server automatically.

That generally means:

• All servers are on the same Notes Named Network, or
• Domino can determine the network path without requiring server-specific addressing information.

If a destination server requires a dedicated network address, custom traget port, or other server-specific connection parameters, things become more complicated.


Why?

A Connection document that targets a group can only contain one set of connection parameters.
There is no way to specify different TCP/IP addresses or network information for each server in the group.
Once replication is initiated via the group Connection document, the server-specific Connection documents are apparently not consulted for network addressing.


When It Works Well

Using server groups is a great fit for environments such as:

• Servers on the same LAN
• Well-connected Notes Named Networks
• Replication where no explicit network addresses are required
• Hub servers that periodically contact many similarly configured servers
   

• Comments [0]

FRITZ!Box installation with ChatGPT

What a fun experience. Even I know FRITZ!Box configurations quite well, having ChatGPT as a chat buddy during install is great.
No matter if the interface is German or English, ChatGPT really knows the FRITZ!Box interface.

The session we had yesterday trying to get my WLAN working again was great.
We ended up deciding I need a new FRITZ!Box 7690 replacing the 7 year old box and set it up together today because of a broken 2,4GHz WLAN I need for IoT devices.

It was fun mixing German and English switching the interface and getting my IPv6 configuration back again.

The next time you have to setup a FRITZ!Box try the ChatGPT voice mode.

• Comments [0]

Gefana Blackbox Exporter Traveler getStatus Probe

Grafana Blackbox exporter is great and supports all kind of probes. This includes HTTPS, TCP probes.
HCL Traveler supports an authenticated end to end status probe, which can be added to the blackbox probe types.
Here is a quick write up. I am adding it to the Domino Grafana GitHub project.




blackbox.yml

In the blackbox configuration you first define the probe.
The only a bit tricky limitation is that you have to define the user and password in the file or in separate user/password files.

traveler_status:
 prober: http
 timeout: 15s

 http:
   preferred_ip_protocol: ip4
   method: GET

   headers:
     Accept-Language: en-US

   basic_auth:
     username: traveler-monitor
     password: change-me

   fail_if_body_not_matches_regexp:
     - "Traveler server is available\\."

   tls_config:
     insecure_skip_verify: false


prometheus.yml

Once defined you can use the probe and define the targets


- job_name: blackbox-traveler

 metrics_path: /probe

 params:
   module: [traveler_status]

 static_configs:

   - targets:
       - https://traveler.example.com/traveler?action=getStatus
     labels:
       pod: traveler.example.com
       namespace: domino

 relabel_configs:
   - source_labels: [__address__]
     target_label: __param_target

   - source_labels: [__param_target]
     target_label: instance

   - target_label: __address__
     replacement: blackbox-exporter:9115

• Comments [0]

Gitea -- Mirror GitHub projects plus a simple to use registry

This project is pretty cool. It isn't only a git server implementation but offers to interesting options.

https://about.gitea.com/products/gitea/

Git Repository Mirror

In contrast to some other free editions of Git servers like GitLab community edition, it allows to mirror public Git projects in your own environment.
This is specially interesting when you don't want to let your servers access the internet and want to control which projects are used inside your company.
The mirror option is part of the "migration" feature and allows you to simply mirror any repository.

Simple to use Container Registry

For larger environments probably the Harbor registry would be the better fit.
But it you only want to host a couple of images, the Gitea container registry works pretty well.

The whole Gitea server is very resource optimized and runs from a single Go binary. Either natively or in a container.
You are up in a minute running it as Docker container for example. The configuration is straightforward. And just needs a reverse proxy like NGINX or Traefik for TLS.

For a simple test I set it up without TLS and on port 3000.

docker push git.lab.nashcom.de:3000/nashcom/domino:latest

On purpose for the screen shots I used a version without TLS to show it's flexibility.
But in production you should leverage TLS and use port 443.

This setup took me a minute. The Docker image already had the container registry configured out of the box.
The container image is Alpine Linux based and the whole environment is very resource optimized in contrast to some larger projects.

I looked at Gitea earlier and liked it already.
But I never looked for the container registry part, which also works very well.

I have already a small compose stack for Gitea in one of my projects.
But maybe it would make sense to add it to my start script repository including a proxy setup. But it would run behind any load-balancer to offload TLS.

• Comments [0]

Use an USB device like a camera from a remote machine

How cool is that. I wanted to use my new camera should work on multiple devices without plugging it into a different machine all the time.

ChatGPT came up with this project https://www.virtualhere.com/
From what I read it is free for one USB device.
It works on multiple platforms. In my case I plug the camera into my M4 and share it with other machines like my Thinkpad.

A camera needs quite some bandwidth and low latency. I my first tests it worked pretty good.

21.07.2026 Update:

When testing it in a meeting there is still some time lag, similar to what you get with playing audio using bluetooth.
At least in my demo showing it to someone over the preview during a meeting causes a delay.
So I would still prefer to use the camera on the same machine. It is still pretty good to have as an option.


-- Daniel

• Comments [0]