Daniel Nashed's Blog

Domino Log Forwarder goes OpenTelemetry

I have been working quite a bit on my Domino Log Forwarder project over the last week.
What started as a small utility to forward Domino console output to Grafana Loki has evolved into a more generic OpenTelemetry log forwarder.

One of the most important changes is that there is no Loki-specific output anymore.

otelfwd now sends OpenTelemetry logs using OTLP/HTTP JSON. Grafana Loki is one of the receivers I use and test with, but the forwarder itself doesn't depend on Loki.

One forwarder, multiple input methods

There are several ways to feed data into otelfwd.

The Domino console is the simplest one. Domino STDOUT is piped into otelfwd, just like with the original version of the project.

The new domfwd Domino server add-in goes a step further. It reads structured events directly from the Domino Event Monitoring queue and sends them to otelfwd through a local UNIX socket.

Other applications can use the same structured interface through a UNIX socket or a TCP connection bound to 127.0.0.1.


There is a UNIX datagram syslog interface. I use this, for example, to send NGINX access and error logs directly into otelfwd.

All those inputs use the same forwarding infrastructure from that point on.

Reliable OTLP delivery

otelfwd converts and groups the incoming records into OpenTelemetry resources and scopes and sends them using OTLP/HTTP JSON.
Before sending, records are batched for efficient transmission.
More importantly, otelfwd has its own write-ahead log (WAL). If the configured OTLP receiver isn't reachable, records are kept locally and automatically replayed when the receiver becomes available again.

The backend is no longer Loki only

This is probably the biggest architectural change compared with the original Domino Log Forwarder.
There is no longer a Loki Push API implementation in the forwarder -> The output is OpenTelemetry only.

For my own environment I currently use Grafana Loki, which provides an OTLP endpoint. But from the perspective of otelfwd, Loki is simply an OTLP receiver.

This also makes it much easier to use the same forwarder with other observability platforms. While testing other backends, VictoriaLogs turned out to be an interesting case.
VictoriaLogs also supports OpenTelemetry logs, but its native OpenTelemetry JSON ingestion format is different from the standard OTLP/HTTP JSON representation used by otelfwd.

Native Domino Events forwarder

The project now contains also a Domino Events Forwarder. It's a native integration into the Domino Event Monitoring Stack and allows you to push enhanced events to any OpenTelemetry backend.

See the updated project on GitHub: https://github.com/nashcom/domino-log-forwarder

• Comments [0]