Daniel Nashed's Blog

Loki Integration next steps

The first integration was based on promtail.
But meanwhile Alloy is the new tool. It is a 400 MB binary.

I have implemented Alloy to read log files. But their might be better ways to integrate it.

Detlev came up with the idea for annotating the process ID of the log line.
That's a bit tricky. But there is a way to implement it. pid.nbf contains the process.ids for all running Domino processes.
But Alloy or promtail can't really annotate PIDs.

A custom program could provided this mapping by reading pid.nbf and evaluating the process-id/thread.id information.

The Domino console.log file rotates in some weird way.
The Start Script logs into notes.log which does not rotate at run-time.

This brings up another idea. Why would we want to write the log first before annotating it.

server | nshlog > /local/notesdata/notes.log

The small C++ program can annotate the log and write it to the log file and also push it to Loki in parallel.

Pushing to the Alloy client didn't work out. Even the Alloy client is 400 MB, the HTTP end-point wasn't configurable.
But I came up with another way.

Loki HTTP API

https://grafana.com/docs/loki/latest/reference/loki-http-api/#ingest-logs

Logs can be also pushed directly to Loki. It turns out to be a more direct way.
The only challenge is to have a way temporary store logs in case Loki is not reachable.

Other benefits of an own annotator

Instead of trying to annotate the time format -- which are not that easy to parse in different locales.
Because the log is almost in real-time the annotation can use the ingestion time instead of parsing the timedates.

This makes annotation a lot easier. The resulting format is JSON and can be pushed directly via LibCurl.

I wrote a first implementation of nshlog. The next step is to decouple the read from the write part and have it in a separate thread to ensure no write operation gets stuck.

The first tests look good. This might end up to be another GitHub OpenSource project which might avoid using the huge Alloy client.

• Comments [0]