Support for Markers in SLF4J backend

[Edarke]

I'm trying to use Flogger with SLF4J+Logback as a backend so that I can output structured logs using logstash-logback-encoder. With the SLF4J API, I can add structured attributes to log statements that can be parsed and efficiently searched.

I want to use the Flogger API for this, so I can take advantage of rate-limiting logging. It looks like flogger's with(MetadataKey, Value) would be a good fit for this. However, it seems the SFL4J-backend doesn't pass the Metadata into the SLF4J's LogEvent, but instead directly appends it to the log message. Would it be possible to pass Flogger's metadata to SLF4J and let the logging configuration choose how to format it?

Here are some concrete examples of what I'm trying to achieve in Logback and what I get with flogger:

Logback Key-value as attributes

logback.atDebug().addKeyValue("key", "value").log("Logback with key values");
 
{"t":"2025-04-06T16:37:25","s":"INFO", "msg":"Logback with key values","attr":{"key":"value"}}

Logback Marker as attributes

if (testLogBack.isDebugEnabled())
  logback.atDebug().addMarker(append("key", "value")).log("Logback with marker");

{"t":"2025-04-06T16:37:07","s":"INFO", "msg":"Logback with marker","attr":{"key":"value"}}

Flogger MetaData gets appended to the formatted message

flogger
    .atFine()
    .with(MetadataKey.single("key", String.class), "value")
    .log("Flogger with key values");

{"t":"2025-04-06T16:37:25","s":"INFO", "msg":"Flogger with key values [CONTEXT key=\"value\" ]"}

[netdpb]

Do you need to use metadata for this information or would it be nice to just log the actual objects passed as arguments to the log() method?

In any case, Flogger supports logging structured data. The default SLF4J logger just formats the whole thing as a string (including metadata), but you are free to implement your own LoggerBackend that intefaces directly with Logback and passes those objects however makes sense.

[cpovirk]

Would it make sense for the default SLF4J logger to pass through all metadata values as key-value pairs nowadays? It looks like the logger's current metadata handling was written back in 2021, and it looks like the SLF4J KeyValuePair class (and associated LoggingEventBuilder) didn't exist until SLF4J 2.0.0 over a year later.

Nowadays, we could probably use logger.atFine().addKeyValue(key, value).log(message, thrown), etc. (It looks like SLF4J also now supports atLevel(Level), so perhaps we can simplify our code a bit.)

It sounds like markers are a separate concept from key-value pairs. Would key-value pairs be sufficient, or would you need support for markers, as well?

[hagbard]

I thought a bit about the whole "marker" thing back when doing the Log4J backend stuff, but concluded there wasn't a semantically clean way to convert Flogger metadata to markers in the general case. However supporting markers explicitly is definitely doable.

My approach would be to create:

1. A custom single-valued MetadataKey<Marker> constant.
2. Implement the appropriate metadata handler changes in the Log4J backend to recognize, extract and handle any markers appropriately (this is all quite well documented and well supported, so not really that hard).

Then you just use:

private static final MetadataKey<Marker> LOG4J_MARKER= Log4jMarkers.marker();
private static final Marker FOO_MARKER = MarkerManager.getMarker("FOO!");

logger.atInfo().with(LOG4J_MARKER, FOO_MARKER).log(...)

or statically imported:

logger.atInfo().with(marker(), FOO_MARKER).log(...)

And if that's too verbose, you can always make a factory to produce some custom marker key subtypes so you can just do:

private static final Log4jMarker BAR_MARKER = Log4jMarkers.create(...);

logger.atInfo().with(BAR_MARKER ).log(...)

In this case, the key type extends MetadataKey<Boolean>, and the marker string/data is in the key itself, the value is always 'true' for the 1-arg with() method, and can be ignored. You WILL need to also implement custom text formatting in the MetadataKey subclass though for this case, since it should end up looking the same in all respects as the first case.

I'd start with the first part though to get things working.

General MetadataKey entries cannot be inferred to be markers though (since they aren't registered with the manager).

And there's some docs for custom metadata keys at:

• https://hagbard.github.io/the-flogger-manual/advanced.html#custom-metadata-keys

The class you'll be specifically interested in for handling markers in the Log4J backend is:
https://github.com/google/flogger/blob/master/api/src/main/java/com/google/common/flogger/backend/MetadataHandler.java
https://github.com/google/flogger/blob/master/api/src/main/java/com/google/common/flogger/backend/MetadataProcessor.java