Docker Log Recursion - Unable to Exclude Fluent Bit's Own Logs Without Container ID

[cyxinda]

Problem Description

When using Fluent Bit to collect Docker container logs via the tail input plugin with Docker_Mode On, Fluent Bit collects its own logs, creating an infinite recursion loop. The only workaround is to exclude logs by container ID, which is not generic and breaks when containers are recreated.

Expected Behavior

Fluent Bit should be able to exclude its own logs using container metadata (labels, container name, etc.) rather than relying on container-specific IDs.

Current Limitations

1. No container metadata fields: The tail input plugin with Docker_Mode On only provides time, log, and stream fields. No container metadata (labels, container_name, etc.) is exposed.
2. No generic exclusion: The grep filter cannot access fields like $labels['app'], $container_name, or $docker['container_labels'] because they don't exist in the parsed records.
3. Forced to use container ID: The only working exclusion is via $file ^.*<container_id>.*$, which is not portable across deployments.

Reproduction Steps

1. Run Fluent Bit with Docker log collection:

# docker-compose.yaml
services:
  fluent-bit:
    image: fluent/fluent-bit:latest
    volumes:
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    command: /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.conf

2. Configuration that fails to exclude Fluent Bit's own logs:

[INPUT]
    Name              tail
    Path              /var/lib/docker/containers/*/*.log
    Parser            docker
    Docker_Mode       On

[FILTER]
    Name          grep
    Match         *
    # These don't work:
    Exclude       $labels['app'] ^fluent-bit$
    Exclude       $container_name ^fluent-bit$

3. Observe recursive logging and infinite loop.

Current Workaround (Not Acceptable)

[FILTER]
    Name          grep
    Match         *
    # Container ID changes on recreation
    Exclude       $file ^.*abcd1234.*$

Error/Warning Messages

[ warn] [env] variable ${labels} is used but not set
[ warn] [env] variable ${container_name} is used but not set
[ warn] [env] variable ${docker['container_labels']} is used but not set

Environment

• Fluent Bit version: 4.2.0 (also observed in earlier versions)
• Docker version: Any
• OS: Linux

Suggested Solutions

Option 1: Enhance Docker Mode to Include Metadata

Extend the tail input plugin's Docker_Mode to add container metadata fields:

• container_id, container_name, image, labels, etc.

Option 2: Add Built-in Self-Exclusion Filter

Add a filter that automatically excludes Fluent Bit's own logs:

[FILTER]
    Name          exclude_self
    Match         *
    # Automatically detects and excludes fluent-bit containers

Option 3: Expose Docker API Fields

Make Docker API metadata available via the docker filter (which currently doesn't exist in some distributions):

[FILTER]
    Name          docker
    Match         *
    # Adds container metadata for filtering

Option 4: Support Label-Based Filtering in Input

Add label filtering directly to the input plugin:

[INPUT]
    Name          tail
    Path          /var/lib/docker/containers/*/*.log
    Parser        docker
    Docker_Mode   On
    Exclude_Labels app=fluent-bit

Use Case Importance

This is critical for:

1. Dynamic environments where container IDs change frequently
2. CI/CD pipelines with disposable containers

Related Issues

• Lack of docker filter plugin in some Fluent Bit distributions
• Inability to access container labels for routing decisions

Additional Context

Users have tried various workarounds:

1. Lua scripts to query Docker API (requires additional tools not in minimal images)
2. External tools like logspout (adds complexity)
3. Custom-built Fluent Bit with docker filter (not standard)
4. Container ID exclusion (breaks on container recreation)

This issue prevents Fluent Bit from being a truly generic Docker logging solution and forces users to adopt fragile, deployment-specific configurations.

---

Impact: High - Affects all Docker deployments using Fluent Bit for log collection
Priority: Critical for production environments
Workaround Exists: Yes, but unacceptable (container ID dependent)