Skip to content

rewrite_tag: splitting a log group across tags drops group metadata #12499

Description

@terrorobe

Bug

When rewrite_tag sends records from one log group to different tags, only the first record's tag gets the group metadata and attributes. Records on other tags lose them.

Reproduces with v5.1.2 (official image, linux/amd64). rewrite_tag.c is unchanged on master.

Reproduction

An info log and a payment-failure log share the resource attributes service.name=checkout and deployment.environment.name=production. Retag them by level and print the exported OTLP JSON:

service:
  flush: 1
  grace: 1
  log_level: error

pipeline:
  inputs:
    - name: dummy
      tag: source
      dummy: '{"level":"info","message":"order accepted"} {"level":"error","message":"payment authorization failed"}'
      samples: 1
      fixed_timestamp: true
      start_time_sec: 1700000000
      processors:
        logs:
          - name: opentelemetry_envelope
          - name: content_modifier
            context: otel_resource_attributes
            action: upsert
            key: service.name
            value: checkout
          - name: content_modifier
            context: otel_resource_attributes
            action: upsert
            key: deployment.environment.name
            value: production

  filters:
    - name: rewrite_tag
      match: source
      rule: $level ^(info|error)$ routed.$level false

  outputs:
    - name: stdout
      match: routed.*
      format: otlp_json
      workers: 0
fluent-bit -c repro.yaml

Observed

Both messages are exported, but the payment-failure record has an empty resource:

Exported message service.name deployment.environment.name
order accepted checkout production
payment authorization failed missing missing

A query filtering on those service and environment attributes would miss the failure record, even though it was delivered.

OTLP JSON output
{
  "resourceLogs": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": {
              "stringValue": "checkout"
            }
          },
          {
            "key": "deployment.environment.name",
            "value": {
              "stringValue": "production"
            }
          }
        ]
      },
      "scopeLogs": [
        {
          "scope": {},
          "logRecords": [
            {
              "timeUnixNano": "1700000000000000000",
              "body": {
                "stringValue": "order accepted"
              }
            }
          ]
        }
      ]
    }
  ]
}

{
  "resourceLogs": [
    {
      "resource": {},
      "scopeLogs": [
        {
          "scope": {},
          "logRecords": [
            {
              "timeUnixNano": "1700000000000000000",
              "body": {
                "stringValue": "payment authorization failed"
              }
            }
          ]
        }
      ]
    }
  ]
}

Retagging both records to routed.same preserves both records' resource attributes. Reversing the input order moves the loss to order accepted.

Expected

Both exported records retain service.name=checkout and deployment.environment.name=production, regardless of their destination tags.

Cause

cb_rewrite_tag_filter() passes process_record() the raw bytes since the previous record (source). Only the first slice contains the group opener, so later records emitted under another tag arrive without one.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions