[processor/logdedup] Use a specific field as deduplication key

[mauri870]

Component(s)

processor/logdedup

Is your feature request related to a problem? Please describe.

Currently the processor uses the entire log message as the deduplication key, allowing the user to remove certain fields from being considered during the hashing part via the exclude_fields option.

One scenario that is not covered is that the deduplication key could be already present as a body or attribute in the log message. You could exclude the other body and attributes via exclude_fields, but this negative match is very error prone and does not scale well.

Describe the solution you'd like

I propose we add a new option to use a certain LogRecord field as deduplication key. This option accepts ottl style path variables:

processors:
    logdedup:
        key:  body["my_id_field"]

Describe alternatives you've considered

Simpler syntax similar to exclude_fields, not ottl needed since we only care about the field value.

Additional context

No response

[github-actions]

Pinging code owners:

• processor/logdedup: @MikeGoldsmith @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

Makes sense to me.

[djaglowski]

I suggest we use ottl style references to fields: body["foo"] instead of body.foo

[mauri870]

@djaglowski Do you have any resources on how to evaluate the OTTL string from the field? I'm struggling to follow up on this part. For example, I have a LogRecord and an OTTL string like body["my_field"] and I need to extract the actual value from the LogRecord. I’d appreciate any help!

[mauri870]

I discussed this privately with @VihasMakwana, and he brought up some interesting points. Since we only care about the value of the field, using OTTL might be overkill. A syntax similar to exclude_fields could be both more familiar to end users and simpler to implement.

[VihasMakwana]

Also, simple syntax such as attributes.foo.bar is more efficient. We can just do value.split(".") and extract the key from attributes.
And we can follow steps similar to what we do with exclude_keys

opentelemetry-collector-contrib/processor/logdedupprocessor/field_remover.go

Lines 33 to 53 in 49b729d

	func newFieldRemover(fieldKeys []string) *fieldRemover {
	fe := &fieldRemover{
	fields: make([]*field, 0, len(fieldKeys)),
	}
	for _, f := range fieldKeys {
	fe.fields = append(fe.fields, &field{
	keyParts: splitField(f),
	})
	}
	return fe
	}
	// RemoveFields removes any body or attribute fields that match in the log record
	func (fe *fieldRemover) RemoveFields(logRecord plog.LogRecord) {
	for _, field := range fe.fields {
	field.removeField(logRecord)
	}
	}

What do you think @djaglowski ?