Propose using a different schema to represent Events in a span

[awangc]

Component(s)

exporter/elasticsearch

Is your feature request related to a problem? Please describe.

When storing Span Events in elasticsearch, the event name becomes the key in the default mapping mode, under which different attributes are stored, e.g. if we have events with name "my-event-1", "my-event-2", then in Elasticsearch we'll have Events.my-event-1.time, Events.my-event-2.time, etc. This does not seem to follow the data format for events for a Span from opentelemetry collector, which are modeled as an array of Span_Event, in which a Span_Event will contain fields like time, name and array of attributes.

The issue I see with this approach is that if name is given arbitrary values (e.g. random UUIDs), then we could see an arbitrary increase in the number of keys, leading to mapping field explosion in Elasticsearch.

Describe the solution you'd like

Store span events as an array in elasticsearch, in which each element is an object with fields with time, name and array of attribute (with dropped attribute counts as another possible field - like the Span_Event class)

Admittedly this format may require nested objects which may bring its own performance issues, but it resembles more the data layout from opentelemetry pdata.

Describe alternatives you've considered

No response

Additional context

The schema proposed above would follow the same format for spans, e.g., we have Span.Name and Span.Attributes, and we'd have Event.Name and Event.Attributes, and more closely represents the Event as defined in opentelemetry

[github-actions]

Pinging code owners:

• exporter/elasticsearch: @JaredTan95 @carsonip @lahsivjar

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[gregkalapos]

Hey 👏

with the default mapping mode (which is the mode none), this is indeed the case, but for OTel traces, I'm not sure how useful that mode is at all. If you look at how data is stored, most things won't follow the shape of the data suggested by OTel - e.g. resource attributes are also stored under Resource.* directly and Kibana is not showing those traces either on the APM UI and the service isn't visible anywhere under Observability - and that's because the none mapping mode.

So for traces, the otel mapping mode would be the way to go.

For that mapping mode there was already a lot of thinking on how data is stored.

When it comes to how OTel data is stored in Elasticsearch, with the otel mapping mode, we have 3 main data streams where data ends up - 1) traces 2) logs, and 3) metrics.

All I'm saying here is meant for the otel mapping mode.

The idea is that events are modelled as log records, therefore they'll end up in the logs data stream - that's very natural for log events, may not be for span events, but I think so far, the idea is that all events will end up in the same data stream. You can of course connect back those span events to the original span via the span id.

This is aligned with what's stated here in the docs.

It's interesting to see that in OTLP, there is a specific type for span events which is totally different from log events.

In any case - with the otel mapping mode, there should not be a cardinality problem, because the event name is a top level field and other fields - e.g. attributes are not embedded under the event name in that case.

So, I'd say the otel mapping mode is the way to go.

Having said that, a few throughs on the default mapping mode (none):

The issue I see with this approach is that if name is given arbitrary values (e.g. random UUIDs), then we could see an arbitrary increase in the number of keys, leading to mapping field explosion in Elasticsearch.

That's technically clearly possible, and would be indeed an issue. Honestly I'd have expected to spec to say something about event name cardinality, all I find is this part here:

It is also recommended that the event names have low-cardinality, so care must be taken to use fields that identify the class of Events but not the instance of the Event.

With that, I'd say, if the user follows the spec, there should not be cardinality explosion. On that other hand, the events API easily allows this, so I think you raise a good point here.