[bwplotka]

Alternative to #3129.

Specify two progressive modes (stateless and stateless-typed) for io.prometheus.write.v2.Request in Remote-Write 2.0-rc.6, following the Prometheus Dev Summit consensus and RFC 9413.

By default, when no ;mode= parameter is set on Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request, all field permutations permitted by the base specification may be sent. 2.0-rc.6 adds two optional, progressive modes where senders MUST guarantee:

1. stateless mode (Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request;mode=stateless):
   • Exemplars are attached to a TimeSeries that contains at least one sample or histogram, so no exemplar-only TimeSeries are sent.
   • Histograms use the Histogram message, so classic histograms are sent as NHCB instead of _bucket, _sum, and _count float series.
2. stateless-typed mode (Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request;mode=stateless-typed):
   • All requirements of stateless mode.
   • Metadata.type is set and not METRIC_TYPE_UNSPECIFIED.
   • start_timestamp is set (non-zero) on every sample and histogram for cumulative and delta metric types (METRIC_TYPE_COUNTER, METRIC_TYPE_HISTOGRAM, and METRIC_TYPE_SUMMARY _sum and _count).

Both modes are optional for senders and receivers and remain valid io.prometheus.write.v2.Request payloads on the wire.

Why two progressive modes instead of one mode or separate capabilities?

• stateless rules, attaching exemplars to samples or histograms and converting classic histograms to NHCB, are sender-side formatting choices. Any sender such as Prometheus or Alloy can enforce them by configuration alone without dropping data, even when scraping legacy targets.
• stateless-typed rules, Metadata.type and start_timestamp, require every series to have a known metric type and counter start time. When scraping targets that expose untyped metrics or omit _created timestamps, a sender can only satisfy stateless-typed by synthesizing state, inferring types, or dropping non-conforming series.
• stateless-typed includes stateless because receivers that require explicit metric types and start timestamps also require structured histograms and attached exemplars, and validating start_timestamp already requires knowing Metadata.type.
• Progressive modes (default < stateless < stateless-typed) give 3 well-defined protocol states rather than $2^N$ capability combinations (RFC 9413), keep Content-Type matching to a single parameter value, and keep prometheus/compliance testing linear.

[ENHANCEMENT] Remote-Write 2.0: Add optional `stateless` and `stateless-typed` modes (`rc.6`).

[bwplotka]

Proposed implementation of those modes in Prom:

• storage/remote: add stateless and stateless-typed modes to remote_write prometheus#20069
• storage/remote: coalesce exemplars onto matching v2 TimeSeries in batch prometheus#20070

[dashpole]

This split makes sense to me

[krajorama]

nit: if the type is not set, it's impossible to validate this, which questions the stateless name a little bit and the usefulness of this mode :(

[bwplotka]

Well, not if we specify a clear heuristic e.g. _bucket, _sum, _count UNTYPED can be rejected. But yes, this is probably the biggest downside of this solution.

[krajorama]

_sum and _count can be summary as well, so nope. This is what we ran into when migrating to native histograms internally, we couldn't tell from PromQL queries alone what the type was, so we had to cross reference with the source code all the time.

[kgeckhart]

I'm a bit naive on this front, are summaries something we still want to support to the point that this would be a blocker beyond updating the spec to say they get dropped too if they are untyped?

[krajorama]

note: I'm unsure about this combined with metadata. As we said, maybe sending ST with every sample isn't a good idea for bandwidth when it comes to cumulative metrics. But I'm unsure about the actual overhead.

[bwplotka]

Well, it's what some backends needs - true ST on cumulatives.

In Prometheus ecosystem I can imagine stateless + metric types (no ST) is the future (aka 3.0)

[krajorama]

Practically, when a client sends the new mode there's no feedback if it was actually verified. Unless of course it's flat out refused with HTTP status 415.

Which means that the user needs to ask the admin of the receiver whether the check was actually performed.

Consider returning X-Prometheus-Remote-Write-Mode-Checked: <stateless|stateless-typed> if a check was actually performed.

[bwplotka]

Practically, when a client sends the new mode there's no feedback if it was actually verified. Unless of course it's flat out refused with HTTP status 415.
Which means that the user needs to ask the admin of the receiver whether the check was actually performed.
Consider returning X-Prometheus-Remote-Write-Mode-Checked: <stateless|stateless-typed> if a check was actually performed.

I don't think we should add a response header for this:

1. ;mode= is a sender-side guarantee (similar to RFC 6906 profiles), not a request for the receiver to validate the sender. Checking that a sender implementation actually satisfies stateless or stateless-typed is the job of prometheus/compliance tests, not production receivers on every request.
2. In practice, receivers validate what their ingestion pipeline needs rather than linting every rule in the mode (for example, a receiver might require Metadata.type and start_timestamp and return 400 when missing, but not burn CPU checking if a float series name ends in _bucket). That makes a single Mode-Checked header hard to define accurately.
3. A sender has nothing actionable to do with that header at runtime on a 2xx response. If a receiver requires a mode, the user already gets immediate feedback via 415 when the mode is not sent and 400 when data cannot be ingested. If a generic receiver (like Prometheus TSDB) accepts all modes without needing the extra guarantees, paying per-response header overhead just to report that is unnecessary.

[kgeckhart]

Do we want to do anything about opening up compression options for rc6 or leave that for a different RC?

[bwplotka]

Yes, let's do in separate PR, can be in same RC, feel free to propose a PR

[bwplotka]

We decided to follow #3147 change instead