feat: experimentation support (experiment flags and events)

[Zaimwa9]

Adds experimentation support to the Go SDK, at parity with the Python, JS, Flutter and Node ports.

• Parse variant, reason and metadata.experiment from remote flag responses onto Flag.
• New opt-in EventProcessor (WithEvents(ctx)) batching $flag_exposure and custom events to the events API.
• New client methods: GetExperimentFlag, TrackEvent, TrackExposureEvent, FlushEvents, DroppedEvents.
• Error handling per the shared rules in Flagsmith/flagsmith-private#318: retry 408/429/502/503/504 and transport errors, 3 attempts with jittered exponential backoff, drop other 4xx/5xx, stop on 401/403, re-queue failed batches, bounded drop-oldest buffer with a dropped counter, rejected[] logged and not resent, dedupe cleared only on 2xx.
• Cancelling the WithEvents context performs a final bounded flush; FlushEvents(ctx) for short-lived processes.

Local evaluation and offline mode carry no experiment metadata and record no exposures. flagengine/ untouched.

Not yet done: manual acceptance against a live environment.

🤖 Generated with Claude Code

[Zaimwa9]

@themis-blindfold review

[themis-blindfold]

⚖️ Themis review: 🟠 Fix before merge

The experimentation API is well covered by the completed build and pre-commit checks, but two event-delivery defects need fixing: a slow events service bypasses the configured buffer bound, and rejection logging can expose identity PII. Local test execution was unavailable because the Go toolchain is not installed.

Area	Score
🎯 Correctness	2/5
🧪 Test coverage	4/5
📐 Code quality	3/5
🚀 Product impact	4/5

🟠 Majors

• events.go:313 — pending event batches are unbounded while sends are slow.
• events.go:509 — rejected events can emit identity PII to host logs.

📝 Walkthrough

• Experiment flag evaluation - carries remote experiment metadata onto flags and records enrolled remote evaluations.
• Event processing - batches custom events and exposures, with retry, rejection, shutdown, and drop accounting behaviour.
• Client configuration - adds opt-in event lifecycle and delivery tuning options, plus public flushing and dropped-event APIs.
• Documentation - describes setup, retry handling, capacity, and shutdown expectations.

🧪 How to verify

1. Run go test ./....
2. Hold /v1/events responses while tracking more than WithEventsMaxBufferSize events; assert pending memory/goroutines and retained events remain bounded.
3. Return a per-event rejection for an email-shaped identifier and assert warning output contains no identifier.

Automate: add the stalled-events capacity regression test alongside the existing event processor tests.

Product take: Experiment exposure and conversion tracking is a meaningful SDK capability. It needs reliable backpressure and privacy-safe observability before applications can enable it at production traffic.

🧭 Assumptions & unverified claims

No unverified assumptions or claims.

A promising experiment, but the queue needs a lab coat before it meets production traffic · reviewed at b1839c5

[Zaimwa9]

@themis-blindfold review

[themis-blindfold]

⚖️ Themis review: 🟠 Fix before merge

This adds a substantial experimentation pipeline with good coverage of normal batching and retries, but two failure paths remain unsafe: shutdown can leave an automatic batch running and uncounted after its deadline, and a rejected API response can bypass the documented log-sanitisation guarantee. CI is still running; pre-commit completed successfully.

Area	Score
🎯 Correctness	2/5
🧪 Test coverage	4/5
📐 Code quality	3/5
🚀 Product impact	4/5

🟠 Majors

• events.go:206 — Automatic sends can outlive shutdown and requeue events after the processor has stopped.
• events.go:570 — Rejected-response bodies can expose identifiers or traits in logs.

📝 Walkthrough

• Flag metadata - remote identity responses now expose experiment assignment details while local evaluation remains metadata-free.
• Client API - opt-in event tracking adds custom events, manual exposures, experiment-aware flag evaluation, explicit flush, and loss accounting.
• Event processor - batches events on a timer or capacity trigger, retries selected transient failures, deduplicates exposures, and stops on rejected credentials.
• Documentation - explains event lifecycle, retry behaviour, buffering, and explicit shutdown flushing.

🧪 How to verify

1. Run go test ./....
2. Hold a buffer-full automatic POST beyond the configured timeout, cancel the events context, then assert the batch is dropped and counted before the processor stops.
3. Return a 400 response body that echoes an identifier and trait value; assert neither reaches the configured logger.
4. Exercise successful remote experiment evaluation, a retryable event failure, and a 401 stop using an events test server.
   Automate: Add the shutdown-timeout and echoed-4xx-log regression tests to events_test.go.

Product take: This is a major capability for experiment measurement, but inaccurate shutdown accounting and potential identity leakage undermine operational trust in it.

🧭 Assumptions & unverified claims

• Unverified: the hosted events API's 4xx error schema does not guarantee that response bodies exclude submitted identifiers or traits; configured self-hosted or proxied endpoints can also return arbitrary error content.

The event queue has stamina; its shutdown needs a firmer curfew. · reviewed at 1092cb4

[Zaimwa9]

@themis-blindfold review

[themis-blindfold]

⚖️ Themis review: 🟠 Fix before merge

TL;DR: This adds the SDK's experimentation surface and a well-tested buffered events pipeline, but concurrent shutdown can leave a caller-started flush running past the advertised cutoff, and a locally unencodable event is retried forever instead of being treated as a bad payload. The existing open privacy concern around logging API-supplied error detail also remains. The recorded checks show pre-commit completed successfully; both build jobs were still running.

Area	Score
🎯 Correctness	2/5
🧪 Test coverage	3/5
📐 Code quality	3/5
🚀 Product impact	4/5

🟠 Majors

• events.go:340 — shutdown does not bound an explicit flush already in flight.
• events.go:613 — locally invalid event payloads are retried and re-queued indefinitely.
• events.go — existing open privacy finding remains unresolved: API-supplied error detail can still be logged.

📝 Walkthrough

• Flag response mapping - surfaces remote experiment metadata and preserves local-evaluation behaviour.
• Client experimentation API - adds opt-in exposure and conversion tracking methods plus event-related options.
• Event processor - batches, retries, deduplicates and accounts for dropped events through a background worker.
• Documentation - describes configuration, batching, failure handling and shutdown expectations for adopters.

🧪 How to verify

1. Run go test ./....
2. Start FlushEvents(context.Background()) against a stalled events endpoint, cancel the WithEvents context, and assert that every request and flush goroutine stops within the configured timeout.
3. Track an event whose metadata contains make(chan int), then assert it is classified as a local payload failure rather than retried and re-queued on every timer tick.
4. Return a 400 JSON response whose detail echoes an identifier and assert no log entry contains that identifier.
5. Exercise a successful remote experiment evaluation and verify its one exposure reaches /v1/events with the expected variant and experiment ID.
   Automate: Add regression tests for shutdown during an explicit flush and for non-JSON-encodable event metadata.

Product take: Experiment exposure and conversion tracking is a meaningful capability for server-side users.
The retry and shutdown fixes matter because inaccurate or lingering telemetry is especially costly when it feeds experiment decisions.

🧭 Assumptions & unverified claims

No unverified assumptions or claims.

A promising events pipeline; it just needs to stop retrying its own impossible payloads · reviewed at 2eefc0e

[Zaimwa9]

@themis-blindfold review

[themis-blindfold]

⚖️ Themis review: 🟠 Fix before merge

The experimentation surface is well covered, but three lifecycle and privacy defects remain: a shutdown-time manual flush can outlive the advertised bound, a concurrent flush can retry after authentication disables events, and a 202 rejection can leak server-provided text to application logs. CI is still running; the completed pre-commit check passed.

Area	Score
🎯 Correctness	2/5
🧪 Test coverage	3/5
📐 Code quality	3/5
🚀 Product impact	4/5

🟠 Majors

• events.go:735 — Predicted: per-event rejection logging could expose server-supplied response content and identifiers.
• events.go:598 — Predicted: a concurrent manual flush could continue retrying after a 401 or 403 disables event tracking.
• events.go:349 — Predicted: a manual FlushEvents call could outlive the documented shutdown deadline.

📝 Walkthrough

• Flag evaluation - exposes remote experiment metadata and records exposures only for enrolled identities.
• Event processor - adds buffering, batching, retry, deduplication, rejection handling, and shutdown flushing.
• Client API and documentation - adds event options and public tracking/flush methods with lifecycle guidance.

🧪 How to verify

1. Run two concurrent FlushEvents(context.Background()) calls; return 401 to one and 503 to the other, then assert no request starts after the 401 and both batches are counted as dropped.
2. Return 202 {"rejected":[{"index":0,"error":"invalid identifier user@example.com"}]} and assert the emitted slog records contain neither the error text nor the identifier.
3. Cancel the WithEvents context while a FlushEvents(context.Background()) request is stalled and assert the request and its goroutine end by the configured timeout.
4. Run go test ./....
   Automate: add the two concurrent-flush regressions to events_test.go.

Product take: Solid experimentation support, but its event-delivery lifecycle needs to honour the stated stop conditions before it is safe for production analytics data.

🧭 Assumptions & unverified claims

No unverified assumptions or claims.

Even background events need to respect a polite goodbye · reviewed at 56894ad

[Zaimwa9]

@themis-blindfold review

[themis-blindfold]

⚖️ Themis review: 🟠 Fix before merge

The new event pipeline has broad focused coverage and the completed build and pre-commit checks passed. One concurrency gap remains: events do not fully snapshot caller-owned nested data, so tracking can race with ordinary caller updates.

Area	Score
🎯 Correctness	3/5
🧪 Test coverage	4/5
📐 Code quality	3/5
🚀 Product impact	3/5

🟠 Majors

• events.go:468 — nested trait and metadata values remain shared with the caller after tracking.

📝 Walkthrough

• Event processor - adds opt-in buffering, retry, shutdown, deduplication, and loss accounting for experimentation events.
• Client and flag models - surface experiment metadata and record eligible identity exposures.
• Configuration and documentation - add event-pipeline options and the lifecycle contract.

🧪 How to verify

1. Run go test ./....
2. Run go test -race ./....
3. Track an event containing a nested metadata or trait map while another goroutine updates that nested map, then flush; the sent event must be the call-time snapshot without a race or crash.
4. Exercise a stalled events endpoint, a retryable response, and shutdown to confirm the bounded flush and loss counter behaviour.
   Automate: add the nested-map snapshot/race regression case to events_test.go.

Product take: Solid experimentation support, but event tracking runs alongside application request paths.
The shared-data race can turn normal metadata enrichment into a process crash or corrupted analytics, so it needs resolving before users rely on it.

🧭 Assumptions & unverified claims

No unverified assumptions or claims.

The event queue is almost ready to mind its own business · reviewed at 73c559d

[Zaimwa9]

@themis-blindfold review

[themis-blindfold]

⚖️ Themis review: 🟠 Fix before merge

This adds a useful opt-in experimentation pipeline, but concurrent manual flushes can send the same exposure twice. The completed build and pre-commit checks passed; Go was not available here for a local test run.

Area	Score
🎯 Correctness	3/5
🧪 Test coverage	4/5
📐 Code quality	4/5
🚀 Product impact	4/5

🟠 Majors

• events.go:626 — a successful batch clears dedupe keys for unrelated exposures that are still in flight.

⚖️ Acknowledged

• Keep in-flight event batches within the configured capacity. — thread resolved by @themis-blindfold
• Do not write identity values to rejection logs. — thread resolved by @themis-blindfold
• Bound shutdown work, not only the final batch. — thread resolved by @themis-blindfold
• Do not log untrusted events API response bodies. — thread resolved by @themis-blindfold
• Shutdown does not bound an explicit flush already in flight. — thread resolved by @themis-blindfold
• Local JSON encoding failures are retried forever. — thread resolved by @themis-blindfold
• Do not log server-supplied rejection text. — thread resolved by @themis-blindfold
• Stop retrying batches once authentication fails. — thread resolved by @themis-blindfold
• Nested event data is not snapshotted before asynchronous encoding. — thread resolved by @themis-blindfold

📝 Walkthrough

• Event delivery - adds opt-in buffered custom events and experiment exposures, with retries, bounded buffering, shutdown flushing, and dropped-event reporting.
• Experiment evaluation - exposes remote experiment metadata on flags and records exposures only for enrolled remote identity evaluations.
• Configuration and docs - adds client options and documents operational behaviour and shutdown use.

🧪 How to verify

• Add a regression test with two concurrent FlushEvents calls, then track the second batch's exposure after the first batch succeeds; assert it is not queued a second time.
• Run go test ./....
• Run go test -race ./... to exercise the event processor's concurrent flush and shutdown paths.
• Exercise a 202 partial-acceptance response and confirm only the relevant exposure keys are released.
  Automate: keep the concurrent-flush dedupe scenario in events_test.go.

Product take: Solid experimentation support, but duplicate exposure events can distort experiment analysis for callers that flush concurrently.

🧭 Assumptions & unverified claims

No unverified assumptions or claims.

One batch should not give every other exposure a fresh passport · reviewed at 9ebe205

[Zaimwa9]

@themis-blindfold review

[themis-blindfold]

⚖️ Themis review: 🟠 Fix before merge

Observed: a successful batch clears the global exposure-deduplication set. Predicted: concurrent flushes could resend the same exposure. CI is still running; the completed pre-commit check passed.

Area	Score
🎯 Correctness	3/5
🧪 Test coverage	4/5
📐 Code quality	3/5
🚀 Product impact	4/5

🟠 Majors

• events.go:432 — concurrent-flush exposure deduplication.

📝 Walkthrough

• Remote flag parsing — surfaces experiment metadata for remote identity evaluations.
• Client API — adds opt-in event configuration plus experiment, tracking, flushing, and dropped-event methods.
• Event delivery — batches events, applies retry classification, and performs bounded shutdown flushing.
• Data handling — snapshots JSON payloads at tracking time and avoids logging event content.

🧪 How to verify

1. Run go test ./....
2. Run go test -race ./... to exercise the asynchronous sender under the race detector.
3. Start two FlushEvents calls with matching exposure events, let one batch succeed first, then track the exposure again; it must not be sent until the other batch resolves.
4. Exercise a live test environment with a successful batch, retryable failure, 401, and cancellation during a stalled request.
   Automate: add the concurrent-flush regression case that keeps per-batch exposure keys until that batch completes.

Product take: This is a major capability for experiment measurement, but duplicate exposures would skew experiment analysis under concurrent flushes.

🧭 Assumptions & unverified claims

Local Go tooling was unavailable, so the test suite was not run in this review.

The events queue is nearly ready to leave the lab; it just needs to remember which exposure is already on its way. · reviewed at 9797f90

[Zaimwa9]

@themis-blindfold review

[themis-blindfold]

⚖️ Themis review: ✅ Ship it

TL;DR: The opt-in experimentation path now keeps event payloads bounded, sanitised, and cancellable through shutdown, while preserving the existing flag-evaluation behaviour. The completed pre-commit check passed; the two build jobs were still running in the captured snapshot.

Area	Score
🎯 Correctness	5/5
🧪 Test coverage	5/5
📐 Code quality	5/5
🚀 Product impact	4/5

📝 Walkthrough

• Event delivery - adds opt-in batching, bounded retries, rejection handling, and shutdown flushing for experimentation events.
• Experiment evaluation - exposes remote experiment metadata and records eligible identity exposures.
• Configuration and docs - adds explicit event options and documents delivery and shutdown semantics.

🧪 How to verify

1. Run go test ./....
2. Run go test -race ./... to exercise concurrent buffering, flushing, and shutdown.
3. Against a live environment, evaluate an enrolled identity with GetExperimentFlag and confirm a single $flag_exposure arrives.
4. Stall the events endpoint, cancel the events context, and confirm outstanding events are dropped within the configured request timeout.
   Automate: Add the live-environment smoke check to the SDK acceptance suite once the test environment is available.

Product take: Solid SDK parity improvement for experimentation users, with a deliberately opt-in integration and clear delivery semantics.

🧭 Assumptions & unverified claims

No unverified assumptions or claims.

Events have learned to leave tidily when the lights go out · reviewed at 8338dcf

[Zaimwa9]

@themis-blindfold review

[themis-blindfold]

⚖️ Themis review: ✅ Ship it

The experimentation event pipeline, experiment metadata parsing, delivery controls, and shutdown paths are coherently covered. The completed pre-commit check passed; I found no remaining unaddressed defect in the reviewed change.

Area	Score
🎯 Correctness	5/5
🧪 Test coverage	5/5
📐 Code quality	5/5
🚀 Product impact	4/5

📝 Walkthrough

• Experiment flags - remote evaluation metadata is surfaced and eligible exposures are recorded.
• Event delivery - buffered batches have bounded retry, deduplication, rejection handling, and loss accounting.
• Lifecycle - cancellation performs a bounded final flush and stops in-flight work.
• SDK configuration - events remain opt-in, isolated from custom request headers, and incompatible with offline mode.

🧪 How to verify

1. Run go test ./....
2. Run go test -race ./... to exercise the event buffer and shutdown concurrency paths.
3. Point WithEventsBaseURL at a test endpoint and verify a 202 batch, a retryable 503, and a 401 disablement.
4. Cancel the WithEvents context with a buffered batch and confirm the final flush completes within the request timeout.
   Automate: run go test -race ./... in PR CI.

Product take: A solid SDK capability for experiment measurement, with clear delivery semantics for server-side users.

🧭 Assumptions & unverified claims

No unverified assumptions or claims.

The event queue now behaves like a queue, including when the network develops opinions. · reviewed at 073aec5

[gagantrivedi]

@coderabbitai review