Skip to content

Closed-loop --search-duration for OpenSearch vector and hybrid (Redis + OpenSearch) - #405

Merged
fcostaoliveira merged 2 commits into
masterfrom
feat/duration-mode-vector-hybrid
Nov 13, 2026
Merged

fcostaoliveira merged 2 commits into
masterfrom
feat/duration-mode-vector-hybrid

Conversation

@fcostaoliveira

Copy link
Copy Markdown
Collaborator

Why

The runbook for the 1M vector campaign (blocker B1) needs 600 s throughput points at 1 and 100 clients, run the same way on Redis and OpenSearch. On master, --search-duration is honoured only by Redis and Vertex (OpenSearch is refused, so its 1,677 MS MARCO 1M queries run once, 1-2 s at 100 clients), and the hybrid workload refuses both --search-duration and --queries on every engine.

What

One engine-neutral closed-loop schedule, reused from fulltext compound.duration_s (#401): the duration_mode::Dispatcher.

  • Schedule. Dispatch k (0-based, across all workers) sends query k mod n of the set. n is the whole query set, or its first --queries N. The deadline is checked before each dispatch, queries in flight at the deadline complete and are counted, and the measured window starts at the shared post-prime gate (as before). Redis, OpenSearch and Vertex all drive their vector loop through engine::CycleMode, so two engines that ran the same set for the same time sent the same query sequence.
  • Scoring. Every completed query, repeats included, is scored against its own ground truth and captured by --dump-results (one record per dispatch, seq gapless, query_id = seq mod n). Failures are still counted in failed_queries / refused by --fail-on-dropped-queries, as in a single pass; they do not stop dispatch.
  • Record. results.duration (duration_mode::cycle_record) holds mode: cycle-query-set, duration_s, queries, dispatched, completed, failed, full_passes, partial_position, repeated, completed_by_deadline / _after_deadline, qps_by_deadline, timed_span_s, latency.{mean,p50,p90,p99}_time, mean_recall (all executed queries), first_pass.{completed,complete,mean_recall}, and 10 s / 60 s intervals. The timing keys share one helper with the fulltext cycle-window record, whose output is unchanged.
  • Hybrid. The duration applies per selectivity bucket: each bucket's pass cycles that bucket's queries for the full duration, so a hybrid point lasts 4 × duration. Each results.hybrid.buckets[] gets a duration record (serialized only when present), and results.duration sums them. --queries stays refused for hybrid (it would apply per bucket), and so does --target-qps.
  • Gates. --target-qps (open loop) stays Redis/Vertex only. Closed-loop duration is now allowed on Redis, OpenSearch and Vertex. A duration on the filter-only workload, or together with --update-search-ratio, is refused at startup: both used to run once and publish the result under a duration they did not run (Redis silently ignored it).
  • Behaviour change with the flag only: Redis/Vertex duration mode used to ignore --queries. It now cycles the first N queries.
  • Without the flag nothing changes: results.duration and buckets[].duration are absent, and the default/open-loop dispatch paths are untouched.

Unrelated test fix: the OpenSearch mock-server unit tests (opensearch_retry_tests, opensearch_maintenance::tests) failed intermittently on macOS. An accepted socket inherits O_NONBLOCK from the non-blocking listener on BSD (not on Linux), so the read raced the client. They now read the accepted socket blocking, bounded by the existing timeout.

600 s points

# vector, 1 and 100 clients (parallel comes from the search config)
vector-db-benchmark --engines <cfg> --datasets msmarco-cohere-1024-1M-cosine \
  --skip-upload --keep-data --search-duration 600 [--warmup-seconds 60] \
  --dump-results <dir> --fail-on-dropped-queries
# hybrid: 600 s per bucket (4 buckets => ~40 min per point)
vector-db-benchmark --engines <hybrid-cfg> --datasets msmarco-cohere-1024-1M-cosine \
  --skip-upload --keep-data --search-duration 600 --dump-results <dir> --fail-on-dropped-queries

--repetitions is forced to 1 in duration mode (unchanged).

Tests

  • Unit: duration_mode (tally/first-pass recall, misaligned samples refused, pass counts, null stats when nothing completed), hybrid::check_params, workload::{supports_search_duration, check_duration_supported}, run_hybrid_buckets (per-bucket records, summary, mixed timed/untimed refused, single pass serializes no duration), the result JSON (results.duration only when present).
  • Integration, identical assertions for both engines (common::assert_vector_duration_point, assert_hybrid_duration_point): Redis and OpenSearch vector (2 s, --queries 5, strict cycled order in the dump, ≥2 full passes, deadline accounting, intervals, recall agreement) and hybrid (1 s per bucket, all 4 buckets, ≥2 passes each, gapless dump).
  • Guard mutation: making Redis duration mode always send query 0 turns test_binary_redis_closed_loop_duration_cycles_the_query_set red ("cycled order").
  • Local: fmt, clippy -D warnings, --lib --bins --release, integration_cli/overhead/harness/streaming suites, full integration_redis (83 passed) and full integration_opensearch (48 passed).

Limitations

  • Memory grows with the run: per completed query it keeps the existing latency/precision/recall/MRR/NDCG samples plus an 8-byte completion time, and under --dump-results one captured record per dispatch. At ~20k QPS × 600 s that is several hundred MB, and gigabytes of dump.
  • Repeats can be served from caches warmed by the first pass. first_pass.mean_recall and repeated are recorded so this is visible.
  • Hybrid rps / bucket rps are completed / timed span (which includes the in-flight tail). Use qps_by_deadline for the exact-deadline figure.
  • Vertex shares the code path but was not integration-tested (cloud-only).

🤖 Generated with Claude Code

… engines

`--search-duration` (closed loop) now runs the vector workload on Redis,
OpenSearch and Vertex, and the hybrid (filter + KNN) workload on Redis and
OpenSearch, through one shared schedule: the duration_mode `Dispatcher`
already used by fulltext `compound.duration_s`. Dispatch k (across all
workers) sends query k mod n of the set (n = the set, or its first
--queries), the deadline is checked before each dispatch, in-flight queries
complete and are counted. Every completed query, repeats included, is
scored against its ground truth and captured by --dump-results.

Each duration point records `results.duration` (duration_mode::cycle_record):
passes, dispatched/completed, completed by/after the deadline,
qps_by_deadline, latency p50/p90/p99, mean recall, first-pass recall and the
10 s / 60 s intervals. Hybrid applies the duration per selectivity bucket;
each bucket report carries its own `duration` record and `results.duration`
sums them. Without the flag no output changes (the new keys are absent).

Also: --target-qps (open loop) stays Redis/Vertex only; a duration on the
filter-only workload or with --update-search-ratio is refused at startup
instead of silently running once; the macOS mock-server tests read their
accepted sockets blocking (BSD inherits O_NONBLOCK from the listener).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

🤖 Automated first-pass review — a human maintainer's review is still required before merge.

I read the full diff against base src/duration_mode.rs (Dispatcher::next / dispatched()) and the warm-up block in experiment.rs. Overall this is the right shape. Redis, OpenSearch and Vertex now share one dispatcher (CycleMode → Dispatcher), so two engines that run the same set for the same time send the same k mod n sequence. That matters for the B1 Redis-vs-OpenSearch comparison. The second commit already reads like a review round. It closes the gaps I would have raised first:

cycle_record refuses misaligned tally/latency/recall lengths instead of quietly producing skewed stats. Shared assert_vector_duration_point / assert_hybrid_duration_point run the same assertions on both engines. The seq-vs-idx mutation on OpenSearch goes red. All good.

Worth fixing before merge (small)

  1. src/hybrid.rs check_search_timeout: the guard is search_timeout < needed with needed = 4 × d. So --search-duration 600 --search-timeout 2400 passes the check. But the watchdog covers the whole point, and that includes per-bucket priming, the in-flight tail after each deadline and profiling. A 2400 s timeout therefore fires near the end of bucket 4, about 40 minutes in. The doc comment itself says "plus the tail, priming and profiling". I'd use <= plus some margin, or refuse anything under needed + Δ. The CLI test only covers 900, far from the boundary, so a test at exactly 4 × d would pin this.

Optional / follow-up

  1. The same watchdog-vs-duration mismatch exists on the vector arm, which has no check. --search-duration 600 --search-timeout 300 on a vector config is killed mid-run. It fails loudly, and Redis/Vertex already behaved this way, but this PR now extends the path to OpenSearch. check_search_timeout could take a bucket count, 1 for vector, and cover both.
  2. Warm-up on hybrid: the closed-loop warm-up at experiment.rs:2067 sets duration_seconds = warmup_seconds and calls search_workload with the hybrid workload. With this PR that warm-up also runs per bucket, so --warmup-seconds 60 costs about 240 s, and it starts at query 0 of each bucket, the same order the measured pass then uses. The runbook's hybrid command omits --warmup-seconds, so it's harmless as written. It's worth one sentence in the --warmup-seconds help, or a refusal, so nobody adds it expecting 60 s.
  3. Vertex now honours --queries in duration mode, which changes behaviour with the flag set. It is only exercised by mirroring Redis line-for-line (no integration test; it's cloud-only). That's fine for now, but a unit-level check that CycleMode::from_params(.., available, n).set_len == min(n, available) lives in src/ already (cycle_set_len). Vertex's wiring is the part with no coverage. Worth a note in the result docs for anyone comparing pre/post Vertex duration points.
  4. mean_recall includes repeats, so the trailing partial pass over-weights the head of the set by one sample. At a plausible 1-client OpenSearch rate (~200 QPS × 600 s ≈ 120k dispatches / 1677 ≈ 72 passes), that is under 1.5% relative weight on part of the set. It's negligible and already visible through first_pass.mean_recall. I'm mentioning it only so the field description doesn't get read as "unweighted per-query mean".

The memory/dump-size and cache-warming caveats in the description are the right ones to call out, and I don't see anything to add there.

Net: a careful generalisation with the right refusals and matched cross-engine assertions. The only thing I'd change before merge is the off-by-the-tail timeout boundary in (1); the accept call is the maintainer's.

@github-actions

Copy link
Copy Markdown

Docker Build Validation

Platforms tested:

  • linux/amd64 (built and tested)
  • linux/arm64 (build validated)

Tests performed:

  • --help command
  • --describe datasets command
  • Integration test (random-100 dataset + Redis)
  • Multi-platform build validation (amd64 + arm64)

… timeout

Review of #405:
- The startup checks read `sp.duration_seconds.or(cli)` but the search stage
  lets --search-duration override the config; a config `duration_seconds: 0`
  plus the flag on a filter-only config slipped through. All startup checks
  in that block (duration and target_qps, incl. the existing fulltext,
  hybrid-rrf and hybrid ones) now use the flag first, as the run does.
- check_duration_supported also checks the engine, per search config, so a
  config-file `duration_seconds` on an engine without the mode (e.g. qdrant),
  which the CLI-only engine gate never saw, is refused instead of running
  once under that label.
- Hybrid: --search-timeout shorter than buckets x duration is refused at
  startup (the watchdog covers the whole 4-bucket point).
- The hybrid run-level duration summary also sums failed and
  completed_after_deadline; the shared integration assertion checks both.
- integration_cli: the refusals fire before any download or connection
  (filter-only, mixed, qdrant config duration, hybrid timeout). Mutations:
  reverting the precedence turns it red; dumping idx instead of seq on
  OpenSearch turns the OpenSearch duration test red.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@fcostaoliveira

fcostaoliveira commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator Author

Addressed the automated review in 5f78fce ( "Duration gates: CLI-first precedence, per-config engine check, hybrid timeout"):

  1. Precedence. Every startup check in that block now uses cli.or(config), which is how the search stage resolves it (the flag overrides the config). That covers duration and target_qps, including the existing fulltext, hybrid-rrf and hybrid checks.
  2. Engine gate for config-file durations. check_duration_supported now also checks the engine for each search config, so a duration_seconds in a config file on qdrant/elasticsearch/... is refused at startup instead of running once under that label.
  3. Guard coverage. A new integration_cli test, duration_where_it_would_run_once_is_refused_before_download_or_connection, covers filter-only + flag, mixed + flag, a qdrant config duration_seconds, and a hybrid --search-timeout that is too short. Each must be refused with no Experiment stage:. Mutations: reverting the precedence turns this test red, and dumping idx instead of seq on OpenSearch turns test_binary_opensearch_closed_loop_duration_cycles_the_query_set red ("gapless dispatch order").
  4. Hybrid timeout. --search-timeout below buckets × duration is refused at startup (hybrid::check_search_timeout). The warm-up is also per bucket, as noted.
  5. Summary. The summary now also sums failed and completed_after_deadline, and the shared integration assertion checks completed_by + after == dispatched.

Re-ran: fmt, clippy, lib/bins release, integration_cli/overhead/harness/streaming, and the Redis + OpenSearch duration/hybrid integration tests.

🤖 Generated with Claude Code

@github-actions

Copy link
Copy Markdown

Docker Build Validation

Platforms tested:

  • linux/amd64 (built and tested)
  • linux/arm64 (build validated)

Tests performed:

  • --help command
  • --describe datasets command
  • Integration test (random-100 dataset + Redis)
  • Multi-platform build validation (amd64 + arm64)

@fcostaoliveira
fcostaoliveira merged commit 9d53e30 into master Nov 13, 2026
37 of 38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant