Repository navigation
Closed-loop --search-duration for OpenSearch vector and hybrid (Redis + OpenSearch) - #405
Conversation
… engines `--search-duration` (closed loop) now runs the vector workload on Redis, OpenSearch and Vertex, and the hybrid (filter + KNN) workload on Redis and OpenSearch, through one shared schedule: the duration_mode `Dispatcher` already used by fulltext `compound.duration_s`. Dispatch k (across all workers) sends query k mod n of the set (n = the set, or its first --queries), the deadline is checked before each dispatch, in-flight queries complete and are counted. Every completed query, repeats included, is scored against its ground truth and captured by --dump-results. Each duration point records `results.duration` (duration_mode::cycle_record): passes, dispatched/completed, completed by/after the deadline, qps_by_deadline, latency p50/p90/p99, mean recall, first-pass recall and the 10 s / 60 s intervals. Hybrid applies the duration per selectivity bucket; each bucket report carries its own `duration` record and `results.duration` sums them. Without the flag no output changes (the new keys are absent). Also: --target-qps (open loop) stays Redis/Vertex only; a duration on the filter-only workload or with --update-search-ratio is refused at startup instead of silently running once; the macOS mock-server tests read their accepted sockets blocking (BSD inherits O_NONBLOCK from the listener). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
🤖 Automated first-pass review — a human maintainer's review is still required before merge. I read the full diff against base
Worth fixing before merge (small)
Optional / follow-up
The memory/dump-size and cache-warming caveats in the description are the right ones to call out, and I don't see anything to add there. Net: a careful generalisation with the right refusals and matched cross-engine assertions. The only thing I'd change before merge is the off-by-the-tail timeout boundary in (1); the accept call is the maintainer's. |
Docker Build ValidationPlatforms tested:
Tests performed:
|
… timeout Review of #405: - The startup checks read `sp.duration_seconds.or(cli)` but the search stage lets --search-duration override the config; a config `duration_seconds: 0` plus the flag on a filter-only config slipped through. All startup checks in that block (duration and target_qps, incl. the existing fulltext, hybrid-rrf and hybrid ones) now use the flag first, as the run does. - check_duration_supported also checks the engine, per search config, so a config-file `duration_seconds` on an engine without the mode (e.g. qdrant), which the CLI-only engine gate never saw, is refused instead of running once under that label. - Hybrid: --search-timeout shorter than buckets x duration is refused at startup (the watchdog covers the whole 4-bucket point). - The hybrid run-level duration summary also sums failed and completed_after_deadline; the shared integration assertion checks both. - integration_cli: the refusals fire before any download or connection (filter-only, mixed, qdrant config duration, hybrid timeout). Mutations: reverting the precedence turns it red; dumping idx instead of seq on OpenSearch turns the OpenSearch duration test red. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Addressed the automated review in 5f78fce ( "Duration gates: CLI-first precedence, per-config engine check, hybrid timeout"):
Re-ran: fmt, clippy, lib/bins release, integration_cli/overhead/harness/streaming, and the Redis + OpenSearch duration/hybrid integration tests. 🤖 Generated with Claude Code |
Docker Build ValidationPlatforms tested:
Tests performed:
|
Why
The runbook for the 1M vector campaign (blocker B1) needs 600 s throughput points at 1 and 100 clients, run the same way on Redis and OpenSearch. On master,
--search-durationis honoured only by Redis and Vertex (OpenSearch is refused, so its 1,677 MS MARCO 1M queries run once, 1-2 s at 100 clients), and the hybrid workload refuses both--search-durationand--querieson every engine.What
One engine-neutral closed-loop schedule, reused from fulltext
compound.duration_s(#401): theduration_mode::Dispatcher.k mod nof the set.nis the whole query set, or its first--queries N. The deadline is checked before each dispatch, queries in flight at the deadline complete and are counted, and the measured window starts at the shared post-prime gate (as before). Redis, OpenSearch and Vertex all drive their vector loop throughengine::CycleMode, so two engines that ran the same set for the same time sent the same query sequence.--dump-results(one record per dispatch,seqgapless,query_id = seq mod n). Failures are still counted infailed_queries/ refused by--fail-on-dropped-queries, as in a single pass; they do not stop dispatch.results.duration(duration_mode::cycle_record) holdsmode: cycle-query-set,duration_s,queries,dispatched,completed,failed,full_passes,partial_position,repeated,completed_by_deadline/_after_deadline,qps_by_deadline,timed_span_s,latency.{mean,p50,p90,p99}_time,mean_recall(all executed queries),first_pass.{completed,complete,mean_recall}, and 10 s / 60 s intervals. The timing keys share one helper with the fulltextcycle-windowrecord, whose output is unchanged.results.hybrid.buckets[]gets adurationrecord (serialized only when present), andresults.durationsums them.--queriesstays refused for hybrid (it would apply per bucket), and so does--target-qps.--target-qps(open loop) stays Redis/Vertex only. Closed-loop duration is now allowed on Redis, OpenSearch and Vertex. A duration on the filter-only workload, or together with--update-search-ratio, is refused at startup: both used to run once and publish the result under a duration they did not run (Redis silently ignored it).--queries. It now cycles the first N queries.results.durationandbuckets[].durationare absent, and the default/open-loop dispatch paths are untouched.Unrelated test fix: the OpenSearch mock-server unit tests (
opensearch_retry_tests,opensearch_maintenance::tests) failed intermittently on macOS. An accepted socket inheritsO_NONBLOCKfrom the non-blocking listener on BSD (not on Linux), so the read raced the client. They now read the accepted socket blocking, bounded by the existing timeout.600 s points
--repetitionsis forced to 1 in duration mode (unchanged).Tests
duration_mode(tally/first-pass recall, misaligned samples refused, pass counts, null stats when nothing completed),hybrid::check_params,workload::{supports_search_duration, check_duration_supported},run_hybrid_buckets(per-bucket records, summary, mixed timed/untimed refused, single pass serializes noduration), the result JSON (results.durationonly when present).common::assert_vector_duration_point,assert_hybrid_duration_point): Redis and OpenSearch vector (2 s,--queries 5, strict cycled order in the dump, ≥2 full passes, deadline accounting, intervals, recall agreement) and hybrid (1 s per bucket, all 4 buckets, ≥2 passes each, gapless dump).test_binary_redis_closed_loop_duration_cycles_the_query_setred ("cycled order").-D warnings,--lib --bins --release, integration_cli/overhead/harness/streaming suites, fullintegration_redis(83 passed) and fullintegration_opensearch(48 passed).Limitations
--dump-resultsone captured record per dispatch. At ~20k QPS × 600 s that is several hundred MB, and gigabytes of dump.first_pass.mean_recallandrepeatedare recorded so this is visible.rps/ bucketrpsare completed / timed span (which includes the in-flight tail). Useqps_by_deadlinefor the exact-deadline figure.🤖 Generated with Claude Code