Skip to content

feat(core): add Voyage AI embedding and reranker providers - #2116

Open
fzowl wants to merge 1 commit into
basicmachines-co:mainfrom
fzowl:feat/voyage-ai-integration
Open

fzowl wants to merge 1 commit into
basicmachines-co:mainfrom
fzowl:feat/voyage-ai-integration

Conversation

@fzowl

@fzowl fzowl commented Nov 14, 2026

Copy link
Copy Markdown

Basic Memory can embed notes and rerank search results through LiteLLM, but Voyage AI has no provider of its own: users need LiteLLM's model strings, per-model dimension settings, and no check that a model name is a real Voyage model.

This adds first-class voyage providers built on the official voyageai SDK, shipped as the optional basic-memory[voyage] extra:

  • Embeddings (semantic_embedding_provider=voyage): voyage-4-large, voyage-4, voyage-4-lite, voyage-code-4, voyage-3-large, voyage-3.5, voyage-3.5-lite, voyage-code-3, voyage-finance-2, voyage-law-2, plus the contextualized voyage-context-4 and voyage-context-3. Matryoshka models accept 2048/1024/512/256; the dimension is requested from Voyage only when semantic_embedding_dimensions is set. Chunks are embedded as document, queries as query.
  • Reranking (reranker_provider=voyage): rerank-3, rerank-3-lite, rerank-2.5, rerank-2.5-lite.
  • No default model. The model must be named; anything else (including the bge-small-en-v1.5 config default), an unsupported dimension, or a batch size outside 1–1000 fails with an error that lists the valid values. Other providers keep their defaults.
  • Requests are packed by the model's per-request token cap (1M -lite, 320K voyage-4/voyage-3.5, 120K the rest) using the SDK tokenizer, and by the 1000-input limit; if the tokenizer cannot be loaded, only the input limit applies.
  • API errors report the text of Voyage's {"detail": ...} body. VOYAGE_API_KEY (or the existing key setting) supplies the credential.
  • Contextualized models send flat one-string documents with enable_auto_chunking=true and chunk_size=32000 for indexed chunks, and no auto-chunking for queries. Basic Memory hands providers flat chunks already mixed across notes, so each chunk is its own document and gets no cross-chunk context; this is documented.

Docs: docs/semantic-search.md and the README rerank section.

Tested with new unit tests for the failure modes (bad or missing model, dimensions, batch size, API-imposed contextualized constraints, malformed or failed rerank responses; 100% coverage of the new modules), and live through bm reindex and bm tool search --mode semantic with every embedding model (native, 256 and 2048 dimensions where supported) and every rerank model. just test-voyage-live runs an opt-in live check for a plain and a contextualized model.

@CLAassistant

CLAassistant commented Nov 14, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-14T14:20:56.820126Z 60513c8 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 60513c8c4b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

client = self._get_client()
try:
encodings = await asyncio.to_thread(client.tokenize, texts, self.model_name)
except Exception as exc:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Restrict tokenizer fallback to expected load failures

When client.tokenize raises an unrelated programming or SDK-contract error rather than the expected offline tokenizer-download failure, this blanket handler permanently disables token-aware packing and proceeds with count-only batches, which can then exceed the model's token ceiling while hiding the original defect. Catch only the specific load/network failures intended for this fallback and let unexpected exceptions propagate.

AGENTS.md reference: AGENTS.md:L133-L134

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 95ad293: the fallback now catches only OSError (every Hugging Face download failure is one) and ImportError (missing tokenizers); any other exception propagates. test_unexpected_tokenizer_errors_are_not_swallowed covers it.

@fzowl
fzowl force-pushed the feat/voyage-ai-integration branch from 60513c8 to 95ad293 Compare November 14, 2026 14:21
@Flynn-Kawolsky

Copy link
Copy Markdown

Thanks for this. I've been running voyage-4-large + rerank-3 through the LiteLLM path on a mostly Traditional Chinese vault, so a first-class provider is welcome.

One data point on why it matters for non-English users. On my own corpus (574 notes, 1,126 generated questions, hybrid search, same reranker), switching embeddings from embeddinggemma-300m to voyage-4-large moved HR@5 from 85.0% to 92.2%. Almost all of the gain was on paraphrased queries without identifiers (69.9% → 88.2%). Queries with an identifier embedded in a long sentence were slightly worse (99.5% → 96.0%). The corpus is private, so I can only share aggregates.

On contextualized models: the description says each chunk is sent as its own document. That is the gap I described in #1999. Without grouping chunks per note, voyage-context-4 gets no cross-chunk context. Would it make sense to link #1999 here and treat per-note grouping as a follow-up?

If per-note grouping lands, I can rerun the same question set against it and report numbers.

@fzowl

fzowl commented Nov 15, 2026

Copy link
Copy Markdown
Author

Thanks, and thanks for the benchmark numbers.

You're right. In this PR voyage-context-4 and voyage-context-3 are supported as models of the voyage provider, but each indexed chunk is sent as its own one-string document. Basic Memory's embedding contract is a flat embed_documents(list[str]), and the vector sync flushes chunks from several notes in one batch. So these models work end to end (the live runs and just test-voyage-live cover them), but they get no cross-chunk context from the note. That limit is documented in docs/semantic-search.md.

I kept the PR to what the current provider contract allows. Embedding all chunks of one note together would mean extending that contract and grouping chunks per note in the vector sync. That is a larger change to the indexing path, so it is not part of this PR. It is the gap described in #1999, which is the right place to track it.

@tonydzi tonydzi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read both providers and the config validation in full (head 95ad293). The embedding side is careful: I checked contextualized_embed against voyageai 0.6.0 (flat List[str] with enable_auto_chunking=True is normalised to one-string documents, so the call is valid) and the tokenizer fallback (from_pretrained failures are OSError subclasses with current tokenizers/huggingface_hub, so except (OSError, ImportError) holds). One gap on the rerank side.

reranker_provider=voyage has no bound on the pool size, so a larger reranker_candidates fails at search time.

  • config_models.py:472: reranker_candidates is only gt=0.
  • config_models.py:1030-1034: the voyage branch validates the model name but not the pool size.
  • voyage_rerank_provider.py:73: client.rerank(query, documents, ...) sends the whole pool in a single request. search_reader.py:1227 and :1657 take up to rerank.candidates rows, so the pool equals reranker_candidates.

Voyage's reranker docs state "The number of documents cannot exceed 1,000", and a total-token cap of 600K for all four supported models, where total = query tokens x number of documents + all document tokens. So reranker_candidates: 1500 is accepted at config load, then every reranked search gets a 4xx from Voyage. That lands in the final except errors.VoyageError branch (line 86) and is raised as SemanticDependenciesMissingError, i.e. reported as a missing dependency or credentials problem. The default of 20 is nowhere near this, but a user widening recall (the field description invites it: "Larger widens recall") hits it, and the 600K product term also binds well below 1000 docs for a long query.

This PR already does the right thing on the embedding side (batch size 1-1000 rejected at construction with the valid range listed, voyage_provider.py:190). Suggested fix, in the same spirit:

  1. In the voyage branch of the reranker validator, reject reranker_candidates > 1000 with the valid range in the message.
  2. Optionally chunk documents in rerank() (score each slice, concatenate in input order) or cap by len(query_tokens) * n + doc_tokens <= 600_000, if you want large pools to work instead of erroring.
  3. A test beside the existing config tests: reranker_provider="voyage", reranker_candidates=1001 raises ValueError.

Caveat: I did not call the live Voyage API (no key here) and did not run your test suite; the 1,000 and 600K figures are from Voyage's published reranker docs, the code paths are from reading the diff.

Add a voyage embedding provider (voyage-4*, voyage-3*, voyage-code-*,
voyage-finance-2, voyage-law-2 and the voyage-context-* models) and a voyage
reranker provider (rerank-3, rerank-3-lite, rerank-2.5, rerank-2.5-lite) built
on the official voyageai SDK, shipped as the optional basic-memory[voyage]
extra.

Neither provider has a default model: the model must be named explicitly and
is validated against the Voyage catalog, along with the supported output
dimensions. API errors surface the text of Voyage's detail field.

Signed-off-by: fzowl <zoltan@voyageai.com>
@fzowl

fzowl commented Nov 17, 2026

Copy link
Copy Markdown
Author

Thanks. Fixed in 8a0ae1b: with reranker_provider=voyage, reranker_candidates above 1000 is rejected at config load with the limit in the message (config_models.py), and a test covers 1000 (accepted) and 1001 (rejected). I did not add slicing or a token-product check: the docs now state the 1000-document limit and the 600K total-token cap, and a request over the token cap still comes back from Voyage with its detail text.

@fzowl
fzowl force-pushed the feat/voyage-ai-integration branch from 95ad293 to 8a0ae1b Compare November 17, 2026 03:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants