Qdrant BM25 average document length

[Topo1995]

Hi,

I’ve been reading through the documentation on setting up a collection for BM25 text search: https://qdrant.tech/documentation/search/text-search/
and I have a couple of questions about the average document length parameter.

1) Why does average document length need to be set for each individual point during upsert?
My understanding is that this value is a collection-level statistic, so every point in the collection should share the same average document length. Wouldn’t it make more sense to define it once at the collection level rather than including it with each point? Is there an option for that?

2) How should updates be handled if the average document length changes significantly over time?
If points are frequently added, removed, or modified, the true average document length of the collection could change substantially. In that case, is it necessary to re-upsert all existing points with the updated value, or is there a mechanism to update this statistic globally without reingesting all data?

Thank you for your time and clarification.

[generall]

Why does average document length need to be set for each individual point during upsert?

The main reason is, that once document is converted into sparse vector representation, document length doesn't have much sense anymore. Document length is not equal to number of elements in sparse vector, as there are models which generate more than one element per token (see SPLADE and MiniCOIL). Document length only exist and text domain, and qdrant doesn't operate with texts directly.

How should updates be handled if the average document length changes significantly over time?

The assumption is, that document length depends on the chunker settings, and even if you change documents, each chunk would stay about the same size. If this is not true for your case, it is unfortunately only possible to change by re-uploading dataset.

[Topo1995]

Thank you generall for your response. I am a confused about the reasoning of your first answer part. If you say document length doesnt have much sense anymore, why would it then be required to set it for each individual point?

[generall]

If you say document length doesnt have much sense anymore

I mean that from Qdrant storage point of view it is not possible to extract document length, as qdrant doesn't see text of the document

[Topo1995]

But we are not storing the document length of the individual point, right? We are storing the average length over all documents.
Is it a mere design decision to have the average document length for each point so that when retrieval is done, they contain the information needed for BM25?

[ibondarenko1]

Adding the operational layer on top of @generall's explanation — what to actually compute and how to keep it healthy in production.

1. Pre-compute avg_len once from your corpus, store it next to your chunker config.

# 1-pass over your corpus; avg_len in the same units the indexer counts
# (tokens AFTER your chunker / tokenizer step, not raw chars)
from statistics import mean

def chunk_token_lengths(corpus, chunker):
    for doc in corpus:
        for chunk in chunker(doc):
            yield len(chunk.tokens)   # whatever your tokenizer/sparse-encoder uses

AVG_LEN = round(mean(chunk_token_lengths(corpus, my_chunker)), 2)
# pin AVG_LEN alongside chunker_version, model_version in your config

Storing it next to chunker_version matters because the moment you change chunker (different overlap, different splitter), avg_len is stale even if the documents are unchanged.

2. Send avg_len per-point at upsert exactly as the maintainer described — but generate it from one constant.

Many users put a per-point computation here, which fights the design. The expected pattern is:

points.append(PointStruct(
    id=chunk_id,
    vector={"text-sparse": sparse_vec},
    payload={"text": chunk_text, "doc_id": doc_id, "avg_len": AVG_LEN},  # same constant for every point in this corpus
))

If you have heterogeneous corpora (e.g., short titles + long bodies in the same collection), give each a separate avg_len constant per source — but it's still a constant per source, not per point.

3. Drift detection — alert before re-uploading.

The maintainer's point about "chunks stay about the same size" holds only as long as your chunker and content distribution don't shift. Watch for drift cheaply:

# Run weekly against a fresh sample of newly upserted chunks
sample = client.scroll("my_collection", limit=10_000, with_payload=["text"]).points
current_avg = mean(len(my_chunker_tokenize(p.payload["text"])) for p in sample)
drift_pct = abs(current_avg - AVG_LEN) / AVG_LEN
# alert when drift > 0.2 — empirically the point where BM25 ranking starts to noticeably miscalibrate

What "wrong" avg_len does to BM25: the length-normalisation term is (1 - b + b * dl/avg_len) (default b=0.75). If your declared avg_len is 2x too small, long docs get over-penalised; if 2x too big, short docs get over-favoured. The retrieval still works — top-K usually shifts only a few positions on broad queries — but on short tail queries you can see relevance regress measurably.

4. Re-upload pattern that avoids downtime.

If drift triggers a re-upload, blue-green at the collection level beats in-place updates:

my_collection_v2  ← fresh upserts with new AVG_LEN
                    + the same vectors / payload as v1
my_collection     → after v2 is fully indexed, alias-swap or
                    flip your search code to point at v2;
                    drop v1 once smoke-tests pass

Doing it in-place via per-point payload updates also works (set_payload(... ["avg_len": NEW_AVG])) — just expect a long-tail of HNSW work as updates land.

5. If your corpus is genuinely heterogeneous (titles + bodies + descriptions in one collection), split by source.

In a single collection where chunk length actually varies by an order of magnitude per source, BM25's single-avg_len assumption is the limiting factor — not the per-point upsert API. Either (a) split into sub-collections per source and search them in parallel, or (b) keep one collection but let the relevance live with the miscalibration on short-tail queries. The chunker assumption @generall mentioned is the design boundary.

Recipe: pin AVG_LEN to your chunker config → drift-alert weekly at 20% threshold → blue-green collection rebuild when the alert fires → split by source if your chunk length is bimodal.

[Topo1995]

Thank you, ibondarenko1, for taking the time to provide such a detailed explanation, it was very helpful. It also raised a few follow-up questions that I hope you can clarify if you have some spare time :D

1) How exactly should avg_len be calculated?

yield len(chunk.tokens) # whatever your tokenizer / sparse-encoder uses

In Qdrant’s text search documentation, avg_len is described as the average number of words per chunk text. Because of that, I’m not sure whether I should literally count whitespace-separated words, or whether the correct approach is to use the tokenizer of the sparse embedding model itself, e.g.:

SparseTextEmbedding(model_name="Qdrant/bm25").token_count(chunk_text)

Would that be the proper way to compute avg_len? Also, does language matter here in any way (for tokenization / stemming / normalization), or is the BM25 tokenizer model-agnostic?

2) Clarification on “separate avg_len constant per source”

You mentioned:

If you have heterogeneous corpora (e.g., short titles + long bodies in the same collection), give each a separate avg_len constant per source — but it's still a constant per source, not per point.

I’m not sure I fully understand what “per source” means in practice.

Do you mean either of the following Options?

• Option A: calculate one average length for all title chunks, another for all body chunks, and store the corresponding constant in payload for each point — even though they all live in the same collection and populate the same sparse vector field?
• Option B: titles and bodies are stored as separate fields within a point, each with its own associated avg_len?

3) Clarification on splitting heterogeneous corpora

You also wrote:

If your corpus is genuinely heterogeneous (titles + bodies + descriptions in one collection), split by source.

Does that mean that, for BM25 length normalization to be accurate, I would need to create separate collections for titles, bodies, descriptions, etc., so that each collection can have its own avg_len?

Right now, my setup uses one sparse vector field, where I combine titles / bodies / descriptions / other text before chunking. That produces significantly different chunk lengths. In that case, is the correct takeaway:

If I want BM25 normalization to be as accurate as possible, I should separate these into distinct collections (or otherwise separate indexes), so each can use a more representative avg_len?

4) Updating average document length

I kinda hope I am mistaken, but I don´t think that the avg_len can be updated in-place as this is not passed to the payload but inside the actual vector definition.

Relevant part taken from https://qdrant.tech/course/essentials/day-3/sparse-retrieval-demo/

                  payload={"text": description},
                  vector={
                         "bm25_sparse_vector": models.Document(
                                text=description,
                                model="Qdrant/bm25",
                               options={"avg_len": avg_document_length} 

Please correct me if it is still updateable in-place :D

Thank you a lot for your time!

[ibondarenko1]

Glad it was useful, and you caught a real error on (4) — getting to that one last.

(1) avg_len units. Qdrant's "average number of words" phrasing is a bit loose. What BM25's length-normalization actually wants is the count from the model's tokenizer. For Qdrant/bm25 via fastembed, the tokenizer does lowercase + whitespace + punctuation split, then stemming runs separately downstream. So avg_len should match the post-tokenize, pre-stem token count, not the count of stemmed unique tokens you'd see if you read len(sparse.indices).

In practice for English, len(text.split()) is within ~5% of what the tokenizer counts and most teams just use that. If you want it exact:

from fastembed import SparseTextEmbedding
m = SparseTextEmbedding(model_name="Qdrant/bm25")
length = len(m.model._tokenize(text))   # private but stable

token_count(text) isn't a public API on SparseTextEmbedding — internally it's the same _tokenize path.

On language: it matters for the stemmer, not the tokenizer. fastembed's BM25 has a language parameter ('english' default; 'french', 'german', etc. via Snowball). The tokenizer itself is whitespace+punctuation regardless of language, which is fine for European languages but not for Chinese/Japanese where you'd want to pre-tokenize externally or use a different sparse model.

(2) Per-source — Option B is the correct one. Sorry, my original wording was sloppy. Option A (same sparse field, different avg_len constants per point) technically goes through at upsert, but it breaks score comparability — BM25 normalization is supposed to use a SINGLE avgdl across the searchable space, and if half the points were normalized against avg_titles=8 and the other half against avg_bodies=200, the scores from those two halves aren't meaningfully comparable in the same query response.

Option B is what I should have written:

client.create_collection(
    collection_name="...",
    sparse_vectors_config={
        "title-bm25": models.SparseVectorParams(...),
        "body-bm25":  models.SparseVectorParams(...),
    },
)

# Pre-compute corpus-level averages once
AVG_TITLES = 8.4
AVG_BODIES = 220.7

points.append(PointStruct(
    id=...,
    vector={
        "title-bm25": models.Document(text=title, model="Qdrant/bm25",
                                       options={"avg_len": AVG_TITLES}),
        "body-bm25":  models.Document(text=body,  model="Qdrant/bm25",
                                       options={"avg_len": AVG_BODIES}),
    },
))

Each named sparse vector keeps its own consistent avgdl, so scores within each source are comparable. At query time you either route to one source or fuse across sources with RRF (next point).

(3) Same collection, named vectors per source — not separate collections. Separate collections force separate query calls and you can't fuse the results in one request. Same collection with named sparse vectors lets you do prefetch + RRF in a single query_points call:

client.query_points(
    collection_name="...",
    prefetch=[
        Prefetch(
            query=Document(text=q, model="Qdrant/bm25",
                           options={"avg_len": AVG_TITLES}),
            using="title-bm25", limit=20),
        Prefetch(
            query=Document(text=q, model="Qdrant/bm25",
                           options={"avg_len": AVG_BODIES}),
            using="body-bm25", limit=20),
    ],
    query=FusionQuery(fusion=Fusion.RRF),
    limit=10,
)

The deeper issue with your current "concatenate titles+bodies+descriptions then chunk" setup isn't really the avg_len — it's that those chunks are semantically heterogeneous, so even with one avg_len that fits everything, the BM25 scoring is inconsistent across chunk types. Chunking per source first, then upserting to per-source named vectors, fixes both the length-normalization problem and the scoring-consistency problem at once.

(4) You're right, my advice on this was wrong. avg_len is part of the sparse vector definition through models.Document(..., options={'avg_len': ...}) and gets baked into the BM25 indices/values when the sparse vector is computed at upsert time. It's not a payload field — set_payload does nothing to it. To "update" avg_len for an existing point you have to re-compute the sparse vector with the new avg_len and call client.update_vectors(...) or re-upsert. There's no cheap in-place path the way I implied.

So the practical cost when drift detection triggers a global avg_len change is "re-vectorize and upsert all affected points", not "set_payload on each one". That makes blue-green collection rebuild even more clearly the right pattern — write everything to a _v2 collection with the new avg_len, swap your alias when indexing finishes, drop _v1. The single-instruction "set_payload" path I described in the original answer doesn't actually exist for avg_len. Apologies for the misdirection there.