Is it good to store user-created data in another collection than system-data?

[Loahrs]

Hey everyone!

I am currently planning on creating a RAG application using Langchain4J and I would like to get some input. The system will have two kinds of data:

1. Data provided by the system. This can be a a manual or other documents. It is data that will be cleaned up, split into chunks and ingested into qdrant. It might be a good idea to prepare all this data during development (or through some pipeline).

2. Data created by the user of the application. Users are able to create articles (e.g FAQ-texts) which we want to also add into our RAG data. If a user deletes such an article, we can delete it from qdrant, modify it etc.

Now when there is an update to the software, there might also be a new version of the manual and related data, which we will have to update then. (Maybe by delivering a new qdrant-snapshot file to replace the old collection or having some scripts to replace the data in the collection).

I thought about creating two collections: one for the users data and one for the systems data. The FAQ says that having multiple collections might be considered an antipattern: https://qdrant.tech/documentation/faq/qdrant-fundamentals/#how-many-collections-can-i-create

So I would like to get some input if that is a terrible idea or a valid plan for my use case. Are there problems with delivering qdrant-snapshots for data updates?

[GioiaZheng]

The "many collections is an antipattern" note in the docs targets per-tenant / per-user collections at scale (hundreds–thousands of them), not two logically distinct corpora. Splitting into a system collection and a user collection is fine.

Two reasonable options:

A — single collection + payload filter. Keep everything in one collection, tag points with source: "system" | "user", add a payload index on that field, and filter at query time. This is the pattern Qdrant recommends for multitenancy. Prefer it when both data types share the same embedding model and dimensionality.

B — two collections. Justified when the two sets have different lifecycles, and yours do: system data is shipped/replaced wholesale on app updates, user data must survive untouched. Keeping them separate lets you snapshot and replace the system corpus independently.

For shipping new system-data versions: snapshots are per-collection, so option B fits your delivery plan directly. Combine it with collection aliases for zero-downtime swaps — load the new snapshot into system_v2, then move the system alias from system_v1 to system_v2 atomically and drop the old one. Your query code always targets the alias, never the versioned name.

So given your update-via-snapshot requirement, two collections + aliases is the cleaner choice here; reach for the single-collection payload-filter approach only if you'd rather avoid managing two collections and the lifecycles converge.

[RossDmello2]

I would decide this from lifecycle boundaries first, then from Qdrant collection count.

For your case, two collections is a reasonable design because the two corpora have different ownership and update semantics:

• system/manual data is versioned with the application and may be replaced as a unit;
• user-created FAQ/articles must survive app upgrades and need per-item CRUD;
• system snapshots should not accidentally overwrite user content.

The anti-pattern is usually one collection per user/project/document at high scale. Two stable collections for two lifecycles is not the same problem.

A practical setup:

1. Use the same embedding model and vector size for both collections if you want to merge results.
2. Query both collections in parallel.
3. Normalize/merge the top results in the application layer.
4. Keep metadata like source = system|user, version, tenant_id, document_id, and updated_at in payloads.
5. For user data, delete/update by stable point IDs or payload filters tied to the source document.
6. For system data, load a new version into a new collection or snapshot, smoke-test it, then switch the application config/alias to the new collection.

If you keep everything in one collection, use payload fields such as source and version and filter during updates. That is simpler for retrieval, but app upgrades become riskier because a bulk replace script must be very careful not to touch user points.

So my default recommendation for this exact scenario would be two collections: knowledge_system_vN and knowledge_user, with the application querying both. It keeps deployment rollback and user-data safety much cleaner.