Repository navigation
Description
Problem
A note has one authoritative history: accepted versions in the database (note_content.db_version / db_checksum). Storage is a projection of it, written asynchronously (file_version / file_checksum). Changes also arrive from storage: WebDAV PUTs, bisync, local editors, and Cloud snapshot restores.
When the indexer observes a file, it has to decide what the observation means. Today NoteContentReconciler decides from content alone: if the observed bytes equal an older accepted version, the rollback guard assumes a stale read and refuses (with no authoritative re-read on object storage). That one rule mixes up two different situations:
| Situation | Example | Right outcome |
|---|---|---|
| Storage is behind the DB | an edit is accepted, materialization is pending, and an index pass reads the previous file | ignore: materialization will overwrite it |
| Someone deliberately wrote older content | snapshot restore, a WebDAV revert, a user pasting an old copy | accept as a new version |
Content can't tell these apart. Both look like "bytes equal to version N-1". Production from Nov 5 to 18 had 4 refusals: one was a lag case the guard correctly protected (an MCP edit followed 50 s later by an index of the previous file), and one was a Cloud snapshot restore the guard wrongly ignored (basicmachines-co/basic-memory-cloud#3222). Two came from WebDAV PUTs, and their intent is unknown.
Timestamps don't fix this. Clocks differ between writers, and object LastModified records arrival, not intent.
Model
Each note has:
- Accepted history: versions
1..db_version, each with a checksum. Only the accepted write path appends to it. Every write through it carries a base revision, the checksum it was based on (optimistic concurrency, already in place). - Materialization record: the accepted version last written to storage (
file_version), and the storage object identity that write produced (new: S3 ETag/version id, or mtime+size+inode locally).
A storage observation is a pair (object identity, bytes). The classification uses identity first, content second:
- Our own write (echo). The identity equals our recorded materialization. No new content: settle the materialization. Nothing to accept.
- Superseded observation. The identity is older than our recorded materialization, or the event's ETag no longer matches the object's current ETag (the existing early gate). Ignore it.
- External write, no pending DB change. An unknown identity and
file_version == db_version: storage held exactly what the DB had, and someone replaced it. The writer's base is our current version, so accept the bytes as a new version, even if they equal an older one. This covers snapshot restores, WebDAV reverts and local edits. - External write over a pending DB change. An unknown identity and
file_version < db_version: the DB accepted a change storage doesn't have yet, and someone wrote storage concurrently. This is a real conflict. Keep the accepted version and record the external bytes as a conflicted version for a person to resolve. Never overwrite either silently.
Content equality only names the case in logs ("revert to version N"). It never decides the outcome.
Invariants
- I1: Accepted history is append-only. Nothing in the indexer rewrites or rolls it back.
- I2: A write that names a base revision is accepted iff that base is current (API path, unchanged).
- I3: An observation of our own materialization never creates a version.
- I4: An external storage write is never dropped silently. It becomes either a new version (case 3) or a recorded conflict (case 4).
- I5: Storage converges to the DB: after accepting a version, materialization eventually writes it, and its identity is recorded.
What it needs
- Record the object identity returned by each materialization write (
note_content.file_etagor similar; local backend: mtime+size+inode). - Pass the observed identity from the storage event (Cloud already has
etagonStorageEventInput) into the reconciler. - Replace the content-based rollback guard with the classification above. Case 2 keeps the protection the guard gives today, keyed on identity instead.
- Tests for each case, including the production sequences above: edit then lagging index (case 1/2), snapshot restore (case 3), concurrent WebDAV write during pending materialization (case 4).
Relation to the Cloud fix
basicmachines-co/basic-memory-cloud#3222 makes Cloud snapshot restore write through the accepted API path with a base revision, so it no longer depends on this model. This issue covers every other storage-originated write.
Activity
claude commentedon Nov 18, 2026
Claude finished @phernandez's task in 18s —— View job
Triage
NoteContentReconciler,note_content(newfile_etag), and the materialization write path. I applied thecloudlabel becauseetagonStorageEventInputand the object-storage ETag are Cloud-specific.Notes for whoever picks this up (from reading the issue; I haven't investigated the code):
file_etagand classifying on it can stay lock-free, with stale writers no-op'ing on generation fences.This is triage only; I haven't started any implementation.
phernandez commentedon Nov 18, 2026
Context for whoever picks this up
Read first: this repo's
CLAUDE.md/AGENTS.md. Commits here need DCO sign-off (git commit -s). Cloud consumes core through a pinned version, so a core change reaches Cloud only when the pin moves (a separate Cloud PR).Code today
src/basic_memory/indexing/note_content_reconciler.py.NoteContentRollbackGuardOutcomeandrollback_guard_allows_promotion(around lines 163 to 178).guard_note_content_promotion(its docstring states the trade-off this issue removes).local_promotion_confirmation_reader(around line 147): object storage gets no authoritative re-read, so the guard defers.log_note_content_promotion(around line 229) asnote_content.promotion_refused.src/basic_memory/indexing/note_content_reconciliation.py.src/basic_memory/indexing/storage_note_changes.py. Each case in the new model has to keep writing the right journal row; case 3 is anupdatedrow withsource="storage".src/basic_memory/models/knowledge.py,NoteContent:db_version,db_checksum(accepted),file_version,file_checksum,file_write_status,file_updated_at(materialized). There is no object identity column yet. That's the schema change, with an Alembic migration.src/basic_memory/runtime/note_materialization_planning.pyandsrc/basic_memory/runtime/note_content.py. The file write goes throughFileService.write_file(src/basic_memory/services/file_service.py, around line 252). It returns a checksum today; the new code needs the storage identity too. Cloud'sS3FileServicesubclassesFileService, and S3PutObjectreturns an ETag and version id.src/basic_memory/runtime/storage_events.py,StorageEventInput.etagalready exists.apps/cloud/src/basic_memory_cloud/services/storage_events.pymaps Tigris events (keepseTag).apps/cloud/src/basic_memory_cloud/pgq/jobs/index_file.pyalready uses the observed ETag in an early gate (checker.detect_observed) to skip observations a newer job has overtaken. That gate is case 2's cheap half.Tests
The existing suites are the place to extend:
tests/indexing/test_note_content_promotion_guard.py,tests/indexing/test_note_content_reconciler.py,tests/indexing/test_note_content_reconciliation.py,tests/indexing/test_note_content_batch_reconciliation.py.Write one test per case using the production sequences in the issue body. Prove each is load-bearing: break the classification and confirm the matching test fails.
Watch out for
AGENTS.md). Case 4 must keep both versions.