Repository navigation
fix(core): journal note changes that arrive through storage - #2121
Conversation
A note created, edited, renamed or deleted straight in storage (a local editor, WebDAV, an rclone bisync, a direct upload) now writes a row into the project change journal (accepted_project_note_change), like an API write does. Before this, those changes updated the database with no journal row, so the wiki projector, the OKF export and the project changes feed never saw them. Rows have source "storage", no actor, and materialized_at equal to accepted_at, because the change is already in storage. An unsettled row would defer every later wiki projection. Each row is written in the transaction that makes the database change, as its last write. - New module indexing/storage_note_changes.py: the "storage" source constant, an identity loader, and one writer that claims consecutive positions and inserts the rows. - NoteContentReconciler journals a bootstrap as "created" (or "updated" when the journal already holds the note) and an applied promotion as "updated". The journal is injected through the constructor; the default is the real one, so existing callers are unchanged. - Project-index move batches journal each replaced destination as "deleted", then each moved note as "moved". Delete batches journal each deleted note as "deleted". Markdown notes only. - ProjectRepository.record_accepted_note_changes takes an optional materialized_at. - Storage-promoted versions now count in the promotion rollback guard's history, so a stale object-storage read of an older API version after a storage edit defers instead of rolling the note back. - docs/DOMAIN_MODEL.md: the journal scope now covers storage changes. Board ticket: BMEM-116. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Drew Cain <groksrc@gmail.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6c8f6d73c7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| file_path=change.note.file_path, | ||
| previous_file_path=change.previous_file_path, | ||
| accepted_at=accepted_at, | ||
| source=ACCEPTED_STORAGE_CHANGE_SOURCE, |
There was a problem hiding this comment.
Preserve projector provenance when journaling generated wiki pages
After a local wiki rebuild, cli/commands/wiki.py immediately indexes the generated index.md and log.md files. Their note-content bootstraps now receive journal rows with source="storage", but wiki_projector._is_projector_change() excludes generated changes only when their source is WIKI_PROJECTOR_SOURCE. Consequently, the next rebuild treats these generated-page permalinks as source-note identities: _plan_wiki_projection() reports identity collisions and removes those pages from its writes, so existing navigation and logs stop updating. These historical journal rows remain on subsequent scans and rebuilds. Preserve the generated pages' projector provenance, or exclude verified projector-owned content from storage journaling, rather than unconditionally labeling it as a user storage change.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 5b9e00f, the commit after the one reviewed here.
is_projector_changeinsrc/basic_memory/indexing/wiki_projector.pynow treats a row at a reservedindex.mdorlog.mdas projector output when its source iswiki_projectororstorage. Generated pages that a local rebuild writes back and indexes no longer claim source-note identities.- The local wiki snapshot (
src/basic_memory/index/local_wiki_projection.py) ends at the last source change, so those rows do not move the watermark either. bm wiki rebuildindexes the project before it plans.- A reserved file a person wrote is still read from the file scan, so it stays covered.
Tests: test_storage_rows_for_locally_written_projection_documents_are_not_source_changes in tests/indexing/test_wiki_projector.py (no identity collisions, no generated page listed as a change), and tests/cli/test_wiki_command.py::test_wiki_rebuild_builds_local_navigation (rebuild, then status current). Both pass on head cdcd4a5.
…aled With storage changes journaled, `bm wiki rebuild` followed by `bm wiki status` reported the fresh rebuild as conflicted, then outdated. CI caught it in tests/cli/test_wiki_command.py::test_wiki_rebuild_builds_local_navigation. Two things caused it. A local rebuild writes index.md and log.md straight into the project folder and then indexes them, so each generated file came back as a storage journal row. The projector only skipped its own rows when their source was "wiki_projector", so these rows claimed the generated documents' identities and moved the watermark. Separately, a note on disk that had never been indexed got its first journal row during the rebuild's closing index pass, after the logs were rendered. - wiki_projector.is_projector_change (made public) also treats storage rows at a reserved index.md or log.md as projector output. The projector already reads reserved documents from the file scan. - The local wiki snapshot ends at the last source change, so the projector's own rows no longer move the watermark or the acceptance time rendered into generated files. - `bm wiki rebuild` indexes the project before it plans, so the generated logs cover every file on disk. A dry run still does not index. - The "storage" journal source constant moves to runtime/project_partition.py, so the projector can use it without importing database code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Drew Cain <groksrc@gmail.com>
groksrc
left a comment
There was a problem hiding this comment.
Changes are required before this pull request can merge. The move path can publish a journal row as materialized before the bytes it describes exist in storage.
Blocking finding
src/basic_memory/indexing/project_index_maintenance.py:836stampsEntityandNoteContentfrom the planned permalink rewrite. It records the moved row before the transaction commits at line 847. The actual file write runs only after that commit at line 856._record_move_batch_changesreads the post-repair permalink and checksum at lines 1005-1026.record_storage_note_changesthen setsmaterialized_at = accepted_atfor that row insrc/basic_memory/indexing/storage_note_changes.py:153-186._write_moved_file_contentscatches a failed post-commit write and still returns success atsrc/basic_memory/indexing/project_index_maintenance.py:1028-1053. The existing tests prove both parts:test_repository_project_index_maintenance_store_writes_moved_content_after_commitrequires commit before write, andtest_repository_project_index_maintenance_store_logs_failed_post_commit_writeaccepts a failed write.
When update_permalinks_on_move changes frontmatter, a consumer can therefore observe a settled row whose checksum is not on disk yet. If the post-commit write fails, the journal permanently claims that a version reached storage when it never did. A later scan can add a corrective version, but it cannot make this historical materialization claim true. The Wiki Projector can also run ahead instead of deferring.
Keep a move row unsettled until any planned post-commit content rewrite succeeds, then mark it materialized. An unchanged-content move can still be born settled. Add a regression test that forces write_moved_file_content to fail and verifies that the journal does not report the planned checksum as materialized and that the projector remains deferred.
Verification
I reviewed head 5b9e00f0f9408db08058891fac5e2ec7b7dcdbd9 against the BMEM-116 acceptance criteria. All 51 GitHub checks pass, including SQLite, PostgreSQL, coverage, static checks, DCO, and CodeQL. The focused journal and move-order tests also pass locally: 15 passed in 46.49s. I found no separate security issue and no board-only identifier in code or tests.
Next step: the author fixes the move settlement boundary and pushes the regression test; Shipright then runs a fresh review of the new head.
A scan move that also rewrites the note's permalink frontmatter (local `update_permalinks_on_move`) stamped the planned bytes into the entity and note_content rows and journaled a settled `moved` row in the batch transaction. The file write ran only after the commit, and a failed write was logged while the move still succeeded. The journal could therefore report a checksum as materialized before storage held it, or forever if the write failed, and the wiki projector could run ahead instead of deferring. Review of PR #2121 found this. - A moved row whose rewrite is planned now starts unsettled (`StorageNoteChange.awaits_storage_write`). A move with unchanged content is still written settled. - `write_moved_file_content` returns the checksum of the stored file. The move settles the row with `settle_stored_note_changes` only when that checksum equals the planned one. A failed write, or stored bytes that differ (CRLF newlines on Windows), leave it unsettled. - A settled storage `created`, `updated` or `deleted` row now settles older unsettled rows of the same note, by the rule every journal settle follows. So when a rewrite never lands, the next scan indexes the file's real bytes as a new version, and that row settles the move's row and lets the projector run. A settled `moved` row settles nothing older: it does not show that an earlier API write reached storage. - A database failure while settling is logged, and the committed move still succeeds, as a failed write already did. Tests force the post-commit write to fail and check that the planned checksum is not settled and that the wiki projector defers on the row, then that the next reconcile settles it. Others cover a landed rewrite, stored bytes that differ from the plan, a failed settle, and the writer's settle rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Drew Cain <groksrc@gmail.com>
|
Fixed in cdcd4a5: a moved note's journal row now settles only once its frontmatter rewrite is in storage. This answers the blocking finding in review 5913290361 (move rows marked materialized before the post-commit rewrite). What changed
Regression testsIn
Verified locally (macOS, head
|
groksrc
left a comment
There was a problem hiding this comment.
Review: two fixes are required before merge
Verdict: review:questions at head cdcd4a5e. The journal transaction boundaries and move settlement now look correct. However, generated Wiki rows leak into OKF history, and the public changes endpoint still documents the old behavior.
Blocking findings
-
Keep locally generated Wiki rows out of OKF history.
record_storage_note_changesassignssource="storage"to every indexed storage row (src/basic_memory/indexing/storage_note_changes.py:180-204). The Wiki planner correctly recognizes storage rows at reservedindex.mdandlog.mdpaths as projector output (src/basic_memory/indexing/wiki_projector.py:795-808). The OKF exporter does not use that classification.recorded_historyexcludes onlysource="wiki_projector"(src/basic_memory/okf/export.py:65-76), so the generated storage rows pass through and render as user history. I reproduced the resulting export entry:- created: [index.md](/index.md)This publishes projector output as note history even though
snapshot_filesexcludes the same generated documents. Reuse one projector-output classification or preserve projector provenance. Add a regression test that keeps real storage edits in OKF history while omitting generatedindex.mdandlog.mdstorage rows. -
Update the published contract for
GET /v2/projects/{project_id}/changes. Its route description still says that the journal records only API writes and omits WebDAV, sync-tool, and direct-upload changes (src/basic_memory/api/v2/routers/project_router.py:417-425). This PR makes those rows visible through the route. Rewrite that scope paragraph from the current behavior so OpenAPI readers do not retain a full-reload workaround that is no longer required for Markdown storage changes.
Verification
- I reviewed
cdcd4a5eagainst currentorigin/mainatc3bf1e26. The new main commit touches no PR paths, andgit merge-treereports a clean merge. - GitHub reports 51 successful checks, one skipped check, and no pending checks.
- Focused storage-journal, move-settlement, rollback-guard, and Wiki tests pass:
25 passed in 16.52s. - The earlier move-settlement issue is fixed: rewritten move rows remain unsettled until the stored checksum matches, and failed writes recover through a later storage promotion.
- I found no separate security issue and no board-only identifier in code, tests, or documentation.
Next step: fix both consumer-facing gaps, add the OKF regression test, and push the new head. Shipright can then run a fresh review.
…rage rows on /changes
A local wiki rebuild writes index.md and log.md to disk, and indexing then
journals them as storage changes. The wiki planner already treated those rows
as projector output, but the OKF export filtered only source "wiki_projector".
So an exported log.md listed the generated pages as note history, for example
"- created: [index.md](/index.md)".
- `is_projector_change` in indexing/wiki_projector.py now takes a source and a
path, so every journal reader can share it. The wiki planner, the local wiki
snapshot and the OKF export's recorded_history all use it. The export's log
now lists the same changes the wiki log does.
- The GET /v2/projects/{project_id}/changes description no longer says storage
writes and folder deletes are missing. It says how each kind of change
appears (`storage`, `delete_directory`, generated wiki pages) and that only
non-Markdown files are still left out. The ProjectChange.source field
description names `storage`.
- docs/DOMAIN_MODEL.md says how generated wiki pages reach the journal and which
rule keeps them out of note history.
Tests: an end-to-end test drives storage edits, a local rebuild and indexing,
then checks OKF history and the rendered log. A unit test covers rows from each
writer. Both fail on the old filter. Route tests cover a storage-written note in
/changes and the published description.
Answers review 5913575834 on #2121. Board ticket: BMEM-116.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
I, Drew Cain <groksrc@gmail.com>, hereby add my Signed-off-by to this commit: ca34229 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Drew Cain <groksrc@gmail.com>
|
Both findings in review 5913575834 are fixed in ca34229. The current head is 62efa59, which adds the DCO sign-off for ca34229 and changes no files. CI is green on it. 1. Generated wiki pages no longer appear in OKF history
Regression tests:
2.
|
groksrc
left a comment
There was a problem hiding this comment.
Review: no blocking findings
Verdict: review:done for PR #2121 at head 62efa590. The storage journal paths preserve the ticket's transaction, ordering, and materialization invariants, and the two prior review findings are fixed.
What I verified
record_storage_note_changeswrites Markdown storage changes withsource="storage", claims consecutive project positions last, and keeps each row in the database transaction that accepts the change (src/basic_memory/indexing/storage_note_changes.py:159).NoteContentReconcilerjournals only successful bootstraps and applied promotions. Deferred, stale, synchronized, and rollback-guard-refused outcomes write no row (src/basic_memory/indexing/note_content_reconciler.py:498).- Move and delete batches preserve the NoteContent-before-Entity lock order. They capture deleted identities before deletion and journal after the database mutation (
src/basic_memory/indexing/project_index_maintenance.py:773,src/basic_memory/indexing/project_index_maintenance.py:1189). - A move with a planned frontmatter rewrite starts unsettled. It settles only after storage returns the planned checksum. A failed or divergent write remains deferred until a later storage version supersedes it (
src/basic_memory/indexing/project_index_maintenance.py:1007). - Generated Wiki
index.mdandlog.mdrows stay out of Wiki and Open Knowledge Format history through the shared projector classification (src/basic_memory/indexing/wiki_projector.py:795,src/basic_memory/okf/export.py:69). The public project-changes description now documents storage rows. - I found no separate security issue and no board-only identifier in code, comments, documentation, or tests.
Verification
- Reviewed current PR head
62efa590against currentorigin/mainatc3bf1e26.git merge-tree --write-tree origin/main HEADcompleted without a conflict. - Focused journal, reconciliation, move/delete, Wiki, Open Knowledge Format, API, and repository tests:
263 passed in 66.05s. - Ruff passed for every changed Python file.
git diff --checkand the board-vocabulary scan passed. - GitHub reports 43 successful status checks with no failed or pending check.
Next step: Shipright reads the review:done label and advances the ticket. This review does not approve, merge, push, or move the ticket.
…indows (#2129) ## Summary Fixes the one test from #2121 that fails on every Windows run of `main`. The product code is fine. The test expected a content-neutral rename, but the test config made indexing rewrite the file during the rename. This PR also makes pull requests run the test file on Windows, so this kind of failure shows up before merge. ## What failed `tests/indexing/test_storage_change_journal.py::test_storage_rename_leaves_one_moved_row` fails on the Windows SQLite unit shard of `main`, most recently in run [44118870413](https://github.com/basicmachines-co/basic-memory/actions/runs/44118870413) at `b4b2c226`: ``` > [moved] = await journal_after(session_maker, test_project, created.partition_position) E ValueError: too many values to unpack (expected 1) ``` ## Why - The shared test config sets `update_permalinks_on_move=True` (`tests/conftest.py:308`). When the rename is indexed, that rewrites the moved note's `permalink` frontmatter. - `write_file_atomic` writes text with the platform's newline (`src/basic_memory/file_utils.py:138-146`), so on Windows the rewrite is stored with CRLF line endings. That behaviour is deliberate. - The stored checksum then differs from the planned checksum. The `moved` row stays unsettled, and the same index run promotes the stored bytes as a new version, which journals an `updated` row. This is the designed recovery path from #2121, not a bug. - So on Windows the test saw two rows after the rename (`moved`, `updated`) where it expected one. ## What changed - **The test.** `test_storage_rename_leaves_one_moved_row` now turns `update_permalinks_on_move` off, so the rename really leaves the file's bytes unchanged. It also asserts that the `moved` row carries the note's existing `db_version` and `db_checksum`, which pins down "unchanged content". The rewrite path is already covered by `test_scan_move_keeps_a_rewritten_note_row_unsettled_when_storage_holds_other_bytes` and `..._when_its_write_fails` in `tests/indexing/test_project_index_maintenance.py`. - **CI.** `test.yml` runs the full Windows unit suite only on `main`. This PR adds `tests/indexing/test_storage_change_journal.py` to the `test-windows-platform` job's file list, so pull requests run it on Windows. No product code changes. ## Verification All local runs on macOS, from `origin/main` at `b4b2c226`: - **Reproduced the Windows failure before the fix.** I used a scratch pytest plugin, kept outside the repo, that makes `write_file_atomic` write CRLF the way Windows does. The old test failed with `ValueError: too many values to unpack (expected 1, got 2)`. - **After the fix, with CRLF forced:** `tests/indexing/test_storage_change_journal.py`, `4 passed`. - **After the fix, native:** SQLite `4 passed`; PostgreSQL (`BASIC_MEMORY_TEST_POSTGRES=1`, testcontainers) `4 passed`. - `ruff check`, `ruff format --check` and `ty check` pass on the changed test. `test.yml` still parses as YAML. - The Windows job on this PR is the proof on real Windows. ## Out of scope `tests/mcp/test_hook_brief_search.py::test_session_start_never_imports_the_mcp_tool_stack` also fails on some Windows runs of `main`. It comes from #2113 and is not touched here. Ticket: BMEM-116 (Drew's board). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Drew Cain <groksrc@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Summary
A note created, edited, renamed or deleted straight in storage now writes a row into the project change journal (
accepted_project_note_change), the same as an API write. Storage here means a local editor, WebDAV, an rclone bisync or a direct upload. Before this change, those edits reached the database through the indexer with no journal row. The wiki projector, the OKF export andGET /v2/projects/{id}/changestherefore never saw them.Board ticket: BMEM-116. Cloud picks this up when its core pin moves; this PR changes core only.
What changed
One shared writer. The new module
src/basic_memory/indexing/storage_note_changes.pyholds:ACCEPTED_STORAGE_CHANGE_SOURCE = "storage". It lives inruntime/project_partition.py, so the pure wiki planner can use it without importing database code.load_storage_note_identities, which reads the entity fields a row names, plus the note'sdb_versionanddb_checksum.record_storage_note_changes, which claims consecutive positions withadvance_partition_position_byand bulk-inserts the rows.StorageNoteChangeJournal, the reconciler's journal.Every storage row has
source="storage"and no actor. A row is settled (materialized_atset) only when storage holds the bytes it names, because an unsettled row defers every later wiki projection (indexing/wiki_projector.py, "Projection deferred until accepted note positions are materialized"). Almost every storage change is already in storage, so its row is written settled, withmaterialized_atequal toaccepted_at. The one exception is a move that rewrites the note's permalink frontmatter; see Review rounds below.Where rows are written. Each row is written in the transaction that makes the database change, as its last write:
db_version/db_checksumNoteContentReconciler.reconcile, bootstrap (firstnote_contentrow)created, orupdatedif the journal already holds the note and its last row is not a deleteNoteContentReconciler.reconcile, applied promotionupdatedapply_project_index_move_batch, replaced destination entitydeleted, written before the move rowsapply_project_index_move_batch, moved entitymoved, withprevious_file_pathapply_project_index_delete_batchdeletedRows within a batch are in path order. Only Markdown notes get rows, matching the API journal.
Outcomes that accept no new content write no row. These are: a file catching up to accepted content, older materialized bytes, a deferred promotion, a stale anchor, a version-guard loss, and a promotion refused by the rollback guard.
The local wiki stays current. CI caught a regression: after
bm wiki rebuild,bm wiki statusreported the fresh rebuild asconflicted(tests/cli/test_wiki_command.py::test_wiki_rebuild_builds_local_navigation). The second commit fixes it in three places:index.mdandlog.mdstraight into the project folder and then indexes them, so each generated file came back as astoragerow.wiki_projector.is_projector_change(now public) treats astoragerow at a reservedindex.mdorlog.mdas projector output, as it already did forsource="wiki_projector". The projector reads reserved documents from the file scan, never from the journal._load_local_wiki_snapshotends the local snapshot at the last source change. Without this, the generated files' own rows moved thesource_watermarkand acceptance time rendered into every generated file, so each rebuild left the next one outdated. Local runs have no projection ledger, so this changes no stored state.bm wiki rebuild(and itsinitandupdatealiases) now indexes the project before it plans. A note on disk that was never indexed otherwise got its first journal row in the rebuild's closing index pass, after the logs were rendered.--dry-runstill does not index.Smaller changes.
NoteContentReconcilertakes an optionalchange_journal. The default is the realStorageNoteChangeJournal, so every existing caller is unchanged.ProjectRepository.record_accepted_note_changestakes an optionalmaterialized_at. I put it on the insert, not onRuntimeAcceptedProjectNoteChange, because that dataclass is embedded in queued job payloads (runtime/job_payloads.py). Adding a field there would change the wire format.docs/DOMAIN_MODEL.md: the journal scope bullet said storage reconciliation does not reach the journal yet. It now describes what does.Design decisions
UPDATE project … RETURNING, which locks the project row until commit. Every storage path claims it after itsnote_contentand entity locks, the same order the accepted-mutation runner and the folder delete use. Nothing takes the project lock and then waits on a note lock, so PostgreSQL cannot deadlock on it.else:of theIntegrityErrorhandler). A losing concurrent bootstrap rolls back and claims no position. The promotion that follows it journals asupdated.find_accepted_content_versionsis unchanged, so storage versions now count as accepted history. This closes a gap: after a storage edit, a stale object-storage read of the API version it replaced used to match the newest journaled version and promote, rolling the note back. It now defers when no authoritative reader exists. The guard's documented trade-off applies to storage versions too: a bisync user who restores a file to an older version's exact bytes is deferred on object storage. The newer content stays accepted.Verification
All commands ran in this branch's worktree on macOS, from
origin/mainat c853f5f.BASIC_MEMORY_ENV=test uv run pytest -n 4 tests/indexing→694 passed, 1 skipped.BASIC_MEMORY_TEST_POSTGRES=1 … pytest tests/indexing/test_storage_note_changes.py tests/indexing/test_storage_change_journal.py tests/indexing/test_note_content_reconciler.py tests/indexing/test_note_content_promotion_guard.py tests/indexing/test_project_index_maintenance.py tests/repository/test_project_partition_repository.py test-int/test_relation_generation_concurrency.py→99 passed.pytest -n 3 tests/services/test_entity_service.py tests/cloud/test_cloud_services.py tests/cloud/test_note_content_materialization.py test-int/read_cache test-int/test_accepted_note_mutation_runner_db.py→270 passed.pytest -n 4 tests/index tests/mcp/test_wiki_after_api_writes.py tests/okf tests/services/test_read_cache_changes.py tests/api/v2/test_project_changes_router.py→434 passed, and 1 failure: "worker 'gw2' crashed while running 'tests/index/test_local_watch_stress_parity.py::test_local_event_index_handles_large_batch_of_file_adds'". That test hit the 120 s pytest timeout while another session's test suite was loading the machine. Run alone it passes:4 passed in 40.40swith this change, and4 passed in 46.98son the original source.assert 'current' == 'deferred', and passes with this change.ruff check .andruff format --check .are clean.ty check src tests test-intreports only the 4pymilvusimport errors from my local environment; CI installs themilvusextra.storage_note_changes.pyis at 100%, and no changed line in the other three source files is uncovered.git diffhas no ticket keys in code, comments or tests.pytest tests/cli/test_wiki_command.py tests/indexing/test_wiki_projector.py tests/index/test_local_wiki_projection.py tests/indexing/test_storage_change_journal.py tests/mcp/test_wiki_after_api_writes.py→ all pass (tests/cli/test_wiki_command.py:9 passed).tests/clias a whole, locally: the subprocess "exits cleanly" tests time out on this machine, for examplesubprocess.TimeoutExpired: Command '['uv', 'run', 'bm', 'reset', '--force']' timed out after 20 seconds.uv run bm --versionalone took 11.5 s at load average 10.8. These tests passed in CI on the first commit.tests/repositoryandtests/runtime. One combined run hit my 580 s limit at about 35% under the same machine load, with no failures by then. CI runs them.New tests:
tests/indexing/test_storage_change_journal.pydrives the real local project indexer against files on disk, with no API calls. It checks three things. Create, edit and delete leavecreated,updatedanddeletedrows in order, carrying thenote_contentrevision. A rename with unchanged content leaves onemovedrow. After storage-only changes,inspect_local_wiki_projectionreportspending_materialization == ().tests/indexing/test_storage_note_changes.pycovers the writer: positions, fields, non-Markdown files, no permalink, andcreatedversusupdated.tests/indexing/test_note_content_reconciler.pyhas real-database tests for the bootstrap and promotion rows and for each no-row outcome. It also has two rollback tests: a failure raised after the journal write leaves nonote_contentchange, no row, and an unchangedproject.partition_position.tests/indexing/test_note_content_promotion_guard.py: a stale read after a journaled storage edit defers, and refused promotions write no row.tests/indexing/test_project_index_maintenance.py: real-database move and delete batch rows, plus a rollback test. Existing fake-session tests now expect the extra identity read.tests/indexing/test_wiki_projector.py: a storage row on a source note projects without deferral, and storage rows for generated documents claim no identity and never appear in a log.currentwith no writes.Review rounds
Round 1 (commit cdcd4a5): a moved row settles only once its rewrite lands. A scan move with
update_permalinks_on_movestamped the planned frontmatter rewrite intoEntityandnote_content. It also journaled a settledmovedrow, then wrote the file after commit. So the journal could call a checksum materialized before storage held it.write_moved_file_contentreturns the stored file's checksum. The move settles the row only when it equals the planned checksum.Round 2 (commit ca34229): generated wiki pages stay out of OKF history, and
/changesdocuments storage rows.index.mdandlog.mdcome back asstoragerows. The OKF export filtered onlysource="wiki_projector", so itslog.mdlisted them as note history.wiki_projector.is_projector_changenow takessourceandpath. The wiki planner, the local wiki snapshot andokf/export.pyrecorded_historyshare it.GET /v2/projects/{project_id}/changesdescription now says how each kind of change appears, includingstorageanddelete_directory. Only non-Markdown files are still missing. TheProjectChange.sourcefield description namesstorage..github/dco.ymlallows.recorded_historytest (both fail on the old filter), a storage-written note read back from/changes, and a check of the published description.tests/index,tests/indexingand the wiki and OKF tests →1041 passed, 1 skipped. PostgreSQL, the changed test files →162 passed.Noted, not changed
note_contentrow, so each gets onecreatedrow. That is one row per note, once, and correct for a client reading the journal from position 0.EntityServicecompatibility writes (create_or_update_entity) write the file and then reconcile it, so their new versions are journaled with sourcestorage. They had no row before.::,[,],|, a backtick,<or>(_validate_canonical_permalink). Frontmatter permalinks are stored raw. A storage note with such a permalink already breaks the projection through its source note while it exists, and API rows have the same exposure. Adeletedrow for it now keeps it in the journal. I have not seen this happen and did not change it.Next step
Review against the board ticket's acceptance criteria. CI's
test.ymlruns the full SQLite and PostgreSQL matrix.🤖 Generated with Claude Code