Skip to content

fix(core): journal note changes that arrive through storage - #2121

Merged
groksrc merged 5 commits into
mainfrom
shipright/BMEM-116
Nov 16, 2026
Merged

groksrc merged 5 commits into
mainfrom
shipright/BMEM-116

Conversation

@groksrc

@groksrc groksrc commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

Summary

A note created, edited, renamed or deleted straight in storage now writes a row into the project change journal (accepted_project_note_change), the same as an API write. Storage here means a local editor, WebDAV, an rclone bisync or a direct upload. Before this change, those edits reached the database through the indexer with no journal row. The wiki projector, the OKF export and GET /v2/projects/{id}/changes therefore never saw them.

Board ticket: BMEM-116. Cloud picks this up when its core pin moves; this PR changes core only.

What changed

One shared writer. The new module src/basic_memory/indexing/storage_note_changes.py holds:

  • The journal source ACCEPTED_STORAGE_CHANGE_SOURCE = "storage". It lives in runtime/project_partition.py, so the pure wiki planner can use it without importing database code.
  • load_storage_note_identities, which reads the entity fields a row names, plus the note's db_version and db_checksum.
  • record_storage_note_changes, which claims consecutive positions with advance_partition_position_by and bulk-inserts the rows.
  • StorageNoteChangeJournal, the reconciler's journal.

Every storage row has source="storage" and no actor. A row is settled (materialized_at set) only when storage holds the bytes it names, because an unsettled row defers every later wiki projection (indexing/wiki_projector.py, "Projection deferred until accepted note positions are materialized"). Almost every storage change is already in storage, so its row is written settled, with materialized_at equal to accepted_at. The one exception is a move that rewrites the note's permalink frontmatter; see Review rounds below.

Where rows are written. Each row is written in the transaction that makes the database change, as its last write:

Path Operation db_version / db_checksum
NoteContentReconciler.reconcile, bootstrap (first note_content row) created, or updated if the journal already holds the note and its last row is not a delete version 1 and the observed checksum
NoteContentReconciler.reconcile, applied promotion updated the promoted version
apply_project_index_move_batch, replaced destination entity deleted, written before the move rows its last accepted revision, if any
apply_project_index_move_batch, moved entity moved, with previous_file_path read after the path updates, so it includes any content repair
apply_project_index_delete_batch deleted its last accepted revision, if any

Rows within a batch are in path order. Only Markdown notes get rows, matching the API journal.

Outcomes that accept no new content write no row. These are: a file catching up to accepted content, older materialized bytes, a deferred promotion, a stale anchor, a version-guard loss, and a promotion refused by the rollback guard.

The local wiki stays current. CI caught a regression: after bm wiki rebuild, bm wiki status reported the fresh rebuild as conflicted (tests/cli/test_wiki_command.py::test_wiki_rebuild_builds_local_navigation). The second commit fixes it in three places:

  • A local rebuild writes index.md and log.md straight into the project folder and then indexes them, so each generated file came back as a storage row. wiki_projector.is_projector_change (now public) treats a storage row at a reserved index.md or log.md as projector output, as it already did for source="wiki_projector". The projector reads reserved documents from the file scan, never from the journal.
  • _load_local_wiki_snapshot ends the local snapshot at the last source change. Without this, the generated files' own rows moved the source_watermark and acceptance time rendered into every generated file, so each rebuild left the next one outdated. Local runs have no projection ledger, so this changes no stored state.
  • bm wiki rebuild (and its init and update aliases) now indexes the project before it plans. A note on disk that was never indexed otherwise got its first journal row in the rebuild's closing index pass, after the logs were rendered. --dry-run still does not index.

Smaller changes.

  • NoteContentReconciler takes an optional change_journal. The default is the real StorageNoteChangeJournal, so every existing caller is unchanged.
  • ProjectRepository.record_accepted_note_changes takes an optional materialized_at. I put it on the insert, not on RuntimeAcceptedProjectNoteChange, because that dataclass is embedded in queued job payloads (runtime/job_payloads.py). Adding a field there would change the wire format.
  • docs/DOMAIN_MODEL.md: the journal scope bullet said storage reconciliation does not reach the journal yet. It now describes what does.

Design decisions

  • Lock order. Claiming a position runs UPDATE project … RETURNING, which locks the project row until commit. Every storage path claims it after its note_content and entity locks, the same order the accepted-mutation runner and the folder delete use. Nothing takes the project lock and then waits on a note lock, so PostgreSQL cannot deadlock on it.
  • The bootstrap journals only after its insert succeeds (the else: of the IntegrityError handler). A losing concurrent bootstrap rolls back and claims no position. The promotion that follows it journals as updated.
  • Rollback guard. find_accepted_content_versions is unchanged, so storage versions now count as accepted history. This closes a gap: after a storage edit, a stale object-storage read of the API version it replaced used to match the newest journaled version and promote, rolling the note back. It now defers when no authoritative reader exists. The guard's documented trade-off applies to storage versions too: a bisync user who restores a file to an older version's exact bytes is deferred on object storage. The newer content stays accepted.
  • Notes with no permalink get no row, with a warning. Databases from before permalinks became mandatory can still hold them, and a journal row needs one. The folder delete raises in this case. A storage change cannot be refused, though: raising would roll back the index transaction and fail the same file on every later scan.

Verification

All commands ran in this branch's worktree on macOS, from origin/main at c853f5f.

  • New and changed tests, SQLite: BASIC_MEMORY_ENV=test uv run pytest -n 4 tests/indexing → 694 passed, 1 skipped.
  • Same tests on PostgreSQL (testcontainers), plus the lock-order test: BASIC_MEMORY_TEST_POSTGRES=1 … pytest tests/indexing/test_storage_note_changes.py tests/indexing/test_storage_change_journal.py tests/indexing/test_note_content_reconciler.py tests/indexing/test_note_content_promotion_guard.py tests/indexing/test_project_index_maintenance.py tests/repository/test_project_partition_repository.py test-int/test_relation_generation_concurrency.py → 99 passed.
  • Journal consumers and other callers, SQLite: pytest -n 3 tests/services/test_entity_service.py tests/cloud/test_cloud_services.py tests/cloud/test_note_content_materialization.py test-int/read_cache test-int/test_accepted_note_mutation_runner_db.py → 270 passed.
  • pytest -n 4 tests/index tests/mcp/test_wiki_after_api_writes.py tests/okf tests/services/test_read_cache_changes.py tests/api/v2/test_project_changes_router.py → 434 passed, and 1 failure: "worker 'gw2' crashed while running 'tests/index/test_local_watch_stress_parity.py::test_local_event_index_handles_large_batch_of_file_adds'". That test hit the 120 s pytest timeout while another session's test suite was loading the machine. Run alone it passes: 4 passed in 40.40s with this change, and 4 passed in 46.98s on the original source.
  • The regression test for the rollback-guard gap fails on the original reconciler with assert 'current' == 'deferred', and passes with this change.
  • ruff check . and ruff format --check . are clean. ty check src tests test-int reports only the 4 pymilvus import errors from my local environment; CI installs the milvus extra.
  • Coverage: storage_note_changes.py is at 100%, and no changed line in the other three source files is uncovered.
  • git diff has no ticket keys in code, comments or tests.
  • Wiki fix, SQLite: pytest tests/cli/test_wiki_command.py tests/indexing/test_wiki_projector.py tests/index/test_local_wiki_projection.py tests/indexing/test_storage_change_journal.py tests/mcp/test_wiki_after_api_writes.py → all pass (tests/cli/test_wiki_command.py: 9 passed).
  • tests/cli as a whole, locally: the subprocess "exits cleanly" tests time out on this machine, for example subprocess.TimeoutExpired: Command '['uv', 'run', 'bm', 'reset', '--force']' timed out after 20 seconds. uv run bm --version alone took 11.5 s at load average 10.8. These tests passed in CI on the first commit.
  • Not run locally: the whole of tests/repository and tests/runtime. One combined run hit my 580 s limit at about 35% under the same machine load, with no failures by then. CI runs them.

New tests:

  • tests/indexing/test_storage_change_journal.py drives the real local project indexer against files on disk, with no API calls. It checks three things. Create, edit and delete leave created, updated and deleted rows in order, carrying the note_content revision. A rename with unchanged content leaves one moved row. After storage-only changes, inspect_local_wiki_projection reports pending_materialization == ().
  • tests/indexing/test_storage_note_changes.py covers the writer: positions, fields, non-Markdown files, no permalink, and created versus updated.
  • tests/indexing/test_note_content_reconciler.py has real-database tests for the bootstrap and promotion rows and for each no-row outcome. It also has two rollback tests: a failure raised after the journal write leaves no note_content change, no row, and an unchanged project.partition_position.
  • tests/indexing/test_note_content_promotion_guard.py: a stale read after a journaled storage edit defers, and refused promotions write no row.
  • tests/indexing/test_project_index_maintenance.py: real-database move and delete batch rows, plus a rollback test. Existing fake-session tests now expect the extra identity read.
  • tests/indexing/test_wiki_projector.py: a storage row on a source note projects without deferral, and storage rows for generated documents claim no identity and never appear in a log.
  • The end-to-end wiki test above also runs rebuild, index and status, and asserts the projection is current with no writes.

Review rounds

Round 1 (commit cdcd4a5): a moved row settles only once its rewrite lands. A scan move with update_permalinks_on_move stamped the planned frontmatter rewrite into Entity and note_content. It also journaled a settled moved row, then wrote the file after commit. So the journal could call a checksum materialized before storage held it.

  • A moved row with a planned rewrite now starts unsettled. A move with unchanged content is still written settled.
  • write_moved_file_content returns the stored file's checksum. The move settles the row only when it equals the planned checksum.
  • If the write fails, the next scan promotes the file's real bytes. That settled storage row settles the older unsettled rows of the same note.

Round 2 (commit ca34229): generated wiki pages stay out of OKF history, and /changes documents storage rows.

  • A local rebuild's generated index.md and log.md come back as storage rows. The OKF export filtered only source="wiki_projector", so its log.md listed them as note history.
  • wiki_projector.is_projector_change now takes source and path. The wiki planner, the local wiki snapshot and okf/export.py recorded_history share it.
  • The GET /v2/projects/{project_id}/changes description now says how each kind of change appears, including storage and delete_directory. Only non-Markdown files are still missing. The ProjectChange.source field description names storage.
  • Commit 62efa59 is an empty DCO remediation commit that adds the sign-off missing from ca34229, as .github/dco.yml allows.
  • Tests: an end-to-end OKF history test through a real local rebuild, a writer-by-writer recorded_history test (both fail on the old filter), a storage-written note read back from /changes, and a check of the published description.
  • Verified locally: SQLite tests/index, tests/indexing and the wiki and OKF tests → 1041 passed, 1 skipped. PostgreSQL, the changed test files → 162 passed.

Noted, not changed

  • Journal growth. The first index after upgrade bootstraps every legacy note that has no note_content row, so each gets one created row. That is one row per note, once, and correct for a client reading the journal from position 0.
  • Contention. Reconcile, move and delete transactions now update the project row, so on PostgreSQL they serialize with API writes to the same project for the length of each short transaction.
  • EntityService compatibility writes (create_or_update_entity) write the file and then reconcile it, so their new versions are journaled with source storage. They had no row before.
  • Unsafe frontmatter permalinks. The wiki projector rejects permalinks containing ::, [, ], |, a backtick, < or > (_validate_canonical_permalink). Frontmatter permalinks are stored raw. A storage note with such a permalink already breaks the projection through its source note while it exists, and API rows have the same exposure. A deleted row for it now keeps it in the journal. I have not seen this happen and did not change it.

Next step

Review against the board ticket's acceptance criteria. CI's test.yml runs the full SQLite and PostgreSQL matrix.

🤖 Generated with Claude Code

A note created, edited, renamed or deleted straight in storage (a local
editor, WebDAV, an rclone bisync, a direct upload) now writes a row into
the project change journal (accepted_project_note_change), like an API
write does. Before this, those changes updated the database with no
journal row, so the wiki projector, the OKF export and the project
changes feed never saw them.

Rows have source "storage", no actor, and materialized_at equal to
accepted_at, because the change is already in storage. An unsettled row
would defer every later wiki projection. Each row is written in the
transaction that makes the database change, as its last write.

- New module indexing/storage_note_changes.py: the "storage" source
  constant, an identity loader, and one writer that claims consecutive
  positions and inserts the rows.
- NoteContentReconciler journals a bootstrap as "created" (or "updated"
  when the journal already holds the note) and an applied promotion as
  "updated". The journal is injected through the constructor; the
  default is the real one, so existing callers are unchanged.
- Project-index move batches journal each replaced destination as
  "deleted", then each moved note as "moved". Delete batches journal
  each deleted note as "deleted". Markdown notes only.
- ProjectRepository.record_accepted_note_changes takes an optional
  materialized_at.
- Storage-promoted versions now count in the promotion rollback guard's
  history, so a stale object-storage read of an older API version after
  a storage edit defers instead of rolling the note back.
- docs/DOMAIN_MODEL.md: the journal scope now covers storage changes.

Board ticket: BMEM-116.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Drew Cain <groksrc@gmail.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-15T20:44:08.906138Z 6c8f6d7 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6c8f6d73c7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

file_path=change.note.file_path,
previous_file_path=change.previous_file_path,
accepted_at=accepted_at,
source=ACCEPTED_STORAGE_CHANGE_SOURCE,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve projector provenance when journaling generated wiki pages

After a local wiki rebuild, cli/commands/wiki.py immediately indexes the generated index.md and log.md files. Their note-content bootstraps now receive journal rows with source="storage", but wiki_projector._is_projector_change() excludes generated changes only when their source is WIKI_PROJECTOR_SOURCE. Consequently, the next rebuild treats these generated-page permalinks as source-note identities: _plan_wiki_projection() reports identity collisions and removes those pages from its writes, so existing navigation and logs stop updating. These historical journal rows remain on subsequent scans and rebuilds. Preserve the generated pages' projector provenance, or exclude verified projector-owned content from storage journaling, rather than unconditionally labeling it as a user storage change.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 5b9e00f, the commit after the one reviewed here.

  • is_projector_change in src/basic_memory/indexing/wiki_projector.py now treats a row at a reserved index.md or log.md as projector output when its source is wiki_projector or storage. Generated pages that a local rebuild writes back and indexes no longer claim source-note identities.
  • The local wiki snapshot (src/basic_memory/index/local_wiki_projection.py) ends at the last source change, so those rows do not move the watermark either.
  • bm wiki rebuild indexes the project before it plans.
  • A reserved file a person wrote is still read from the file scan, so it stays covered.

Tests: test_storage_rows_for_locally_written_projection_documents_are_not_source_changes in tests/indexing/test_wiki_projector.py (no identity collisions, no generated page listed as a change), and tests/cli/test_wiki_command.py::test_wiki_rebuild_builds_local_navigation (rebuild, then status current). Both pass on head cdcd4a5.

…aled

With storage changes journaled, `bm wiki rebuild` followed by
`bm wiki status` reported the fresh rebuild as conflicted, then
outdated. CI caught it in
tests/cli/test_wiki_command.py::test_wiki_rebuild_builds_local_navigation.

Two things caused it. A local rebuild writes index.md and log.md
straight into the project folder and then indexes them, so each
generated file came back as a storage journal row. The projector only
skipped its own rows when their source was "wiki_projector", so these
rows claimed the generated documents' identities and moved the
watermark. Separately, a note on disk that had never been indexed got
its first journal row during the rebuild's closing index pass, after
the logs were rendered.

- wiki_projector.is_projector_change (made public) also treats storage
  rows at a reserved index.md or log.md as projector output. The
  projector already reads reserved documents from the file scan.
- The local wiki snapshot ends at the last source change, so the
  projector's own rows no longer move the watermark or the acceptance
  time rendered into generated files.
- `bm wiki rebuild` indexes the project before it plans, so the
  generated logs cover every file on disk. A dry run still does not
  index.
- The "storage" journal source constant moves to
  runtime/project_partition.py, so the projector can use it without
  importing database code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Drew Cain <groksrc@gmail.com>

@groksrc groksrc left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes are required before this pull request can merge. The move path can publish a journal row as materialized before the bytes it describes exist in storage.

Blocking finding

  • src/basic_memory/indexing/project_index_maintenance.py:836 stamps Entity and NoteContent from the planned permalink rewrite. It records the moved row before the transaction commits at line 847. The actual file write runs only after that commit at line 856.
  • _record_move_batch_changes reads the post-repair permalink and checksum at lines 1005-1026. record_storage_note_changes then sets materialized_at = accepted_at for that row in src/basic_memory/indexing/storage_note_changes.py:153-186.
  • _write_moved_file_contents catches a failed post-commit write and still returns success at src/basic_memory/indexing/project_index_maintenance.py:1028-1053. The existing tests prove both parts: test_repository_project_index_maintenance_store_writes_moved_content_after_commit requires commit before write, and test_repository_project_index_maintenance_store_logs_failed_post_commit_write accepts a failed write.

When update_permalinks_on_move changes frontmatter, a consumer can therefore observe a settled row whose checksum is not on disk yet. If the post-commit write fails, the journal permanently claims that a version reached storage when it never did. A later scan can add a corrective version, but it cannot make this historical materialization claim true. The Wiki Projector can also run ahead instead of deferring.

Keep a move row unsettled until any planned post-commit content rewrite succeeds, then mark it materialized. An unchanged-content move can still be born settled. Add a regression test that forces write_moved_file_content to fail and verifies that the journal does not report the planned checksum as materialized and that the projector remains deferred.

Verification

I reviewed head 5b9e00f0f9408db08058891fac5e2ec7b7dcdbd9 against the BMEM-116 acceptance criteria. All 51 GitHub checks pass, including SQLite, PostgreSQL, coverage, static checks, DCO, and CodeQL. The focused journal and move-order tests also pass locally: 15 passed in 46.49s. I found no separate security issue and no board-only identifier in code or tests.

Next step: the author fixes the move settlement boundary and pushes the regression test; Shipright then runs a fresh review of the new head.

@groksrc groksrc added the review:questions Automated review found blocking questions or required changes label Nov 15, 2026
A scan move that also rewrites the note's permalink frontmatter (local
`update_permalinks_on_move`) stamped the planned bytes into the entity
and note_content rows and journaled a settled `moved` row in the batch
transaction. The file write ran only after the commit, and a failed
write was logged while the move still succeeded. The journal could
therefore report a checksum as materialized before storage held it,
or forever if the write failed, and the wiki projector could run
ahead instead of deferring. Review of PR #2121 found this.

- A moved row whose rewrite is planned now starts unsettled
  (`StorageNoteChange.awaits_storage_write`). A move with unchanged
  content is still written settled.
- `write_moved_file_content` returns the checksum of the stored file.
  The move settles the row with `settle_stored_note_changes` only when
  that checksum equals the planned one. A failed write, or stored bytes
  that differ (CRLF newlines on Windows), leave it unsettled.
- A settled storage `created`, `updated` or `deleted` row now settles
  older unsettled rows of the same note, by the rule every journal
  settle follows. So when a rewrite never lands, the next scan indexes
  the file's real bytes as a new version, and that row settles the
  move's row and lets the projector run. A settled `moved` row settles
  nothing older: it does not show that an earlier API write reached
  storage.
- A database failure while settling is logged, and the committed move
  still succeeds, as a failed write already did.

Tests force the post-commit write to fail and check that the planned
checksum is not settled and that the wiki projector defers on the row,
then that the next reconcile settles it. Others cover a landed rewrite,
stored bytes that differ from the plan, a failed settle, and the
writer's settle rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Drew Cain <groksrc@gmail.com>
@groksrc

groksrc commented Nov 15, 2026

Copy link
Copy Markdown
Member Author

Fixed in cdcd4a5: a moved note's journal row now settles only once its frontmatter rewrite is in storage. This answers the blocking finding in review 5913290361 (move rows marked materialized before the post-commit rewrite).

What changed

  • _record_move_batch_changes writes the moved row unsettled when the batch plans a frontmatter rewrite for that note (StorageNoteChange.awaits_storage_write in src/basic_memory/indexing/storage_note_changes.py). A move with unchanged content is still written settled.
  • ProjectIndexMoveContentUpdater.write_moved_file_content now returns the checksum of the stored file. After the post-commit write, _settle_stored_move_rows settles the row only when that checksum equals the planned one (src/basic_memory/indexing/project_index_maintenance.py). A failed write leaves the row unsettled. So do stored bytes that differ from the plan, such as CRLF newlines on Windows.
  • Recovery when the rewrite never lands. The next scan sees the file's checksum differ from the stamped one, and the reconciler promotes the file's real bytes to a new version. That settled updated row now settles older unsettled rows of the same note. This is the rule mark_accepted_note_change_materialized already applies: a newer materialized generation supersedes every older one. Settled storage created and deleted rows do the same. A settled moved row settles nothing older, because a move does not show that an earlier API write reached storage.
  • A failed settle. A database failure while settling is logged, and the committed move still succeeds, as a failed write already did. The row then waits for the next storage change to the note. The local API write path has the same window between its file write and its settle (settle_local_project_change in src/basic_memory/index/note_content_materialization.py).

Regression tests

In tests/indexing/test_project_index_maintenance.py, against a real database:

  • test_scan_move_keeps_a_rewritten_note_row_unsettled_when_its_write_fails forces write_moved_file_content to fail. It asserts the row keeps the planned checksum with materialized_at NULL, and that a wiki projector plan over the journal waits on that position. It then runs the real NoteContentReconciler on the file's real bytes. It asserts version 4 is journaled, every row is settled, and the projector waits on nothing.
  • test_scan_move_settles_a_rewritten_note_row_only_after_its_write reads the journal from inside the write: the rewritten row is unsettled there, the unchanged move is settled, and both are settled afterwards.
  • test_scan_move_keeps_a_rewritten_note_row_unsettled_when_storage_holds_other_bytes and test_scan_move_logs_a_failed_settle_and_keeps_the_committed_move cover the other two exits.

tests/indexing/test_storage_note_changes.py covers the writer's settle rule for each operation.

Verified locally (macOS, head cdcd4a5e)

  • SQLite: tests/indexing, plus the local move, local wiki projection and wiki command tests: 743 passed, 1 skipped.
  • SQLite: tests/index, tests/index/test_local_project_index.py, tests/services/test_initialization.py and the two move and permalink integration tests: 265 passed.
  • PostgreSQL: the five changed journal test files: 192 passed.
  • ruff check is clean. ty check src tests test-int reports only the 4 known pymilvus import errors. storage_note_changes.py has 100% line coverage.

One consequence to know

After recovery, the superseded move row has materialized_at set although its planned bytes never reached storage. That is what a settled row already means once a newer settled version of the note exists. It is also what lets the projector run again.

Next step: CI on the new head, then a fresh review.

@groksrc groksrc removed the review:questions Automated review found blocking questions or required changes label Nov 15, 2026

@groksrc groksrc left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: two fixes are required before merge

Verdict: review:questions at head cdcd4a5e. The journal transaction boundaries and move settlement now look correct. However, generated Wiki rows leak into OKF history, and the public changes endpoint still documents the old behavior.

Blocking findings

  1. Keep locally generated Wiki rows out of OKF history. record_storage_note_changes assigns source="storage" to every indexed storage row (src/basic_memory/indexing/storage_note_changes.py:180-204). The Wiki planner correctly recognizes storage rows at reserved index.md and log.md paths as projector output (src/basic_memory/indexing/wiki_projector.py:795-808). The OKF exporter does not use that classification. recorded_history excludes only source="wiki_projector" (src/basic_memory/okf/export.py:65-76), so the generated storage rows pass through and render as user history. I reproduced the resulting export entry:

    - created: [index.md](/index.md)
    

    This publishes projector output as note history even though snapshot_files excludes the same generated documents. Reuse one projector-output classification or preserve projector provenance. Add a regression test that keeps real storage edits in OKF history while omitting generated index.md and log.md storage rows.

  2. Update the published contract for GET /v2/projects/{project_id}/changes. Its route description still says that the journal records only API writes and omits WebDAV, sync-tool, and direct-upload changes (src/basic_memory/api/v2/routers/project_router.py:417-425). This PR makes those rows visible through the route. Rewrite that scope paragraph from the current behavior so OpenAPI readers do not retain a full-reload workaround that is no longer required for Markdown storage changes.

Verification

  • I reviewed cdcd4a5e against current origin/main at c3bf1e26. The new main commit touches no PR paths, and git merge-tree reports a clean merge.
  • GitHub reports 51 successful checks, one skipped check, and no pending checks.
  • Focused storage-journal, move-settlement, rollback-guard, and Wiki tests pass: 25 passed in 16.52s.
  • The earlier move-settlement issue is fixed: rewritten move rows remain unsettled until the stored checksum matches, and failed writes recover through a later storage promotion.
  • I found no separate security issue and no board-only identifier in code, tests, or documentation.

Next step: fix both consumer-facing gaps, add the OKF regression test, and push the new head. Shipright can then run a fresh review.

@groksrc groksrc added the review:questions Automated review found blocking questions or required changes label Nov 15, 2026
groksrc and others added 2 commits November 15, 2026 18:24
…rage rows on /changes

A local wiki rebuild writes index.md and log.md to disk, and indexing then
journals them as storage changes. The wiki planner already treated those rows
as projector output, but the OKF export filtered only source "wiki_projector".
So an exported log.md listed the generated pages as note history, for example
"- created: [index.md](/index.md)".

- `is_projector_change` in indexing/wiki_projector.py now takes a source and a
  path, so every journal reader can share it. The wiki planner, the local wiki
  snapshot and the OKF export's recorded_history all use it. The export's log
  now lists the same changes the wiki log does.
- The GET /v2/projects/{project_id}/changes description no longer says storage
  writes and folder deletes are missing. It says how each kind of change
  appears (`storage`, `delete_directory`, generated wiki pages) and that only
  non-Markdown files are still left out. The ProjectChange.source field
  description names `storage`.
- docs/DOMAIN_MODEL.md says how generated wiki pages reach the journal and which
  rule keeps them out of note history.

Tests: an end-to-end test drives storage edits, a local rebuild and indexing,
then checks OKF history and the rendered log. A unit test covers rows from each
writer. Both fail on the old filter. Route tests cover a storage-written note in
/changes and the published description.

Answers review 5913575834 on #2121. Board ticket: BMEM-116.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
I, Drew Cain <groksrc@gmail.com>, hereby add my Signed-off-by to this commit: ca34229

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Drew Cain <groksrc@gmail.com>
@groksrc

groksrc commented Nov 16, 2026

Copy link
Copy Markdown
Member Author

Both findings in review 5913575834 are fixed in ca34229. The current head is 62efa59, which adds the DCO sign-off for ca34229 and changes no files. CI is green on it.

1. Generated wiki pages no longer appear in OKF history

  • is_projector_change (src/basic_memory/indexing/wiki_projector.py) now takes source and path as keyword arguments, so any journal reader can call it. The rule is unchanged: a row at a reserved index.md or log.md is projector output when its source is wiki_projector or storage.
  • recorded_history (src/basic_memory/okf/export.py) now filters with that rule instead of source != "wiki_projector". The wiki planner, the local wiki snapshot and the OKF export now share one classification, so the exported log.md lists the same changes as the wiki log.
  • One side effect: the export used to drop a wiki_projector row at any path. Now it drops one only at a reserved name. The projector writes nothing else, so this changes no real output.
  • snapshot_files already left the generated files out of the bundle, and it refuses to export a hand-written concept at a reserved name. So the history and the files now agree.

Regression tests:

  • tests/indexing/test_storage_change_journal.py::test_okf_history_keeps_storage_edits_and_omits_generated_wiki_pages runs the real local indexer: a storage create and edit, a local wiki rebuild, then indexing. It checks that the generated pages were journaled as storage rows. Then it checks that the OKF history and rendered log lines list only the note's created and updated.
  • tests/okf/test_export_failures.py::test_history_keeps_note_edits_and_omits_generated_wiki_pages covers hand-written rows from each writer: storage note edits and moves, generated pages at the root and in a folder, and a wiki_projector row.
  • I ran both tests against the old filter, and both failed. The first failure was At index 1 diff: (2, 'created', 'index.md') != (4, 'updated', 'guides/start.md').

2. GET /v2/projects/{project_id}/changes describes the current behaviour

  • I rewrote the scope paragraph (src/basic_memory/api/v2/routers/project_router.py). It says the journal records Markdown note changes however they arrive, and source tells them apart:
    • API writes carry the route name;
    • folder deletes carry delete_directory;
    • storage changes carry storage and appear when the project index picks them up;
    • a rename with unchanged content is one moved change, and a rename with edits is a delete plus a create;
    • generated wiki pages carry wiki_projector, or storage after a local rebuild.
  • Only non-Markdown files are still described as missing. The old text also listed folder deletes as missing, but fix(core): journal a delete for every note a folder delete removes #2120 already journals them.
  • The ProjectChange.source field description (src/basic_memory/schemas/v2/project_changes.py) now names storage.
  • tests/api/v2/test_project_changes_router.py adds a test that writes a note straight into the project folder, runs the indexer, and reads it back from /changes with source storage. Another test checks the published description and the source field.
  • docs/DOMAIN_MODEL.md says how generated wiki pages reach the journal and which rule keeps them out of note history.

Verification

  • SQLite, local: tests/index, tests/indexing, and the wiki and OKF tests in tests/cli and tests/services: 1041 passed, 1 skipped. tests/okf and the changes route tests also passed.
  • PostgreSQL, local: the OKF, changes route, storage journal and wiki projector tests: 162 passed.
  • ruff is clean. ty reports only the 4 known pymilvus import errors.
  • GitHub, head 62efa59: all 41 check runs passed, and license/cla passed. SQLite lifecycle probes filters on paths, so it skipped the empty sign-off commit. It ran on ca34229 and passed.

@groksrc groksrc removed the review:questions Automated review found blocking questions or required changes label Nov 16, 2026

@groksrc groksrc left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: no blocking findings

Verdict: review:done for PR #2121 at head 62efa590. The storage journal paths preserve the ticket's transaction, ordering, and materialization invariants, and the two prior review findings are fixed.

What I verified

  • record_storage_note_changes writes Markdown storage changes with source="storage", claims consecutive project positions last, and keeps each row in the database transaction that accepts the change (src/basic_memory/indexing/storage_note_changes.py:159).
  • NoteContentReconciler journals only successful bootstraps and applied promotions. Deferred, stale, synchronized, and rollback-guard-refused outcomes write no row (src/basic_memory/indexing/note_content_reconciler.py:498).
  • Move and delete batches preserve the NoteContent-before-Entity lock order. They capture deleted identities before deletion and journal after the database mutation (src/basic_memory/indexing/project_index_maintenance.py:773, src/basic_memory/indexing/project_index_maintenance.py:1189).
  • A move with a planned frontmatter rewrite starts unsettled. It settles only after storage returns the planned checksum. A failed or divergent write remains deferred until a later storage version supersedes it (src/basic_memory/indexing/project_index_maintenance.py:1007).
  • Generated Wiki index.md and log.md rows stay out of Wiki and Open Knowledge Format history through the shared projector classification (src/basic_memory/indexing/wiki_projector.py:795, src/basic_memory/okf/export.py:69). The public project-changes description now documents storage rows.
  • I found no separate security issue and no board-only identifier in code, comments, documentation, or tests.

Verification

  • Reviewed current PR head 62efa590 against current origin/main at c3bf1e26. git merge-tree --write-tree origin/main HEAD completed without a conflict.
  • Focused journal, reconciliation, move/delete, Wiki, Open Knowledge Format, API, and repository tests: 263 passed in 66.05s.
  • Ruff passed for every changed Python file. git diff --check and the board-vocabulary scan passed.
  • GitHub reports 43 successful status checks with no failed or pending check.

Next step: Shipright reads the review:done label and advances the ticket. This review does not approve, merge, push, or move the ticket.

@groksrc groksrc added the review:done Automated review found no blocking issues label Nov 16, 2026
@groksrc
groksrc merged commit 8a9e3ea into main Nov 16, 2026
44 checks passed
@groksrc
groksrc deleted the shipright/BMEM-116 branch November 16, 2026 01:02
groksrc added a commit that referenced this pull request Nov 18, 2026
…indows (#2129)

## Summary

Fixes the one test from #2121 that fails on every Windows run of `main`.
The product code is fine. The test expected a content-neutral rename,
but the test config made indexing rewrite the file during the rename.
This PR also makes pull requests run the test file on Windows, so this
kind of failure shows up before merge.

## What failed


`tests/indexing/test_storage_change_journal.py::test_storage_rename_leaves_one_moved_row`
fails on the Windows SQLite unit shard of `main`, most recently in run
[44118870413](https://github.com/basicmachines-co/basic-memory/actions/runs/44118870413)
at `b4b2c226`:

```
>       [moved] = await journal_after(session_maker, test_project, created.partition_position)
E       ValueError: too many values to unpack (expected 1)
```

## Why

- The shared test config sets `update_permalinks_on_move=True`
(`tests/conftest.py:308`). When the rename is indexed, that rewrites the
moved note's `permalink` frontmatter.
- `write_file_atomic` writes text with the platform's newline
(`src/basic_memory/file_utils.py:138-146`), so on Windows the rewrite is
stored with CRLF line endings. That behaviour is deliberate.
- The stored checksum then differs from the planned checksum. The
`moved` row stays unsettled, and the same index run promotes the stored
bytes as a new version, which journals an `updated` row. This is the
designed recovery path from #2121, not a bug.
- So on Windows the test saw two rows after the rename (`moved`,
`updated`) where it expected one.

## What changed

- **The test.** `test_storage_rename_leaves_one_moved_row` now turns
`update_permalinks_on_move` off, so the rename really leaves the file's
bytes unchanged. It also asserts that the `moved` row carries the note's
existing `db_version` and `db_checksum`, which pins down "unchanged
content". The rewrite path is already covered by
`test_scan_move_keeps_a_rewritten_note_row_unsettled_when_storage_holds_other_bytes`
and `..._when_its_write_fails` in
`tests/indexing/test_project_index_maintenance.py`.
- **CI.** `test.yml` runs the full Windows unit suite only on `main`.
This PR adds `tests/indexing/test_storage_change_journal.py` to the
`test-windows-platform` job's file list, so pull requests run it on
Windows.

No product code changes.

## Verification

All local runs on macOS, from `origin/main` at `b4b2c226`:

- **Reproduced the Windows failure before the fix.** I used a scratch
pytest plugin, kept outside the repo, that makes `write_file_atomic`
write CRLF the way Windows does. The old test failed with `ValueError:
too many values to unpack (expected 1, got 2)`.
- **After the fix, with CRLF forced:**
`tests/indexing/test_storage_change_journal.py`, `4 passed`.
- **After the fix, native:** SQLite `4 passed`; PostgreSQL
(`BASIC_MEMORY_TEST_POSTGRES=1`, testcontainers) `4 passed`.
- `ruff check`, `ruff format --check` and `ty check` pass on the changed
test. `test.yml` still parses as YAML.
- The Windows job on this PR is the proof on real Windows.

## Out of scope


`tests/mcp/test_hook_brief_search.py::test_session_start_never_imports_the_mcp_tool_stack`
also fails on some Windows runs of `main`. It comes from #2113 and is
not touched here.

Ticket: BMEM-116 (Drew's board).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Drew Cain <groksrc@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review:done Automated review found no blocking issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant