Repository navigation
fix(core): journal a delete for every note a folder delete removes - #2120
Conversation
A folder delete removed its notes but wrote no rows to the project's change
journal. Every journal consumer (the wiki log, OKF export history and the
/v2/projects/{id}/changes feed) kept treating those notes as present.
The guarded bulk delete now returns each removed entity's identity. In the
same transaction it claims one block of consecutive journal positions and
writes one `deleted` row per Markdown note, in file-path order, with source
"delete_directory". Non-Markdown files and notes the guard leaves in place
get no row, and a delete that removes no note claims no position.
Each row rides on its file's cleanup request as project_change, so Cloud's
existing delete-file job settles it. Where no queued job exists (local
inline cleanup of any outcome, or a failed enqueue), DirectoryDeleteService
settles the rows itself, so they cannot stall wiki projection (GitHub #1625).
ProjectRepository gains advance_partition_position_by, a bulk journal
insert, and a bulk settle that follows the single-position settle rule.
Board ticket: BMEM-117.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 13cfd67f22
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if result.project_changes_to_settle: | ||
| async with db.scoped_session(self.session_maker) as session: | ||
| await settle_directory_delete_project_changes(session, result) |
There was a problem hiding this comment.
Recover unsettled directory deletes after interrupted cleanup
If a local folder delete is cancelled during inline cleanup (for example, Ctrl-C during a large deletion), the entity deletions and journal rows have already committed, but settlement is skipped because it runs only after the entire cleanup loop returns. Even notes whose files were successfully deleted retain materialized_at=NULL, permanently blocking subsequent wiki projections in plan_wiki_projection. Startup recovery cannot repair these rows: recover_stuck_materializations enumerates NoteContent, which this delete has cascaded away; retrying the folder delete also finds no entities for the removed files. Add a recovery path for these committed delete rows so interruption cannot leave the project's wiki projection permanently stalled.
AGENTS.md reference: AGENTS.md:L185-L187
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 9f72e87. Local startup recovery (recover_project_materializations) now settles every unsettled deleted journal row through settle_interrupted_note_deletes, so an interrupted cleanup can no longer stall wiki projection. Details: #2120 (comment)
The read-cache integration tests failed in CI with "Accepted note is missing permalink". Databases from before the disable_permalinks setting was removed can still hold Markdown notes with no permalink. A journal row needs one, so raising there rolled back the whole folder delete. Such a note was never journaled, so the folder delete now removes it without a journal row and logs a warning. Every other note is journaled as before. DCO Remediation Commit for Drew Cain <groksrc@gmail.com> I, Drew Cain <groksrc@gmail.com>, hereby add my Signed-off-by to this commit: 13cfd67f224b5b53f03c4c52daf1e0bdc8c3b7a5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Drew Cain <groksrc@gmail.com>
The previous commit named the wrong SHA in its remediation line. DCO Remediation Commit for Drew Cain <groksrc@gmail.com> I, Drew Cain <groksrc@gmail.com>, hereby add my Signed-off-by to this commit: 13cfd67 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Drew Cain <groksrc@gmail.com>
groksrc
left a comment
There was a problem hiding this comment.
Verdict: questions. The transactional journal write, consecutive position claim, cleanup propagation, and local settlement paths are otherwise consistent with the ticket and repository invariants.
Blocking question — preserve one journal row per deleted Markdown note (src/basic_memory/indexing/directory_delete_runner.py:468). The new permalink is None branch deletes a legacy Markdown note but deliberately omits its deleted row. That leaves the exact journal gap this ticket is meant to close: consumers never observe that delete, and a folder with N Markdown notes can advance the partition by fewer than N positions. It also reverses the ticket’s explicit fail-fast rule for a missing permalink and differs from the existing single-note delete.
Please either keep the delete atomic by raising before the position claim, as the accepted ticket specifies, or obtain an explicit scope decision and implement a valid journal identity for legacy notes. The regression test should assert the chosen complete-journal behavior instead of accepting a silent omission.
Verified at head ac2c53799ee9756631adf1bbae4794e1ef047e42: all GitHub checks are green and the PR is mergeable. The focused SQLite review run passed 37 tests in tests/indexing/test_directory_delete_runner.py and tests/repository/test_project_partition_repository.py. Next step: the PR author resolves the legacy no-permalink behavior, then requests another review.
…eletes Answers the review on this PR. A folder delete again fails, and rolls back, when it would remove a Markdown note that has no permalink. A journal row needs a permalink, and skipping the row left consumers treating the note as present. This matches the single-note delete. The two read-cache tests that seeded such notes now give them a permalink. Local startup recovery now settles `deleted` journal rows left unsettled when the process exited between a delete's commit and its inline cleanup. note_content is gone by then, so materialization recovery could not reach them, and one unsettled row defers every later wiki projection (GitHub #1625). The same gap affected single-note deletes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Drew Cain <groksrc@gmail.com>
Review answered in commit 9f72e87Both findings are fixed on this branch. 1. A note with no permalink got no journal row (blocking,
2. An interrupted cleanup leaves rows unsettled for good (Codex,
Verified locally:
|
groksrc
left a comment
There was a problem hiding this comment.
Verdict: done. I found no blocking issues at head 9f72e87556ec07b7ca271ec63f16fb8614953877.
The prior blocking question is resolved: a Markdown note without a permalink now raises inside the delete transaction, so the entity deletes, journal rows, and partition position all roll back together. The interrupted-cleanup finding is also resolved: local startup recovery settles committed deleted rows that no cleanup path can reach, without settling unrelated note changes.
The transaction, consecutive position allocation, journal identity, cleanup propagation, and local settlement behavior match the ticket. GitHub reports the pull request mergeable. All required checks are successful; the optional Claude check is skipped.
Verification: BASIC_MEMORY_ENV=test uv run pytest tests/indexing/test_directory_delete_runner.py tests/repository/test_project_partition_repository.py tests/services/test_initialization.py passed 52 tests locally. Next step: the ticket owner can continue the pipeline.
Summary
A folder delete now writes one
deletedrow per removed Markdown note into the project's change journal. Before this change it wrote none. The wiki log, OKF export history and the/v2/projects/{id}/changesfeed all missed those deletes and kept treating the notes as present.Each row is written in the delete's own transaction and settled once its file cleanup has run. A row left unsettled would defer every later wiki projection for the project, which is the failure in GitHub #1625.
Board ticket: BMEM-117.
What changed
RepositoryDirectoryDeleteAcceptanceStore.delete_directory_entitiesreads each note'sdb_versionanddb_checksumduring the lock-check read it already did. It must read them first, becausenote_contentcascades away with the entity. The guardedDELETE … RETURNINGnow returns id, external id, permalink, title, file path and content type. The newrecord_directory_delete_project_changesthen claims positions and inserts the rows (src/basic_memory/indexing/directory_delete_runner.py).operation=deletedandprevious_file_path=NULL, and all rows in a batch share oneaccepted_at. The source is the newACCEPTED_DIRECTORY_DELETE_SOURCE = "delete_directory". Actor fields stay NULL, because the folder-delete route carries no actor today.DirectoryEntityDeleteResultandDirectoryDeleteAcceptance, keyed by entity id.RuntimeDirectoryFileSnapshot.to_pending_note_file_deletetakes aproject_changeargument, so eachRuntimeNoteFileDeleteJobRequestcarries its row. Cloud's existingdelete_note_filejob already settles a request'sproject_change.enqueue_directory_file_delete_jobscollects the rows of files whose cleanup ran inline (deleted, missing or skipped) or failed to enqueue. They reachDirectoryDeleteAcceptedResult.project_changes_to_settle, andDirectoryDeleteService.delete_directorysettles them in one statement. A queued cleanup (the enqueuer returnsNone, as in Cloud) is left to its job.run_directory_deletesettles the same way, in its own session.ProjectRepositoryhas three new methods (src/basic_memory/repository/project_repository.py):advance_partition_position_by(count)claims N positions in oneUPDATE … RETURNING.record_accepted_note_changesdoes one bulk insert.mark_accepted_note_changes_materializedsettles a list of positions with the same rule as the single-position settle: each target, plus older unsettled rows of the same note. It works in chunks of 500 positions.The protocol
DirectoryDeleteAcceptanceStore.delete_directory_entitiesgains a required keyword,project_external_id. Cloud uses core'sRepositoryDirectoryDeleteAcceptanceStoreandDirectoryDeleteService, so it needs no code change for the journal rows. It picks this up when its core pin moves.Notes with no permalink
Databases from before the
disable_permalinkssetting was removed (#313 added it) can still hold Markdown notes with no permalink. No migration backfilled them. A journal row needs a permalink, so no journal row ever named these notes.Raising on them would roll back the whole folder delete, where today it succeeds. CI caught this:
test-int/read_cache/test_runtime_invalidation.pyseeds such notes and failed withRuntimeError: Accepted note is missing permalink for entity_id=1. So the folder delete now removes them without a journal row and logs a warning.test_directory_delete_skips_journal_row_for_note_without_permalinkcovers this. The single-note delete still raises, unchanged.Verification
All commands ran in this branch's worktree on macOS.
BASIC_MEMORY_ENV=test uv run pytest tests/indexing/test_directory_delete_runner.py tests/repository/test_project_partition_repository.py tests/cloud/test_cloud_services.py tests/mcp/test_wiki_after_api_writes.py→57 passed.BASIC_MEMORY_TEST_POSTGRES=1, without the wiki test →56 passed.BASIC_MEMORY_TEST_POSTGRES=1 … pytest test-int/test_relation_generation_concurrency.py→4 passed. It skips on SQLite by design.pytest tests/api/v2/test_knowledge_router.py tests/api/v2/test_delete_unresolves_inbound_relations.py tests/mcp/test_tool_bm_delete.py tests/services/test_entity_service.py -k "directory or folder"→25 passed.pytest tests/indexing/test_directory_delete_runner.py test-int/read_cache/test_runtime_invalidation.py→43 passedon Postgres, and the directory-delete read-cache tests pass on SQLite.ruff checkandruff formatare clean.ty check src tests test-intreports only the 4pymilvusimport errors from my local environment. CI installs themilvusextra.New tests:
test_directory_delete_journals_each_markdown_note_and_carries_rows_to_cleanup: rows at positions 6 and 7 after head 5, in path order, with their fields and revisions. The PNG gets no row. Each cleanup request carries its row, and queued rows stay unsettled.test_directory_delete_journal_rolls_back_with_failed_transaction: a failure after the entity delete keeps the note, the journal and the head unchanged.test_directory_delete_claims_no_position_without_deleted_markdown: an empty folder, and a folder of only non-Markdown files.test_local_directory_delete_settles_every_row_whatever_the_cleanup_outcome: the real local enqueuer against files on disk, with one file deleted, one missing, one changed on disk (skipped) and one whose delete raisesOSError. Every row is settled, andinspect_local_wiki_projectionreportspending_materialization == ().test_run_directory_delete_settles_inline_cleanup_rows.test_directory_delete_skips_journal_row_for_note_without_permalink.Noted, not changed
accepted_atandmaterialized_atcome back with their UTC wall-clock time labelled as local time. For example, 20:15 UTC reads back as 20:15 CDT.AcceptedProjectNoteChange.__getattribute__→ensure_timezone_awaretreats naive SQLite values as local time. The single-row ORM insert and the new bulk insert behave identically: I checked both with a throwaway probe. So this PR does not cause it.delete_note_filejob will schedule a wiki projection once the core pin moves. That is one projection request per note in the folder. Whether those requests coalesce is a question for the Cloud pin bump.🤖 Generated with Claude Code