Skip to content

fix(core): journal a delete for every note a folder delete removes - #2120

Merged
groksrc merged 4 commits into
mainfrom
shipright/BMEM-117
Nov 15, 2026
Merged

groksrc merged 4 commits into
mainfrom
shipright/BMEM-117

Conversation

@groksrc

@groksrc groksrc commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

Summary

A folder delete now writes one deleted row per removed Markdown note into the project's change journal. Before this change it wrote none. The wiki log, OKF export history and the /v2/projects/{id}/changes feed all missed those deletes and kept treating the notes as present.

Each row is written in the delete's own transaction and settled once its file cleanup has run. A row left unsettled would defer every later wiki projection for the project, which is the failure in GitHub #1625.

Board ticket: BMEM-117.

What changed

  • Rows written with the delete. RepositoryDirectoryDeleteAcceptanceStore.delete_directory_entities reads each note's db_version and db_checksum during the lock-check read it already did. It must read them first, because note_content cascades away with the entity. The guarded DELETE … RETURNING now returns id, external id, permalink, title, file path and content type. The new record_directory_delete_project_changes then claims positions and inserts the rows (src/basic_memory/indexing/directory_delete_runner.py).
  • Row shape. Positions are consecutive and follow file-path order. Each row has operation=deleted and previous_file_path=NULL, and all rows in a batch share one accepted_at. The source is the new ACCEPTED_DIRECTORY_DELETE_SOURCE = "delete_directory". Actor fields stay NULL, because the folder-delete route carries no actor today.
  • What gets no row. Non-Markdown files get no row, the same as a single non-Markdown delete. Notes the guarded delete leaves in place also get none. A delete that removes no Markdown note claims no position. A Markdown note with no permalink is deleted without a row, with a warning: see below.
  • Journal last. The position claim and insert run after the search-index and vector cleanup. The project-row lock is therefore held only to the end of the transaction, in the same NoteContent → Entity → Project order as the single-note delete.
  • Rows carried to cleanup. The changes travel on DirectoryEntityDeleteResult and DirectoryDeleteAcceptance, keyed by entity id. RuntimeDirectoryFileSnapshot.to_pending_note_file_delete takes a project_change argument, so each RuntimeNoteFileDeleteJobRequest carries its row. Cloud's existing delete_note_file job already settles a request's project_change.
  • Settling where no job will. enqueue_directory_file_delete_jobs collects the rows of files whose cleanup ran inline (deleted, missing or skipped) or failed to enqueue. They reach DirectoryDeleteAcceptedResult.project_changes_to_settle, and DirectoryDeleteService.delete_directory settles them in one statement. A queued cleanup (the enqueuer returns None, as in Cloud) is left to its job. run_directory_delete settles the same way, in its own session.
  • Why a failed or skipped cleanup still settles. The entity delete is final. A file left on disk is re-indexed later as a new note, so waiting on it would stall projection for good.
  • Repository. ProjectRepository has three new methods (src/basic_memory/repository/project_repository.py):
    • advance_partition_position_by(count) claims N positions in one UPDATE … RETURNING.
    • record_accepted_note_changes does one bulk insert.
    • mark_accepted_note_changes_materialized settles a list of positions with the same rule as the single-position settle: each target, plus older unsettled rows of the same note. It works in chunks of 500 positions.

The protocol DirectoryDeleteAcceptanceStore.delete_directory_entities gains a required keyword, project_external_id. Cloud uses core's RepositoryDirectoryDeleteAcceptanceStore and DirectoryDeleteService, so it needs no code change for the journal rows. It picks this up when its core pin moves.

Notes with no permalink

Databases from before the disable_permalinks setting was removed (#313 added it) can still hold Markdown notes with no permalink. No migration backfilled them. A journal row needs a permalink, so no journal row ever named these notes.

Raising on them would roll back the whole folder delete, where today it succeeds. CI caught this: test-int/read_cache/test_runtime_invalidation.py seeds such notes and failed with RuntimeError: Accepted note is missing permalink for entity_id=1. So the folder delete now removes them without a journal row and logs a warning. test_directory_delete_skips_journal_row_for_note_without_permalink covers this. The single-note delete still raises, unchanged.

Verification

All commands ran in this branch's worktree on macOS.

  • SQLite: BASIC_MEMORY_ENV=test uv run pytest tests/indexing/test_directory_delete_runner.py tests/repository/test_project_partition_repository.py tests/cloud/test_cloud_services.py tests/mcp/test_wiki_after_api_writes.py → 57 passed.
  • Postgres (testcontainers): the same command with BASIC_MEMORY_TEST_POSTGRES=1, without the wiki test → 56 passed.
  • Lock order: BASIC_MEMORY_TEST_POSTGRES=1 … pytest test-int/test_relation_generation_concurrency.py → 4 passed. It skips on SQLite by design.
  • Related routes: pytest tests/api/v2/test_knowledge_router.py tests/api/v2/test_delete_unresolves_inbound_relations.py tests/mcp/test_tool_bm_delete.py tests/services/test_entity_service.py -k "directory or folder" → 25 passed.
  • After the permalink change: pytest tests/indexing/test_directory_delete_runner.py test-int/read_cache/test_runtime_invalidation.py → 43 passed on Postgres, and the directory-delete read-cache tests pass on SQLite.
  • ruff check and ruff format are clean. ty check src tests test-int reports only the 4 pymilvus import errors from my local environment. CI installs the milvus extra.

New tests:

  • test_directory_delete_journals_each_markdown_note_and_carries_rows_to_cleanup: rows at positions 6 and 7 after head 5, in path order, with their fields and revisions. The PNG gets no row. Each cleanup request carries its row, and queued rows stay unsettled.
  • test_directory_delete_journal_rolls_back_with_failed_transaction: a failure after the entity delete keeps the note, the journal and the head unchanged.
  • test_directory_delete_claims_no_position_without_deleted_markdown: an empty folder, and a folder of only non-Markdown files.
  • test_local_directory_delete_settles_every_row_whatever_the_cleanup_outcome: the real local enqueuer against files on disk, with one file deleted, one missing, one changed on disk (skipped) and one whose delete raises OSError. Every row is settled, and inspect_local_wiki_projection reports pending_materialization == ().
  • test_run_directory_delete_settles_inline_cleanup_rows.
  • test_directory_delete_skips_journal_row_for_note_without_permalink.
  • The guarded-delete survivor test and the move-out test now also assert that no position was claimed.
  • Repository tests cover the bulk claim, the bulk insert and the bulk settle rule.

Noted, not changed

  • Under SQLite, accepted_at and materialized_at come back with their UTC wall-clock time labelled as local time. For example, 20:15 UTC reads back as 20:15 CDT. AcceptedProjectNoteChange.__getattribute__ → ensure_timezone_aware treats naive SQLite values as local time. The single-row ORM insert and the new bulk insert behave identically: I checked both with a throwaway probe. So this PR does not cause it.
  • In Cloud, each folder-delete row's delete_note_file job will schedule a wiki projection once the core pin moves. That is one projection request per note in the folder. Whether those requests coalesce is a question for the Cloud pin bump.

🤖 Generated with Claude Code

A folder delete removed its notes but wrote no rows to the project's change
journal. Every journal consumer (the wiki log, OKF export history and the
/v2/projects/{id}/changes feed) kept treating those notes as present.

The guarded bulk delete now returns each removed entity's identity. In the
same transaction it claims one block of consecutive journal positions and
writes one `deleted` row per Markdown note, in file-path order, with source
"delete_directory". Non-Markdown files and notes the guard leaves in place
get no row, and a delete that removes no note claims no position.

Each row rides on its file's cleanup request as project_change, so Cloud's
existing delete-file job settles it. Where no queued job exists (local
inline cleanup of any outcome, or a failed enqueue), DirectoryDeleteService
settles the rows itself, so they cannot stall wiki projection (GitHub #1625).

ProjectRepository gains advance_partition_position_by, a bulk journal
insert, and a bulk settle that follows the single-position settle rule.

Board ticket: BMEM-117.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-15T06:55:51.231860Z 13cfd67 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 13cfd67f22

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +113 to +115
if result.project_changes_to_settle:
async with db.scoped_session(self.session_maker) as session:
await settle_directory_delete_project_changes(session, result)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Recover unsettled directory deletes after interrupted cleanup

If a local folder delete is cancelled during inline cleanup (for example, Ctrl-C during a large deletion), the entity deletions and journal rows have already committed, but settlement is skipped because it runs only after the entire cleanup loop returns. Even notes whose files were successfully deleted retain materialized_at=NULL, permanently blocking subsequent wiki projections in plan_wiki_projection. Startup recovery cannot repair these rows: recover_stuck_materializations enumerates NoteContent, which this delete has cascaded away; retrying the folder delete also finds no entities for the removed files. Add a recovery path for these committed delete rows so interruption cannot leave the project's wiki projection permanently stalled.

AGENTS.md reference: AGENTS.md:L185-L187

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9f72e87. Local startup recovery (recover_project_materializations) now settles every unsettled deleted journal row through settle_interrupted_note_deletes, so an interrupted cleanup can no longer stall wiki projection. Details: #2120 (comment)

groksrc and others added 2 commits November 15, 2026 01:11
The read-cache integration tests failed in CI with "Accepted note is missing
permalink". Databases from before the disable_permalinks setting was removed
can still hold Markdown notes with no permalink. A journal row needs one, so
raising there rolled back the whole folder delete.

Such a note was never journaled, so the folder delete now removes it without
a journal row and logs a warning. Every other note is journaled as before.

DCO Remediation Commit for Drew Cain <groksrc@gmail.com>

I, Drew Cain <groksrc@gmail.com>, hereby add my Signed-off-by to this commit: 13cfd67f224b5b53f03c4c52daf1e0bdc8c3b7a5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Drew Cain <groksrc@gmail.com>
The previous commit named the wrong SHA in its remediation line.

DCO Remediation Commit for Drew Cain <groksrc@gmail.com>

I, Drew Cain <groksrc@gmail.com>, hereby add my Signed-off-by to this commit: 13cfd67

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Drew Cain <groksrc@gmail.com>

@groksrc groksrc left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: questions. The transactional journal write, consecutive position claim, cleanup propagation, and local settlement paths are otherwise consistent with the ticket and repository invariants.

Blocking question — preserve one journal row per deleted Markdown note (src/basic_memory/indexing/directory_delete_runner.py:468). The new permalink is None branch deletes a legacy Markdown note but deliberately omits its deleted row. That leaves the exact journal gap this ticket is meant to close: consumers never observe that delete, and a folder with N Markdown notes can advance the partition by fewer than N positions. It also reverses the ticket’s explicit fail-fast rule for a missing permalink and differs from the existing single-note delete.

Please either keep the delete atomic by raising before the position claim, as the accepted ticket specifies, or obtain an explicit scope decision and implement a valid journal identity for legacy notes. The regression test should assert the chosen complete-journal behavior instead of accepting a silent omission.

Verified at head ac2c53799ee9756631adf1bbae4794e1ef047e42: all GitHub checks are green and the PR is mergeable. The focused SQLite review run passed 37 tests in tests/indexing/test_directory_delete_runner.py and tests/repository/test_project_partition_repository.py. Next step: the PR author resolves the legacy no-permalink behavior, then requests another review.

@groksrc groksrc added the review:questions Automated review found blocking questions or required changes label Nov 15, 2026
…eletes

Answers the review on this PR.

A folder delete again fails, and rolls back, when it would remove a
Markdown note that has no permalink. A journal row needs a permalink, and
skipping the row left consumers treating the note as present. This matches
the single-note delete. The two read-cache tests that seeded such notes now
give them a permalink.

Local startup recovery now settles `deleted` journal rows left unsettled
when the process exited between a delete's commit and its inline cleanup.
note_content is gone by then, so materialization recovery could not reach
them, and one unsettled row defers every later wiki projection
(GitHub #1625). The same gap affected single-note deletes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Drew Cain <groksrc@gmail.com>
@groksrc

groksrc commented Nov 15, 2026

Copy link
Copy Markdown
Member Author

Review answered in commit 9f72e87

Both findings are fixed on this branch.

1. A note with no permalink got no journal row (blocking, directory_delete_runner.py:468). Fixed by going back to fail-fast, as the review asked.

  • record_directory_delete_project_changes now raises RuntimeError("Accepted note is missing permalink …") before it claims any position. The delete's transaction rolls back, so no entity, journal row or position changes. This matches the single-note delete.
  • The test test_directory_delete_without_permalink_rolls_back_whole_delete replaces the old skip test. It asserts that both notes and their note content survive, the head stays put, no journal row is written and no cleanup is queued.
  • The two read-cache integration tests that hit this in CI only left the permalink out of their seed data. They now seed one (test-int/read_cache/test_runtime_invalidation.py).

2. An interrupted cleanup leaves rows unsettled for good (Codex, directory_deletes.py:115). Fixed with a local startup recovery step.

  • New settle_interrupted_note_deletes in index/note_content_materialization.py, backed by ProjectRepository.settle_unmaterialized_deletes. It settles every unsettled deleted row in the project, plus older unsettled rows of the same note, using the existing per-position rule.
  • recover_project_materializations runs it first, on every local startup and before bm wiki rebuild. The local runtime has no durable cleanup queue, so nothing else would ever settle these rows. The entity delete is final whatever happened to the file.
  • The same gap affected single-note deletes, which settle after their inline cleanup in the same way. This step covers both.
  • Tests: test_startup_recovery_settles_rows_of_an_interrupted_folder_delete cancels a folder delete mid-cleanup, then runs startup recovery. The deleted rows end settled, and a surviving note's unsettled row is left alone. test_settle_unmaterialized_deletes_retires_deleted_notes_only covers the repository method.

Verified locally:

  • SQLite: tests/indexing/test_directory_delete_runner.py, tests/repository/test_project_partition_repository.py and tests/services/test_initialization.py, 52 passed. The read-cache directory-delete tests, 2 passed.
  • Postgres (BASIC_MEMORY_TEST_POSTGRES=1): the same files plus test-int/read_cache/test_runtime_invalidation.py, 71 passed.
  • ruff check and ty check on the changed files are clean.

@groksrc groksrc removed the review:questions Automated review found blocking questions or required changes label Nov 15, 2026

@groksrc groksrc left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: done. I found no blocking issues at head 9f72e87556ec07b7ca271ec63f16fb8614953877.

The prior blocking question is resolved: a Markdown note without a permalink now raises inside the delete transaction, so the entity deletes, journal rows, and partition position all roll back together. The interrupted-cleanup finding is also resolved: local startup recovery settles committed deleted rows that no cleanup path can reach, without settling unrelated note changes.

The transaction, consecutive position allocation, journal identity, cleanup propagation, and local settlement behavior match the ticket. GitHub reports the pull request mergeable. All required checks are successful; the optional Claude check is skipped.

Verification: BASIC_MEMORY_ENV=test uv run pytest tests/indexing/test_directory_delete_runner.py tests/repository/test_project_partition_repository.py tests/services/test_initialization.py passed 52 tests locally. Next step: the ticket owner can continue the pipeline.

@groksrc groksrc added the review:done Automated review found no blocking issues label Nov 15, 2026
@groksrc
groksrc merged commit c853f5f into main Nov 15, 2026
54 checks passed
@groksrc
groksrc deleted the shipright/BMEM-117 branch November 15, 2026 10:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review:done Automated review found no blocking issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant