Skip to content

pathlib migration: migrate importer to use pathlib.Path - #6994

Open
snejus wants to merge 3 commits into
masterfrom
migrate-importer-to-pathlib
Open

snejus wants to merge 3 commits into
masterfrom
migrate-importer-to-pathlib

Conversation

@snejus

@snejus snejus commented Sep 5, 2026 •

Copy link
Copy Markdown
Member

Depends on: #6991

Pathlib Migration: Importer Paths 🛠️

This continues the pathlib migration by moving the whole importer - session, tasks, state, stages and the import command - from raw bytes paths to pathlib.Path. The PathBytes alias is gone, along with most of the syspath / displayable_path / os.fsencode noise around it.

Key Changes

1. sorted_walk and albums_in_dir work with Path 📁

  • Change: both used to be generic over AnyStr, which forced albums_in_dir to keep a duplicate set of bytes multi-disc regexes (MULTIDISC_BYTES_PATTERNS) and to encode ignore patterns on the fly. Internally they now deal with Path only.
  • Change: sorted_walk uses Path.iterdir() and Path.match() for ignore rules. Yielded dirs / files are still relative to root.
  • Impact: MULTIDISC_BYTES_PATTERNS is removed, is_subdir_of_any_in_list uses Path.parents instead of util.ancestry, and fetchart no longer converts to str before walking.

2. sorted_walk compatibility shim 🧩

  • Purpose: beets-filetote and beets-copyartifacts call sorted_walk with bytes paths (and bytes ignore patterns in filetote's case) and then concatenate the results as bytes.
  • Compatibility: str and bytes paths and bytes ignore patterns keep working. Results come back in the representation of the given path, with a DeprecationWarning until 3.0.0 - same approach as the ArtResizer and art utils migration.

3. Importer uses Path throughout 🔄

  • Change: ImportSession.paths, ImportTask.toppath / paths / old_paths, replaced_albums keys, ImportTaskFactory, ArchiveImportTask and ImportState are all typed as Path.
  • Change: tasks built from library items use item.filepath / album.filepath instead of item.path.
  • Change: CLI and --from-logfile paths are normalised into Path objects up front, so the separate byte_paths / byte_paths_from_logfiles lists are gone.
  • Change: extension.fix_extension and extension.remux_mpeglayer3_wav take and return Path.

4. State file conversion 💾

  • Change: ImportState converts tagprogress and taghistory entries to Path when the state file is read.
  • Compatibility: state files written by previous versions contain bytes paths. Without the conversion incremental imports would re-import every previously imported directory and interrupted imports would not offer to resume.

5. Plugins and events 🔌

Module Before After
import event paths list[bytes] list[Path]
convert _temp_files list[bytes] list[Path]
importadded replaced_album_paths set[bytes] set[Path]
mbsubmit displayable_path(p) passes Path to subprocess directly
replaygain import_end list[bytes] list[Path]

Testing Strategy 🧪

  1. Handle only pathlib.Path in sorted_walk, albums_in_dir
    • test/test_util.py: sorted_walk tests updated for Path input and output.
    • test/test_importer.py: albums_in_dir tests use Path.
  2. typing: Use pathlib.Path in beets.importer
    • test/test_importer.py: updated for Path signatures, plus TestImportState covering conversion of a legacy bytes state file.
    • test/ui/commands/test_import.py, test/plugins/test_convert.py, test/util/test_extension.py updated for the new signatures.
    • beets/test/helper.py and beets/test/_common.py pass Path objects to the import session instead of os.fsencode(...).
  3. Add compatibility shim for sorted_walk
    • test/test_util.py: str and bytes paths and bytes ignore patterns return results in the given representation and emit a DeprecationWarning.

Notes 📝

  • Breaking for plugins: import event listeners and anything reading task.paths, task.toppath or task.old_paths now receive Path instead of bytes. There's no shim for these, same as Library.directory in pathlib migration: migrate Library.directory to pathlib.Path #6991. Of the plugins listed in our docs, beets-originquery (os.path.commonpath(task.paths).decode(...)) will break.
  • PathBytes is removed from beets.util and beets.util.extension.
  • Changelog entries are under "For plugin developers", and docs/dev/plugins/events.rst is updated for the import event.

Copilot AI lite review requested due to automatic review settings September 5, 2026 20:08
@snejus
snejus requested a review from a team as a code owner September 5, 2026 20:08
@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown

Thank you for the PR! The changelog has not been updated, so here is a friendly reminder to check if you need to add an entry.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

grug see few real behavior bugs/regressions (import state ordering/legacy decode, CLI path normalization, fetchart path join, archive cleanup check) that can break resume/import/art scan.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

PR move importer world from bytes path to pathlib.Path, so one path type go through importer session/tasks/state/UI and importer-facing plugins.

Changes:

  • Switch importer task/session/stage/state APIs from bytes/str paths to Path.
  • Update util + plugins to use Path ops (.exists(), .is_file(), .rglob()) instead of os.path + encode/decode.
  • Update tests and helpers to assert Path values at runtime boundary.
File summaries
File Description
test/ui/commands/test_import.py Update logfile parsing test to expect Path outputs.
test/test_util.py Update sorted_walk tests for Path roots and relative Path entries.
test/test_importer.py Update importer tests to pass/compare Path and use Path joins.
test/plugins/test_convert.py Update convert plugin test messaging for Path paths.
beetsplug/replaygain.py Update import_end hook signature to list[Path].
beetsplug/mbsubmit.py Pass Path task paths through picard launch flow.
beetsplug/importadded.py Update replaced-album path tracking to Path and use album.filepath.
beetsplug/fetchart.py Use sorted_walk with Path inputs while collecting candidate images.
beetsplug/convert.py Track temp files as Path and cleanup via Path.is_file().
beets/util/extension.py Make extension fix/remux helpers operate on Path only.
beets/util/init.py Change sorted_walk to take/return Path values and simplify ignore logic.
beets/ui/commands/import_/session.py Update resume prompt to accept Path.
beets/ui/commands/import_/init.py Change import command plumbing to Path paths + logfile parsing yields Path.
beets/test/helper.py Update import session fixtures to pass Path paths.
beets/test/_common.py Update _common.import_session helper to accept Sequence[Path].
beets/importer/tasks.py Convert task objects + archive handling + directory discovery to Path.
beets/importer/state.py Convert import state file path handling to Path and update state APIs to Path.
beets/importer/stages.py Update pipeline stages to log and build tasks with Path.
beets/importer/session.py Update import session bookkeeping/merge tracking/resume to Path.
beets/events.py Update importer event typed args to expose Path list.
Review details

Suppressed comments (1)

beets/importer/state.py:78

  • grug see state._open load pickle then assign tagprogress/taghistory direct. old state file likely have bytes/str, but new code expect Path. this break "legacy import state remains readable" from issue/PR text. grug want coerce loaded keys/values to pathlib.Path (decode bytes) at load boundary.
            with self.path.open("rb") as f:
                state = pickle.load(f)
                # Read the states
                self.tagprogress = state.get("tagprogress", {})
                self.taghistory = state.get("taghistory", set())
  • Files reviewed: 20/20 changed files
  • Comments generated: 5
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread beets/importer/state.py Resolved
Comment thread beets/importer/tasks.py Resolved
Comment thread beets/ui/commands/import_/__init__.py Resolved
Comment thread beetsplug/fetchart.py Resolved
Comment thread beetsplug/importadded.py Resolved
@codecov

codecov Bot commented Sep 5, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.34356% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.87%. Comparing base (ee6e08a) to head (2c11dc8).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
beets/ui/commands/import_/__init__.py 33.33% 10 Missing ⚠️
beets/importer/tasks.py 94.54% 2 Missing and 1 partial ⚠️
beets/util/__init__.py 86.36% 2 Missing and 1 partial ⚠️
beets/importer/state.py 96.42% 0 Missing and 1 partial ⚠️
beetsplug/convert.py 66.66% 0 Missing and 1 partial ⚠️
beetsplug/mbsubmit.py 50.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6994      +/-   ##
==========================================
- Coverage   81.02%   80.87%   -0.16%     
==========================================
  Files         150      150              
  Lines       21228    21221       -7     
  Branches     3242     3241       -1     
==========================================
- Hits        17201    17163      -38     
- Misses       3212     3245      +33     
+ Partials      815      813       -2     
Files with missing lines Coverage Δ
beets/events.py 100.00% <100.00%> (ø)
beets/importer/session.py 93.57% <100.00%> (-0.05%) ⬇️
beets/importer/stages.py 95.38% <100.00%> (+0.02%) ⬆️
beets/ui/commands/import_/session.py 72.76% <100.00%> (ø)
beets/util/extension.py 83.05% <100.00%> (-0.83%) ⬇️
beetsplug/fetchart.py 74.11% <100.00%> (-0.11%) ⬇️
beetsplug/importadded.py 94.25% <100.00%> (ø)
beetsplug/replaygain.py 76.12% <100.00%> (ø)
beets/importer/state.py 92.00% <96.42%> (+0.69%) ⬆️
beetsplug/convert.py 76.04% <66.66%> (ø)
... and 4 more

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@snejus
snejus force-pushed the migrate-importer-to-pathlib branch from 658cb4f to 370eb44 Compare October 9, 2026 00:34
@snejus
snejus force-pushed the migrate-importer-to-pathlib branch from 370eb44 to 1097461 Compare October 9, 2026 00:50
@snejus
snejus force-pushed the migrate-importer-to-pathlib branch 2 times, most recently from efce878 to fa1d3fc Compare October 12, 2026 21:23
@snejus
snejus force-pushed the migrate-importer-to-pathlib branch from fa1d3fc to 60569a1 Compare October 26, 2026 08:39
@snejus
snejus added this pull request to stack #7115 October 27, 2026 04:09
@snejus
snejus force-pushed the migrate-importer-to-pathlib branch from 60569a1 to a6f89bc Compare October 27, 2026 12:08
@snejus
snejus force-pushed the migrate-importer-to-pathlib branch from a6f89bc to 8a71663 Compare November 1, 2026 14:05
@snejus
snejus force-pushed the migrate-importer-to-pathlib branch from 8a71663 to fe3d555 Compare November 1, 2026 14:27
Base automatically changed from migrate-library-directory-path to master November 1, 2026 14:29
@snejus
snejus force-pushed the migrate-importer-to-pathlib branch 2 times, most recently from b1dfe2e to dff4065 Compare November 1, 2026 14:35
@snejus
snejus force-pushed the migrate-importer-to-pathlib branch from dff4065 to 33cdb90 Compare November 1, 2026 16:00

@semohr semohr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Two small requests/additions.

Comment thread beets/importer/tasks.py

all_files_imported = move and not any(
files for _, _, files in os.walk(util.syspath(self.toppath))
p.is_file() for p in self.toppath.rglob("*")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldnt this be something like this?

all_files_imported = move and not any(
    files for _, _, files in self.toppath.walk()
)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pathlib.Path does not have such a method :(

Comment thread beets/importer/state.py Outdated
# Read the states
self.tagprogress = state.get("tagprogress", {})
self.taghistory = state.get("taghistory", set())
# Older state files store `bytes` paths. Re-sort the progress

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems a bit complicated imo. Also I dont really understand why we need a dict now. Can you elaborate?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was needed because I changed tagprocess and taghistory to store paths as Path instead of bytes so everything had to be re-sorted.

I kinda changed my mind now and reverted this thing - I think it may make more sense (especially regarding compatibility) to continue storing these paths as bytes?

@snejus
snejus force-pushed the migrate-importer-to-pathlib branch 3 times, most recently from ecb44eb to 5e29dad Compare November 15, 2026 19:47
@snejus
snejus force-pushed the migrate-importer-to-pathlib branch from 5e29dad to 2c11dc8 Compare November 15, 2026 21:00
@snejus

snejus commented Nov 15, 2026

Copy link
Copy Markdown
Member Author

@semohr I'm gonna create a separate master-like branch for pathlib migration - I realise that this work is going to cause hell for plugins so we'd rather release it as part of v3, methinks.

What do you reckon?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Migrate importer path APIs and state to pathlib.Path

3 participants