[yarikoptic-gitmate]

The goal is to run the mirror update in CI instead of the drogon cron, working from a checkout of the superdataset with none of the ~1400 mirrors installed. This PR has two parts:

• The plan: docs/ci-mirroring-plan.md, now v3. It includes the review findings on v2 and the changes that landed on main since.
• A prototype: the backups2datalad side of the plan, plus the workflow around it.

The workflow only wraps subcommands. Deciding what to sync, cloning, and recording all happen in backups2datalad, so drogon and CI run the same logic.

What changes in backups2datalad

• Per-Dandiset state in the superdataset's .gitmodules (mirrorstate.py):
  • The keys are dandiset-last-modified, dandiset-tagged-releases, and dandiset-stats (<commit>,<files>,<bytes>).
  • Embargoed Dandisets get no stats, because .gitmodules is public.
  • On drogon, update-from-backup writes the keys into the same commit as the gitlinks they describe. Keys are only written for the gitlinked commit.
  • The first drogon run with this version bootstraps the keys for every mirror. Drogon's behavior is otherwise unchanged.
• backups2datalad plan decides which Dandisets need a sync from .gitmodules plus one archive listing.
  • It skips the per-Dandiset /versions/ calls. Section 3.2 of the plan has the evidence that publish and unembargo always bump modified.
  • --embargoed skip is the default: embargoed Dandisets are only counted, never named.
  • --max caps the batch. A quarter of the batch goes to the Dandisets stale longest; the rest is filled smallest first.
• update-from-backup --clone-missing: when the superdataset (or, for a Zarr, its Dandiset) records a mirror that is not present, it is cloned rather than refused by Refuse to create a mirror from scratch when it already exists elsewhere #124's guard.
  • annex.private is set before git-annex first runs in the clone, so the clone's UUID never reaches the pushed git-annex branch.
  • The clone's stats cache is seeded from .gitmodules.
• update-from-backup --results-dir DIR plus record:
  • --results-dir syncs and pushes the mirrors and writes one JSON file per Dandiset, without touching the superdataset.
  • record applies those results to a checkout with no subdatasets: gitlinks and keys.
  • It then runs the existing pass that makes deleted Dandisets' repositories private.
  • It commits and pushes. If the push is rejected, it fetches, resets to the remote branch, and re-applies.
• Superdataset description: for mirrors that are not installed, it now falls back to the .gitmodules keys. Before, it required every mirror to be installed, even on runs given explicit IDs.
• docs/ci/mirror.yml: the GitHub Actions workflow for dandi/dandisets.
  • Three jobs: plan, then a matrix of one sync job per Dandiset, then record. It is dispatch-only while the cron still runs.
  • It lives here because this session cannot push to dandi/dandisets.

Not done yet

These are listed in section 10 of the plan:

• Zarr stats without the Zarr mirrors: a recount still clones every Zarr of a changed Dandiset (section 4.8).
• --time-limit.
• Embargoed Dandisets: routing them to a private repository.
• Using plan on drogon: having update-from-backup use the planner there.

Testing

• New unit tests cover parsing and writing .gitmodules, the planner's reasons, skips, cap and embargo policies, and record. The record test covers registering a new Dandiset, updating an existing one, a rejected push followed by a retry over a concurrent push, and re-recording idempotently.
• New end-to-end test (test/test_ci_flow.py) runs against the local Docker archive:
  • A full backup records the keys.
  • The next steps run on a checkout of the superdataset with no mirrors: plan → upload → plan again → update-from-backup --clone-missing --results-dir → record, which has to retry over a concurrent push.
  • Finally, a fresh checkout plans nothing.
• Full suite locally: 205 passed. The one failure, test_read_annexed_file_from_commit, fails on main too in that sandbox, because it can't download its test file.
• Workflow: checked with actionlint.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KoaLJfNiVVnwSDHNMg2bpK

[yarikoptic-gitmate]

CI note: this PR only changes docs/ci-mirroring-plan.md. The test (3.x, py) failures come from fixture setup: docker compose pull for dandi-cli's test archive fails with minio Error unauthorized on quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z.

That image comes from dandi/dandi-cli#1921, merged 2026-09-14 after Docker Hub's minio/minio was deleted, and Quay now refuses the pull as well. The same failure hits #124. No fix exists yet; it needs another MinIO image source in dandi-cli's test docker-compose.yml. I'm triggering one re-run to confirm.

---

Generated by Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 85.37074% with 73 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.89%. Comparing base (3a0cc4d) to head (b110f4b).

Files with missing lines	Patch %	Lines
src/backups2datalad/datasetter.py	75.24%	33 Missing and 18 partials ⚠️
src/backups2datalad/__main__.py	76.74%	5 Missing and 5 partials ⚠️
src/backups2datalad/adataset.py	88.52%	5 Missing and 2 partials ⚠️
src/backups2datalad/adandi.py	60.00%	2 Missing ⚠️
src/backups2datalad/asyncer.py	81.81%	1 Missing and 1 partial ⚠️
src/backups2datalad/mirrorstate.py	99.40%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #125      +/-   ##
==========================================
+ Coverage   80.67%   81.89%   +1.21%     
==========================================
  Files          19       20       +1     
  Lines        3617     4048     +431     
  Branches      568      651      +83     
==========================================
+ Hits         2918     3315     +397     
- Misses        500      519      +19     
- Partials      199      214      +15     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[yarikoptic-gitmate]

codecov/project is red on 045b5b4: 76.77%, 0.19% below main (90aacaf). This PR only adds docs/ci-mirroring-plan.md and touches no code, so the drop is coverage noise from the test run, probably from timing-dependent paths. Nothing in this PR can fix it. All CI jobs on this head passed: 3.11, 3.12 and 3.13 py, plus lint and typing. I have no way to re-run the Codecov status myself. The failed py runs from 2026-09-24 are from an earlier head that hit the MinIO/Quay pull problem, since fixed by dandi-cli 0.80.1.

---

Generated by Claude Code

[github-actions]

CI mirroring dry run

Run 13.1 (pull_request, 2026-11-10T21:37:09Z): backups2datalad baa1aa8, dandisets draft at d2613ea3 (2026-11-10T16:35:36-05:00), mode timestamp, anonymous (embargoed Dandisets invisible).

Plan: 3 to sync (3 forced (history)); 0 deferred, 1 quiescent, 0 embargoed skipped, 0 unembargoing; 0 no longer on the archive.

Dandiset	planned as	size	CI sync (from recorded commit)	vs. drogon's mirror
000018	forced	0	no change (9fb09e02) ⚠️ recorded ccaaeba4 not on GitHub	❌ differs from drogon's e53a22b1 at the same state: 1 file changed, 3 insertions(+), 1 deletion(-)
000024	forced	0	no change (eed92acf) ⚠️ recorded c525a946 not on GitHub	❌ differs from drogon's 5c0c72f4 at the same state: 1 file changed, 3 insertions(+), 1 deletion(-)
000030	forced	0	no change (f71c2a7c) ⚠️ recorded 56a5111f not on GitHub	❌ differs from drogon's a60b00b3 at the same state: 1 file changed, 3 insertions(+), 1 deletion(-)

⚠️ The superdataset records commits GitHub lacks for 000018, 000024, 000030: CI started from GitHub's latest instead, so differences from drogon's mirror may be what drogon never pushed.

Details

000018, CI's new commits:

• Set .gitattributes to the dandiset default policy (backups2datalad)

000018, differences from drogon's e53a22b1 (at the same state):

diff --git a/.gitattributes b/.gitattributes
index 47bb253..4483632 100644
--- a/.gitattributes
+++ b/.gitattributes
@@ -1,4 +1,6 @@
 
 * annex.backend=SHA256E
 **/.git* annex.largefiles=nothing
-* annex.largefiles=((mimeencoding=binary)and(largerthan=0))
\ No newline at end of file
+### BEGIN dandiset default policy (backups2datalad)
+* annex.largefiles=((mimeencoding=binary)and(largerthan=0))or(largerthan=10MiB)
+### END dandiset default policy (backups2datalad)

000024, CI's new commits:

• Set .gitattributes to the dandiset default policy (backups2datalad)

000024, differences from drogon's 5c0c72f4 (at the same state):

diff --git a/.gitattributes b/.gitattributes
index 47bb253..4483632 100644
--- a/.gitattributes
+++ b/.gitattributes
@@ -1,4 +1,6 @@
 
 * annex.backend=SHA256E
 **/.git* annex.largefiles=nothing
-* annex.largefiles=((mimeencoding=binary)and(largerthan=0))
\ No newline at end of file
+### BEGIN dandiset default policy (backups2datalad)
+* annex.largefiles=((mimeencoding=binary)and(largerthan=0))or(largerthan=10MiB)
+### END dandiset default policy (backups2datalad)

000030, CI's new commits:

• Set .gitattributes to the dandiset default policy (backups2datalad)

000030, differences from drogon's a60b00b3 (at the same state):

diff --git a/.gitattributes b/.gitattributes
index 47bb253..4483632 100644
--- a/.gitattributes
+++ b/.gitattributes
@@ -1,4 +1,6 @@
 
 * annex.backend=SHA256E
 **/.git* annex.largefiles=nothing
-* annex.largefiles=((mimeencoding=binary)and(largerthan=0))
\ No newline at end of file
+### BEGIN dandiset default policy (backups2datalad)
+* annex.largefiles=((mimeencoding=binary)and(largerthan=0))or(largerthan=10MiB)
+### END dandiset default policy (backups2datalad)

What record would commit to the superdataset (not pushed)

3 updated (000018, 000024, 000030)

000018:
 - Set .gitattributes to the dandiset default policy (backups2datalad)

000024:
 - Set .gitattributes to the dandiset default policy (backups2datalad)

000030:
 - Set .gitattributes to the dandiset default policy (backups2datalad)


 .gitmodules | 6 ++++++
 000018      | 2 +-
 000024      | 2 +-
 000030      | 2 +-
 4 files changed, 9 insertions(+), 3 deletions(-)