bake_and_save rewrites the entire Arrow IPC file on every commit, which takes about 15 seconds at 10 million rows, versus about 17ms for a delta autosave. The goal is to accumulate deltas and bake lazily instead: stop baking on every commit, write chained delta files, replay deltas on open, background-compact past a threshold (roughly 50 deltas or 1MB), dedupe during replay, track a bake version somewhere for crash detection, consider moving delta storage into SQLite, and use thin undo entries (IDs, not full clones) for large imports.
Related: #236
bake_and_save rewrites the entire Arrow IPC file on every commit, which takes about 15 seconds at 10 million rows, versus about 17ms for a delta autosave. The goal is to accumulate deltas and bake lazily instead: stop baking on every commit, write chained delta files, replay deltas on open, background-compact past a threshold (roughly 50 deltas or 1MB), dedupe during replay, track a bake version somewhere for crash detection, consider moving delta storage into SQLite, and use thin undo entries (IDs, not full clones) for large imports.
Related: #236