[Marc-André Moreau (mamoreau-devolutions)]

Summary

Fixes the hot paths of a connection-per-operation workload over GUID-keyed rowid tables (Remote Desktop Manager's SQLite data source), from the RDM engine benchmark report: Windows 8.3 TEMP paths, GetValue schema PRAGMAs, slow pooled opens, table-scanning GUID lookups, quadratic inserts, O(n) updates, per-commit lock and checkpoint overhead, unpooled encrypted opens, and a misleading error for SEE-encrypted files.

Reader and query paths

• Temp directory: VdbeExecutionOptions resolves the spill directory lazily through a cache keyed on TMP/TEMP/USERPROFILE (an 8.3 short-form TEMP made each GetTempPath() ~1.5 ms).
• SqliteDataReader: declared types, undeclared-source flags and select-source PRAGMA results are cached per result set (keyed on statement + SQL, reset by NextResult).
• Index equality SEARCH (GetManagedIndexRows): seeks the durable index b-tree instead of materializing every row and sorting the whole index; inside a transaction it reads through the transaction's own writes. Rowid tables without method indexes only.

Write paths

• INSERT: unique-index validation uses a key set cached on the table (keyed by an index-definition signature, valid only for one row-store identity), so imports are linear.
• UPDATE: plain UPDATE ... WHERE on rowid tables (no FROM/LIMIT/ORDER BY/RETURNING, no rowid change, no FK-relevant assignment) runs PerformUpdateInPlace: matched rows only, same rowid-order WHERE/SET interleaving, incremental unique checks, in-place row replacement. Conflict handling (IGNORE/FAIL partial publish) mirrors the full path. The full path also prunes WHERE evaluation to equality candidates.

Storage

• Pooled open/close and commit-time version checks skip the second-pager catalog probe when the shared generation and durable view token show nothing committed.
• WAL lock leases reuse idle, unlocked handles of a mapped -shm carrier (≤16 per access mode, closed with the final mapping, identity-checked, never reused after a failed unlock; macOS keeps its brokered descriptors). File opens cost 1–4.5 ms under Windows file-system filters, while a lock on an open handle costs ~25 µs.
• Checkpoints no longer bump the pager's local generation, which forced a full rescan (header check + WAL reopen + recovery scan) after every checkpoint and could skip a needed rescan.
• Turso checkpoint parity for SqliteConnection/AhtolaConnection local files (core/storage/pager.rs commit_wal, wal.rs should_checkpoint, threshold 1000): ordinary row commits checkpoint once the WAL holds more than 1000 committed frames, plus a best-effort checkpoint on close. Replicas, MVCC and structural changes (full rewrites, REINDEX, VACUUM, page-size and journal-mode changes) keep checkpointing immediately. This needed four changes:
  • WAL validation scans resume after the last validated commit frame (trusted only while the on-disk WAL header and boundary frame header are unchanged; reset, truncation and tail recovery always re-validate every frame);
  • wal-index header validation reuses the commit-frame header that scan validated instead of reading the whole frame on every page access;
  • opening a writable pager keeps a wal-index that already validates against the WAL instead of rebuilding it, which reset live readers' read marks, as SQLite's walIndexReadHdr;
  • writers publish an idle read mark at the committed boundary so read-only readers (backup snapshots) can share it.
• Encrypted connections are pooled by default, keyed by cipher + SHA-256 key fingerprint (a different or wrong key never gets the pooled database). Pooled entries own their encryption file system. AhtolaConnection now accepts Pooling=True with encryption.
• An encrypted open of non-Ahtola ciphertext (e.g. SQLite SEE) now reports "not in the Ahtola encrypted page format" instead of claiming a plaintext SQLite header.

Behavior changes

• Between checkpoints the .db file alone does not contain the latest commits (as in SQLite/Turso WAL mode): copy the -wal with it or run PRAGMA wal_checkpoint(TRUNCATE). Documented in docs/dotnet-packages.md.
• Idle pooled encrypted databases keep their key and decrypted page cache in memory until ClearPool/ClearAllPools. Documented.
• In-place UPDATE checks NOT NULL only on the written row, as SQLite does.

Measurements

RDM-shaped harness (GUID blob PK, ~1.3 KB XML DATA, connection per operation), 10,000 entries, Windows 11 / .NET 10 Release / Defender on. The machine was noisy, so comparisons were made back to back.

Operation	Before	After	System.Data.SQLite (RDM today)
Pooled open + SELECT 1 + close	34 ms (1k rows)	0.4–0.7 ms	8.0 ms
Load every entry via DataTable (GetValue)	9.9 s (1k rows)	61–87 ms	81 ms
GUID lookup	3–4 MB alloc/op (1k rows); 95 ms (report)	0.6–0.7 ms	0.3 ms
Import 10,000 rows (one tx)	186 s (report); 11 s at 5k here	4.2 s	192 ms
Save (tx: history insert + update)	108 ms	7–9 ms	29 ms
Log row (autocommit insert)	57–75 ms	3–6 ms	19 ms
Encrypted open	69–105 ms	0.09 ms	14 ms
Search (LIKE)	~38 ms	~38 ms	40 ms

Not addressed

• Import is still ~20× slower than System.Data.SQLite (per-statement table cloning and validation).
• Files encrypted by System.Data.SQLite (SEE) still need an export/re-encrypt step or a custom IPageCodec.
• The UPDATE candidate scan and commit-time row diff are still O(table rows) (about 1 ms and 2% at 10k).
• Turso.Data.Sqlite issues from the report are upstream.

Test plan

• Full managed suite on net10.0 / win-x64: 8,756 passed, 0 failed
• net8.0/net9.0/net10.0 build with no warnings
• New ConnectionPerOperationWorkloadTests (seek correctness incl. transactions and triggers, unique-key cache, in-place UPDATE conflict semantics and FK enforcement, reader declared types, pooled reopen, lock-handle pool, deferred checkpoints and peer visibility across WAL restarts, error text, temp dir) and ManagedConnectionPoolingTests encrypted-pooling cases
• CI on Linux/macOS: lock-handle pooling and WAL changes have only run on Windows
• ./build.ps1 validate-package / validate-trim
• ./build.ps1 format-check (already fails on master for unrelated files)

🤖 Generated with Claude Code