Repository navigation
feat(lance): add Lance-backed graph storage extension - #140
Merged
Merged
Conversation
Implements the icebug-disk storage design (docs: node datasets, CSR indices+indptr datasets, flat rel datasets, immutable tables) on Lance datasets instead of Parquet files: - LANCE_SCAN table function (also enables LOAD FROM / COPY FROM with file_format='lance') reading through DuckDB's lance core extension, the same engine built by lance-duckdb. - LanceNodeTable / LanceRelTable (CSR + FLAT) on ColumnarNodeTableBase / ColumnarRelTableBase, registered via ExternalTableHookRegistry for StorageFormat::LANCE; range-limited CSR scans, degree summaries, PK lookup and recursive traversal all work through the shared columnar scan machinery. - e2e coverage (test/test_files/lance.test, fixtures mirror dataset/ice-disk-test row-for-row), path unit tests, and a fixture generator script (scripts/gen_test_data.sh). Requires the core external-table-hooks API (stacked ladybug PR); validated locally against it (9 e2e cases + 12 unit tests green).
adsharma
force-pushed
the
lance-extension
branch
from
October 27, 2026 05:16
9ef13ca to
f6dcf33
Compare
Replace the DuckDB-mediated reads with a native reader: new rust/ crate (lance_ladybug_ffi, Lance 13 + Arrow 58, same versions as lance-duckdb) exposing open/count/schema/scan-to-Arrow-batches over a C ABI, built and linked by CMake following the lance-duckdb Cargo integration. Batches cross the FFI via the Arrow C data interface and decode with core ArrowConverter. - Tables and LANCE_SCAN keep their structure; only the read primitive swaps (SQL CASTs become exact-type validation at CREATE, which now also enforces the stamped lance_version marker). - Fixtures regenerated with rust/examples/gen_fixtures.rs (identical rows, native int widths, v1 markers) plus a v999 dataset for the version-mismatch test; e2e expectations unchanged. - Drop the DuckDB connector, loader, installer and find_package.
Contributor
Author
|
Yes — extension/lance/rust/target/release/liblance_ladybug_ffi.a (304MB, gitignored build output, not committed). Our FFI code itself is ~700 lines; the bulk is the statically-linked dependency tree it pulls in:
The final linked extension is 176MB (linker drops unreferenced symbols; thin LTO + strip are on) — same weight class as the DuckDB-linked extensions. |
The manylinux container has no cargo, so the lance_ladybug_ffi custom command died mid-build with a bare 'no such file or directory'. Install rustup stable plus the matrix-arch std target, and ensure protoc exists (prost-build needs it) via dnf with an official-release fallback. CMake now locates cargo at configure time and fails fast with a clear message instead of a mid-build custom-command error.
Calling project() redefined PROJECT_SOURCE_DIR to extension/lance, so build_extension_lib() wrote liblance.lbug_extension to extension/lance/extension/lance/build instead of extension/lance/build where LOAD_DYNAMIC_EXTENSION resolves it. All 9 lance e2e cases failed with 'neither an official extension, nor does the extension path exist'. Other extensions never call project(); match that convention.
…456) Core generalized per-format storage beyond LANCE (LadybugDB/ladybug#1456): StorageFormat::LANCE is gone, replaced by EXTERNAL plus a persisted format name; the hook surface is renamed to ColumnarTableHooks / ColumnarTableHookRegistry, registered by name, with supportsMixedColumnarScan replacing the scanStateRank lattice and ColumnarRelTableScanState as the shared multi-rel scan-state base. - Register under "lance" with supportsMixedColumnarScan instead of StorageFormat::LANCE + scanStateRank 3. - LanceRelTableScanState extends ColumnarRelTableScanState directly (was IceDiskRelTableScanState, now an empty subclass kept for compat).
adsharma
force-pushed
the
lance-extension
branch
from
November 1, 2026 19:05
48753fa to
1225a35
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #16 (storage and scan portion; vector/FTS/hybrid search functions remain future work).
Stack 2/3 of the Lance storage work. Depends on LadybugDB/ladybug#1445 (core external-table-hooks API); CI here builds against prebuilt main artifacts, so this goes green once #1445 merges.
What
Lance-backed read-only graph storage: the icebug-disk design (node datasets, CSR
indices+indptrdatasets, flat rel datasets, immutable tables) on Lance files instead of Parquet — the storage-side analogue of lance-csr. Reads are native: newrust/crate (lance_ladybug_ffi, Lance 13 + Arrow 58, same versions as lance-duckdb) exposes open/count/schema/scan over a C ABI, built and linked by CMake following the lance-duckdb Cargo integration. Batches cross via the Arrow C data interface and decode with coreArrowConverter; no DuckDB layer anywhere.LANCE_SCAN(path)table function (also enablesLOAD FROM ... (file_format='lance')andCOPY ... FROM ...).LanceNodeTable/LanceRelTable(CSR + FLAT) on the columnar bases, registered viaExternalTableHookRegistryforStorageFormat::LANCE. Range-limited CSR scans (limit/offset pushed into Lance, in-order for positional mapping), in-memory indptr, degree summaries, PK lookup, recursive traversal, and mixed multi-rel scans (state extendsIceDiskRelTableScanState, rank 3).lance_versionmarker is enforced (missing warns, mismatch errors), mirroring icebug-disk's footer version.test/test_files/lance.test(9 e2e cases: scans, 1-hop/2-hop/backward/undirected/varlen/counts/flat/mixed-tables/invalid-storage/version-mismatch/remote-deferred/read-only) with fixtures fromrust/examples/gen_fixtures.rs(same rows asdataset/ice-disk-test), pluslance_utils_testpath unit tests. CI list updated.Testing
Validated locally against the #1445 branch: full e2e file green (one order-dependent 'extension not loaded' case removed — hook registration is process-global, so it can only be covered in a fresh process), 12/12 path unit tests green,
ice_diske2e still 9/9. Rustcargo check+ release build warning-free; C++ files clang-format-clean.Follow-up (3/3, main repo after this merges): Makefile list,
OFFICIAL_EXTENSIONentry,docs/bullet, submodule bump.