Skip to content

About

Standalone cargo-fuzz harness for lance-format/lance, retesting fork bugs against upstream

Resources

Code of conduct

Stars

2 stars

Watchers

0 watching

Forks

Repository files navigation

Lance Fuzz

Stateful, model-based fuzzing for Lance datasets, inspired by ClickHouse's BuzzHouse.

Buzz-style fuzzing generates weighted sequences of operations against a data system while maintaining a model of the expected state. After every successful operation, the harness updates its model and checks Lance through logical and structural oracles.

Resources:

How it works

dataset_ops runs stateful operation sequences against a fresh Lance dataset. It mirrors the expected rows in a BTreeMap<u64, Row> and compares Lance with that model throughout the session. A finding is an error from an eligible operation, a logical mismatch, or a structural validation failure.

How a session works

Each input is generated from an RNG seed, column specifications, and operation candidates. The initial schema has one to five value columns, which AddColumn can grow to eight. The session examines at most 64 candidates and executes the first 24 whose preconditions are met.

Session:

  1. Creates a fresh dataset with a non-null u64 id column.
  2. Generates data and executes eligible operations in order.
  3. Applies every successful mutation to the in-memory row model.
  4. Verifies Lance after every executed operation.
  5. Runs structural validation and reopens file-backed datasets for a final independent read.

What it exercises

  • Writes: Append, full and partial merge-insert, conditional update, replace-all, merge-delete, SQL delete, and SQL update.
  • Schema and history: Add or remove columns, restore an earlier version, and migrate between Lance file versions.
  • Indices: Create, optimize, drop, and query scalar or vector indices.
  • Maintenance: Compact files with varied physical and concurrency settings.
  • Reads: Filter, count, project, limit, offset, take, and KNN search.

Scalar index selection is type-aware and covers BTree, Bitmap, ZoneMap, BloomFilter, NGram, Inverted, and LabelList. Float32 fixed-size lists can use an IVF_FLAT vector index.

Oracle boundaries

The model checks logical dataset contents; it does not independently recompute generated values.

KNN validation checks result safety and cardinality — live, unique, non-null rows, and at most min(k, live_non_null_rows) of them — not the exact neighbor set. The single-partition IVF_FLAT index is itself exhaustive; the looseness covers rows the index does not yet include, tied distances, and null or non-finite vector values.

Float equality is never probed — float literals do not round-trip reliably through SQL — so equality-family indices on float columns are exercised through null probes only. Temporal and decimal index probes use exact arrow_cast literals; Duration and Decimal128 equality is checked for correctness but not for index usage, because Lance does not plan those predicates as sargable.

Limit/offset and take-by-offset checks compare one plan's rows against another plan's at the same offsets; every scan pins scan_in_order to keep that comparison meaningful.

The schema excludes types such as Map that cannot be written by every initial file version, and storage coverage is limited to the local filesystem and the in-memory object store.

Normalizations

Some model behavior is deliberately shaped around what Lance can represent:

  • Null structs are modeled as valid structs with null children, because the Lance file format does not preserve struct-level validity.
  • SourceDedupeBehavior::FirstSeen is exercised only with duplicate keys that already exist; duplicate unmatched source rows would both be inserted and violate the unique-key model.
  • Equality predicates avoid integer literals outside the SQL parser's safe i64 range.
  • Inverted indices use the raw tokenizer with every normalization filter disabled, so a document is exactly one token and full-text matches are plain string equality. Full-text probes run only on plain string columns: list documents are tokenized inconsistently between the index build and the flat-match path, so they have no oracle.
  • Dictionary arrays are canonicalized before comparison, because equivalent values can use different dictionary encodings.
  • Local generator wrappers avoid zero-row generation failures, alignment problems for decimal buffers, timezone loss in generated timestamp arrays, and wall-clock-dependent temporal values.

Configuration

LANCE_FUZZ_STORAGE selects one storage profile for the entire campaign. Storage is not encoded in FuzzInput, so the same input can be replayed against every profile.

Value Storage
memory (default) In-memory object store through memory://
tmp Unique temporary directory addressed by filesystem path
uri Unique temporary directory addressed through a file:// URI

The input seed selects physical settings without consuming the RNG stream used for data generation:

Setting Values
Lance file version 2.0, 2.1, 2.2, 2.3
Rows per row group 1, 4, 16
Rows per file 2, 4, 8
Stable row IDs Disabled or enabled
Compaction profile Default or two independently selected knobs

SOFT_STATS downgrades the row-count checks on merge, update, and delete from assertions to warnings. Those counts are hard assertions by default; softening them lets a run continue to the scan and probe checks, which distinguishes a genuine data divergence from a merely miscounted stat.

Running

Setup

Install Pixi, nightly Rust, and cargo-fuzz:

rustup install nightly
rustup +nightly component add llvm-tools-preview
cargo install cargo-fuzz

Fuzzing

Build the fuzz target or run it across all CPU cores:

pixi run fuzz-build
pixi run fuzz       # run until stopped
pixi run fuzz 300   # run for 300 seconds

Fuzz output is written to logs/dataset_ops-<timestamp>/, one fuzz-<n>.log per worker. Workers share corpus/dataset_ops/; crashes land in artifacts/dataset_ops/.

On aarch64 Linux (e.g. AWS Graviton), install lld first (apt install lld). The instrumented binary is too large for GNU ld's aarch64 branch range, so pixi links it with hack/cc-lld.

Automated campaign

The fuzz Claude skill runs a full campaign: it updates every lance-* dependency to the latest upstream tag, confirms that the tracked crash suite still reproduces, fuzzes dataset_ops for a specified duration while periodically triaging crashes, and minimizes each finding into a standalone reproducer in bugs/:

/fuzz until tomorrow 8am

Reproducing a finding

A finding is a saved artifact. Replay it on stable, without nightly or cargo-fuzz:

cargo run --bin replay -- <artifact-path>
REPLAY_DUMP=1 cargo run --bin replay -- <artifact-path>   # print the decoded input, do not run it

The dev profile keeps debug_assert! and overflow checks on. Replay has no coverage guidance or sanitizer instrumentation, so memory-safety findings must still be confirmed under cargo-fuzz.

Importing crashes

Import new findings from the ignored artifacts/dataset_ops/ directory:

pixi run crash-import
uv run hack/crash-check.py dataset_ops --import path/to/artifact

By default, imports keep up to three inputs per crash class. Named artifacts are always kept. Inputs are content-addressed, so re-importing them is safe and does not create duplicates.

Maintain the bug column in manifest.tsv by hand:

  • ?: not yet linked to a bug
  • open: an uninvestigated divergence
  • harness: a known harness false positive

Coverage from the corpus

pixi run fuzz-coverage
pixi run fuzz-coverage-report

pixi run fuzz-coverage [target] [min-dark-lines] builds the target with coverage instrumentation, then replays every input in corpus/<target>. It prints total Lance line coverage, a per-crate table, and every fully uncovered file with at least 400 lines by default.

Collection can take several minutes. pixi run fuzz-coverage-report re-renders the report from the last collection, so changing the dark-file threshold is fast. The full per-file report is written to coverage/<target>/per-file.txt.

Coverage counters are tied to one build. Collect a fresh baseline after updating Lance or changing the harness.

Updating source code

Use the same upstream Git tag for every direct lance-* dependency and lance-datagen in Cargo.toml. To fuzz a local Lance checkout instead, rename .cargo/local-lance.toml to .cargo/config.toml — see the caveats in that file. Then refresh and verify the dependencies:

cargo update
pixi run fuzz-build
pixi run crash-check

After changing the source code, run:

cargo fmt --all --check
cargo nextest run --lib
pixi run fuzz-build
pixi run crash-check

Keep the FuzzInput decoding layout when possible so the existing corpus and crash suite retain their meaning. If it must change, rebuild the corpus and re-import representative crashes with the new harness.

Run the tracked crash suite after updating Lance or changing the harness:

pixi run crash-check
pixi run crash-check dataset_ops 3       # retry mismatches three times
uv run hack/crash-check.py dataset_ops index   # check matching inputs only

The report shows inputs that still reproduce, no longer crash, or have a different class. crash-check exits with status 1 when an input no longer matches, so it can run in CI.

Crash classes are normalized across Lance versions, but serialized inputs depend on the FuzzInput layout. If the layout changes, the same bytes can decode into a different session. The manifest stores a hash of src/dataset_ops/ops.rs (the file defining the input types), and crash-check warns if it changes. In that case, re-import inputs instead of treating widespread class changes as Lance regressions.

About

Standalone cargo-fuzz harness for lance-format/lance, retesting fork bugs against upstream

Resources

Code of conduct

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages