You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
bug: vector index builds leak their shuffle scratch directory #8992
Every vector index build leaves its shuffle scratch directory behind in the OS temp directory. Nothing removes it, so a process that builds indexes repeatedly accumulates one directory per build.
Reproduced on abcd800cd by pointing TMPDIR at an empty directory and running a single index build:
The test process exited normally and the directory is still there.
prepare_vector_segment_build creates the scratch directory as a local TempStdDir (rust/lance/src/index/vector.rs:590), hands create_ivf_shuffler only its path, and returns the shuffler alone (:599). The guard drops when the function returns, so the directory is removed before anything has been written into it. The shuffle writer recreates it on the way to writing shuffle_data.lance, and from that point nothing owns it.
Both build paths go through that helper: build_vector_index_impl (:1006) and build_distributed_vector_index (:605). The incremental path (:1344) and the optimize path in index/vector/ivf.rs:759 keep their guards in the enclosing function scope, so they are not affected.
Each leaked directory holds a shuffled copy of the column that was indexed, so a long-running process that rebuilds indexes fills its temp filesystem with copies of its own data.
#6980 fixed the same thing for the legacy IVF_HNSW partition staging directory.
Activity
fix(index): keep the shuffler scratch directory alive for the whole b…