Skip to content

perf(hasher): reduce Git file enumeration checks - #1254

Open
djordjem88 wants to merge 1 commit into
microsoft:mainfrom
djordjem88:user/djmaks/lage-git-file-enumeration
Open

djordjem88 wants to merge 1 commit into
microsoft:mainfrom
djordjem88:user/djmaks/lage-git-file-enumeration

Conversation

@djordjem88

Copy link
Copy Markdown
Contributor

Summary

Requested reviewers: Nemanja Tesic (@nemanjatesic) and Elizabeth Craig (@ecraig12345).

Use git ls-files --cached --deleted --stage -t -z to enumerate tracked inputs and filter deleted paths without an individual JavaScript existsSync call for every regular tracked file. Retain existence checks for skip-worktree and indexed symlink entries. Run the tracked and optional untracked queries concurrently while retaining tracked-before-untracked insertion order.

NUL-delimited parsing preserves spaces, tabs, and Unicode paths. Git failures continue to reject through execa. Git still performs filesystem work; this removes separate JavaScript checks, not all stat I/O.

Part of the broader Lage performance effort explored in #1238. This change is self-contained and based directly on current main. The dependency-queue cursor originally implemented alongside it is deliberately excluded into its own PR.

Validation

  • Independent branch based on main at cf155216; extracted source and test files exactly match the previously validated implementation.
  • Final independent branch: all 20 hasher tests passed across two suites.
  • Coverage includes tracked/staged deletions, renames, untracked and ignored paths, skip-worktree/assume-unchanged entries, indexed symlinks, conflict stages, reinitialization, and Git failures.
  • Full local CI passed with the separate queue optimization during implementation (30 successful targets, 91 cached/skipped, zero failures). This is not a claim about GitHub PR CI.
  • Includes a Beachball patch note. No end-to-end speedup claim.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7b26a45b-25aa-45e3-95b4-f9a1b55dcb58
Comment on lines +62 to 92
const [lsFilesResults, lsOtherResults] = await Promise.all([
execa("git", ["ls-files", "--cached", "--deleted", "--stage", "-t", "-z"], { cwd: root }),
includeUntracked ? execa("git", ["ls-files", "-o", "-z", "--exclude-standard"], { cwd: root }) : undefined,
]);

const files: string[] = [];
const deletedFiles = new Set<string>();
for (const entry of lsFilesResults.stdout.split("\0")) {
if (!entry) {
continue;
}

if (lsFilesResults.exitCode === 0) {
const files = lsFilesResults.stdout.split("\0").filter((f) => Boolean(f) && fs.existsSync(path.join(root, f)));
this.addToPackageTree(files);
// Staged records separate metadata from the unquoted path with a tab.
const file = entry.slice(entry.indexOf("\t") + 1);
if (entry.startsWith("R ")) {
// Deleted paths also have cached records; exclude them after collecting both.
deletedFiles.add(file);
} else if (entry.startsWith("S ") || entry.slice(2, 8) === "120000") {
// Git skips deletion checks for skip-worktree entries and does not follow symlinks.
if (fs.existsSync(path.join(root, file))) {
files.push(file);
}
} else {
files.push(file);
}
}
this.addToPackageTree(files.filter((file) => !deletedFiles.has(file)));

if (includeUntracked) {
// Also get all untracked files in the workspace according to git
const lsOtherResults = await execa("git", ["ls-files", "-o", "-z", "--exclude-standard"], { cwd: root });
if (lsOtherResults.exitCode === 0) {
const files = lsOtherResults.stdout.split("\0").filter(Boolean);
this.addToPackageTree(files);
}
if (lsOtherResults) {
this.addToPackageTree(lsOtherResults.stdout.split("\0").filter(Boolean));
}

@ecraig12345 Elizabeth Craig (ecraig12345) Nov 6, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great, thanks! Even better, this can be combined into one command. With -t, untracked records have a ? prefix but no stage metadata or tab separator, so they need their own parsing branch.

Suggested change
const [lsFilesResults, lsOtherResults] = await Promise.all([
execa("git", ["ls-files", "--cached", "--deleted", "--stage", "-t", "-z"], { cwd: root }),
includeUntracked ? execa("git", ["ls-files", "-o", "-z", "--exclude-standard"], { cwd: root }) : undefined,
]);
const files: string[] = [];
const deletedFiles = new Set<string>();
for (const entry of lsFilesResults.stdout.split("\0")) {
if (!entry) {
continue;
}
if (lsFilesResults.exitCode === 0) {
const files = lsFilesResults.stdout.split("\0").filter((f) => Boolean(f) && fs.existsSync(path.join(root, f)));
this.addToPackageTree(files);
// Staged records separate metadata from the unquoted path with a tab.
const file = entry.slice(entry.indexOf("\t") + 1);
if (entry.startsWith("R ")) {
// Deleted paths also have cached records; exclude them after collecting both.
deletedFiles.add(file);
} else if (entry.startsWith("S ") || entry.slice(2, 8) === "120000") {
// Git skips deletion checks for skip-worktree entries and does not follow symlinks.
if (fs.existsSync(path.join(root, file))) {
files.push(file);
}
} else {
files.push(file);
}
}
this.addToPackageTree(files.filter((file) => !deletedFiles.has(file)));
if (includeUntracked) {
// Also get all untracked files in the workspace according to git
const lsOtherResults = await execa("git", ["ls-files", "-o", "-z", "--exclude-standard"], { cwd: root });
if (lsOtherResults.exitCode === 0) {
const files = lsOtherResults.stdout.split("\0").filter(Boolean);
this.addToPackageTree(files);
}
if (lsOtherResults) {
this.addToPackageTree(lsOtherResults.stdout.split("\0").filter(Boolean));
}
const lsFilesResults = await execa(
"git",
["ls-files", "--cached", "--deleted", "--stage", "-t", "-z", ...(includeUntracked ? ["--others", "--exclude-standard"] : [])],
{ cwd: root }
);
const files: string[] = [];
const deletedFiles = new Set<string>();
for (const entry of lsFilesResults.stdout.split("\0")) {
if (!entry) {
continue;
}
if (entry.startsWith("? ")) {
files.push(entry.slice(2));
continue;
}
// Staged records separate metadata from the unquoted path with a tab.
const file = entry.slice(entry.indexOf("\t") + 1);
if (entry.startsWith("R ")) {
// Deleted paths also have cached records; exclude them after collecting both.
deletedFiles.add(file);
} else if (entry.startsWith("S ") || entry.slice(2, 8) === "120000") {
// Git skips deletion checks for skip-worktree entries and does not follow symlinks.
if (fs.existsSync(path.join(root, file))) {
files.push(file);
}
} else {
files.push(file);
}
}
this.addToPackageTree(files.filter((file) => !deletedFiles.has(file)));

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants