Summary
TestImagesHandoffRoundTrip and TestCompletionHandoffRoundTrip in pkg/utils intermittently fail on Windows only. Both fail on the reader side: the writer has already delivered its payload, and the reader never observes it.
This is not a new regression. It reproduces on main and predates the cancellation work in #1237 — it was found while chasing a red check there, and the analysis is buried in that now-merged PR, so filing it here where it can be found.
Rate
2 failures in 12 iterations of go test -count=12 ./pkg/utils/ on windows/amd64, consistent with a ~1-in-5 rate measured earlier with -count=150 against origin/main at 0bc94c49.
Measured with and without #1237 applied, against a matched baseline. The rate did not change, which is what rules that PR out as the cause.
Symptom
handoff_test.go:76: ReadImagesPipe did not return within the deadline: context deadline exceeded
--- FAIL: TestImagesHandoffRoundTrip (30.00s)
handoff_test.go:115: Await: context deadline exceeded
--- FAIL: TestCompletionHandoffRoundTrip (30.00s)
The 30 seconds is testContext's deadline. Before Await and ReadImagesPipe took a context, the same hang ran to the package-wide ten minute timeout with no indication of which test was stuck:
FAIL github.com/eraser-dev/eraser/pkg/utils 601.225s
So the failure is now bounded and attributable, but the underlying bug is untouched.
What is known
A goroutine dump taken during the hang showed Await parked in internal/poll.(*FD).Read with the writer goroutine already returned — the payload was written and the connection closed, but the reader saw neither the bytes nor EOF.
Two hypotheses were tested and disproved:
- Not a zero-byte read. Instrumentation showed
bytes=8 on every passing iteration; the hanging iteration never reaches the read completion at all.
- Not a missing half-close. Adding an explicit
CloseWrite() on the writer before Close() made no difference to the rate.
Scope
Windows only. The Unix FIFO implementation has not reproduced it. It affects the unit suite rather than a deployed worker, but the same code path is what a Windows collector and remover use to hand off, so it is worth understanding before Windows workers are relied on.
Reproducer
$env:GOOS='windows'
go test -count=50 -timeout 30m ./pkg/utils/
Expect roughly one failure per five iterations on windows/amd64.
Summary
TestImagesHandoffRoundTripandTestCompletionHandoffRoundTripinpkg/utilsintermittently fail on Windows only. Both fail on the reader side: the writer has already delivered its payload, and the reader never observes it.This is not a new regression. It reproduces on
mainand predates the cancellation work in #1237 — it was found while chasing a red check there, and the analysis is buried in that now-merged PR, so filing it here where it can be found.Rate
2 failures in 12 iterations of
go test -count=12 ./pkg/utils/onwindows/amd64, consistent with a ~1-in-5 rate measured earlier with-count=150againstorigin/mainat0bc94c49.Measured with and without #1237 applied, against a matched baseline. The rate did not change, which is what rules that PR out as the cause.
Symptom
The 30 seconds is
testContext's deadline. BeforeAwaitandReadImagesPipetook a context, the same hang ran to the package-wide ten minute timeout with no indication of which test was stuck:So the failure is now bounded and attributable, but the underlying bug is untouched.
What is known
A goroutine dump taken during the hang showed
Awaitparked ininternal/poll.(*FD).Readwith the writer goroutine already returned — the payload was written and the connection closed, but the reader saw neither the bytes nor EOF.Two hypotheses were tested and disproved:
bytes=8on every passing iteration; the hanging iteration never reaches the read completion at all.CloseWrite()on the writer beforeClose()made no difference to the rate.Scope
Windows only. The Unix FIFO implementation has not reproduced it. It affects the unit suite rather than a deployed worker, but the same code path is what a Windows collector and remover use to hand off, so it is worth understanding before Windows workers are relied on.
Reproducer
Expect roughly one failure per five iterations on
windows/amd64.