Skip to content

Windows: handoff round-trip reader intermittently never observes a delivered payload #1283

Description

@charleswool

Summary

TestImagesHandoffRoundTrip and TestCompletionHandoffRoundTrip in pkg/utils intermittently fail on Windows only. Both fail on the reader side: the writer has already delivered its payload, and the reader never observes it.

This is not a new regression. It reproduces on main and predates the cancellation work in #1237 — it was found while chasing a red check there, and the analysis is buried in that now-merged PR, so filing it here where it can be found.

Rate

2 failures in 12 iterations of go test -count=12 ./pkg/utils/ on windows/amd64, consistent with a ~1-in-5 rate measured earlier with -count=150 against origin/main at 0bc94c49.

Measured with and without #1237 applied, against a matched baseline. The rate did not change, which is what rules that PR out as the cause.

Symptom

handoff_test.go:76: ReadImagesPipe did not return within the deadline: context deadline exceeded
--- FAIL: TestImagesHandoffRoundTrip (30.00s)

handoff_test.go:115: Await: context deadline exceeded
--- FAIL: TestCompletionHandoffRoundTrip (30.00s)

The 30 seconds is testContext's deadline. Before Await and ReadImagesPipe took a context, the same hang ran to the package-wide ten minute timeout with no indication of which test was stuck:

FAIL github.com/eraser-dev/eraser/pkg/utils 601.225s

So the failure is now bounded and attributable, but the underlying bug is untouched.

What is known

A goroutine dump taken during the hang showed Await parked in internal/poll.(*FD).Read with the writer goroutine already returned — the payload was written and the connection closed, but the reader saw neither the bytes nor EOF.

Two hypotheses were tested and disproved:

  • Not a zero-byte read. Instrumentation showed bytes=8 on every passing iteration; the hanging iteration never reaches the read completion at all.
  • Not a missing half-close. Adding an explicit CloseWrite() on the writer before Close() made no difference to the rate.

Scope

Windows only. The Unix FIFO implementation has not reproduced it. It affects the unit suite rather than a deployed worker, but the same code path is what a Windows collector and remover use to hand off, so it is worth understanding before Windows workers are relied on.

Reproducer

$env:GOOS='windows'
go test -count=50 -timeout 30m ./pkg/utils/

Expect roughly one failure per five iterations on windows/amd64.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions