Skip to content

fix(http): enforce absolute read header timeout against slow HTTP DoS - #195

Closed
Vonng wants to merge 1 commit into
mainfrom
codex/slow-http-dos-hardening
Closed

Vonng wants to merge 1 commit into
mainfrom
codex/slow-http-dos-hardening

Conversation

@Vonng

@Vonng Vonng commented Sep 15, 2026

Copy link
Copy Markdown
Member

Fixes #183

Root cause

A slow HTTP DoS scanner holds connections open by dribbling request bytes (slowloris). Two inherited gaps allowed exactly that:

  1. deadlineconn neutralized the absolute header deadline. httpListener.Accept() wrapped every accepted connection with deadlineconn.WithReadDeadline(idle), which re-extends the read deadline on every partial read. net/http sets ReadHeaderTimeout as an absolute deadline (headers must complete within N seconds in total), but the wrapper converted it into an inter-byte activity timeout — a client sending 1 byte every 25 s keeps the connection (goroutine + fd) alive forever.
  2. the timeout was never actually configured. The --read-header-timeout flag (default 30 s) is defined but never copied into globalServerCtxt in common-main.go, so the server ran with ReadHeaderTimeout = 0, i.e. disabled. Upstream MinIO master shares both defects (verified against minio/minio@master).

Fix

  • internal/http/listener.go: keep deadlineconn on the write side only. Write deadlines stay activity-based (big downloads must not be capped in total), and native read deadlines are honored again.
  • cmd/common-main.go: wire --read-header-timeout into the server context.
  • cmd/server-main.go: drop UseReadTimeout. With native read deadlines honored, an absolute ReadTimeout would cap the whole request including large uploads; request bodies are bounded per-read instead (next point). WriteTimeout stays: it only resets the activity-based write deadline per request.
  • internal/http/server.go: wrap request bodies in idleTimeoutBody, which refreshes the connection read deadline ahead of every body read via http.NewResponseController. Stalled uploads are cut off after the idle timeout while progressing ones extend indefinitely. Under HTTP/2 the controller is supported natively per stream; on hijacked connections it is a no-op.

HTTP/2 audit: x/net/http2 only sets zero (infinite) read deadlines on the conn, and Server.IdleTimeout is wired into the http2 idle timer, so idle h2 connections are still cleaned up after 30 s — no regression from dropping the read-side wrapper.

Verification

Unit tests (internal/http/server-deadline_test.go, full xhttp stack incl. listener and TCPOptions wiring):

  • slow-header trickle killed at ReadHeaderTimeout — verified the test fails on the old listener code (connection survives) and passes with the fix
  • stalled request body cut off at the idle deadline, handler observes the error
  • slow-but-progressing body (4× the idle timeout in total) completes

Live single-node run (default flags):

probe before after
slowloris, 1 partial header / 3 s alive ≥ 90 s (unbounded) cut at 33 s
2 MiB PUT, 64 KiB chunks 1 s apart (32 s total) ok ok (no absolute body cap)
PUT stalled mid-body ~30 s cleanup 30 s cleanup (idleTimeoutBody)
GET with stalled reader (70 s) ~30 s write cutoff ~30 s write cutoff (unchanged)
regular mcli mb/cp/cat/stat ok ok

Regression: go build ./..., go vet, ./internal/http ./internal/deadlineconn suites, and the full ./cmd suite (720 s) all pass.

Residual: deliberate slow-body/slow-read trickle (inter-byte gap < idle) still holds a connection, as before — an object store cannot cap total transfer time without breaking legitimate large/slow uploads. Perimeter timeouts (LB/reverse proxy) remain the recommendation for strict environments; this change closes the unbounded slow-header case the scanner flagged.

Slow HTTP DoS scanners (issue #183) hold connections open by dribbling
request bytes, and two inherited gaps let them:

1. deadlineconn re-extended the read deadline on every partial read,
   converting net/http's absolute ReadHeaderTimeout into an inter-byte
   activity timeout that a trickle bypasses entirely.
2. the --read-header-timeout flag (default 30s) was never copied into
   the server context, so the server actually ran with header timeout
   disabled (upstream shares this bug).

Keep deadlineconn on the write side only so response writes stay
activity-based, and leave ReadTimeout at zero now that native read
deadlines are honored, since an absolute cap would terminate large
uploads. Request bodies keep an activity-based bound instead via a
per-read deadline refresh in xhttp, and the flag is wired through
common-main.

With this, a slow-header connection is cut at ReadHeaderTimeout
(verified live: 33s kill vs. unlimited before), while slow progressing
uploads (32s for 2 MiB), stalled bodies (30s cleanup), slow readers
(~30s write cutoff) and plain S3 traffic are unaffected.

Fixes #183

Signed-off-by: Feng Ruohang <rh@vonng.com>
@Vonng

Vonng commented Sep 15, 2026 •

Copy link
Copy Markdown
Member Author

The request-header timeout fix has landed through PR #196, merged as 40220bd836cb. This alternative implementation is closed as superseded; its branch is retained.

The merged implementation preserves the absolute header deadline, wires --read-header-timeout / MINIO_READ_HEADER_TIMEOUT, and retains rolling idle timeouts for active HTTP/1 transfers. See the integration and validation record and the published documentation on timeout behavior and release status. Issue #183 now points to the merged fix.

As of September 16, these changes are on main but are not included in Server RELEASE.2026-09-03T13-18-01Z. Final release acceptance remains #203.

@Vonng Vonng closed this Sep 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Vulnerability: Slow HTTP DOS

1 participant