Dgraph Zero - Connection reset by peer

[arighosh2014]

I am deploying dgraph in k8s. Dgraph is running successfully, however I am getting the following error log always in dgraph zero service. This error comes at a constant rate of approx 900 per 15 minutes. Have you seen this before?

E0614 13:18:28.777374 1 x.go:358] Could not send error msg=write tcp 10.19.17.249:6080->10.19.48.240:41088: write: connection reset by peer code=Error due to http error write tcp 10.19.17.249:6080->10.19.48.240:41088: write: connection reset by peer

[matthewmcneely]

Something is hammering the state endpoint. A constant ~900/15min ≈ 1 request/second from a single source IP is the signature of an automated poller hammering /state and tearing the connection down early.

Usual suspects:

• A k8s liveness/readiness probe pointed at /state (not /health), with a timeoutSeconds shorter than how long /state takes. /state does a WaitLinearizableRead through Raft (up to 10s) and marshals the entire membership state — far heavier than a probe should call. When the kubelet's timeout fires, it RSTs mid-response.
• A load balancer / service-mesh health check doing the same.
• Ratel or a monitoring dashboard auto-refreshing /state.