Skip to content

fix(cluster): make migration response timeout configurable - #3624

Open
wengsht wants to merge 1 commit into
apache:unstablefrom
wengsht:user/shitao/configurable-migration-response-timeout
Open

wengsht wants to merge 1 commit into
apache:unstablefrom
wengsht:user/shitao/configurable-migration-response-timeout

Conversation

@wengsht

@wengsht wengsht commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add mutable migrate-response-timeout-ms configuration with a 5000 ms default and a 1000-300000 ms range
  • apply the configured receive timeout before migration response parsing and before every APPLYBATCH
  • add focused coverage for live config updates and responses that arrive after the previous one-second deadline

This is the timeout-only split from #3613. It intentionally excludes the subkey-iterator optimization.

Motivation

Slot migration currently sets SO_RCVTIMEO to one second while checking destination responses, then reuses that socket for APPLYBATCH. A production canary failed after about 1.14 seconds with Resource temporarily unavailable even though the destination was lightly loaded. This indicates an EAGAIN receive timeout rather than destination saturation.

An explicit bounded setting lets operators test a longer response deadline without introducing low-level retries. Reapplying it before every batch also makes CONFIG SET migrate-response-timeout-ms ... effective during an active migration.

Validation

  • ./x.py check format --clang-format-path clang-format-18
  • built the kvrocks and unittest targets
  • ./build/unittest --gtest_filter="IOUtil.SockReadLineWaitsForConfiguredReceiveTimeout:Config.GetAndSet": 2 tests passed
  • git diff --check

🤖 Generated with Codex (gpt-5.6-sol)

The destination handshake left SO_RCVTIMEO at one second on the socket later used by APPLYBATCH. Slow-but-successful writes therefore surfaced as EAGAIN and aborted migration.

Use a bounded mutable timeout and reapply it before each batch so CONFIG SET takes effect immediately.

Model: gpt-5.6-sol
Co-Authored-By: Codex (gpt-5.6-sol) <noreply@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant