[chenBright]

What problem does this PR solve?

Issue Number: resolve

Problem Summary:

The existing RDMA transport allocates receive queues, buffers and completion queues per connection. These resources remain reserved even when connections are idle, making this model expensive for workloads with many connections and sparse RPC traffic.

What is changed and the side effects?

Changed:

This PR introduces an optional SRQ mode for many established connections with sparse RPC traffic and a low active-connection ratio. High concurrency concentrated on a single connection is outside the primary optimization target.

• Introduce a common RdmaConnection interface for private RQ and SRQ connections.
• Keep one RC QP and SQ per connection, while sharing an SRQ, receive buffer pool, TX CQ, and RX CQ within each worker.
• Reference UCX's RC/Active Message design for shared receive resources, completion-queue credits, and per-connection flow control:
  • Initialize each connection with a send window, avoiding an extra credit-request round trip before its first request.
  • Send soft/hard credit requests with data and respond with piggybacked or standalone grants.
  • Restore the negotiated window on a grant.
• Use event-driven worker progress, batched posting and completion accounting, and reserved control capacity.
• Preserve zero-copy transmission and reception for eligible buffers. Small messages can use inline sends and copying on receive.
• Keep receive slots independent of application-held buffers, allowing a slot to be replenished with another registered block.

Send path

flowchart LR
    A["Socket write<br/>RdmaEndpoint"] --> B["SrqConnection::Send<br/>Acquire PostLock"]
    B --> C["Apply FC events<br/>Check SQ capacity<br/>Reserve shared TX CQ credits"]
    C --> D{"Capacity available?"}

    D -->|Yes| E["Prepare SGEs and IMM<br/>Publish send records<br/>ibv_post_send"]
    E --> F["Account accepted prefix<br/>Roll back unposted suffix"]
    F -->|More input| C
    F -->|Done| G["Unlock and return"]

    D -->|No| H["Short write or EAGAIN<br/>Wait for resources"]
    H -->|Retry| B

    E -.->|TX completion| I["Worker polls TX CQ<br/>CompleteSend"]
    I --> J["Recycle SQ slots<br/>Return shared CQ credits"]
    J --> K["FinishRound outside worker lock<br/>Release payload references<br/>Wake writers"]
    K -.-> H
    L["Received grant"] -.->|Wake writer| H

Loading

Normal send completion does not acquire the connection's submit lock. Send records are published before posting, so completion can safely reclaim them even before ibv_post_send returns. Shared-CQ wakeups indicate availability, and resumed writers still reserve credits before submitting.

Receive path

flowchart LR
    A["NIC consumes SRQ WQE<br/>DMA into registered block"]
    A --> B["CQ event → RunEvent<br/>Poll RX CQ"]
    B --> C["HandleReceive<br/>Find slot and connection<br/>Release slot and validate"]
    C --> D["ReceiveControl<br/>Publish FC events"]

    D -.->|Grant| W["Wake sender"]
    D -.->|Hard request| H["Schedule progress<br/>Send pure grant if pending"]

    D --> E{"Payload?"}
    E -->|No| R["Reuse registered block"]
    E -->|Yes| F{"Zero-copy eligible?"}

    F -->|No| G["Copy payload into IOBuf"]
    G --> R
    F -->|Yes| Z["TakeReceived<br/>Transfer BlockRef"]

    G --> Q["EnqueueReceived<br/>Per-connection MPSC"]
    Z --> Q
    Q --> S["FinishRound → StartDelivery<br/>Outside worker lock"]
    S --> T["Single consumer bthread<br/>Consume → Deliver → RPC"]

    C -.->|Slot available| P["Refill<br/>Reuse or acquire another block<br/>ibv_post_srq_recv"]
    R --> P
    P --> A

    T -.->|Last zero-copy reference released| U["Return block to pool<br/>Notify worker if needed"]
    U --> P

Loading

Soft requests are answered by piggybacking grants on outgoing data. Hard requests schedule a standalone grant if the request remains pending. Grants do not wait for application-held receive buffers to return. When no receive buffers are available, replenishment pauses and RC RNR retries cover temporary SRQ exhaustion.

Side effects:

Performance effects: Recorded development benchmarks used 10,000 connections, 64-byte echo attachments, and four senders sleeping for 1 ms after each RPC. Client and server used different physical NICs on the same host, with event-driven progress. Each configuration ran twice with reversed execution order.

Mode	Client RSS after connection setup	Mean latency	P50	P99
Private RQ	~12,252 MiB	62.1–63.4 µs	58–60 µs	105–118 µs
SRQ	~1,095 MiB	62.7–64.5 µs	60 µs	89–94 µs

Client RSS decreased by approximately 91%. Mean and median latency remained close, while P99 improved in this sparse-request workload.

These are historical measurements documented in docs/cn/rdma_srq.md, rather than a performance rerun of the latest changes. The ranges represent individual runs

Breaking backward compatibility:
SRQ is disabled by default. Existing private-RQ behavior and v2 handshakes remain supported. SRQ requires v3-capable peers with compatible SRQ settings. Failed SRQ negotiation between v3 peers falls back to TCP. SRQ cannot be combined with polling mode.

---

Check List:

• Please make sure your changes are compilable.
• When providing us with a new feature, it is best to add related tests.
• Please follow Contributor Covenant Code of Conduct.

[Copilot]

🟡 Changes recommended

Approval is blocked by three critical issues: an unsynchronized shared_ptr teardown race, missing cancellation for blocked handshake tasks, and a missing <unordered_set> include that breaks clean compilation.

3 open issues

• Synchronize connection ownership checks with teardown · New
• Cancel in-flight handshake tasks when stopping a connection · New
• Include <unordered_set> for SocketId set · New

What changed in this PR

This pull request adds opt-in, event-driven RDMA Shared Receive Queue (SRQ) support while preserving private-RQ behavior and compatible handshake fallbacks.

Changes:

• Adds shared SRQ workers, receive pools, completion queues, flow-control credits, zero-copy buffer reuse, and asynchronous teardown.
• Extends v3 handshake negotiation with SRQ parameters, credit management, and TCP fallback.
• Refactors private-RQ handling behind a common RdmaConnection interface and expands unit and integration test coverage.

Blocking review findings:

• Critical — src/brpc/rdma/rdma_endpoint.cpp:756: Endpoint teardown can race with asynchronous delivery while accessing the same std::shared_ptr, causing undefined behavior.
• Critical — src/brpc/rdma/rdma_endpoint.h:69: The no-op StopConnect hook can leave a blocked handshake running after socket failure or recycling.
• Critical — src/brpc/rdma/rq_connection.cpp:1122: The translation unit uses std::unordered_set without including <unordered_set>, preventing a clean build.

File	Reviewed changes
test/​rdma_test_helper.h	Adds shared RDMA test fixtures, polling helpers, and RPC utilities.
test/​mpsc_queue_unittest.cc	Tests move-only IOBuf ownership transfer through queue emplacement.
test/​BUILD.bazel	Registers RDMA/SRQ test dependencies and scheduling configuration.
test/​brpc_srq_receive_pool_unittest.cpp	Tests receive-pool registration, growth, recycling, and asynchronous events.
test/​brpc_srq_flow_control_unittest.cpp	Tests SRQ credit requests, grants, malformed messages, and concurrent state transitions.
src/​butil/​containers/​mpsc_queue.h	Adds variadic in-place queue construction for ownership-preserving moves.
src/​brpc/​socket.h	Updates socket friendship and copy restrictions for RDMA connection implementations.
src/​brpc/​rdma/​srq_worker.h	Declares shared SRQ worker and connection state.
src/​brpc/​rdma/​srq_receive_pool.h	Declares bounded, registered receive-memory pools.
src/​brpc/​rdma/​srq_receive_pool.cpp	Implements receive-region growth, recycling, notifications, and deregistration retries.
src/​brpc/​rdma/​srq_flow_control.h	Defines SRQ request, grant, and credit-window management.
src/​brpc/​rdma/​rq_connection.h	Declares the private-RQ connection backend.
src/​brpc/​rdma/​rq_connection.cpp	Implements private-RQ allocation, polling, sending, receiving, and cleanup.
src/​brpc/​rdma/​rdma_helper.cpp	Loads SRQ verbs, dispatches asynchronous events, and manages RDMA shutdown.
src/​brpc/​rdma/​rdma_handshake.proto	Adds SRQ protocol and flow-control handshake fields.
src/​brpc/​rdma/​rdma_handshake.h	Extends parsed handshake state for SRQ negotiation.
src/​brpc/​rdma/​rdma_handshake.cpp	Validates, translates, and serializes SRQ handshake parameters.
src/​brpc/​rdma/​rdma_handshake_server.h	Updates server handshake documentation.
src/​brpc/​rdma/​rdma_handshake_server.cpp	Handles v2/v3 fallback handshake behavior.
src/​brpc/​rdma/​rdma_handshake_constants.h	Clarifies handshake constants and wire-format documentation.
src/​brpc/​rdma/​rdma_endpoint.h	Refactors endpoint ownership around selectable RDMA connection backends.
src/​brpc/​rdma/​rdma_connection.h	Defines the common RDMA connection interface and shared helpers.
src/​brpc/​rdma/​rdma_connection.cpp	Implements common QP setup, MTU/ECE handling, and SGE conversion.
src/​brpc/​rdma/​rdma_async_event.h	Declares registered asynchronous verbs-event contexts.
src/​brpc/​rdma/​rdma_async_event.cpp	Implements asynchronous event registration and dispatch.
src/​brpc/​rdma/​block_pool.cpp	Integrates SRQ memory-region lookup and recycling with IOBuf allocation.

This pull request requires changes.

_{Review effort: Lite}

[chenBright]

@wwbmmm @yanglimingcn PTAL. This PR is ready.

[Copilot]

🟡 Changes recommended

Two critical lifetime-safety issues remain active: RDMA shutdown can race with in-flight growth work, and the test socket helper returns a pointer after releasing its owner, risking use-after-recycle.

2 open issues

• Wait for live SRQ pools before releasing the RDMA context · New
• Retain SocketUniquePtr ownership when returning socket addresses · New

3 resolved since last review

• Include <unordered_set> for SocketId set
• Cancel in-flight handshake tasks when stopping a connection
• Synchronize connection ownership checks with teardown

_{Review effort: Lite}

[chenBright]

The latest Copilot review comments have been addressed:

• Verified that the initial registered region and growth-task references protect the RDMA PD/context throughout asynchronous pool growth.
• Fixed socket ownership in the shared test helpers and updated all affected callers.
• Extended the close test to verify reference lifetime and subsequent recycling.

@wwbmmm PTAL.