Skip to content

nat44-ed: address-only static mappings looked up with the packet's protocol (all flows on one worker; other IP protocols dropped) #3743

Description

@anvanster

Summary

nat44_ed_add_static_mapping stores an address-only static mapping with port
0 and protocol 0 (e_port = l_port = proto = 0). Three lookups then
search for it with the packet's protocol instead of 0, so they never match it:

Code Effect
nat44_ed_get_out2in_worker_index (nat44_ed.c), "first try static mappings without port" Misses, falls back to get_thread_idx_by_port. All new inbound flows to a given destination port go to the same worker, whatever the mapping's own worker is.
nat44_ed_out2in_slowpath_unknown_proto (nat44_ed_out2in.c) Inbound packets of protocols other than TCP/UDP/ICMP to an address-only mapping are dropped as no translation.
unknown-protocol path of nat44-ed-in2out-slowpath (nat44_ed_in2out.c) Outbound packets of other protocols from the mapped host miss the mapping and fall back to dynamic translation; with no pool address they are dropped, with no NAT error counter.

nat44_ed_external_sm_lookup in nat44_ed_in2out.c already does the right
thing: it retries with nat44_ed_sm_o2i_lookup (sm, addr, 0, 0, 0) for
address-only mappings.

Impact

With many address-only 1:1 mappings serving the same port (for example one
VRF per tenant or per VM, each with a 1:1 mapping), every new inbound flow to
that port is handed off to one worker. In our tests that capped throughput at
0.61 Mpps with 4 workers, and in bursts of 1,000 simultaneous new connections
the handoff queue to that worker (default 64) overflowed and dropped SYNs.
Protocols such as GRE or SCTP do not work through an address-only 1:1
mapping in either direction.

Version and environment

Version:                  v26.06-release
Compiler:                 Clang/LLVM 18.1.3 (1ubuntu1)

fd.io release packages on Ubuntu 24.04, x86_64, 4 workers. The same three
lookups are present on master (025b64cdbe).

Reproduction

Self-contained script (veth pairs and network namespaces on one host, plain
vppctl, iproute2 and python3), included at the end of this issue as
nat44-ed-repro.sh. It
needs a running VPP with at least 2 workers, nat_plugin and
af_packet_plugin, and NAT44-ED not yet enabled. It creates:

  • an outside client namespace at 10.10.10.2, connected to host-vout
    (10.10.10.1, nat44 out);
  • four inside namespaces, each at 172.16.0.2/30 in its own VRF (1–4),
    connected to host-vin$i (172.16.0.1, nat44 in), with
    nat44 add static mapping local 172.16.0.2 external 10.20.0.$i vrf $i
    and ip route add 0.0.0.0/0 table $i via 10.10.10.2 next-hop-table 0.

It then sends 200 new UDP flows (50 source ports × 4 external addresses),
20 inbound packets of IP protocol 253 to 10.20.0.1, and 20 outbound packets
of IP protocol 253 from 172.16.0.2 in VRF 1.

Expected

  • Flows spread over the workers (each mapping's worker is hashed from its
    local address and FIB, so different VRFs land on different workers).
  • Protocol 253 is translated in both directions by the address-only 1:1
    mapping.

Actual (v26.06)

== 1. 200 UDP flows (50 source ports x 4 external addresses), sessions per thread:
-------- thread 0 vpp_main: 0 sessions --------
-------- thread 1 vpp_wk_0: 200 sessions --------
-------- thread 2 vpp_wk_1: 0 sessions --------
-------- thread 3 vpp_wk_2: 0 sessions --------
-------- thread 4 vpp_wk_3: 0 sessions --------
== 2. inbound IP protocol 253 x20 to 10.20.0.1, delivered to nat-in1:
   received: 0 of 20
        20      nat44-ed-out2in-slowpath                no translation             error
== 3. outbound IP protocol 253 x20 from 172.16.0.2 (VRF 1), source seen in nat-out:
   received: 0 of 20, sources:

Trace of an outbound packet, from our original setup (an af_packet inside
interface host-wp-tap0, host B at 10.10.0.2): the slow path does not match
the mapping and drops the packet; the only counter is at the drop node.

00:00:05:584854: nat44-ed-in2out
  NAT44_IN2OUT_ED_FAST_PATH: sw_if_index 2, next index 3
  search key local 172.16.0.2:0 remote 10.10.0.2:0 proto EXP1 fib 1 thread-index 0 session-index 0
00:00:05:584863: nat44-ed-in2out-slowpath
  NAT44_IN2OUT_ED_SLOW_PATH: sw_if_index 2, next index 0
00:00:05:584870: error-drop
  rx:host-wp-tap0
00:00:05:584871: drop
  ethernet-input: no error

With the fix (proposed in a PR)

Looking all three up with protocol 0, the same script gives:

-------- thread 1 vpp_wk_0: 50 sessions --------
-------- thread 2 vpp_wk_1: 50 sessions --------
-------- thread 3 vpp_wk_2: 50 sessions --------
-------- thread 4 vpp_wk_3: 50 sessions --------
== 2. ... received: 20 of 20   (no 'no translation' errors)
== 3. ... received: 20 of 20, sources: 10.20.0.1

We also ran TCP traffic through it with VMs behind the inside interfaces
(24/24 transfers of 200 KB, sessions spread evenly).

Possibly related, different bug in a neighbouring path: #3728.

nat44-ed-repro.sh
#!/usr/bin/env bash
# Reproduce: NAT44-ED looks up address-only static mappings with the packet's
# protocol, but stores them with protocol 0.
#
# Needs: a running VPP with at least 2 workers and nat_plugin and
# af_packet_plugin loaded, NAT44-ED not yet enabled; root; iproute2; python3.
# Builds everything on one host with veth pairs and network namespaces:
#
#   nat-out (10.10.10.2) --veth-- VPP host-vout 10.10.10.1   [nat44 outside]
#   nat-in$i (172.16.0.2) --veth-- VPP host-vin$i 172.16.0.1  [nat44 inside, VRF $i]
#   static mapping: local 172.16.0.2 external 10.20.0.$i vrf $i (address-only)
#
# Checks:
#   1. 200 new UDP flows to the 4 external addresses: NAT sessions per worker.
#   2. 20 inbound packets of IP protocol 253 to 10.20.0.1.
#   3. 20 outbound packets of IP protocol 253 from 172.16.0.2 in VRF 1.
#
# Usage: sudo ./nat44-ed-repro.sh   (VPPCTL="vppctl -s /run/vpp/cli.sock" to override)
set -euo pipefail
VPPCTL=${VPPCTL:-vppctl}
N=4
vpp() { $VPPCTL "$@" | tr -d '\r'; }

cleanup() {
	set +e
	for i in $(seq $N); do ip link del "vin$i" 2>/dev/null; ip netns del "nat-in$i" 2>/dev/null; done
	ip link del vout 2>/dev/null
	ip netns del nat-out 2>/dev/null
}
trap cleanup EXIT
cleanup

vpp nat44 plugin enable sessions 65536

# Outside: a client namespace.
ip netns add nat-out
ip link add vout type veth peer name vout-ns
ip link set vout-ns netns nat-out
ip -n nat-out addr add 10.10.10.2/24 dev vout-ns
ip -n nat-out link set vout-ns up
ip -n nat-out link set lo up
ip -n nat-out route add 10.20.0.0/24 via 10.10.10.1
ip link set vout up
vpp create host-interface name vout >/dev/null
vpp set interface ip address host-vout 10.10.10.1/24
vpp set interface state host-vout up
vpp set interface nat44 out host-vout

# Inside: N servers, each in its own VRF, all with the same address.
for i in $(seq $N); do
	ip netns add "nat-in$i"
	ip link add "vin$i" type veth peer name "vin$i-ns"
	ip link set "vin$i-ns" netns "nat-in$i"
	ip -n "nat-in$i" addr add 172.16.0.2/30 dev "vin$i-ns"
	ip -n "nat-in$i" link set "vin$i-ns" up
	ip -n "nat-in$i" link set lo up
	ip -n "nat-in$i" route add default via 172.16.0.1
	ip link set "vin$i" up
	vpp ip table add "$i"
	vpp create host-interface name "vin$i" >/dev/null
	vpp set interface ip table "host-vin$i" "$i"
	vpp set interface ip address "host-vin$i" 172.16.0.1/30
	vpp set interface state "host-vin$i" up
	vpp set interface nat44 in "host-vin$i"
	vpp ip route add 0.0.0.0/0 table "$i" via 10.10.10.2 next-hop-table 0
	vpp nat44 add static mapping local 172.16.0.2 external "10.20.0.$i" vrf "$i"
done

# Resolve neighbors first so no test packet waits on ARP.
ip netns exec nat-out ping -c1 -W1 10.10.10.1 >/dev/null || true
for i in $(seq $N); do ip netns exec "nat-in$i" ping -c1 -W1 172.16.0.1 >/dev/null || true; done
sleep 1

echo "== 1. 200 UDP flows (50 source ports x $N external addresses), sessions per thread:"
ip netns exec nat-out python3 - "$N" <<'EOF'
import socket, sys
for i in range(1, int(sys.argv[1]) + 1):
    for p in range(50):
        s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
        s.bind(("", 40000 + i * 50 + p))
        s.sendto(b"x", ("10.20.0.%d" % i, 8080))
        s.close()
EOF
sleep 1
vpp show nat44 sessions | grep -E '^-------- thread'

echo "== 2. inbound IP protocol 253 x20 to 10.20.0.1, delivered to nat-in1:"
vpp clear errors
ip netns exec nat-in1 timeout 3 tcpdump -n -l -i vin1-ns 'ip proto 253' 2>/dev/null >/tmp/nat-repro-in.txt &
sleep 1
ip netns exec nat-out python3 -c '
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_RAW, 253)
for _ in range(20): s.sendto(b"proto-253-inbound", ("10.20.0.1", 0))'
wait
echo "   received: $(grep -c ' IP ' /tmp/nat-repro-in.txt) of 20"
vpp show errors | grep -i 'no translation' || echo "   (no 'no translation' errors)"

echo "== 3. outbound IP protocol 253 x20 from 172.16.0.2 (VRF 1), source seen in nat-out:"
ip netns exec nat-out timeout 3 tcpdump -n -l -i vout-ns 'ip proto 253' 2>/dev/null >/tmp/nat-repro-out.txt &
sleep 1
ip netns exec nat-in1 python3 -c '
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_RAW, 253)
for _ in range(20): s.sendto(b"proto-253-outbound", ("10.10.10.2", 0))'
wait
echo "   received: $(grep -c ' IP ' /tmp/nat-repro-out.txt) of 20, sources: $(awk '/ IP / {print $3}' /tmp/nat-repro-out.txt | sort -u | tr '\n' ' ')"
rm -f /tmp/nat-repro-in.txt /tmp/nat-repro-out.txt

Activity

  1. anvanster commented on Oct 1, 2026

    @anvanster
    Author

    Proposed fix on Gerrit: https://gerrit.fd.io/r/c/vpp/+/46487 (nat: look up address-only static mappings with protocol 0). It looks up all three address-only mappings with protocol 0. With it, the reproduction script above spreads the 200 flows 50/50/50/50 and translates IP protocol 253 in both directions. test_nat44_ed passes 85/85 and test_nat44_ed_output 1/1, with EXTENDED_TESTS=1 so the multi-worker class runs too.

    (PR #3744 was closed automatically because this mirror only accepts changes through Gerrit.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions