VPP container fails to start

[chenlein]

Environment

• Calico/VPP version: v3.27.0
• Kubernetes version: v1.27.12
• Deployment type: cloud, hosted, VM, bare-metal...
  bare-metal with bond device
• Network configuration: Calico / host-local IPAM? IPIP / VXLAN enabled? Specific BGP settings?
  Calico BGP with None encapsulation

Issue description
VPP container fails to start

To Reproduce
vpp container logs:

[root@master-0 ~]# kubectl logs -f -n calico-vpp-dataplane calico-vpp-node-fb77h vpp 
time="2024-04-25T12:18:21Z" level=info msg="Version info\nImage tag                   : ab81a775fbdeba932888690c68ddf7e9f4bd8d2b\nVPP-dataplane version       : ab81a77 Release v3.27.0\nVPP Version                 : 24.02-rc0~8-g9db45f6ae\nBinapi-generator version    : v0.8.0\nVPP Base commit             : 06efd532e gerrit:34726/3 interface: add buffer stats api\n------------------ Cherry picked commits --------------------\ncapo: Calico Policies plugin\nacl: acl-plugin custom policies\ncnat: [WIP] no k8s maglev from pods\npbl: Port based balancer\ngerrit:40078/3 vnet: allow format deleted swifidx\ngerrit:40090/3 cnat: undo fib_entry_contribute_forwarding\ngerrit:39507/13 cnat: add flow hash config to cnat translation\ngerrit:34726/3 interface: add buffer stats api\n-------------------------------------------------------------\n"
time="2024-04-25T12:18:21Z" level=info msg="Config:CALICOVPP_FEATURE_GATES={}"
time="2024-04-25T12:18:21Z" level=info msg="Config:CALICOVPP_INITIAL_CONFIG={\n  \"vppStartupSleepSeconds\": 1,\n  \"corePattern\": \"/var/lib/vpp/vppcore.%e.%p\",\n  \"extraAddrCount\": 0,\n  \"ifConfigSavePath\": \"\",\n  \"defaultGWs\": \"\",\n  \"redirectToHostRules\": null\n}"
time="2024-04-25T12:18:21Z" level=info msg="Config:CALICOVPP_GRACEFUL_SHUTDOWN_TIMEOUT=10s"
time="2024-04-25T12:18:21Z" level=info msg="Config:CALICOVPP_INIT_SCRIPT_TEMPLATE="
time="2024-04-25T12:18:21Z" level=info msg="Config:CALICOVPP_HOOK_BEFORE_IF_READ=#!/bin/sh\n\nHOOK=\"$0\"\nchroot /host /bin/sh <<EOSCRIPT\n\nfix_dns () {\n    if systemctl status NetworkManager > /dev/null 2>&1; then\n        echo \"default_hook: system is using NetworkManager; fixing dns...\"\n        sed -i \"s/\\[main\\]/\\[main\\]\\ndns=none/\" /etc/NetworkManager/NetworkManager.conf\n        systemctl daemon-reload\n        systemctl restart NetworkManager\n    fi\n}\n\nundo_dns_fix () {\n    if systemctl status NetworkManager > /dev/null 2>&1; then\n        echo \"default_hook: system is using NetworkManager; undoing dns fix...\"\n        sed -i \"0,/dns=none/{/dns=none/d;}\" /etc/NetworkManager/NetworkManager.conf\n        systemctl daemon-reload\n        systemctl restart NetworkManager\n    fi\n}\n\nrestart_network () {\n    if systemctl status systemd-networkd > /dev/null 2>&1; then\n        echo \"default_hook: system is using systemd-networkd; restarting...\"\n        systemctl restart systemd-networkd\n    elif systemctl status NetworkManager > /dev/null 2>&1; then\n        echo \"default_hook: system is using NetworkManager; restarting...\"\n        systemctl restart NetworkManager\n    elif systemctl status networking > /dev/null 2>&1; then\n        echo \"default_hook: system is using networking service; restarting...\"\n        systemctl restart networking\n    elif systemctl status network > /dev/null 2>&1; then\n        echo \"default_hook: system is using network service; restarting...\"\n        systemctl restart network\n    else\n        echo \"default_hook: Networking backend not detected, network configuration may fail\"\n    fi\n}\n\nif which systemctl > /dev/null; then\n    echo \"default_hook: using systemctl...\"\nelse\n    echo \"default_hook: Init system not supported, network configuration may fail\"\n    exit 1\nfi\n\nif [ \"$HOOK\" = \"BEFORE_VPP_RUN\" ]; then\n    fix_dns\nelif [ \"$HOOK\" = \"VPP_RUNNING\" ]; then\n    restart_network\nelif [ \"$HOOK\" = \"VPP_DONE_OK\" ]; then\n    undo_dns_fix\n    restart_network\nelif [ \"$HOOK\" = \"VPP_ERRORED\" ]; then\n    undo_dns_fix\n    restart_network\nfi\n\nEOSCRIPT\n"
time="2024-04-25T12:18:21Z" level=info msg="Config:CALICOVPP_HOOK_VPP_ERRORED=#!/bin/sh\n\nHOOK=\"$0\"\nchroot /host /bin/sh <<EOSCRIPT\n\nfix_dns () {\n    if systemctl status NetworkManager > /dev/null 2>&1; then\n        echo \"default_hook: system is using NetworkManager; fixing dns...\"\n        sed -i \"s/\\[main\\]/\\[main\\]\\ndns=none/\" /etc/NetworkManager/NetworkManager.conf\n        systemctl daemon-reload\n        systemctl restart NetworkManager\n    fi\n}\n\nundo_dns_fix () {\n    if systemctl status NetworkManager > /dev/null 2>&1; then\n        echo \"default_hook: system is using NetworkManager; undoing dns fix...\"\n        sed -i \"0,/dns=none/{/dns=none/d;}\" /etc/NetworkManager/NetworkManager.conf\n        systemctl daemon-reload\n        systemctl restart NetworkManager\n    fi\n}\n\nrestart_network () {\n    if systemctl status systemd-networkd > /dev/null 2>&1; then\n        echo \"default_hook: system is using systemd-networkd; restarting...\"\n        systemctl restart systemd-networkd\n    elif systemctl status NetworkManager > /dev/null 2>&1; then\n        echo \"default_hook: system is using NetworkManager; restarting...\"\n        systemctl restart NetworkManager\n    elif systemctl status networking > /dev/null 2>&1; then\n        echo \"default_hook: system is using networking service; restarting...\"\n        systemctl restart networking\n    elif systemctl status network > /dev/null 2>&1; then\n        echo \"default_hook: system is using network service; restarting...\"\n        systemctl restart network\n    else\n        echo \"default_hook: Networking backend not detected, network configuration may fail\"\n    fi\n}\n\nif which systemctl > /dev/null; then\n    echo \"default_hook: using systemctl...\"\nelse\n    echo \"default_hook: Init system not supported, network configuration may fail\"\n    exit 1\nfi\n\nif [ \"$HOOK\" = \"BEFORE_VPP_RUN\" ]; then\n    fix_dns\nelif [ \"$HOOK\" = \"VPP_RUNNING\" ]; then\n    restart_network\nelif [ \"$HOOK\" = \"VPP_DONE_OK\" ]; then\n    undo_dns_fix\n    restart_network\nelif [ \"$HOOK\" = \"VPP_ERRORED\" ]; then\n    undo_dns_fix\n    restart_network\nfi\n\nEOSCRIPT\n"
time="2024-04-25T12:18:21Z" level=info msg="Config:CALICOVPP_BGP_LOG_LEVEL=INFO"
time="2024-04-25T12:18:21Z" level=info msg="Config:SERVICE_PREFIX=[10.96.0.0/12]"
time="2024-04-25T12:18:21Z" level=info msg="Config:CALICOVPP_IPSEC_IKEV2_PSK="
time="2024-04-25T12:18:21Z" level=info msg="Config:CALICOVPP_INTERFACES={\n  \"defaultPodIfSpec\": {\n    \"rx\": 1,\n    \"tx\": 1,\n    \"rxqsz\": 0,\n    \"txqsz\": 0,\n    \"isl3\": true,\n    \"rxMode\": 0\n  },\n  \"maxPodIfSpec\": {\n    \"rx\": 10,\n    \"tx\": 10,\n    \"rxqsz\": 1024,\n    \"txqsz\": 1024,\n    \"isl3\": null,\n    \"rxMode\": 0\n  },\n  \"vppHostTapSpec\": {\n    \"rx\": 1,\n    \"tx\": 1,\n    \"rxqsz\": 1024,\n    \"txqsz\": 1024,\n    \"isl3\": false,\n    \"rxMode\": 0\n  },\n  \"uplinkInterfaces\": [\n    {\n      \"rx\": 0,\n      \"tx\": 0,\n      \"rxqsz\": 0,\n      \"txqsz\": 0,\n      \"isl3\": null,\n      \"rxMode\": 0,\n      \"isMain\": false,\n      \"physicalNetworkName\": \"\",\n      \"interfaceName\": \"nm-bond\",\n      \"vppDriver\": \"af_packet\",\n      \"newDriver\": \"\",\n      \"annotations\": null,\n      \"mtu\": 0\n    }\n  ]\n}"
time="2024-04-25T12:18:21Z" level=info msg="Config:CALICOVPP_IPSEC={\n  \"nbAsyncCryptoThreads\": 0,\n  \"extraAddresses\": 0\n}"
time="2024-04-25T12:18:21Z" level=info msg="Config:CALICOVPP_NATIVE_DRIVER="
time="2024-04-25T12:18:21Z" level=info msg="Config:CALICOVPP_HOOK_VPP_RUNNING=#!/bin/sh\n\nHOOK=\"$0\"\nchroot /host /bin/sh <<EOSCRIPT\n\nfix_dns () {\n    if systemctl status NetworkManager > /dev/null 2>&1; then\n        echo \"default_hook: system is using NetworkManager; fixing dns...\"\n        sed -i \"s/\\[main\\]/\\[main\\]\\ndns=none/\" /etc/NetworkManager/NetworkManager.conf\n        systemctl daemon-reload\n        systemctl restart NetworkManager\n    fi\n}\n\nundo_dns_fix () {\n    if systemctl status NetworkManager > /dev/null 2>&1; then\n        echo \"default_hook: system is using NetworkManager; undoing dns fix...\"\n        sed -i \"0,/dns=none/{/dns=none/d;}\" /etc/NetworkManager/NetworkManager.conf\n        systemctl daemon-reload\n        systemctl restart NetworkManager\n    fi\n}\n\nrestart_network () {\n    if systemctl status systemd-networkd > /dev/null 2>&1; then\n        echo \"default_hook: system is using systemd-networkd; restarting...\"\n        systemctl restart systemd-networkd\n    elif systemctl status NetworkManager > /dev/null 2>&1; then\n        echo \"default_hook: system is using NetworkManager; restarting...\"\n        systemctl restart NetworkManager\n    elif systemctl status networking > /dev/null 2>&1; then\n        echo \"default_hook: system is using networking service; restarting...\"\n        systemctl restart networking\n    elif systemctl status network > /dev/null 2>&1; then\n        echo \"default_hook: system is using network service; restarting...\"\n        systemctl restart network\n    else\n        echo \"default_hook: Networking backend not detected, network configuration may fail\"\n    fi\n}\n\nif which systemctl > /dev/null; then\n    echo \"default_hook: using systemctl...\"\nelse\n    echo \"default_hook: Init system not supported, network configuration may fail\"\n    exit 1\nfi\n\nif [ \"$HOOK\" = \"BEFORE_VPP_RUN\" ]; then\n    fix_dns\nelif [ \"$HOOK\" = \"VPP_RUNNING\" ]; then\n    restart_network\nelif [ \"$HOOK\" = \"VPP_DONE_OK\" ]; then\n    undo_dns_fix\n    restart_network\nelif [ \"$HOOK\" = \"VPP_ERRORED\" ]; then\n    undo_dns_fix\n    restart_network\nfi\n\nEOSCRIPT\n"
time="2024-04-25T12:18:21Z" level=info msg="Config:CALICOVPP_LOG_LEVEL=info"
time="2024-04-25T12:18:21Z" level=info msg="Config:CALICOVPP_SRV6={\n  \"localsidPool\": \"\",\n  \"policyPool\": \"\"\n}"
time="2024-04-25T12:18:21Z" level=info msg="Config:CALICOVPP_LOG_FORMAT="
time="2024-04-25T12:18:21Z" level=info msg="Config:CALICOVPP_CONFIG_TEMPLATE=unix {\n  nodaemon\n  full-coredump\n  cli-listen /var/run/vpp/cli.sock\n  pidfile /run/vpp/vpp.pid\n  exec /etc/vpp/startup.exec\n}\napi-trace { on }\ncpu {\n    workers 0\n}\nsocksvr {\n    socket-name /var/run/vpp/vpp-api.sock\n}\nplugins {\n    plugin default { enable }\n    plugin dpdk_plugin.so { disable }\n    plugin calico_plugin.so { enable }\n    plugin ping_plugin.so { disable }\n    plugin dispatch_trace_plugin.so { enable }\n}\nbuffers {\n  buffers-per-numa 131072\n}"
time="2024-04-25T12:18:21Z" level=info msg="Config:CALICOVPP_HOOK_BEFORE_VPP_RUN=#!/bin/sh\n\nHOOK=\"$0\"\nchroot /host /bin/sh <<EOSCRIPT\n\nfix_dns () {\n    if systemctl status NetworkManager > /dev/null 2>&1; then\n        echo \"default_hook: system is using NetworkManager; fixing dns...\"\n        sed -i \"s/\\[main\\]/\\[main\\]\\ndns=none/\" /etc/NetworkManager/NetworkManager.conf\n        systemctl daemon-reload\n        systemctl restart NetworkManager\n    fi\n}\n\nundo_dns_fix () {\n    if systemctl status NetworkManager > /dev/null 2>&1; then\n        echo \"default_hook: system is using NetworkManager; undoing dns fix...\"\n        sed -i \"0,/dns=none/{/dns=none/d;}\" /etc/NetworkManager/NetworkManager.conf\n        systemctl daemon-reload\n        systemctl restart NetworkManager\n    fi\n}\n\nrestart_network () {\n    if systemctl status systemd-networkd > /dev/null 2>&1; then\n        echo \"default_hook: system is using systemd-networkd; restarting...\"\n        systemctl restart systemd-networkd\n    elif systemctl status NetworkManager > /dev/null 2>&1; then\n        echo \"default_hook: system is using NetworkManager; restarting...\"\n        systemctl restart NetworkManager\n    elif systemctl status networking > /dev/null 2>&1; then\n        echo \"default_hook: system is using networking service; restarting...\"\n        systemctl restart networking\n    elif systemctl status network > /dev/null 2>&1; then\n        echo \"default_hook: system is using network service; restarting...\"\n        systemctl restart network\n    else\n        echo \"default_hook: Networking backend not detected, network configuration may fail\"\n    fi\n}\n\nif which systemctl > /dev/null; then\n    echo \"default_hook: using systemctl...\"\nelse\n    echo \"default_hook: Init system not supported, network configuration may fail\"\n    exit 1\nfi\n\nif [ \"$HOOK\" = \"BEFORE_VPP_RUN\" ]; then\n    fix_dns\nelif [ \"$HOOK\" = \"VPP_RUNNING\" ]; then\n    restart_network\nelif [ \"$HOOK\" = \"VPP_DONE_OK\" ]; then\n    undo_dns_fix\n    restart_network\nelif [ \"$HOOK\" = \"VPP_ERRORED\" ]; then\n    undo_dns_fix\n    restart_network\nfi\n\nEOSCRIPT\n"
time="2024-04-25T12:18:21Z" level=info msg="Config:CALICOVPP_HOOK_VPP_DONE_OK=#!/bin/sh\n\nHOOK=\"$0\"\nchroot /host /bin/sh <<EOSCRIPT\n\nfix_dns () {\n    if systemctl status NetworkManager > /dev/null 2>&1; then\n        echo \"default_hook: system is using NetworkManager; fixing dns...\"\n        sed -i \"s/\\[main\\]/\\[main\\]\\ndns=none/\" /etc/NetworkManager/NetworkManager.conf\n        systemctl daemon-reload\n        systemctl restart NetworkManager\n    fi\n}\n\nundo_dns_fix () {\n    if systemctl status NetworkManager > /dev/null 2>&1; then\n        echo \"default_hook: system is using NetworkManager; undoing dns fix...\"\n        sed -i \"0,/dns=none/{/dns=none/d;}\" /etc/NetworkManager/NetworkManager.conf\n        systemctl daemon-reload\n        systemctl restart NetworkManager\n    fi\n}\n\nrestart_network () {\n    if systemctl status systemd-networkd > /dev/null 2>&1; then\n        echo \"default_hook: system is using systemd-networkd; restarting...\"\n        systemctl restart systemd-networkd\n    elif systemctl status NetworkManager > /dev/null 2>&1; then\n        echo \"default_hook: system is using NetworkManager; restarting...\"\n        systemctl restart NetworkManager\n    elif systemctl status networking > /dev/null 2>&1; then\n        echo \"default_hook: system is using networking service; restarting...\"\n        systemctl restart networking\n    elif systemctl status network > /dev/null 2>&1; then\n        echo \"default_hook: system is using network service; restarting...\"\n        systemctl restart network\n    else\n        echo \"default_hook: Networking backend not detected, network configuration may fail\"\n    fi\n}\n\nif which systemctl > /dev/null; then\n    echo \"default_hook: using systemctl...\"\nelse\n    echo \"default_hook: Init system not supported, network configuration may fail\"\n    exit 1\nfi\n\nif [ \"$HOOK\" = \"BEFORE_VPP_RUN\" ]; then\n    fix_dns\nelif [ \"$HOOK\" = \"VPP_RUNNING\" ]; then\n    restart_network\nelif [ \"$HOOK\" = \"VPP_DONE_OK\" ]; then\n    undo_dns_fix\n    restart_network\nelif [ \"$HOOK\" = \"VPP_ERRORED\" ]; then\n    undo_dns_fix\n    restart_network\nfi\n\nEOSCRIPT\n"
time="2024-04-25T12:18:21Z" level=info msg="Config:NODENAME=node-3"
time="2024-04-25T12:18:21Z" level=info msg="Config:CALICOVPP_INTERFACE="
time="2024-04-25T12:18:21Z" level=info msg="Config:CALICOVPP_SWAP_DRIVER="
time="2024-04-25T12:18:21Z" level=info msg="Config:CALICOVPP_CONFIG_EXEC_TEMPLATE="
time="2024-04-25T12:18:21Z" level=info msg="Config:CALICOVPP_DEBUG={}"
default_hook: using systemctl...
time="2024-04-25T12:18:21Z" level=info msg="No pci device for interface nm-bond"
time="2024-04-25T12:18:21Z" level=info msg="-- Environment --"
time="2024-04-25T12:18:21Z" level=info msg="Hugepages            1192"
time="2024-04-25T12:18:21Z" level=info msg="KernelVersion        4.19.90-52"
time="2024-04-25T12:18:21Z" level=info msg="Drivers              map[uio_pci_generic:false vfio-pci:false]"
time="2024-04-25T12:18:21Z" level=info msg="initial iommu status disabled"
time="2024-04-25T12:18:21Z" level=info msg="-- Interface Spec --"
time="2024-04-25T12:18:21Z" level=info msg="Interface Name:      nm-bond"
time="2024-04-25T12:18:21Z" level=info msg="Native Driver:       af_packet"
time="2024-04-25T12:18:21Z" level=info msg="New Drive Name:      "
time="2024-04-25T12:18:21Z" level=info msg="PHY target #Queues   rx:0 tx:0"
time="2024-04-25T12:18:21Z" level=info msg="Tap MTU:             0"
time="2024-04-25T12:18:21Z" level=info msg="-- Interface config --"
time="2024-04-25T12:18:21Z" level=info msg="Node IP4:            10.83.3.44/24"
time="2024-04-25T12:18:21Z" level=info msg="Node IP6:            "
time="2024-04-25T12:18:21Z" level=info msg="PciId:               "
time="2024-04-25T12:18:21Z" level=info msg="Driver:              "
time="2024-04-25T12:18:21Z" level=info msg="Linux IF was up ?    true"
time="2024-04-25T12:18:21Z" level=info msg="Promisc was on ?     false"
time="2024-04-25T12:18:21Z" level=info msg="DoSwapDriver:        false"
time="2024-04-25T12:18:21Z" level=info msg="Mac:                 9c:c2:c4:55:f0:60"
time="2024-04-25T12:18:21Z" level=info msg="Addresses:           [10.83.3.44/24 nm-bond]"
time="2024-04-25T12:18:21Z" level=info msg="Routes:              [{Ifindex: 9 Dst: 10.83.3.0/24 Src: 10.83.3.44 Gw: <nil> Flags: [] Table: 254 Realm: 0}, {Ifindex: 9 Dst: 10.233.36.64/26 Src: <nil> Gw: 10.83.3.48 Flags: [] Table: 254 Realm: 0}, {Ifindex: 9 Dst: 10.233.38.64/26 Src: <nil> Gw: 10.83.3.55 Flags: [] Table: 254 Realm: 0}, {Ifindex: 9 Dst: 10.233.39.0/26 Src: <nil> Gw: 10.83.3.49 Flags: [] Table: 254 Realm: 0}, {Ifindex: 9 Dst: 10.233.43.0/26 Src: <nil> Gw: 10.83.3.51 Flags: [] Table: 254 Realm: 0}, {Ifindex: 9 Dst: 10.233.80.0/26 Src: <nil> Gw: 10.83.3.43 Flags: [] Table: 254 Realm: 0}, {Ifindex: 9 Dst: 10.233.97.192/26 Src: <nil> Gw: 10.83.3.54 Flags: [] Table: 254 Realm: 0}, {Ifindex: 9 Dst: 10.233.101.64/26 Src: <nil> Gw: 10.83.3.42 Flags: [] Table: 254 Realm: 0}, {Ifindex: 9 Dst: 10.233.112.0/26 Src: <nil> Gw: 10.83.3.41 Flags: [] Table: 254 Realm: 0}, {Ifindex: 9 Dst: 10.233.112.64/26 Src: <nil> Gw: 10.83.3.47 Flags: [] Table: 254 Realm: 0}, {Ifindex: 9 Dst: 10.233.133.192/26 Src: <nil> Gw: 10.83.3.53 Flags: [] Table: 254 Realm: 0}, {Ifindex: 9 Dst: 10.233.161.128/26 Src: <nil> Gw: 10.83.3.46 Flags: [] Table: 254 Realm: 0}, {Ifindex: 9 Dst: 10.233.205.192/26 Src: <nil> Gw: 10.83.3.50 Flags: [] Table: 254 Realm: 0}, {Ifindex: 9 Dst: 10.233.226.64/26 Src: <nil> Gw: 10.83.3.52 Flags: [] Table: 254 Realm: 0}, <Dst: nil (default), Ifindex: 9, Gw: 10.83.3.1, >]"
time="2024-04-25T12:18:21Z" level=info msg="PHY original #Queues rx:16 tx:16"
time="2024-04-25T12:18:21Z" level=info msg="MTU                  1500"
time="2024-04-25T12:18:21Z" level=info msg="isTunTap             false"
time="2024-04-25T12:18:21Z" level=info msg="isVeth               false"
time="2024-04-25T12:18:21Z" level=info msg="Running with uplink af_packet"
default_hook: using systemctl...
default_hook: system is using NetworkManager; fixing dns...
time="2024-04-25T12:18:21Z" level=info msg="VPP started [PID 3538088]"
vpp[3538088]: buffer: numa[0] falling back to non-hugepage backed buffer pool (vlib_physmem_shared_map_create: pmalloc_map_pages: Unable to lock pages: Cannot allocate memory)
time="2024-04-25T12:18:22Z" level=info msg="Waiting for VPP... [0/10]"
vpp[3538088]: buffer: numa[1] falling back to non-hugepage backed buffer pool (vlib_physmem_shared_map_create: pmalloc_map_pages: Unable to lock pages: Cannot allocate memory)
vpp[3538088]: buffer: numa[2] falling back to non-hugepage backed buffer pool (vlib_physmem_shared_map_create: pmalloc_map_pages: Unable to lock pages: Cannot allocate memory)
time="2024-04-25T12:18:24Z" level=info msg="Waiting for VPP... [1/10]"
vpp[3538088]: buffer: numa[3] falling back to non-hugepage backed buffer pool (vlib_physmem_shared_map_create: pmalloc_map_pages: Unable to lock pages: Cannot allocate memory)
time="2024-04-25T12:18:26Z" level=info msg="Waiting for VPP... [2/10]"
time="2024-04-25T12:18:28Z" level=info msg="Waiting for VPP... [3/10]"
vpp[3538088]: buffer: numa[4] falling back to non-hugepage backed buffer pool (vlib_physmem_shared_map_create: pmalloc_map_pages: Unable to lock pages: Cannot allocate memory)
time="2024-04-25T12:18:30Z" level=info msg="Waiting for VPP... [4/10]"
time="2024-04-25T12:18:32Z" level=warning msg="Waiting for VPP... [5/10] cannot connect to VPP on socket /var/run/vpp/vpp-api.sock: VPP API socket file /var/run/vpp/vpp-api.sock does not exist"
vpp[3538088]: buffer: numa[5] falling back to non-hugepage backed buffer pool (vlib_physmem_shared_map_create: pmalloc_map_pages: Unable to lock pages: Cannot allocate memory)
time="2024-04-25T12:18:34Z" level=warning msg="Waiting for VPP... [6/10] cannot connect to VPP on socket /var/run/vpp/vpp-api.sock: VPP API socket file /var/run/vpp/vpp-api.sock does not exist"
time="2024-04-25T12:18:36Z" level=warning msg="Waiting for VPP... [7/10] cannot connect to VPP on socket /var/run/vpp/vpp-api.sock: VPP API socket file /var/run/vpp/vpp-api.sock does not exist"
vpp[3538088]: buffer: numa[6] falling back to non-hugepage backed buffer pool (vlib_physmem_shared_map_create: pmalloc_map_pages: Unable to lock pages: Cannot allocate memory)
time="2024-04-25T12:18:38Z" level=warning msg="Waiting for VPP... [8/10] cannot connect to VPP on socket /var/run/vpp/vpp-api.sock: VPP API socket file /var/run/vpp/vpp-api.sock does not exist"
time="2024-04-25T12:18:40Z" level=warning msg="Waiting for VPP... [9/10] cannot connect to VPP on socket /var/run/vpp/vpp-api.sock: VPP API socket file /var/run/vpp/vpp-api.sock does not exist"
vpp[3538088]: buffer: numa[7] falling back to non-hugepage backed buffer pool (vlib_physmem_shared_map_create: pmalloc_map_pages: Unable to lock pages: Cannot allocate memory)
time="2024-04-25T12:18:42Z" level=error msg="Error connecting to VPP: Cannot connect to VPP after 10 tries"
time="2024-04-25T12:18:42Z" level=info msg="Terminating Vpp 1 (SIGINT 3538088)"
time="2024-04-25T12:18:42Z" level=info msg="Restoring configuration"
time="2024-04-25T12:18:42Z" level=info msg="Received signal interrupt, vpp index 1"
time="2024-04-25T12:18:42Z" level=info msg="Signaled vpp (PID 3538088) interrupt"
time="2024-04-25T12:18:42Z" level=info msg="Done with signal interrupt"
vpp[3538088]: received signal SIGINT, PC 0x7fcc346265f2
time="2024-04-25T12:18:42Z" level=info msg="Setting promisc off"
time="2024-04-25T12:18:42Z" level=info msg="restoring address 10.83.3.44/24 nm-bond"
time="2024-04-25T12:18:42Z" level=error msg="cannot add address 10.83.3.44/24 nm-bond back to nm-bond : file exists"
time="2024-04-25T12:18:42Z" level=info msg="restoring route {Ifindex: 9 Dst: 10.83.3.0/24 Src: 10.83.3.44 Gw: <nil> Flags: [] Table: 254 Realm: 0}"
time="2024-04-25T12:18:42Z" level=info msg="restoring routes : {Ifindex: 9 Dst: 10.83.3.0/24 Src: 10.83.3.44 Gw: <nil> Flags: [] Table: 254 Realm: 0} already exists"
time="2024-04-25T12:18:42Z" level=info msg="restoring route {Ifindex: 9 Dst: 10.233.36.64/26 Src: <nil> Gw: 10.83.3.48 Flags: [] Table: 254 Realm: 0}"
time="2024-04-25T12:18:42Z" level=info msg="restoring routes : {Ifindex: 9 Dst: 10.233.36.64/26 Src: <nil> Gw: 10.83.3.48 Flags: [] Table: 254 Realm: 0} already exists"
vpp[3538088]: #0  0x00007fcc349781c2 0x7fcc349781c2
time="2024-04-25T12:18:42Z" level=info msg="restoring route {Ifindex: 9 Dst: 10.233.38.64/26 Src: <nil> Gw: 10.83.3.55 Flags: [] Table: 254 Realm: 0}"
time="2024-04-25T12:18:42Z" level=info msg="restoring routes : {Ifindex: 9 Dst: 10.233.38.64/26 Src: <nil> Gw: 10.83.3.55 Flags: [] Table: 254 Realm: 0} already exists"
time="2024-04-25T12:18:42Z" level=info msg="restoring route {Ifindex: 9 Dst: 10.233.39.0/26 Src: <nil> Gw: 10.83.3.49 Flags: [] Table: 254 Realm: 0}"
time="2024-04-25T12:18:42Z" level=info msg="restoring routes : {Ifindex: 9 Dst: 10.233.39.0/26 Src: <nil> Gw: 10.83.3.49 Flags: [] Table: 254 Realm: 0} already exists"
time="2024-04-25T12:18:42Z" level=info msg="restoring route {Ifindex: 9 Dst: 10.233.43.0/26 Src: <nil> Gw: 10.83.3.51 Flags: [] Table: 254 Realm: 0}"
time="2024-04-25T12:18:42Z" level=info msg="restoring routes : {Ifindex: 9 Dst: 10.233.43.0/26 Src: <nil> Gw: 10.83.3.51 Flags: [] Table: 254 Realm: 0} already exists"
time="2024-04-25T12:18:42Z" level=info msg="restoring route {Ifindex: 9 Dst: 10.233.80.0/26 Src: <nil> Gw: 10.83.3.43 Flags: [] Table: 254 Realm: 0}"
time="2024-04-25T12:18:42Z" level=info msg="restoring routes : {Ifindex: 9 Dst: 10.233.80.0/26 Src: <nil> Gw: 10.83.3.43 Flags: [] Table: 254 Realm: 0} already exists"
time="2024-04-25T12:18:42Z" level=info msg="restoring route {Ifindex: 9 Dst: 10.233.97.192/26 Src: <nil> Gw: 10.83.3.54 Flags: [] Table: 254 Realm: 0}"
time="2024-04-25T12:18:42Z" level=info msg="restoring routes : {Ifindex: 9 Dst: 10.233.97.192/26 Src: <nil> Gw: 10.83.3.54 Flags: [] Table: 254 Realm: 0} already exists"
time="2024-04-25T12:18:42Z" level=info msg="restoring route {Ifindex: 9 Dst: 10.233.101.64/26 Src: <nil> Gw: 10.83.3.42 Flags: [] Table: 254 Realm: 0}"
time="2024-04-25T12:18:42Z" level=info msg="restoring routes : {Ifindex: 9 Dst: 10.233.101.64/26 Src: <nil> Gw: 10.83.3.42 Flags: [] Table: 254 Realm: 0} already exists"
time="2024-04-25T12:18:42Z" level=info msg="restoring route {Ifindex: 9 Dst: 10.233.112.0/26 Src: <nil> Gw: 10.83.3.41 Flags: [] Table: 254 Realm: 0}"
time="2024-04-25T12:18:42Z" level=info msg="restoring routes : {Ifindex: 9 Dst: 10.233.112.0/26 Src: <nil> Gw: 10.83.3.41 Flags: [] Table: 254 Realm: 0} already exists"
time="2024-04-25T12:18:42Z" level=info msg="restoring route {Ifindex: 9 Dst: 10.233.112.64/26 Src: <nil> Gw: 10.83.3.47 Flags: [] Table: 254 Realm: 0}"
time="2024-04-25T12:18:42Z" level=info msg="restoring routes : {Ifindex: 9 Dst: 10.233.112.64/26 Src: <nil> Gw: 10.83.3.47 Flags: [] Table: 254 Realm: 0} already exists"
time="2024-04-25T12:18:42Z" level=info msg="restoring route {Ifindex: 9 Dst: 10.233.133.192/26 Src: <nil> Gw: 10.83.3.53 Flags: [] Table: 254 Realm: 0}"
time="2024-04-25T12:18:42Z" level=info msg="restoring routes : {Ifindex: 9 Dst: 10.233.133.192/26 Src: <nil> Gw: 10.83.3.53 Flags: [] Table: 254 Realm: 0} already exists"
time="2024-04-25T12:18:42Z" level=info msg="restoring route {Ifindex: 9 Dst: 10.233.161.128/26 Src: <nil> Gw: 10.83.3.46 Flags: [] Table: 254 Realm: 0}"
time="2024-04-25T12:18:42Z" level=info msg="restoring routes : {Ifindex: 9 Dst: 10.233.161.128/26 Src: <nil> Gw: 10.83.3.46 Flags: [] Table: 254 Realm: 0} already exists"
time="2024-04-25T12:18:42Z" level=info msg="restoring route {Ifindex: 9 Dst: 10.233.205.192/26 Src: <nil> Gw: 10.83.3.50 Flags: [] Table: 254 Realm: 0}"
time="2024-04-25T12:18:42Z" level=info msg="restoring routes : {Ifindex: 9 Dst: 10.233.205.192/26 Src: <nil> Gw: 10.83.3.50 Flags: [] Table: 254 Realm: 0} already exists"
time="2024-04-25T12:18:42Z" level=info msg="restoring route {Ifindex: 9 Dst: 10.233.226.64/26 Src: <nil> Gw: 10.83.3.52 Flags: [] Table: 254 Realm: 0}"
time="2024-04-25T12:18:42Z" level=info msg="restoring routes : {Ifindex: 9 Dst: 10.233.226.64/26 Src: <nil> Gw: 10.83.3.52 Flags: [] Table: 254 Realm: 0} already exists"
time="2024-04-25T12:18:42Z" level=info msg="restoring route {Ifindex: 9 Dst: <nil> Src: <nil> Gw: 10.83.3.1 Flags: [] Table: 254 Realm: 0}"
time="2024-04-25T12:18:42Z" level=info msg="restoring routes : {Ifindex: 9 Dst: <nil> Src: <nil> Gw: 10.83.3.1 Flags: [] Table: 254 Realm: 0} already exists"
time="2024-04-25T12:18:42Z" level=error msg="Error pinging calico-vpp: Error kill -SIGUSR1 3537865: no such process"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x19a61f7]

goroutine 1 [running]:
main.(*VppRunner).runVpp(0xc00079fef0)
        /vpp-dataplane/vpp-manager/vpp_runner.go:851 +0x4b7
main.(*VppRunner).Run(0xc00079fef0, {0xc000489fb0, 0x1, 0x1})
        /vpp-dataplane/vpp-manager/vpp_runner.go:112 +0x2a7
main.main()
        /vpp-dataplane/vpp-manager/main.go:224 +0x74c
vpp[3538088]: #1  0x00007fcc34554520 0x7fcc34554520
vpp[3538088]: #2  0x00007fcc346265f2 read + 0x12
vpp[3538088]: #3  0x00007fcc34866782 0x7fcc34866782
[root@master-0 ~]# 

Additional context

[onong]

Hi @chenlein,

"KernelVersion 4.19.90-52"

Looks like a pretty old kernel version. Which OS is the system running on?

[sknat]

It seems like there is an issue in the way we handle VPP failure (should be fixed by #696).
But your first issue seems to be that VPP itself failed. As @onong stated, maybe the kernel version is at play.
You can also look at getting a coredump from VPP

  CALICOVPP_INITIAL_CONFIG: |-
    {
      "corePattern": "/var/lib/vpp/vppcore.%e.%p",
      ...
    }

and mounting /var/lib/vpp on the host

[chenlein]

@onong Yes, my kernel version is quite old. I am using the operating system "Kylin Linux Advanced Server V10 (Lance)". I learned from the "System requirements" that the kernel requirement is "Linux kernel 3.10 or later with required dependencies." Will VPP not work with the kernel version I am using?
@sknat Do I need to upgrade to a newer kernel version? Or can I just wait for #696 to be released with my current os version?

[onong]

@chenlein, the "System requirements" part is from Calico really and theoretically it holds true for Calico/VPP too and where it doesn't we call it out. The "Linux kernel 3.10" is one such which we need to call out in our docs I guess. VPP is constantly evolving and may not work with very old kernels.

Do you see a coredump by any chance? Refer to post from @sknat .

Also, could you try and grab the vpp logs if possible. Immediately after the VPP pod starts(and before it terminates), you could execute the following:

kubectl exec -it -n calico-vpp-dataplane calico-vpp-node-fb77h -c vpp -- bash
vppctl sh log

Lastly, #696 just fixes a bug in the way we handle vpp closure while exiting/terminating so no, it wont solve your problem.

[chenlein]

@onong Thank you for your help. I will try to get the coredump and vppctl sh log information tomorrow. I will update here once I have it.

[chenlein]

@onong @sknat FYI

[root@master-0 vpp]# kubectl exec -it -n calico-vpp-dataplane calico-vpp-node-kwbf9 -c vpp -- vppctl sh log
2024/04/27 05:31:21:943 notice     plugin/load    Loaded plugin: abf_plugin.so (Access Control List (ACL) Based Forwarding)
2024/04/27 05:31:21:945 notice     plugin/load    Loaded plugin: acl_plugin.so (Access Control Lists (ACL))
2024/04/27 05:31:21:945 notice     plugin/load    Loaded plugin: adl_plugin.so (Allow/deny list plugin)
2024/04/27 05:31:21:946 notice     plugin/load    Loaded plugin: af_packet_plugin.so (af-packet)
2024/04/27 05:31:21:946 notice     plugin/load    Loaded plugin: af_xdp_plugin.so (AF_XDP Device Plugin)
2024/04/27 05:31:21:946 notice     plugin/load    Loaded plugin: arping_plugin.so (Arping (arping))
2024/04/27 05:31:21:947 notice     plugin/load    Loaded plugin: avf_plugin.so (Intel Adaptive Virtual Function (AVF) Device Driver)
2024/04/27 05:31:21:948 notice     plugin/load    Loaded plugin: bpf_trace_filter_plugin.so (BPF Trace Filter Plugin)
2024/04/27 05:31:21:948 notice     plugin/load    Loaded plugin: bufmon_plugin.so (Buffers monitoring plugin)
2024/04/27 05:31:21:948 notice     plugin/load    Loaded plugin: builtinurl_plugin.so (vpp built-in URL support)
2024/04/27 05:31:21:948 notice     plugin/load    Loaded plugin: capo_plugin.so (Calico Policy)
2024/04/27 05:31:21:948 notice     plugin/load    Loaded plugin: cdp_plugin.so (Cisco Discovery Protocol (CDP))
2024/04/27 05:31:21:948 notice     plugin/load    Loaded plugin: cnat_plugin.so (CNat Translate)
2024/04/27 05:31:21:953 notice     plugin/load    Loaded plugin: crypto_ipsecmb_plugin.so (Intel IPSEC Multi-buffer Crypto Engine)
2024/04/27 05:31:21:953 notice     plugin/load    Loaded plugin: crypto_native_plugin.so (Intel IA32 Software Crypto Engine)
2024/04/27 05:31:21:954 notice     plugin/load    Loaded plugin: crypto_openssl_plugin.so (OpenSSL Crypto Engine)
2024/04/27 05:31:21:954 notice     plugin/load    Loaded plugin: crypto_sw_scheduler_plugin.so (SW Scheduler Crypto Async Engine plugin)
2024/04/27 05:31:21:954 notice     plugin/load    Loaded plugin: ct6_plugin.so (IPv6 Connection Tracker)
2024/04/27 05:31:21:954 notice     plugin/load    Loaded plugin: det44_plugin.so (Deterministic NAT (CGN))
2024/04/27 05:31:21:954 notice     plugin/load    Loaded plugin: dhcp_plugin.so (Dynamic Host Configuration Protocol (DHCP))
2024/04/27 05:31:21:954 notice     plugin/load    Loaded plugin: dispatch_trace_plugin.so (Dispatch Trace)
2024/04/27 05:31:21:954 notice     plugin/load    Loaded plugin: dma_intel_plugin.so (Intel DSA Backend)
2024/04/27 05:31:21:954 notice     plugin/load    Loaded plugin: dns_plugin.so (Simple DNS name resolver)
2024/04/27 05:31:21:961 notice     plugin/load    Plugin disabled: dpdk_plugin.so
2024/04/27 05:31:21:961 notice     plugin/load    Loaded plugin: dslite_plugin.so (Dual-Stack Lite)
2024/04/27 05:31:21:961 notice     plugin/load    Plugin disabled (default): fateshare_plugin.so
2024/04/27 05:31:21:961 notice     plugin/load    Loaded plugin: flowprobe_plugin.so (Flow per Packet)
2024/04/27 05:31:21:961 notice     plugin/load    Loaded plugin: geneve_plugin.so (GENEVE Tunnels)
2024/04/27 05:31:21:962 notice     plugin/load    Loaded plugin: gre_plugin.so (Generic Routing Encapsulation (GRE) plugin)
2024/04/27 05:31:21:962 notice     plugin/load    Loaded plugin: gtpu_plugin.so (GPRS Tunnelling Protocol, User Data (GTPv1-U))
2024/04/27 05:31:21:962 notice     plugin/load    Loaded plugin: hs_apps_plugin.so (Host Stack Applications)
2024/04/27 05:31:21:962 notice     plugin/load    Loaded plugin: hsi_plugin.so (Host Stack Intercept (HSI))
2024/04/27 05:31:21:962 notice     plugin/load    Loaded plugin: http_plugin.so (Hypertext Transfer Protocol (HTTP))
2024/04/27 05:31:21:962 notice     plugin/load    Loaded plugin: http_static_plugin.so (HTTP Static Server)
2024/04/27 05:31:21:962 notice     plugin/load    Plugin disabled (default): idpf_plugin.so
2024/04/27 05:31:21:962 notice     plugin/load    Loaded plugin: igmp_plugin.so (Internet Group Management Protocol (IGMP))
2024/04/27 05:31:21:963 notice     plugin/load    Loaded plugin: ikev2_plugin.so (Internet Key Exchange (IKEv2) Protocol)
2024/04/27 05:31:21:963 notice     plugin/load    Loaded plugin: ila_plugin.so (Identifier Locator Addressing (ILA) for IPv6)
2024/04/27 05:31:21:963 notice     plugin/load    Loaded plugin: ioam_plugin.so (Inbound Operations, Administration, and Maintenance (OAM))
2024/04/27 05:31:21:963 notice     plugin/load    Loaded plugin: ip_session_redirect_plugin.so (IP session redirect)
2024/04/27 05:31:21:963 notice     plugin/load    Loaded plugin: l2tp_plugin.so (Layer 2 Tunneling Protocol v3 (L2TP))
2024/04/27 05:31:21:963 notice     plugin/load    Loaded plugin: l3xc_plugin.so (L3 Cross-Connect (L3XC))
2024/04/27 05:31:21:963 notice     plugin/load    Loaded plugin: lacp_plugin.so (Link Aggregation Control Protocol (LACP))
2024/04/27 05:31:21:963 notice     plugin/load    Loaded plugin: lb_plugin.so (Load Balancer (LB))
2024/04/27 05:31:21:963 notice     plugin/load    Plugin disabled (default): linux_cp_plugin.so
2024/04/27 05:31:21:964 notice     plugin/load    Plugin disabled (default): linux_cp_unittest_plugin.so
2024/04/27 05:31:21:964 notice     plugin/load    Plugin disabled (default): linux_nl_plugin.so
2024/04/27 05:31:21:964 notice     plugin/load    Loaded plugin: lisp_plugin.so (Locator ID Separation Protocol (LISP))
2024/04/27 05:31:21:964 notice     plugin/load    Plugin disabled (default): lisp_unittest_plugin.so
2024/04/27 05:31:21:964 notice     plugin/load    Loaded plugin: lldp_plugin.so (Link Layer Discovery Protocol (LLDP))
2024/04/27 05:31:21:964 notice     plugin/load    Loaded plugin: mactime_plugin.so (Time-based MAC Source Address Filter)
2024/04/27 05:31:21:964 notice     plugin/load    Loaded plugin: map_plugin.so (Mapping of Address and Port (MAP))
2024/04/27 05:31:21:964 notice     plugin/load    Loaded plugin: mdata_plugin.so (Buffer metadata change tracker.)
2024/04/27 05:31:21:965 notice     plugin/load    Loaded plugin: memif_plugin.so (Packet Memory Interface (memif) -- Experimental)
2024/04/27 05:31:21:965 notice     plugin/load    Loaded plugin: mss_clamp_plugin.so (TCP MSS clamping plugin)
2024/04/27 05:31:21:965 notice     plugin/load    Loaded plugin: nat44_ei_plugin.so (IPv4 Endpoint-Independent NAT (NAT44 EI))
2024/04/27 05:31:21:965 notice     plugin/load    Loaded plugin: nat64_plugin.so (NAT64)
2024/04/27 05:31:21:965 notice     plugin/load    Loaded plugin: nat66_plugin.so (NAT66)
2024/04/27 05:31:21:966 notice     plugin/load    Loaded plugin: nat_plugin.so (Network Address Translation (NAT))
2024/04/27 05:31:21:966 notice     plugin/load    Plugin disabled (default): npt66_plugin.so
2024/04/27 05:31:21:966 notice     plugin/load    Loaded plugin: nsh_plugin.so (Network Service Header (NSH))
2024/04/27 05:31:21:966 notice     plugin/load    Loaded plugin: nsim_plugin.so (Network Delay Simulator)
2024/04/27 05:31:21:966 notice     plugin/load    Plugin disabled (default): oddbuf_plugin.so
2024/04/27 05:31:21:966 notice     plugin/load    Loaded plugin: pbl_plugin.so (Port based balancer (PBL))
2024/04/27 05:31:21:967 notice     plugin/load    Loaded plugin: perfmon_plugin.so (Performance Monitor)
2024/04/27 05:31:21:967 notice     plugin/load    Plugin disabled: ping_plugin.so
2024/04/27 05:31:21:967 notice     plugin/load    Loaded plugin: pnat_plugin.so (Policy 1:1 NAT)
2024/04/27 05:31:21:967 notice     plugin/load    Loaded plugin: pppoe_plugin.so (PPP over Ethernet (PPPoE))
2024/04/27 05:31:21:967 notice     plugin/load    Loaded plugin: prom_plugin.so (Prometheus Stats Exporter)
2024/04/27 05:31:21:967 notice     plugin/load    Plugin disabled (default): quic_plugin.so
2024/04/27 05:31:21:968 notice     plugin/load    Loaded plugin: rdma_plugin.so (RDMA IBverbs Device Driver)
2024/04/27 05:31:21:968 notice     plugin/load    Loaded plugin: srv6ad_plugin.so (Dynamic Segment Routing for IPv6 (SRv6) Proxy)
2024/04/27 05:31:21:968 notice     plugin/load    Loaded plugin: srv6adflow_plugin.so (Dynamic Segment Routing for IPv6 (SRv6) Proxy)
2024/04/27 05:31:21:968 notice     plugin/load    Loaded plugin: srv6am_plugin.so (Masquerading Segment Routing for IPv6 (SRv6) Proxy)
2024/04/27 05:31:21:968 notice     plugin/load    Loaded plugin: srv6as_plugin.so (Static Segment Routing for IPv6 (SRv6) Proxy)
2024/04/27 05:31:21:968 notice     plugin/load    Loaded plugin: srv6mobile_plugin.so (SRv6 GTP Endpoint Functions)
2024/04/27 05:31:21:968 notice     plugin/load    Loaded plugin: stn_plugin.so (VPP Steals the NIC (STN) for Container Integration)
2024/04/27 05:31:21:969 notice     plugin/load    Loaded plugin: svs_plugin.so (Source Virtual Routing and Forwarding (VRF) Select)
2024/04/27 05:31:21:969 notice     plugin/load    Loaded plugin: tlsopenssl_plugin.so (Transport Layer Security (TLS) Engine, OpenSSL Based)
2024/04/27 05:31:21:969 notice     plugin/load    Loaded plugin: tlspicotls_plugin.so (Transport Layer Security (TLS) Engine, Picotls Based)
2024/04/27 05:31:21:969 notice     plugin/load    Loaded plugin: tracedump_plugin.so (Streaming packet trace dump plugin)
2024/04/27 05:31:21:970 notice     plugin/load    Loaded plugin: tracenode_plugin.so (Tracing packet node)
2024/04/27 05:31:21:970 notice     plugin/load    Plugin disabled (default): unittest_plugin.so
2024/04/27 05:31:21:970 notice     plugin/load    Loaded plugin: urpf_plugin.so (Unicast Reverse Path Forwarding (uRPF))
2024/04/27 05:31:21:970 notice     plugin/load    Loaded plugin: vhost_plugin.so (Vhost-User)
2024/04/27 05:31:21:971 notice     plugin/load    Loaded plugin: vmxnet3_plugin.so (VMWare Vmxnet3 Device Driver)
2024/04/27 05:31:21:971 notice     plugin/load    Loaded plugin: vrrp_plugin.so (VRRP v3 (RFC 5798))
2024/04/27 05:31:21:971 notice     plugin/load    Loaded plugin: vxlan_plugin.so (VxLan Tunnels)
2024/04/27 05:31:21:971 notice     plugin/load    Loaded plugin: wireguard_plugin.so (Wireguard Protocol)
2024/04/27 05:31:22:114 warn       buffer         numa[1] falling back to non-hugepage backed buffer pool (vlib_physmem_shared_map_create: pmalloc_map_pages: Unable to lock pages: Cannot allocate memory)
2024/04/27 05:31:22:696 warn       buffer         numa[2] falling back to non-hugepage backed buffer pool (vlib_physmem_shared_map_create: pmalloc_map_pages: Unable to lock pages: Cannot allocate memory)
2024/04/27 05:31:24:390 warn       buffer         numa[3] falling back to non-hugepage backed buffer pool (vlib_physmem_shared_map_create: pmalloc_map_pages: Unable to lock pages: Cannot allocate memory)
2024/04/27 05:31:26:717 warn       buffer         numa[4] falling back to non-hugepage backed buffer pool (vlib_physmem_shared_map_create: pmalloc_map_pages: Unable to lock pages: Cannot allocate memory)
2024/04/27 05:31:29:625 warn       buffer         numa[5] falling back to non-hugepage backed buffer pool (vlib_physmem_shared_map_create: pmalloc_map_pages: Unable to lock pages: Cannot allocate memory)
2024/04/27 05:31:33:186 warn       buffer         numa[6] falling back to non-hugepage backed buffer pool (vlib_physmem_shared_map_create: pmalloc_map_pages: Unable to lock pages: Cannot allocate memory)
2024/04/27 05:31:37:415 warn       buffer         numa[7] falling back to non-hugepage backed buffer pool (vlib_physmem_shared_map_create: pmalloc_map_pages: Unable to lock pages: Cannot allocate memory)
2024/04/27 05:31:42:312 warn       perfmon        skipping source 'intel-uncore' - intel_uncore_init: no uncore units found
2024/04/27 05:31:42:319 warn       perfmon        skipping source 'intel-core' - intel_core_init: not a IA-32 CPU
2024/04/27 05:31:42:360 notice     vat-plug/load  Loaded plugin: acl_test_plugin.so
2024/04/27 05:31:42:360 notice     vat-plug/load  Loaded plugin: adl_test_plugin.so
2024/04/27 05:31:42:360 notice     vat-plug/load  Loaded plugin: af_xdp_test_plugin.so
2024/04/27 05:31:42:360 notice     vat-plug/load  Loaded plugin: arping_test_plugin.so
2024/04/27 05:31:42:361 notice     vat-plug/load  Loaded plugin: avf_test_plugin.so
2024/04/27 05:31:42:361 notice     vat-plug/load  Loaded plugin: builtinurl_test_plugin.so
2024/04/27 05:31:42:361 notice     vat-plug/load  Loaded plugin: capo_test_plugin.so
2024/04/27 05:31:42:361 notice     vat-plug/load  Loaded plugin: cdp_test_plugin.so
2024/04/27 05:31:42:361 notice     vat-plug/load  Loaded plugin: ct6_test_plugin.so
2024/04/27 05:31:42:361 notice     vat-plug/load  Loaded plugin: dhcp_test_plugin.so
2024/04/27 05:31:42:361 notice     vat-plug/load  Loaded plugin: dns_test_plugin.so
2024/04/27 05:31:42:361 notice     vat-plug/load  Loaded plugin: flowprobe_test_plugin.so
2024/04/27 05:31:42:361 notice     vat-plug/load  Loaded plugin: geneve_test_plugin.so
2024/04/27 05:31:42:361 notice     vat-plug/load  Loaded plugin: gtpu_test_plugin.so
2024/04/27 05:31:42:361 notice     vat-plug/load  Loaded plugin: http_static_test_plugin.so
2024/04/27 05:31:42:361 error      vat-plug/load  vat_plugin_register: idpf plugin not loaded...
2024/04/27 05:31:42:368 notice     vat-plug/load  Loaded plugin: ikev2_test_plugin.so
2024/04/27 05:31:42:368 notice     vat-plug/load  Loaded plugin: ioam_test_plugin.so
2024/04/27 05:31:42:369 notice     vat-plug/load  Loaded plugin: ip_session_redirect_test_plugin.so
2024/04/27 05:31:42:369 notice     vat-plug/load  Loaded plugin: l2tp_test_plugin.so
2024/04/27 05:31:42:369 notice     vat-plug/load  Loaded plugin: lacp_test_plugin.so
2024/04/27 05:31:42:369 notice     vat-plug/load  Loaded plugin: lb_test_plugin.so
2024/04/27 05:31:42:369 notice     vat-plug/load  Loaded plugin: lisp_test_plugin.so
2024/04/27 05:31:42:369 notice     vat-plug/load  Loaded plugin: lldp_test_plugin.so
2024/04/27 05:31:42:369 notice     vat-plug/load  Loaded plugin: mactime_test_plugin.so
2024/04/27 05:31:42:369 notice     vat-plug/load  Loaded plugin: mdata_test_plugin.so
2024/04/27 05:31:42:369 notice     vat-plug/load  Loaded plugin: memif_test_plugin.so
2024/04/27 05:31:42:369 notice     vat-plug/load  Loaded plugin: nsh_test_plugin.so
2024/04/27 05:31:42:369 notice     vat-plug/load  Loaded plugin: nsim_test_plugin.so
2024/04/27 05:31:42:369 error      vat-plug/load  vat_plugin_register: oddbuf plugin not loaded...
2024/04/27 05:31:42:376 notice     vat-plug/load  Loaded plugin: pppoe_test_plugin.so
2024/04/27 05:31:42:376 notice     vat-plug/load  Loaded plugin: rdma_test_plugin.so
2024/04/27 05:31:42:376 notice     vat-plug/load  Loaded plugin: stn_test_plugin.so
2024/04/27 05:31:42:376 notice     vat-plug/load  Loaded plugin: tlsopenssl_test_plugin.so
2024/04/27 05:31:42:376 notice     vat-plug/load  Loaded plugin: tracedump_test_plugin.so
2024/04/27 05:31:42:377 notice     vat-plug/load  Loaded plugin: tracenode_test_plugin.so
2024/04/27 05:31:42:377 notice     vat-plug/load  Loaded plugin: vlib_vlibapi_test_plugin.so
2024/04/27 05:31:42:377 notice     vat-plug/load  Loaded plugin: vmxnet3_test_plugin.so
2024/04/27 05:31:42:377 notice     vat-plug/load  Loaded plugin: vnet_arp_test_plugin.so
2024/04/27 05:31:42:377 notice     vat-plug/load  Loaded plugin: vnet_interface_test_plugin.so
2024/04/27 05:31:42:377 notice     vat-plug/load  Loaded plugin: vnet_ip6_nd_test_plugin.so
2024/04/27 05:31:42:377 notice     vat-plug/load  Loaded plugin: vnet_ip_test_plugin.so
2024/04/27 05:31:42:377 notice     vat-plug/load  Loaded plugin: vnet_ipsec_test_plugin.so
2024/04/27 05:31:42:377 notice     vat-plug/load  Loaded plugin: vnet_l2_test_plugin.so
2024/04/27 05:31:42:377 notice     vat-plug/load  Loaded plugin: vnet_session_test_plugin.so
2024/04/27 05:31:42:377 notice     vat-plug/load  Loaded plugin: vnet_sr_mpls_test_plugin.so
2024/04/27 05:31:42:377 notice     vat-plug/load  Loaded plugin: vpp_api_test_plugin.so
2024/04/27 05:31:42:377 notice     vat-plug/load  Loaded plugin: vrrp_test_plugin.so
[root@master-0 vpp]# ll /var/lib/vpp
total 0

[onong]

Thanks for the logs @chenlein. Sorry I missed out the fact that you seem to be using bond interface (nm-bond). bond interfaces are not supported at the moment so thats probably the root cause :( Could you pls try it out with a normal interface to verify?

[chenlein]

@onong Sorry, I am unable to modify the network configuration of the current environment at will. If there is a chance in the future, I will try other testing environments. If there are any new situations, I will come here to update. For now, I will close the current issuer. Thank you again for your help!