VPP Deployment Reference

Source: 83 blog articles from ipng.ch (Pim van Pelt / IPng Networks, AS8298 / AS50869), VyOS VPP integration, and GoVPP documentation.

Full article archive: ~/Code/site/ipng.ch/articles/ (external, not checked in) Consolidated notes: docs/research/vpp-deployment-notes.md Feasibility analysis: docs/research/ze-vpp-analysis.md

Production Architecture (IPng AS8298)

IPng Networks runs VPP as the production forwarding plane on all 12 backbone routers across Europe and US. VPP replaces Linux kernel routing, achieving 8-35x forwarding performance.

startup.conf Reference

Production Values (Supermicro SYS-5018D-FN8T, Xeon D1518 4C)

startup.conf Syntax

VPP's startup.conf uses a custom format (not INI, not YAML):

section-name {
  key value
  key value value
  flag-without-value
  nested-section {
    key value
  }
}

String values are unquoted. Multi-word values use spaces. Booleans are bare flags (present = true). Dev entries use PCI address as the key with a nested block for per-device options.

System Prerequisites

Hugepage Sizing

Formula: main_heap + buffers * data_size + statseg + overhead. Round up generously.

NIC Support and Driver Binding

Driver Matrix

DPDK Bind Sequence (vfio-pci)

Per configured PCI address:

1. Read current driver: /sys/bus/pci/devices/<addr>/driver symlink basename
2. Save original driver to persistent state
3. Load kernel modules: vfio, vfio_pci, vfio_iommu_type1
4. Unbind from current driver: write PCI address to /sys/bus/pci/devices/<addr>/driver/unbind
5. Bind to vfio-pci: write vendor:device to /sys/bus/pci/drivers/vfio-pci/new_id

Reverse on teardown: unbind vfio-pci, PCI rescan (/sys/bus/pci/rescan), rebind original driver.

Source: VyOS control_host.py (Python, ported to Go for ze).

Mellanox RDMA Exception

mlx5 NICs use bifurcated driver mode. The kernel mlx5_core driver stays loaded. VPP creates the interface via create interface rdma host-if <pci-addr> instead of a DPDK dev entry.

This means: - No vfio binding step - No driver save/restore - DPDK section in startup.conf does NOT include mlx5 devices - Interface creation uses GoVPP RdmaCreate API instead of DPDK auto-discovery - Performance is equivalent to DPDK for mlx5

This describes VPP's general Mellanox handling. Ze's own VPP integration does not yet implement RDMA/mlx5 interface creation (there is no RdmaCreate call in the codebase); this section is background for a future deployment path.

NIC-Specific Quirks

LCP (Linux Control Plane) Plugin

Architecture

LCP creates TAP mirrors in Linux for every VPP interface. Routing daemons (ze's BGP) run in Linux and see TAP devices. VPP's netlink plugin syncs kernel route changes to VPP's FIB.

VPP interface (TenGigabitEthernet3/0/0) <---> Linux TAP (in dataplane netns)
                                                  |
                                              ze BGP reactor (TCP bind on TAP)

lcpng vs Upstream LCP

Recommendation for ze: Start with upstream LCP. Evaluate lcpng when vpp-3 (MPLS) needs LCP MPLS support.

TAP Interface Naming

LCP creates TAPs with VPP's interface name, truncated to Linux's 15-char limit. TenGigabitEthernet3/0/0 becomes a truncated name. vppcfg uses short names. Ze's naming module (vpp-4 spec) handles bidirectional mapping.

Network Namespace

All LCP TAPs are created in the dataplane netns. This isolates the forwarding plane from the management plane. Ze's BGP reactor, SSH, web UI all run in this netns.

GoVPP Connection

Socket Transport

GoVPP connects via Unix socket at /run/vpp/api.sock. Pure Go, no CGo.

AsyncConnect pattern: - 10 connection attempts, 1 second interval (initial connect only) - Returns event channel for connection state changes (Connected, Disconnected) - Ze consumes the first state event and returns an error on any non-Connected state; there is no automatic reconnect-with-backoff loop after an established session drops (reconnection is the caller's responsibility)

API Clients

GoVPP provides typed RPC service clients generated from VPP's .api.json files:

Stats Client (separate from binary API)

Stats use shared memory, not the binary API socket. Separate connection:

Stats socket default: /run/vpp/stats.sock

Performance Reference

Efficiency Under Load

VPP gets more efficient as load increases due to instruction/data cache reuse in the vector processing model. vectors_per_call metric shows this: 1.00 = idle (one packet at a time), 256 = saturated (processing full vectors). Production typically 4-16.

Hardware Reference

What VPP Does NOT Provide

• No routing protocols (BGP, OSPF, IS-IS). Ze's job.
• No firewall (nftables/iptables bypassed). VPP ACLs are basic match/action.
• No hierarchical QoS. VPP policers are flat token buckets, not HTB/HFSC.
• No kernel features on fast path (tc, XDP, conntrack, eBPF).
• DPDK/RDMA NIC drivers only (not all hardware supported).
• Minimum 4 physical CPU cores, 8 GB RAM.