VPP Data Plane

Status: the VPP component manages the VPP process lifecycle (startup, crash recovery, DPDK NIC binding, GoVPP connection), the fib-vpp plugin programs routes from ze's system RIB directly into VPP's FIB via the GoVPP binary API, and the stats segment is polled for per-interface, per-node and system-wide Prometheus metrics. MPLS label programming, a VPP-native interface backend, and VPP-native features (L2XC, bridge domains, VXLAN, policers, ACLs, SRv6, sFlow) are designed but not yet wired.

Why this matters

Ze is a BGP daemon. BGP produces forwarding decisions; something else has to carry the packets. The default answer on Linux is the kernel route table, programmed via netlink by the fib-kernel plugin. That works, but it puts every packet through the kernel's software forwarding path, which tops out at a few Mpps on commodity hardware and starts dropping under bursty load.

VPP (the FD.io Vector Packet Processor) is a user-space software router built on DPDK. It takes the NICs away from the kernel, batches packets into vectors, and walks each vector through a graph of forwarding nodes with hot caches. On the same hardware where the kernel loses packets, VPP forwards at line rate: roughly 35 Mpps on a 4-core Xeon D-1518, with numbers like 18 Mpps per thread for MPLS and around 14 Mpps per thread for VXLAN. IPng Networks has run this stack in production on AS8298 for several years.

Adding VPP to ze is not about replacing BGP. It is about giving ze a credible answer when someone asks "can I use this for an IXP route server, a production edge router, or a gokrazy appliance on an N100 mini-PC?" The control plane stays in ze (BGP, RIB, config, CLI, web UI). The forwarding plane becomes VPP. The two talk through a small, typed interface (GoVPP over a Unix socket).

How the two halves fit together

┌──────────────────────────────────────────────────────────┐
│                          ze                              │
│                                                          │
│   BGP reactor ──▶ protocol RIB ──▶ sysRIB                │
│                                     │                    │
│                              ┌──────┴──────┐             │
│                              │             │             │
│                         fib-kernel     fib-vpp           │
│                         (netlink)      (GoVPP)           │
│                              │             │             │
│                              ▼             ▼             │
│                       Linux FIB       VPP FIB            │
│                        (local)        (transit)          │
│                                                          │
│   vpp component: starts VPP, binds NICs, watches health  │
└──────────────────────────────────────────────────────────┘

Ze keeps two FIB backends and runs them side by side. fib-kernel still programs the Linux route table so local services (SSH, the web UI, BGP TCP sessions) keep working. fib-vpp pushes the same best routes into VPP's FIB via the GoVPP binary API, so transit packets are forwarded by VPP at DPDK speed. Both plugins subscribe to the (system-rib, best-change) event on the EventBus and react independently.

BGP sessions themselves still use Linux sockets. VPP's Linux Control Plane (LCP) plugin mirrors every VPP interface as a TAP device in a dataplane network namespace, so ze's BGP reactor can bind() and connect() exactly as it does today. VPP is invisible to the BGP code path.

What ze does for you

When vpp.enabled is true, ze takes ownership of the whole VPP lifecycle. You do not run systemd start vpp; you do not edit /etc/vpp/startup.conf; you do not dpdk-devbind your NICs by hand. Ze does the following, in order, every time it starts or VPP crashes:

The point is that VPP's system-level prerequisites (vfio module load, NIC unbind, driver save, rescan-on-teardown) are part of ze's job, not the operator's. This matters on a gokrazy appliance where there is no systemd and ze is PID 1 for the data plane.

Running against an externally supervised VPP

Set vpp.external true when something other than ze owns the VPP process -- a systemd unit, a container sidecar, or the Python stub the functional tests use. In this mode ze skips steps 3, 4, 5, 6 and 11 of the table above: no startup.conf generation, no vfio module load, no NIC unbind, no exec vpp, no PCI rescan on shutdown. Ze still connects via GoVPP at step 7, emits the same lifecycle events at step 8, runs the stats poller at step 9, and blocks on context cancellation instead of cmd.Wait at step 10.

Typical configurations:

vpp {
  enabled  true;
  external true;
  api-socket /run/vpp/api.sock;
}

The operator owns startup.conf, owns the systemd unit, owns the vfio bind. Ze owns the API socket conversation and the FIB programming.

Use this for: - Systemd deployments: the system VPP unit starts before ze, ze connects to its API socket on startup. - Gokrazy containers: a sidecar image bundles vpp and ze; the supervisor starts VPP first, then ze with external true. - Functional tests: bin/ze-test vpp runs test/vpp/*.ci which drive a Python vpp_stub.py listening on the API socket instead of a real VPP. See docs/functional-tests.md for the harness.

Configuring VPP

The vpp { ... } container lives in the main ze config. Minimal example:

vpp {
  enabled true;
  dpdk {
    interface 0000:03:00.0 {
      name xe0;
    }
    interface 0000:03:00.1 {
      name xe1;
    }
  }
}

This is enough to boot VPP with the default heap, default buffer count, default stats segment and LCP enabled. Add cores, tune memory, or change the stats poll interval only when the defaults do not fit the workload.

Every leaf, what it does, what it defaults to

Enabling FIB programming

The VPP component starts VPP, but it does not program routes. That is fib-vpp's job, and it has its own switch under the fib container:

fib {
  vpp {
    enabled true;
    table-id 0;
  }
}

fib-vpp depends on the vpp subsystem and on the RIB plugin. If VPP is disabled or the GoVPP channel fails to open, the plugin falls back to a noop backend and logs a warning instead of blocking the rest of ze.

System prerequisites

VPP is not a user-space toy; DPDK needs real kernel cooperation. Ze validates its own config, but it cannot set kernel boot parameters or allocate hugepages. Before enabling VPP:

Supported NICs through DPDK include Intel i210/i350, X520/X540, X710/XL710 and E810 families, plus VirtIO for VMs. Mellanox ConnectX-5 and later use RDMA in a bifurcated driver mode; the current ze implementation is DPDK only and does not yet bind Mellanox NICs.

Observing what VPP is doing

Two places expose VPP state through ze:

1. Prometheus metrics. The stats poller exports per-interface counters (rx/tx packets, bytes, drops, errors), per-node clocks and vectors-per-call, and system-wide vector and input rates. Poll interval is configurable via vpp.stats.poll-interval.

2. The show fib vpp command. Dumps the routes fib-vpp believes it has installed in VPP, as JSON [{"prefix": ..., "next-hop": ...}, ...].

Direct VPP introspection (vppctl show int, vppctl show ip fib) is still available through the CLI socket ze writes to /run/vpp/cli.sock.

What is not yet wired

Today, VPP process lifecycle, IPv4/IPv6 FIB programming, and stats telemetry are in the tree. The remaining phases:

The three-strategy framing in docs/research/ze-vpp-analysis.md explains the bigger picture: strategy 1 (IPng / VyOS style, netlink intermediary) is the safe default, strategy 3 (direct FIB programming, what ze does) is where ze differentiates by skipping the kernel entirely and converging sub-second on a full table.

References and further reading