VPP Data Plane

Why VPP is in Ze

Ze is a BGP daemon. BGP produces forwarding decisions; something else has to carry the packets. The default answer on Linux is the kernel route table, programmed via netlink by the fib-kernel plugin. That works, but it puts every packet through the kernel's software forwarding path, which tops out at a few Mpps on commodity hardware and starts dropping under bursty load.

VPP (the FD.io Vector Packet Processor) is a user-space software router built on DPDK. It takes the NICs away from the kernel, batches packets into vectors, and walks each vector through a graph of forwarding nodes with hot caches. On the same hardware where the kernel loses packets, VPP forwards at line rate: roughly 35 Mpps on a 4-core Xeon D-1518, with numbers like 18 Mpps per thread for MPLS and around 14 Mpps per thread for VXLAN. IPng Networks has run this stack in production on AS8298 for several years.

Adding VPP to ze is not about replacing BGP. It gives ze a forwarding-plane answer for IXP route servers, production edge routers, and gokrazy appliances on small hardware such as an N100 mini-PC. The control plane stays in ze (BGP, RIB, config, CLI, web UI). The forwarding plane becomes VPP. The two talk through a small, typed interface (GoVPP over a Unix socket).

How the two halves fit together

┌──────────────────────────────────────────────────────────┐
│                          ze                              │
│                                                          │
│   BGP reactor ──▶ protocol RIB ──▶ sysRIB                │
│                                     │                    │
│                              ┌──────┴──────┐             │
│                              │             │             │
│                         fib-kernel     fib-vpp           │
│                         (netlink)      (GoVPP)           │
│                              │             │             │
│                              ▼             ▼             │
│                       Linux FIB       VPP FIB            │
│                        (local)        (transit)          │
│                                                          │
│   vpp component: starts VPP, binds NICs, watches health  │
└──────────────────────────────────────────────────────────┘

Ze keeps two FIB backends and runs them side by side. fib-kernel still programs the Linux route table so local services (SSH, the web UI, BGP TCP sessions) keep working. fib-vpp pushes the same best routes into VPP's FIB via the GoVPP binary API, so transit packets are forwarded by VPP at DPDK speed. Both plugins subscribe to the (system-rib, best-change) event on the EventBus and react independently.

BGP sessions themselves still use Linux sockets. VPP's Linux Control Plane (LCP) plugin mirrors every VPP interface as a TAP device, so ze's BGP reactor bind()s and connect()s exactly as it does on a kernel NIC. VPP is invisible to the BGP code path.

Which namespace those TAPs land in is a separate question, and it is the one live limitation here. LCP creates them in the namespace named by vpp.lcp.netns, which defaults to dataplane, following the IPng production convention that isolates the forwarding plane from the management plane. Ze's BGP reactor has no network-namespace awareness: it binds in whatever namespace the ze process itself runs in. So on the shipped default, ze's BGP cannot bind on an LCP-shadowed interface unless ze is itself running in the dataplane namespace.

Setting vpp.lcp.netns to host or root does not work around this. To VPP those are ordinary namespace names: it resolves the value to the path /var/run/netns/<name>, and ze passes the same leaf to VPP as its global default netns, which VPP falls back to for any pair that carries no namespace of its own. So host asks VPP for a namespace literally called host, which normally does not exist.

Two remedies work today. Leave vpp.lcp.netns empty: that clears VPP's global default netns, and LCP then creates the TAPs in VPP's own namespace, where ze runs. Or run ze's BGP in the namespace the leaf names. Nothing else does, until BGP learns to bind inside a named namespace (specced, not implemented).

ze doctor reports doctor-vpp-lcp-netns for any non-empty vpp.lcp.netns, and a BGP stanza is not required to see it: host and root get one warning of their own, because the value is wrong even where nothing binds. On Linux the check also opens the namespace directory, and reports a namespace that is absent (LCP pair creation would fail at apply) separately from a probe that could not answer. Run ze explain doctor-vpp-lcp-netns for the detail.

What ze does for you

When vpp.enabled is true, ze takes ownership of the whole VPP lifecycle. You do not run systemd start vpp; you do not edit /etc/vpp/startup.conf; you do not dpdk-devbind your NICs by hand. Ze does the following, in order, every time it starts or VPP crashes:

The point is that VPP's system-level prerequisites (vfio module load, NIC unbind, driver save, rescan-on-teardown) are part of ze's job, not the operator's. This matters on a gokrazy appliance where there is no systemd and ze is PID 1 for the data plane.

Running against an externally supervised VPP

Set vpp.external true when something other than ze owns the VPP process -- a systemd unit, a container sidecar, or the Python stub the functional tests use. In this mode ze skips steps 3, 4, 5, 6 and 11 of the table above: no startup.conf generation, no vfio module load, no NIC unbind, no exec vpp, no PCI rescan on shutdown. Ze still connects via GoVPP at step 7, emits the same lifecycle events at step 8, runs the stats poller at step 9, and blocks on context cancellation instead of cmd.Wait at step 10.

Typical configurations:

vpp {
  enabled  true;
  external true;
  api-socket /run/vpp/api.sock;
}

The operator owns startup.conf, owns the systemd unit, owns the vfio bind. Ze owns the API socket conversation and the FIB programming.

Use this for: - Systemd deployments: the system VPP unit starts before ze, ze connects to its API socket on startup. - Gokrazy containers: a sidecar image bundles vpp and ze; the supervisor starts VPP first, then ze with external true. - Functional tests: bin/ze-test vpp runs test/vpp/*.ci which drive a Python vpp_stub.py listening on the API socket instead of a real VPP. See docs/functional-tests.md for the harness.

Configuring VPP

The vpp { ... } container lives in the main ze config. Minimal example:

vpp {
  enabled true;
  dpdk {
    interface 0000:03:00.0 {
      name xe0;
    }
    interface 0000:03:00.1 {
      name xe1;
    }
  }
}

This is enough to boot VPP with the default heap, default buffer count, default stats segment and LCP enabled. Add cores, tune memory, or change the stats poll interval only when the defaults do not fit the workload.

Every leaf, what it does, what it defaults to

Enabling FIB programming

The VPP component starts VPP, but it does not program routes. That is fib-vpp's job, and it has its own switch under the fib container:

fib {
  vpp {
    enabled true;
    table-id 0;
  }
}

fib-vpp depends on the vpp subsystem and on the RIB plugin. If VPP is disabled or the GoVPP channel fails to open, the plugin falls back to a noop backend and logs a warning instead of blocking the rest of ze.

System prerequisites

VPP is not a user-space toy; DPDK needs real kernel cooperation. On a general-purpose host Ze cannot set kernel boot parameters, so you reserve hugepages yourself. On the gokrazy appliance Ze owns the kernel cmdline: set image.hugepages (and optionally image.memory-bytes) in appliance.json and ze appliance build bakes default_hugepagesz/hugepagesz/hugepages into the boot cmdline for you (see the appliance guide). ze doctor reports doctor-vpp-hugepages when VPP is enabled but the reservation is missing, insufficient, or clamped. Before enabling VPP:

Supported NICs through DPDK include Intel i210/i350, X520/X540, X710/XL710 and E810 families, plus VirtIO for VMs. Mellanox ConnectX-5 and later use RDMA in a bifurcated driver mode; the current ze implementation is DPDK only and does not yet bind Mellanox NICs.

Observing what VPP is doing

Two places expose VPP state through ze:

1. Prometheus metrics. The stats poller exports per-interface counters (rx/tx packets, bytes, drops, errors), per-node clocks and vectors-per-call, and system-wide vector and input rates. Poll interval is configurable via vpp.stats.poll-interval.

2. The show fib vpp command. Dumps the routes fib-vpp believes it has installed in VPP, as JSON [{"prefix": ..., "next-hop": ...}, ...].

Direct VPP introspection (vppctl show int, vppctl show ip fib) is still available through the CLI socket ze writes to /run/vpp/cli.sock.

What is not yet wired

Today, VPP process lifecycle, IPv4/IPv6 FIB programming, and stats telemetry are in the tree. The remaining phases:

| vpp-5 | L2 cross-connect, bridge domains, policers, ACLs, SRv6, sFlow | Depends on vpp-4. Each feature is independent. (VXLAN tunnels landed with spec-followup-vpp-iface.) | | (netns) | Netns-aware BGP listening: BGP binds inside a named network namespace | Not in tree. RealListenerFactory.Listen binds via net.ListenConfig in whatever namespace the ze process runs in; nothing in the BGP reactor references a network namespace. So with the default vpp.lcp.netns of dataplane, BGP reaches an LCP TAP only if ze itself runs in that namespace (see "How the two halves fit together"). Specced, not implemented; ze doctor reports doctor-vpp-lcp-netns meanwhile. |

The three-strategy framing in docs/research/ze-vpp-analysis.md explains the bigger picture: strategy 1 (IPng / VyOS style, netlink intermediary) is the safe default, strategy 3 (direct FIB programming, what ze does) is where ze differentiates by skipping the kernel entirely and converging sub-second on a full table.

References and further reading