ZeBGP Debugging Tools
This document describes the debugging tools available in ZeBGP for troubleshooting config parsing, BGP message encoding, and session issues.
Quick Reference
| Tool | Purpose | Usage |
|---|---|---|
ze config dump |
Inspect parsed config | ze config dump config.conf, or ze cli -c "show config dump config.conf | json" |
le test peer --decode |
Decode BGP messages | le test peer --decode --sink |
ze.log.* |
Per-subsystem logging | ze.log.bgp.routes=debug ze bgp server config.conf |
| Functional test diff | Test failure analysis | Automatic on message mismatch |
--server N / --client N |
Interactive test debugging | le test bgp encode --server 1 |
1. Config Dump Command
Location: internal/component/config/cli/cmd_dump.go
Parses a config file and displays the interpreted values. Useful for verifying that config parsing works correctly.
Usage
# Human-readable output
ze config dump config.conf
# JSON output (for scripting)
ze cli -c "show config dump config.conf | json"
Example Output
Warnings:
line 5: freeform 'flow' contains nested block 'route match' - data may be lost
router-id: 1.2.3.4
local-as: 65000
peer 10.0.0.1:
local-as: 65000
peer-as: 65001
static-routes:
- prefix: 192.168.0.0/24
next-hop: 10.0.0.254
local-preference: 100
When to Use
- Config not being parsed correctly
- Static routes not appearing
- Verifying attribute values after parsing
- Checking for parser warnings about data loss
2. BGP Message Decoder
Decodes raw BGP messages (hex) into human-readable format. Used automatically by functional tests and available via --decode flag.
Usage
# In ze-peer, show decoded messages
le test peer --decode --sink --port 1790
Example Output
msg recv FFFFFFFF...:0030:02:00000014400101004002060201000000014003040A000001...
UPDATE (len=48)
ORIGIN: IGP
AS_PATH: [1]
NEXT_HOP: 10.0.0.1
NLRI: 192.168.0.0/24
Decoded Attributes
| Code | Name | Decoded Format |
|---|---|---|
| 1 | ORIGIN | IGP, EGP, INCOMPLETE |
| 2 | AS_PATH | [1 2 3] or [{1 2}] for AS_SET |
| 3 | NEXT_HOP | IP address |
| 4 | MED | Integer |
| 5 | LOCAL_PREF | Integer |
| 8 | COMMUNITIES | high:low format |
| 16 | EXT_COMMUNITIES | Hex (0x...) |
3. Test Failure Diff
When a functional test fails with "message mismatch", the output now includes a decoded diff showing exactly what differed.
Example Output
failed: message mismatch
--- Expected vs Received ---
Expected: UPDATE (len=64)
ORIGIN: IGP
AS_PATH: []
NEXT_HOP: 10.0.0.1
LOCAL_PREF: 100
NLRI: 193.0.2.1/32
Received: KEEPALIVE (len=19)
Differences:
- ORIGIN: IGP (missing)
- AS_PATH: [] (missing)
- NEXT_HOP: 10.0.0.1 (missing)
- LOCAL_PREF: 100 (missing)
~ NLRI: expected=193.0.2.1/32, got=
Diff Symbols
-Missing attribute (expected but not received)+Unexpected attribute (received but not expected)~Different value (both present but values differ)
4. Per-Subsystem Logging (ze.log.*)
Environment variables that enable structured logging at key points in the pipeline.
Usage
# Enable all subsystems at debug level
ze.log=debug ze bgp server config.conf
# Enable specific subsystems
ze.log.bgp.config=debug ze.log.bgp.routes=debug ze bgp server config.conf
Subsystems
Names follow <domain>.<component> convention. Run ze env for the full list.
| Variable | What it logs |
|---|---|
ze.log |
Base level for ALL subsystems |
ze.log.bgp.config |
Config parsing and loading |
ze.log.bgp.routes |
Static route handling, UPDATE sending |
ze.log.bgp.reactor.session |
BGP session handling |
ze.log.bgp.reactor.peer |
Peer FSM transitions, session events |
ze.log.bgp.reactor |
Reactor operations (reload, etc.) |
ze.log.plugin.server |
Plugin RPC server |
ze.log.plugin.relay |
Plugin stderr relay |
Levels: disabled, debug, info, warn, err (case-insensitive)
Example Output
time=2024-01-15T13:53:41.182Z level=DEBUG msg="config parsed" subsystem=bgp.config warnings=0
time=2024-01-15T13:53:41.183Z level=DEBUG msg="config loaded" subsystem=bgp.config peers=1
time=2024-01-15T13:53:41.183Z level=DEBUG msg="peer routes configured" subsystem=bgp.config peer=127.0.0.1 routes=2
time=2024-01-15T13:53:41.184Z level=DEBUG msg="FSM transition" subsystem=bgp.reactor.peer peer=127.0.0.1 from=OPENSENT to=OPENCONFIRM
time=2024-01-15T13:53:41.184Z level=INFO msg="session established" subsystem=bgp.reactor.peer peer=127.0.0.1 localAS=1 peerAS=1
time=2024-01-15T13:53:41.184Z level=DEBUG msg="route sent" subsystem=bgp.routes peer=127.0.0.1 prefix=193.0.2.1/32 nextHop=10.0.0.1
Hierarchical Priority
Most specific wins:
ze.log.bgp.reactor.peer=warn- Peer logging at WARNze.log.bgp.reactor=debug- All reactor.* at DEBUGze.log.bgp=info- All bgp.* at INFOze.log=debug- All subsystems at DEBUG
Shell-compatible: ze_log_bgp_routes=debug also works (dot→underscore)
Logging Locations
Currently instrumented:
internal/component/config/loader.go: Config parsing, peer routes (configLogger)internal/component/bgp/reactor/peer.go: FSM, session, route operations (peerLogger, routesLogger)internal/component/bgp/reactor/session.go: RFC 7606 handling (sessionLogger)internal/component/bgp/reactor/reactor.go: Reload, route operations (reactorLogger, routesLogger)
5. Parser Warnings
The parser now collects warnings for potentially problematic patterns, accessible via Parser.Warnings().
Warning Types
| Pattern | Warning |
|---|---|
| Nested block in Freeform | freeform 'X' contains nested block 'Y' - data may be lost |
Viewing Warnings
# config-dump shows warnings
ze config dump config.conf
# Output:
# Warnings:
# line 5: freeform 'flow' contains nested block 'route match' - data may be lost
# Also visible with debug logging
ze.log.config=debug ze bgp server config.conf
5. Granular Debug (runtime)
Set debug per-module with filters and named profiles (verb-first, VyOS-style):
ze set debug module bgp.reactor # Enable debug
ze set debug module bgp.reactor flag update # Add a flag filter
ze set debug module bgp.reactor scope neighbor 192.0.2.1 # Add a scope filter
ze delete debug module bgp.reactor # Disable debug
ze show debug profile name default # Show stored profile (offline)
show debug # Show live daemon state (YANG RPC)
ze set debug profile name deep-trace # Save named profile
ze set debug active name deep-trace # Apply a saved profile after restart
show debug profile name <name> reads from debug.zefs (works offline). show debug
queries the running daemon's actual slogutil state (requires daemon).
Debugging Workflow
1. Config Issues
# Step 1: Check parsed config
ze config dump config.conf
# Step 2: Look for warnings
ze config dump config.conf 2>&1 | grep -i warning
# Step 3: Check JSON for exact values
ze cli -c "show config dump config.conf | json" | jq '.Neighbors[0].StaticRoutes'
2. Route Not Being Sent
# Step 1: Enable route logging
ze.log.bgp.routes=debug ze.log.bgp.config=debug ze bgp server config.conf
# Expected output when working:
# level=DEBUG msg="peer routes configured" subsystem=bgp.config peer=X routes=2
# level=DEBUG msg="route sent" subsystem=bgp.routes peer=X prefix=10.0.0.0/24 nextHop=192.168.0.1
3. Session Issues
# Step 1: Enable session + peer logging
ze.log.bgp.reactor.peer=debug ze.log.bgp.reactor.session=debug ze bgp server config.conf
# Look for FSM transitions and session establishment
A session closed because the peer stopped reading
Ze logs this line at WARN, with no debug logging needed, when it closes a session whose peer accepted no data:
level=WARN msg="BGP session closed: the peer accepted no data for the write budget; ze connects to it again by itself" subsystem=bgp.reactor.session peer=192.0.2.1 budget=30s fix="..." code=doctor-bgp-peer-not-reading
What it means: ze had data to send, and for the whole write budget the peer took none of it while it kept the TCP connection open. Its BGP process is not reading. Ze closed the connection with no NOTIFICATION, because the peer could not have read one, and connects again by itself. The routes are sent again when the session is Established.
A hold timer, a send hold timer or an operator can end the session before the
write budget passes. The blocked route write then still holds the session, and
ze logs this line at ERROR instead, with the code doctor-bgp-write-path-held:
level=ERROR msg="no NOTIFICATION was sent to the peer: another writer held the session's write path, so the connection is closed without one" subsystem=bgp.reactor.session peer=192.0.2.1 fix="..." code=doctor-bgp-write-path-held
Ze did not wait for the blocked write: it failed that write, closed the
connection and connects again by itself. The cause and the fix are the same as
above. ze show diagnostic doctor-bgp-write-path-held names the second cause,
a plugin that does not take its events.
| Where | What it shows |
|---|---|
ze show errors, ze show doctor |
One event per close, code doctor-bgp-peer-not-reading, with the peer and the budget that passed |
ze show bgp peer, field last-error |
peer accepted no data for the write budget (no NOTIFICATION sent), until a later NOTIFICATION replaces it |
ze show diagnostic doctor-bgp-peer-not-reading |
The description of the code |
What to do: on the peer, find why its BGP process stopped reading (CPU load,
a stalled process, a full input queue). The budget of a route write is
ze.fwd.write.deadline (30s by default); a KEEPALIVE, OPEN or NOTIFICATION
has a third of the hold time, between 10s and 30s. Raise
ze.fwd.write.deadline only for a peer that is known to be slow and healthy:
a longer budget keeps a dead session up for longer.
4. Test Failures
# Run single test
le test bgp encode 1
# Output now includes decoded diff automatically
# Look at "Differences:" section to see what's wrong
5. Interactive Functional Test Debugging
Run server and client separately to see live output:
# Terminal 1: Start test server (le test peer)
le test bgp encode --server 1
# Terminal 2: Start test client (zebgp)
le test bgp encode --client 1
Behavior:
- Server waits for client, prints messages received, exits when test completes
- Client connects, sends configured messages, exits when server disconnects
Press Ctrl+C in the client terminal once the server has finished validating: the runner sends SIGTERM to ze for a graceful shutdown.
Use --port to avoid conflicts:
le test bgp encode --server 1 --port 11790
le test bgp encode --client 1 --port 11790
Adding New Logging
To add logging to new code:
import "github.com/ze-software/ze/internal/core/slogutil"
// Create a subsystem logger (once per package)
var myLogger = slogutil.Logger("my.subsystem")
// Log with structured key-value pairs
myLogger.Debug("operation completed", "peer", addr, "count", n)
myLogger.Info("session established", "peer", addr, "localAS", localAS)
myLogger.Warn("unexpected state", "state", state, "expected", expected)
Subsystem naming convention: ze.log. + simplified package path (e.g., bgp.reactor.peer).
File Locations
| File | Purpose |
|---|---|
internal/test/runner/decode.go |
BGP message decoder |
internal/test/peer/peer.go |
Test peer with decode support |
internal/core/slogutil/slogutil.go |
Logging infrastructure |
internal/component/config/cli/cmd_dump.go |
config dump command |
internal/test/cli/cmd_peer.go |
--decode flag |
internal/component/config/parser.go |
Parser warnings |
internal/component/bgp/config/loader.go |
Config logging (configLogger) |
internal/component/bgp/reactor/peer.go |
Peer/route logging (peerLogger, routesLogger) |
internal/component/bgp/reactor/session.go |
Session logging (sessionLogger) |
internal/component/bgp/reactor/reactor.go |
Reactor logging (reactorLogger, routesLogger) |