Ze System Architecture

Legacy document. The hub/orchestrator concepts below are still broadly accurate, but several concrete details have changed and its examples no longer match the shipping system:

• The daemon is started with ze start <config-file>. The bare ze <config-file> launch form was removed from the CLI; the examples below have been corrected to the surviving form.
• The local-as / peer-as / peer-group config grammar shown here has been removed; local AS is now session { asn { local ... } } and peer groups are group blocks (see configuration syntax changes and config syntax).
• Shipped plugins (bgp, rib, gr) run in-process today rather than being forked as separate external ... { run "ze bgp" } processes; the external block is for third-party out-of-process plugins.
• The source tree is organised under internal/core, internal/component, and internal/plugins; there is no internal/bgp/... layout as sketched below.

For the current architecture see Core Design (canonical) and Hub Architecture.

Build personalities

The repository builds ze and le from the one cmd/ze codebase. The root ./ze and ./le launchers execute the cached bin/ze and bin/le personalities and build only when that cache is absent.

./le --name <name> takes one session out of that cache. The launcher consumes the option, builds bin/le-<name>/le with the same tags and toolchain pin as the shared build, and rebuilds it on every call. The file keeps the name le because defaultDispatch selects the personality with registry.LookupRoot(binaryName()), so the session name goes on the directory. The launcher carries the name into the process, and refuseWrongBuildName refuses to answer when the running binary is a different build.

./le --update moves the cache forward instead of stepping around it. It builds the working tree beside bin/le and renames the result into place, so a peer session executing the old binary keeps its inode. The launcher never rebuilds on its own, because one peer's unfinished source would then fail every call in every session. About one call in sixteen compares the binary with the build inputs, after the command has answered. It prints one stderr line when a file git holds unmodified is newer.

Both personalities use the command registry and pipe engine. Their composition roots remain separate: a normal ze build imports no internal/le package, while the non-default ze_le build companion imports internal/le/register.go and exposes its inventory under ze le. Shipped builds do not enable ze_le.

Overview

Ze supports two operating modes:

This document describes Hub mode.

In hub mode, Ze runs as a hub process (ze) that orchestrates separate plugin processes communicating via pipes. This architecture enables:

• Language freedom (plugins can be Go, Python, Rust, etc.)
• Crash isolation (BGP crash doesn't kill RIB)
• Independent development and testing
• Third-party extensibility

Request Metadata Ownership

Request-scoped metadata is owned by the transport wiring, not by command payloads. REST, gRPC, SSH exec, and SSH interactive sessions extract trusted caller information (username, remoteAddr, request cancellation context) at the edge, then pass it through the shared API engine or dispatcher.

The hub composition root converts that metadata into pluginserver.CommandContext; builtin handlers, subsystem dispatch, and plugin RPC routing derive child contexts from it. Plugin JSON payloads and command strings do not carry or override identity metadata.

                           ┌─────────────────────┐
                           │ ze start config.conf│
                           │      (hub)          │
                           └──────────┬──────────┘
                                      │
           ┌──────────────────────────┼──────────────────────────┐
           │                          │                          │
           ▼                          ▼                          ▼
    ┌─────────────┐           ┌─────────────┐           ┌─────────────┐
    │   ze bgp    │           │   ze rib    │           │  ze gr      │
    │  (process)  │           │  (process)  │           │  (process)  │
    └─────────────┘           └─────────────┘           └─────────────┘

Running Ze

Basic Usage

# Start Ze with a config file
ze start config.conf

# The hub process:
# 1. Parses the config file
# 2. Forks plugin processes (ze bgp, ze rib, etc.)
# 3. Routes config to each plugin
# 4. Coordinates startup via 5-stage protocol
# 5. Routes commands and events between plugins

Process Hierarchy

When running, you'll see these processes:

$ ps aux | grep ze
user  1234  ze start config.conf    # Hub process
user  1235  ze bgp                  # BGP protocol handler
user  1236  ze rib                  # Adj-RIB tracking
user  1237  ze gr                   # Graceful Restart
user  1238  /opt/acme/plugin        # Third-party plugin

Configuration File

The hub parses the entire config file (like VyOS):

1. Parse config syntax
2. Validate against combined YANG schema (all plugins)
3. Convert to internal map-of-maps structure
4. Route JSON subtrees to plugins based on handler registration

The config file has three sections, parsed in order:

Section 1: Environment

Global settings applied before forking processes. The block is environment, modeled in ze-hub-conf.yang like every other top-level block.

environment {
    log {
        level debug
    }
    daemon {
        pid /var/run/ze.pid
    }
}

An older env { } block existed for a second runtime that parsed the config itself. That runtime is deleted, and env { } is not a top-level keyword: it fails ze config validate and it fails at boot, with the same message.

Section 2: Plugin Declarations

Which processes to fork. Uses YANG schema from ze-plugin-conf.yang.

plugin {
    # Built-in plugins (shipped with ze)
    external bgp {
        run "ze bgp";
    }
    external rib {
        run "ze rib";
    }
    external gr {
        run "ze gr";
    }

    # Third-party plugins
    external acme {
        run "/opt/acme/monitor-plugin";
        respawn true;           # Restart if it exits, if the plugin permits it
        timeout 60;             # Startup timeout
    }
}

A plugin declares in its Stage-1 registration what its own failure means: restart, ignore or fatal. That declaration decides what ze does. The respawn leaf states what the operator expects, and it can only ask for less than the declaration permits: respawn false leaves a plugin stopped that would have been started again, and respawn true against a plugin that declares it must not be restarted stops ze at startup with an error naming both sides.

Section 3: Plugin Configuration

Configuration for each plugin. Routed by the hub to the appropriate process.

# Routed to ze bgp (handles "bgp" container per ze-bgp-conf.yang)
bgp {
    local-as 65001;
    router-id 1.1.1.1;

    peer transit-a {
        remote {
            ip 192.0.2.1;
            as 65002;
        }
        passive;

        capability {
            # GR plugin handles this path (augments BGP schema)
            graceful-restart {
                enabled true;
                restart-time 120;
            }
        }
    }

    peer-group upstream {
        peer-as 65000;
    }
}

# Routed to ze rib (handles "rib" container per ze-rib.yang)
rib {
    # RIB-specific settings
}

# Routed to /opt/acme/plugin (handles "acme" container per acme.yang)
acme {
    endpoint "https://monitor.example.com";
    interval 30;
}

Plugin Types

Built-in Plugins

Shipped with Ze, same binary:

Third-Party Plugins

Any executable that speaks the plugin protocol:

plugin {
    external my-plugin {
        run "/path/to/plugin";
        respawn true;   # Only if the plugin declares failure-policy restart
    }
}

my-plugin {
    # Config routed here based on plugin's YANG schema
}

Augmenting Plugins

Some plugins (like GR) don't have their own root config block. They augment another plugin's schema:

// ze-gr.yang
augment "/bgp:bgp/bgp:peer/bgp:capability" {
    container graceful-restart {
        leaf enabled { type boolean; }
        leaf restart-time { type uint16; }
    }
}

Config for augmenting plugins appears nested within the augmented schema:

bgp {
    peer transit-a {
        remote {
            ip 192.0.2.1;
            as 65002;
        }
        capability {
            graceful-restart {      # Handled by ze gr, not ze bgp
                enabled true;
            }
        }
    }
}

How augmenting plugins work:

GR registers handler for bgp.peer.capability.graceful-restart. Hub sends just that JSON subtree to GR:

{"enabled": true, "restart-time": 120}

GR then uses the capability API to inject the capability:

# GR sends command:
capability hex 64 0078 peer 192.168.1.1
               │    │        └─ target peer
               │    └─ restart-time (120) in 12-bit format
               └─ capability code 64 (graceful restart, RFC 4724)

BGP stores registered capabilities and includes them in OPEN messages to peers.

GR Plugin Coordination

GR plugin coordinates with BGP and RIB via commands and events:

ze gr                         ze (hub)                      ze bgp
   │                             │                             │
   │◄── config (JSON subtree) ───│                             │
   │                             │                             │
   │── capability hex 64 ... ───►│─── (routes to BGP) ────────►│
   │   peer 192.168.1.1          │      (BGP stores for OPEN)  │
   │                             │                             │
   │── request subscribe bgp.peer.*►│                             │
   │                             │                             │
   │                             │◄── event bgp.peer.restart ──│
   │◄── event bgp.peer.restart ──│                             │
   │                             │                             │
   │── rib defer peer X ────────►│─────────────────────────────│──► ze rib
   │                             │                             │

Key points:

• GR uses capability hex <code> <value> peer <addr> format
• Hub routes capability commands to BGP
• GR subscribes to peer events (restart, up, down)
• GR coordinates with RIB for route deferral during restart

CLI Commands

See Command Ownership for the plugin self-containment pattern: how each component owns its CLI commands, YANG schema, and tests.

Command Routing

CLI commands are routed to plugins by prefix:

# Routed to ze bgp
ze bgp peer list
ze bgp peer upstream1 show
ze send bgp upstream1 update ...

# Routed to ze rib
ze rib show
ze rib replay upstream1

# Routed to hub (system commands)
ze system schema list
ze system process list
ze config reload

How CLI Works

┌─────────────────────────────────────────────────────────────────────┐
│ $ ze bgp peer list                                                  │
│                                                                     │
│  1. CLI connects to daemon via SSH (127.0.0.1:2222)                 │
│  2. CLI sends: bgp peer list                                        │
│  3. Daemon looks up "bgp" in handler map → ze bgp process           │
│  4. Daemon forwards command via stdin to ze bgp                     │
│  5. ze bgp executes, sends response via stdout                      │
│  6. Daemon returns response to CLI via SSH session                  │
│  7. CLI displays result                                             │
└─────────────────────────────────────────────────────────────────────┘

Same binary, two modes:

• ze start config.conf → starts as daemon, listens on SSH port
• ze bgp peer list → connects to running daemon as SSH client

SSH target configurable via env vars ze_ssh_host and ze_ssh_port

System Commands

Commands handled directly by the hub:

# List registered schemas
ze system schema list
ze-bgp: bgp, bgp.peer, bgp.peer-group
ze-rib: rib
ze-gr: bgp.peer.capability.graceful-restart

# Show a schema
ze system schema show ze-bgp

# List running processes
ze system process list
bgp: pid=1235 state=ready
rib: pid=1236 state=ready
gr:  pid=1237 state=ready

# Reload configuration
ze config reload

Event Flow

Plugins communicate via events routed through the hub.

Event Subscription

Plugins subscribe to events during startup:

# ze rib subscribes to BGP events
request subscribe bgp.event.*

Event Publishing

When something happens, plugins publish events:

# ze bgp publishes peer state change
event bgp.peer.up peer=192.0.2.1 asn=65002

Event Routing

Hub routes events to subscribers:

ze bgp                        ze (hub)                      ze rib
   │                             │                             │
   │ (peer establishes)          │                             │
   │── event bgp.peer.up ───────>│                             │
   │   peer=192.0.2.1            │                             │
   │                             │── event bgp.peer.up ───────>│
   │                             │   peer=192.0.2.1            │
   │                             │                             │
   │ (UPDATE received)           │                             │
   │── event bgp.update ────────>│                             │
   │   {...}                     │── event bgp.update ────────>│
   │                             │   {...}                     │

Startup Sequence

10-Step Protocol

1. Hub parses environment { }      → Set global settings (api-socket, log level, etc.)
2. Hub parses plugin { } block     → Build process list from ze-plugin-conf.yang
3. Hub forks each plugin           → ze bgp, ze rib, ze gr, third-party, ...
4. Each plugin: Stage 1            → Declare YANG module + handlers
5. Hub registers schemas           → Build handler routing table (SchemaRegistry)
6. Hub parses remaining config     → Full parse, validate against combined YANG
7. Hub converts to JSON            → Map-of-maps structure
8. Hub routes JSON to plugins      → Each plugin gets its subtree (Stage 2)
9. Plugins: Stage 3-4              → Capability declarations, registry sharing
10. Plugins: Stage 5               → Ready, start operating

After startup, GR and other plugins use commands to configure BGP:

• capability hex <code> <value> peer <addr> → inject capabilities
• Plugins subscribe to events for runtime coordination

5-Stage Protocol (Per Plugin)

Each plugin follows this protocol with the hub:

Priority: Determines verify/apply order. Lower = first. Example: BGP=100, RIB=200, GR=300.

Config Notification to Plugins

Pull Model (Hub Never Pushes)

Hub notifies plugins of config changes, plugins query for config data.

On commit (startup, SIGHUP, or ze config commit), hub sends config verify / config apply notifications. Plugins query hub for config.

Based on handler registration, plugins query for their JSON config:

On-Demand Query

Plugins query hub for specific config paths using text protocol:

# Query live (running) config:
#1 query config live path "bgp.peer[address=192.0.2.1]"
@1 done data '{"address": "192.0.2.1", "peer-as": 65002, "timers": {...}}'

# Query edit (candidate) config:
#2 query config edit path "bgp.peer[address=192.0.2.1]"
@2 done data '{"address": "192.0.2.1", "peer-as": 65003, "timers": {...}}'

Hub stores config as map-of-maps internally, provides JSON in data field.

Live/Edit Configuration (VyOS-style)

Hub maintains two configuration states:

Commit Workflow

1. User modifies edit config (via CLI or file)
2. User requests commit
3. For each plugin (by priority, lower first):
   a. Hub sends: config verify
   b. Plugin queries hub for live and edit config (its section)
   c. Plugin computes diff using shared library
   d. Plugin validates changes are acceptable
   e. Plugin responds: done or error
4. If all plugins verify ok:
   a. For each plugin (by priority):
      - Hub sends: config apply
      - Plugin applies changes
      - Plugin responds: done
   b. Edit becomes new live
5. If any verify fails:
   a. Hub aborts commit
   b. Edit unchanged, live unchanged

Priority examples: BGP=100 (first), RIB=200, GR=300 (last).

Plugin Diff Responsibility

Hub provides: Raw config states (live and edit) on request.

Plugin is responsible for:

1. Query the config sections it needs
2. Compute diff using shared library code
3. Validate and apply changes

Shared Diff Library

Plugins use shared library code (not reimplemented per plugin) to compute differences:

Location: internal/component/config/diff/

Usage (Go):
  // Send query, receive JSON in data field
  live := sendQuery("query config live path \"bgp.peer\"")
  edit := sendQuery("query config edit path \"bgp.peer\"")
  changes := diff.Compare(live, edit)
  // changes = []Change{{Action: "create", Path: "bgp.peer[addr=X]", Data: {...}}, ...}

This library is part of the Ze codebase, available to all plugins.

YANG Schema

See Command Ownership for how command YANG schemas use container merge to live in their owning component rather than a central verb package.

What YANG Provides

• Type validation (ranges, patterns, enums)
• Cross-reference validation (leafref)
• Schema-driven config routing

Schema Registration

During Stage 1, plugins declare their YANG schema:

declare schema yang ze-bgp
declare schema handler bgp
declare schema handler bgp.peer
declare done

Existing YANG Modules

Note: Plugin YANG schemas augment ze-bgp-conf to extend the configuration tree. Each plugin owns its YANG in a schema/ subdirectory.

YANG Augment Merging

Plugins can augment other plugins' YANG schemas. Hub merges all YANG modules into a single consistent view:

1. Each plugin declares its YANG module in Stage 1
2. Hub collects all modules
3. Hub merges augments into base modules
4. Hub validates combined schema is consistent

Conflict handling: If two plugins define conflicting augments (same path, different definitions), hub refuses to start. The plugins are incompatible.

Package Structure

Planned (aspirational). This is the target package structure, not the current layout. See docs/architecture/overview.md for the actual directory structure.

internal/
├── hub/                    # Hub/orchestrator (protocol-agnostic)
│   ├── hub.go              # Core hub
│   ├── process.go          # Fork and manage child processes
│   ├── router.go           # Route commands/events
│   └── config.go           # Parse env and plugin blocks
│
├── plugin/
│   ├── bgp/                # BGP plugin (moved from internal/bgp/)
│   │   ├── message/        # BGP wire format
│   │   ├── attribute/      # Path attributes
│   │   ├── nlri/           # NLRI types
│   │   ├── capability/     # BGP capabilities
│   │   ├── fsm/            # State machine
│   │   ├── rib/            # Peer-to-peer routing (moved from internal/rib/)
│   │   └── reactor/        # BGP-specific reactor (moved from internal/reactor/)
│   │
│   ├── rib/                # Adj-RIB tracking plugin
│   │   ├── rib.go
│   │   └── storage/
│   │
│   ├── gr/                 # Graceful Restart plugin
│   │
│   ├── server.go           # Plugin server (reused by hub)
│   ├── handler.go          # Command/event dispatch
│   ├── schema.go           # SchemaRegistry
│   └── subsystem.go        # 5-stage protocol
│
├── yang/                   # YANG loader and validator
│   ├── loader.go
│   └── validator.go
│
└── config/                 # Config parsing (shared)

Signals

Config Reload (SIGHUP)

1. Hub receives SIGHUP
2. Hub re-parses config file
3. Hub diffs current vs new config
4. Hub sends verify/apply for changes to affected plugins
5. Plugins apply changes (add/remove peers, etc.)

Benefits of This Architecture

Security Model

Privilege Dropping (ExaBGP Pattern)

Ze follows the standard Unix daemon privilege separation model:

1. Start as root (or with CAP_NET_BIND_SERVICE) to bind port 179
2. Bind the BGP listening socket
3. Drop privileges to the configured user/group
4. All subsequent work -- including plugin spawning -- runs as the unprivileged user

The target user/group is configured via environment variables:

When ze.user is not set, no privilege dropping occurs.

Implementation: internal/core/privilege/ -- calls setgid then setuid after reactor.Start() binds port 179.

Plugin TLS Transport

External plugins connect back to the engine via TLS. The engine binds TLS listeners (configured via plugin { hub { server <name> { ip ...; port ...; secret ...; } } }), forks child processes with ZE_PLUGIN_HUB_HOST/ZE_PLUGIN_HUB_PORT/ZE_PLUGIN_HUB_TOKEN/ZE_PLUGIN_CA_PEM env vars, and waits for authenticated connect-back. ZE_PLUGIN_CA_PEM carries the certificate authority root that issued the listener's certificate, and the SDK validates the chain against it and nothing else; the full contract is in Process Protocol. Each plugin uses a single bidirectional TLS connection with MuxConn for concurrent RPCs.

Plugin Process Isolation

Each external plugin runs in its own process group (Setpgid) for clean signal handling and inherits the daemon's (already-dropped) uid/gid. All plugins run as the same unprivileged user.

Example Session

# Start Ze
$ ze start /etc/ze/config.conf
[hub] Starting with config /etc/ze/config.conf
[hub] Forking ze bgp (pid 1235)
[hub] Forking ze rib (pid 1236)
[hub] Forking ze gr (pid 1237)
[hub] All plugins ready

# Check status
$ ze system process list
bgp: pid=1235 state=ready uptime=5m
rib: pid=1236 state=ready uptime=5m
gr:  pid=1237 state=ready uptime=5m

# List BGP peers
$ ze bgp peer list
192.0.2.1  AS65002  Established  5m
192.0.2.2  AS65003  Active       -

# Show routes
$ ze rib show
Prefix          Next-Hop      AS-Path        Peer
10.0.0.0/24     192.0.2.1     65002          192.0.2.1
10.0.1.0/24     192.0.2.1     65002 65004    192.0.2.1

# Reload config
$ ze config reload
[hub] Reloading configuration
[hub] Added peer 192.0.2.3
[hub] Reload complete

# Graceful shutdown
$ kill -TERM $(pgrep -f "ze start config.conf")
[hub] Received SIGTERM, shutting down
[hub] Notifying plugins...
[bgp] Sending NOTIFICATION to peers
[hub] All plugins stopped

Related Documents

• Core Design - Canonical architecture (describes in-process mode)
• Hub Architecture - Hub mode internal design details
• Process Protocol - 5-stage protocol specification
• YANG Config Design - Schema design
• Config Dispatch - mode selection by config content (completed). This was summary 189, retired on 2026-08-01 and not carried into Design History, whose header gives the git-recovery route

Last Updated: 2026-01-30