VM Appliance

Suitable for N100-class mini PCs, Proxmox VMs, or QEMU testing.

What's in the image

Ze owns network configuration in the appliance. The gokrazy default DHCP and NTP packages are excluded from the image; the shipped Ze seed template enables interface DHCP auto-discovery with set interface dhcp-auto true and leaves Ze NTP disabled (set environment ntp enabled false) until the operator enables it in Ze config.

The root filesystem is read-only (SquashFS). Persistent data lives on a separate ext4 partition mounted at /perm.

Prerequisites

Install once on the build machine (macOS):

brew install e2fsprogs    # ext4 filesystem tools
brew install qemu         # VM runtime (testing only)

For appliance ISO creation, install grub-mkstandalone (or grub2-mkstandalone) plus xorriso. ze appliance iso checks those tools before it stages an ISO.

The gokrazy build tool (gok) is vendored in the repo at vendor/github.com/gokrazy/ and built automatically by Make. No separate install needed.

First-time setup

After cloning the repo, download gokrazy system packages (Linux kernel, init, serial console) into the Go module cache. This is a one-time ~42MB download. The exact versions are pinned in gokrazy/ze/builddir/*/go.mod (tracked in git, verified by go.sum).

make ze-gokrazy-deps

After this, builds work offline.

L2TP Kernel Support

Ze's L2TP LNS path needs kernel PPPoL2TP support in the appliance runtime kernel: CONFIG_PPP, CONFIG_PPPOL2TP, CONFIG_L2TP, and CONFIG_L2TP_V3. The shared runtime proof kernel also keeps CONFIG_PPPOE built in for PPPoE evidence. The pinned upstream gokrazy kernel is not assumed to provide these options. Build the repo-local kernel before building an appliance intended to terminate L2TP subscribers:

make ze-kernel                                   # default runtime build: docker, amd64
make ze-kernel KERNEL_BUILDER=qemu               # force the shared QEMU backend
make ze-kernel KERNEL_ARCH=arm64                 # runtime arm64 kernel
make ze-kernel KERNEL_ARCH=arm64 KERNEL_BUILDER=qemu
make ze-gokrazy USER=admin PASS=secret

On Apple Silicon, use a native arm64 VM image to avoid x86_64 emulation while still building the kernel with the same L2TP/PPP options:

make ze-kernel KERNEL_ARCH=arm64                 # default builder is docker
make ze-kernel KERNEL_ARCH=arm64 KERNEL_BUILDER=qemu
make ze-gokrazy GOKRAZY_ARCH=arm64 USER=admin PASS=secret
make ze-gokrazy-run GOKRAZY_ARCH=arm64 GOKRAZY_QEMU_ACCEL=hvf

make ze-kernel delegates to gokrazy/kernel/Makefile, which calls the single shared driver tools/kernel-builder/run.py. The driver reads the kernel version from internal/appliance/kernel.version, selects the Docker backend by default (or the QEMU backend with KERNEL_BUILDER=qemu), resolves the tracked gokrazy/kernel/kernel.config + runtime.config fragments (plus the shared # ze-include: efi-console console fragment) and matching .require manifests, and emits vmlinuz, lib/modules/, and DTBs. make ze-kernel then assembles those into an out-of-tree kernel package (tmp/kernel/pkg, a copy of the pinned rtr7/kernel module with our artifacts overlaid) and points gok at it via a go.mod replace. The pinned module cache is never mutated in place and there is no backup to restore; make ze-kernel-clean drops the replace and removes tmp/kernel.

On a Linux runner with QEMU, xl2tpd, pppd, /dev/ppp, and PPPoL2TP kernel support, the deployment proof target builds an L2TP-enabled appliance image and drives a real LAC against it:

make ze-deployment-gokrazy-l2tp-ppp-test

The proof image is built from a temporary gokrazy instance config so the normal appliance config is left unchanged. It disables IPv6CP in that proof image because the current static L2TP pool is IPv4-only. Set ZE_GOKRAZY_SKIP_BUILD=1 to run against an existing tmp/gokrazy/ze.img that was already built with the L2TP proof template and proof runtime environment.

Build an image

First build (creates SSH credentials and a TLS certificate):

make ze-gokrazy USER=admin PASS=secret

Subsequent rebuilds reuse the existing database (same credentials, same TLS cert):

make ze-gokrazy

To use a database from a running instance or another machine:

make ze-gokrazy ZEFS=/path/to/database.zefs

To build with a different first-boot template without editing gokrazy/ze/ze.conf:

make ze-gokrazy USER=admin PASS=secret GOKRAZY_TEMPLATE=tmp/my-ze.conf

The legacy Make first build:

1. Builds bin/ze for the host
2. Runs ze init --seed with credentials and generates a self-signed TLS certificate. --seed skips baking the build host's discovered interfaces into the active config; otherwise that active config would hold the wrong host's NICs and shadow the seed template, leaving the appliance without web/L2TP. The appliance instead builds its active config at first boot from the template merged with its own on-device discovery.
3. Cross-compiles Ze for linux/GOKRAZY_ARCH and builds a 2GB disk image
4. Formats the persistent /perm partition
5. Injects database.zefs (credentials + TLS cert) into /perm/ze/

The database is kept at tmp/gokrazy/init/database.zefs between builds. Browsers that trust the certificate on first use will not prompt again after image rebuilds. Structured ze appliance build also writes a build manifest into /perm/ze/build.json; the legacy make ze-gokrazy flow does not.

The image lands at tmp/gokrazy/ze.img.

Test in QEMU

make ze-gokrazy-run

This boots the image with these legacy Make forwards:

Quit QEMU with Ctrl-A X.

The Gokrazy management UI shows process status, stdout/stderr ring buffers, and resource usage. In appliance mode it is exposed under Ze's authenticated web UI at /gokrazy/; the proxy reads Gokrazy's password from the same password-file locations Gokrazy uses when it needs to inject upstream Basic Auth.

Deploy to hardware

Write the image to a USB drive or internal disk on your N100 machine:

# Linux
sudo dd if=tmp/gokrazy/ze.img of=/dev/sdX bs=4M status=progress

# macOS
sudo dd if=tmp/gokrazy/ze.img of=/dev/rdiskN bs=4m

Or import into Proxmox:

qm importdisk <vmid> tmp/gokrazy/ze.img <storage>

The machine boots to a serial console (115200 baud). Ze starts automatically, gets a DHCP address, and loads its active configuration from /perm/ze/database.zefs (bootstrapped from the seed template on first boot). The serial console requires authentication with the local admin credentials before granting shell access. If the credentials database is missing or unreadable, access is granted without authentication for emergency recovery. When admin-enabled: false is set in the appliance config, the serial console denies the built-in admin (fail-closed) and prints "local admin login disabled".

Configuration

Seed config

The initial Ze config is stored as the seed template in gokrazy/ze/ze.conf. Legacy Make writes that file into file/template/ze.conf in ZeFS during make ze-gokrazy; structured ze appliance assemble uses the same default when no base or per-appliance overlay config is present. Because make ze-gokrazy runs ze init --seed, the seed DB has no file/active/ze.conf to shadow the template, so the template becomes the effective config on first boot (ze appliance assemble never wrote an active config, so it was already correct).

set environment log level info
set environment web enabled true
set environment web server default ip 0.0.0.0
set environment web server default port 8080
set environment ssh enabled true
set environment ssh server default ip 0.0.0.0
set environment ssh server default port 22
set environment ntp enabled false
set interface dhcp-auto true

To change the seed config, edit gokrazy/ze/ze.conf, pass GOKRAZY_TEMPLATE=/path/to/ze.conf to the legacy Make workflow, or use the structured workflow's config-base and per-appliance ze.conf files.

Runtime config

Once booted, use ze config edit over SSH to modify the running configuration. Changes are stored in /perm/ze/database.zefs and persist across reboots and image updates.

Environment variables

Ze's environment is set in gokrazy/ze/config.json under PackageConfig:

Updating

Gokrazy supports atomic A/B partition updates over the network:

bin/ze-setup appliance push <name>

This pushes the most recent image to the device. The system reboots into the new version. If the update fails mid-flight, the previous root partition is still intact.

For full image rebuilds (when you also want to update the kernel or partition layout), use ze appliance build <name> again and re-flash.

Architecture notes

Internal plugins only

Gokrazy has no shell and no PATH. Ze's external plugin mechanism (which uses /bin/sh -c to fork processes) does not work. All Ze plugins (bgp-rib, bgp-gr, bgp-adj-rib-in, etc.) are compiled into the ze binary as internal plugins and run as goroutines. This is the default and covers all standard BGP functionality.

Process supervision

Gokrazy's init restarts Ze if it exits with a non-zero status (except 125, which means "don't restart"). Ze handles SIGTERM for graceful shutdown. Logs (stdout/stderr) are captured in ring buffers visible through the gokrazy web UI.

Persistent storage

The /perm partition (ext4) survives image updates. Ze stores its database (database.zefs), TLS certificates, and config state there via the ze.config.dir=/perm/ze environment variable.

Repo layout

gokrazy/
  .gitignore              # excludes *.img
  ze/
    config.json           # gokrazy instance config (what to build, how to start)
    builddir/
      codeberg.org/thomas-mangin/ze/
        go.mod            # ze dependency pins + relative replace directive
        go.sum
      github.com/rtr7/kernel/
        go.mod, go.sum    # linux kernel version pin
      github.com/gokrazy/gokrazy/
        go.mod, go.sum    # gokrazy init system version pin
        cmd/dhcp/         # DHCP client
        cmd/ntp/          # NTP client
        cmd/heartbeat/    # watchdog heartbeat
        cmd/randomd/      # entropy seeder
cmd/ze-serial-shell/        # serial console login gate (replaces serial-busybox)
  main.go                   # gokrazy wrapper: symlink + DontStartOnBoot
  _gokrazy/                 # renamed busybox extrafiles per arch
cmd/ze-gok/
  main.go                   # gok wrapper (built by make bin/gok)

The gok build tool source is vendored in the main vendor/github.com/gokrazy/ directory. The builddir/ files are small text (go.mod + go.sum, ~27KB). System packages (kernel, init) live in the Go module cache after make ze-gokrazy-deps.

ze-setup binary

Appliance build commands (ze appliance) and PXE provisioning (ze install remote) are part of the ze-setup binary. This keeps build-host tooling out of the on-device ze binary.

Build ze-setup from the repo root:

make bin/ze-setup          # produces bin/ze-setup

Building and installing an appliance (end to end)

From a JSON config (recommended)

Write an appliance config file (arch, kernel profile, credentials, networking):

{
    "credentials": { "username": "exa", "admin-enabled": true },
    "ssh":   { "host": "0.0.0.0", "port": "2222" },
    "web":   { "enabled": true, "host": "0.0.0.0", "port": "8080" },
    "tls":   { "cert-name": "router.local", "validity-years": 10 },
    "identity": { "hostname": "ze-prod" },
    "device": { "address": "10.12.104.10", "update-port": 443 },
    "image": { "arch": "amd64", "size-bytes": 2147483648, "kernel-profile": "hardware-kms" }
}

Reserving hugepages for VPP

When the appliance runs VPP, reserve hugepages at boot by adding image.hugepages to the config. ze appliance build bakes default_hugepagesz/hugepagesz/hugepages into the boot cmdline (via a derived gokrazy instance config; the checked-in gokrazy/ze/config.json is never modified). Declare image.memory so the build rejects a reservation over 50% of target RAM and ze appliance run sizes QEMU's -m to match. Sizes are byte-size strings (10b, 512mb, 1gb, 1tb; case-insensitive, 1024-based):

"image": {
    "arch": "amd64",
    "size-bytes": 2147483648,
    "memory": "8gb",
    "hugepages": { "size": "1gb", "page-size": "2mb" }
}

hugepages.size is the total reservation and page-size is 2mb or 1gb; the page count is size / page-size (so size must be a whole multiple of page-size). The reservation is bounded to 512 GiB and, when memory is set, to 50% of it. 1gb pages need CPU pdpe1gb support and CONFIG_HUGETLBFS in the kernel profile (both surfaced by ze doctor).

Build the full ISO in one command:

make ze-iso CONFIG=prod.json SSH_PASSWORD='choose-a-strong-one'

This runs the entire pipeline: init, kernel build, initrd, disk image, and ISO. The appliance name is derived from the config filename (prod.json creates appliance prod). Subsequent builds with the same config reinitialize from scratch.

Rebuild after code changes (appliance config unchanged):

make ze-iso-build NAME=prod

Set up PXE boot (optional, after the ISO build):

make ze-pxe NAME=prod

See make help-deploy for all variables (APPLIANCE_BUILDER, PXE_DIR, etc.).

Manual steps

The Makefile targets call these ze-setup commands under the hood. Run them individually when you need finer control:

# 1. Build bin/ze-setup
make bin/ze-setup

# 2. Create an appliance with its config and secrets
env ze.appliance.ssh.password='choose-a-strong-one' \
  bin/ze-setup appliance init --config prod.json prod

# 3. Optional readiness check for ISO prerequisites
bin/ze-setup appliance iso --check

# 4. Prepare ISO prerequisites
bin/ze-setup appliance kernel prod                # download or build the installer kernel
bin/ze-setup appliance initrd                    # download or build the installer initrd

# 5. Build the full disk image (gokrazy + ZeFS credentials)
bin/ze-setup appliance build prod

# 6. Build the bootable installer ISO
bin/ze-setup appliance iso prod

# 7. Install: either boot the ISO on the target machine, or PXE provision
#    Option A: copy the ISO to a USB stick or mount in a VM
#    Option B: serve over the network with PXE
bin/ze-setup install remote \
  --interface eth0 \
  --network 10.0.0.0/24 \
  --image ~/.config/ze/appliances/prod/ze-*.img \
  --ssh-username admin \
  --ssh-password 'choose-a-strong-one'

The kernel and initrd commands first check the XDG cache. If ze.appliance.kernel.url or ze.appliance.initrd.url is set, the matching command then tries that configured prebuilt-artifact URL; otherwise it builds locally. Kernel local builds use the shared Docker-or-QEMU builder selection, and initrd local builds compile and pack cmd/ze-installer. Once cached, subsequent runs are instant. See "ISO prerequisites" below for details.

ze appliance (structured workflow)

The ze appliance command provides structured appliance management. Each appliance has its own directory with a JSON config, secrets that are optionally encrypted at rest, and a TLS certificate.

Quick start

bin/ze-setup appliance init lab                  # interactive wizard
bin/ze-setup appliance build lab                 # full image (assemble + gok + ext4)
bin/ze-setup appliance kernel lab                 # download or build installer kernel
bin/ze-setup appliance initrd                    # download or build installer initrd
bin/ze-setup appliance iso lab                   # bootable installer ISO from latest image
bin/ze-setup appliance list                      # show all appliances
bin/ze-setup appliance show lab                  # config summary + cert expiry

Appliance directory

By default, appliances live in ~/.config/ze/appliances/. Override with --dir or ZE_APPLIANCE_DIR.

~/.config/ze/appliances/
  _shared/
    ze.conf                    # optional base config for all appliances
  lab/
    appliance.json             # config (no credentials)
    ze.conf                    # per-device config overrides
    secrets/                   # 0700 permissions
      .encrypted               # marker (present = secrets encrypted)
      tls/
        cert.pem               # public certificate (plaintext)
        key.pem                # private key (encrypted if passphrase set)
      password.hash            # bcrypt hash (encrypted if passphrase set)
      update.token             # gokrazy OTA token (encrypted if passphrase set)
      authorized_keys          # SSH public keys (plaintext)

Encryption

Secrets are encrypted at rest with Argon2id + XChaCha20-Poly1305 when an encryption passphrase is set during ze appliance init. The passphrase is never stored on disk. For fleet operations, ze appliance unlock starts a passphrase agent (like ssh-agent) that holds the derived key in memory.

bin/ze-setup appliance unlock                    # start agent (prompts for passphrase)
bin/ze-setup appliance unlock --duration 15m     # auto-expire after 15 minutes
bin/ze-setup appliance unlock --stop             # stop agent

Day-2 operations

bin/ze-setup appliance passwd lab                # rotate SSH password
bin/ze-setup appliance replace-cert lab          # regenerate self-signed cert
bin/ze-setup appliance replace-cert lab --cert ca.pem --key ca.key   # use CA-signed cert
bin/ze-setup appliance rekey lab                 # change encryption passphrase
bin/ze-setup appliance clone lab lab2            # copy config (not secrets)

Config layering

Set config-base in appliance.json to share a base config across appliances:

{
  "config-base": "../_shared/ze.conf"
}

The base config is read first, then per-appliance ze.conf is appended. Later set commands override earlier ones; delete commands remove settings from the base.

Commands reference

ISO prerequisites

The ISO build requires an installer kernel, an initrd, grub-mkstandalone, and xorriso. Use ze appliance iso --check to see what is ready and what is missing. The kernel and initrd commands handle downloading or building these artifacts automatically:

bin/ze-setup appliance iso --check               # report readiness
bin/ze-setup appliance kernel lab                 # reads arch/profile from appliance config
bin/ze-setup appliance kernel --profile hardware lab   # explicit hardware profile
bin/ze-setup appliance kernel --builder qemu --arch arm64 lab
bin/ze-setup appliance kernel --target runtime    # build the verified runtime kernel tree
bin/ze-setup appliance initrd                    # download or build initrd
bin/ze-setup appliance iso lab                   # build ISO

ze appliance kernel defaults to --target installer (the monolithic PXE Image); --target runtime builds the gokrazy runtime kernel tree (modules + vmlinuz) from gokrazy/kernel/ with the runtime requirement floor enforced. The command reports the target it built (kernel ready: ... (target=installer, profile=qemu, version=7.1.1)). The installer target tries cache first, then a configured prebuilt-artifact URL if ze.appliance.kernel.url is set, then local build. Every local build runs through the shared driver tools/kernel-builder/run.py, which selects Docker when available and falls back to QEMU; use --builder docker or --builder qemu to force one path. Resolved installer artifacts are cached under $XDG_CACHE_HOME/ze/ (default ~/.cache/ze/) and copied to build/kernel/ when appropriate.

ze appliance initrd uses the same cache, optional configured URL, then local build pattern for the initrd artifact. The download URL has no built-in release server default; set ze.appliance.kernel.url or ze.appliance.initrd.url to use prebuilt artifacts.

ze doctor includes checks for kernel, initrd, grub, xorriso, and e2fsprogs availability, reporting warnings with actionable hints when prerequisites are missing.

ISO installer media

Create an installer ISO from an image already produced by ze appliance build. By default the command selects the latest ze-*.img in the appliance directory, verifies its .sha256 sidecar, and writes ze-*.iso next to the image. Use --image to select a specific image filename and --output to write the ISO elsewhere. The output path must not overwrite the selected .img, and the image filename must stay within [A-Za-z0-9._-] so the initrd can pass it on the kernel command line. By default, ze appliance iso resolves a matching installer kernel from the cache, or from build/kernel/Image only when its variant metadata matches the appliance arch/profile/version. ze appliance kernel and make -C tools/installer-kernel both delegate to tools/kernel-builder/; pass --kernel to keep multiple kernels side by side.

bin/ze-setup appliance build lab
bin/ze-setup appliance kernel --builder docker --profile hardware lab
bin/ze-setup appliance iso lab
bin/ze-setup appliance iso --image ze-20260601-120000.img lab
bin/ze-setup appliance iso --output /path/to/lab.iso lab
bin/ze-setup appliance iso --kernel build/kernel/Image lab

The ISO is an installer envelope around the existing raw gokrazy image. The image is gzip-compressed inside the ISO to reduce media size (a 2 GiB image with ~100 MiB of content compresses to roughly 100 MiB). The installer initrd decompresses the image during installation. The ISO does not rebuild the appliance, regenerate credentials, fetch a separate ZeFS database, or mutate /perm after writing the disk image. The installed disk receives the selected image bytes, including the /perm/ze/database.zefs and /perm/ze/build.json manifest that build already injected.

The ISO boot path accepts an optional explicit target disk. If no target is set, the installer writes only when exactly one non-removable candidate disk remains after excluding the ISO source media. The initrd also matches the booted ISO by a builder-generated ze.media-id token before it trusts a mounted installer volume, so identical image filenames on multiple attached installer media do not confuse the source selection. With multiple fixed disks, pass a whole disk path such as /dev/vda at ISO creation time:

bin/ze-setup appliance iso --target /dev/vda lab

After the installer writes the disk in ISO mode, it powers off instead of rebooting. Remove the installer media, then power the target back on so the firmware boots from the written disk.

The ISO contains the full provisioned appliance image, including the embedded ZeFS database. Handle the ISO with the same care as the .img file.

USB write method: the ISO can be written with dd, Etcher, or Rufus in DD mode. Ventoy is also supported when the installer kernel includes loop device and FAT/exFAT filesystem support (the hardware kernel profile has this). The initrd detects the ISO file on the Ventoy data partition, loop-mounts it, and proceeds with the installation. When using the qemu kernel profile, Ventoy is not supported.

Remote operations (push, config-push)

Push a built image to a running gokrazy device via its HTTPS update endpoint:

bin/ze-setup appliance push lab
bin/ze-setup appliance push --image ze-20260427-143022.img lab   # rollback to older image
bin/ze-setup appliance push --all                                # all devices with address
bin/ze-setup appliance push --all --parallel 4                   # 4 concurrent uploads

Push uses the update token (from secrets/update.token) for HTTP basic auth, and verifies the device TLS certificate against the stored cert.pem. No system CA pool is consulted.

When --image is set, the file name must resolve to a regular file inside the appliance directory. Path traversal and symlinks escaping that directory are rejected after the local update token is read, but before TLS setup or any network request starts.

Preview the effective configuration (base + overlay merged) without building:

bin/ze-setup appliance config lab --merged

Push a config change to a running device without rebuilding the image:

bin/ze-setup appliance config-push lab
bin/ze-setup appliance config-push --dry-run lab    # preview only, no SSH connection
bin/ze-setup appliance config-push --all            # all addressed devices
bin/ze-setup appliance config-push --all --parallel 4

Config-push uses SSH (operator's key via ssh-agent) to upload the merged config to the device, which validates and applies it. No secrets are transmitted over SSH.

Device-side config behavior

At boot, unmanaged devices resolve the active config in ZeFS. If no active config exists, Ze bootstraps one from the seed template or interface discovery. If /perm/ze/config-pushed.conf exists and parses as Ze config, Ze writes it over the active config. If the pushed config fails validation, Ze deletes that pushed file and continues with the existing active config.

After loading the effective config, the device writes its SHA-256 hash to /perm/ze/config-active-hash for fleet drift detection.

Last-known-good hash: at build time, ze appliance build writes the SHA-256 of the assembled seed config to meta/config/last-known-good in ZeFS. This serves as the build-time integrity baseline.

Config push and health monitor: config-push connects over SSH, stages the merged config, validates it, and applies it on the device. The source-backed health monitor is armed when a pushed config file is consumed at boot; it watches BGP peer close events for 30 seconds. If a peer closes during that window, the device reverts to the previous config, or to the seed config if no previous config was saved. If the window completes, the active config hash is written to /perm/ze/last-known-good-pushed.

Batch init

Initialize multiple appliances from a JSON manifest:

bin/ze-setup appliance init --batch manifest.json

Manifest format (array of entries):

[
  {"name": "edge-01", "hostname": "edge-01.lab", "password": "secret1", "device.address": "10.0.0.1"},
  {"name": "edge-02", "hostname": "edge-02.lab", "password": "generate"}
]

Use "password": "generate" for per-device random passwords (printed to stdout once, never stored in plaintext). When an encryption passphrase is set, each encrypted secret write receives a fresh random salt and nonce.

Disaster recovery (export/import)

Export creates an encrypted archive of an appliance directory for offsite backup or bastion migration. Archives include config, secrets, and build metadata, but exclude images and ZeFS databases (both are rebuildable).

Export a single appliance:

bin/ze-setup appliance export lab
# creates lab.ze.enc in the current directory

Export all appliances:

bin/ze-setup appliance export --all
# creates appliances-YYYYMMDD-HHMMSS.ze.enc

Import restores from an archive:

bin/ze-setup appliance import lab.ze.enc

Import to a different bastion (migration):

bin/ze-setup appliance import lab.ze.enc --dir /path/to/new/bastion

Archives are always encrypted using the same Argon2id + XChaCha20-Poly1305 scheme as secrets at rest. The archive passphrase can differ when provided separately from the secrets passphrase. import --force overwrites files present in the archive, but it does not delete extra files already present in existing appliance directories.