Graceful Restart

Configuration

plugin {
    internal gr {
        use bgp-gr
    }
    internal rib {
        use bgp-rib
    }
}

bgp {
    peer upstream1 {
        remote { ip 10.0.0.1; as 65001; }
        ...
        capability {
            graceful-restart {
                restart-time 120;
            }
        }

        attach process gr {
            receive [ open-received state eor ]
            send [ update ]
        }
        attach process rib {
            receive [ update state refresh ]
            send [ update ]
        }
    }
}

Config Reference

How It Works

What Ze Advertises

The code-64 capability Ze sends carries the Restart Time, then one <AFI, SAFI, Flags> tuple for each address family the peer carries. A peer configured for ipv4/unicast and ipv6/unicast with the default restart time receives 0078 00010100 00020100. The family list is the peer's own family configuration, so the code-64 and the code-71 (LLGR) capabilities list the same families. RFC 9494 Section 4.2 reads a family that code 64 omits as a Restart Time of zero.

The graceful-restart family container narrows that list, and it governs code 64 alone:

capability {
    graceful-restart {
        family {
            name ipv6/unicast
        }
    }
}

That peer receives 0078 00020100, whatever else its family configuration carries. Writing the container with no name in it makes Ze name no address family at all, which RFC 4724 Section 3 reads as a speaker that runs the Receiving Speaker procedures and preserves nothing of its own. Writing no container is the default, and the default names every family the peer carries.

A name the peer's own family list does not carry is refused, because RFC 4724 Section 3 scopes a tuple to routes "advertised with the same AFI and SAFI" and the session carries none of them. Ze names the family in the error. The refusal runs at config commit, so the commit fails and the running router is untouched; a configuration that reaches startup carrying one stops the daemon rather than leaving it running without Graceful Restart.

The families the peer carries include the ones it inherits from its group, and exclude any entry written mode disable. Both are what the session really negotiates.

Narrowing code 64 is how an operator asks for LLGR with no conventional GR phase. RFC 9494 Section 4.1: "the conventional GR phase can be skipped by omitting all AFIs/SAFIs from the GR Capability, advertising a Restart Time of zero, or both". So the container never touches the code-71 list.

The Two Bits Ze Sets at Connection Time

The payload above is built when the configuration is loaded, so it carries what the operator configured and nothing about a restart. Two bits say what happened to Ze, and both are written on the way to the OPEN.

The Restart State bit says Ze has restarted. ze signal restart writes the restart marker, and the reactor sets the bit while that marker is live. Outside the window a new connection gets 0, which is a cold start.

The Forwarding State bit of an address family says the routes of that family were still being forwarded while Ze was down. Ze sets it inside the same window, and only when the forwarding plane kept its routes. The kernel FIB does keep them by default: fib { kernel { } } leaves every route it installed in place as Ze stops, marks them on the way back up and removes only the ones that do not return (flush-on-stop, sweep-delay). Writing flush-on-stop true, or configuring no FIB at all, means nothing was preserved and the bit stays 0.

The bit decides what the peer does the moment Ze comes back. RFC 4724 Section 4.2: if it "is not set in the newly received Graceful Restart Capability ... then the Receiving Speaker MUST immediately remove all the stale routes from the peer that it is retaining for that address family". So a peer holds Ze's routes while the session is down either way, and a clear bit makes it drop them at re-establishment rather than waiting for Ze to re-advertise.

Normal Session

1. Peer session establishes, GR capability negotiated in OPEN
2. Routes received and installed in RIB normally

Peer Restarts

1. Peer goes down -- GR plugin sends retain-routes to RIB
2. RIB marks routes as stale -- routes kept in forwarding but flagged
3. Restart timer starts -- countdown from restart-time seconds
4. Peer reconnects -- new session established, fresh routes received
5. Fresh routes replace stale -- each new route implicitly clears its stale flag
6. End-of-RIB received -- GR plugin sends purge-stale to RIB
7. Remaining stale routes removed -- any route not refreshed is withdrawn

Restart Timer Expiry

If the peer does not reconnect within restart-time seconds, all stale routes are purged. A safety margin of 5 seconds is added to account for processing delays.

Fail-Safe

If the GR plugin crashes or fails to issue purge-stale, the RIB automatically expires stale routes after restart-time + 5s.

Plugin Bindings

The GR plugin requires:

• receive [ open-received state eor ] -- needs the peer's OPEN, its up/down events, and End-of-RIB markers
• The RIB plugin must also be loaded with receive [ update state refresh ] and send [ update ]

The GR plugin depends on bgp-rib (declared in its registration). The engine ensures bgp-rib starts first.

Keep the bgp-gr engine in the daemon process

Load the plugin with internal <name> { use bgp-gr; }, as every example on this page does. run "ze plugin bgp-gr" starts the plugin engine in a child process, and that arrangement changes what the daemon sends to every peer.

The RFC 9494 egress filter runs in the daemon. The peer capability state it reads is written by the plugin engine, so with run that state stays empty for the life of the daemon process. The filter then treats every neighbor as a neighbor from which the LLGR Capability was never received: it withdraws stale routes from external peers (Section 4.3) and attaches NO_EXPORT with LOCAL_PREF 0 to the routes it sends to internal peers (Section 4.6). Peers that negotiated LLGR with the child engine get that same treatment.

ze doctor reports the arrangement as doctor-bgp-gr-out-of-process. Run ze explain doctor-bgp-gr-out-of-process for the full text.

CLI

 $ ze cli -c "show bgp rib received"
# Shows routes with stale flag when applicable

Long-Lived Graceful Restart (RFC 9494)

LLGR extends standard GR with a second, much longer stale period. When the GR restart-time expires without the peer reconnecting, instead of purging all stale routes, LLGR keeps them for up to ~194 days (per-family configurable) with reduced priority.

Configuration

Add long-lived-stale-time under the graceful-restart block:

capability {
    graceful-restart {
        restart-time 120;
        long-lived-stale-time 3600;    # LLGR period in seconds (0-16777215)
    }
}

LLGR is only active when both peers negotiate it. Ze advertises LLGR capability (code 71) in OPEN when long-lived-stale-time is configured. LLGR requires GR capability (code 64) to also be present -- LLGR without GR is ignored per RFC 9494.

How It Works

1. GR period expires -- peer has not reconnected within restart-time seconds
2. LLGR begins -- for each family with long-lived-stale-time > 0:
   • Routes carrying the NO_LLGR community (0xFFFF0007) are deleted
   • LLGR_STALE community (0xFFFF0006) is attached to remaining stale routes
   • Routes are marked as stale level 2 (deprioritized in best-path selection)
   • Per-family LLST timer starts
3. During LLGR -- LLGR-stale routes lose to any non-stale route in best-path selection. Between two LLGR-stale routes, normal tiebreaking applies.
4. LLST timer expires -- stale routes for that family are purged
5. Peer reconnects during LLGR -- standard RFC 4724 procedures apply; families with F-bit=0 or missing from the new OPEN are purged

Stale Levels

Ze uses a graduated stale level system for route prioritization:

Readvertising a Stale Route

RFC 9494 Section 4.3 governs a stale route that ze forwards on to a neighbor from which the LLGR Capability was not received. The stale level on the ROUTE decides, not the restart state, so a route staled by request bgp rib mark-stale takes the same treatment as one staled by a peer restart.

The filter fails closed. When it cannot read the peer capability state, it answers "LLGR was not received" for every destination. It raises one warning per process and then withdraws or depreferences. An unread state used to accept, which put a long-lived stale route into a neighbor that never agreed to hold one.

The one arrangement that leaves the state permanently unread is run "ze plugin bgp-gr". See Keep the bgp-gr engine in the daemon process.

Special Case: Skip GR

If restart-time is 0 but long-lived-stale-time is nonzero, the GR period is skipped entirely. On session drop, LLGR begins immediately.

Well-Known Communities

CLI

Decode LLGR capability from hex:

$ ze plugin bgp-gr --capa 00010180000e10

Shows per-family LLST values and F-bit flags.

Without Graceful Restart

When GR is not configured or the peer does not advertise the GR capability, routes are withdrawn immediately on session down. No stale state, no restart timer.