Behavioral Anomaly Detection

Pre-Alpha. This page describes behavior that may change.

Ze has a behavioral anomaly detector: a report-only subsystem that builds a per-source behavioral baseline, scores how far each source deviates from its own history and from its peers, and correlates weak signals into a single incident. It is a separate domain from the volumetric DDoS subsystem, which reacts to raw packet and bandwidth rates. DDoS asks whether an interface is being flooded. Anomaly detection asks whether a source is behaving unlike itself and unlike its neighbours.

For the volumetric subsystem, see DDoS detection and mitigation.

Traffic-feature facts layer

Both detectors read from a neutral facts layer. trafficfeature is a second, independent consumer of the observation feed that derives domain-neutral per-source feature signals. These are measurable facts rather than verdicts. Detection plugins apply judgement on top, and this layer never decides that traffic is an attack.

The service is lazy and consumer-refcounted, so it costs nothing until a consumer attaches. It aggregates on a one-second tick and bounds memory with per-source cardinality and idle-eviction caps. Operators can read it directly:

show traffic feature
show traffic feature 203.0.113.7   // one source

The output lists the top source IPs by activity with their feature vector and a degraded flag when no data has arrived yet.

How detection scores a source

Each tick, the detector folds one traffic-feature snapshot into per-source, per-feature baselines and scores every source in two ways:

• Self-deviation. A z-score of the current value above the source's own running mean, in units of standard deviation, from an EWMA baseline. Below-baseline values score zero because only excess is anomalous. Each feature has a standard-deviation floor so a near-constant feature does not produce a divide-by-tiny-variance spike.
• Cohort rarity. How far the value sits above the source's cohort, other sources in the same source-prefix bucket. It is computed leave-one-out so an outlier is never compared against a baseline it dominates. Rarity is scored only once the cohort has at least min-cohort-size other members.

A continuous feature fires at the larger of its self-deviation and cohort-rarity score when that reaches deviation-threshold. The binary features, new-peer and rare-port, each contribute exactly one unit of evidence, enough to satisfy the correlation gate without dominating a continuous signal.

Correlation into one incident

A source is anomalous for a tick only when at least min-features-to-correlate features fire together. The fired z-scores are fused into one incident score: the strongest at full weight plus a discounted sum of the rest through corroboration-weight. The score is capped so a benign multi-feature move cannot saturate into a critical incident. The incident carries a graded severity derived from the score.

Confirm and clear lifecycle

An incident is confirmed only after confirm-duration consecutive anomalous ticks and cleared only after clear-consecutive consecutive normal ticks. This drives an AnomalyDetected, AnomalyOngoing, and AnomalyCleared event stream plus a bounded recent-incident ring.

Poisoning and false-positive guards

Freeze-learn. The per-source baseline is updated with the current tick only when the source is not anomalous, or while it is still warming up. A sustained attack cannot drift the source's baseline upward until it looks normal again.

No first-sight false positive. A newly seen source has no baseline to deviate from. Self-deviation is suppressed until the baseline has accumulated a short warmup of samples. Early values seed the baseline before the detector can flag the source.

Viewing anomaly incidents

show anomaly detect

The command reports whether the detector is enabled and lists the recent-incident ring. Each incident includes the entity, cohort, fused score and severity, timestamp, and fired features with their z-scores.

Shadow-first responder

The anomaly-shape responder can act on confirmed incidents. It defaults to shadow mode, where it logs the action it would take and installs nothing. Set it explicitly to armed before it can alter traffic.

anomaly {
    shape {
        mode shadow;        // default; armed installs live per-source rules
        action limit;       // rate-limit (default) or drop
        limit-rate 1000;
        limit-unit second;
        auto-revert-ttl 300;
        blast-radius-cap 16;
        allowlist [ 192.0.2.0/24 ];
    }
}

When armed, it installs a per-source firewall rule, a rate limit by default or a drop, for the anomalous source. Its safety properties are:

• Per-entity arming. Each source is armed independently as a whole owner set. Re-arming an already armed source refreshes its timer without reinstalling or double-counting.
• Timed auto-revert. Every armed rule carries an auto-revert-ttl. The rule is withdrawn when the TTL elapses regardless of any clear event, and an ongoing incident extends the deadline. A cleared incident withdraws the rule early.
• Blast-radius cap. At most blast-radius-cap sources are armed at once. Beyond the cap, arming is refused and logged.
• Kill switch. Engaging the kill switch reverts every armed rule and forces the responder back to shadow.
• Allowlist. Sources in the allowlist are never armed.

Inspect the responder state with:

show anomaly shape

The output reports the mode, configured action, kill-switch state, armed count, and armed sources.

Configuration reference

anomaly detect

anomaly shape

See also

• DDoS detection and mitigation covers the volumetric subsystem that shares the traffic-feature facts layer.
• Firewall covers the nftables backend used by the responder.
• Monitoring covers Ze health, metrics, and operational visibility.