Notification Mechanics

Notification delivery has five distinct decisions: event severity, state transition, routing, template resolution, and delivery outcome. Use the settings pages to inspect each one directly.

Severity describes how serious an observation is; event type describes what happened; status describes the alert lifecycle. The shared resolver maps healthy metrics to SUCCESS, warnings to WARNING, critical readings to CRITICAL, stale or unknown data to NOTICE, and collector failures to ERROR.

• Hysteresis uses separate clear thresholds so a boundary reading does not oscillate.
• Dwell requires a condition to hold for the configured duration or sample count.
• A telemetry gap is a data gap, not a recovery, and never silently resolves an incident.
• Flapping is collapsed into one event until the condition stabilizes.

Routing rules match server, team, device class, location, event type, and minimum severity. Rules run by priority and can stop or continue evaluation. Multiple matching gateways receive separately rendered deliveries.

Templates resolve per field through server, gateway, team, user, and built-in defaults. Every provider and event type has a built-in fallback, so templates are never required setup.

With NOTIFICATION_PIPELINE_V2=on, evaluation persists a delivery record before the outbox worker sends it. The default shadow mode evaluates and records decisions without changing live delivery.

• Retryable failures use bounded exponential backoff and honor Retry-After.
• Authentication, invalid-target, configuration, and payload failures do not retry.
• Exhausted attempts become DEAD_LETTER; every attempt retains redacted error detail and template source.
• Gateway health is derived as healthy, degraded, failing, disabled, or unconfigured.

Account events use preferences separate from infrastructure routing. Security and data categories cannot be disabled or assigned a per-category gateway; they use the account default destination. Team and lifecycle categories are optional.

Security, invitation, membership, server registration, server deletion, and agent-key rotation events are modeled. The remaining identity and platform call sites are being added incrementally.