Notification Reliability
Queue behavior, retry decisions, and practical recovery steps for failed delivery.
Last updated
2026-08-28 · Audience: Operators + SRE
Delivery lifecycle
PENDING -> RETRYING -> SENT
-> DEAD_LETTER
SUPPRESSED = recorded, intentionally not sentEvaluation records the decision first. The outbox worker claims due deliveries, sends them with a bounded timeout, and records the attempt. A claim older than five minutes can be reclaimed after a worker crash.
Retries and dead letters
Timeouts, network failures, provider 5xx responses, and rate limits retry with bounded exponential backoff and jitter. Authentication failures, invalid targets, invalid configuration, and rejected payloads do not retry. A provider Retry-After value takes precedence when it is safe to honor.
After the attempt limit is exhausted, the delivery becomes DEAD_LETTER. Fix the gateway, verify it, then inspect the delivery log before retrying through the normal test or operational workflow.
Fairness and safety limits
- Each drain processes at most 50 entries.
- Each provider has at most four concurrent sends.
- Each account contributes at most ten entries per batch.
- Idempotency is stable per history, gateway, and attempt.
These limits keep one alert storm from starving other tenants or making a slow provider block the whole queue.