Skip to content
Docs
Start free

Guides

Incidents in PagerDuty

Connect PagerDuty and the kill switch, over-settlements, held high-value payments and spent budgets open one incident each, resolved when they end.

#What opens an incident

TriggerOpensResolvesSeverity
Kill switchan agent frozen (agent_freeze) or a fleet (agent_freeze_batch)its unfreeze, or the batch liftedwarning for a pause, error for a security hold, critical for a fleet
Incidentan over-settlement, an over-volume, unmatched Ramp charges (agent_incident)(by a person in PagerDuty)critical for an over-settlement, otherwise error
Held requesta payment held for approval at or above the threshold (default 1,000.00)its approval, denial, expiry or cancellationwarning
Budgetthe gateway held or refused calls because a budget is spent(by a person)error when refused, warning when held

Drills never page. A freeze Immiscible makes because of an incident is left to that incident. Each incident has a dedup key built from the ledger (imm:<workspace>:freeze:<agent>, imm:<workspace>:budget:<scope>:<id>:<month>, and so on), and Immiscible remembers which are open, so a budget refusing a thousand calls is one incident, and a resolve is only sent for one Immiscible opened. Manage chooses the triggers and the threshold.

Delivery is off the request path, in order, with up to six attempts and backoff on a 429 or a 5xx. PagerDuty deduplicates on the key, so a retried delivery that did arrive changes nothing.

#Connect

  • Connect PagerDuty (when the operator has registered an app): sign in to PagerDuty, choose a service, and Immiscible adds an Events API v2 integration named Immiscible to it. It keeps only that integration’s key and lets the sign-in go.
  • Or paste a key: on a service, Integrations, add Events API v2, and paste the 32-character Integration Key. Choose US or EU.

Either way a test incident is opened and resolved before anything is saved.

#For the operator: register the PagerDuty app once (optional)

  1. In PagerDuty, Integrations, Developer Tools, App Registration, New App (Manager, Global Admin or Account Owner).
  2. Add OAuth 2.0, choose Classic User OAuth with the write scope (classic apps work on any account without PagerDuty’s review), and the redirect URL https://<your PUBLIC_URL host>/connect/pagerduty/callback.
  3. Set PAGERDUTY_CLIENT_ID and PAGERDUTY_CLIENT_SECRET. Without them the card offers the pasted key only.

Sources: Events API v2, rate limits, OAuth, register an app, REST schema: createServiceIntegration.