An alarm fires at 3am. Everything the responder needs already exists — the runbook, the review of the last time this happened, who owns the service, what it depends on — scattered across systems they now have to search one at a time, while the graphs get worse.
The retrieval is the easy part. What makes an incident copilot safe to point at a production service is everything that refuses: a gate that turns away the wrong person, an expiry that escalates instead of quietly approving itself, an action registry the planner cannot talk its way out of, and an audit entry for every attempt including the ones that were refused.
Scaffold the project
npx -y tiged@2.12.8 --mode=git gleanwork/glean-cookbook/recipes/incident-copilot incident-copilotInstall dependencies
cd incident-copilot && npm installWatch the governance hold, with no credentials
Replays recorded responses and asserts the parts that matter: the gate refuses the wrong actor, expiry escalates without executing, an unregistered action is refused, a mutating action with no supported cause is downgraded, and every attempt is audited.
cd incident-copilot && npm run verify:fixtureSet credentials
Only for a live run. Use npm run login for direct Search + Chat, or npm run login:agent when the user selected an existing Glean agent. Set WATCHED_SERVICES to the service already supplied and GLEAN_AGENT_ID only for the agent path.
cd incident-copilot && npm run login -- --email "<work-email>"Run it
cd incident-copilot && npm startVerify
Fire the sample alarm and check three things: the probable cause cites a past incident rather than a runbook, approving as someone who is not on call returns 403, and forcing expiry escalates without executing anything.
Only a matching past incident may support a causal claim. Runbooks support procedures, not causes.
The demo restricts who may approve but trusts the asserted actor. Production deployments must authenticate the actor and define which credential executes actions.
- Persist incidents and the audit log. An audit log you can lose by restarting a process is not an audit log.
- Add the rest of the dashboard: response-time rollups, an expiring-soon lane, and an end-of-shift handoff summary.
- Replace the simulated actions with real governed tools, keeping the registry boundary and the approval log exactly where they are.
- Write the postmortem draft back into your knowledge base so the next incident retrieves it as a precedent — the loop that makes the evidence rules get better over time.
- Expose the copilot itself over A2A so another agent can request triage, with the same gate in front of every action.
Triage a real alert from one of your services
The copilot acknowledges in the channel, resolves the service to its on-call engineer and owner from your own service catalog, fans out retrieval, and posts a triage card. A past incident is cited as the precedent that supports whatever it proposes — the proposal has to point at evidence, not at a hunch.
Triage an alert with no matching past incident
The highest-scoring retrieved document will often be a runbook, so a relevance-ranked copilot would confidently blame whatever that runbook is about. With no matching signature and nothing in flight, this copilot asserts no cause. Ranking is not evidence.
Approve the proposed action as someone who is not on call
Refused with 403 and audited against that actor. The allowed set is the on-call engineer and service owner read from your service catalog, not a config file. The incident stays awaiting approval. The recipe implements the authorization itself — nothing upstream does it for you.
Let the approval window expire
The proposal escalates to the escalation target named in your service catalog and is NOT executed. Auto-approving on timeout would invert the point of a gate. The escalation is audited and posted to the channel.
Have the Glean agent propose an action that is not registered
Refused at proposal time and audited, with no approval card offered. An agent that can describe arbitrary actions into existence is an agent with production access, whatever its prompt says.