How to Build a DeFi Incident Response Runbook

An incident response runbook for a lending protocol is not a PR template. It is a short document that tells a tired person what to do when a risk alert, a war-room ping, or a messy Twitter thread arrives.

This article proposes a structure you can copy. It assumes Ethereum lending, a multisig, and a preference for human review. It does not assume BlockSentinel is in the loop, though we built our guard workflow to match this shape.

Why This Matters

During an incident, organizations regress to whoever shouts. Signers get pulled into explorer tabs. Someone pastes an unverified address. A parameter change is drafted because "we should do something".

A runbook is how you pre-decide the boring parts: who is in the room, what evidence is required, what is out of bounds, and how you close the incident even when you took no chain action.

Roles

Keep the list small.

Role	Job	Not their job
Incident lead	Clock, decisions, comms out	Deep reversing in the first 15 minutes
On-chain reviewer	Logs, txs, freshness	Tweeting
Risk reviewer	Thresholds, markets, user impact	Inventing calldata
Signer liaison	Only if a proposal is actually going to signers	Being the war room

If two of those are the same human, write that down. Do not pretend you have a SOC.

What Teams Should Watch in the First 15 Minutes

1. Is ingest/freshness healthy? If the monitor is stale, say so immediately.
2. What actually moved: borrows, liquidations, same-tx clusters, or something off-chain?
3. Which market and asset?
4. Is this already a known market event (listings, rate changes, paired-protocol stress)?
5. Do we have an evidence packet, or are we staring at a chart?

Do not open governance forums in minute one. Do not draft pause calldata in minute one unless your written policy already says that exact asset and action are allowlisted.

The scoring side of this is borrow pressure and liquidation stress.

Evidence Standard

Before anyone says "we should freeze":

• Protocol, chain, time window
• Score, confidence, freshness
• Top factors
• At least a few transaction hashes or a log range
• What the model did not see (oracles, traces, mempool)

If you cannot produce that, you are still in discovery. Discovery is allowed. Signing is not.

Escalation and Comms

Pre-write channels:

• Internal: one incident channel, not three.
• External: who is allowed to post, and that nobody else should.
• Signers: only after a proposal exists, with expiry visible.

A useful external sentence: "We are reviewing live lending-market stress with an evidence packet. We have not executed protocol changes." That is better than silence and better than theater.

Decision Options

Every incident should end in one of these, written down:

• Watch — thresholds not met or confidence too low.
• Escalate — humans stay on, no proposal yet.
• Propose — allowlisted advisory package, expiry set.
• Execute — only signers, only after review, never the monitor.
• Reject / expire — still recorded.

"Nothing happened" is not a close state unless you record Watch or Reject. Future you will not remember.

Tabletop: Hypothetical Cascade

T+0: liquidations cluster on a stablecoin reserve. Risk 7d 79, confidence 0.68, freshness 2 minutes. T+5: incident lead names on-chain reviewer. Twitter is ignored except as a pointer to tx hashes already in the packet. T+12: borrow pressure is also elevated. Same-tx count is 1. Policy allows freeze of that reserve, not a global pause. T+20: JSON proposal created, 120-minute expiry. Signer liaison notified, not the whole tokenholder chat. T+40: liquidations slow. Proposal expires unused. Close state: Propose then expire. Thresholds unchanged until a postmortem.

No heroics. That is the point.

Aftercare

Within 48 hours:

• Store the packet.
• List what confused people.
• Decide whether thresholds or allowlists change.
• Note ingest gaps.
• Thank the people who did not tweet.

The operator checklist that feeds this runbook is here. Safety constraints are on security.

How BlockSentinel Approaches This

We emit alerts with severity, factors, and links. We can emit advisory guard JSON. We do not execute. We do not want to be the system that "handles the incident" for you. We want the packet in the runbook to exist when you need it.

Next Step

If you want a pilot that produces those packets for an Ethereum lending market, request early access. Bring your roles table, even if it currently has two names on it.