Incident Response

The First Hour of an Incident: What Lean Security Teams Get Wrong

Ryan Zahrai 7 min read
A dark SOC environment with time indicators suggesting an active incident being managed

The alert fires at 3:14am. Your on-call engineer wakes up, opens PagerDuty. What happens in the next twenty minutes determines whether the incident gets contained in two hours or eight.

We have watched this play out more times than we can count, and the pattern is almost always the same. Not because the engineer is slow, or because the team is under-resourced, or even because the runbook is bad. The first twenty minutes are expensive because they are almost entirely spent rediscovering context that existed before the alert fired.

The context-gathering loop

Here is what that first stretch actually looks like, in rough sequence. The on-call opens the alert. The alert title says something like "EC2 Instance High Outbound Traffic" or "Failed logins above threshold." There are no attached logs. There is a link to the Datadog dashboard, which requires re-authentication after a forced timeout. By the time the dashboard loads, the engineer is checking Slack to see if anyone else is awake. No one is. They open CloudTrail manually. They try to remember which region the affected instance is in. They check the asset register, which is a spreadsheet last updated six months ago.

Twenty minutes in, the engineer now has a rough picture of what might be happening. They have not taken any containment action yet. The attacker, if there is one, has had twenty minutes of uncontested time on the host.

This is not a failure of the engineer. It is a structural problem: the information needed to act is not where it is needed, at the time it is needed.

Why documentation does not close the gap

Most teams respond to this problem by improving their runbooks. They write more detail. They add screenshots. They update the escalation paths. These are all genuinely useful things to do, and we are not arguing against good runbooks.

But a runbook that lives in Confluence is not the same as a runbook that executes. When your on-call is half-asleep at 3am, reading a document is not the natural action. The natural action is to start investigating the thing that woke them up. The runbook becomes something to reference after the adrenaline has passed, not something that guides the first five minutes.

There is a specific failure mode we see often: the runbook says "Step 1: isolate the affected host." The engineer reads this, then opens a terminal, then has to figure out the correct AWS CLI command for their environment, then realizes they do not have the instance ID to hand, then goes back to Datadog. The runbook gave the right instruction but could not execute it.

What the first hour actually costs

If you model the blast radius of a compromised host over time, the curve is not linear. The first ten minutes are the highest-value window for interruption. Lateral movement typically starts within fifteen minutes of initial access, depending on the attack pattern. By thirty minutes, a determined attacker has usually established persistence and begun exfiltration attempts.

This is why the context-gathering loop is so costly. It is not just wasted engineer time. Every minute spent rediscovering what the affected host is, what it connects to, and what normal traffic looks like, is a minute the attacker is not being interrupted.

A useful benchmark: if your team's mean-time-to-isolate a confirmed compromised host is above fifteen minutes, you have a structural problem in your first-response process, not a skill problem. The fix is not training. It is changing what happens before the engineer opens the terminal.

What changes when the first steps run themselves

The shift is specific. When Parachute receives an alert from PagerDuty or Datadog, it does not wait for a human to decide what to investigate. The runbook starts immediately: the affected host is identified, relevant CloudTrail events are pulled, the host's network connections are captured, and if the runbook calls for it, the host is isolated from the network before the on-call has finished reading the alert title.

Consider a hypothetical but plausible scenario: a growing SaaS company, three-person security team, alert fires on unusual outbound traffic from a production EC2 instance at 2am. In a manual-first workflow, the on-call spends twelve minutes gathering context before isolating the host. In a runbook-automated workflow, isolation and log collection complete in under ninety seconds. The engineer wakes up to an alert that already includes the isolation timestamp, the pulled CloudTrail events, and a Slack message with the evidence package attached. Their job is now to review and decide next steps, not to collect.

The difference in the first hour is not just speed. It is the engineer's cognitive state. Arriving at a contained, documented incident is a fundamentally different problem than arriving at an open, uncontained one. The decisions required are different, the pressure is different, and the error rate is lower.

The mistakes we see most in first-response processes

Running isolation before evidence collection is the most common. The instinct to contain is correct but the order matters. If you isolate a host before pulling logs, you may lose volatile data that only exists in memory or in open network connections. The sequence should be: collect first, isolate second, at least for the automated steps.

The second mistake is alert fatigue from low-fidelity triggers. Teams that have automated host isolation on a poorly-tuned alert end up with isolated hosts and angry engineers. Automation needs to be paired with alert quality work, not used as a substitute for it. If you automate a bad signal, you automate a bad response.

Third: not having a clear escalation branch in the runbook. Automated steps handle the first five minutes well. But when Parachute completes the evidence collection and posts to Slack, what happens if the on-call is offline? The runbook needs an explicit path: who gets paged next, after how many minutes, and under what conditions is an incident automatically escalated to P0.

The boundary between automation and human judgment

We are not arguing that automation should replace the on-call. The job of incident response is ultimately a judgment problem: is this actual compromise or a misconfigured rule? What data is actually at risk? Who needs to know? Those are human questions.

What automation handles well is the undifferentiated first work: the log pulls, the asset identification, the initial containment, the evidence package. These steps are the same every time, regardless of what kind of incident it turns out to be. They should not require human intervention to start.

The place where human judgment becomes critical is after the initial automated steps have completed. The engineer now has a contained host, a populated evidence package, and a timeline of what Parachute executed. From that position, they are making decisions with information, not hunting for information while the clock runs.

A practical starting point

If you are not running any automation in your first-response process, the most valuable place to start is not with complex conditional branching. Start with the three steps that are always true for any alert involving a potentially compromised host: identify the asset, pull the last hour of CloudTrail events for that asset, and post the summary to your incident Slack channel with the on-call tagged.

That alone compresses the context-gathering phase from fifteen to twenty minutes down to seconds. Everything after that is your team doing the actual work of incident response, rather than setup.

The first hour of an incident is the most expensive. Getting the rote steps out of the way automatically is the only way to give your team the time they need to use judgment where judgment actually matters.