Digital Forensics

Evidence-First Incident Response: Collecting Before You Contain

Marcus Webb 7 min read
Abstract dark concept of digital evidence collection and forensic layers

The standard incident response advice on containment is to act fast. Isolate the host. Cut off the attacker's access. Stop the bleeding. This is broadly correct advice, and speed of containment is one of the most important factors in limiting blast radius.

But there is a sequencing error embedded in how most teams execute it. The instinct to isolate immediately, before anything else, often destroys the evidence you need to understand what actually happened. By the time the post-mortem starts, you are working from an incomplete forensic record, not because the attacker cleaned up, but because your own containment action cleared it.

What isolation actually does to your evidence

When you isolate a host, you are cutting its network connections. This is the right outcome for containment. The problem is that isolation, especially when done by modifying security group rules or pulling a host from a VPC, does not preserve the host's in-memory state. Processes continue running. Network connection tables reflect the post-isolation state, not the active connection state at the time of compromise. Log buffers that were not yet flushed to disk may be lost if the process is killed.

More concretely: if an attacker established a reverse shell from your EC2 instance to an external C2 server, the active connection details, including the destination IP and port, exist in the host's network connection table right up until you modify the security group. After isolation, that connection is gone. If you did not capture the netstat output before cutting the network, you have lost one of the most actionable pieces of evidence: where the attacker was calling home to.

Similarly, process memory on a running host contains artefacts that exist nowhere else. Injected shellcode, decrypted credentials that were never written to disk, the full command history of an attacker's shell session. These are volatile. They do not survive a reboot, and they do not survive a forensic acquisition taken after the fact if the host has been running and processing new activity since compromise.

The evidence collection window is short regardless of what you do

The counterargument to evidence-first is: if the attacker is actively exfiltrating data or moving laterally, every second of delay is additional exposure. That is true and it is not a trivial concern.

But the evidence collection we are describing is not a lengthy forensic investigation. It is a set of targeted, fast captures of the most volatile data: active network connections (netstat or equivalent), running process list, recently modified files in common malware staging directories, any open file handles associated with unusual processes. This data can be collected via a single script execution in seconds, not minutes.

In Parachute's default evidence-first runbook sequence, the evidence capture step runs before the isolation step, but the gap is measured in seconds. The runbook sends a command to the host agent to capture process list and network connections, receives the output, and then immediately triggers the isolation. The total time from alert to isolation is slightly longer than isolation-first, but the difference is rarely more than ten to fifteen seconds. Against that, you have captured evidence that would otherwise be permanently lost.

What to collect, specifically

The volatile data that matters most, in rough priority order:

Active network connections: Captures the destination IPs and ports of all established and listening connections. This is the most actionable evidence for identifying C2 infrastructure. After isolation, this data is gone.

Running process list: Includes parent-child relationships, full command line arguments, and user context for each process. An injected process often shows an unusual parent-child relationship (e.g., cmd.exe spawned by a legitimate service process) that disappears if the host is rebooted or the processes are killed.

Recently modified files: A targeted search for files modified in the last two to four hours in directories commonly used for staging (temp directories, user profile directories, system32). This catches droppers and toolkits that were written to disk.

CloudTrail events for the affected identity: If the compromise involves a cloud principal, pulling the last few hours of CloudTrail events for that principal before taking any remediation action preserves the API call history. Isolating the host does not affect CloudTrail, but if you also rotate or invalidate the credentials as part of containment, some CloudTrail metadata may be harder to reconstruct.

Active database connections and recent queries (if applicable): For hosts running database services, capturing open connections and recent query history before isolation can confirm whether data exfiltration was in progress. This is particularly relevant for incidents triggered by anomalous database activity.

The post-mortem cost of skipping this step

Post-mortem quality degrades directly with the quality of the forensic record. When the forensic record is incomplete because evidence was destroyed during containment, the post-mortem answers the question "what did we observe before we acted?" rather than "what actually happened?"

Consider a hypothetical: a growing SaaS team responds to a Datadog alert about unusual outbound traffic from a production instance. They isolate immediately. The network connection table is cleared on isolation. Twelve hours later, writing the post-mortem, they know the traffic was unusual but cannot identify where it was going. They cannot determine if data was exfiltrated because the destination is unknown. The post-mortem concludes with "possible data exfiltration, destination unconfirmed."

If the team had captured the active network connections before isolating, that destination IP would be in the evidence package. The post-mortem would know whether the IP was a known C2 address, a cloud storage service suggesting exfiltration, or a false positive from a misconfigured service. That is the difference between an incident with an unknown outcome and an incident with a known one.

When isolation-first is the right call anyway

Evidence-first is the right default. There are situations where isolation-first is correct, and being clear about when they apply is important.

If ransomware is actively encrypting files, every second before isolation is additional encrypted data. The evidence destruction from encryption in progress is worse than the evidence loss from isolation. Isolate first.

If the compromised host is actively exfiltrating large volumes of data over a high-bandwidth connection, the evidence collection delay permits additional exfiltration. In this case, evaluate whether a brief capture window (under thirty seconds) is acceptable given the data transfer rate. In most cases it is; large-scale exfiltration at a rate that makes thirty seconds significant is a specific scenario, not the common case.

If the host is part of critical infrastructure where any additional connectivity, even the seconds required for evidence collection, carries regulatory or contractual risk, isolation-first may be required by policy. This is a legitimate constraint, though teams in this position should work to pre-collect as much context as possible through their monitoring stack so the post-mortem has something to work from.

Building evidence-first into your runbooks

The practical implementation is straightforward. In your Parachute runbook, the evidence capture action precedes the isolation action. The runbook collects netstat output, process list, and recent file modifications, stores them to the evidence package, and then triggers host isolation. The sequence is deterministic and fast.

The result is that when your on-call opens the incident, they have both a contained host and a populated evidence package. They are not starting the investigation from zero. The decisions they need to make, about scope, about impact, about whether to notify anyone, can be made from evidence rather than from guesswork. That is a better starting point for the post-mortem, and a better outcome for the people running it at 4am.