This post covers the mechanics of how Parachute handles an EC2 isolation workflow: what happens in what order, what CloudTrail events we pull, and where the edge cases are. This is a technical walkthrough aimed at teams evaluating whether to build this themselves or use Parachute's built-in action.
What We Are Trying to Accomplish
An automated EC2 isolation workflow needs to do three things in order, and it needs to do them quickly:
- Capture the current security group state of the instance before modifying anything
- Swap the instance to a quarantine security group that blocks all traffic except a defined management channel
- Pull the relevant CloudTrail events from the period around the alert, so the responding engineer has context when they open the incident
The ordering matters. If you collect evidence after isolation, you risk the CloudTrail events being delayed in delivery (CloudTrail has a typical delivery lag of 5 to 15 minutes for management events). If you modify the security group without saving the original state, you make recovery harder and the audit trail incomplete. If you do not define a management channel before isolation, your own SSM-based access to the instance may be cut off.
Step One: Snapshot the Instance Security Group State
Before touching the instance, Parachute calls ec2:DescribeInstances and ec2:DescribeSecurityGroups to record the current security groups attached to each network interface. This snapshot is written to the incident record.
The call looks like this against the AWS SDK:
// describe_instance_network_interfaces returns the ENI list
// each ENI may have multiple security groups
instance_details = ec2.describe_instances(InstanceIds=[instance_id])
for reservation in instance_details['Reservations']:
for instance in reservation['Instances']:
for ni in instance.get('NetworkInterfaces', []):
eni_id = ni['NetworkInterfaceId']
current_sgs = [sg['GroupId'] for sg in ni['Groups']]
# store eni_id -> current_sgs mapping in incident recordMulti-ENI instances are common for hosts that bridge a production VPC and a management VPC. If you only snapshot the primary ENI, you may isolate the production interface but leave the management interface open, which is not the isolation you intended. Pull all ENIs.
Step Two: Apply the Quarantine Security Group
The quarantine security group should already exist in your account before an incident occurs. Creating it on the fly during an incident adds latency and creates a risk that the creation call fails or produces an incorrectly configured group under pressure.
The quarantine security group configuration Parachute recommends:
- No inbound rules (block all inbound traffic)
- One outbound rule: TCP port 443 to the SSM VPC endpoint (to preserve Systems Manager access)
- No other outbound rules
The SSM carve-out is important. Without it, you lose the ability to run SSM Run Command against the isolated instance, which removes your primary mechanism for collecting volatile evidence without opening network access. If you use a bastion host model instead of SSM, substitute the bastion host's IP for the SSM endpoint carve-out.
The security group swap is applied to each ENI independently:
ec2.modify_network_interface_attribute(
NetworkInterfaceId=eni_id,
Groups=[quarantine_sg_id]
)Step Three: CloudTrail Evidence Collection
CloudTrail evidence collection for a host compromise centers on a few specific event types. Parachute's default evidence collection window is 4 hours before the alert timestamp.
The events we look for, in priority order:
IAM and credential events related to the instance role: AssumeRole calls from the instance's IAM role, particularly calls that originate from an unexpected source IP or an unexpected UserAgent. If an attacker has compromised instance credentials, the most visible evidence in CloudTrail is credential use from outside AWS infrastructure or from an unexpected region.
EC2 control plane activity on the instance: ModifyInstanceAttribute, AuthorizeSecurityGroupIngress (if the attacker tried to open inbound rules), CreateKeyPair, ImportKeyPair. Any control plane modification to the instance or its associated resources in the window before the alert is significant.
VPC flow log correlation: If VPC Flow Logs are enabled and shipped to CloudWatch Logs or S3, Parachute queries for the instance's private IP address as a source across the 4-hour window, looking for connections to unusual destination IPs or unusual ports. Flow log delivery is typically much faster than CloudTrail management events, so this is often the richest evidence set for active exfiltration or lateral movement.
The CloudTrail lookup call itself:
cloudtrail.lookup_events(
LookupAttributes=[
{'AttributeKey': 'ResourceName', 'AttributeValue': instance_id}
],
StartTime=alert_time - timedelta(hours=4),
EndTime=alert_time + timedelta(minutes=30)
)One important limitation: CloudTrail lookup_events only returns management events by default, and has a maximum results cap of 50 events. For high-activity accounts, important events can be cut off. For production use, the more reliable approach is to query CloudTrail Lake or a SIEM with a direct CloudTrail S3 ingest, rather than relying on the lookup API alone.
What the Responding Engineer Sees
When a Parachute runbook completes this workflow, the incident record in the platform contains:
- Original security group IDs for each ENI on the instance (for reverting after the incident)
- Confirmation that the quarantine security group was applied, with timestamp
- Formatted CloudTrail event list grouped by event type, sorted by time
- Any anomalous AssumeRole calls flagged automatically (different source IP from the instance's usual activity pattern)
- A Slack notification with a summary and link to the full incident record
The engineer opens the Slack notification, clicks through to the incident, and has the evidence in front of them before they have opened a terminal. The first thing they do is review the evidence, not gather it.
The Edge Cases Worth Knowing
Several configurations cause the default workflow to behave unexpectedly. Worth documenting before you hit them during an incident.
Instances with EBS-backed root volumes that are also mounted by other instances: Isolation does not prevent a running process from writing to the root volume. If the instance is running a destructive payload, local disk damage continues after network isolation. The only way to stop local writes is to stop the instance, which requires balancing evidence preservation against ongoing damage.
RDS instances in the same security group as the EC2 instance: Applying a quarantine security group to the EC2 instance does not automatically reroute traffic from the EC2 instance to the RDS instance. But if the RDS security group references the original EC2 security group by ID (a common pattern), those rules still exist. Verify the RDS security group rules separately.
CloudTrail delivery lag: For events that occurred within the last 15 minutes before the alert, CloudTrail may not have delivered them yet. The evidence set collected immediately after alert receipt may be incomplete. Parachute retries the CloudTrail collection 20 minutes after initial collection to catch delayed events and appends them to the incident record.
These are edge cases we have run into building and testing the EC2 isolation action. Most of them have workarounds that can be encoded in the runbook configuration. The important thing is to know about them before the incident, not during it.
