Seeing the Invisible: VPC Flow Logs

When network packets are getting dropped, how do you know why? You cannot run Wireshark on the AWS underlying infrastructure — the physical network is abstracted away. AWS gives you the next best thing: VPC Flow Logs.

VPC Flow Logs capture metadata about IP traffic flowing to and from network interfaces in your VPC. You can enable them at three levels of granularity:
- VPC level: Captures all traffic across all ENIs in the VPC.
- Subnet level: Captures all traffic for ENIs in a specific subnet.
- ENI level: Captures traffic for a single network interface.

What Flow Logs Capture (Metadata, Not Payload)

Flow logs do not capture packet content. You cannot see passwords, HTTP bodies, or application data. What you can see is the connection metadata:

version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status
2       123456789012 eni-abc123de 10.0.1.50 203.0.113.5 54321 443 6 20 1000 1620000000 1620000060 REJECT OK

Breaking down the key fields:
* srcaddr / dstaddr: Source and destination IP addresses.
* srcport / dstport: Source and destination ports.
* protocol: 6 = TCP, 17 = UDP, 1 = ICMP.
* packets / bytes: Volume of traffic in the flow.
* action: ACCEPT or REJECT — determined by Security Groups and NACLs.
* log-status: OK means the log was delivered successfully.

Where Flow Logs Are Sent

You can publish Flow Logs to:
- Amazon CloudWatch Logs: Best for real-time querying with CloudWatch Logs Insights.
- Amazon S3: Best for long-term retention and bulk analysis with Amazon Athena.
- Amazon Kinesis Data Firehose: Best for streaming to third-party SIEM tools.

Use Cases

1. Troubleshooting connectivity: Find REJECT entries to identify which firewall rule is blocking traffic and at which layer (Security Group vs. NACL).
2. Security monitoring: Detect port scanning (many REJECT entries from a single source IP across many ports), data exfiltration (unusually large bytes values on outbound flows), or unexpected outbound connections to unknown IPs.
3. Cost optimization: Identify which resources are generating the most cross-AZ or NAT Gateway traffic.

Important limitation: Flow Logs have a delay — typically 5–15 minutes before records appear in CloudWatch Logs or S3. They are not a real-time packet capture tool.