Building a security data lake is not just a storage decision — it is an architectural commitment with operational implications that affect query cost, data quality, and analyst productivity.
The Hot / Warm / Cold Tiering Pattern
Not all security data has equal query frequency. Structuring storage in tiers dramatically reduces cost:
| Tier | Data Age | Storage Class | Query Frequency | Cost |
|---|---|---|---|---|
| Hot | 0–7 days | S3 Standard or SSD | Daily (active investigations) | Higher |
| Warm | 7–90 days | S3 Infrequent Access | Weekly (threat hunting) | Medium |
| Cold | 90 days–7 years | S3 Glacier | Rarely (compliance, forensics) | Very low |
S3 Lifecycle Policies automate this transition: a single policy rule scoped to the security-lake/ prefix can declare that objects transition to Infrequent Access storage after 7 days and to Glacier after 90, with no application code involved.
Partitioning Strategy
How you partition your data in S3 determines how fast (and how cheaply) your queries run. Athena charges per byte scanned — a well-partitioned table can reduce query cost by 99% compared to a flat directory.
Bad partitioning (scans everything):
s3://security-lake/vpc-flow/all-logs/
Good partitioning (Athena prunes irrelevant partitions):
s3://security-lake/vpc-flow/year=2024/month=03/day=15/hour=14/
A query with WHERE year=2024 AND month=03 AND day=15 only scans that day's data, not the entire history.
Schema Evolution
Security log schemas change. AWS adds new fields to CloudTrail. Your EDR vendor releases a new agent version with additional telemetry. Your Parquet schema must accommodate this without breaking existing queries.
The standard approach is to use schema-on-read with a catalog (AWS Glue Data Catalog or Apache Hive Metastore): the schema is defined at query time, not at write time. New fields are simply ignored by old queries, and new queries can reference them immediately. This is fundamentally different from a traditional database where adding a column requires a migration.
Senior engineers are often asked: "How do you handle schema drift in a security data lake?" The answer demonstrates architectural maturity: use a schema registry (like AWS Glue) with schema-on-read semantics, write data in a format that supports schema evolution (Parquet with Avro-compatible field addition rules), and version your ETL normalization jobs so you can reprocess historical data if the schema changes significantly. Avoid schema-on-write for raw security logs — you cannot predict what fields a vendor will add next quarter.