Alerting and threat hunting are fundamentally different activities, even though both use the same data lake.
Alerting is reactive and automated. You define a known-bad pattern in advance, and the system notifies you when it occurs. It answers: "Did the thing I expected happen?"
Threat Hunting is proactive and human-driven. A skilled analyst starts with a hypothesis — an educated guess about attacker behavior — and goes looking for evidence of it, even if no alert has fired. It answers: "Is something bad happening that I haven't thought to look for yet?"
The critical enabler of threat hunting is long-term data retention. Sophisticated attackers (APT groups, nation-state actors) operate on timescales of months. They may compromise a system in January, sit dormant until March, and exfiltrate data in June. A 30-day SIEM retention window makes this attack invisible in hindsight. A data lake retaining 2 years of logs makes it reconstructable.
The Hypothesis-Driven Hunting Workflow
Step 1 — Form a hypothesis from intelligence.
A threat intelligence report describes a new ransomware group that uses a specific User-Agent string (Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) — an outdated IE9 string no legitimate browser sends) to download their second-stage payload.
Step 2 — Translate to a query.
SELECT
src_addr,
dst_addr,
http_user_agent,
COUNT(*) as request_count,
MIN(start_time) as first_seen,
MAX(start_time) as last_seen
FROM load_balancer_logs
WHERE
http_user_agent LIKE '%MSIE 9.0%'
AND start_time >= NOW() - INTERVAL '90' DAY
GROUP BY src_addr, dst_addr, http_user_agent
ORDER BY first_seen ASC
Step 3 — Investigate matches.
The query returns 3 hits from 60 days ago — before the threat intelligence report was even published. This means the attacker was already in the environment.
Step 4 — Pivot and correlate.
Take the source IP from the match and join across datasets to reconstruct the full attack chain.
Cross-Dataset Correlation: Reconstructing an Attack
The real power of a data lake is the ability to join across the three telemetry planes to tell a complete story:
-- Full attack chain reconstruction
WITH initial_access AS (
-- Step 1: Find the suspicious HTTP request (Load Balancer logs)
SELECT src_addr as attacker_ip, dst_addr as compromised_instance_ip, start_time
FROM load_balancer_logs
WHERE http_user_agent LIKE '%MSIE 9.0%'
),
c2_traffic AS (
-- Step 2: Find outbound C2 connections from that instance (VPC Flow Logs)
SELECT src_addr, dst_addr as c2_ip, bytes, start_time
FROM vpc_flow_logs
WHERE src_addr IN (SELECT compromised_instance_ip FROM initial_access)
AND bytes BETWEEN 40 AND 200 -- Small beaconing packets
),
endpoint_activity AS (
-- Step 3: Find what the malware did on the machine (EDR logs)
SELECT hostname, process_name, command_line, event_time
FROM edr_process_events
WHERE src_ip IN (SELECT src_addr FROM c2_traffic)
AND event_time > (SELECT MIN(start_time) FROM initial_access)
)
SELECT * FROM initial_access
JOIN c2_traffic ON initial_access.compromised_instance_ip = c2_traffic.src_addr
JOIN endpoint_activity ON c2_traffic.src_addr = endpoint_activity.hostname
ORDER BY event_time ASC
This single query, running across three separate datasets, reconstructs the complete attack timeline: initial web shell delivery → C2 beaconing → malware execution on the endpoint.