Log Parsing: Streaming Large Files

The "Out of Memory" Trap

Imagine a production log file that is 100GB. Your laptop has 16GB of RAM. If you try to load the entire file at once, the operating system will kill your process — or worse, it will start swapping to disk, grinding the machine to a halt. This is a real failure mode that has caused production incidents.

The interviewer who hands you this problem is specifically testing whether you know to avoid this trap.

The Solution: Line-by-Line Streaming

Python's file iterator is lazy by default — it reads one line at a time from disk, processes it, and discards it before reading the next. Memory usage stays constant regardless of file size:

from collections import Counter

def top_ips(filepath, k=5):
    ip_counts = Counter()
    with open(filepath) as f:
        for line in f:          # Reads ONE line at a time — O(1) memory per line
            line = line.strip()
            if not line:
                continue
            parts = line.split()
            if parts:
                ip_counts[parts[0]] += 1
    return ip_counts.most_common(k)

results = top_ips("/var/log/nginx/access.log", k=5)
for ip, count in results:
    print(f"{ip}: {count} requests")

Memory profile:
- ip_counts dict: O(U) where U = number of unique IPs (bounded, manageable)
- Line buffer: O(L) where L = length of one line (tiny, constant)
- Total: effectively O(U), not O(file size)

Generators for Pipeline Processing

For more complex pipelines, Python generators let you compose streaming operations without materializing intermediate results:

def read_lines(filepath):
    with open(filepath) as f:
        for line in f:
            yield line.strip()

def parse_ip(lines):
    for line in lines:
        parts = line.split()
        if parts:
            yield parts[0]

def count_top_k(items, k):
    return Counter(items).most_common(k)

# The pipeline — no intermediate list is ever fully materialized
lines = read_lines("/var/log/nginx/access.log")
ips = parse_ip(lines)
print(count_top_k(ips, k=5))

This generator pipeline pattern is the foundation of tools like awk, grep, and Unix pipes — each stage processes one record at a time and passes it downstream.

Senior insight: For truly massive files (terabytes), even the Counter dict can exceed RAM if there are billions of unique keys. In that case, you would use an external sort (write chunks to disk, merge-sort them) or a probabilistic data structure like Count-Min Sketch, which approximates frequency counts in fixed, sub-linear memory.