Log Parsing & String Manipulation

The Most Common SRE Task

Logs are the primary source of truth in a running system. When something breaks at 3 AM, you are not reading dashboards — you are reading logs. The ability to quickly extract structured information from unstructured text is one of the most practical skills an SRE can have.

Anatomy of a Log Line

Consider a standard Apache Combined Log Format entry:

192.168.1.1 - alice [10/Oct/2023:13:55:36 -0700] "GET /index.html HTTP/1.1" 200 1024

The fields, in order, are: client IP, identity, user, timestamp, request line, HTTP status code, and response size. Your job is to reliably extract one or more of these fields from millions of such lines.

Manual Split vs. Regular Expressions

Approach 1: String Split — Fast and Predictable

Split on whitespace and access fields by index:

line = '192.168.1.1 - alice [10/Oct/2023:13:55:36 -0700] "GET /index.html HTTP/1.1" 200 1024'
parts = line.split()
ip = parts[0]       # '192.168.1.1'
status = parts[8]   # '200'
size = parts[9]     # '1024'

• Time complexity: O(N) where N is line length.
• Pro: Extremely fast, easy to debug, readable.
• Con: Fragile. If a URL contains a space (e.g., GET /path with spaces), the index shifts and you silently extract the wrong field.

Approach 2: Regular Expressions — Powerful but Costly

import re

pattern = re.compile(
    r'(?P<ip>\S+) \S+ \S+ \[.*?\] "\S+ \S+ \S+" (?P<status>\d{3}) (?P<size>\d+)'
)
match = pattern.match(line)
if match:
    ip = match.group('ip')
    status = match.group('status')

• Pro: Handles variable-length fields, quoted strings, and format variations correctly.
• Con: Slower (regex engine overhead), harder to read, and regex bugs are notoriously difficult to debug. As the saying goes: "Some people, when confronted with a problem, think 'I know, I'll use regular expressions.' Now they have two problems."

The Interview Strategy: Always start with split() and explain its limitations. Only reach for regex when the interviewer introduces a format variation that breaks the index-based approach. This demonstrates pragmatism — a core SRE trait.

Counting Frequencies: The Right Data Structure

Once you can extract a field (like an IP address), counting frequencies is a standard pattern:

from collections import Counter

ip_counts = Counter()
with open("access.log") as f:
    for line in f:
        parts = line.split()
        if len(parts) > 0:
            ip_counts[parts[0]] += 1

top_5 = ip_counts.most_common(5)
print(top_5)
# [('192.168.1.1', 4821), ('10.0.0.5', 3102), ...]

Counter.most_common(k) uses a heap internally and runs in O(N log k) time — far more efficient than sorting the entire dictionary when k is small.