Turning Log Files into Answers
In ops work, you frequently need to answer questions from data: "Which servers are overloaded?", "What is the most common error code in the last hour?", "Which IP addresses are generating the most traffic?" You could answer these with shell pipelines (grep | awk | sort | uniq -c), but those pipelines break on malformed lines, struggle with structured formats like JSON logs, and become unreadable past a certain complexity.
pandas is a data analysis library that treats structured data like an in-memory database table. Think of it as a programmable spreadsheet — you load data into a DataFrame (rows and columns), then filter, sort, group, and aggregate it with readable Python code.
Core Operations for Ops Work
Loading data:
import pandas as pd
# From a CSV file (e.g., an exported CloudWatch metrics report)
df = pd.read_csv('server_metrics.csv')
# From a JSON Lines file (one JSON object per line — common log format)
df = pd.read_json('app.log', lines=True)
# From a Python list of dicts (e.g., parsed boto3 API responses)
instances = [
{"id": "i-001", "type": "t3.micro", "state": "running", "cpu": 15},
{"id": "i-002", "type": "t3.large", "state": "running", "cpu": 94},
{"id": "i-003", "type": "t3.micro", "state": "stopped", "cpu": 0},
]
df = pd.DataFrame(instances)
Filtering rows (like SQL WHERE):
# Find all running instances with CPU above 80%
overloaded = df[(df['state'] == 'running') & (df['cpu'] > 80)]
Aggregating data (like SQL GROUP BY):
# Average CPU usage per instance type
avg_cpu_by_type = df.groupby('type')['cpu'].mean()
# Count instances in each state
state_counts = df['state'].value_counts()
Sorting:
# Find the top 5 highest-CPU instances
top_cpu = df.sort_values('cpu', ascending=False).head(5)
The power of pandas is that each of these operations works identically whether your DataFrame has 100 rows or 10 million rows. The library uses optimized C code under the hood, making it far faster than equivalent Python loops for large datasets.