The Limits of Text Matching
Imagine you're a security auditor reviewing a building's access logs. You could search for the word "unauthorized" in every log file — but what if someone wrote a comment in a log entry that said "this door is NOT unauthorized"? A naive text search would flag it. What you really want is to understand the meaning of each entry, not just scan for keywords.
The same problem applies to code analysis. When you want to enforce a rule like "never use eval() in production code," your first instinct might be to reach for a Regular Expression (regex) — a recursive text search for the string eval( across every file. This feels fast and simple, but regex treats your source code as a flat string of characters. It has no understanding of the language's structure, so it makes systematic mistakes:
- It flags
# Don't use eval() here— a comment that will never execute. - It flags
logger.info("eval() is dangerous")— a string literal, not a function call. - It misses
eval ("user_input")— valid Python with extra whitespace. - It misses
builtins.eval(x)— the same function accessed through a different path.
These aren't edge cases you can patch with a cleverer regex. They're fundamental limitations of treating code as text. The solution is to analyze code the way the Python interpreter does: as a structured tree of logical operations.
Abstract Syntax Trees (ASTs) give you exactly that. An AST is the data structure a language parser produces when it reads your source file — a hierarchical representation of every statement, expression, and operation, stripped of all formatting noise. By walking this tree, you can write rules that understand what code does, not just what it looks like.
Interviewers at companies with large Python codebases (data platforms, ML infrastructure teams) sometimes ask: "How would you enforce a custom coding standard across a monorepo?" The wrong answer is "write a regex and run it in CI." The right answer explains AST-based analysis: you parse the code into a tree, visit specific node types, and integrate with a tool like Pylint or Flake8 so the check runs alongside standard linting. This demonstrates you understand the difference between syntactic and semantic analysis.