Why Raw Counters Are Useless on a Graph
A Counter only ever goes up (or resets to zero on restart). If you graph http_requests_total directly, you get a line that climbs forever to the upper right. It tells you the cumulative total since the process started — not how busy the service is right now.
What you actually care about is the rate of change: how many requests per second is the service handling at this moment?
The rate() Function
rate() takes a Range Vector and computes the per-second average rate of increase over that window. It is the single most important function in PromQL. Applying it to a request counter filtered to a specific job over a 5-minute window means: "For each time series matching this selector, look at the last 5 minutes of samples. Fit a line through them. Report the slope in units of requests per second."
Counter reset handling: If a pod crashes and restarts, the counter resets from (say) 50,000 back to 0. Without special handling, this would look like a massive negative spike. rate() automatically detects resets and corrects for them — it knows a drop to zero is a restart, not negative traffic.
Choosing the window size [5m]:
- Too short (e.g., [1m]): Noisy, spiky graph. Sensitive to individual scrape gaps.
- Too long (e.g., [30m]): Smooth but slow to react. A spike at minute 1 is still visible at minute 30.
- [5m] is the standard default. Match it to your alerting evaluation interval.
The increase() Function
increase() answers a different question: not "how fast?" but "how many total?" Applying it to the same counter over a 1-hour window answers: "How many total requests arrived in the last hour?"
Mathematically, increase(metric[d]) ≈ rate(metric[d]) * duration_in_seconds. It is syntactic sugar — useful for dashboards showing "requests in the last hour" as a stat panel rather than a rate graph.
Senior nuance: Both rate() and increase() use linear interpolation between samples. If your scrape interval is 15s and you use a [30s] window, you only have 2 data points — the result is noisy and unreliable. The rule of thumb: your range window should be at least 4× your scrape interval. With a 15s scrape interval, use [1m] minimum, [5m] for stability.