What is a Token?
LLMs don't process text character by character or word by word — they operate on tokens, which are chunks of text produced by a tokenizer algorithm (typically Byte-Pair Encoding, or BPE).
- Common short words are usually one token: "cat" → 1 token, "the" → 1 token.
- Longer or rarer words are split: "unbelievable" → ["un", "believ", "able"] → 3 tokens.
- Code and special characters often tokenize inefficiently:
{"key": "value"}might be 8–10 tokens. - Rule of thumb: 1,000 tokens ≈ 750 English words ≈ roughly one page of text.
Understanding tokenization matters because every API call and every self-hosted inference run is billed or budgeted in tokens, not words or characters.
Input vs. Output: Why They Cost Differently
When you send a request to an LLM, there are two distinct phases:
-
Prefill (Input Tokens): The model processes your entire prompt in parallel. All input tokens are processed simultaneously across the GPU's parallel compute units. This is fast and relatively cheap.
-
Decode (Output Tokens): The model generates the response one token at a time, sequentially. Each token requires a full forward pass through the model. This is slow and expensive — typically 3–5x the cost per token compared to input.
A typical commercial API pricing structure reflects this asymmetry:
| Token type | Example price |
|---|---|
| Input | $0.01 per 1K tokens |
| Output | $0.03 per 1K tokens |
Practical implication: Asking the model to summarize a 10-page document (large input, small output) is much cheaper than asking it to write a 10-page report (small input, large output). Designing prompts to minimize output verbosity — for example, asking for bullet points instead of prose — directly reduces costs.
Cost Modeling at Scale
At production scale, token costs compound quickly. Consider a customer support chatbot:
- Average conversation: 500 input tokens + 200 output tokens
- Cost per conversation: (0.5 × $0.01) + (0.2 × $0.03) = $0.005 + $0.006 = $0.011
- At 100,000 conversations/day: $1,100/day or ~$400,000/year
This is why LLM Ops teams obsess over prompt efficiency, output length constraints (max_tokens), and caching repeated prompts. A 20% reduction in average output length translates directly to a 20% reduction in the most expensive part of your bill.
A common interview question is: "How would you reduce LLM API costs in production?" Weak answers focus only on using a cheaper model. Strong answers address the full cost surface: (1) prompt caching for repeated system prompts, (2) output length constraints, (3) routing simple queries to smaller/cheaper models and complex queries to larger ones (model routing), and (4) RAG to reduce context size. Mention that output tokens are 3–5x more expensive than input tokens — this signals you understand the underlying economics.