Tokenization: Protecting Data at Rest

The Difference from Encryption

Encryption is a mathematical transformation: given a key and an algorithm (like AES-256), you can deterministically scramble data and unscramble it. The security relies on the secrecy of the key and the strength of the math. If the key is compromised, all encrypted data is exposed.

Tokenization takes a fundamentally different approach: it replaces sensitive data with a random, non-sensitive placeholder called a token. There is no mathematical relationship between the token and the original value. The only way to recover the original data is to query the Token Vault — a highly secured, access-controlled system that holds the mapping table.

Original PAN:  4111 1111 1111 1111
Token:         Tok_7f3a9b2c1d4e5f6a

If an attacker steals the token, they have a random string. There is no algorithm to run, no key to brute-force. The token is cryptographically useless without access to the vault.

Use Case: PCI-DSS Compliance

The Payment Card Industry Data Security Standard (PCI-DSS) requires merchants to protect Primary Account Numbers (PANs — credit card numbers). Tokenization is the dominant solution:

1. Customer enters card number at checkout.
2. The payment system immediately forwards the PAN to a Tokenization Service Provider (e.g., Stripe, Braintree, or an in-house vault).
3. The provider returns a token: Tok_7f3a9b2c1d4e5f6a.
4. The merchant's application stores only the token in its database.
5. For future charges (subscriptions, refunds), the merchant sends the token to the provider, which looks up the real PAN and processes the transaction.

The merchant's database — the most commonly breached system — contains only tokens. Even a complete database dump yields nothing useful to an attacker. The Token Vault, which holds the real mappings, is a hardened system with strict access controls, audit logging, HSM (Hardware Security Module) protection, and typically operated by a specialized provider.

Format-Preserving Tokenization

A practical challenge: many legacy systems validate that a "credit card number" is exactly 16 digits. A random token like Tok_7f3a9b2c1d4e5f6a would break these systems. Format-Preserving Tokenization (FPT) generates tokens that match the format of the original data:

Original PAN:  4111 1111 1111 1111
FPT Token:     4999 3847 2910 5623

The token looks like a valid card number (same length, same character set) but maps to a random value in the vault. This allows tokenization to be dropped into legacy systems with minimal code changes.

Tokenization vs. Encryption: When to Use Which