The Anti-Pattern: Queue in Database

The Temptation

It starts innocently. You need to process background jobs — sending emails, resizing images, generating reports. You already have a PostgreSQL or MySQL database running. Why spin up another piece of infrastructure? Just create a jobs table with a status column, and have workers poll it continuously with something like SELECT * FROM jobs WHERE status = 'PENDING' LIMIT 1 FOR UPDATE SKIP LOCKED.

This works. It's transactional, it's simple, and it reuses existing infrastructure. At 10 jobs per minute with 2 workers, you'll never notice a problem.

Why It Kills Databases at Scale

At scale, this pattern becomes a Reliability Anti-Pattern for three compounding reasons:

1. Polling Thundering Herd

If you have 500 workers each polling every second, that's 500 queries per second (QPS) hitting your database just to say "nothing to do." These queries aren't free — each one requires the database to parse SQL, plan the query, acquire locks, scan the index, and return an empty result set. This is pure overhead with zero business value.

2. MVCC Bloat and Vacuum Pressure

PostgreSQL uses Multi-Version Concurrency Control (MVCC), which means that when you UPDATE or DELETE a row, the old version isn't immediately removed — it's marked as dead and left in place until the VACUUM process cleans it up. A job queue is extremely high-churn: rows are inserted as PENDING, updated to PROCESSING, then deleted or updated to DONE. This generates enormous numbers of dead tuples. The autovacuum process must work overtime to reclaim space, consuming CPU and I/O that should be serving real queries. Left unchecked, this causes table bloat — the physical table file grows to many times its logical size, degrading scan performance.

3. Lock Contention

Even with SKIP LOCKED (which tells workers to skip rows already locked by another worker), the lock management overhead on a hot table is significant. The database must maintain lock state for every in-flight row, and this internal bookkeeping competes with your actual application queries.

The Fix

Use a purpose-built message broker. Redis (with the Sidekiq or Celery client libraries), RabbitMQ, Apache Kafka, or AWS SQS are all designed specifically for high-churn, ephemeral data. They store messages in memory (with optional persistence), support thousands of consumers without lock contention, and don't suffer from MVCC bloat. Keep your relational database for what it's good at: durable, queryable, relational data.