Provisioning New Cells
Cells are not static. As your user base grows, you provision new cells. This process should be fully automated — a "cell stamp" is a parameterized infrastructure template (Terraform module, CloudFormation stack, Helm chart) that can be instantiated with a single command, parameterized by a new cell ID, target region, and capacity.
Once provisioned, the cell is registered in the routing table as available. New tenants are assigned to it. Existing tenants are not automatically migrated — migration is a separate, careful operation.
Tenant Migration
Sometimes you need to move a tenant from one cell to another — for example, to rebalance load or to give a growing customer their own dedicated cell. The migration process follows a read-write split pattern:
- Snapshot: Take a consistent snapshot of the tenant's data in the source cell.
- Sync: Continuously replicate changes from source to destination cell (the tenant is still live on the source).
- Cutover: Update the routing table to point the tenant to the destination cell. New requests go to the destination. In-flight requests on the source complete normally.
- Drain: Wait for the source cell to drain all in-flight requests for this tenant.
- Cleanup: Remove the tenant's data from the source cell.
This process can be done with zero downtime if implemented carefully. The routing table update is the atomic moment of cutover.
Cell Health and Observability
Each cell should emit its own metrics, logs, and traces — tagged with the cell ID. Your monitoring system should be able to answer: "Show me the error rate for Cell 7 over the last hour." This per-cell observability is what makes it possible to detect and isolate a misbehaving cell quickly.
Key metrics to track per cell:
- Request latency (p50, p99, p999)
- Error rate (4xx, 5xx)
- Capacity utilization (current tenants / max tenants)
- Database connection pool saturation
When a cell's error rate spikes above a threshold, an automated system can mark it as unhealthy in the routing table, preventing new traffic from being assigned to it while the on-call engineer investigates.