The Failover Timeline
Understanding the exact sequence of events during a MemoryDB failover is critical for designing resilient applications. Here is what happens when a Primary node fails:
T+0s: Primary node in AZ-A becomes unreachable (hardware failure, AZ outage, etc.).
T+0s to T+5s: MemoryDB's cluster management detects the failure. It uses a quorum-based detection mechanism — Replica nodes and the control plane must agree the Primary is gone before acting. This prevents "split-brain" scenarios where a slow Primary is incorrectly declared dead.
T+5s to T+15s: A Replica is selected for promotion. Before accepting writes, the promoted Replica replays any transaction log entries it has not yet consumed. This is the key step that guarantees the new Primary has exactly the same state as the old Primary had at the moment of its last acknowledged write.
T+15s to T+20s: The new Primary begins accepting writes. DNS for the cluster endpoint is updated to point to the new Primary. Clients using the cluster endpoint automatically reconnect.
Total client impact: Writes are blocked for approximately 10–20 seconds. Reads from Replicas continue uninterrupted during this window (if your client is configured to read from Replicas).
Designing Applications for Failover
A 10–20 second write outage is not zero. Your application must handle it gracefully:
import redis
from redis.exceptions import ConnectionError, TimeoutError
import time
def resilient_write(r, key, value, max_retries=5, base_delay=0.5):
"""
Write to MemoryDB with exponential backoff retry.
Handles the ~10-20 second failover window gracefully.
"""
for attempt in range(max_retries):
try:
result = r.set(key, value)
return result
except (ConnectionError, TimeoutError) as e:
if attempt == max_retries - 1:
raise # Re-raise on final attempt
delay = base_delay * (2 ** attempt) # Exponential backoff
print(f"Write failed (attempt {attempt + 1}): {e}. Retrying in {delay:.1f}s...")
time.sleep(delay)
This retry pattern with exponential backoff handles the failover window: the first few retries fail during the promotion process, and by the time the backoff reaches 4–8 seconds, the new Primary is usually ready.
Interviewers frequently ask: "How do you handle the write outage during a MemoryDB or ElastiCache failover?" The answer has two parts: (1) implement client-side retry with exponential backoff to absorb the 10–20 second window, and (2) if your application cannot tolerate any write blocking, consider a write-ahead buffer (e.g., a local queue or SQS) that absorbs writes during the failover and replays them once the cluster recovers. The second approach adds complexity and is only justified for extremely write-sensitive workloads.
The Transaction Log's Role in Failover
The transaction log is what makes MemoryDB's failover semantically different from ElastiCache's:
- ElastiCache failover: The Replica becomes Primary with whatever state it had at the moment of promotion. If replication lag was 500ms, the last 500ms of writes are lost.
- MemoryDB failover: The Replica replays the transaction log to catch up to the exact last acknowledged write before accepting new writes. Zero data loss.
This replay step is why MemoryDB failover takes slightly longer than ElastiCache failover — it is doing more work to provide a stronger guarantee.