The Federated Averaging Algorithm
The foundational FL algorithm is Federated Averaging (FedAvg), introduced by Google researchers in 2017. It is elegant in its simplicity. Training proceeds in discrete rounds, each consisting of six steps:
Step 1 — Selection: The central server selects a random subset of eligible devices. "Eligible" typically means: device is charging, on unmetered Wi-Fi, and idle. For a global deployment, this might mean selecting 1,000 phones from a pool of 10 million.
Step 2 — Broadcast: The server sends the current global model weights w_t to each selected device. This is a one-way push — the model travels to the data.
Step 3 — Local Training: Each device independently trains the model on its own local dataset for a fixed number of steps (called local epochs, typically 1–5). Starting from w_t, each device computes a locally improved set of weights w_t^k for device k.
Step 4 — Reporting: Each device computes its update: Δw^k = w_t^k - w_t (the delta between its locally trained model and the starting global model). It sends only Δw^k back to the server — not its local data, not its full model weights in a way that reveals training data.
Step 5 — Aggregation: The server aggregates all received updates using a weighted average:
w_{t+1} = w_t + (1/N) * Σ(n_k / n * Δw^k)
Where n_k is the number of local data points on device k and n is the total data points across all participating devices. Devices with more data contribute proportionally more to the global update.
Step 6 — Repeat: The updated global model w_{t+1} becomes the starting point for the next round.
Why Averaging Works
The intuition: if 1,000 people each independently discover that "rizz" tends to follow "no" in text, and each of their local models shifts slightly to predict that pattern, averaging those 1,000 small shifts produces a reliable global signal. No single user's data dominates. Rare patterns that appear on only a few devices get diluted, while common patterns that appear across many devices get amplified — exactly the behavior you want.
The number of local epochs is a critical hyperparameter. More local epochs means less communication (fewer rounds needed) but increases the risk of client drift — local models diverging so far from the global model that their updates conflict rather than cooperate. This trade-off between communication efficiency and model convergence is one of the central engineering challenges in FL system design.