Privacy-Preserving AI: Differential Privacy and Secure Aggregation

The Gradient Leakage Problem

Federated Learning's privacy guarantee seems intuitive: raw data never leaves the device, so privacy is preserved. This intuition is wrong in its naive form.

In 2019, researchers demonstrated Gradient Inversion Attacks (also called Model Inversion Attacks): given only the gradient update Δw sent by a device, an adversary can reconstruct the original training images with high fidelity — including recognizable faces from medical images. The gradients encode information about the training data more than initially assumed.

This means FL's privacy guarantee is not automatic. It requires deliberate cryptographic and statistical defenses layered on top of the basic FedAvg protocol.

Defense Layer 1: Differential Privacy (DP)

Differential Privacy is a mathematical framework that provides a provable privacy guarantee: the output of a computation (in this case, the gradient update) reveals negligibly more information about any single individual than if that individual had not participated at all.

In practice, DP is implemented via two operations applied to the gradient before it leaves the device:

1. Gradient Clipping: Clip the gradient norm to a maximum value C. This bounds the maximum influence any single training example can have on the update: Δw_clipped = Δw * min(1, C / ||Δw||).

2. Gaussian Noise Addition: Add random noise sampled from a Gaussian distribution scaled to C: Δw_private = Δw_clipped + N(0, σ²C²I).

The noise makes it statistically impossible to determine whether any specific data point was in the training set. The privacy guarantee is quantified by parameters (ε, δ) — lower ε means stronger privacy but more noise and worse model accuracy. This is the fundamental privacy-utility trade-off in DP.

# Conceptual pseudocode for DP-SGD on device
for batch in local_data:
    gradients = compute_gradients(model, batch)
    # Step 1: Clip
    clipped = clip_by_norm(gradients, max_norm=C)
    # Step 2: Add noise
    noisy = clipped + gaussian_noise(scale=sigma * C)
    apply_update(model, noisy)

Defense Layer 2: Secure Aggregation (SecAgg)

Even with DP, the server still receives individual device updates. A compromised server (or a man-in-the-middle attacker) could analyze individual updates to infer information about specific users.

Secure Aggregation uses cryptographic protocols (specifically, Secure Multi-Party Computation, SMPC) to ensure the server can compute the sum of all device updates without ever seeing any individual update in plaintext.

The mechanism works via secret sharing with pairwise masks:

1. Each pair of devices (i, j) agrees on a random mask r_{ij} using a key agreement protocol (e.g., Diffie-Hellman).
2. Device i adds +r_{ij} to its update for each partner j, and subtracts r_{ji} for each partner where j > i.
3. When the server sums all masked updates, the masks cancel out algebraically, revealing only the true sum.

Analogy: Ten people want to compute their average salary without revealing individual salaries. Each person adds a secret random number to their salary before announcing it. The random numbers are coordinated so they sum to zero across the group. The announced numbers sum to the true total salary, but no individual's true salary is revealed.

Device i sends: Δw_i + Σ_j(r_{ij}) - Σ_j(r_{ji})
Server computes: Σ_i(masked updates) = Σ_i(Δw_i)  [masks cancel]

SecAgg adds communication overhead (the key agreement phase) and fails gracefully when devices drop out mid-round — a significant engineering challenge in cross-device deployments where dropout rates can exceed 30%.