The Hardest Join in Data Engineering
When you train a model, you are not analyzing current data — you are replaying history. You are asking: "Given what we knew at this exact moment in the past, what would the model have predicted?" Getting this right requires a type of join that most databases do not natively support.
Consider a concrete example. You want to train a model to predict whether a user will click an ad.
- Observation: User 55 clicked an ad on June 1st at 10:00 AM.
- Feature you want:
total_clicks_lifetimefor User 55.
If you query the database today (July 1st), User 55 has 100 lifetime clicks. But on June 1st at 10:00 AM, they had only 50 clicks. The other 50 clicks happened after the event you are trying to model.
If you train your model with the value 100, you are teaching it a pattern that did not exist at prediction time. This is Data Leakage — future information contaminating the training signal. The model will appear to perform well in offline evaluation (because the leaked signal is genuinely predictive) but will fail in production (because the future is not available at inference time).
AS OF Joins
The Feature Store solves this with Point-in-Time Joins, sometimes called AS OF joins. The mechanism works as follows:
- You provide a DataFrame of observations — each row is an event with an entity ID and a timestamp.
- The Feature Store joins each observation against the feature history, finding the feature value that was valid at or before that observation's timestamp.
- It respects the feature's TTL (Time To Live) — if the most recent feature update before the observation timestamp is older than the TTL, it returns
NULLrather than a stale value.
In Feast, this is a single call: you pass a DataFrame of observations (entity ID + timestamp) to get_historical_features(), and it returns a training-ready DataFrame where each row's feature value reflects what was true at that row's specific timestamp — so User 55's June 1st row gets the June 1st click count, not whatever the count is today.
The result is a historically accurate training dataset where every feature value reflects what was knowable at the moment of each observation.
A common interview question is: "Your fraud model has 99% accuracy in offline evaluation but only 72% in production. What would you investigate?" Point-in-Time correctness is the first thing to check. If the training pipeline did not use AS OF joins, the model was trained on leaked future data, producing artificially inflated offline metrics. Ask to see the training pipeline's join logic and verify that feature timestamps are strictly less than or equal to the observation timestamp.