A practical look at feature and history leakage, and how to test for it before your model or your report does.
A churn model trains at 94% accuracy, ships, and quietly underperforms for months. Nobody can find the bug because there isn't one, in the usual sense: the code runs, the pipeline completes, the numbers look plausible. The problem is a feature that was computed using information the model wouldn't have had at prediction time.
This is feature leakage, and it's the most common reason a model that tests well fails to perform in production. It's also one of the hardest bugs to catch, because it doesn't throw an error. It just makes your offline metrics lie to you.
Say you're predicting churn using "support tickets in the last 30 days" as a feature. If that feature is computed by querying the tickets table as it looks today, and a customer who churned in March had a ticket logged in April explaining why, that feature now contains information from after the event you're trying to predict. Your model just learned to use the future to predict the past. It will never be able to do that in production, because in production, April hasn't happened yet.
The fix isn't "be more careful with the query." It's structural: features need to be computed as of a specific timestamp, using only data that existed at that timestamp, every time, for every row in a training set.
Most homegrown feature pipelines are built around convenience: a scheduled job queries "current state" tables and writes a snapshot. That's fine for an AI learning pipeline, where "current state" is exactly what you want. It's wrong for a training set, where every row needs its own "as of" boundary, and those boundaries are different for every example.
Point-in-time correctness has to be a property of the feature store itself, not a discipline every analyst has to remember. That means every feature carries its own valid-time metadata, training and serving paths read from the same versioned store, and a validation step checks for leakage before a feature set is allowed to ship, the same way a manifest's other requirements get checked.
A useful test: pick any row in your training set and ask, "if I re-ran this exact query on the date this row is dated, would I get the same feature value?" If the honest answer is "I'm not sure," you likely have leakage somewhere in the pipeline.
Want the full reference, or ready to see it on your own data?
See the Feature and Training Product