Classic ML to Feature Store and Serving Blueprint
Centralize duplicated, drift-prone feature engineering for classic ML into a feature store that defines features once for offline training and online serving. Point-in-time-correct joins and parity monitoring eliminate training/serving skew.
What and Why
Classic ML teams (gradient boosting, logistic regression, and similar) often re-implement features per project, in notebooks for training and again in services for serving. The two implementations drift apart, causing training/serving skew and silent accuracy loss. A feature store defines features once, materializes them offline for training and online for low-latency serving, and tracks lineage.
This blueprint centralizes feature engineering so models share consistent, reusable, governed features.
Phases
Assessment. Inventory features across existing models, noting duplicates and inconsistent definitions. Identify which need online (low-latency) access versus offline-only.
Feature definition. Define features declaratively (entities, feature views, transformations) in a feature store such as Feast. Attach data contracts and ownership. Version definitions in git.
Offline store. Materialize features to an offline store (Parquet/warehouse) for point-in-time-correct training datasets, eliminating label leakage. Generate training sets via the store, not bespoke joins.
Online serving. Sync features to an online store (Redis) updated by streaming (Kafka) or batch jobs. Models at serving time read the exact same feature definitions.
Governance. Add lineage, freshness monitoring, and drift detection on features. Establish review for new features to prevent duplication.
Key Risks and Mitigations
- Training/serving skew: the core problem. The feature store's single definition for both paths, plus monitoring of feature distributions, keeps them aligned.
- Feature duplication: teams recreate existing features. Enforce a feature catalog with discovery and review before adding new ones.
- Data consistency: online and offline can diverge. Use point-in-time-correct joins offline and monitor online freshness and parity.
Recommended Tooling
Feast (or an equivalent feature store), an offline store on Parquet/warehouse, Redis as the online store, Kafka for streaming feature updates, DVC for dataset versioning, and drift/freshness monitoring with OpenTelemetry.
Success Metrics
Measure feature reuse across models, training/serving consistency (parity checks passing), time to add and ship a new feature, and model accuracy stability.
Prerequisites
Defined entities and data sources, an offline and online store, streaming or batch pipelines to materialize features, and ownership for the feature catalog.