Streaming ML Feature Pipeline on GCP
A real-time feature engineering pipeline on GCP using Pub/Sub and Dataflow to compute streaming aggregates served from Bigtable and stored in BigQuery. It keeps training and serving consistent via a feature store and tracks freshness as the key SLO.
Overview
Many models need features that reflect the last few seconds or minutes of behavior, such as recent click counts, rolling spend, or session activity. A streaming feature pipeline computes these aggregates continuously and serves them to online models, while writing the same definitions to an offline store for training. Use this architecture when freshness materially improves predictions and stale, batch-computed features are not good enough.
Components
- Pub/Sub: ingests the raw event stream from applications.
- Dataflow: runs windowed, stateful stream processing to compute aggregates with exactly-once semantics.
- Bigtable: the online feature store for millisecond serving lookups.
- BigQuery: the offline store of feature history for training and analysis.
- Dataproc: optional heavy batch backfills and reprocessing.
- Vertex AI Feature Store: registers feature definitions so training and serving stay consistent.
- Cloud Monitoring: tracks freshness, lag, and pipeline health.
Data Flow
Events arrive on Pub/Sub. Dataflow groups them into sliding or tumbling windows per entity, computes aggregates, and upserts results to Bigtable for serving and to BigQuery for history. The same feature definitions are registered in the feature store so a training job and a serving call use identical logic. When logic changes, Dataproc backfills historical features from the event log so training data reflects the new definition.
Scaling and Resilience
Dataflow autoscales workers with stream volume and uses checkpoints so it can recover without losing or double-counting events. Pub/Sub buffers spikes and replays events for reprocessing. Watermarks handle late-arriving data gracefully. Monitor end-to-end lag as the key SLO, since stale features silently degrade model quality. Keep the event log as the durable source of truth so any feature can be recomputed from scratch.
Security
Apply data contracts between event producers and the pipeline so a schema change cannot silently break features. Encrypt data in transit and at rest, and restrict feature access by team. Validate and quarantine malformed events rather than letting them corrupt aggregates. Track lineage from event to feature for audit and debugging. Monitor null rates and value ranges as data-quality guards that catch upstream breakage early.
Trade-offs and Alternatives
Streaming features add real-time value but cost more and are harder to operate than batch features; reserve them for signals where freshness changes outcomes. Exactly-once stream processing is powerful but complex; some use cases tolerate at-least-once with idempotent writes for simpler operations. Maintaining matching batch and streaming logic risks skew, which a unified definition in the feature store mitigates. For mostly-stable features, a daily batch job is cheaper and entirely adequate.