Streaming
52 items tagged with "streaming"
Patterns4
Change Data Capture (CDC)
Captures row-level changes from a database's transaction log and streams them as events to downstream consumers.
Lambda Architecture
A big-data design that runs a batch layer for accurate historical views and a speed layer for low-latency recent data, merging both at query time.
Kappa Architecture
A streaming-first data design that uses a single processing path over an immutable log, reprocessing history by replay instead of a separate batch layer.
Backpressure
Lets a slow consumer signal upstream producers to slow down, preventing unbounded queues and memory exhaustion when demand exceeds processing capacity.
Tutorials4
Building Event-Driven Systems with Kafka
Create scalable event-driven architectures using Apache Kafka
How to build a Kafka producer and consumer
Create a Kafka topic, produce messages with keys, and consume them in a group, covering offsets, partitions, and delivery basics.
How to load Kafka topics into a database with Kafka Connect
Use a Kafka Connect sink connector to stream topic data into a relational database with no custom consumer code.
How to Build a gRPC Service
Define a service in Protocol Buffers, generate code, implement unary and streaming methods, and call the service from a client.
Blueprints8
Batch to Streaming Blueprint
Migration blueprint for converting batch processing to real-time streaming
Synchronous REST to Event-Driven Blueprint
Convert tightly-coupled synchronous REST call chains into an event-driven architecture using a broker, async messaging, and the outbox pattern.
Batch to Streaming with Apache Kafka Blueprint
Replace nightly batch ETL with real-time event streaming on Apache Kafka for low-latency data movement.
Batch to Streaming with Amazon Kinesis Blueprint
Convert scheduled batch pipelines to real-time streaming on Amazon Kinesis Data Streams for managed, low-latency ingestion.
Build a Change Data Capture (CDC) Pipeline Blueprint
Stand up a CDC pipeline with Debezium and Kafka to stream database changes in real time to downstream consumers.
Synchronous Calls to Event-Driven (Kafka) Blueprint
Replace brittle synchronous service-to-service calls with asynchronous events on Apache Kafka to decouple services and improve resilience.
Webhooks to Event Streaming Blueprint
Evolve unreliable HTTP webhook integrations into a durable event-streaming backbone with replay, ordering, and at-least-once delivery.
Batch File Transfer to Streaming CDC Blueprint
Replace nightly batch file transfers between systems with real-time change data capture streamed onto an event backbone.
Products2
Reference Architectures13
Serverless Event Processing Pipeline on AWS
A fully serverless event-ingestion and processing pipeline on AWS using Lambda, EventBridge, and DynamoDB with no servers to operate.
Real-Time Streaming Platform with Kafka
A real-time streaming platform on Kafka with stream processing, a schema registry, and exactly-once pipelines on Kubernetes.
Event Streaming on Amazon Kinesis
A managed event streaming platform on Amazon Kinesis with serverless processing and real-time analytics on AWS.
Streaming Lakehouse with Flink and Iceberg
A streaming lakehouse on AWS where Apache Flink writes continuously into Apache Iceberg tables for unified streaming and batch analytics.
IoT Time-Series Analytics Platform
An IoT analytics platform on Azure ingesting device telemetry into a time-series store for real-time monitoring and forecasting.
Feature Store and Online Serving on AWS
A reference design for a dual offline/online feature store on AWS that keeps training and serving features consistent and serves them at low latency.
Recommendation System on AWS
A reference design for a two-stage recommender on AWS combining candidate retrieval and ranking, with streaming feedback and real-time serving.
Real-Time Fraud Detection on GCP
A reference design for streaming fraud detection on GCP that scores transactions in milliseconds using a feature store, rules, and an ML model.
Streaming ML Feature Pipeline on GCP
A reference design for a real-time feature engineering pipeline on GCP that computes streaming aggregates and serves them to online models consistently.
Event-Driven Backbone with Kafka
An organization-wide event streaming backbone on Kafka that decouples producers and consumers through durable, replayable topics.
High-Volume Streaming Ingestion Pipeline
A streaming ingestion pipeline that captures high-velocity event data, buffers it durably, and lands it for real-time and batch use.
Change Data Capture Streaming Pipeline
A change-data-capture pipeline that streams database changes as events to keep downstream systems and caches in sync in near real time.
IoT Ingestion Platform
A platform that ingests telemetry from large device fleets over MQTT, processes it as a stream, and stores it for analytics and control.
Playbooks7
Event-Driven Architecture Adoption Program Playbook
A program to introduce event-driven communication, a schema registry, and async workflows into a synchronous backend estate.
Batch to Streaming Program Playbook
Transform nightly batch data pipelines into real-time streaming pipelines using event-driven architecture and stream processing.
CDC and Real-Time Pipeline Program Playbook
Build change data capture pipelines to stream database changes into analytics and event systems with low latency and strong consistency.
Schema Evolution and Versioning Program Playbook
Adopt a schema registry and safe evolution practices for event and data schemas to prevent breaking changes across producers and consumers.
Real-Time Inference Program Playbook
A program to deliver low-latency, high-throughput real-time ML inference with autoscaling, feature freshness, and strict SLOs.
Event-Driven Architecture Adoption Playbook
A program to adopt event-driven architecture with a streaming backbone, schema governance, and resilient async patterns.
Feature Store Rollout Playbook
A program to roll out a feature store that unifies feature engineering for training and serving with consistency and reuse.
Checklists4
Event-Driven Architecture Readiness Checklist
Confirm readiness to adopt event-driven communication between services, covering schemas, delivery semantics, and observability.
Streaming Pipeline Readiness Checklist
Production-readiness checks for real-time streaming data pipelines built on platforms such as Apache Kafka or Pulsar.
Data Pipeline Observability Checklist
Checks for instrumenting data pipelines with freshness, volume, schema, lineage, and quality monitoring to detect issues early.
Message Broker Migration Checklist
Migration items for moving messaging workloads to a new broker while preserving delivery guarantees and ordering.
Stacks2
Kafka + Flink + Iceberg Streaming Stack
Real-time streaming architecture: Kafka transports events, Flink processes them with stateful low-latency compute, and Iceberg lands them in an open lakehouse.
Apache Pinot Real-Time Analytics
A user-facing analytics stack built on Apache Pinot for ultra-low-latency, high-throughput queries over fresh streaming data.
Comparisons4
Kafka vs RabbitMQ
Distributed event-streaming log versus a flexible, feature-rich message broker for traditional queuing.
Kafka vs Pulsar
Mature event-streaming standard versus Apache Pulsar's segmented storage and built-in multi-tenancy.
Spark vs Flink
The unified batch-and-streaming engine with micro-batch roots versus Apache Flink's true stream-first processing engine.
Pulsar vs Kinesis
Open-source, self-hostable streaming with built-in multi-tenancy versus AWS Kinesis, a fully managed streaming service on AWS.
FAQs3
What is change data capture (CDC)?
Change data capture is a technique that detects and streams row-level changes (inserts, updates, deletes) from a source database so downstream systems...
What is a data pipeline?
A data pipeline is a series of automated steps that move data from sources to destinations, transforming and validating it along the way so it is read...
What is the difference between batch and stream processing?
Batch processing handles large volumes of data in scheduled chunks, such as a nightly job that aggregates the previous day's sales; it is simple and e...