Batch to Streaming with Apache Kafka Blueprint

A blueprint to replace nightly batch ETL with Apache Kafka streaming pipelines using Kafka Connect/Debezium, schema registry contracts, and Streams/Flink processing, validated by a parallel run.

From

Batch

Streaming

Difficulty

Advanced

Duration

16 weeks

Team Size

medium

Overview

Batch ETL runs on a schedule, so downstream data is always hours stale and a failed batch delays everything. Apache Kafka is a distributed, durable log that moves events continuously, enabling real-time pipelines, event-driven services, and stream processing. This blueprint converts scheduled batch jobs into streaming pipelines.

Phases

Assessment. Identify batch jobs that benefit from freshness (fraud, inventory, analytics) versus those fine as batch. Map sources, sinks, and current volumes. Decide processing semantics needed: at-least-once versus exactly-once.

Topic design. Model events as topics with partition keys chosen for ordering and parallelism. Define schemas in Avro or JSON Schema and register them in a schema registry to enforce compatibility as events evolve. Set retention and compaction policies.

Pipeline build. Use Kafka Connect for source/sink connectors (databases via Debezium CDC, object stores, warehouses). Build transformations with Kafka Streams or Apache Flink. Apply data contracts so producers and consumers agree on shape.

Parallel run. Run streaming alongside batch and reconcile outputs to prove correctness before retiring batch.

Cutover. Switch downstream consumers to the streaming outputs, then decommission batch jobs.

Key Risks and Mitigations

Data consistency: Use exactly-once semantics in Kafka Streams/Flink and idempotent sinks; reconcile against batch during parallel run.
Ordering guarantees: Kafka orders within a partition only. Choose partition keys carefully so related events stay ordered.
Skills gap: Streaming requires new operational skills (consumer lag, rebalancing). Train teams and add lag monitoring before cutover.

Recommended Tooling

Apache Kafka with a schema registry; Kafka Connect and Debezium for CDC sources; Kafka Streams or Apache Flink for processing; consumer-lag and throughput dashboards in Prometheus/Grafana.

Success Metrics

Lower end-to-end latency (seconds versus hours), higher sustained throughput, and improved data freshness for downstream consumers.

Prerequisites

A Kafka cluster, a schema registry, defined data contracts, and a parallel-run reconciliation plan against existing batch outputs.

See a real scan run

A replay of the actual CLI running against our test repositories — live progress, real findings, a genuine DriftScore. Nothing executes in your browser.

Replay

demo@vibgrate — bash

❯ npx @vibgrate/cli scan

╭──────────────────────────────────────────╮

│ Vibgrate Drift Report │

╰──────────────────────────────────────────╯

── node-turborepo (node) .

Runtime: >=18.0.0 (6 majors behind)

Frameworks:

Turbo: 1.13.4 → 2.10.12 (1 behind)

TypeScript: 5.9.3 → 7.0.2 (2 behind)

Dependencies:

1 current 1 1-behind 3 2+ behind 1 unknown

── @repo/admin (node) apps/admin

Frameworks:

TanStack Query: 5.102.5 → 5.102.5 (current)

React: 18.3.1 → 19.2.8 (1 behind)

React DOM: 18.3.1 → 19.2.8 (1 behind)

TypeScript: 5.9.3 → 7.0.2 (2 behind)

Vite: 5.4.21 → 8.2.2 (3 behind)

Dependencies:

3 current 9 1-behind 3 2+ behind 4 unknown

── @repo/api (node) apps/api

Frameworks:

Express: 4.22.2 → 5.2.1 (1 behind)

TypeScript: 5.9.3 → 7.0.2 (2 behind)

Vitest: 1.6.1 → 4.1.11 (3 behind)

Dependencies:

7 current 5 1-behind 3 2+ behind 4 unknown

── @repo/web (node) apps/web

Frameworks:

Next.js: 14.2.35 → 16.3.3 (2 behind)

React: 18.3.1 → 19.2.8 (1 behind)

React DOM: 18.3.1 → 19.2.8 (1 behind)

TypeScript: 5.9.3 → 7.0.2 (2 behind)

Dependencies:

2 current 6 1-behind 3 2+ behind 5 unknown

── @repo/config (node) packages/config

Frameworks:

TypeScript: 5.9.3 → 7.0.2 (2 behind)

Dependencies:

2 current 2 1-behind 5 2+ behind 0 unknown

── @repo/database (node) packages/database

Frameworks:

Prisma: 5.22.0 → 7.10.0 (2 behind)

TypeScript: 5.9.3 → 7.0.2 (2 behind)

Dependencies:

1 current 0 1-behind 3 2+ behind 1 unknown

── @repo/types (node) packages/types

Frameworks:

TypeScript: 5.9.3 → 7.0.2 (2 behind)

Dependencies:

0 current 0 1-behind 1 2+ behind 1 unknown

── @repo/ui (node) packages/ui

Frameworks:

React: 18.3.1 → 19.2.8 (1 behind)

TypeScript: 5.9.3 → 7.0.2 (2 behind)

React: 18.3.1 → 19.2.8 (1 behind)

Dependencies:

1 current 4 1-behind 1 2+ behind 1 unknown

── @repo/utils (node) packages/utils

Frameworks:

TypeScript: 5.9.3 → 7.0.2 (2 behind)

Vitest: 1.6.1 → 4.1.11 (3 behind)

Dependencies:

0 current 1 1-behind 2 2+ behind 1 unknown

Tech Stack

Frontend: React, React DOM

Meta-frameworks: Next.js

Bundlers: tsx, Turbo, Vite

CSS / UI: Autoprefixer, PostCSS, Tailwind CSS

Backend: Express

ORM / Database: Prisma, Prisma Client

Testing: Vitest

Lint & Format: ESLint, ESLint Prettier, ESLint React, Prettier, typescript-eslint

Services & Integrations

Auth: JWT 9.0.3

Databases: Prisma 5.22.0

TypeScript

v5.3.3 · strict ✔ · MIXED · target: ES2022

Build & Deploy

Package Managers: pnpm

Monorepo: npm-workspaces, pnpm-workspaces, turbo

Product Purpose Signals

Frameworks: react, nextjs

Evidence: 177

Top Signals:

- [heading] Dashboard (apps/admin/src/pages/Dashboard.tsx)

- [title] Revenue Overview (apps/admin/src/pages/Dashboard.tsx)

- [copy] workspace:* (packages/ui/package.json)

- [copy] ./dist (packages/ui/tsconfig.json)

- [copy] ./src/index.ts (packages/ui/package.json)

- [copy] @repo/config/tsconfig-base.json (packages/ui/tsconfig.json)

- [copy] @repo/ui (packages/ui/package.json)

- [copy] #3b82f6 (apps/admin/src/pages/Dashboard.tsx)

Unknowns:

- No pricing or billing evidence found.

- No integrations/connectors evidence found.

- No route structure evidence found.

Security Posture

Lockfile ✖ · .env ✔ · node_modules ✔

Platform

Native modules: turbo

Code Quality

Files: 36 · Functions: 183 · Avg complexity: 2.62 · Avg length: 21.13 lines

Max nesting: 2 · Circular deps: 0 · Dead code: 0%

God files: apps/admin/src/pages/Products (448 lines)

Database Schema

postgresql · 8 models · 1 enum

Models: Address, CartItem, Category, Order, OrderItem (+3 more)

Findings (16 errors, 11 warnings)

✖ Node.js runtime ">=18.0.0" reached end-of-life on 2025-04-30 (latest: 24.0.0).

vibgrate/runtime-eol in .

⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).

vibgrate/framework-major-lag in .

✖ 60% of dependencies are 2+ major versions behind in node-turborepo.

vibgrate/dependency-rot in .

✖ @types/node is 6 major versions behind (spec: ^20.11.0, latest: 26.3.0).

vibgrate/dependency-major-lag in .

⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).

vibgrate/framework-major-lag in apps/admin

✖ Vite is 3 major versions behind (current: 5.4.21, latest: 8.2.2).

vibgrate/framework-major-lag in apps/admin

✖ vite is 3 major versions behind (spec: ^5.0.12, latest: 8.2.2).

vibgrate/dependency-major-lag in apps/admin

⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).

vibgrate/framework-major-lag in apps/api

✖ Vitest is 3 major versions behind (current: 1.6.1, latest: 4.1.11).

vibgrate/framework-major-lag in apps/api

✖ @types/node is 6 major versions behind (spec: ^20.11.0, latest: 26.3.0).

vibgrate/dependency-major-lag in apps/api

✖ vitest is 3 major versions behind (spec: ^1.2.1, latest: 4.1.11).

vibgrate/dependency-major-lag in apps/api

⚠ Next.js is 2 major versions behind (current: 14.2.35, latest: 16.3.3).

vibgrate/framework-major-lag in apps/web

⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).

vibgrate/framework-major-lag in apps/web

✖ @types/node is 6 major versions behind (spec: ^20.11.0, latest: 26.3.0).

vibgrate/dependency-major-lag in apps/web

⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).

vibgrate/framework-major-lag in packages/config

✖ 56% of dependencies are 2+ major versions behind in @repo/config.

vibgrate/dependency-rot in packages/config

✖ eslint-plugin-react-hooks is 3 major versions behind (spec: ^4.6.0, latest: 7.1.1).

vibgrate/dependency-major-lag in packages/config

⚠ Prisma is 2 major versions behind (current: 5.22.0, latest: 7.10.0).

vibgrate/framework-major-lag in packages/database

⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).

vibgrate/framework-major-lag in packages/database

✖ 75% of dependencies are 2+ major versions behind in @repo/database.

vibgrate/dependency-rot in packages/database

⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).

vibgrate/framework-major-lag in packages/types

✖ 100% of dependencies are 2+ major versions behind in @repo/types.

vibgrate/dependency-rot in packages/types

⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).

vibgrate/framework-major-lag in packages/ui

⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).

vibgrate/framework-major-lag in packages/utils

✖ Vitest is 3 major versions behind (current: 1.6.1, latest: 4.1.11).

vibgrate/framework-major-lag in packages/utils

✖ 67% of dependencies are 2+ major versions behind in @repo/utils.

vibgrate/dependency-rot in packages/utils

✖ vitest is 3 major versions behind (spec: ^1.2.1, latest: 4.1.11).

vibgrate/dependency-major-lag in packages/utils

╭──────────────────────────────────────────╮

│ Top Priority Actions │

╰──────────────────────────────────────────╯

1. Upgrade EOL runtime in node-turborepo

End-of-life runtimes no longer receive security patches and block ecosystem upgrades.

./.

>=18.0.0 → 24.0.0 (6 majors behind)

Impact: −10 drift points (runtime & EOL)

2. Fix security posture: no lockfile found

Without a lockfile, installs are non-deterministic. Run the install command to generate one and commit it.

Missing: package-lock.json, pnpm-lock.yaml, or yarn.lock

3. Upgrade Vite 5.4.21 → 8.2.2 in @repo/admin (+2 more)

3 major versions behind. Major framework drift increases breaking change risk and blocks access to security fixes and performance improvements.

./apps/admin

Vite: 5.4.21 → 8.2.2 (3 majors behind)

./apps/api

Vitest: 1.6.1 → 4.1.11 (3 majors behind)

./packages/utils

Vitest: 1.6.1 → 4.1.11 (3 majors behind)

Impact: −5–15 drift points

4. Reduce dependency rot in @repo/types (100% severely outdated)

1 of 1 dependencies are 2+ majors behind. Run `npm outdated` and prioritise packages with known CVEs or breaking API changes.

./packages/types

typescript: 5.9.3 → 7.0.2 (2 majors behind)

Impact: −5–10 drift points

5. Reduce dependency rot in @repo/database (75% severely outdated)

3 of 4 dependencies are 2+ majors behind. Run `npm outdated` and prioritise packages with known CVEs or breaking API changes.

./packages/database

@prisma/client: 5.22.0 → 7.10.0 (2 majors behind)

prisma: 5.22.0 → 7.10.0 (2 majors behind)

typescript: 5.9.3 → 7.0.2 (2 majors behind)

Impact: −5–10 drift points

╭──────────────────────────────────────────╮

│ Architecture Layers │

╰──────────────────────────────────────────╯

Archetype: nextjs (80% confidence)

Files classified: 24 (11 unclassified)

Folders classified: 8

apps/admin/src presentation 100% 4 files

apps/admin/src/pages presentation 100% 2 files

apps/api/src/middleware middleware 100% 2 files

apps/api/src/routes routing 100% 2 files

apps/web/src/app presentation 100% 4 files

apps/web/src/app/products presentation 100% 2 files

apps/web/src/app/products/[id] presentation 100% 1 file

packages/ui/src presentation 100% 6 files

Unclassified source (sample): 11

presentation 15 files drift ████████████████████ 100 risk high

routing 4 files drift ████████████████████ 100 risk high

middleware 2 files drift ███████▍░░░░░░░░░░░░ 37 risk moderate

config 2 files drift ░░░░░░░░░░░░░░░░░░░░ 0 risk none

shared 1 file drift ████████████████████ 100 risk high

╭──────────────────────────────────────────╮

│ DriftScore Summary │

╰──────────────────────────────────────────╯

DriftScore: 66/100

Risk Level: HIGH

Projects: 9

Classified: 8 nano · 1 micro · 0 small · 0 standard

Billable: 0.42 · 9 detected → 0.42 billable projects (micro-project pricing)

0.1 micro · 0.32 nano

These fractions add up across repositories, then round down to whole billable projects.

Score Breakdown

Runtime: ████████████████████ 100

Frameworks: █████████▏░░░░░░░░░░ 46

Dependencies: ██████▏░░░░░░░░░░░░░ 31

EOL Risk: ████████████████████ 100

Scanned at 2026-08-26T09:08:28.481Z · 7.1s · 286 files scanned · 56 workspace files · 27 dirs

❯

Press Run to start.

Explore the Vibgrate CLI

Batch to Streaming with Apache Kafka Blueprint

Overview

Phases

Key Risks and Mitigations

Recommended Tooling

Success Metrics

Prerequisites

Tags

See a real scan run