Kafka + Flink Streaming Lakehouse

An end-to-end streaming lakehouse stack where Kafka ingests, Flink processes in real time, and Iceberg tables on S3 serve analytics. It unifies real-time pipelines and lakehouse analytics on open formats.

Kafka + Flink Streaming Lakehouse

This stack delivers an end-to-end streaming lakehouse where events flow from ingestion to analytical tables in near real time. Apache Kafka transports events, Apache Flink processes and enriches them with low latency, and the results land as Apache Iceberg tables on S3 that both streaming and batch engines can query. It unifies real-time pipelines with lakehouse analytics on open formats.

Components

Apache Kafka: The durable event backbone that ingests high-volume streams with replay and partitioned parallelism.
Apache Flink: A stateful stream processor with event-time semantics and exactly-once guarantees. It filters, joins, aggregates, and enriches events, then writes to lakehouse tables.
Apache Iceberg: An open table format on S3 providing ACID snapshots, schema evolution, and multi-engine reads, serving as the sink for processed data.
Amazon S3: Durable, cheap object storage for Iceberg data and metadata.
Query engines: Trino, Spark, or Flink query the resulting tables for analytics.

Strengths

Real-time to analytics. Events become queryable lakehouse tables with low latency.
Open formats. Iceberg avoids warehouse lock-in and enables multi-engine access.
Strong processing semantics. Flink's exactly-once and event-time handling ensure correctness.
Scalable backbone. Kafka and Flink scale horizontally for high-volume streams.

Trade-offs

Operational heft. Running Kafka, Flink, and Iceberg maintenance demands significant expertise.
Tuning. State sizing, checkpoints, and small-file compaction need ongoing attention.
Complexity for simple cases. Batch-only needs do not justify this much machinery.
Cost. Always-on streaming compute adds expense versus periodic batch.

When to Use It

Choose this stack when you need both real-time event processing and open lakehouse analytics over the same data, such as streaming ETL, live metrics, and event-driven analytics that also feed BI. It suits teams with streaming expertise and high-volume event flows. For purely batch or low-volume workloads, simpler stacks fit better. For unified real-time and analytical pipelines on open formats, Kafka plus Flink plus Iceberg is a powerful combination.

See a real scan run

A replay of the actual CLI running against our test repositories — live progress, real findings, a genuine DriftScore. Nothing executes in your browser.

Replay

demo@vibgrate — bash

❯ npx @vibgrate/cli scan

╭──────────────────────────────────────────╮

│ Vibgrate Drift Report │

╰──────────────────────────────────────────╯

── node-turborepo (node) .

Runtime: >=18.0.0 (6 majors behind)

Frameworks:

Turbo: 1.13.4 → 2.10.12 (1 behind)

TypeScript: 5.9.3 → 7.0.2 (2 behind)

Dependencies:

1 current 1 1-behind 3 2+ behind 1 unknown

── @repo/admin (node) apps/admin

Frameworks:

TanStack Query: 5.102.5 → 5.102.5 (current)

React: 18.3.1 → 19.2.8 (1 behind)

React DOM: 18.3.1 → 19.2.8 (1 behind)

TypeScript: 5.9.3 → 7.0.2 (2 behind)

Vite: 5.4.21 → 8.2.2 (3 behind)

Dependencies:

3 current 9 1-behind 3 2+ behind 4 unknown

── @repo/api (node) apps/api

Frameworks:

Express: 4.22.2 → 5.2.1 (1 behind)

TypeScript: 5.9.3 → 7.0.2 (2 behind)

Vitest: 1.6.1 → 4.1.11 (3 behind)

Dependencies:

7 current 5 1-behind 3 2+ behind 4 unknown

── @repo/web (node) apps/web

Frameworks:

Next.js: 14.2.35 → 16.3.3 (2 behind)

React: 18.3.1 → 19.2.8 (1 behind)

React DOM: 18.3.1 → 19.2.8 (1 behind)

TypeScript: 5.9.3 → 7.0.2 (2 behind)

Dependencies:

2 current 6 1-behind 3 2+ behind 5 unknown

── @repo/config (node) packages/config

Frameworks:

TypeScript: 5.9.3 → 7.0.2 (2 behind)

Dependencies:

2 current 2 1-behind 5 2+ behind 0 unknown

── @repo/database (node) packages/database

Frameworks:

Prisma: 5.22.0 → 7.10.0 (2 behind)

TypeScript: 5.9.3 → 7.0.2 (2 behind)

Dependencies:

1 current 0 1-behind 3 2+ behind 1 unknown

── @repo/types (node) packages/types

Frameworks:

TypeScript: 5.9.3 → 7.0.2 (2 behind)

Dependencies:

0 current 0 1-behind 1 2+ behind 1 unknown

── @repo/ui (node) packages/ui

Frameworks:

React: 18.3.1 → 19.2.8 (1 behind)

TypeScript: 5.9.3 → 7.0.2 (2 behind)

React: 18.3.1 → 19.2.8 (1 behind)

Dependencies:

1 current 4 1-behind 1 2+ behind 1 unknown

── @repo/utils (node) packages/utils

Frameworks:

TypeScript: 5.9.3 → 7.0.2 (2 behind)

Vitest: 1.6.1 → 4.1.11 (3 behind)

Dependencies:

0 current 1 1-behind 2 2+ behind 1 unknown

Tech Stack

Frontend: React, React DOM

Meta-frameworks: Next.js

Bundlers: tsx, Turbo, Vite

CSS / UI: Autoprefixer, PostCSS, Tailwind CSS

Backend: Express

ORM / Database: Prisma, Prisma Client

Testing: Vitest

Lint & Format: ESLint, ESLint Prettier, ESLint React, Prettier, typescript-eslint

Services & Integrations

Auth: JWT 9.0.3

Databases: Prisma 5.22.0

TypeScript

v5.3.3 · strict ✔ · MIXED · target: ES2022

Build & Deploy

Package Managers: pnpm

Monorepo: npm-workspaces, pnpm-workspaces, turbo

Product Purpose Signals

Frameworks: react, nextjs

Evidence: 177

Top Signals:

- [heading] Dashboard (apps/admin/src/pages/Dashboard.tsx)

- [title] Revenue Overview (apps/admin/src/pages/Dashboard.tsx)

- [copy] workspace:* (packages/ui/package.json)

- [copy] ./dist (packages/ui/tsconfig.json)

- [copy] ./src/index.ts (packages/ui/package.json)

- [copy] @repo/config/tsconfig-base.json (packages/ui/tsconfig.json)

- [copy] @repo/ui (packages/ui/package.json)

- [copy] #3b82f6 (apps/admin/src/pages/Dashboard.tsx)

Unknowns:

- No pricing or billing evidence found.

- No integrations/connectors evidence found.

- No route structure evidence found.

Security Posture

Lockfile ✖ · .env ✔ · node_modules ✔

Platform

Native modules: turbo

Code Quality

Files: 36 · Functions: 183 · Avg complexity: 2.62 · Avg length: 21.13 lines

Max nesting: 2 · Circular deps: 0 · Dead code: 0%

God files: apps/admin/src/pages/Products (448 lines)

Database Schema

postgresql · 8 models · 1 enum

Models: Address, CartItem, Category, Order, OrderItem (+3 more)

Findings (16 errors, 11 warnings)

✖ Node.js runtime ">=18.0.0" reached end-of-life on 2025-04-30 (latest: 24.0.0).

vibgrate/runtime-eol in .

⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).

vibgrate/framework-major-lag in .

✖ 60% of dependencies are 2+ major versions behind in node-turborepo.

vibgrate/dependency-rot in .

✖ @types/node is 6 major versions behind (spec: ^20.11.0, latest: 26.3.0).

vibgrate/dependency-major-lag in .

⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).

vibgrate/framework-major-lag in apps/admin

✖ Vite is 3 major versions behind (current: 5.4.21, latest: 8.2.2).

vibgrate/framework-major-lag in apps/admin

✖ vite is 3 major versions behind (spec: ^5.0.12, latest: 8.2.2).

vibgrate/dependency-major-lag in apps/admin

⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).

vibgrate/framework-major-lag in apps/api

✖ Vitest is 3 major versions behind (current: 1.6.1, latest: 4.1.11).

vibgrate/framework-major-lag in apps/api

✖ @types/node is 6 major versions behind (spec: ^20.11.0, latest: 26.3.0).

vibgrate/dependency-major-lag in apps/api

✖ vitest is 3 major versions behind (spec: ^1.2.1, latest: 4.1.11).

vibgrate/dependency-major-lag in apps/api

⚠ Next.js is 2 major versions behind (current: 14.2.35, latest: 16.3.3).

vibgrate/framework-major-lag in apps/web

⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).

vibgrate/framework-major-lag in apps/web

✖ @types/node is 6 major versions behind (spec: ^20.11.0, latest: 26.3.0).

vibgrate/dependency-major-lag in apps/web

⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).

vibgrate/framework-major-lag in packages/config

✖ 56% of dependencies are 2+ major versions behind in @repo/config.

vibgrate/dependency-rot in packages/config

✖ eslint-plugin-react-hooks is 3 major versions behind (spec: ^4.6.0, latest: 7.1.1).

vibgrate/dependency-major-lag in packages/config

⚠ Prisma is 2 major versions behind (current: 5.22.0, latest: 7.10.0).

vibgrate/framework-major-lag in packages/database

⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).

vibgrate/framework-major-lag in packages/database

✖ 75% of dependencies are 2+ major versions behind in @repo/database.

vibgrate/dependency-rot in packages/database

⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).

vibgrate/framework-major-lag in packages/types

✖ 100% of dependencies are 2+ major versions behind in @repo/types.

vibgrate/dependency-rot in packages/types

⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).

vibgrate/framework-major-lag in packages/ui

⚠ TypeScript is 2 major versions behind (current: 5.9.3, latest: 7.0.2).

vibgrate/framework-major-lag in packages/utils

✖ Vitest is 3 major versions behind (current: 1.6.1, latest: 4.1.11).

vibgrate/framework-major-lag in packages/utils

✖ 67% of dependencies are 2+ major versions behind in @repo/utils.

vibgrate/dependency-rot in packages/utils

✖ vitest is 3 major versions behind (spec: ^1.2.1, latest: 4.1.11).

vibgrate/dependency-major-lag in packages/utils

╭──────────────────────────────────────────╮

│ Top Priority Actions │

╰──────────────────────────────────────────╯

1. Upgrade EOL runtime in node-turborepo

End-of-life runtimes no longer receive security patches and block ecosystem upgrades.

./.

>=18.0.0 → 24.0.0 (6 majors behind)

Impact: −10 drift points (runtime & EOL)

2. Fix security posture: no lockfile found

Without a lockfile, installs are non-deterministic. Run the install command to generate one and commit it.

Missing: package-lock.json, pnpm-lock.yaml, or yarn.lock

3. Upgrade Vite 5.4.21 → 8.2.2 in @repo/admin (+2 more)

3 major versions behind. Major framework drift increases breaking change risk and blocks access to security fixes and performance improvements.

./apps/admin

Vite: 5.4.21 → 8.2.2 (3 majors behind)

./apps/api

Vitest: 1.6.1 → 4.1.11 (3 majors behind)

./packages/utils

Vitest: 1.6.1 → 4.1.11 (3 majors behind)

Impact: −5–15 drift points

4. Reduce dependency rot in @repo/types (100% severely outdated)

1 of 1 dependencies are 2+ majors behind. Run `npm outdated` and prioritise packages with known CVEs or breaking API changes.

./packages/types

typescript: 5.9.3 → 7.0.2 (2 majors behind)

Impact: −5–10 drift points

5. Reduce dependency rot in @repo/database (75% severely outdated)

3 of 4 dependencies are 2+ majors behind. Run `npm outdated` and prioritise packages with known CVEs or breaking API changes.

./packages/database

@prisma/client: 5.22.0 → 7.10.0 (2 majors behind)

prisma: 5.22.0 → 7.10.0 (2 majors behind)

typescript: 5.9.3 → 7.0.2 (2 majors behind)

Impact: −5–10 drift points

╭──────────────────────────────────────────╮

│ Architecture Layers │

╰──────────────────────────────────────────╯

Archetype: nextjs (80% confidence)

Files classified: 24 (11 unclassified)

Folders classified: 8

apps/admin/src presentation 100% 4 files

apps/admin/src/pages presentation 100% 2 files

apps/api/src/middleware middleware 100% 2 files

apps/api/src/routes routing 100% 2 files

apps/web/src/app presentation 100% 4 files

apps/web/src/app/products presentation 100% 2 files

apps/web/src/app/products/[id] presentation 100% 1 file

packages/ui/src presentation 100% 6 files

Unclassified source (sample): 11

presentation 15 files drift ████████████████████ 100 risk high

routing 4 files drift ████████████████████ 100 risk high

middleware 2 files drift ███████▍░░░░░░░░░░░░ 37 risk moderate

config 2 files drift ░░░░░░░░░░░░░░░░░░░░ 0 risk none

shared 1 file drift ████████████████████ 100 risk high

╭──────────────────────────────────────────╮

│ DriftScore Summary │

╰──────────────────────────────────────────╯

DriftScore: 66/100

Risk Level: HIGH

Projects: 9

Classified: 8 nano · 1 micro · 0 small · 0 standard

Billable: 0.42 · 9 detected → 0.42 billable projects (micro-project pricing)

0.1 micro · 0.32 nano

These fractions add up across repositories, then round down to whole billable projects.

Score Breakdown

Runtime: ████████████████████ 100

Frameworks: █████████▏░░░░░░░░░░ 46

Dependencies: ██████▏░░░░░░░░░░░░░ 31

EOL Risk: ████████████████████ 100

Scanned at 2026-08-26T09:08:28.481Z · 7.1s · 286 files scanned · 56 workspace files · 27 dirs

❯

Press Run to start.

Explore the Vibgrate CLI

Kafka + Flink Streaming Lakehouse