Data Mesh Platform

Data Mesh Platform

Data mesh treats data as a product owned by the domain teams that know it best, rather than funneling everything through a central data team. Four principles guide it: domain ownership, data as a product, a self-serve data platform, and federated computational governance. This platform spans multiple clouds with open table formats so domains can publish interoperable data products. Use it in large organizations where a central team has become a bottleneck and domains need autonomy.

Components

• Data products: discoverable, documented datasets owned by domains, each with a clear interface and SLAs.
• Object storage: per-cloud buckets (S3, GCS, ADLS) holding data product tables.
• Apache Iceberg: an open table format giving ACID tables portable across engines and clouds.
• Trino: a federated query engine that joins data products across domains and clouds.
• Kafka: the event backbone for streaming data products and inter-domain integration.
• Data catalog: central discovery, schemas, ownership, and lineage for all products.
• Open Policy Agent (OPA): federated access policy enforced consistently across domains.

Data Flow

Each domain ingests and transforms its own source data, publishing curated Iceberg tables or Kafka topics as versioned data products described by data contracts. The catalog registers every product with its schema, owner, and quality metrics. Consumers discover products in the catalog and query them through Trino, which federates across storage and clouds without copying data. Cross-domain compositions become new data products with their own contracts.

Scaling and Resilience

The mesh scales organizationally by adding domains without overloading a central team. Trino scales horizontally with worker nodes, and Iceberg's metadata layer handles large tables with partition pruning and snapshot isolation. Because storage is open and per-cloud, a single cloud or domain outage isolates impact. Snapshots and contract versioning let consumers pin stable versions while producers iterate.

Security

Federated governance encodes policies as code in OPA so access, classification, and retention are enforced uniformly across domains. Each cloud's IAM secures storage, and the catalog tags sensitive fields to drive masking. Data contracts make schema and quality guarantees explicit, reducing breaking changes. Audit logs from the catalog and query engine support compliance.

Trade-offs and Alternatives

Data mesh improves scalability and domain ownership but raises coordination, tooling, and platform-engineering cost; without strong governance and a real self-serve platform it can fragment into silos. It suits large, mature organizations more than small teams, for whom a centralized warehouse or lakehouse is simpler. Open formats like Iceberg and engines like Trino keep the mesh vendor-neutral. Choose data mesh when domain autonomy and cross-cloud interoperability outweigh the added platform investment.