Orchestration
17 items tagged with "orchestration"
Tutorials2
How to orchestrate workflows with AWS Step Functions
Build a state machine with AWS Step Functions to orchestrate Lambda tasks with retries, branching, and error handling.
How to author a data pipeline DAG in Apache Airflow
Write an Airflow DAG with scheduled tasks and dependencies, run it locally, and monitor execution in the UI.
Stacks4
Airflow + Spark Batch Pipeline Stack
Classic big-data batch pattern: Apache Airflow orchestrates scheduled DAGs that run distributed Apache Spark jobs over data-lake storage.
LLM Agent Stack
Agentic AI stack: an orchestration framework drives an LLM to reason, call tools and APIs, and use vector memory to complete multi-step tasks autonomously.
Dagster + dbt
An asset-oriented data orchestration stack where Dagster schedules and observes pipelines built around dbt's transformation models.
Prefect + Spark
A Python-native orchestration stack where Prefect schedules and monitors large-scale data processing jobs running on Apache Spark.
FAQs5
What is Kubernetes?
Kubernetes is an open-source platform for automating the deployment, scaling, and operation of containerized applications. It groups containers into l...
What is the difference between a Kubernetes pod and a container?
A container is a single packaged process, while a pod is the smallest deployable unit in Kubernetes and can hold one or more containers that share the...
When should I use Kubernetes?
Kubernetes makes sense when you run many containerized services that need automated scaling, self-healing, rolling updates, and consistent deployment ...
What is container orchestration?
Container orchestration is the automated management of the lifecycle of containers across a cluster of machines, including scheduling, scaling, networ...
What is a data pipeline?
A data pipeline is a series of automated steps that move data from sources to destinations, transforming and validating it along the way so it is read...