Unified orchestration engine

The problem

Orchestration for the organization had grown by accretion: Airflow for batch, Step Functions for some AWS-native flows, Dagster for a newer slice of the estate. None of them was wrong; together they were expensive. Three scheduling systems meant three mental models, three retry semantics, three places to look when something did not run, and three upgrade paths to maintain. On-call answered the question “why didn’t last night’s jobs finish?” across three consoles.

The approach

I built a Rust job scheduler deployed as a service on ECS. Its purpose was to consolidate scheduling previously spread across Airflow, Step Functions, and Dagster. The result was one operational interface with consistent alerting and error handling, rather than separate conventions for each orchestration tool.

The important boundary in this design is between executing a workload and operating it. A transformation can remain SQL or Python while the scheduling system decides when it runs and provides the operational context around it. That distinction lets orchestration change without making every underlying data transformation a rewrite.

A shared scheduler also makes consistency an explicit engineering responsibility. With several engines, each tool supplies its own behavior and the platform team has to reconcile the differences. With a custom service, that responsibility moves into code the team owns. The benefit depends on whether the shared behavior removes more operating work than the new service adds.

This project sat alongside my broader ownership of the data platform: streaming and batch pipelines, customer-facing analytics, governance, and production operations. Orchestration was one part of that system. Its value was in giving those operations a common interface, rather than adding another tool for the team to learn.

The outcome

Scheduling was unified in the Rust service on ECS. The platform gained a shared operational interface and consistent handling of errors and alerts. I do not have a retained measurement for incident reduction or time saved, so this case study does not attach a percentage to the result.

Tradeoffs and lessons

You must want the operational semantics more than the ecosystem. Airflow and Dagster ship rich UIs and operator libraries; the case for building is owning scheduling, retries, and history exactly. That trade only makes sense when consistency across every pipeline is the product.

A custom scheduler creates an ownership obligation. The team gains control over behavior, but also owns upgrades, testing, and operational diagnosis. That continuing cost belongs in the decision.

Consolidation is a migration project, not a cutover. A migration needs explicit checks for schedules, dependencies, and failure behavior. A shared interface is useful only if workloads still run correctly behind it.