Data modeling and lineage framework

The problem

The team’s standard delivery pattern was Athena queries feeding QuickSight dashboards. AWS gives you the compute; the surrounding discipline was missing. Three gaps recurred: how is the code tracked and promoted between environments; how are assets materialized, orchestrated, and monitored; and what documentation, definitions, and access rules exist around each dataset. Each gap could be patched with effort - manual runs, tribal knowledge, stale Confluence pages - but the effort compounded with every pipeline.

dbt solved this shape of problem for warehouses, but there was no straightforward managed path for Athena, and running dbt against Athena meant containers and custom integrations that reproduced most of the framework anyway.

The approach

I built a standalone, Python-native modeling framework:

• Models as Python objects: queries defined in model definitions; the framework handles rendering, execution, and environment promotion, and runs in CI and inside orchestration tools.
• Column-level lineage: each model declares its inputs; the framework propagates types, names, and column-level lineage from sources through every downstream model.
• Generated documentation: definitions written once in models synced to Confluence, and datasets pushed to the Glue catalog and QuickSight - so the catalog and the BI layer were generated from the same model definitions.
• Orchestration-friendly: build-pipeline integration meant every change ran the same path to test and production.

Years later I led the migration of this estate to SQLMesh. The in-house framework had served the data platform; by then the maintained open-source ecosystem offered the same discipline without our maintenance burden.

The outcome

The framework became the default way data was built for the organization: modeled, versioned, documented, and lineage-traced from source to dashboard. Questions that previously required archaeology - “which pipeline feeds this column?” - became lookups. The SQLMesh migration moved the modeling work onto an open-source engine.

Tradeoffs and lessons

Building was the right call then; migrating was the right call later. The framework existed because the gap existed. When the ecosystem closed the gap, owning a modeling engine stopped being differentiation and became liability.

Lineage at the column level, not the table level, is what changes conversations. Impact analysis for a rename or a type change is a graph query instead of a search party.

Documentation that is generated gets read. Generating documentation from model definitions reduced the work required to keep it aligned with code.