Pipeline automation at API-limit scale

The problem

The organization’s dealer advertising ran through Datorama: pipelines with per-dealership credentials, typically three to four per dealership, at a scale of over 4,000 pipelines. Datorama caps resources and pipelines per workspace - limits the organization hit early. Manual management meant someone opening streams in the UI, judging statuses by eye, and rerunning failures by hand; state and reruns were unmanageable at that volume.

The binding constraint was the API: 20,000 requests per 24 hours. A naive automation that re-fetched configuration each run would burn roughly 12,000 daily calls just reconstructing object state, leaving almost nothing for the actual work - reruns and pipeline creation.

The approach

I wrote a Python wrapper around Datorama’s API that made the platform scriptable within those limits:

• Incremental configuration caching: the wrapper stores the last definition of every object it has seen and rebuilds objects from that definition on startup, calling the API for lists and requesting definitions only for objects it has never seen. Startup dropped from ~12,000 calls to a few dozen.
• Adaptive job checking: previous per-stream job durations, generated from stored history, estimated how long to wait before checking a job for completion, instead of polling on a fixed interval.
• Queue-aware scheduling: workspace resources are finite and jobs queue as the daily schedule fires; the wrapper evaluated the queue depth before adding jobs, preventing starvation and stuck runs.
• Specific recovery behavior: on schedule, it identified failed jobs and reran them; repeated failures - the ones retrying would not fix - were flagged for a person. Detection, retry, and escalation were explicit behaviors, not an unqualified “self-healing” claim.

The outcome

Routine management - status checks, reruns, pipeline creation - became a repeatable, scheduled operation instead of a full-time manual grind. API usage stayed well inside the daily cap because state reconstruction was nearly free. Failures still reached a human, but only the ones that needed one.

Tradeoffs and lessons

Cache invalidation is cheaper than rate-limit exile. The design bet that object definitions change rarely paid off; the trade is a stale-definition risk when the platform changes an object outside the wrapper, which requires an explicit refresh when existing definitions change.

Adaptive waiting beats polling. Modeling wait times from historical durations reduced unnecessary status requests.

Automate the escalation, not just the fix. Separating “retryable” from “needs a human” is what kept trust in the automation high.