Airflow patterns that survive 300+ pipelines

When a single Airflow deployment grows past a few hundred DAGs, the failure mode stops being 'a pipeline breaks' and becomes 'noise drowns out the signal of a pipeline breaking'. Most of the patterns that helped weren't clever — they were boring, consistent, and enforced everywhere.

Standardising on a small set of custom operators — rather than letting every team write bespoke PythonOperators — turned out to be the highest-leverage change. It meant retries, alerting, and logging behaved the same way across every stakeholder team's DAGs, which made on-call rotations viable instead of miserable.

The second big lever was routing failures somewhere a human could actually triage in seconds, not minutes. Wiring PagerDuty directly into DAG-level callbacks, with enough context in the payload to skip the 'which pipeline, which task, which run' scavenger hunt, cut incident detection time dramatically.

None of this is exotic. It's the unglamorous work of making the 'good path' the only path.