# Pathrule Pattern: Apache Airflow Data Workflows (1.0.0) # ::pathrule:package:apache-airflow ### [RULE] Derive every read and write from the logical data interval (path: /src/tasks) A task may run late, retry tomorrow, or backfill last year. Reading the process clock makes the same task instance operate on different data depending on when infrastructure executes it. - Accept interval start, interval end, logical date, and run identity as explicit task inputs rather than calling the current date for partition choice. - Query sources with closed or half-open boundaries that prevent overlap and gaps across adjacent intervals. - Write to a deterministic partition or operation key derived from the logical interval and transformation version. - Include DAG, task, run, map, attempt, and interval identity in logs and data-quality evidence so reruns remain attributable. See /dags for the adjacent decision or procedure that completes this constraint. --- ### [RULE] Make every task safe to retry and clear (path: /src/tasks) Airflow retries and operators can clear task instances manually. A task that appends blindly or sends an unkeyed notification produces duplicates even though orchestration is behaving correctly. - Stage output for the task instance and publish it through an atomic replace, partition overwrite, merge, or committed manifest. - Use stable idempotency keys for remote APIs, notifications, and messages and record their outcomes outside transient worker memory. - Do not mark orchestration success until output completeness and required quality checks are durable. - Define cleanup for failed staging data without deleting a prior successful interval or another concurrent run. See /tests/dags for the adjacent decision or procedure that completes this constraint. --- ### [MEMORY] DAG parsing is pure, fast configuration (path: /dags) Schedulers repeatedly parse DAG files. Top-level database queries, cloud listing, secrets calls, or large computation slow the control plane and can make a temporary dependency outage hide the entire workflow. - Keep top-level code limited to imports, constants, lightweight configuration, and deterministic graph construction. - Move source discovery and data-dependent branching into tasks or approved dynamic mapping based on bounded task output. - Load secrets at task execution unless graph construction genuinely needs a non-secret configuration value. - Test that importing the DAG module performs no external I/O and remains within the scheduler's parsing budget. See /tests/dags for the rule or workflow that puts this decision into practice. --- ### [MEMORY] Airflow metadata carries references, not datasets (path: /dags) Orchestration metadata is optimized for task coordination, not large tables, files, or model objects. Passing bulk data through it burdens the metadata database and web interface. - Return object locations, partition identifiers, row counts, checksums, schema versions, and quality summaries from tasks. - Store datasets in the warehouse, object store, or database appropriate to their format and retention. - Keep control payloads bounded and free of secrets or large personal records because metadata is widely visible to operators. - Make downstream tasks verify the referenced object's identity and completeness rather than trusting a path string alone. See /src/tasks for the rule or workflow that puts this decision into practice. --- ### [SKILL] plan-airflow-backfill (path: /) --- name: plan-airflow-backfill description: Plan and execute an Airflow backfill after logic repair, late data, or a new destination. --- # Plan Airflow Backfill Run this procedure when the affected surface changes, before the result is promoted to production. Record evidence for every step instead of accepting a plausible-looking result. 1. Define the exact DAG version, interval range, expected partitions, source retention, destination behavior, and prior successful state. 2. Estimate per-interval reads, writes, connections, memory, and external calls, then cap parallelism through pools and batch windows. 3. Run a small representative interval set, compare row counts, checksums, quality rules, and downstream visibility, and prove rerun idempotency. 4. Expand in bounded batches while watching scheduler, workers, metadata database, source, destination, and downstream freshness. 5. Reconcile every intended interval and output, record gaps or exceptions, and remove temporary concurrency or routing changes after completion. ## Exit criteria The change is complete only when the expected behavior, failure behavior, and rollback path have all been exercised with representative data. Preserve the evidence with the change so the next operator can repeat the same checks. --- ### [SKILL] triage-airflow-failure (path: /) --- name: triage-airflow-failure description: Triage a failed, missing, delayed, or repeatedly retried Airflow task or DAG run. --- # Triage Airflow Failure Run this procedure when the affected surface changes, before the result is promoted to production. Record evidence for every step instead of accepting a plausible-looking result. - [ ] Identify DAG, task, run, interval, map index, attempt, worker, queue, and code version before reading a generic stack trace. - [ ] Separate not-scheduled, queued, worker-start, execution, timeout, infrastructure, dependency, data-quality, and publish failures. - [ ] Inspect upstream task state, pools, concurrency, sensors, external dependencies, and metadata health before rerunning the task. - [ ] Decide whether the failure is transient and idempotent, permanently invalid, or requires code or data repair; do not clear state as the first diagnostic action. - [ ] After repair, rerun the smallest safe interval, verify output and downstream state, then document the signal that would detect recurrence earlier. ## Exit criteria The change is complete only when the expected behavior, failure behavior, and rollback path have all been exercised with representative data. Preserve the evidence with the change so the next operator can repeat the same checks.