Quick answer: Find the first failed task, inspect its logs and recent changes, identify whether the issue is code, permissions, data or infrastructure, then fix and rerun only the safe failed portion.
1. Start with the failed task, not the whole DAG
Open the Airflow grid or graph view and inspect the specific failed task instance. Check its try number, start time, error trace and upstream dependencies. A downstream failure may be a symptom of an earlier task.
2. Categorize the failure
- Connection or IAM: expired credentials, missing BigQuery/GCS permissions, or unavailable endpoints.
- Code or environment: import errors, missing dependencies, Python exceptions, incorrect configuration.
- Input data: missing file, schema drift, malformed records, unexpected nulls.
- Platform: memory pressure, scheduling capacity, timeouts, transient service incidents.
3. Validate the latest upstream inputs
For a pipeline landing files in Cloud Storage and loading BigQuery, verify the expected object exists, matches the naming/date convention and can be read by the service account. Reconcile staging row counts before relying on final business tables.
4. Rerun only after checking idempotency
Re-running an ingestion task can create duplicates if it only appends. Design tasks with stable load identifiers, deduplication, partition replacement or MERGE logic so retries produce the intended state.
5. Prevent the repeat incident
Add actionable error notifications, input validations, bounded retries, documented runbooks and monitoring for missing or late data. Record root cause and preventive change separately from immediate recovery.
Interview-ready explanation
“I begin at the failed task and read the traceback. I check dependencies, the input file, permissions and recent deployments, then fix the root cause and rerun the affected task safely. Finally I validate row counts and add prevention where needed.”
This is a general educational example, not a disclosed client production incident.