Top 50 Airflow Interview Questions and Answers

What is Apache Airflow?
Airflow is an open-source workflow automation and orchestration tool that allows users to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs).
What are DAGs in Airflow?
DAGs (Directed Acyclic Graphs) are collections of tasks organized to reflect their relationships and dependencies.
How does Airflow handle dependencies between tasks?
Dependencies are defined in DAGs using set_upstream() or set_downstream() methods or by using >> and << operators.
What are the main components of Airflow?
- Scheduler: Orchestrates the execution of tasks.
- Executor: Handles the execution of tasks.
- Worker: Executes the tasks.
- Web Server: Provides a user interface.
- Metadata Database: Stores metadata.
What are Operators in Airflow?
Operators are predefined tasks in Airflow that define what is executed. Types include BashOperator, PythonOperator, EmailOperator, etc.

What is a Task Instance in Airflow?
A specific run of a task in a DAG for a particular execution date.
Explain the difference between Operators and Sensors.
- Operator: Executes an action or operation.
- Sensor: Waits for a condition to be met before executing downstream tasks.
What are the different types of Executors in Airflow?
- SequentialExecutor
- LocalExecutor
- CeleryExecutor
- KubernetesExecutor
How does Airflow handle retries?
Retries are configured using parameters like retries, retry_delay, and retry_exponential_backoff.
How can you parameterize a DAG?
By using dagrun.conf or Variable objects and passing arguments dynamically.
What is XCom in Airflow?
XCom (Cross-Communication) allows tasks to exchange small amounts of data during DAG runs.
What are Airflow hooks?
Hooks are interfaces to interact with external systems like databases, cloud services, etc.
How do you trigger a DAG manually?
Using the Airflow UI, CLI, or API.
What is the difference between depends_on_past and wait_for_downstream?
- depends_on_past: Ensures a task runs only if the previous instance of the same task succeeded.
- wait_for_downstream: Ensures a task runs only if all downstream tasks from the previous instance succeeded.
How is the Airflow Scheduler different from the Executor?
The Scheduler determines what tasks to execute, while the Executor actually executes the tasks.

How do you monitor workflows in Airflow?
Using the web UI, logs, and metrics exposed through the monitoring tab.
What is a SubDag?
A SubDag is a DAG within a DAG that allows hierarchical workflows.
Explain the concept of TaskGroup.
TaskGroup is a feature that groups tasks visually in the DAG UI to improve readability.
How do you manage Airflow configurations?
Using the airflow.cfg file or environment variables.
What are the best practices for writing Airflow DAGs?
- Keep DAGs idempotent.
- Use modular code.
- Limit DAG size.
- Use error handling and retries.
How does Airflow handle backfilling?
By running tasks for past dates where they haven’t been executed yet.
What is SLA in Airflow?
SLA (Service Level Agreement) defines the maximum allowed time for a task to complete.
How do you deploy Airflow in production?
Using a distributed setup with CeleryExecutor or KubernetesExecutor, along with proper monitoring and scaling.
Explain Dynamic DAG generation.
Dynamically generating DAGs based on external inputs or configurations using Python logic.
What is the role of Airflow Plugins?
Plugins extend Airflow functionalities like creating custom operators, sensors, hooks, etc.

How would you handle task failure in Airflow?
Configure retries, use failure callbacks, or set up alerting mechanisms.
How can you ensure data integrity in workflows?
Use Sensors to check data availability and implement robust error handling.
How do you set up a custom Operator?
Subclass the BaseOperator class and define the execute() method.
How would you optimize DAG performance?
- Avoid large DAG files.
- Use parallelism and concurrency.
- Offload heavy computations.
What are DAG Run states?
- Running
- Success
- Failed

How do you integrate Airflow with Kubernetes?
Use the KubernetesExecutor or KubernetesPodOperator.
How would you migrate workflows to Airflow?
Rewrite the workflows as Python code and define dependencies using DAGs.
What is Airflow’s role in ETL pipelines?
Orchestrating and automating data extraction, transformation, and loading tasks.
How do you handle high availability in Airflow?
Use CeleryExecutor with multiple workers and a resilient database backend.
What are some alternative tools to Airflow?
Prefect, Luigi, Dagster, and AWS Step Functions.

How do you debug Airflow tasks?
Use logs, inspect code in the execute() method, and test tasks locally.
How do you resolve database connection issues in Airflow?
Check database credentials, connectivity, and proper hook configuration.
How do you handle timezone differences in Airflow?
Use the start_date and end_date parameters with pendulum for timezone-aware DAGs.
How do you manage dependencies across DAGs?
Use ExternalTaskSensor or pass data between DAGs using shared databases.
What are the default Airflow directories?
- DAGs folder: Stores DAG files.
- Logs folder: Stores task logs.
- Plugins folder: Stores custom plugins.

Can Airflow handle streaming data?
Airflow is designed for batch processing, but workarounds like periodic DAG runs can be implemented.
What is the role of Celery in Airflow?
Celery handles distributed task execution in Airflow’s CeleryExecutor setup.
How do you archive old DAG runs?
Clean metadata from the database using the airflow db clean command.
What is Airflow’s API used for?
Automating tasks like triggering DAGs, fetching DAG statuses, and monitoring.
How can you secure an Airflow environment?
Enable RBAC, use secure authentication, and encrypt sensitive data.