Removing Default Example DAGs from Airflow: A Comprehensive Guide
Written on
Chapter 1: Introduction to Airflow
Apache Airflow is a widely-used batch-oriented workflow orchestrator that has seen significant adoption in recent years. The setup process for Airflow is quite simple. However, upon launching Airflow, you might notice several example Directed Acyclic Graphs (DAGs) displayed in the user interface.
These example DAGs are designed to help newcomers become acquainted with the platform. As you grow more comfortable with Airflow, you may wish to eliminate these samples and start developing your own workflows and DAGs.
Section 1.1: Disabling Example DAGs in Airflow
In this section, we will guide you on how to disable the loading of example DAGs in Airflow.
Subsection 1.1.1: Modifying the airflow.cfg File
The airflow.cfg file serves as a configuration container for running Airflow instances. To prevent Airflow from loading example DAGs, simply add the following configuration option under the core section of the file:
[core]
load_examples = False
Section 1.2: Disabling Example DAGs in Docker
If you are operating Airflow using Docker, you can deactivate example DAGs by setting the AIRFLOW__CORE__LOAD_EXAMPLES environment variable. In the docker-compose.yml file, under the environment key, you would add:
version: '3'
x-airflow-common:
&airflow-common
image: apache/airflow:2.3.0
environment:
- AIRFLOW__CORE__LOAD_EXAMPLES=False
# Additional configuration can be added here...
Chapter 2: Managing DAGs in Airflow
This video titled "How to REMOVE airflow example DAGs locally and in docker" provides a practical walkthrough on removing example DAGs from your local Airflow setup.
In this video, titled "How to remove default example dags in airflow," you will find further insights on the process of eliminating default example DAGs.
Deleting Example DAGs from the Airflow Database
Airflow relies on a database backend utilized by the scheduler, executors, and web server to manage state information. This database features a table named dags, which includes all DAGs, including the example ones loaded during startup. You also have the option to directly access the database and remove the example DAGs, ensuring they are no longer loaded by Airflow.
Final Thoughts
When Airflow launches, it automatically loads several example DAGs to assist new users in getting started with the orchestrator. If these examples are no longer needed, you can adjust the configuration settings to prevent Airflow from loading them. In this guide, we explored various methods to achieve this, including altering the airflow.cfg file, setting the appropriate environment variable, or deleting them directly from the Airflow database.