How to create a conditional task in Airflow - Stack Overflow I would like to create a conditional task in Airflow as described in the schema below The expected scenario is the following: Task 1 executes If Task 1 succeed, then execute Task 2a Else If Task 1
Airflow backfill clarification - Stack Overflow I'm just getting started with Airbnb's airflow, and I'm still not clear on how when backfilling is done Specifically, there are 2 use-cases that confuse me: If I run airflow scheduler for a few
airflow - Run DAG at specific time each day - Stack Overflow I've read multiple examples about schedule_interval, start_date and the Airflow docs multiple times aswell, and I still can't wrap my head around: How do I get to execute my DAG at a specific time each day?
Proper way to create dynamic workflows in Airflow Problem Is there any way in Airflow to create a workflow such that the number of tasks B * is unknown until completion of Task A? I have looked at subdags but it looks like it can only work with a
Airflow: how to delete a DAG? - Stack Overflow In Airflow versions < 1 10 , its a two step process: 1 Remove the Dag from airflow dags folder This will remove the dag from airflow list_dags command But it will still be visible on GUI with a message that since its state is active, it is shown on Airflow GUI In order to remove follow the step below: 2) Go to mysql instance of airflow cluster and look for database name "airflow" In that
python - setting up s3 for logs in airflow - Stack Overflow 40 UPDATE Airflow 1 10 makes logging a lot easier For s3 logging, set up the connection hook as per the above answer and then simply add the following to airflow cfg [core] # Airflow can store logs remotely in AWS S3