For some reason, Airflow doesn't seem to trigger the latest run for a dag with a weekly schedule interval.
Current Date:
$ date
$ Tue Aug 9 17:09:55 UTC 2016
DAG:
from datetime import datetime
from datetime import timedelta
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
dag = DAG(
dag_id='superdag',
start_date=datetime(2016, 7, 18),
schedule_interval=timedelta(days=7),
default_args={
'owner': 'Jon Doe',
'depends_on_past': False
}
)
BashOperator(
task_id='print_date',
bash_command='date',
dag=dag
)
Run scheduler
$ airflow scheduler -d superdag
You'd expect a total of four DAG Runs as the scheduler should backfill for 7/18, 7/25, 8/1, and 8/8. However, the last run is not scheduled.
EDIT 1:
I understand that Vineet although that doesn’t seem to explain my issue.
In my example above, the DAG’s start date is July 18.
- First DAG Run: July 18
- Second DAG Run: July 25
- Third DAG Run: Aug 1
- Fourth DAG Run: Aug 8 (not run)
Where each DAG Run processes data from the previous week.
Today being Aug 9, I would expect the Fourth DAG Run to have executed with a execution date of Aug 8 which processes data for the last week (Aug 1 until Aug 8) but it doesn’t.