Repository navigation
Description
Apache Airflow version
3.0.6
If "Other Airflow 2 version" selected, which one?
No response
What happened?
For a DAG which allows for only one Dag run at a time (max_active_runs=1), if multiple events are posted created during processing of the current DagRun, only oldest one is processed while other are ignored.
Detailed scenario:
- Create asset event 1 (timestamp: T1)
- DAG Run 1 triggers for asset event 1 (run_after = T1)
- While DAG Run 1 is running: Create asset events 2, 3, and 4 (timestamps: T2, T3, T4)
- DAG Run 1 completes (consumes only event 1)
- DAG Run 2 triggers for asset event 2 (run_after = T2) consuming only event 2
- DAG Run 2 completes
- No further DAG runs are triggered automatically by the scgheduler
- Events 3 and 4 are lost forever unless a new asset event (event 5) is created
- If event 5 is created (timestamp: T5) DAG Run 3 is triggered consuming events 3, 4 and 5
- Result: Events 3, 4, and 5 are all processed together in DAG Run 3
Timeline:
T1=10:00 T2=10:01 T3=10:02 T4=10:03 T5=10:10
│ │ │ │ │
Event1 Event2 Event3 Event4 Event5
│ │ │ │ │
│ │ └─────────┴─────────────► │
│ │ (skipped by DAG Run 2) │
│ │ │
└─►DR1 └─►DR2 └─►DR3
(E1) (E2 only) (E3,E4,E5)
Made some SQL queries to confirm that run_after for DagRun 2 matches timestamp of event 2 (instead of lastest event 4)
SELECT start_date, run_after, run_id FROM dag_run WHERE dag_id = '<my_dag_name>' ORDER BY run_after DESC LIMIT 3;
"2025-09-24 11:54:00.343726+00" "2025-09-24 11:53:59.679149+00" "asset_triggered__2025-09-24T11:53:59.679149+00:00_mnhsp7QE" # DagRun 3
"2025-09-24 11:48:34.438651+00" "2025-09-24 11:45:29.587827+00" "asset_triggered__2025-09-24T11:45:29.587827+00:00_Pw3gZ43A" # DagRun 2 (run_after) matches timestamp of event 2, even though latest event is Event 4
"2025-09-24 11:44:57.051966+00" "2025-09-24 11:44:56.471468+00" "asset_triggered__2025-09-24T11:44:56.471468+00:00_jGlPAYkN" # DagRun 1
SELECT ae.id, ae.timestamp FROM asset_event ae JOIN asset a ON ae.asset_id = a.id WHERE a.name = '<my_asset_name>' ORDER BY ae.timestamp DESC LIMIT 5;
63 "2025-09-24 11:53:59.671414+00" # Event 5
62 "2025-09-24 11:45:33.58892+00" # Event 4
61 "2025-09-24 11:45:31.583102+00" # Event 3
60 "2025-09-24 11:45:29.579845+00" # Event 2
59 "2025-09-24 11:44:56.457813+00" # Event 1
What you think should happen instead?
All AssetEvents posted during DagRun execution should be processed by next DagRun (eventually each event should create separate DagRun as discussed in #55956 but I guess this is not how event scheduling is designed currently).
How to reproduce
- Create a DAG to be scheduled based on an Asset and ensure
max_active_runs=1 - Create asset event 1
- While DAG run 1 is processing asset event 1, create asset events 2 and 3
- After DAG run 1 processing ends, verify that DAG run 2 consumed only event 2
- After DAG run 2 processing ends, create asset event 4
- Verify that DAG run 3 processed events 3 and 4
Operating System
Debian GNU/Linux 12 (bookworm)
Versions of Apache Airflow Providers
apache-airflow-providers-amazon==9.12.0
apache-airflow-providers-celery==3.12.2
apache-airflow-providers-cncf-kubernetes==10.7.0
apache-airflow-providers-common-compat==1.7.3
apache-airflow-providers-common-io==1.6.2
apache-airflow-providers-common-messaging==1.0.5
apache-airflow-providers-common-sql==1.27.5
apache-airflow-providers-docker==4.4.2
apache-airflow-providers-elasticsearch==6.3.2
apache-airflow-providers-fab==2.4.1
apache-airflow-providers-ftp==3.13.2
apache-airflow-providers-git==0.0.6
apache-airflow-providers-google==17.1.0
apache-airflow-providers-grpc==3.8.2
apache-airflow-providers-hashicorp==4.3.2
apache-airflow-providers-http==5.3.3
apache-airflow-providers-microsoft-azure==12.6.1
apache-airflow-providers-mysql==6.3.3
apache-airflow-providers-odbc==4.10.2
apache-airflow-providers-openlineage==2.6.1
apache-airflow-providers-postgres==6.2.3
apache-airflow-providers-redis==4.2.0
apache-airflow-providers-sendgrid==4.1.3
apache-airflow-providers-sftp==5.3.4
apache-airflow-providers-slack==9.1.4
apache-airflow-providers-smtp==2.2.0
apache-airflow-providers-snowflake==6.4.0
apache-airflow-providers-ssh==4.1.3
apache-airflow-providers-standard==1.2.0
Deployment
Official Apache Airflow Helm Chart
Deployment details
Deployment to Kubernetes cluster using official helm chart, version 1.18.0.
Anything else?
Possible root cause - for some reason triggered_date_by_dag (code) contains timestamp of the oldest instead of most recent one AssetEvent from AssetDagRunQueue, which results in ignoring later events during scheduling (code) while removing all asset for DAG from AssetDagRunQueue (code).
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
phanikumv commentedon Nov 12, 2025
Cc @Lee-W could you please take a look at this?
phanikumv commentedon Nov 12, 2025
Had a chat with @Lee-W . This will need a re-design. We will use #56750 to track these issues
dschneider-wxs commentedon Feb 6, 2026
i ran into this too! and it blew up my scheduler because it couldn't process all the accumulated events at once. hoping this scenario can be handled with #55956
running airflow 3.1.6
Keep queued asset events while a Dag is at max_active_runs
LipuFei commentedon Oct 16, 2026
I ran into the same problem. I'm running Airflow 3.3.2.
It seems like, if there are many events and the DAG is running, some new events are deferred by the scheduler, so no new DAG run will be queued immediately, but then these events were kinda forgotten... Until another event for that asset arrives and "unblocks" or "awakens" the scheduler.
Initially I thought it's related to
max_active_runs = 1, but I also did a PoC locally withoutmax_active_runs = 1and I can still produce the problem. I think it's also a problem even whenmax_active_runsis not set.My PoC is very simple with two DAGs, one emitting multiple events and one job:
How to reproduce:
sendDAG 2 to 3 times. Wait a little bit between each trigger.jobDAG runs to complete.Assetson the GUI. A few events do not have DAG runs (until you send more events to the asset to unblock them).