-
Notifications
You must be signed in to change notification settings - Fork 14.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix incorrect data interval alignment due to assumption on input time alignment #22658
Conversation
I am very sorry, because my wrong rebase caused the old request to be closed. 😭 |
8a59c00
to
c16d11f
Compare
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
This PR fixes the incorrect first run of data_interval_start after changing the scheduling time. * Added _align_to_prev function for scheduling alignment after time change. * renamed _align to _align_to_next.
0d243d5
to
e3271e6
Compare
Anyone else wants to take a look at this one? It’s a relatively straightforward change, since we established that this additional method is needed. |
I think what woudl be great here is to add a few more tests showing the different cases and maybe explaining the context in the test a bit better? The case here is either not tested with the edge case or it's not clear that it is tested I think:
|
Actually this is also the cause of #23689. I’ve added a few test cases here to illustrate those issues, and modified the PR description to better explain the problems this solve. |
This introduces a new
_align_to_prev
util function for cron-based interval alignment. Previously we assumed_get_prev(_align_to_next(t))
is enough for this purpose, but this proved incorrect (see discussions in #21715), and an additional alignment logic is needed to avoid the inferred interval being off by one period when the input value has unexpected alignment (unexpectedly aligning for a manual run, and unexpected not aligning for an auto one).See also #21715
Fix #23689