Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expand tasks in mapped group at parse time #27158

Merged
merged 2 commits into from
Nov 10, 2022

Conversation

uranusjr
Copy link
Member

@uranusjr uranusjr commented Oct 20, 2022

Depends on #27027. Merged; rebased to main.

This involves a few things:

  • The mapped task group needs to implement parse-time and run-time ti count accessors, similar to the mapped operator.
  • Since a non-mapped operator can be in a mapped task group and thus needs to be expanded, the parse-time and run-time ti count accessors are "raised" from MappedOperator to AbstractOperator (so they can be used by BaseOperator). These now also considers an operator's parent task groups.
  • If a ti count accessor is called on a non-mapped operator that's not in any mapped task group, the accessor should return a special value of a sort. Since None is already used, we raise an exception in this case instead (simply called NotMapped). For consistency, the previous case where None was returned (upstreams not available) is also changed to raise an exception instead (reusing NotFullyPopulated). This means the ti count accessors now always return an int, and raise an exception whenever the count is not available.
  • All of the above result in a coherant interface in AbstractOperator that can be used in DagRun to perform ti count revision.

@uranusjr uranusjr changed the title [WIP] Add mapped task group info to serialization [WIP] Expand tasks in mapped group at parse time Oct 20, 2022
@uranusjr uranusjr force-pushed the mapped-task-group-literal-expand-ti branch from dbd5ab4 to 895876a Compare October 21, 2022 02:22
@uranusjr uranusjr changed the title [WIP] Expand tasks in mapped group at parse time Expand tasks in mapped group at parse time Oct 21, 2022
@uranusjr
Copy link
Member Author

uranusjr commented Oct 21, 2022

I think this is ready, hopefully CI agrees. (Edit: it does.)

@uranusjr uranusjr marked this pull request as ready for review October 21, 2022 09:18
@uranusjr uranusjr force-pushed the mapped-task-group-literal-expand-ti branch 2 times, most recently from 2ddef0b to 6e4d651 Compare October 24, 2022 11:27
This involves a few things:

* The mapped task group needs to implement parse-time and run-time ti
  count accessors, similar to the mapped operator.
* Since a non-mapped operator can be in a mapped task group and thus
  needs to be expanded, the parse-time and run-time ti count accessors
  are "raised" from MappedOperator to AbstractOperator (so they can be
  used by BaseOperator). These now also considers an operator's parent
  task groups.
* If a ti count accessor is called on a non-mapped operator that's not
  in any mapped task group, the accessor should return a special value
  of a sort. Since None is already used, we raise an exception in this
  case instead (simply called NotMapped). For consistency, the previous
  case where None was returned (upstreams not available) is also changed
  to raise an exception instead (reusing NotFullyPopulated). This means
  the ti count accessors now always return an int, and raise an
  exception whenever the count is not available.
* All of the above result in a coherant interface in AbstractOperator
  that can be used in DagRun to perform ti count revision.

I also removed caching from get_mapped_ti_count since (judging from the
various cache-clearing calls) it seems to be cause more trouble than
benefits.
@uranusjr uranusjr merged commit a16aa73 into apache:main Nov 10, 2022
@uranusjr uranusjr deleted the mapped-task-group-literal-expand-ti branch November 10, 2022 08:17
Adityamalik123 pushed a commit to Adityamalik123/airflow that referenced this pull request Nov 12, 2022
@ephraimbuddy ephraimbuddy added the type:new-feature Changelog: New Features label Nov 16, 2022
@ashb ashb added this to the Airflow 2.5.0 milestone Nov 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants