Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update description of installing providers separately from core #24454

Merged
merged 1 commit into from
Jun 20, 2022

Conversation

potiuk
Copy link
Member

@potiuk potiuk commented Jun 14, 2022

The description of how to install providers separately from the
core was wrong. It suggested installing main constraints, but
in fact we are not able to give any guarantees when installing
providers this way so no constraints can guarantee consistent
set of dependencies and the user is on their own in this case.

This PR corrects the installation docs.


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragement file, named {pr_number}.significant.rst, in newsfragments.

The description of how to install providers separately from the
core was wrong. It suggested installing main constraints, but
in fact we are not able to give any guarantees when installing
providers this way so no constraints can guarantee consistent
set of dependencies and the user is on their own in this case.

This PR corrects the installation docs.
@potiuk
Copy link
Member Author

potiuk commented Jun 14, 2022

This is a result of talking to a user :).

Comment on lines +134 to +136
to install, upgrade or downgrade any of the providers you need. We release providers independently from the
core of Airflow, so often new versions of providers are released before Airflow is, also if you do not want
yet to upgrade Airflow to the latest version, you might want to install newly released providers separately.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
to install, upgrade or downgrade any of the providers you need. We release providers independently from the
core of Airflow, so often new versions of providers are released before Airflow is, also if you do not want
yet to upgrade Airflow to the latest version, you might want to install newly released providers separately.
to install, upgrade or downgrade any of the providers you need. We release providers independently from
core Airflow, so often new versions of providers are released before Airflow is. You also may might want to
install newly released providers separately without also upgrading your core Airflow version.

@eladkal
Copy link
Contributor

eladkal commented Jun 14, 2022

What is the effort in creating:
Constraint-2.3.2-latest?

Currently we have constraint-2.3.2 which created one time only when version is released (so providers versions are never updated). Using constraint-main may cause problems as you deacribed in this PR.

Basicly the idea/question/suggestion is to have a latest version of updated constraints per specific core version which will be updated regularly. This will save users the big headeach if doable.

@potiuk
Copy link
Member Author

potiuk commented Jun 14, 2022

What is the effort in creating:
Constraint-2.3.2-latest?

1st of all this is not for installing latest providers on "latest" airflow, but installing them on ANY compatible airflow. For this last scenario we are rather limited in what we can do.

The overhead is not that big for just "latest" airflow - we are actually doing it every time when we release a bugfix release already - so 2.3.3 constraints are exactly that when we prepare them.
The question is - if we start adding constraints for "previous" releases of Airflow - should we do it for "latest" stable release only, or for "all versions". Which versions? I think it's a no-go for anything but the latest release due to overhead "per version" it needs.

And to be honest - it does not help a lot to anyone but those who are on the latest version of it is the "latest" only. Plus it is not always possible, we would have to likely manually remove some providers - pip resolution is not perfect, as it might get to bactracking cycles, and I am afraid if we "commit" to releasing such constraints, we are going to have a lot of work to solve those. And in some cases (like k8s provider with conflicting kubernets library update) it won't be possible at all.

And it's not really problematic actually. we have roughly 1.5- monthly release cycle for Airlfow bugfixes. So we are usually talking about max 2 weeks gap between those two events. And it will automatically use the latest providers, so I'd rather encourage people to update to latest bugfix release.

Another possibility (hinted in the survey) - maybe even better we could also adapt our release schedule for providers - it does not have to be "exactly monthly" - even now it is "roughly monthly". So we could make sure we release all providers a week (or 1.5 week) before a new bugfix/feature release of Airlfow - making sure that when we release the "latest" it always has "just released" providers as well. That might actually be the easiest and best way to do.

@potiuk
Copy link
Member Author

potiuk commented Jun 19, 2022

Should we merge it @eladkal ?

@github-actions github-actions bot added the okay to merge It's ok to merge this PR as it does not require more tests label Jun 20, 2022
@github-actions
Copy link

The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

@potiuk potiuk merged commit 28d3236 into apache:main Jun 20, 2022
@potiuk potiuk deleted the better-description-for-provider-upgrade branch June 20, 2022 08:09
ephraimbuddy pushed a commit that referenced this pull request Jun 29, 2022
The description of how to install providers separately from the
core was wrong. It suggested installing main constraints, but
in fact we are not able to give any guarantees when installing
providers this way so no constraints can guarantee consistent
set of dependencies and the user is on their own in this case.

This PR corrects the installation docs.

(cherry picked from commit 28d3236)
@ephraimbuddy ephraimbuddy added the type:doc-only Changelog: Doc Only label Jun 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind:documentation okay to merge It's ok to merge this PR as it does not require more tests type:doc-only Changelog: Doc Only
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants