Use sql_alchemy_conn for celery result backend when result_backend is not set

[Aakcht]

This PR adds possibility to use the same connection for celery result_backend and metadata by automatically adding db+ for result_backend connection.

closes #18283

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragement file, named {pr_number}.significant.rst, in newsfragments.

@wip ready for review

[dstandish]

A little unsure about this one. On one hand I get the pain, and like the idea of making it more convenient. On the other hand it's good to be explicit. Another way of approaching this... what if we just made it so that if celery result backend is not configured, then it would take the database as the default (with prefix)?

So then, if it's configured, we use what we are given (even if it's wrong because missing db prefix, thus forcing the user to update the config to be correct). But if it's not configured, i.e. so that there is no value for conf.get('celery', 'RESULT_BACKEND'), then we use the database sql_alchemy_conn value and add prefix.

cc @jedcunningham

[Aakcht]

@dstandish For me the issue is mostly related to airflow helm chart/kubernetes: I have the same problem as described in #18283 - I want to use the same pre-created secret for both metadata and resultBackendConnection in helm chart, see https://github.com/apache/airflow/blob/main/chart/values.yaml#L322 . Currently it's not possible because one of the secrets has to start with "db+".

I suppose using sql_alchemy_conn when result_backend is not configured would work, but in this case some changes will be required in the helm chart too, will it be ok?

For example currently with celery executor this secret for result_backend connection is created every time when custom result backend secret is not provided and the env for the connection is always defined.

[dstandish]

i see re https://github.com/apache/airflow/blob/main/chart/templates/secrets/result-backend-connection-secret.yaml#L21 we create the secret when none referenced in your config. good point.

so yeah if you like this suggestion, i.e. to deal with this in airflow config, then i think in the chart we just have to look at airflow version and if greater than 2.4 (or whenever this hypothetical change implemented) then don't create the secret (because sql_alchemy_conn will be used by default anyway)

[Aakcht]

Sure, I'll try implementing the change so sql_alchemy_conn is used when result_backend is not set. Looks like I'll have to change default value of result_backend to None (currently it is set to db+postgresql://postgres:airflow@postgres/airflow - hopefully it won't be too much of a problem.

[Aakcht]

The PR is ready for review - not sure why WIP check is still in progress :/ I can retrigger the checks if needed.

As @potiuk suggested, I modified helm chart tests with parameterized to make it easier to update helm charts to a future airflow version.

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[potiuk]

Nice! You made it with parameterizzed tests :)

[Aakcht]

All the tests have finally passed 😌