Improve DB migration tests in CI

[potiuk]

Body

Our DB migration tests in CI run on empty tables and some of the migrations perform some data manipulation during down/up migration. By having empty tables we are skipping over this logic.

We should likely add some steps before and in-between the migration to populate the database witth data - likely by running a few dags maybe runniing a few other commands that modify the data (connections/variables etc.).

Also, it seems that some of the migrations might behave differently depending on what is the "to" migration - for example in case of 2.9.0 -> 2.8.4 migration, the original migration in #38743 failed, while 2.0.0 did not and we have to either change our CI testing or (I'd say more likely) make it a bit more "Target independent"

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[potiuk]

cc: @ephraimbuddy