Use dependabot to bump dependencies

[antonymilne]

In #1805 it was generally agreed that we should use dependabot to automatically bump our dependencies. To be honest I don't know why we didn't just do this ages ago - it will hopefully make our lives way easier. We have a good enough test suite that we can rely on it to tell us whether it's safe to bump versions or not.

My original plan was to do it on kedro-viz first, which is a bit simpler in terms of requirements: kedro-org/kedro-viz#814. And then, if it seems to work well for us there, introduce it on kedro also.

However, I'd be happy to try it out on kedro first. I'm not sure exactly how dependabot works so don't know which dependencies exactly it would be able to track since we have them spread in a few places (requirements.txt, test_requirements.txt [mainly needed for kedro-datasets], setup.py). Highest priority is to get it working on requirements.txt since those are the core requirements.

Warning. Make sure that dependabot does not bump the major version (breaking changes) automatically for dataset dependences.

[datajoely]

I would like to push that we also get into the habit of doing smaller, more frequent patch releases for this sort of thing.

[SajidAlamQB]

Dependabot monitors these files in kedro:

• pyproject.toml
• requirements.txt
• test_requirements.txt
• features/windows_reqs.txt
• setup.py

I've been testing dependabot on a forked kedro repo you can view the dependency graph here: https://github.com/SajidAlamQB/kedro/network/dependencies

Currently dependabot opens 46 PRs for kedro which you can view here:
https://github.com/SajidAlamQB/kedro/pulls

[SajidAlamQB]

Dependabot remarks:

Dependabot version updates for Kedro will help us bump dependencies automatically but has some limitations. With the work done on kedro-datasets and the #1498, we decided to simplify dependabot to just requirments.txt. Due to the dependabot/dependabot-core#4364, we can't tell it what files it can access explicitly so we get around this by limiting which directory dependabot can scan. With this in mind, we have moved requirments.txt from root into dependency/requirments.txt forcing dependabot to only scan in this directory.

Furthermore, after some testing on starters, dependabot could not work properly. Dependabot can only run version updates on manifest files if it can access all of the dependencies in the file, even if you add inaccessible dependencies to the ignore option of your configuration file. In our starters requirments.txt we have kedro[pandas.CSVDataSet]~={{ cookiecutter.kedro_version }} dependency which dependabot couldn't parse hence it is failing. Following this, we have decided to forgo implementing dependabot on starters for now.

For now, we can leave this issue open so that when work on datasets is completed we can come back and look at a more comprehensive way of using dependabot.

[astrojuanlu]

This is basically in operation already. Can we close?

[SajidAlamQB]

Discussed in backlog-grooming, this has been completed.