Create dependabot.yml configuration file for version updates

[SajidAlamQB]

Description

Related Issue: #1810

This PR aims to add Dependabot Version updates for Kedro.
Dependabot will monitor only these files in kedro:

• requirements.txt

As this is the initial PR for dependabot it has identified 10 different dependencies that need updating. As discussed with @rashidakanchwala the configuration for dependabot will be initially set to daily checks with a limit of 50 PRs that the bot can open. Follow-up PRs will be made to change the configuration into a weekly check for updates and remove the maximum PR limit.

Note
When this PR is merged 10 new PRs will be opened by dependabot.

Development notes

To keep things similar between framework and viz a branch, dependency-update, will be created and used by dependabot. We will also keep the dependency label consistent.

I have tested this out on a forked kedro repo see the result here: https://github.com/SajidAlamQB/kedro-3/pulls

After some testing on starters, I was unable to get dependabot working properly. Dependabot can only run version updates on manifest files if it can access all of the dependencies in the file, even if you add inaccessible dependencies to the ignore option of your configuration file. In our starters requirments.txt we have kedro[pandas.CSVDataSet]~={{ cookiecutter.kedro_version }} dependency which dependabot couldn't parse hence it is failing.

Checklist

• Read the contributing guidelines
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes

[antonymilne]

Thanks for the explanation @SajidAlamQB! At a glance it looks like the vast majority of these updates are for dataset requirements - is that right?

Given that all these requirements would move to kedro-datasets I wonder whether we should just make those changes on that repo instead. I'm not sure exactly when we're going to remove the datasets tests (and hence the requirements) from this repo though.

There's also a problem here that the bumps to dataset requirements are just altering test_requirements.txt and not the ones in setup.py which are the ones that actually matter to users e.g. https://github.com/SajidAlamQB/kedro/pull/2. We have a ticket to clean up this setup but again I'm not sure when we'll be doing it: #1498.

Ideally I'd say that here we just limit dependabot to looking at requirements.txt and not test_requirements.txt for now, and deal with the dataset dependencies separately. Problem is that from what @rashidakanchwala said there didn't seem to be a good way to do that... Maybe there's something clever we could do with ignore though? https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file I don't even see documented anywhere how dependabot works out which files to try and modify - can you? Does looking at their source code make this clearer?

@MerelTheisenQB thoughts?

[merelcht]

You raise a lot of great points here @AntonyMilneQB !

Given that all these requirements would move to kedro-datasets I wonder whether we should just make those changes on that repo instead. I'm not sure exactly when we're going to remove the datasets tests (and hence the requirements) from this repo though.

I agree it would be better to make the changes on kedro-datasets, however, we're only fully removing the datasets in 0.19.0 so in the meantime we'll need to bump dataset requirements on Kedro as well.

There's also a problem here that the bumps to dataset requirements are just altering test_requirements.txt and not the ones in setup.py which are the ones that actually matter to users e.g. SajidAlamQB#2. We have a ticket to clean up this setup but again I'm not sure when we'll be doing it: #1498.

This is a very crucial point. If dependabot only bumps test_requirements.txt users won't benefit and we'll still have to manually go and change the extra_requires.. It would be great if we can find a way that it bumps both.

Ideally I'd say that here we just limit dependabot to looking at requirements.txt and not test_requirements.txt for now, and deal with the dataset dependencies separately. Problem is that from what @rashidakanchwala said there didn't seem to be a good way to do that... Maybe there's something clever we could do with ignore though? https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file I don't even see documented anywhere how dependabot works out which files to try and modify - can you? Does looking at their source code make this clearer?

☝️ This would be a good solution in the meantime. But I definitely feel that we should look at how extra_requires could be bumped automatically, because that's what really affects/benefits our users.

[SajidAlamQB]

Thanks @AntonyMilneQB that makes a lot of sense.

Ideally I'd say that here we just limit dependabot to looking at requirements.txt and not test_requirements.txt for now, and deal with the dataset dependencies separately. Problem is that from what @rashidakanchwala said there didn't seem to be a good way to do that... Maybe there's something clever we could do with ignore though? https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file I don't even see documented anywhere how dependabot works out which files to try and modify - can you? Does looking at their source code make this clearer?

Looking through this thread dependabot/dependabot-core#4364 there isn't a direct way through the dependabot.yml to exclude files/directories. I had a brief look through their source code but hasn't helped me identify how exactly they choose what files to modify. I will continue to look for any workarounds we can do in the meantime.

[rashidakanchwala]

Thanks for the explanation @SajidAlamQB! At a glance it looks like the vast majority of these updates are for dataset requirements - is that right?

Given that all these requirements would move to kedro-datasets I wonder whether we should just make those changes on that repo instead. I'm not sure exactly when we're going to remove the datasets tests (and hence the requirements) from this repo though.

There's also a problem here that the bumps to dataset requirements are just altering test_requirements.txt and not the ones in setup.py which are the ones that actually matter to users e.g. SajidAlamQB#2. We have a ticket to clean up this setup but again I'm not sure when we'll be doing it: #1498.

Ideally I'd say that here we just limit dependabot to looking at requirements.txt and not test_requirements.txt for now, and deal with the dataset dependencies separately. Problem is that from what @rashidakanchwala said there didn't seem to be a good way to do that... Maybe there's something clever we could do with ignore though? https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file I don't even see documented anywhere how dependabot works out which files to try and modify - can you? Does looking at their source code make this clearer?

@MerelTheisenQB thoughts?

you could also ignore test_requirements.txt if you change directory to the specific directory which only has requirements.txt (if that's the case)

[SajidAlamQB]

A user on the GitHub Issue I posted earlier recommended using renovate GitHub app: https://github.com/renovatebot/renovate which is an alternative to Dependabot. It does allow for more customizability but wanted to know everyone's thoughts on using this. @AntonyMilneQB @MerelTheisenQB

[merelcht]

Renovate looks really good! It has lots of stars on Github and seems to be maintained actively as well. I also read in a blog that it creates bigger PRs merging updates rather than a single PR for each update, which might be easier to get merged. If we go for this, I do think it would be better to use it for Viz as well so there's consistency in which tools we use.

[SajidAlamQB]

Below I have highlighted three options we can take for this issue (Please let me know if you think of any others):

Option 1 (Simple solution): We use dependabot as is and manually go through its PRs selecting the ones we want to be merged.

Option 2 (Breaking change?): The only real way we can tell dependabot which manifest files to use is by specifying the directory. Therefore if we place the requirements.txt or any other files in a specific directory rather than root we can limit which files dependabot uses.

Option 3 (Complex solution): We can alternatively forgo dependabot in favour of renovate, which is a GitHub app that is more configurable than dependabot. But this solution may mean we also have to change kedro-viz.

Note
After some discussion we have decided to go with Option 2 for the framework side, this change will only benefit development, we have also decided to add dependabot to the kedro-starters repo, so each new starter will have an updated template we can use the PRs opened there as a guide for the ones to add on the framework. Moreover, once work on the datasets is done we will implement dependabot there as well.

EDIT: After some testing on starters I was unable to get dependabot working properly. Dependabot can only run version updates on manifest files if it can access all of the dependencies in the file, even if you add inaccessible dependencies to the ignore option of your configuration file. In our starters requirments.txt we have kedro[pandas.CSVDataSet]~={{ cookiecutter.kedro_version }} dependency which dependabot couldn't parse hence it is failing.

[merelcht]

I wonder if this needs to be changed as well:
https://github.com/kedro-org/kedro/blob/main/.circleci/continue_config.yml#L149

I'm not actually sure how that checksum stuff works. The file is mentioned in a string so I don't know if it matters that it's now in a different directory.. 🤔

[noklam]

Looks good!

The fork branch is not accessible to me, I think it helps if I can see the result of the dependent bot PR.

A couple of questions:

1. Is it possible to do a batch update since you mentioned it will raise 46 PRs, it may take a lot of time to just manually merge it one by one.

2. Does it addresses the problem Antony raised now? I.e. the setup.py - I want to see an example PR.

3.I suggest do a global search of requirements.txt and other dependencies file to make sure we didn't miss any doc.
I think we need to update the MAKEFILE as well as this affect the contributor guide.

[SajidAlamQB]

Looks good!

The fork branch is not accessible to me, I think it helps if I can see the result of the dependent bot PR.

A couple of questions:

1. Is it possible to do a batch update since you mentioned it will raise 46 PRs, it may take a lot of time to just manually merge it one by one.
2. Does it addresses the problem Antony raised now? I.e. the setup.py - I want to see an example PR.

3.I suggest do a global search of requirements.txt and other dependencies file to make sure we didn't miss any doc. I think we need to update the MAKEFILE as well as this affect the contributor guide.

Thanks for the reply @noklam, after some discussion with the team we decided to limit dependabot just to update requirements.txt hence why requirements.txt has been moved to its own directory so dependabot can only access that.

With this change, there are no longer 46 PRs only 10 you can view the updated fork here: https://github.com/SajidAlamQB/kedro-3/pulls

Once requirements move to kedro-datasets we will implement dependabot there as well. There is still some lingering work to be done before we can fully embrace dependabot, I think we simplify this issue to have a basic implementation of it for now though.

[noklam]

Thank you @SajidAlamQB, I have had a quick look at the PRs and I have a follow-up question.

What PR are we going to merge? Do we accept all PRs as long as it passes the tests? @MerelTheisenQB

Update fsspec requirement from <=2022.7.1,>=2021.4 to >=2021.4,<2022.8.3 in /dependency
https://github.com/SajidAlamQB/kedro-3/pull/10

For example, I think this is a useful PR - as it updates the upper bound and user may need it. But why is it also changing the lower bound to 2021.4?

Update rich requirement from ~=12.0 to ~=12.6 in /dependency
https://github.com/SajidAlamQB/kedro-3/pull/4

For this PR, does it matters since we use ~=, I don't think it affects the users unless it is changing to ~=13.0.

[merelcht]

I think in practice we will merge most PRs, but we should be critical and have a proper look at what dependabot is actually changing. It shouldn't change lower bounds, but if it's a change that doesn't do much but isn't harmful either, we'll just get it in.

[SajidAlamQB]

We can add versioning strategies to avoid updating lower bounds and constraining it to just upper bounds.

[merelcht]

We can add versioning strategies to avoid updating lower bounds and constrain it just upper bounds.

I think the only problem with this, is that sometimes you have to bound the lower version, because of some vulnerability that was exposed and you should only use newer versions going forward.

[SajidAlamQB]

Dependabot remarks:

Dependabot version updates for Kedro will help us bump dependencies automatically but has some limitations. With the work being done on kedro-datasets and the improvements to test_requirments.txt setup, we decided to simplify dependabot to just requirments.txt. Due to the limitation of dependabot configuration, we can't tell it what files it can access explicitly so we get around this by limiting which directory dependabot can scan. With this in mind, we have moved requirments.txt from root into dependency/requirments.txt forcing dependabot to only scan in this directory.

Have a look at a forked Kedro repo with the above-mentioned changes: https://github.com/SajidAlamQB/kedro-3/pulls

Furthermore, after some testing on starters, dependabot was not able to work properly. Dependabot can only run version updates on manifest files if it can access all of the dependencies in the file, even if you add inaccessible dependencies to the ignore option of your configuration file. In our starters requirments.txt we have kedro[pandas.CSVDataSet]~={{ cookiecutter.kedro_version }} dependency which dependabot couldn't parse hence it is failing. Following this, we have decided to forgo implementing dependabot on starters for now.

[merelcht]

Thanks for investigating all options with dependabot @SajidAlamQB ! This is already a step closer in improving our dependency management 👍

[noklam]

Thanks for explaining and nice write up! I think it is a good first step even if it's not automating everything.