Reduce number of commits in upload large folder

[Wauplin]

Should fix #2518 (or at least make it more reasonable).

This PR reduces the number of commits produced for upload-large-folder command. The main change is to make the commit size bigger, going from 25 files to either 75 regular files or 150 LFS files. Also, the rules have now been slightly modified to avoid commits with only 1 or 2 files at the beginning of the process.

This should greatly reduce the number of rate limits when using this tool while staying robust.

[LysandreJik]

The change makes sense to me! Thanks @Wauplin

[hanouticelina]

Still need to dig more into this part of the code but it seems all good to me :)

[Wauplin]

Thanks for the reviews! 🤗

[FurkanGozukara]

@Wauplin thank you so much

i just made a test and uploading 20400 files

it went very smooth until 5700 files were submitted

then errors started to pop up - i hadn't had used any api quote today

[Wauplin]

Has the 504 gateway timeout started to be a blocker? When such an error happens, the commit is usually retried. I suspect that with the git repository getting bigger and bigger (5700 files to handle), the commit size starts to be too big to process under the X second before the timeout is triggered. What we could do is a dynamic threshold that gets lower when a 504 is triggered.

But at least it's a completely different problem than the rate-limit one!

[FurkanGozukara]

Has the 504 gateway timeout started to be a blocker? When such an error happens, the commit is usually retried. I suspect that with the git repository getting bigger and bigger (5700 files to handle), the commit size starts to be too big to process under the X second before the timeout is triggered. What we could do is a dynamic threshold that gets lower when a 504 is triggered.

But at least it's a completely different problem than the rate-limit one!

yes it was a huge blockler, it really made it way way slower

[Wauplin]

Ok thanks for the feedback 👍 We'll think about how to update this.