Skip to content

[Prototyping] Play around with google drive and AWS testing with github secrets #528

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 39 commits into
base: main
Choose a base branch
from

Conversation

JoeZiminski
Copy link
Member

@JoeZiminski JoeZiminski commented Jun 20, 2025

Hey @cs7-shrey this is PR to prototype a possible approach to testing. This PR will not be merged, it's just here as a rough example.

I set up a NIU google drive and AWS with the credentials as github secrets (I can give you these credentials later you can play with this outside of the CI). The tests connect programmatically to google drive or AWS, setting up the connection and transferring some files. Later we can fill this in to perform some checks that the expected files were transferred.
To have a look at the changes, you can check the diff on the last two commits (one for google drive and one for AWS).

For google drive, I had to change the way that the non-browser connection worked in order to make it programmatic. Now, the user must generate a credentials json through google, and pass the path of this to rclone. A benefit is that it does not require installing rclone on a second machine, but a downside is that it requires a bit more set up on the google end. For now, I'm leaning towards using this method because it can be performed programmatically, and it's roughly equivalent in convenience, and the non-browser case should in any case be infrequently used. Let me know what you think about this, thanks!

shrey added 30 commits June 14, 2025 19:04
@JoeZiminski JoeZiminski force-pushed the test_gdrive_aws_secrets branch from d26159a to 1921f6f Compare June 20, 2025 17:43
@JoeZiminski JoeZiminski force-pushed the test_gdrive_aws_secrets branch from 1921f6f to 086fcd2 Compare June 20, 2025 18:03
@JoeZiminski JoeZiminski force-pushed the test_gdrive_aws_secrets branch from 086fcd2 to 34d158e Compare June 20, 2025 18:04
@JoeZiminski JoeZiminski changed the title Test gdrive aws secrets Play around with google drive and AWS testing with github secrets Jun 20, 2025
@JoeZiminski
Copy link
Member Author

JoeZiminski commented Jun 20, 2025

Some general thoughts:

  1. We could create a new branch /connection-dev (I'll make sure you have permissions to branch off the main repo) and squash merge Add Google Drive and AWS S3 as a Remote Storage option #503 into it, and then start a new PR for tests. This might make it easier to track, because the implementation is basically done and it will be nice to draw a line under it and then start on tests. But, we cannot merge into main yet as I think there will be a quick release done before this is complete, and we can't release it without the tests.
  2. The search you have using rclone is really nice, if we can centralise local filesystem and ssh tests through this function, it would be fantastic. We could then perform all of the heavy-duty tests for local filesystem, and then do a few key tests for ssh, aws and gdrive, because they will all share most of the underlying code.
  3. I'm working on the ssh tests, I can refactor these according to (2) assuming that it is possible. Then we will have the 'connections' implementation and test really tight!
  4. We can also use rclone to implement Check that central_path can be written to during SSH setup. #449 in a cross-connection fashion

@JoeZiminski JoeZiminski changed the title Play around with google drive and AWS testing with github secrets [Prototyping] Play around with google drive and AWS testing with github secrets Jun 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant