Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add mover/copy stage #67

Merged
merged 36 commits into from
Jan 18, 2024
Merged

Add mover/copy stage #67

merged 36 commits into from
Jan 18, 2024

Conversation

jbusecke
Copy link
Collaborator

Towards #33 and will help keeping the ultimate target store 'clean' as in there will be no empty or half-written stores.

@jbusecke
Copy link
Collaborator Author

Ok some of the jobs (example) fail with

FileNotFoundError: [Errno 2] No such file or directory: 'gsutil' [while running 'Creating CMIP6.ScenarioMIP.MPI-M.MPI-ESM1-2-LR.ssp126.r16i1p1f1.Omon.o2.gn.v20210901|OpenURLWithFSSpec|OpenWithXarray|Preprocessor|StoreToZarr|MoveStore|Logging to non-QC table|TestDataset|Logging to QC table/MoveStore/Moving Zarr Store-ptransform-59']

what is the best way to get gustil on the runner? @yuvipanda @cisaacstern

If that is hard I might consider using the beam IO module... but would be nice if this simple stage would work...

@jbusecke
Copy link
Collaborator Author

Lets see if adding gustil to the requirements.txt just works...

@jbusecke jbusecke changed the title Add mover stage Add mover/copy stage Nov 18, 2023
@jbusecke
Copy link
Collaborator Author

So I am running into this issue when calling gusti via subprocess.run:

FIND THIS: ServiceException: 401 Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket. Permission 'storage.objects.list' denied on resource (or it may not exist).

So we do not have the right permissions. How is access actually set within a Ptransform?

Wondering if @yuvipanda or @cisaacstern have an idea here.

Quick summary: I just want to copy the zarr store from a given path to another within a Ptransform

@jbusecke
Copy link
Collaborator Author

Ok so using beam.io worked quite well. Thanks to @sharkinsspatial for the initial push, and @cisaacstern for the pair programming today.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant