Moving data to Pangeo from google bucket that is "requester pays"

[kdlamb]

Hi, I am interested in accessing a data set that is currently in a google bucket. The bucket access is set to "requester pays". It is relatively large (150+ GB). Is there a good way to directly access it in its current location or to move it over to Pangeo so that it could be accessed directly?

[jbusecke]

Thanks for raising this @kdlamb.

I think that we should avoid actually moving the data, since that will cause recurring costs. Instead we should figure out some guidance on how to deal with such data.

I am seeing two pathways here:

• The user who wants to access the data has a Google Cloud account and can use that to charge the egress costs too. This is absolutely possible right now, but some additional guidance about how to best authenticate from within the hub would be very helpful to add to the docs.
• I think there needs to be a process through which we can decide and implement to pay the egress from the LEAP account. In fact I think this is already set up for certain pangeo buckets which are requester pays. I believe it is worth surfacing where this is configured and then discussing a process how users could request changes to this config.

I will bring this discussion to 2i2c's attention and hope they can provide some more insight.

[rabernat]

Is the point here to:

1. Provide access to this data to LEAP researchers from the LEAP hub? or...
2. Provide access to the general public over the internet?

If 2, it makes sense to move to OSN. If 1, we are already in Google Cloud, so we don't have to pay egress charges to access from our hub.

[kdlamb]

I'm mainly interested in having access to the data myself, rather than providing access to the general public.

[jbusecke]

I believe @kdlamb was mostly looking at 1, if I understood correctly.

If 1, we are already in Google Cloud, so we don't have to pay egress charges to access from our hub.

Right, thanks for correcting me here. I just realized that I made a stupid mistake in testing this yesterday:

Trying

import xarray as xr
import gcsfs
fs = gcsfs.GCSFileSystem(requester_pays=True)
mapper = fs.get_mapper('gs://eu-aerosols-spaceml/Geostationary_imagery/100_1')
ds = xr.open_dataset(mapper, engine='zarr', chunks={})

gives

/tmp/ipykernel_1010/3727539910.py:5: RuntimeWarning: Failed to open Zarr store with consolidated metadata, but successfully read with non-consolidated metadata. This is typically much slower for opening a dataset. To silence this warning, consider: 1. Consolidating metadata in this existing store with zarr.consolidate_metadata(). 2. Explicitly setting consolidated=False, to avoid trying to read consolidate metadata, or 3. Explicitly setting consolidated=True, to raise an error in this case instead of falling back to try reading non-consolidated metadata. ds = xr.open_dataset(mapper, engine='zarr', chunks={}) --------------------------------------------------------------------------- KeyError Traceback (most recent call last) File /srv/conda/envs/notebook/lib/python3.10/site-packages/xarray/backends/zarr.py:210, in _get_zarr_dims_and_attrs(zarr_obj, dimension_key, try_nczarr) 208 try: 209 # Xarray-Zarr --> 210 dimensions = zarr_obj.attrs[dimension_key] 211 except KeyError as e:

File /srv/conda/envs/notebook/lib/python3.10/site-packages/zarr/attrs.py:73, in Attributes.getitem(self, item)
72 def getitem(self, item):
---> 73 return self.asdict()[item]

KeyError: '_ARRAY_DIMENSIONS'

During handling of the above exception, another exception occurred:

KeyError Traceback (most recent call last)
File /srv/conda/envs/notebook/lib/python3.10/site-packages/xarray/backends/zarr.py:224, in _get_zarr_dims_and_attrs(zarr_obj, dimension_key, try_nczarr)
221 try:
222 # NCZarr uses Fully Qualified Names
223 dimensions = [
--> 224 os.path.basename(dim) for dim in zarray["_NCZARR_ARRAY"]["dimrefs"]
225 ]
226 except KeyError as e:

KeyError: '_NCZARR_ARRAY'

The above exception was the direct cause of the following exception:

KeyError Traceback (most recent call last)
Cell In[4], line 5
3 fs = gcsfs.GCSFileSystem(requester_pays=True)
4 mapper = fs.get_mapper('gs://eu-aerosols-spaceml/Geostationary_imagery/100_1')
----> 5 ds = xr.open_dataset(mapper, engine='zarr', chunks={})

File /srv/conda/envs/notebook/lib/python3.10/site-packages/xarray/backends/api.py:570, in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, inline_array, chunked_array_type, from_array_kwargs, backend_kwargs, **kwargs)
558 decoders = _resolve_decoders_kwargs(
559 decode_cf,
560 open_backend_dataset_parameters=backend.open_dataset_parameters,
(...)
566 decode_coords=decode_coords,
567 )
569 overwrite_encoded_chunks = kwargs.pop("overwrite_encoded_chunks", None)
--> 570 backend_ds = backend.open_dataset(
571 filename_or_obj,
572 drop_variables=drop_variables,
573 **decoders,
574 **kwargs,
575 )
576 ds = _dataset_from_backend_dataset(
577 backend_ds,
578 filename_or_obj,
(...)
588 **kwargs,
589 )
590 return ds

File /srv/conda/envs/notebook/lib/python3.10/site-packages/xarray/backends/zarr.py:980, in ZarrBackendEntrypoint.open_dataset(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta, group, mode, synchronizer, consolidated, chunk_store, storage_options, stacklevel, zarr_version)
978 store_entrypoint = StoreBackendEntrypoint()
979 with close_on_error(store):
--> 980 ds = store_entrypoint.open_dataset(
981 store,
982 mask_and_scale=mask_and_scale,
983 decode_times=decode_times,
984 concat_characters=concat_characters,
985 decode_coords=decode_coords,
986 drop_variables=drop_variables,
987 use_cftime=use_cftime,
988 decode_timedelta=decode_timedelta,
989 )
990 return ds

File /srv/conda/envs/notebook/lib/python3.10/site-packages/xarray/backends/store.py:43, in StoreBackendEntrypoint.open_dataset(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta)
29 def open_dataset( # type: ignore[override] # allow LSP violation, not supporting **kwargs
30 self,
31 filename_or_obj: str | os.PathLike[Any] | BufferedIOBase | AbstractDataStore,
(...)
39 decode_timedelta=None,
40 ) -> Dataset:
41 assert isinstance(filename_or_obj, AbstractDataStore)
---> 43 vars, attrs = filename_or_obj.load()
44 encoding = filename_or_obj.get_encoding()
46 vars, attrs, coord_names = conventions.decode_cf_variables(
47 vars,
48 attrs,
(...)
55 decode_timedelta=decode_timedelta,
56 )

File /srv/conda/envs/notebook/lib/python3.10/site-packages/xarray/backends/common.py:210, in AbstractDataStore.load(self)
188 def load(self):
189 """
190 This loads the variables and attributes simultaneously.
191 A centralized loading function makes it easier to create
(...)
207 are requested, so care should be taken to make sure its fast.
208 """
209 variables = FrozenDict(
--> 210 (_decode_variable_name(k), v) for k, v in self.get_variables().items()
211 )
212 attributes = FrozenDict(self.get_attrs())
213 return variables, attributes

File /srv/conda/envs/notebook/lib/python3.10/site-packages/xarray/backends/zarr.py:517, in ZarrStore.get_variables(self)
516 def get_variables(self):
--> 517 return FrozenDict(
518 (k, self.open_store_variable(k, v)) for k, v in self.zarr_group.arrays()
519 )

File /srv/conda/envs/notebook/lib/python3.10/site-packages/xarray/core/utils.py:471, in FrozenDict(*args, **kwargs)
470 def FrozenDict(*args, **kwargs) -> Frozen:
--> 471 return Frozen(dict(*args, **kwargs))

File /srv/conda/envs/notebook/lib/python3.10/site-packages/xarray/backends/zarr.py:518, in (.0)
516 def get_variables(self):
517 return FrozenDict(
--> 518 (k, self.open_store_variable(k, v)) for k, v in self.zarr_group.arrays()
519 )

File /srv/conda/envs/notebook/lib/python3.10/site-packages/xarray/backends/zarr.py:494, in ZarrStore.open_store_variable(self, name, zarr_array)
492 data = indexing.LazilyIndexedArray(ZarrArrayWrapper(name, self))
493 try_nczarr = self._mode == "r"
--> 494 dimensions, attributes = _get_zarr_dims_and_attrs(
495 zarr_array, DIMENSION_KEY, try_nczarr
496 )
497 attributes = dict(attributes)
499 # TODO: this should not be needed once
500 # zarr-developers/zarr-python#1269 is resolved.

File /srv/conda/envs/notebook/lib/python3.10/site-packages/xarray/backends/zarr.py:227, in _get_zarr_dims_and_attrs(zarr_obj, dimension_key, try_nczarr)
223 dimensions = [
224 os.path.basename(dim) for dim in zarray["_NCZARR_ARRAY"]["dimrefs"]
225 ]
226 except KeyError as e:
--> 227 raise KeyError(
228 f"Zarr object is missing the attribute {dimension_key} and the NCZarr metadata, "
229 "which are required for xarray to determine variable dimensions."
230 ) from e
232 nc_attrs = [attr for attr in zarr_obj.attrs if attr.lower().startswith("_nc")]
233 attributes = HiddenKeyDict(zarr_obj.attrs, [dimension_key] + nc_attrs)

KeyError: 'Zarr object is missing the attribute _ARRAY_DIMENSIONS and the NCZarr metadata, which are required for xarray to determine variable dimensions.'

which to me indicates that the access works but I am missing something about the organization of the actual data (I will raise a separate issue to discuss that).

But just so that I understand this correctly: If I am accessing e.g. data from the legacy pangeo catalog I do not have to specify requester_pays=True.

xr.open_dataset('gs://cmip6/GFDL_CM2_6/control/ocean', engine='zarr')

just works. So I assume there must be something that is configured specifically for this store/bucket?