Trying to read in a zarr store from s3 using xarray. Getting a Key Error. Any thoughts
import fsspec
import xarray as xr
uri = "s3://era5-pds/zarr/2020/12/data/eastward_wind_at_10_metres.zarr"
ds = xr.open_zarr(fsspec.get_mapper(uri, anon=True), consolidated=True)
I can open it locally ok if I download it first:
import s3fs
fs = s3fs.S3FileSystem(anon=True)
fs.get("s3://era5-pds/zarr/2020/12/data/eastward_wind_at_10_metres.zarr/*", "eastward_wind_at_10_metres.zarr", recursive=True)
Here's the Traceback associated with the top command
Traceback (most recent call last):
File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/s3fs/core.py", line 234, in _call_s3
return await method(**additional_kwargs)
File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/aiobotocore/client.py", line 154, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/fsspec/mapping.py", line 132, in __getitem__
result = self.fs.cat(k)
File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/fsspec/asyn.py", line 241, in cat
raise ex
File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/s3fs/core.py", line 737, in _cat_file
resp = await self._call_s3(
File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/s3fs/core.py", line 252, in _call_s3
raise translate_boto_error(err) from err
FileNotFoundError: The specified key does not exist.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/xarray/backends/zarr.py", line 675, in open_zarr
ds = open_dataset(
File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/xarray/backends/api.py", line 572, in open_dataset
store = opener(filename_or_obj, **extra_kwargs, **backend_kwargs)
File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/xarray/backends/zarr.py", line 294, in open_group
zarr_group = zarr.open_consolidated(store, **open_kwargs)
File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/zarr/convenience.py", line 1178, in open_consolidated
meta_store = ConsolidatedMetadataStore(store, metadata_key=metadata_key)
File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/zarr/storage.py", line 2680, in __init__
meta = json_loads(store[metadata_key])
File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/fsspec/mapping.py", line 136, in __getitem__
raise KeyError(key)
KeyError: '.zmetadata'
