使用xarray，我ds.to_zarr()用于将数据集写入 S3，然后xr.open_zarr()查看是否得到相同的数据集。
我的数据集xarray如下所示：
<xarray.Dataset>
Dimensions: (nv: 2, reference_time: 11, time: 11, x: 4608, y: 3840)
Coordinates:
* reference_time (reference_time) datetime64[ns] 2018-04-01T18:00:00 ...
* x (x) float64 -2.304e+06 -2.303e+06 -2.302e+06 ...
* y (y) float64 -1.92e+06 -1.919e+06 -1.918e+06 ...
* time (time) datetime64[ns] 2018-04-01T19:00:00 ...
Dimensions without coordinates: nv
Data variables:
time_bounds (time, nv) datetime64[ns] dask.array<shape=(11, 2), chunksize=(1, 2)>
ProjectionCoordinateSystem (time) |S64 b'' b'' b'' b'' b'' b'' b'' b'' ...
T2D (time, y, x) float64 dask.array<shape=(11, 3840, 4608), chunksize=(1, 3840, 4608)>
我把它写成zarr使用：
fs = s3fs.S3FileSystem(anon=False)
d = s3fs.S3Map(f_zarr, s3=fs)
ds.to_zarr(store=d, mode='w')
当我尝试使用以下方法读回它时：
ds2 = xr.open_zarr(d)
我回来了：
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-17-8198db1c8578> in <module>()
----> 1 ds2 = xr.open_zarr(d)
/opt/conda/lib/python3.6/site-packages/xarray/backends/zarr.py in open_zarr(store, group, synchronizer, auto_chunk, decode_cf, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables)
476
477 variables = OrderedDict([(k, maybe_chunk(k, v))
--> 478 for k, v in ds.variables.items()])
479 return ds._replace_vars_and_dims(variables)
480 else:
/opt/conda/lib/python3.6/site-packages/xarray/backends/zarr.py in <listcomp>(.0)
476
477 variables = OrderedDict([(k, maybe_chunk(k, v))
--> 478 for k, v in ds.variables.items()])
479 return ds._replace_vars_and_dims(variables)
480 else:
/opt/conda/lib/python3.6/site-packages/xarray/backends/zarr.py in maybe_chunk(name, var)
471 token2 = tokenize(name, var._data)
472 name2 = 'zarr-%s' % token2
--> 473 return var.chunk(chunks, name=name2, lock=None)
474 else:
475 return var
/opt/conda/lib/python3.6/site-packages/xarray/core/variable.py in chunk(self, chunks, name, lock)
820 data = indexing.ImplicitToExplicitIndexingAdapter(
821 data, indexing.OuterIndexer)
--> 822 data = da.from_array(data, chunks, name=name, lock=lock)
823
824 return type(self)(self.dims, data, self._attrs, self._encoding,
/opt/conda/lib/python3.6/site-packages/dask/array/core.py in from_array(x, chunks, name, lock, asarray, fancy, getitem)
1977 >>> a = da.from_array(x, chunks=(1000, 1000), lock=True) # doctest: +SKIP
1978 """
-> 1979 chunks = normalize_chunks(chunks, x.shape)
1980 if name in (None, True):
1981 token = tokenize(x, chunks)
/opt/conda/lib/python3.6/site-packages/dask/array/core.py in normalize_chunks(chunks, shape)
1907 raise ValueError(
1908 "Chunks and shape must be of the same length/dimension. "
-> 1909 "Got chunks=%s, shape=%s" % (chunks, shape))
1910
1911 if shape is not None:
ValueError: Chunks and shape must be of the same length/dimension. Got chunks=(11, 64), shape=(11,)
如果我设置，我可以读取数据集auto_chunk=False：
ds2 = xr.open_zarr(d, auto_chunk=False)
ds2
结果是
<xarray.Dataset>
Dimensions: (nv: 2, reference_time: 11, time: 11, x: 4608, y: 3840)
Coordinates:
* reference_time (reference_time) datetime64[ns] 2018-04-01T18:00:00 ...
* time (time) datetime64[ns] 2018-04-01T19:00:00 ...
* x (x) float64 -2.304e+06 -2.303e+06 -2.302e+06 ...
* y (y) float64 -1.92e+06 -1.919e+06 -1.918e+06 ...
Dimensions without coordinates: nv
Data variables:
LWDOWN (time, y, x) float64 ...
但是我是否不了解有关分块和方式的一些知识xarray，dask并且zarr应该一起工作？
为了上班，我必须做些什么不同的auto_chunk=True事情？