2

我正在尝试将 dask 数组存储在 zarr 文件中。

当 dask 数组具有定义的形状时，我已经设法做到了。


import dask
import dask.array as da
import numpy as np
from tempfile import TemporaryDirectory
import zarr


np_array = np.random.randint(1, 10, size=1000)
array = da.from_array(np_array)

with TemporaryDirectory() as tmpdir:
    delayed = da.to_zarr(array, url=tmpdir,
                         compute=False, component='/data')
    dask.compute(delayed)

     z_object = zarr.open_group(tmpdir, mode='r')

     assert np.all(np_array == z_object.data[:])

但是，如果我对 dask 数组执行了任何操作，形状就会丢失，并且 zarr 会抱怨形状中的 Nans。

# this will fail

np_array = np.random.randint(1, 10, size=1000)
array = da.from_array(np_array)

array = array[array > 5]

with TemporaryDirectory() as tmpdir:
    delayed = da.to_zarr(array, url=tmpdir,
                         compute=False, component='/data')
    dask.compute(delayed)

    z_object = zarr.open_group(tmpdir, mode='r')

    assert np.all(np_array[np_array > 5] == z_object.data[:])

这是引发的错误：

Traceback (most recent call last):
  File "/home/peio/devel/variation/variation6/variation6/tests/test_zarr.py", line 38, in <module>
    without_shape()
  File "/home/peio/devel/variation/variation6/variation6/tests/test_zarr.py", line 29, in without_shape
    compute=False, component='/data')
  File "/home/peio/devel/variation/pyenv3/lib/python3.7/site-packages/dask/array/core.py", line 2808, in to_zarr
    **kwargs
  File "/home/peio/devel/variation/pyenv3/lib/python3.7/site-packages/zarr/creation.py", line 120, in create
    chunk_store=chunk_store, filters=filters, object_codec=object_codec)
  File "/home/peio/devel/variation/pyenv3/lib/python3.7/site-packages/zarr/storage.py", line 323, in init_array
    object_codec=object_codec)
  File "/home/peio/devel/variation/pyenv3/lib/python3.7/site-packages/zarr/storage.py", line 343, in _init_array_metadata
    shape = normalize_shape(shape) + dtype.shape
  File "/home/peio/devel/variation/pyenv3/lib/python3.7/site-packages/zarr/util.py", line 58, in normalize_shape
    shape = tuple(int(s) for s in shape)
  File "/home/peio/devel/variation/pyenv3/lib/python3.7/site-packages/zarr/util.py", line 58, in <genexpr>
    shape = tuple(int(s) for s in shape)
ValueError: cannot convert float NaN to integer

有没有办法将没有已知形状的 dask 数组存储到 zarr 文件中？

提前致谢！

4

1 回答 1

2

Zarr 期望块的形状是统一的并且是事先知道的。Dask 目前通过将数组重新分块使其统一来促进这一点。但是array[array > 5]会创建一个具有未知块形状的 Dask Array。因此，由于不存在所需的信息，因此无法预先将其重新划分为统一的。也就是说，我们可以更好地解释这一点。

可以通过使用返回已知块形状的 Dask 操作来解决此问题（正如 David 所建议的那样）。或者，可以在存储之前确定块形状（以计算为代价）。我们也可以讨论扩展 Zarr 来处理这种情况，但这是一个长期的解决方案。

于 2019-07-23T17:19:39.853 回答