0

我想像这样加载bookcorpus

train_ds, test_ds = load_dataset('bookcorpus', split=['train', 'test']),

但是,得到以下错误:

Traceback (most recent call last):             
  File "<stdin>", line 1, in <module>
  File "/home/marcelbraasch/.local/lib/python3.8/site-packages/datasets/load.py", line 1627, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/marcelbraasch/.local/lib/python3.8/site-packages/datasets/builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File "/home/marcelbraasch/.local/lib/python3.8/site-packages/datasets/builder.py", line 709, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/home/marcelbraasch/.local/lib/python3.8/site-packages/datasets/utils/info_utils.py", line 74, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=4853859824, num_examples=74004228, dataset_name='bookcorpus'), 'recorded': SplitInfo(name='train', num_bytes=2982081448, num_examples=45726619, dataset_name='bookcorpus')}]

我想继续将它保存到磁盘,因为我不想每次使用它时都下载它。是什么导致了这个错误?

4

1 回答 1

0

BookCorpus 不再公开可用。

这是一个解决方法:

https://github.com/soskek/bookcorpus

于 2021-12-02T20:55:32.400 回答