2

当我尝试使用 xmltodict 将一些 xml 转换为数据框时，会发生特定列包含我需要的所有信息作为 dict 或 dict 列表。我可以使用 pandas 将此列转换为多个列，但我无法在 dask 中执行类似的操作。

不可能使用 meta，因为我不知道 xml 中可用的所有可能字段，并且 dask 是必要的，因为真正的 xml 文件每个都大于 1Gb。

示例.xml：

<?xml version="1.0" encoding="UTF-8"?>
<itemList>
  <eventItem uid="1">
    <timestamp>2019-07-04T09:57:35.044Z</timestamp>
    <eventType>generic</eventType>
    <details>
      <detail>
        <name>columnA</name>
        <value>AAA</value>
      </detail>
      <detail>
        <name>columnB</name>
        <value>BBB</value>
      </detail>
    </details>
  </eventItem>
  <eventItem uid="2">
    <timestamp>2019-07-04T09:57:52.188Z</timestamp>
    <eventType>generic</eventType>
    <details>
      <detail>
        <name>columnC</name>
        <value>CCC</value>
      </detail>
    </details>
  </eventItem>
</itemList>

工作熊猫代码：

import xmltodict
import collections
import pandas as pd

def pd_output_dict(details):
    detail = details.get("detail", [])
    ret_value = {}
    if type(detail) in (collections.OrderedDict, dict):
        ret_value[detail["name"]] = detail["value"]
    elif type(detail) == list:
        for i in detail:
            ret_value[i["name"]] = i["value"]
    return pd.Series(ret_value)

with open("example.xml", "r", encoding="utf8") as f:
    df_dict_list = xmltodict.parse(f.read()).get("itemList", {}).get("eventItem", [])
    df = pd.DataFrame(df_dict_list)
    df = pd.concat([df, df.apply(lambda row: pd_output_dict(row.details), axis=1, result_type="expand")], axis=1)
    print(df.head())

不工作的代码：

import xmltodict
import collections
import dask
import dask.bag as db
import dask.dataframe as dd

def dd_output_dict(row):
    detail = row.get("details", {}).get("detail", [])
    ret_value = {}
    if type(detail) in (collections.OrderedDict, dict):
        row[detail["name"]] = detail["value"]
    elif type(detail) == list:
        for i in detail:
            row[i["name"]] = i["value"]
    return row

with open("example.xml", "r", encoding="utf8") as f:
    df_dict_list = xmltodict.parse(f.read()).get("itemList", {}).get("eventItem", [])
    df_bag = db.from_sequence(df_dict_list)
    df = df_bag.to_dataframe()
    df = df.apply(lambda row: dd_output_dict(row), axis=1)

这个想法是在大熊猫中获得类似的结果，但在我收到错误的那一刻：

>>> df = df.apply(lambda row: output_dict(row), axis=1)
Traceback (most recent call last):
  File "C:\Anaconda3\lib\site-packages\dask\dataframe\utils.py", line 169, in raise_on_meta_error
    yield
  File "C:\Anaconda3\lib\site-packages\dask\dataframe\core.py", line 4711, in _emulate
    return func(*_extract_meta(args, True), **_extract_meta(kwargs, True))
  File "C:\Anaconda3\lib\site-packages\dask\utils.py", line 854, in __call__
    return getattr(obj, self.method)(*args, **kwargs)
  File "C:\Anaconda3\lib\site-packages\pandas\core\frame.py", line 6487, in apply
    return op.get_result()
  File "C:\Anaconda3\lib\site-packages\pandas\core\apply.py", line 151, in get_result
    return self.apply_standard()
  File "C:\Anaconda3\lib\site-packages\pandas\core\apply.py", line 257, in apply_standard
    self.apply_series_generator()
  File "C:\Anaconda3\lib\site-packages\pandas\core\apply.py", line 286, in apply_series_generator
    results[i] = self.f(v)
  File "<stdin>", line 1, in <lambda>
  File "<stdin>", line 4, in output_dict
AttributeError: ("'str' object has no attribute 'get'", 'occurred at index 0')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Anaconda3\lib\site-packages\dask\dataframe\core.py", line 3964, in apply
    M.apply, self._meta_nonempty, func, args=args, udf=True, **kwds
  File "C:\Anaconda3\lib\site-packages\dask\dataframe\core.py", line 4711, in _emulate
    return func(*_extract_meta(args, True), **_extract_meta(kwargs, True))
  File "C:\Anaconda3\lib\contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\Anaconda3\lib\site-packages\dask\dataframe\utils.py", line 190, in raise_on_meta_error
    raise ValueError(msg)
ValueError: Metadata inference failed in `apply`.

You have supplied a custom function and Dask is unable to
determine the type of output that that function returns.

To resolve this please provide a meta= keyword.
The docstring of the Dask function you ran should have more information.

Original error is below:
------------------------
AttributeError("'str' object has no attribute 'get'", 'occurred at index 0')

Traceback:
---------
  File "C:\Anaconda3\lib\site-packages\dask\dataframe\utils.py", line 169, in raise_on_meta_error
    yield
  File "C:\Anaconda3\lib\site-packages\dask\dataframe\core.py", line 4711, in _emulate
    return func(*_extract_meta(args, True), **_extract_meta(kwargs, True))
  File "C:\Anaconda3\lib\site-packages\dask\utils.py", line 854, in __call__
    return getattr(obj, self.method)(*args, **kwargs)
  File "C:\Anaconda3\lib\site-packages\pandas\core\frame.py", line 6487, in apply
    return op.get_result()
  File "C:\Anaconda3\lib\site-packages\pandas\core\apply.py", line 151, in get_result
    return self.apply_standard()
  File "C:\Anaconda3\lib\site-packages\pandas\core\apply.py", line 257, in apply_standard
    self.apply_series_generator()
  File "C:\Anaconda3\lib\site-packages\pandas\core\apply.py", line 286, in apply_series_generator
    results[i] = self.f(v)
  File "<stdin>", line 1, in <lambda>
  File "<stdin>", line 4, in output_dict
4

1 回答 1

0

对，所以像 map_partitions 这样的操作需要知道列名和数据类型。正如您所提到的，您可以使用meta=关键字指定它。

也许您可以遍历您的数据一次以计算它们将是什么，然后构造一个适当的元对象并将其传递进去？这是低效的，需要阅读所有数据，但我不确定是否有另一种方法。

于 2019-08-29T14:12:32.687 回答