python - 如何将以下多维类似xml的数据解析为数据框

Question

我探索了一个基于 XML 的 API，用于与工作相关的事物，它来自仓库数据。理想情况下，我想用 pandas 在 python 中做一些分析。

Aggregate(aggregate_dimension_value_list=[ DateAggregateDimensionValue(value=datetime.datetime(2013, 8, 28, 19, 30, tzinfo= UTC )) , None,  StringAggregateDimensionValue(value=u'VIRTUALLY_LABELED_CASE') ], quantity=127) ,  

Aggregate(aggregate_dimension_value_list=[ DateAggregateDimensionValue(value=datetime.datetime(2013, 8, 28, 19, 30, tzinfo= UTC )) ,  StringAggregateDimensionValue(value=u'PPTransMergeNonCon') ,  StringAggregateDimensionValue(value=u'PRIME_BIN_RANDOM_STOW') ], quantity=15)   

Aggregate(aggregate_dimension_value_list=[ DateAggregateDimensionValue(value=datetime.datetime(2013, 8, 27, 21, 0, tzinfo= UTC )) ,  StringAggregateDimensionValue(value=u'PPTransFRA1') ,  StringAggregateDimensionValue(value=u'PRIME_BIN_RANDOM_STOW') ], quantity=8) ,

数据看起来像上面的流，在我在 VIM 中做了一些查找和替换之后（我知道我可以在 python 中编写脚本）。如何最好地将这种奇怪的格式放入 Pandas 中？理想情况下，我想要日期时间、字符串聚合维度值和数量。但是在这个需要解析的数据中，有很多 None。在数据框中进行一些分析很容易，但我在这里有点难过（感觉很像 n00b）。

编辑：这是我得到并想要解析的未正则表达式和未替换的数据。它不是真正的 XML，所以 XML 不起作用。

[<DateAggregateDimensionValue(value=datetime.datetime(2013, 8, 26, 20, 30, tzinfo=<UTC
>))>, <StringAggregateDimensionValue(value=u'PPTransCGN1')>, <
StringAggregateDimensionValue(value=u'PRIME_BIN_RANDOM_STOW')>], quantity=992)>, <
StringAggregateDimensionValue(value=u'PPTransLEJ1')>, <StringAggregateDimensionValue(
value=u'PRIME_BIN_RANDOM_STOW')>], quantity=945)>, <Aggregate(
aggregate_dimension_value_list=[<DateAggregateDimensionValue(value=datetime.datetime(2013
, 8, 23, 19, 30, tzinfo=<UTC>))>, None, <StringAggregateDimensionValue(value=u'TOTE')>],
 quantity=87)>, <Aggregate(aggregate_dimension_value_list=[<DateAggregateDimensionValue(
value=datetime.datetime(2013, 8, 27, 17, 30, tzinfo=<UTC>))>, <
StringAggregateDimensionValue(value=u'PPTransMUC3')>, <StringAggregateDimensionValue(
value=u'TOTE')>], quantity=14)>, <Aggregate(aggregate_dimension_value_list=[<
DateAggregateDimensionValue(value=datetime.datetime(2013, 8, 27, 20, 30, tzinfo=<UTC
>))>, <StringAggregateDimensionValue(value=u'PPTransEUK5')>, <
StringAggregateDimensionValue(value=u'PRIME_BIN_RANDOM_STOW')>], quantity=339)>, <
Aggregate(aggregate_dimension_value_list=[<DateAggregateDimensionValue(value=datetime.
datetime(2013, 8, 26, 20, 30, tzinfo=<UTC>))>, <StringAggregateDimensionValue(value=u
'PPTransCGN1')>, <StringAggregateDimensionValue(value=u'TOTE')>], quantity=1731)>, <
Aggregate(aggregate_dimension_value_list=[<DateAggregateDimensionValue(value=datetime.
datetime(2013, 8, 26, 19, 30, tzinfo=<UTC>))>, <StringAggregateDimensionValue(value=u
'PPTransEUK5')>, quantity=444)>, <Aggregate(aggregate_dimension_value_list=[<
DateAggregateDimensionValue(value=datetime.datetime(2013, 8, 26, 19, 30, tzinfo=<UTC
>))>, <StringAggregateDimensionValue(value=u'PPTransEUK5')>, <
StringAggregateDimensionValue(value=u'TOTE')>], quantity=28)>, <Aggregate(
aggregate_dimension_value_list=[<DateAggregateDimensionValue(value=datetime.datetime(2013
, 8, 28, 19, 30, tzinfo=<UTC>))>, <StringAggregateDimensionValue(value=u'PPTransORY1')>,
 <StringAggregateDimensionValue(value=u'PRIME_BIN_RANDOM_STOW')>], quantity=69)>, <
Aggregate(aggregate_dimension_value_list=<Aggregate(aggregate_dimension_value_list=[<
DateAggregateDimensionValue(value=datetime.datetime(2013, 8, 26, 19, 30, tzinfo=<UTC
>))>, <StringAggregateDimensionValue(value=u'PPTransMAD4')>, <
StringAggregateDimensionValue(value=u'PRIME_BIN_RANDOM_STOW')>], quantity=47)>, <
Aggregate(aggregate_dimension_value_list=[<DateAggregateDimensionValue(value=datetime.
datetime(2013, 8, 26, 21, 0, tzinfo=<UTC>))>, None, None], quantity=78)>

score 1 · Accepted Answer

如果您更喜欢解析器的内容，这里有一个 pyparsing 解决您的问题：

from pyparsing import Suppress,QuotedString,Word,alphas,nums,alphanums,Keyword,Optional
import datetime

# define UTC timezone for sake of eval
if hasattr(datetime,"timezone"):
    UTC = datetime.timezone(datetime.timedelta(0),"UTC")
else:
    UTC = None

_ = Suppress
evaltokens = lambda s,l,t: eval(''.join(t))

timevalue = 'datetime.datetime' + QuotedString('(', endQuoteChar=')', unquoteResults=False)
timevalue.setParseAction(evaltokens)

strvalue = 'u' + QuotedString("'", unquoteResults=False)
strvalue.setParseAction(evaltokens)

nonevalue = Keyword("None").setParseAction(lambda s,l,t: [None])
intvalue = Word(nums).setParseAction(lambda s,l,t: int(t[0]))

COMMA = Optional(_(","))

valuedexpr = lambda expr: (Word(alphas) + "(" + "value" + "=" + expr + ")").setParseAction(lambda t: t[4])

lineexpr = (_("Aggregate(aggregate_dimension_value_list=[") +
            valuedexpr(timevalue)("timestamp") + COMMA +
            (nonevalue | valuedexpr(strvalue))("s1") + COMMA +
            (nonevalue | valuedexpr(strvalue))("s2") + COMMA +
        "]" + COMMA +
        "quantity=" + intvalue("qty"))

用于lineexpr.searchString从每个聚合中提取数据：

for data in lineexpr.searchString(sample):
    print data.dump()
    print data.qty
    print

给予：

[datetime.datetime(2013, 8, 28, 19, 30), None, u'VIRTUALLY_LABELED_CASE', ']', 'quantity=', 127]
- qty: 127
- s1: None
- s2: VIRTUALLY_LABELED_CASE
- timestamp: 2013-08-28 19:30:00
127

[datetime.datetime(2013, 8, 28, 19, 30), u'PPTransMergeNonCon', u'PRIME_BIN_RANDOM_STOW', ']', 'quantity=', 15]
- qty: 15
- s1: PPTransMergeNonCon
- s2: PRIME_BIN_RANDOM_STOW
- timestamp: 2013-08-28 19:30:00
15

[datetime.datetime(2013, 8, 27, 21, 0), u'PPTransFRA1', u'PRIME_BIN_RANDOM_STOW', ']', 'quantity=', 8]
- qty: 8
- s1: PPTransFRA1
- s2: PRIME_BIN_RANDOM_STOW
- timestamp: 2013-08-27 21:00:00
8

dump()将显示所有可用的命名结果值 - 请注意如何使用直接访问数量属性data.qty。这是为您设置的结果名称“数量”的定义"quantity=" + intvalue("qty")。timestamp, s1, 和s2可以类似地访问。（这里还有一点eval，清理它留给读者作为练习。）

编辑：

这是修改后的 pyparsing 解析器，用于处理原始的类似 XML 的原始内容。这些变化真的很小：

from pyparsing import Suppress,QuotedString,Word,alphas,nums,alphanums,Keyword,Optional, ungroup
import datetime

# define UTC timezone for sake of eval
if hasattr(datetime,"timezone"):
    UTC = datetime.timezone(datetime.timedelta(0),"UTC")
else:
    UTC = None

_ = Suppress
evaltokens = lambda s,l,t: eval(''.join(t))

timevalue = 'datetime.datetime' + QuotedString('(', endQuoteChar=')', unquoteResults=False)
replUTC = lambda s,l,t: ''.join(t).replace("< UTC>","UTC").replace("<UTC >","UTC").replace("<UTC>","UTC")
timevalue.setParseAction(replUTC, evaltokens)

strvalue = 'u' + QuotedString("'", unquoteResults=False)
strvalue.setParseAction(evaltokens)

nonevalue = Keyword("None").setParseAction(lambda s,l,t: [None])
intvalue = Word(nums).setParseAction(lambda s,l,t: int(t[0]))

COMMA = Optional(_(","))
LT,GT,LPAR,RPAR,LBRACK,RBRACK = map(Suppress,"<>()[]")

#~ valuedexpr = lambda expr: (Word(alphas) + "(" + "value" + "=" + expr + ")").setParseAction(lambda t: t[4])
valuedexpr = lambda expr: ungroup(LT + (Word(alphas) + "(" + "value" + "=" + expr("value") + ")" + GT).setParseAction(lambda t: t.value))

#~ lineexpr = (_("Aggregate(aggregate_dimension_value_list=[") +
            #~ valuedexpr(timevalue)("timestamp") + COMMA +
            #~ (nonevalue | valuedexpr(strvalue))("s1") + COMMA +
            #~ (nonevalue | valuedexpr(strvalue))("s2") + COMMA +
        #~ "]" + COMMA +
        #~ "quantity=" + intvalue("qty"))

lineexpr = (LT + "Aggregate" + LPAR + "aggregate_dimension_value_list" + "=" + LBRACK +
            valuedexpr(timevalue)("timestamp") + COMMA +
            (nonevalue | valuedexpr(strvalue))("s1") + COMMA +
            (nonevalue | valuedexpr(strvalue))("s2") + 
        RBRACK + COMMA +
        "quantity=" + intvalue("qty") + RPAR + GT)

从您粘贴的文本（其中一些格式错误）中，给出：

['Aggregate', 'aggregate_dimension_value_list', '=', datetime.datetime(2013, 8, 26, 20, 30), u'PPTransCGN1', u'PRIME_BIN_RANDOM_STOW', 'quantity=', 992]
- qty: 992
- s1: PPTransCGN1
- s2: PRIME_BIN_RANDOM_STOW
- timestamp: 2013-08-26 20:30:00
992

['Aggregate', 'aggregate_dimension_value_list', '=', datetime.datetime(2013, 8, 23, 19, 30), None, u'TOTE', 'quantity=', 87]
- qty: 87
- s1: None
- s2: TOTE
- timestamp: 2013-08-23 19:30:00
87

['Aggregate', 'aggregate_dimension_value_list', '=', datetime.datetime(2013, 8, 27, 17, 30), u'PPTransMUC3', u'TOTE', 'quantity=', 14]
- qty: 14
- s1: PPTransMUC3
- s2: TOTE
- timestamp: 2013-08-27 17:30:00
14

['Aggregate', 'aggregate_dimension_value_list', '=', datetime.datetime(2013, 8, 27, 20, 30), u'PPTransEUK5', u'PRIME_BIN_RANDOM_STOW', 'quantity=', 339]
- qty: 339
- s1: PPTransEUK5
- s2: PRIME_BIN_RANDOM_STOW
- timestamp: 2013-08-27 20:30:00
339

['Aggregate', 'aggregate_dimension_value_list', '=', datetime.datetime(2013, 8, 26, 20, 30), u'PPTransCGN1', u'TOTE', 'quantity=', 1731]
- qty: 1731
- s1: PPTransCGN1
- s2: TOTE
- timestamp: 2013-08-26 20:30:00
1731

['Aggregate', 'aggregate_dimension_value_list', '=', datetime.datetime(2013, 8, 26, 19, 30), u'PPTransEUK5', u'TOTE', 'quantity=', 28]
- qty: 28
- s1: PPTransEUK5
- s2: TOTE
- timestamp: 2013-08-26 19:30:00
28

['Aggregate', 'aggregate_dimension_value_list', '=', datetime.datetime(2013, 8, 28, 19, 30), u'PPTransORY1', u'PRIME_BIN_RANDOM_STOW', 'quantity=', 69]
- qty: 69
- s1: PPTransORY1
- s2: PRIME_BIN_RANDOM_STOW
- timestamp: 2013-08-28 19:30:00
69

['Aggregate', 'aggregate_dimension_value_list', '=', datetime.datetime(2013, 8, 26, 19, 30), u'PPTransMAD4', u'PRIME_BIN_RANDOM_STOW', 'quantity=', 47]
- qty: 47
- s1: PPTransMAD4
- s2: PRIME_BIN_RANDOM_STOW
- timestamp: 2013-08-26 19:30:00
47

['Aggregate', 'aggregate_dimension_value_list', '=', datetime.datetime(2013, 8, 26, 21, 0), None, None, 'quantity=', 78]
- qty: 78
- s1: None
- s2: None
- timestamp: 2013-08-26 21:00:00
78

score 0 · Accepted Answer

Aggregate您可以为、DateAggregateDimensionValue和定义最小类StringAggregateDimensionValue，然后依次为eval每一行：

import datetime

# define UTC timezone for sake of eval
if hasattr(datetime,"timezone"):
    UTC = datetime.timezone(datetime.timedelta(0),"UTC")
else:
    UTC = None

# define minimal classes to eval initializers
class AggregateDimensionValue(object):
    def __init__(self, value):
        self.value = value
class StringAggregateDimensionValue(AggregateDimensionValue): pass
class DateAggregateDimensionValue(AggregateDimensionValue): pass
class Aggregate(object):
    def __init__(self, aggregate_dimension_value_list, quantity):
        self.timestamp, self.s1, self.s2 = aggregate_dimension_value_list
        # pull values out of parsed "aggregate" instances
        self.timestamp = self.timestamp.value
        if self.s1 is not None:
            self.s1 = self.s1.value
        if self.s2 is not None:
            self.s2 = self.s2.value
        self.quantity = quantity

eval对输入字符串使用这些最小类：

for line in sample.splitlines():
    if not line.strip():
        continue
    obj = eval(line.strip(' ,'))
    print obj.__dict__

给出：

{'timestamp': datetime.datetime(2013, 8, 28, 19, 30), 's1': None, 'quantity': 127, 's2': u'VIRTUALLY_LABELED_CASE'}
{'timestamp': datetime.datetime(2013, 8, 28, 19, 30), 's1': u'PPTransMergeNonCon', 'quantity': 15, 's2': u'PRIME_BIN_RANDOM_STOW'}
{'timestamp': datetime.datetime(2013, 8, 27, 21, 0), 's1': u'PPTransFRA1', 'quantity': 8, 's2': u'PRIME_BIN_RANDOM_STOW'}

当然，这带有关于使用的所有常见警告eval，例如提防任何可能的恶意代码注入。但我怀疑你自己已经控制了这个输入文件，所以如果你注入自己的恶意代码，你只能怪你自己。

python - 如何将以下多维类似xml的数据解析为数据框

2 回答 2

Related

Reference