感谢您在以下方面的帮助:我需要读取一个大型 XML 文件并将其转换为 CSV。
我有两个函数应该做同样的事情,只有一个(function1)使用 iterparse(因为我需要处理大约 2GB 的文件)和另一个不使用(function2)。
Function2 对于相同的 XML 文件(但最大 150 MB)工作得非常好,但在该大小之后,它会因内存而失败。
我遇到的问题是,尽管代码(对于 function1)没有给出错误,但它会丢失一些孩子(这是一个大问题!)。另一方面,Function2 读取所有子项,并且不会“松动”或失败。
问:你能在function1的代码中看到一些孩子会丢失(或阅读不正确,或被忽略)的原因吗?
注意 1:我准备好发送一个 50 KB 的 XML 示例,以备不时之需。
注2:变量“nchil_count”只是为了计算孩子的数量。
代码(功能1):
def function1 ():
# This function uses Iterparse
# Doesn't give errors but looses some children. Why?
# prints output to csv file, WCEL.csv
from xml.etree.cElementTree import iterparse
fname = "C:\Leonardo\Input data\Xml input data\NetactFiles\Netact_3g_rnc11_t1.xml"
# ELEMENT_LIST = ["WCEL"]
# Delete contents from exit file
open("C:\Leonardo\Input data\Xml input data\WCEL.csv", 'w').close()
# Open exit file
with open("C:\Leonardo\Input data\Xml input data\WCEL.csv", "a") as exit_file:
with open(fname) as xml_doc:
context = iterparse(xml_doc, events=("start", "end"))
context = iter(context)
event, root = context.next()
for event, elem in context:
if event == "start" and elem.tag == "{raml20.xsd}managedObject":
# if event == "start":
if elem.get('class') == 'WCEL':
print elem.attrib
# print elem.tag
element = elem.getchildren()
nchil_count = 0
for child in element:
if child.tag == "{raml20.xsd}p":
nchil_count = nchil_count + 1
# print child.tag
# print child.attrib
val = child.text
# print val
val = str (val)
exit_file.write(val + ",")
exit_file.write('\n')
print nchil_count
elif event == "end" and elem.tag == "{raml20.xsd}managedObject":
# Clear Memory
root.clear()
xml_doc.close()
exit_file.close()
return ()
代码(功能2):
def function2 (xmlFile):
# Using Element Tree
# Successful
# Works well with files of 150 MB, like an XML (RAML) RNC export from Netact (1 RNC only)
# It fails with huge files due to Memory
import xml.etree.cElementTree as etree
import shutil
with open("C:\Leonardo\Input data\Xml input data\WCEL.csv", "a") as exit_file:
# Populate the values per cell:
tree = etree.parse(xmlFile)
for value in tree.getiterator(tag='{raml20.xsd}managedObject'):
if value.get('class') == 'WCEL':
print value.attrib
element = value.getchildren()
nchil_count = 0
for child in element:
if child.tag == "{raml20.xsd}p":
nchil_count = nchil_count + 1
# print child.tag
# print child.attrib
val = child.text
# print val
val = str (val)
exit_file.write(val + ",")
exit_file.write('\n')
print nchil_count
exit_file.close() ## File closing after writing.
return ()