11

在维基百科中，您可以找到一些有趣的数据进行排序、过滤、...

这是一个 wikitable 的示例

{| class="wikitable sortable"
|-
! Model !! Mhash/s !! Mhash/J !! Watts !! Clock !! SP !! Comment
|-
| ION || 1.8 || 0.067 || 27 ||  || 16 || poclbm;  power consumption incl. CPU
|-
| 8200 mGPU || 1.2 || || || 1200 || 16 || 128 MB shared memory, "poclbm -w 128 -f 0"
|-
| 8400 GS || 2.3 || || ||  ||  || "poclbm -w 128"
|-
|}

我正在寻找一种将此类数据导入 Python Pandas DataFrame 的方法

4

4 回答 4

13

这是一个使用py-wikimarkup和PyQuery从 wikimarkup 字符串中提取所有表作为 pandas DataFrames 的解决方案，忽略非表内容。

import wikimarkup
import pandas as pd
from pyquery import PyQuery

def get_tables(wiki):
    html = PyQuery(wikimarkup.parse(wiki))
    frames = []
    for table in html('table'):
        data = [[x.text.strip() for x in row]
                for row in table.getchildren()]
        df = pd.DataFrame(data[1:], columns=data[0])
        frames.append(df)
    return frames

给定以下输入，

wiki = """
=Title=

Description.

{| class="wikitable sortable"
|-
! Model !! Mhash/s !! Mhash/J !! Watts !! Clock !! SP !! Comment
|-
| ION || 1.8 || 0.067 || 27 ||  || 16 || poclbm;  power consumption incl. CPU
|-
| 8200 mGPU || 1.2 || || || 1200 || 16 || 128 MB shared memory, "poclbm -w 128 -f 0"
|-
| 8400 GS || 2.3 || || || || || "poclbm -w 128"
|-
|}

{| class="wikitable sortable"
|-
! A !! B !! C
|-
| 0
| 1
| 2
|-
| 3
| 4
| 5
|}
"""

get_tables返回以下数据帧。

       Model Mhash/s Mhash/J Watts Clock  SP                                     Comment
0        ION     1.8   0.067    27        16        poclbm;  power consumption incl. CPU
1  8200 mGPU     1.2                1200  16  128 MB shared memory, "poclbm -w 128 -f 0"
2    8400 GS     2.3                                                     "poclbm -w 128"

 

   A  B  C
0  0  1  2
1  3  4  5
于 2013-03-31T04:04:52.340 回答
4

你可以直接使用熊猫。像这样的东西...

pandas.read_html(url, attrs={"class": "wikitable"})

于 2018-11-08T19:22:27.667 回答
2

用于re进行一些预处理，然后用于read_csv将其转换为DataFrame：

table = """{| class="wikitable sortable"
|-
! Model !! Mhash/s !! Mhash/J !! Watts !! Clock !! SP !! Comment
|-
| ION || 1.8 || 0.067 || 27 ||  || 16 || poclbm;  power consumption incl. CPU
|-
| 8200 mGPU || 1.2 || || || 1200 || 16 || 128 MB shared memory, "poclbm -w 128 -f 0"
|-
| 8400 GS || 2.3 || || ||  ||  || "poclbm -w 128"
|-
|}"""

data = StringIO(re.sub("^\|.|^!.", "", table.replace("|-\n", ""), flags=re.MULTILINE))
df = pd.read_csv(data, delimiter="\|\||!!", skiprows=1)

输出：

       Model    Mhash/s   Mhash/J   Watts   Clock    SP                                       Comment
0        ION         1.8    0.067      27            16          poclbm;  power consumption incl. CPU
1  8200 mGPU         1.2                     1200    16    128 MB shared memory, "poclbm -w 128 -f 0"
2    8400 GS         2.3                                                              "poclbm -w 128"
于 2013-03-31T02:54:39.363 回答
1

已编辑 - 下面的完整答案。我没有安装 Panda，所以让我知道这是否适合你。

from pandas import *

wikitable = '''
{| class="wikitable sortable"
|-
! Model !! Mhash/s !! Mhash/J !! Watts !! Clock !! SP !! Comment
|-
| ION || 1.8 || 0.067 || 27 ||  || 16 || poclbm;  power consumption incl. CPU
|-
| 8200 mGPU || 1.2 || || || 1200 || 16 || 128 MB shared memory, "poclbm -w 128 -f 0"
|-
| 8400 GS || 2.3 || || ||  ||  || "poclbm -w 128"
|-
|}'''
rows = wikitable.split('|-')
header = []
table = []
for i in rows:
     line = i.strip()
     if line.startswith('!'):
         header = line.split('!!')
     elif line.startswith('|') and line.strip() != '|}':
         table.append(line[2:].split('||'))

data = {}
for i in range(len(header) - 1):
    col = []
    for row in table:
        col.append(row[i])
    data[header[i]] = col

print(data)

df = DataFrame(data)
于 2013-03-30T23:24:38.060 回答