regex - 如何将抓取的多行内容转换为列表？

Question

我试图将抓取的内容转换为数据操作列表，但出现以下错误：TypeError: 'NoneType' object is not callable

#! /usr/bin/python

from urllib import urlopen
from BeautifulSoup import BeautifulSoup
import os
import re

# Copy all of the content from the provided web page
webpage = urlopen("http://www.optionstrategist.com/calculators/free-volatility-    data").read()

# Grab everything that lies between the title tags using a REGEX
preBegin = webpage.find('<pre>') # Locate the pre provided
preEnd = webpage.find('</pre>') # Locate the /pre provided

# Copy the content between the pre tags
voltable = webpage[preBegin:preEnd] 

# Pass the content to the Beautiful Soup Module
raw_data = BeautifulSoup(voltable).splitline()

score 0 · Accepted Answer

从第一个pre元素中获取文本：

#!/usr/bin/env python
from urllib2 import urlopen
from BeautifulSoup import BeautifulSoup

url = "http://www.optionstrategist.com/calculators/free-volatility-data"
soup = BeautifulSoup(urlopen(url))
print soup.pre.string

要提取带有数据的行：

from itertools import dropwhile

lines = soup.pre.string.splitlines()
# drop lines before the data table header
lines = dropwhile(lambda line: not line.startswith("Symbol"), lines)
# extract lines with data
lines = (line for line in lines if '%ile' in line)

现在每一行都包含固定列格式的数据。您可以使用切片和/或正则表达式来解析/验证每行中的各个字段。

score 0 · Accepted Answer

代码非常简单。这是 BeautifulSoup4 的代码：

# Find all <pre> tag in the HTML page
preTags = webpage.find_all('pre')

for tag in preTags:
    # Get the text inside the tag
    print(tag.get_text())

参考：

regex - 如何将抓取的多行内容转换为列表？

2 回答 2

Related

Reference