2

我终于发现 BS4 不再像 BS3 那样使用“标记按摩”。但是我仍然需要一种类似的方式来处理不需要的 document.write。您会在 BS3 中执行以下操作，但在 BS4 中如何执行？

# Javascript code in ths page generates HTML markup  
# that isn't parsed correctly by BeautifulSoup.
# To avoid this problem, all document.write fragments are removed
my_massage = copy(BeautifulSoup.MARKUP_MASSAGE)
my_massage.append((re.compile(u"document.write(.+);"), lambda match: ""))
my_massage.append((re.compile(u'alt=".+">'), lambda match: ">"))

另外，由于 BS4 BeautifulSoup 构造函数不再支持 markupmassage 参数，我应该在我的程序中的哪个位置处理 document.write 问题？我假设这是问题所在，因为我只是想打印出表格标记，并且在运行 windmill 时遇到线程异常。

这就是我的代码的样子：

#!/usr/bin/env python
# Generated by the windmill services transformer
#from windmill.authoring import WindmillTestClient
from bs4 import BeautifulSoup

import re, urlparse
from copy import copy
from windmill.authoring import setup_module, WindmillTestClient
from windmill.conf import global_settings
import sys


global_settings.START_CHROME = True # This makes it use Firefox
setup_module(sys.modules[__name__])


def get_table_info(client):
        """
    Parse HTML page and extract featured image name and link
    """
    # Get Javascript updated HTML page
    client.waits.forElement(xpath=u"//table[@id='trades']",
                        timeout=40000)
    response = client.commands.getPageText()
    assert response['status']
    assert response['result']

    # Create soup from HTML page and get desired information
    soup = BeautifulSoup(response['result'])

    table_info = soup.select("#trades")
    return table_info


def test_scrape():
    """
    Scrape site
    """

    # Open main gallery page
    client = WindmillTestClient(__name__)
    client.open(url='http://www.zulutrade.com/trader/128391')


    table_info = {}
    table_info = get_table_info(client)


    print table_info



test_scrape()
4

1 回答 1

0

你不需要告诉 BeautifulSoup 如何按摩标记——你可以在将它提供给BeautifulSoup构造函数之前自己修改它：

html = response['result']
html = re.sub(r'document.write(.+);', '', html)
html = re.sub(r'alt=".+">', '>', html)
soup = BeautifulSoup(html)
于 2014-02-02T15:21:00.997 回答