0

我正在尝试使用以下代码。但我得到了第一个选定日期的 5 倍相同数据。

import re
import datetime
import mechanicalsoup

def get_EminiTable(soup):
    allValues = []
    colnames = soup.find('thead').get_text()
    allValues.append([i for i in colnames.split('\n') if len(i) > 1])
    lnr = 0
    for line in soup.tbody.find_all('tr'):
        allValues.append([i for i in line.get_text().split('\n') if len(i) > 0])
        if 'UNCH' in allValues:
            allValues.pop()
        if lnr > 1: break
        lnr += 1
    return allValues

def get_settldays(soup):
    settlDays = soup.find('select', id="cmeTradeDate").get
    settlDays = re.findall('\d\d/\d\d/\d\d\d\d',str(settlDays))
    return [datetime.datetime.strptime(adat, '%m/%d/%Y') for adat in settlDays]


url =  "http://www.cmegroup.com/trading/equity-index/us-index/e-mini-sandp500_quotes_settlements_futures.html"
browser = mechanicalsoup.StatefulBrowser()
if str(browser.open(url)) != '<Response [200]>':
    print('Error')
    quit()
soup = browser.get_current_page()
settlDays = get_settldays(soup)
for adate in settlDays:
    form = browser.select_form('form[id="quotesoptionsform1"]')
    form.set("tradeDate",adate.__format__('%m/%d/%Y'))
    browser.submit_selected()
    soup = browser.get_current_page()
    tabvals = get_EminiTable(soup)
    print(adate)
    for each in tabvals:
        print(each)
browser.session.close()
browser.close()

获取不同表格的正确日期的任何想法，或者它是机械汤中的错误。

4

1 回答 1

0

我怀疑原因是因为这个表单是由 JavaScript 处理的，而不是 HTTP，所以用 HTTP 请求提交表单（这就是 MechanicalSoup 的工作方式）实际上并没有做任何事情。

MechanicalSoup 使用以下语法创建请求 URL：

http://www.cmegroup.com/trading/equity-index/us-index/e-mini-sandp500_quotes_settlements_futures.html?action=Submit&tradeDate=12%2F15%2F2017

如果单击此链接，您可以看到查询字符串&action=Submit&tradeDate=12%2F15%2F2017被忽略。而如果我们在真实的网络浏览器中手动设置交易日期，我们会看到它附加#tradeDate=12/15/2017到 URL。

即使我从 Web 浏览器中获取该 URL 并使用 MechanicalSoup 打开它，它也无法正确显示，因为未处理 JavaScript。您可以通过以下方式修改最后一个 for 循环来看到这一点：

for adate in settlDays:
    # Open the URL for each date directly
    date = adate.__format__('%m/%d/%Y')
    resp = browser.open(url + "#tradeDate={}".format(date))

    # Make sure we constructed the URL correctly
    print(resp.url)

    # Print the date that is being displayed
    soup = browser.get_current_page()
    print(soup.find('select', id='cmeTradeDate').find('option', attrs={'selected': 'selected'}).text)

输出看起来像（忽略语法突出显示......）：

http://www.cmegroup.com/trading/equity-index/us-index/e-mini-sandp500_quotes_settlements_futures.html#tradeDate=12/21/2017
Thursday, 21 Dec 2017 (Prelim)
http://www.cmegroup.com/trading/equity-index/us-index/e-mini-sandp500_quotes_settlements_futures.html#tradeDate=12/20/2017
Thursday, 21 Dec 2017 (Prelim)
http://www.cmegroup.com/trading/equity-index/us-index/e-mini-sandp500_quotes_settlements_futures.html#tradeDate=12/19/2017
Thursday, 21 Dec 2017 (Prelim)
http://www.cmegroup.com/trading/equity-index/us-index/e-mini-sandp500_quotes_settlements_futures.html#tradeDate=12/18/2017
Thursday, 21 Dec 2017 (Prelim)
http://www.cmegroup.com/trading/equity-index/us-index/e-mini-sandp500_quotes_settlements_futures.html#tradeDate=12/15/2017
Thursday, 21 Dec 2017 (Prelim)

除非有人知道使用 HTTP 处理此页面的替代方法，否则我认为您最好的选择是使用可以与 JavaScript 交互的工具，例如模拟真实浏览器的 Selenium。

于 2017-12-21T22:22:58.893 回答