1

我正在尝试为测试网站编写我的第一个网络爬虫。它涉及登录，我遵循了有关如何处理此类情况的教程。

import requests
from lxml import html



payload = {
"email": "test_test@test.com",
"password": "123qweasd",
"_token": "3ow4dl7COwnRHa8a6nvNGp4eLkF3wQapT3otGXjR"
 }

rs = requests.session()

login_url = 'https://cloud.webscraper.io/login'
log_page = rs.get(login_url)

tree = html.fromstring(log_page.content)
auth_token = list(set(tree.xpath("//input[@name='_token']/@value")))[0]

login = rs.post(login_url,data=payload, headers=dict(referer=login_url))

url = "https://cloud.webscraper.io/sitemaps"
result = rs.get(url, headers=dict(referer=url))

tree = html.fromstring(result.text)
sidebar_cat = tree.xpath('//*[@id="main-menu-inner"]/ul')

print(sidebar_cat)

我希望这个脚本列出侧边栏中的类别。似乎脚本每次都返回并清空列表。电流输出为

"[] 
Process finished with exit code 0"
4

2 回答 2

1

您已经提取_token了值，但使用了硬编码值。尝试将提取的值传递给payload：

import requests
from lxml import html

rs = requests.session()

login_url = 'https://cloud.webscraper.io/login'
log_page = rs.get(login_url)

tree = html.fromstring(log_page.content)
auth_token = tree.xpath("//input[@name='_token']/@value")[0]

payload = {
    "email": "test_test@test.com",
    "password": "123qweasd",
    "_token": auth_token
 }

login = rs.post(login_url,data=payload, headers=dict(referer=login_url))

url = "https://cloud.webscraper.io/sitemaps"
result = rs.get(url, headers=dict(referer=url))

tree = html.fromstring(result.text)
sidebar_cat = tree.xpath('//*[@id="main-menu-inner"]/ul')

print(sidebar_cat)
于 2019-01-21T15:30:39.410 回答
0

尝试打印“结果”并在之后暂时注释掉所有内容。这样您就可以知道请求部分是否有错误，或者问题是否来自响应处理。如果响应按预期打印，则对最后三行代码进行故障排除。如果不是，则对代码的请求部分进行故障排除。

于 2019-01-21T15:29:53.183 回答