1

我一般是编程新手，所以我希望这不是一个愚蠢的问题。我已经用谷歌搜索并花了最后 4 个小时试图解决这个问题，但我真的很感激我应该尝试解决这个问题的建议/步骤。谢谢！

到目前为止，这是我对蜘蛛的了解： from scrapy.spider import BaseSpider from scrapy.selector import HtmlXPathSelector from tutorial.items import TutorialItem from scrapy.http import FormRequest, Request

class LoginSpider(BaseSpider):
    name = 'pinterest'
    start_urls = ['https://www.pinterest.com/login/']

    def parse(self, response):
        return FormRequest.from_response(response,
                    formdata={'username_or_email': '...', 'password': '...'},
                    callback=self.after_login, dont_filter = True)

    def after_login(self, response):
        print response.url

据我了解，Scrapy 会自动处理 cookie，因此 CSRF 令牌会通过。我在设置中将 COOKIES_ENABLED 和 COOKIES_DEBUG 设置为 True：

SPIDER_MIDDLEWARES = {'scrapy.contrib.downloadermiddleware.cookies.CookiesMiddleware':     
700,}
USER_AGENT = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like 
Gecko) Chrome/29.0.1547.66 Safari/537.36"
COOKIES_ENABLED = True
COOKIES_DEBUG = True

这是调试的输出：

2013-09-27 11:11:42-0700 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
2013-09-27 11:11:43-0700 [pinterest] DEBUG: Received cookies from: <200 https://
www.pinterest.com/login/>
        Set-Cookie: csrftoken=1FBJIzKqxH7XQ5tdXNtUIDHEJsL1210K; Domain=.pinteres
t.com; expires=Fri, 26-Sep-2014 18:11:46 GMT; Max-Age=31449600; Path=/
        Set-Cookie: _pinterest_sess="eJwr9UotN47SN0rUjzJ3ciwo109N8UixNPM1znK0tY8
vycxNtfUN8TXxdfEt9wsJLfdLt7VVK04tLs5MsfXMyjb0c/c0AIpX+Ia4ZfpmBeX4uqSbRFYlG0SFuFb
4ZjlWRLkHGkZWuRp6AvUBAEY1IrA="; Domain=.pinterest.com; expires=Mon, 22-Sep-2014
18:11:46 GMT; Max-Age=31103999; Path=/
2013-09-27 11:11:43-0700 [pinterest] DEBUG: Crawled (200) <GET https://www.pinte
rest.com/login/> (referer: None)
2013-09-27 11:11:43-0700 [pinterest] DEBUG: Sending cookies to: <POST https://ww
w.pinterest.com/login/>
        Cookie: csrftoken=1FBJIzKqxH7XQ5tdXNtUIDHEJsL1210K; _pinterest_sess="eJw
r9UotN47SN0rUjzJ3ciwo109N8UixNPM1znK0tY8vycxNtfUN8TXxdfEt9wsJLfdLt7VVK04tLs5MsfX
Myjb0c/c0AIpX+Ia4ZfpmBeX4uqSbRFYlG0SFuFb4ZjlWRLkHGkZWuRp6AvUBAEY1IrA="
2013-09-27 11:11:43-0700 [pinterest] DEBUG: Redirecting (302) to <GET http://www
.pinterest.com/csrf_error/> from <POST https://www.pinterest.com/login/>
2013-09-27 11:11:43-0700 [pinterest] DEBUG: Sending cookies to: <GET http://www.
pinterest.com/csrf_error/>
        Cookie: csrftoken=1FBJIzKqxH7XQ5tdXNtUIDHEJsL1210K; _pinterest_sess="eJw
r9UotN47SN0rUjzJ3ciwo109N8UixNPM1znK0tY8vycxNtfUN8TXxdfEt9wsJLfdLt7VVK04tLs5MsfX
Myjb0c/c0AIpX+Ia4ZfpmBeX4uqSbRFYlG0SFuFb4ZjlWRLkHGkZWuRp6AvUBAEY1IrA="
2013-09-27 11:11:44-0700 [pinterest] DEBUG: Crawled (200) <GET http://www.pinter
est.com/csrf_error/> (referer: https://www.pinterest.com/login/)
http://www.pinterest.com/csrf_error/

问题是设置cookie并发送到登录页面后，我仍然收到CSRF错误并被重定向。我做错了什么我无法像浏览器一样模拟登录过程吗？我尝试将用户代理设置为 iPhone 并获得代码 200 并且没有重定向，但 response.url 显示“ https://www.pinterest.com/login/?next=/login/ ”所以它仍然没有登录适当地。

非常感谢我能得到的所有帮助。谢谢！

4

1 回答 1

1

看起来非移动登录页面使用 XHR 请求来执行登录。您可以尝试深入研究 XHR 请求，也许还有 javascript 代码，以查看您必须做什么才能在 scrapy 中重现请求。

但是，正如您所指出的，有一个通过更改用户代理启用的移动登录页面。

后一种方法的问题是电子邮件的字段email不是username_or_email.

这是有细微变化的蜘蛛：

from scrapy.http import FormRequest
from scrapy.spider import BaseSpider


class LoginSpider(BaseSpider):
    name = 'pinterest'
    start_urls = ['https://www.pinterest.com/login/']
    # you can set the user agent either in the settings or the spider
    user_agent = ('Mozilla/5.0 (iPhone; CPU iPhone OS 5_0 like Mac OS X) '
                  'AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 '
                  'Mobile/9A334 Safari/7534.48.3')

    def parse(self, response):
        data = {'email': 'XXX@xxx', 'password': 'xxx'}
        # no need for dont_filter
        return FormRequest.from_response(response, formdata=data, callback=self.after_login)

    def after_login(self, response):
        print response.url

输出：

$ scrapy runspider pinterest.py
2013-09-28 19:16:58-0400 [scrapy] INFO: Scrapy 0.16.5 started (bot: scrapybot)
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware, HttpCompressionMiddleware, ChunkedTransferMiddleware, DownloaderStats
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Enabled item pipelines: 
2013-09-28 19:16:58-0400 [pinterest] INFO: Spider opened
2013-09-28 19:16:58-0400 [pinterest] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023
2013-09-28 19:16:58-0400 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
2013-09-28 19:17:01-0400 [pinterest] DEBUG: Crawled (200) <GET https://www.pinterest.com/login/> (referer: None)
2013-09-28 19:17:09-0400 [pinterest] DEBUG: Redirecting (302) to <GET http://www.pinterest.com/> from <POST https://www.pinterest.com/login/?next=%2Flogin%2F>
2013-09-28 19:17:09-0400 [pinterest] DEBUG: Redirecting (302) to <GET http://www.pinterest.com/join/discover/> from <GET http://www.pinterest.com/>
2013-09-28 19:17:10-0400 [pinterest] DEBUG: Crawled (200) <GET http://www.pinterest.com/join/discover/> (referer: https://www.pinterest.com/login/)
http://www.pinterest.com/join/discover/
于 2013-09-28T23:21:19.407 回答