专家们,
使用谷歌应用引擎检索即使是最基本的网站似乎也非常具有挑战性!
就我而言,我想通过以下网址检索网站:
http://tdbank.mortgagewebcenter.com/PowerSite/CheckRates.aspx/Index/9809
我想接受所有 cookie,然后 POST 对此 url 的响应(这是一个简单的表单帖子):
http://tdbank.mortgagewebcenter.com/PowerSite/CheckRates.aspx/Search
我想发布的字符串是:
'POSTDATA': 'Q585=1&Q2926=1&Q586=200000&Q587=240000&Q588=&Q9166=07071&Q591=1&Q592=1&Q594=3&searchButton=Search'
我遇到的问题是网页在我的网络浏览器上声明“未启用 Cookie”。
正如您从下面的代码中看到的那样,我尝试手动添加 cookie,但这并不成功。
请帮忙!-托德
import cgi
import webapp2
import gzip
import StringIO
from google.appengine.api import users
from google.appengine.api import urlfetch
from BeautifulSoup import BeautifulSoup
class MainPage(webapp2.RequestHandler):
def get(self):
# self.response.headers['Content-Type'] = 'text/html'
url = "http://tdbank.mortgagewebcenter.com/PowerSite/CheckRates.aspx/Index/9809"
result = urlfetch.fetch(url,
headers={'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language': 'en-us',
'Accept-Encoding': 'gzip',
'Connection': 'keep-alive'})
cookie = result.headers.get('set-cookie')
input_text = {'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language': 'en-us',
'Accept-Encoding': 'gzip',
'Connection': 'keep-alive',
'Content-Type': 'application/x-www-form-urlencoded',
'Content-Length': '97',
'POSTDATA': 'Q585=1&Q2926=1&Q586=200000&Q587=240000&Q588=&Q9166=07071&Q591=1&Q592=1&Q594=3&searchButton=Search',
'SiteProfile':'ProfileId=9809',
's_sess':'c_m=undefinedfeedity.comfeedity.com; s_sq=; s_cc=true;;',
'bhCookieSaveSess':'1',
'bhPrevResults':'bhjs=1&bhrf=http://www.google.com/'}
input_text['set-cookie'] = cookie
## self.response.out.write(input_text)
url2 = "http://tdbank.mortgagewebcenter.com/PowerSite/CheckRates.aspx/Search"
result2 = urlfetch.fetch(url2, method='POST',headers=input_text)
# self.response.out.write(result.headers)
f = StringIO.StringIO(result2.content)
c = gzip.GzipFile(fileobj=f)
content = c.read()
self.response.out.write(content)
# self.response.out.write(result.content)
app = webapp2.WSGIApplication([('/', MainPage)],
debug=True)
yaml 文件:
application: fimrates
version: 1
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /.*
script: fimrates.app