我正在从Chris Reeves的系列教程中学习网络抓取技术。真是好东西,你应该看看。
我遇到了教程号中的示例问题。10克里斯解释了与 mySQL 数据库的连接。首先,我遇到了不向数据库中的表提交值的问题。然后在评论中我发现我错过了conn.commit()
哪个视频作者没有包含在他的程序中。我已将这部分代码添加到我的程序中,效果很好,现在看起来像这样:
from threading import Thread
import urllib
import re
import MySQLdb
conn = MySQLdb.connect(host="127.0.0.1",port=3307,user="root",passwd="root",db="stock_data")
query = "INSERT INTO tutorial (symbol) values('AAPL')"
x = conn.cursor()
x.execute(query)
conn.commit()
row = x.fetchall()
它连接到我的本地数据库,并成功地将AAPL添加到列符号下的表教程中。
我的问题始于 Chris 教程的第二部分,您应该在其中添加代码的多线程部分,该部分从外部 .txt 文件中读取 4 个字母的符号并将所有内容添加到同一个数据库中。
现在当我的程序看起来像这样时:
from threading import Thread
import urllib
import re
import MySQLdb
gmap = {}
def th(ur):
base = "http://finance.yahoo.com/q?s="+ur
regex = '<span id="yfs_l84_'+ur.lower()+'">(.+?)</span>'
pattern = re.compile(regex)
htmltext = urllib.urlopen(base).read()
results = re.findall(pattern,htmltext)
try:
gmap[ur] = results [0]
except:
print "got an error"
symbolslist = open("threads/symbols.txt").read()
symbolslist = symbolslist.replace(" ","").split(",")
print symbolslist
threadlist = []
for u in symbolslist:
t = Thread(target=th,args=(u,))
t.start()
threadlist.append(t)
for b in threadlist:
b.join()
conn = MySQLdb.connect(host="127.0.0.1",port=3307,user="root",passwd="root",db="stock_data")
for key in gmap.keys():
print key,gmap[key]
query = "INSERT INTO tutorial (symbol,last) values("
query = query+"'"+key+"',"+gmap[key]+")"
x = conn.cursor()
x.execute(query)
conn.commit()
row = x.fetchall()
这几乎与Chris 示例完全相同(除了我不使用外部登录数据,而是直接在代码中,但这不是问题),我收到所有线程的错误,它们看起来像这样:
Exception in thread Thread-474:
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
self.run()
File "C:\Python27\lib\threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "threads/threads2.py", line 12, in th
htmltext = urllib.urlopen(base).read()
File "C:\Python27\lib\urllib.py", line 87, in urlopen
return opener.open(url)
File "C:\Python27\lib\urllib.py", line 208, in open
return getattr(self, name)(url)
File "C:\Python27\lib\urllib.py", line 345, in open_http
h.endheaders(data)
File "C:\Python27\lib\httplib.py", line 969, in endheaders
self._send_output(message_body)
File "C:\Python27\lib\httplib.py", line 829, in _send_output
self.send(msg)
File "C:\Python27\lib\httplib.py", line 791, in send
self.connect()
File "C:\Python27\lib\httplib.py", line 772, in connect
self.timeout, self.source_address)
File "C:\Python27\lib\socket.py", line 571, in create_connection
raise err
IOError: [Errno socket error] [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
正如我所说,这只是 Thread-474 的一个错误,但我在 IDE 中的多个线程中得到它,对于 Thread-441、Thread-390、Thread-391 等也是如此……
我错过了什么?它是在代码中还是在我的 MySql 服务器设置中?因为根据克里斯示例中的所有内容,它应该可以工作
帮助任何人?