python - eventlet 抓取和保存文件的问题

Question

我可以使用 evenlet 从网站上删除 img 但未能将它们保存到国内目录中。以下是代码。有谁熟悉 tasklets 模型中的 I/O 操作？谢谢

import pyquery
import eventlet
from eventlet.green import urllib2

#fetch img urls............ works fine

print "loading page..."
html=urllib2.urlopen("http://www.meinv86.com/meinv/yuanchuangmeinvzipai/").read()
print "Parsing urls..."
d=pyquery.PyQuery(html)
count=0
urls=[]
url=''
for i in d('img'):
 count=count+1
 print i.attrib["src"]
 urls.append(i.attrib["src"])


def fetch(url):
 try:
  print "start feteching %s" %(url)
  urlfile = urllib2.urlopen(url)
  size=int(urlfile.headers['content-length'])
  print 'downloading %s, total file size: %d' %(url,size)
  data = urlfile.read()
  print 'download complete - %s' %(url)

##########################################
#file save just won't work

  f=open("/head2/"+url+".jpg","wb")
  f.write(body)  
  f.close()
  print "file saved"
##########################################  

  return data

 except:
  print "fail to download..."




pool = eventlet.GreenPool()

for body in pool.imap(fetch, urls):
  print "done"

Answer 1

确保它url适合作为文件名，例如：

import hashlib
import os

def url2filename(url, ext=''):
    return hashlib.md5(url).hexdigest() + ext # anything that removes '\/'

# ...
with open(os.path.join("/head2", url2filename(url, '.jpg')), 'wb') as f:
     f.write(body)
print "file saved"

注意：您可能不想将文件写入顶级目录，例如'/head2'.

你也可以考虑urllib.urlretrieve()。