4

出于学术和性能的考虑，鉴于这种爬网递归网络爬网功能（仅在给定域内爬网），使其迭代运行的最佳方法是什么？目前，当它运行时，当它完成时，python 已经使用超过 1GB 的内存，这对于在共享环境中运行是不可接受的。

   def crawl(self, url):
    "Get all URLS from which to scrape categories."
    try:
      links = BeautifulSoup(urllib2.urlopen(url)).findAll(Crawler._match_tag)
    except urllib2.HTTPError:
      return
    for link in links:
      for attr in link.attrs:
        if Crawler._match_attr(attr):
          if Crawler._is_category(attr):
            pass
          elif attr[1] not in self._crawled:
            self._crawled.append(attr[1])
            self.crawl(attr[1])
4

4 回答 4

12

使用 BFS 而不是递归爬行 (DFS)：http ://en.wikipedia.org/wiki/Breadth_first_search

您可以为 BFS 队列使用外部存储解决方案（例如数据库）来释放 RAM。

算法是：

//pseudocode:
var urlsToVisit = new Queue(); // Could be a queue (BFS) or stack(DFS). (probably with a database backing or something).
var visitedUrls = new Set(); // List of visited URLs.

// initialization:
urlsToVisit.Add( rootUrl );

while(urlsToVisit.Count > 0) {
  var nextUrl = urlsToVisit.FetchAndRemoveNextUrl();
  var page = FetchPage(nextUrl);
  ProcessPage(page);
  visitedUrls.Add(nextUrl);
  var links = ParseLinks(page);
  foreach (var link in links)
     if (!visitedUrls.Contains(link))
        urlsToVisit.Add(link); 
}
于 2009-03-29T09:16:53.920 回答
5

您可以将要抓取的新 URL 放入队列中，而不是递归。然后运行直到队列为空而不递归。如果将队列放入文件中，则几乎不使用任何内存。

于 2009-03-29T09:54:17.650 回答
2

@Mehrdad - 感谢您的回复，您提供的示例简洁易懂。

解决方案：

  def crawl(self, url):
    urls = Queue(-1)
    _crawled = []

    urls.put(url)

    while not urls.empty():
      url = urls.get()
      try:
        links = BeautifulSoup(urllib2.urlopen(url)).findAll(Crawler._match_tag)
      except urllib2.HTTPError:
        continue
      for link in links:
        for attr in link.attrs:
          if Crawler._match_attr(attr):
            if Crawler._is_category(attr):
              continue
            else:
              Crawler._visit(attr[1])
              if attr[1] not in _crawled:
                urls.put(attr[1])
于 2009-03-30T12:33:44.733 回答
0

links只需将其用作队列，您就可以很容易地做到这一点：

def get_links(url):
    "Extract all matching links from a url"
    try:
        links = BeautifulSoup(urllib2.urlopen(url)).findAll(Crawler._match_tag)
    except urllib2.HTTPError:
        return []

def crawl(self, url):
    "Get all URLS from which to scrape categories."
    links = get_links(url)
    while len(links) > 0:
        link = links.pop()
        for attr in link.attrs:
            if Crawler._match_attr(attr):
                if Crawler._is_category(attr):
                    pass
            elif attr[1] not in self._crawled:
                self._crawled.append(attr[1])
                # prepend the new links to the queue
                links = get_links(attr[1]) + links

当然，这并不能解决内存问题......

于 2009-03-29T19:26:10.257 回答