0

我试图通过以下代码进行网页抓取：

from bs4 import BeautifulSoup
import requests
import pandas as pd

page = requests.get('https://www.google.com/search?q=phagwara+weather')
soup = BeautifulSoup(page.content, 'html-parser')
day = soup.find(id='wob_wc')

print(day.find_all('span'))

但不断收到以下错误：

 File "C:\Users\myname\Desktop\webscraping.py", line 6, in <module>
    soup = BeautifulSoup(page.content, 'html-parser')
  File "C:\Users\myname\AppData\Local\Programs\Python\Python38-32\lib\site-packages\bs4\__init__.py", line 225, in __init__
    raise FeatureNotFound(
bs4.FeatureNotFound: Couldn't find a tree builder with the features you requested: html-parser. Do you need to install a parser library?

我安装了lxml 和 html5lib这个问题仍然存在。

4

3 回答 3

2

您需要将 'html-parser' 更改为soup = BeautifulSoup(page.content, 'html.parser')

于 2020-03-29T11:18:58.190 回答
0

您需要提及标签，所以soup.find(id="wob_wc")应该是soup.find("div", id="wob_wc"))

和解析器名称html.parser没有html-parser区别的是点。

同样默认情况下，Google通常会给你一个响应，200以防止你知道你是否被阻止。通常你必须检查r.content。

我已经包括了headers，现在它可以工作了。

import requests
from bs4 import BeautifulSoup

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0'}
r = requests.get(
    "https://www.google.com/search?q=phagwara+weather", headers=headers)
soup = BeautifulSoup(r.content, 'html.parser')

print(soup.find("div", id="wob_wc"))
于 2020-03-29T11:55:52.200 回答
0

实际上，您不需要遍历整个事物："div #wob_wc"，因为当前位置、天气、日期、温度、降水量、湿度和风由一个元素组成，不要在其他任何地方重复，您可以使用select()orfind()代替。

如果你想迭代某些东西，那么迭代温度预测是一个好主意，例如：

for forecast in soup.select('.wob_df'):
  high_temp = forecast.select_one('.vk_gy .wob_t:nth-child(1)').text
  low_temp = forecast.select_one('.QrNVmd .wob_t:nth-child(1)').text
  print(f'High: {high_temp}, Low: {low_temp}')

'''
High: 67, Low: 55
High: 65, Low: 56
High: 68, Low: 55
'''

查看SelectorGadget Chrome 扩展程序，您可以在其中CSS通过单击浏览器中所需的元素来获取选择器。CSS选择器参考。

在线 IDE 中的代码和完整示例：

from bs4 import BeautifulSoup
import requests, lxml

headers = {
  "User-Agent":
  "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}

params = {
  "q": "phagwara weather",
  "hl": "en",
  "gl": "us"
}

response = requests.get('https://www.google.com/search', headers=headers, params=params)
soup = BeautifulSoup(response.text, 'lxml')

weather_condition = soup.select_one('#wob_dc').text
tempature = soup.select_one('#wob_tm').text
precipitation = soup.select_one('#wob_pp').text
humidity = soup.select_one('#wob_hm').text
wind = soup.select_one('#wob_ws').text
current_time = soup.select_one('#wob_dts').text

print(f'Weather condition: {weather_condition}\n'
      f'Tempature: {tempature}°F\n'
      f'Precipitation: {precipitation}\n'
      f'Humidity: {humidity}\n'
      f'Wind speed: {wind}\n'
      f'Current time: {current_time}\n')

for forecast in soup.select('.wob_df'):
  day = forecast.select_one('.QrNVmd').text
  weather = forecast.select_one('img.uW5pk')['alt']
  high_temp = forecast.select_one('.vk_gy .wob_t:nth-child(1)').text
  low_temp = forecast.select_one('.QrNVmd .wob_t:nth-child(1)').text
  print(f'Day: {day}\nWeather: {weather}\nHigh: {high_temp}, Low: {low_temp}\n')

---------
'''
Weather condition: Partly cloudy
Temperature: 87°F
Precipitation: 5%
Humidity: 70%
Wind speed: 4 mph
Current time: Tuesday 4:00 PM

Forcast temperature:
Day: Tue
Weather: Partly cloudy
High: 90, Low: 76
...
'''

或者，您可以使用来自 SerpApi的Google Direct Answer Box API来实现相同的目的。这是一个带有免费计划的付费 API。

您的示例中的主要区别在于，您只需要迭代已经提取的数据，而不是从头开始做所有事情，或者弄清楚如何绕过 Google 的块。

要集成的代码：

params = {
  "engine": "google",
  "q": "phagwara weather",
  "api_key": os.getenv("API_KEY"),
  "hl": "en",
  "gl": "us",
}

search = GoogleSearch(params)
results = search.get_dict()

loc = results['answer_box']['location']
weather_date = results['answer_box']['date']
weather = results['answer_box']['weather']
temp = results['answer_box']['temperature']
precipitation = results['answer_box']['precipitation']
humidity = results['answer_box']['humidity']
wind = results['answer_box']['wind']

forecast = results['answer_box']['forecast']

print(f'{loc}\n{weather_date}\n{weather}\n{temp}°F\n{precipitation}\n{humidity}\n{wind}\n')

print(json.dumps(forecast, indent=2))



---------
'''
Phagwara, Punjab, India
Tuesday 4:00 PM
Partly cloudy
87°F
5%
70%
4 mph

[
  {
    "day": "Tuesday",
    "weather": "Partly cloudy",
    "temperature": {
      "high": "90",
      "low": "76"
    },
    "thumbnail": "https://ssl.gstatic.com/onebox/weather/48/partly_cloudy.png"
  }
...
]
'''

免责声明，我为 SerpApi 工作。

于 2021-08-31T11:00:03.430 回答