0

我正在尝试用 Beautiful Soup 抓取网站。打印完容器后，它给了我一个空列表。我怎样才能解决这个问题？

import requests
from bs4 import BeautifulSoup
import lxml

URL = 'https://www.monster.com/jobs/search/?q=Software-Developer&where=Australia'
page = requests.get(URL)

soup = BeautifulSoup(page.content, 'lxml')
container = soup.find_all('div', class_="results-list")
print(container)
4

1 回答 1

1

数据通过 Ajax 调用从外部 URL 加载。您可以使用requests模块来模拟它并加载数据：

import json
import requests
from textwrap import wrap
from bs4 import BeautifulSoup


api_url = "https://services.monster.io/jobs-svx-service/v2/monster/search-jobs/samsearch/en-us"

payload = {
    "fingerprintId": "",
    "jobAdsRequest": {
        "placement": {"appName": "monster", "component": "JSR_SPLIT_VIEW"},
        "position": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    },
    "jobQuery": {
        "companyDisplayNames": [],
        "excludeJobs": [],
        "locations": [{"address": "Australia", "country": "us"}],
        "query": "Software-Developer",
    },
    "offset": 0,
    "pageSize": 20,
    "searchId": "",
}


data = requests.post(api_url, json=payload).json()

# uncomment to print all data:
# print(json.dumps(data, indent=4))

for r in data["jobResults"]:
    s = BeautifulSoup(r["jobPosting"]["description"], "html.parser")
    print("\n".join(wrap(s.get_text(strip=True, separator=" "))))
    print("-" * 80)

印刷：

Description: The role of the Mobile Developer is to analyze business
requirements, develop a design plan, translate plan into program
specifications (or a manual process), code, test, and coordinate
implementation. They will be focusing on enhancements within existing
applications (not building out anything new). Working with multiple
applications- all very client specific. This developer could also be
tasked with some frontend work for a specific application. Need to be
able to utilize Angular skills for frontend development. -Must be
highly proficient with mobile platform Application Programming
Interfaces (API) such as Apple iOS and Android Mobile -Hands on
experience with latest iOS and Android tech -Deep knowledge of Angular


...and so on
于 2021-06-18T15:12:42.333 回答