python-3.x - 如何使用 pyppeteer 异步获取 url（一个浏览器多个选项卡）

Question

我希望我的脚本

1. 打开说 3 个标签

2. 异步获取一个 url（每个选项卡都相同）

3. 保存响应

4. 睡眠 4 秒

5. 用正则表达式解析响应（我试过 BeautifulSoup，但它太慢了）并返回一个令牌

6. 在 3 个选项卡中循环多次

我的问题是 2。我有一个示例脚本，但它同步获取 url。我想让它异步。

from pyppeteer import launch

urls = ['https://www.example.com']


async def main():
    browser = await launch(headless=False)
    for url in urls:
        page1 = await browser.newPage()
        page2 = await browser.newPage()
        page3 = await browser.newPage()

        await page1.goto(url)
        await page2.goto(url)
        await page3.goto(url)

        title1= await page1.title()
        title2= await page2.title()
        title3= await page3.title()

        print(title1)
        print(title2)
        print(title3)

    #await browser.close()


asyncio.get_event_loop().run_until_complete(main())

此外，如您所见，代码不是那么简洁。我该如何让它异步？

另外，如果有帮助，我还有其他不符合我需要的 pyppeteer 脚本，以防万一转换这些脚本会更容易

import asyncio
from pyppeteer import launch

url = 'http://www.example.com'
browser = None

async def fetchUrl(url):
    # Define browser as a global variable to ensure that the browser window is only created once in the entire process
    global browser
    if browser is None:
        browser = await launch(headless=False)

    page = await browser.newPage()

    await page.goto(url)
    #await asyncio.wait([page.waitForNavigation()])
    #str = await page.content()
    #print(str)

 # Execute this function multiple times for testing
asyncio.get_event_loop().run_until_complete(fetchUrl(url))
asyncio.get_event_loop().run_until_complete(fetchUrl(url))

该脚本是异步的，但它一次执行一个事件循环，因此它与同步一样好。

# cat test.py
import asyncio
import time
from pyppeteer import launch

WEBSITE_LIST = [
    'http://envato.com',
    'http://amazon.co.uk',
    'http://example.com',
]

start = time.time()

async def fetch(url):
    browser = await launch(headless=False, args=['--no-sandbox'])
    page = await browser.newPage()
    await page.goto(f'{url}', {'waitUntil': 'load'})
    print(f'{url}')
    await asyncio.sleep(1)
    await page.close()
    #await browser.close()

async def run():
    tasks = []

    for url in WEBSITE_LIST:
        task = asyncio.ensure_future(fetch(url))
        tasks.append(task)

    responses = await asyncio.gather(*tasks)

loop = asyncio.get_event_loop()
future = asyncio.ensure_future(run())
loop.run_until_complete(future)

print(f'It took {time.time()-start} seconds.')

该脚本是异步的，但它会为每个 url 启动一个单独的浏览器，这最终会占用太多资源。

Answer 1

这将在单独的选项卡中打开每个 URL：

import asyncio
import traceback

from pyppeteer import launch

URLS = [
    "http://envato.com",
    "http://amazon.co.uk",
    "http://example.com",
]


async def fetch(browser, url):
    page = await browser.newPage()

    try:
        await page.goto(f"{url}", {"waitUntil": "load"})
    except Exception:
        traceback.print_exc()
    else:
        html = await page.content()
        return (url, html)
    finally:
        await page.close()


async def main():
    tasks = []
    browser = await launch(headless=True, args=["--no-sandbox"])

    for url in URLS:
        tasks.append(asyncio.create_task(fetch(browser, url)))

    for coro in asyncio.as_completed(tasks):
        url, html = await coro
        print(f"{url}: ({len(html)})")

    await browser.close()


if __name__ == "__main__":
    main = asyncio.run(main())