3

我最近遇到了需要尽快获取 URL 列表的问题。

所以很自然地，我设置了一个小测试，看看什么效果最好。

方法 1 - asyncio

async def test_async():
    async with httpx.AsyncClient() as client:
        await asyncio.gather(*(fetch_async(client, symbol) for symbol in symbols))


async def fetch_async(client, symbol):
    await client.get(
        f"https://query1.finance.yahoo.com/v8/finance/chart/{symbol}.NS", timeout=None,
    )

方法 2 - ThreadPoolExecutor

async def test_threads():
    with ThreadPoolExecutor(max_workers=len(symbols)) as pool, httpx.Client() as client:
        loop = asyncio.get_event_loop()

        await asyncio.gather(
            *(
                loop.run_in_executor(pool, fetch_sync_fn(client, symbol))
                for symbol in symbols
            )
        )


def fetch_sync_fn(client, symbol):
    def fn():
        client.get(
            f"https://query1.finance.yahoo.com/v8/finance/chart/{symbol}.NS",
            timeout=None,
        )

    return fn

2013 MacBook pro 上的结果

In [3]: %timeit asyncio.run(test_threads())                                                                                                                                                          
1.41 s ± 87.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [4]: %timeit asyncio.run(test_async())                                                                                                                                                            
1.24 s ± 62.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

数字海洋 5 美元服务器上的结果

In [4]: %timeit asyncio.run(test_threads())
5.94 s ± 66.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [3]: %timeit asyncio.run(test_async())
10.7 s ± 97.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Google colab上的结果

%timeit loop.run_until_complete(test_threads())
1 loop, best of 3: 723 ms per loop

%timeit loop.run_until_complete(test_async())
1 loop, best of 3: 597 ms per loop

问题

  1. 这种不一致的原因是什么？为什么服务器和本地机器上的赢家不同？
  2. 为什么服务器上的两个测试都比较慢？在具有更快网络连接的服务器上，纯网络任务不应该更快吗？

完整代码为 github gist

4

0 回答 0