0

这是我正在使用的示例代码。我想快速解析页面并将结果数据输入数据库。但是，添加一行后，我的代码开始运行明显变慢。我了解这与数据库的工作有关。但我不明白如何解决它。

如果您对加速此代码有任何其他建议，我将不胜感激。

import asyncpg
import asyncio
import aiohttp
from settings import URI, SQL, URLS


class Singleton(type):
    _instances = {}

    def __call__(cls, *args, **kwargs):
        if cls not in cls._instances:
            cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
        return cls._instances[cls]


class DBManager(metaclass=Singleton):
    """ Class for interacting with Postgres database """

    def __init__(self, dsn) -> None:
        """ The constructor takes the database """
        self.dsn = dsn
        self.pool = None

    async def connect(self):
        self.pool = await asyncpg.create_pool(dsn=self.dsn)


    async def insert_data(self, peson: str, address: str):
        async with self.pool.acquire() as connect:
            return await connect.execute(SQL, type_tx, address)


db = DBManager(URI)


async def check_address(url, session):
    async with session.get(url) as result:
        try:
            result = await result.json()
            person = 'adult' if result['age'] >= 21 else 'child'
            address = result['address']
            await db.insert_data(person, address)
            return print(address, person)
        except Exception as e:
            print(e)


async def bound_fetch(sem, url, session):
    async with sem:
        return await check_address(url, session)


async def main():
    await db.connect()
    urls = [url for i in URLS]
    sem = asyncio.Semaphore(50)
    tasks = []
    async with aiohttp.ClientSession() as session:
        for url in urls:
            task = asyncio.ensure_future(bound_fetch(sem, url, session))
            tasks.append(task)
        responses = asyncio.gather(*tasks)
        await responses

if __name__ == '__main__':
    asyncio.run(main())
4

1 回答 1

0

等待任务完成很慢。必要时 await 将等待任务完成。当不需要等待时，简单地继续而不阻塞您的代码通常是可取的。

正如您所提到的，这条线：await db.insert_data(person, address)很慢。那是因为它正在等待 db.insert。但是您没有使用此结果，因此您可以省略 await 并允许插入异步进行，而不是等待。

您提到 asynchio，python 文档在此处对此进行了很好的概述。

于 2020-12-23T00:09:36.667 回答