3 
>>> import asyncio
>>> help(asyncio.wait)

..

Help on function wait in module asyncio.tasks:

wait(fs, *, loop=None, timeout=None, return_when='ALL_COMPLETED')
    Wait for the Futures and coroutines given by fs to complete.

    Coroutines will be wrapped in Tasks.

    Returns two sets of Future: (done, pending).

    Usage:

        done, pending = yield from asyncio.wait(fs)

    Note: This does not raise TimeoutError! Futures that aren't done
    when the timeout occurs are returned in the second set.
(END)

I dont quite understand last Note in this help (what is second set? is it pending/reprocessing set? how do I execute pending tasks and combine the results of both done and pending and then save in DB)

My problem: I'm using asyncio with aiohttp, have millions of urls , few of them might raise timeout error. I want to send them in a queue for reprocessing or it should take care by eventpool.

import asyncio
import aiohttp
sem = asyncio.Semaphore(10)

def process_data(url):
    with (yield from sem):
          response = yield from aiohttp.request('GET', url)
          print(response)

loop = asyncio.get_event_loop()
c = asyncio.wait([process_data(url) for url in url_list], timeout=10)
loop.run_until_complete(c)

PS: I'm not using wait_for method.

4

1 回答 1

4

以下是帮助中的两组：

Returns two sets of Future: (done, pending).

第二组是pending在超时时间内未完成的一组作业。它将返回一个包含两个期货列表的元组，一个已经完成，一个仍在等待中。

代替：

c = asyncio.wait([process_data(url) for url in url_list], timeout=10)
loop.run_until_complete(c)

你可能应该：

def dostuff():
    done, pending = yield from asyncio.wait([process_data(url) for url in url_list], timeout=10)

    # do something with pending

loop.run_until_complete(dostuff())

以下是更多信息：

https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.wait

于 2015-03-03T18:04:20.517 回答