python - Celery tasks vanishing - Django/Celery

Question

here is my code:

my task

from celery.decorators import task

@task()
def add(x, y):
    return x + y

using my tasks

from core.tasks import add

results = []

for i in range(100):
    results.append(add.delay(i, i))

---

problem 1

after a few seconds I perform the following:

for result in results:
    print result.result

prints this:

(the values are not returned in what looks like a pattern)

None
None
None
None
8
None
None
None
None
18
None
None
None
None
28
None
None
None
None
38
...

Second clean OS install+setup:

everything is working as expected, still not sure what happened here...

---

Tasks are also randomly missing within the Django admin interface...

---

Anybody know what is going on? :|

Answer 1

The task.delay() is asynchronous. BTW, the whole AMQP thing is about making tasks asynchronous. If you want synchronous behavior, what is the point of using celery?

from celery.decorators import task

@task()
def add(x, y, results):
    results.append(x + y)

------------8<-------------
from core.tasks import add

results = []

for i in range(100):
    add.delay(i, i, results)

Wait a few seconds before printing (while the consumers do their work) and be aware that results may came up out of order.

Method task.delay will return a celery.result.AsyncResult instance. Before using AsyncResult.result you should check for AsyncResult.state == SUCCESS.

Your final print loop could be:

for result in results:
    while not result.state.ready():
        time.sleep(secs)
    if result.state == u'SUCCESS':
        print result.result
    else:
        print "Something Rotten in the State of Denmark..."

This is (almost) the same as:

for result in results:
    print result.get()

Look at TaskSets, they are better than storing results in a list.

In practice people store results in the database and place the wait loop in the client that will then hammer the server via AJAX every few seconds until completion.