0

So my problem is:

  1. I have existing Flask application that has several Celery periodic tasks (so Flask app running and Celery workers for process tasks are running).

Now in our infrastructure appears NSQ and 3rd application that pushes messages to NSQ, so I have to add functionality to existing Flask app that can read those messages from NSQ and do some work (probably create entities in DB or initiate Celery tasks)

  1. I looked at pynsq official client, it based on tornado.IOLoop, so it requires that IOLoop is started all time, the example from documentation works well, but I don't have an idea how to run Reader with existing Flask app in one thread.

  2. I looked ad gnsq library that based on gevent, it seems possible to start its Reader in separate thread.

So possible ways to do smth I see:

  1. Probably run Reader (run_forever) in separate thread, so when Reader founds message - some work I can delegate to callback handler function that starts on_message event.

  2. Probably exists way when I can read messages from queue and process it, for example from Celery task, that I run once in 30 sec. But I cannot find how, all libraries I saw provide long-polling way, when Reader starts like infinite loop and reads messages.

  3. Make separate application that will be responsible for reading messages from NSQ and re-use some code from existing Flask app in it. (or just make that application like 'portal' from NSQ to my Flask app).

P.S. I've tried to start Reader in thread when Flask app starts, I got some success with gnsq library, its Reader connects to NSQ and on message runs handler's code, but with pynsq that based on IOLoop - when I'm trying to run nsq.run() in thread it fails, seems I cannot to run "pynsq" nsq.run() not in main thread)

Please point me in right direction, I'm stuck a bit how to make it properly.

P.S. Also probably somebody can give me advice of proper way to read messages from NSQ from synchronous code, I mean for example when some code starts in celery task, in that code can I read and process NSQ messages somehow ?

4

0 回答 0