1

我正在尝试在 Django 3.2.3、Python 3.7.9 中使用 uWSGI Spooler 而不是 Celery 运行并发任务。我找到了一些类似this、this和this的资源，但没有任何效果。我在这个旅程中遇到了很多错误，我已经使用我在网上找到的解决方案修复了它们，现在这就是我所拥有的：

设置

uwsgi.ini

[uwsgi]

pythonpath          = /path/to/djproj
wsgi-file           = /path/to/djproj/wsgi.py
uid                 = myuid
module              = wsgi:application

master              = true
processes           = 1
threads             = 10
lazy-apps           = true
http                = 0.0.0.0:8080

vacuum              = true
log-format          = %(ltime) Worker: %(wid) %(status) %(method) %(uri) Size: %(size)
log-date            = %%Y %%m %%d %%H:%%M:%%S.000
# Let django handle most of the logging
disable-logging     = true
log-5xx             = true

harakiri            = 60
harakiri-verbose    = true

stats               = /tmp/djproj_stats.socket

# Spooling
spooler             = /path/to/tasks
spooler-harakiri    = 600
import              = djproj.tasks

任务.py

import logging

logger = logging.getLogger(__name__)

try:
    from uwsgidecorators import spool

    logger.warning("Imported spool successfully.")
except Exception:
    logger.warning("Couldn't import spool.")

    def spool(func):
        def func_wrapper(**arguments):
            return func(arguments)

        return func_wrapper


@spool
def run_task(arguments):
    logger.warning("Running in spool.")
    from djproj.myapp.models import MyModel

    obj = MyModel.objects.get(id=arguments["obj_id"])
    obj.run()

djproj/myapp/models.py

# ...

def prepare_spooler_args(**kwargs):
    args = {}
    for name, value in kwargs.items():
        args[name.encode("utf-8")] = str(value).encode("utf-8")
    return args

class MyModel(models.Model):
    # ...
    def start_run_in_spooler(self):
        args = prepare_spooler_args(task_id=self.id)
        run_task(args)

结果

当我运行uwsgi --ini uwsgi.ini并访问触发此代码的端点时，我得到：

...
2021 09 20 12:56:20.000 - *** Stats server enabled on /tmp/djproj_stats.socket fd: 16 ***
2021 09 20 12:56:20.000 - spawned uWSGI http 1 (pid: 919)
2021 09 20 12:56:20.000 - [spooler /path/to/tasks pid: 917] managing request uwsgi_spoolfile_on_5a22c167ad32_826_2_189168444_1632124468_886229 ...
2021 09 20 12:56:20.000 - unable to find the spooler function, have you loaded it into the spooler process ?

我觉得很奇怪，网上关于如何完成这项工作的资源如此之少。每次我寻找解决方案时，每个人都推荐 Celery，就像它是任何 Python 应用程序中并发的灵丹妙药，尽管 Django + uWSGI 是一个非常常见的组合，而且 Spooler 似乎是一个简单而轻量级的解决方案。如果有人对如何使这项工作有任何提示，那就太棒了。

4

1 回答 1

0

在同事的帮助下，我们终于成功了。以下是必要的更改：

uwsgi.ini

import = djproj.tasks已更改为spooler-import = djproj.tasks.

任务.py

必须将导入从假脱机函数中移出，并且需要初始化 Django 才能使导入工作。这是代码的最终版本：

import logging
from functools import wraps

import django

logger = logging.getLogger(__name__)

try:
    from djproj.myapp.models import MyModel
except Exception:
    logger.warning(f"Django is not loaded yet! Setting up...")
    try:
        django.setup(set_prefix=False)
    except Exception:
        pass
    from djproj.myapp.models import MyModel

try:
    from uwsgidecorators import spool
except Exception:
    logger.warning("Couldn't import uwsgidecorators!")

    def spool(pass_arguments):
        def decorator(method):
            if callable(pass_arguments):
                method.gw_method = method.__name__
            else:
                method.gw_method = pass_arguments

            @wraps(method)
            def wrapper(*args, **kwargs):
                method(*args, **kwargs)

            return wrapper

        if callable(pass_arguments):
            return decorator(pass_arguments)
        return decorator


@spool(pass_arguments=True)
def run_task(task_id):
    task = MyModel.objects.get(id=task_id)
    task.run()

模型.py 和视图.py

删除了调用假脱机函数的模型方法。相反，我们只是直接在视图中调用假脱机函数：

class MyModelViewSet(viewsets.ModelViewSet):
    queryset = models.MyModel.objects.all()
    serializer_class = serializers.MyModelSerializer

    def create(self, request, *args, **kwargs):
        # ...

        run_task(task_id=task.id)

        # ...
        return Response(serializer.data, status=status_code, headers=headers)

希望这对任何面临类似问题的人有用。

于 2021-09-21T12:26:36.983 回答