2

我要构建一个 Django 应用程序，它提供一个 REST api 来添加一些处理作业。

在每个请求上，都会将一个作业添加到数据库中以供以后处理。

可以通过自定义命令处理作业 ( ./manage.py runJobs)

这些作业是 CPU 密集型的，所以如果我在一台有 4 核的机器上运行，我希望最多有4 个工作人员并行运行作业。

如果我需要扩展，我需要能够在不同的机器上运行更多的操作系统相同的工人。

处理作业会占用大量内存，因此最好为每个作业创建一个新进程 - 也就是说 - 我认为最好./manage.py runJobs运行一个作业并退出，而不是在同一个进程中运行多个作业。

我需要确保比赛条件不会导致两名工人得到同一份工作。

我对linux进程之间的通信不是很熟悉，所以我认为来这里寻求技巧是个好主意。

你建议用什么架构来解决这个问题？

4

1 回答 1

0

I think I found a good enough solution that allows me to use the database as a semaphore, and doesn't required me to install additional software on the server.

It gos like this:

------ The custom command ------

from django.db import transaction

def run_one_job():
    candidate_jobs = Job.objects.filter(status='PENDING')
    job = lock_one_job_from_list(candidate_jobs)
    if job:
        process(job)  # whatever that means
        job.status = 'DONE'
        job.save()

def lock_one_job_from_list(jobs):
    for job in jobs:
        _job = attempt_lock_job(job.id)
        if _job:
            return _job


@transaction.commit_on_success
def attempt_lock_job(id):
    j = Job.objects.select_for_update(id=id)[0]  # wait until you get a write lock for that record
    if j.status == 'PENDING':
        j.status = 'RUNNING'
        j.save()
        return j

------ worker.sh -------

while [ 1 ]
do
./manage.py runJobs
sleep 1
done

Then I can spawn as many worker.sh instances as cores are available. This is not ideal because there will be a bunch of workers polling the database every second, but it does solve the biggest risk: running the same job twice. I'll think I'll go with this :-).

Please let me know if you see any holes in this approach.

于 2013-09-25T16:10:28.410 回答