1

我在多进程环境中对以下 Python/Django 代码有疑问。它尝试更新 InnoDB 表，就好像它是一个队列一样，弹出顶部值。它工作正常，直到负载达到一定水平并且多个进程同时在此代码中。当一个进程持有要删除的表，而另一个进程试图更新表时，就会发生死锁。

我以前对 InnoDB 不了解的是索引与表是分开的，因此一个进程可以锁定表并等待索引锁定，而另一个进程可能相反，从而产生死锁。这似乎与下面代码中的 UPDATE/DELETE 死锁有关。

简而言之，代码首先更新表中的单行（LIMIT 1），防止任何其他进程更新同一行。然后它选择该行以检索其他内容，执行一些工作，然后从表中删除该行。

@classmethod
@transaction.commit_manually
def pop(cls, unique_id):
    """Retrieves the next item and removes it from the queue."""
    transaction.commit()    # This has the side effect of clearing the object cache

    if cls.objects.all().filter(unique_id__isnull=True).count() == 0:  return None

    sql = "UPDATE queue SET unique_id=%d" % unique_id
    sql += " WHERE unique_id IS NULL"
    sql += " ORDER BY id LIMIT 1"
    cursor = connection.cursor()
    try:
        cursor.execute(sql)
        row = cursor.fetchone()
    except OperationalError, oe:    # deadlock, retry later
        transaction.rollback()
        return None
    # If an item is available, it is now marked with our process_id.

    # Retrieve any items with this process id
    items = cls.objects.all().filter(process_id=process_id)
    if len(items) == 0:     # No items available
        transaction.rollback()
        return None
    top = items[0]      # there should only be one item
    # ... perform some other actions here ...
    top.delete()        # This item is deleted from the queue
    transaction.commit()
    return item

如上所述，当两个进程同时运行相同的代码时，就会发生死锁。top.delete()当另一个进程正在执行UPDATE查询时，一个进程正在尝试执行。InnoDB 行级锁定不会阻止第二个进程尝试在此处更新表。这是来自的输出show engine innodb status;：

------------------------
LATEST DETECTED DEADLOCK
------------------------
130625 13:30:22
*** (1) TRANSACTION:
TRANSACTION 0 821802296, ACTIVE 0 sec, process no 27565, OS thread id 139724816692992 starting index read
mysql tables in use 2, locked 2
LOCK WAIT 38 lock struct(s), heap size 14320, 14936 row lock(s)
MySQL thread id 2984294, query id 3142696474 example.com 10.0.1.1 mysql Sending data
UPDATE queue SET unique_id=100804 WHERE unique_id IS NULL ORDER BY id LIMIT 1
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1476 page no 2469 n bits 280 index `PRIMARY` of table `mysql`.`queue` trx id 0 821802296 lock_mode X locks rec but not gap waiting
Record lock, heap no 208 PHYSICAL RECORD: n_fields 9; compact format; info bits 32
 0: len 4; hex 80290a1b; asc  )  ;; 1: len 6; hex 000030fbb547; asc   0  G;; 2: len 7; hex 0000002a471282; asc    *G  ;; 3: len 4; hex 80000005; asc     ;; 4: len 4; hex 800
00051; asc    Q;; 5: len 4; hex 07a27ca6; asc   | ;; 6: len 8; hex 8000124f06c3361e; asc    O  6 ;; 7: SQL NULL; 8: SQL NULL;

*** (2) TRANSACTION:
TRANSACTION 0 821802311, ACTIVE 0 sec, process no 27565, OS thread id 139724817225472 updating or deleting, thread declared inside InnoDB 499
mysql tables in use 1, locked 1
3 lock struct(s), heap size 1216, 2 row lock(s), undo log entries 1
MySQL thread id 2984268, query id 3142696582 example.com 10.0.1.1 mysql updating
DELETE FROM `queue` WHERE `id` IN (2689563)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 1476 page no 2469 n bits 280 index `PRIMARY` of table `mysql`.`queue` trx id 0 821802311 lock_mode X locks rec but not gap
Record lock, heap no 208 PHYSICAL RECORD: n_fields 9; compact format; info bits 32
 0: len 4; hex 80290a1b; asc  )  ;; 1: len 6; hex 000030fbb547; asc   0  G;; 2: len 7; hex 0000002a471282; asc    *G  ;; 3: len 4; hex 80000005; asc     ;; 4: len 4; hex 800
00051; asc    Q;; 5: len 4; hex 07a27ca6; asc   | ;; 6: len 8; hex 8000124f06c3361e; asc    O  6 ;; 7: SQL NULL; 8: SQL NULL;

*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1476 page no 351 n bits 1200 index `queue_queue_id` of table `mysql`.`queue` trx id 0 821802311 lock_mode X locks rec but not gap waiting
Record lock, heap no 1128 PHYSICAL RECORD: n_fields 2; compact format; info bits 0
 0: len 4; hex 80000005; asc     ;; 1: len 4; hex 80290a1b; asc  )  ;;

*** WE ROLL BACK TRANSACTION (2)

pop()我的问题是：在 Python/Django 和 InnoDB 中实现队列和执行操作的正确方法是什么？特别是，需要对此代码进行哪些更改？

4

0 回答 0