0

我数据库中的一些表是从一些在后台不断运行的后台 python 脚本更新的（进行数据挖掘）。Django 很难知道数据是在 Django 之外更新的，并且总是显示旧数据。经过长时间的研究，大多数在线缓存禁用方法对我不起作用，但这种解决方法非常有效：

from django.db import connection
connection.close()

这仍然不是一个完美的解决方案，但它确实在 Django 的视图中起作用。如果你知道如何解决这个问题，请在此处评论Django: how to refresh or reload models from database。但是，数据也可以通过 REST 访问。在 Tastypie 中，资源仍未更新。

JSON 结构总是返回过时的时间戳，我可以看到数据库中的最新值是不同的。我已经尝试了以下方法，但没有奏效。

from tastypie.cache import NoCache
cache = NoCache()

现在唯一的刷新方法就是手动重启uwsgi服务，这显然不是解决办法。这个问题快把我逼疯了。任何评论将不胜感激。谢谢你。

更新

我已将代码和模型简化为仅包含时间戳列。这是完整的代码和日志。

Tastypie 的资源.py

from lp.models import LatestPrices
from tastypie.resources import ModelResource
from tastypie.serializers import Serializer
from django.db import connection

class LpResource(ModelResource):
    class Meta:

        # forces Django to make a new connection, fixes OperationalError: (2006, 'MySQL server has gone away') after a long time with no access
        connection.close() 

        queryset = LatestPrices.objects.all()

        include_resource_uri = False
        resource_name = 'lp'
        allowed_methods = ['get']
        excludes = ['slug']
        serializer = Serializer(formats=['json'])
        connection.close()

模型.py：

from django.db import models

class LatestPrices(models.Model):
    id = models.AutoField(primary_key=True)
    slug = models.SlugField(unique=True)

    # timestamp is updated every 10 seconds
    timestamp = models.DateTimeField(null=True, blank=True)

    # Removed all other columns for debugging.

    # The data to show in admin
    def __unicode__(self):
        return str(self.timestamp)

    class Meta:
        db_table = 'latest_prices'

Django 之外的数据库注入器脚本

while True:
    try:
        # Get date & time
        timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
        logger.info("------" + timestamp)

        # Inject to local database
        logger.debug("Injecting...")
        con = None
        con = db.connect('localhost', 'user', 'pw', 'tablename')
        con.autocommit(True)
        cursor = con.cursor(MySQLdb.cursors.DictCursor)
        update_query = "UPDATE latest_prices SET slug='latest', timestamp='" + timestamp \
                             + "' WHERE id=1"
        rows_affected = cursor.execute(update_query)
        con.commit()
        if rows_affected != 1:
            logger.critical("Couldn't inject! Rows affected: " + str(rows_affected))
            cursor.close()
            time.sleep(WAIT_TIME)
            continue

        cursor.close()

    except Exception as e:
        # Code removed. Email admin the error string...
        pass
    finally:    
        pass

    time.sleep(WAIT_TIME)

mysql.log

          124 Query UPDATE latest_prices SET slug='latest', timestamp='2013-04-04 18:38:16' WHERE id=1
          124 Query commit
130404 18:38:26   125 Connect   dev@localhost on gs
          125 Query set autocommit=0
          125 Query set autocommit=1
          124 Quit  
          125 Query UPDATE latest_prices SET slug='latest', timestamp='2013-04-04 18:38:26' WHERE id=1
          125 Query commit
130404 18:38:36   126 Connect   dev@localhost on gs
          126 Query set autocommit=0
          126 Query set autocommit=1
          125 Quit  
          126 Query UPDATE latest_prices SET slug='latest', timestamp='2013-04-04 18:38:36' WHERE id=1
          126 Query commit
130404 18:38:46   127 Connect   dev@localhost on gs
          127 Query set autocommit=0
          127 Query set autocommit=1
          126 Quit  
          127 Query UPDATE latest_prices SET slug='latest', timestamp='2013-04-04 18:38:46' WHERE id=1
          127 Query commit

# Click the browser refresh button one time here
130404 18:38:53    73 Query SELECT `latest_prices`.`id`, `latest_prices`.`slug`, `latest_prices`.`timestamp` FROM `latest_prices` WHERE (`latest_prices`.`id` = 1  AND `latest_prices`.`id` = 1 )

130404 18:38:56   128 Connect   dev@localhost on gs
          128 Query set autocommit=0
          128 Query set autocommit=1
          127 Quit  
          128 Query UPDATE latest_prices SET slug='latest', timestamp='2013-04-04 18:38:56' WHERE id=1
          128 Query commit

# Click the browser refresh button a few times here
130404 18:38:58    70 Query SELECT `latest_prices`.`id`, `latest_prices`.`slug`, `latest_prices`.`timestamp` FROM `latest_prices` WHERE (`latest_prices`.`id` = 1  AND `latest_prices`.`id` = 1 )
130404 18:39:02    70 Query SELECT `latest_prices`.`id`, `latest_prices`.`slug`, `latest_prices`.`timestamp` FROM `latest_prices` WHERE (`latest_prices`.`id` = 1  AND `latest_prices`.`id` = 1 )
130404 18:39:04    73 Query SELECT `latest_prices`.`id`, `latest_prices`.`slug`, `latest_prices`.`timestamp` FROM `latest_prices` WHERE (`latest_prices`.`id` = 1  AND `latest_prices`.`id` = 1 )

130404 18:39:06   129 Connect   dev@localhost on gs
          129 Query set autocommit=0
          129 Query set autocommit=1
          128 Quit  
          129 Query UPDATE latest_prices SET slug='latest', timestamp='2013-04-04 18:39:06' WHERE id=1
          129 Query commit
130404 18:39:16   130 Connect   dev@localhost on gs
          130 Query set autocommit=0
          130 Query set autocommit=1
          130 Query UPDATE latest_prices SET slug='latest', timestamp='2013-04-04 18:39:16' WHERE id=1
          129 Quit  
          130 Query commit

浏览器输出（总是一样的）

{"id": 1, "timestamp": "2013-04-04T18:29:45"}

HTTP 标头

Cache-Control →no-cache
Connection →keep-alive
Content-Type →application/json
Date →Fri, 05 Apr 2013 00:27:48 GMT
Server →nginx
Transfer-Encoding →
Transfer-Encoding
The form of encoding used to safely transfer the entity to the user. Currently defined methods are: chunked, compress, deflate, gzip, identity.

chunked
Vary →Accept

我还在注入器脚本中添加了显式提交。它在 mysql.log 中说 commit 和我正在使用的现有“set autocommit = 1”并没有太大区别。

现在我正在查看 mysql.log，我看到每次点击浏览器的刷新按钮时，我都会看到 Django 发送的新 SELECT 语句。但是这些对象仍然很旧。我想不通，为什么？

4

2 回答 2

1

Django 将缓存查询集中的结果，但不缓存不相关查询集中的结果。

以下代码将导致两个数据库查询：

list(SomeTable.objects.all())  # First db query.
list(SomeTable.objects.all())  # Second db query unrelated to first.

症状和您的connection.close()解决方案与 MySQL 事务问题一致。每个查询的新连接将导致不同的事务。

似乎在您的 Django 代码中启动了一个事务，但从未提交或回滚。当事务开始时，该事务中的进一步读/写只会看到事务开始时数据库的状态，以及该事务中所做的更新。如果另一个进程启动会话（例如您的数据挖掘脚本）并插入数据，则 Django 事务将不会看到它，直到现有事务关闭。

如果没有看到您的代码，就不可能知道为什么会有一个事务开始但没有完成。要验证此假设，请打开 MySQL 查询日志并检查其内容。您应该会看到一个start transaction没有相应commitor的查询rollback。

于 2013-04-05T00:28:21.057 回答
0

将“transaction-isolation = READ-COMMITTED”添加到 my.cnf。此处有更多详细信息：Django：如何从数据库刷新或重新加载模型

于 2013-04-05T05:23:04.987 回答