29

每次我使用 Admin 列出模型的条目时,Admin 都会计算表中的行数。更糟糕的是,即使您正在过滤查询,它似乎也会这样做。

例如,如果我只想显示 id 为 123、456、789 的模型,我可以这样做:

/admin/myapp/mymodel/?id__in=123,456,789

但是运行的查询(除其他外)是:

SELECT COUNT(*) FROM `myapp_mymodel` WHERE `myapp_mymodel`.`id` IN (123, 456, 789) # okay
SELECT COUNT(*) FROM `myapp_mymodel` # why???

这是杀死mysql + innodb。这个问题似乎在这张票中得到了部分承认,但我的问题似乎更具体,因为它计算了所有行,即使它不应该计算在内。

有没有办法禁用全局行数?

注意:我使用的是 django 1.2.7。

4

7 回答 7

28

Django 1.8 允许您通过设置禁用此功能show_full_result_count = False

https://docs.djangoproject.com/en/stable/ref/contrib/admin/#django.contrib.admin.ModelAdmin.show_full_result_count

于 2015-01-26T16:23:10.860 回答
25

好的,我想我找到了解决方案。正如彼得建议的那样,最好的方法是处理该count属性,并且可以通过使用自定义查询集(如 本文中所见)覆盖它来完成,该查询集专门使用近似等价的计数:

from django.db import connections, models
from django.db.models.query import QuerySet

class ApproxCountQuerySet(QuerySet):
    """Counting all rows is very expensive on large Innodb tables. This
    is a replacement for QuerySet that returns an approximation if count()
    is called with no additional constraints. In all other cases it should
    behave exactly as QuerySet.

    Only works with MySQL. Behaves normally for all other engines.
    """

    def count(self):
        # Code from django/db/models/query.py

        if self._result_cache is not None and not self._iter:
            return len(self._result_cache)

        is_mysql = 'mysql' in connections[self.db].client.executable_name.lower()

        query = self.query
        if (is_mysql and not query.where and
                query.high_mark is None and
                query.low_mark == 0 and
                not query.select and
                not query.group_by and
                not query.having and
                not query.distinct):
            # If query has no constraints, we would be simply doing
            # "SELECT COUNT(*) FROM foo". Monkey patch so the we
            # get an approximation instead.
            cursor = connections[self.db].cursor()
            cursor.execute("SHOW TABLE STATUS LIKE %s",
                    (self.model._meta.db_table,))
            return cursor.fetchall()[0][4]
        else:
            return self.query.get_count(using=self.db)

然后在管理员中:

class MyAdmin(admin.ModelAdmin):

    def queryset(self, request):
        qs = super(MyAdmin, self).queryset(request)
        return qs._clone(klass=ApproxCountQuerySet)

近似函数可能会在页码 100000 上搞砸,但对于我的情况来说已经足够了。

于 2012-05-04T09:32:19.380 回答
10

我发现 Nova 的回答很有帮助,但我使用的是 postgres。我稍微修改了它以适用于 postgres,对处理表命名空间进行了一些细微的改动,以及稍微不同的“检测 postgres”逻辑。

这是pg版本。

class ApproxCountPgQuerySet(models.query.QuerySet):
  """approximate unconstrained count(*) with reltuples from pg_class"""

  def count(self):
      if self._result_cache is not None and not self._iter:
          return len(self._result_cache)

      if hasattr(connections[self.db].client.connection, 'pg_version'):
          query = self.query
          if (not query.where and query.high_mark is None and query.low_mark == 0 and
              not query.select and not query.group_by and not query.having and not query.distinct):
              # If query has no constraints, we would be simply doing
              # "SELECT COUNT(*) FROM foo". Monkey patch so the we get an approximation instead.
              parts = [p.strip('"') for p in self.model._meta.db_table.split('.')]
              cursor = connections[self.db].cursor()
              if len(parts) == 1:
                  cursor.execute("select reltuples::bigint FROM pg_class WHERE relname = %s", parts)
              else:
                  cursor.execute("select reltuples::bigint FROM pg_class c JOIN pg_namespace n on (c.relnamespace = n.oid) WHERE n.nspname = %s AND c.relname = %s", parts)
          return cursor.fetchall()[0][0]
      return self.query.get_count(using=self.db)
于 2014-04-16T19:44:40.407 回答
6

Nova 的解决方案(ApproxCountQuerySet)效果很好,但是在较新版本的 Django 中,queryset 方法被 get_queryset 取代,所以现在应该是:

class MyAdmin(admin.ModelAdmin):

    def get_queryset(self, request):
        qs = super(MyAdmin, self).get_queryset(request)
        return qs._clone(klass=ApproxCountQuerySet)
于 2017-06-14T14:54:40.997 回答
4

如果这是一个严重的问题,您可能必须采取 Drastic Actions™。

查看 1.3.1 安装的代码,我看到管理代码正在使用由get_paginator(). 默认分页器类似乎在django/core/paginator.py. 该类有一个名为的私有值_count,该值设置在Paginator._get_count()(我的副本中的第 120 行)中。这又用于设置Paginator类的一个名为count. 我认为那_get_count()是你的目标。现在舞台已经搭建好了。

你有几个选择:

  1. 直接修改源码。我建议这样做,但由于您似乎被困在 1.2.7 上,您可能会发现它是最方便的。请记住记录此更改!未来的维护者(可能包括您自己)会感谢您的提醒。

  2. Monkeypatch 班级。这比直接修改要好,因为a)如果您不喜欢更改,只需注释掉monkeypatch,b)它更有可能与Django的未来版本一起使用。我有一个可以追溯到 4 年多的猴子补丁,因为他们仍然没有修复模板变量_resolve_lookup()代码中的错误,该错误无法识别顶层评估的可调用对象,仅在较低级别。尽管补丁(包装类的方法)是针对 0.97-pre 编写的,但它仍然适用于 1.3.1。

我没有花时间确切地弄清楚您必须为您的问题做出哪些更改,但它可能类似于将_approx_count成员添加到适当的类class META,然后测试以查看该 attr 是否存在。如果是并且是,None那么您执行sql.count()并设置它。如果您在(或接近)列表的最后一页,您可能还需要重置它。如果您需要更多帮助,请与我联系;我的电子邮件在我的个人资料中。

于 2012-05-03T17:04:20.533 回答
3

可以更改管理类使用的默认分页器。这是一个在短时间内缓存结果的方法:https ://gist.github.com/e4c5/6852723

于 2013-10-06T11:20:08.937 回答
0

我设法创建了一个自定义分页器,它显示当前页面编号、一个下一步按钮和一个显示完整计数链接。如果需要,它允许使用原始分页器。

在此处输入图像描述

使用的技巧是per_page + 1从 db 中获取元素,以查看是否有更多元素,然后提供虚假计数。

假设我们想要第三页并且页面有 25 个元素 => 我们想要object_list[50:75]。调用查询集时Paginator.count,将对查询集进行评估object_list[50:76](请注意,我们采用75+1 个元素),然后如果我们从 db 获得 25+1 个元素,则返回计数为 76,如果没有收到 26,则返回 50 + 收到的元素数元素。


TL;DR:我为以下内容创建了一个 mixin ModelAdmin

from django.core.paginator import Paginator
from django.utils.functional import cached_property


class FastCountPaginator(Paginator):
    """A faster paginator implementation than the Paginator. Paginator is slow
    mainly because QuerySet.count() is expensive on large queries.

    The idea is to use the requested page to generate a 'fake' count. In
    order to see if the page is the final one  it queries n+1 elements
    from db then reports the count as page_number * per_page + received_elements.
    """

    use_fast_pagination = True

    def __init__(self, page_number, *args, **kwargs):
        self.page_number = page_number
        super(FastCountPaginator, self).__init__(*args, **kwargs)

    @cached_property
    def count(self):
        # Populate the object list when count is called. As this is a cached property,
        # it will be called only once per instance
        return self.populate_object_list()

    def page(self, page_number):
        """Return a Page object for the given 1-based page number."""
        page_number = self.validate_number(page_number)
        return self._get_page(self.object_list, page_number, self)

    def populate_object_list(self):
        # converts queryset object_list to a list and return the number of elements until there
        # the trick is to get per_page elements + 1 in order to see if the next page exists.
        bottom = self.page_number * self.per_page
        # get one more object than needed to see if we should show next page
        top = bottom + self.per_page + 1
        object_list = list(self.object_list[bottom:top])
        # not the last page
        if len(object_list) == self.per_page + 1:
            object_list = object_list[:-1]
        else:
            top = bottom + len(object_list)
        self.object_list = object_list
        return top


class ModelAdminFastPaginationMixin:
    show_full_result_count = False  # prevents root_queryset.count() call

    def changelist_view(self, request, extra_context=None):
        # strip count_all query parameter from the request before it is processed
        # this allows all links to be generated like this parameter was not present and without raising errors
        request.GET = request.GET.copy()
        request.GET.paginator_count_all = request.GET.pop('count_all', False)

        return super().changelist_view(request, extra_context)

    def get_paginator(self, request, queryset, per_page, orphans=0, allow_empty_first_page=True):
        # use the normal paginator if we want to count all the ads
        if hasattr(request.GET, 'paginator_count_all') and request.GET.paginator_count_all:
            return Paginator(queryset, per_page, orphans, allow_empty_first_page)
        page = self._validate_page_number(request.GET.get('p', '0'))
        return FastCountPaginator(page, queryset, per_page, orphans, allow_empty_first_page)

    def _validate_page_number(self, number):
        # taken from Paginator.validate_number and adjusted
        try:
            if isinstance(number, float) and not number.is_integer():
                raise ValueError
            number = int(number)
        except (TypeError, ValueError):
            return 0
        if number < 1:
            number = 0
        return number

pagination.html模板:

{% if cl and cl.paginator and cl.paginator.use_fast_pagination %}
    {# Fast paginator with only next button and show the total number of results#}
    {% load admin_list %}
    {% load i18n %}
    {% load admin_templatetags %}
    <p class="paginator">
        {% if pagination_required %}
            {% for i in page_range %}
                {% if forloop.last %}
                    {% fast_paginator_number cl i 'Next' %}
                {% else %}
                    {% fast_paginator_number cl i %}
                {% endif %}
            {% endfor %}
        {% endif %}
        {% show_count_all_link cl "showall" %}
    </p>
{% else %}
    {#  use the default pagination template if we are not using the FastPaginator  #}
    {% include "admin/pagination.html" %}
{% endif %}

和使用的模板标签:

from django import template
from django.contrib.admin.views.main import PAGE_VAR
from django.utils.html import format_html
from django.utils.safestring import mark_safe

register = template.Library()

DOT = '.'


@register.simple_tag
def fast_paginator_number(cl, i, text_display=None):
    """Generate an individual page index link in a paginated list.

    Allows to change the link text by setting text_display
    """
    if i == DOT:
        return '… '
    elif i == cl.page_num:
        return format_html('<span class="this-page">{}</span> ', i + 1)
    else:
        return format_html(
            '<a href="{}"{}>{}</a> ',
            cl.get_query_string({PAGE_VAR: i}),
            mark_safe(' class="end"' if i == cl.paginator.num_pages - 1 else ''),
            text_display if text_display else i + 1,
        )


@register.simple_tag
def show_count_all_link(cl, css_class='', text_display='Show the total number of results'):
    """Generate a button that toggles between FastPaginator and the normal
    Paginator."""
    return format_html(
        '<a href="{}"{}>{}</a> ',
        cl.get_query_string({PAGE_VAR: cl.page_num, 'count_all': True}),
        mark_safe(f' class="{css_class}"' if css_class else ''),
        text_display,
    )

你可以这样使用它:

class MyVeryLargeModelAdmin(ModelAdminFastPaginationMixin, admin.ModelAdmin):
# ...

或者更简单的版本,不显示Next按钮和Show the total number of results

from django.core.paginator import Paginator
from django.utils.functional import cached_property


class FastCountPaginator(Paginator):
    """A faster paginator implementation than the Paginator. Paginator is slow
    mainly because QuerySet.count() is expensive on large queries.

    The idea is to use the requested page to generate a 'fake' count. In
    order to see if the page is the final one  it queries n+1 elements
    from db then reports the count as page_number * per_page + received_elements.
    """

    use_fast_pagination = True

    def __init__(self, page_number, *args, **kwargs):
        self.page_number = page_number
        super(FastCountPaginator, self).__init__(*args, **kwargs)

    @cached_property
    def count(self):
        # Populate the object list when count is called. As this is a cached property,
        # it will be called only once per instance
        return self.populate_object_list()

    def page(self, page_number):
        """Return a Page object for the given 1-based page number."""
        page_number = self.validate_number(page_number)
        return self._get_page(self.object_list, page_number, self)

    def populate_object_list(self):
        # converts queryset object_list to a list and return the number of elements until there
        # the trick is to get per_page elements + 1 in order to see if the next page exists.
        bottom = self.page_number * self.per_page
        # get one more object than needed to see if we should show next page
        top = bottom + self.per_page + 1
        object_list = list(self.object_list[bottom:top])
        # not the last page
        if len(object_list) == self.per_page + 1:
            object_list = object_list[:-1]
        else:
            top = bottom + len(object_list)
        self.object_list = object_list
        return top


class ModelAdminFastPaginationMixin:
    show_full_result_count = False  # prevents root_queryset.count() call

    def get_paginator(self, request, queryset, per_page, orphans=0, allow_empty_first_page=True):
        page = self._validate_page_number(request.GET.get('p', '0'))
        return FastCountPaginator(page, queryset, per_page, orphans, allow_empty_first_page)

    def _validate_page_number(self, number):
        # taken from Paginator.validate_number and adjusted
        try:
            if isinstance(number, float) and not number.is_integer():
                raise ValueError
            number = int(number)
        except (TypeError, ValueError):
            return 0
        if number < 1:
            number = 0
        return number

于 2019-11-19T08:15:32.153 回答