7

我正在寻找一些关于如何更好地优化此查询的建议。

对于每条_piece_detail记录：

  1. _scan在 (zip, zip_4, zip_delivery_point, serial_number) 上包含至少一条匹配记录
  2. 属于一家公司mailing_groups（通过一系列关系）
  3. 有：
    1. first_scan_date_time大于MIN(scan_date_time)相关的 _scan 记录
    2. latest_scan_date_time小于MAX(scan_date_time)相关_scan记录的

我将需要：

  1. 设置_piece_detail.first_scan_date_time为MIN(_scan.scan_date_time)
  2. 设置_piece_detail.latest_scan_date_time为MAX(_scan.scan_date_time)

由于我正在处理数以百万计的记录，因此我试图减少我实际必须搜索的记录数量。以下是有关数据的一些事实：

  1. _piece_details 表由 分区job_id，因此按 , 的顺序运行这些检查似乎是最有意义 _piece_detail.job_id的_piece_detail.piece_id。
  2. 扫描记录表现在包含超过 100,000,000 条记录，并按 (zip, zip_4, zip_delivery_point, serial_number, scan_date_time) 进行分区，这是用于将 _scan 与 a 匹配的相同键_piece_detail（除了 scan_date_time）。
  3. 只有大约 40% 的_piece_detail记录属于 a mailing_group，但在我们运行完整的连接关系之前，我们不知道这些记录是哪些。
  4. 只有大约 30% 的 _scan 记录属于_piece_detail带有 a 的 a mailing_group。
  5. _scan每个_piece_detail. _

现在，我正在寻找一种以体面的方式执行此操作的方法。我最初是从这样的事情开始的：

UPDATE _piece_detail
    INNER JOIN (
        SELECT _piece_detail.job_id, _piece_detail.piece_id, MIN(_scan.scan_date_time) as first_scan_date_time, MAX(_scan.scan_date_time) as latest_scan_date_time
        FROM _piece_detail
            INNER JOIN _container_quantity 
                ON _piece_detail.cqt_database_id = _container_quantity.cqt_database_id 
                AND _piece_detail.job_id = _container_quantity.job_id
            INNER JOIN _container_summary 
                ON _container_quantity.container_id = _container_summary.container_id 
                AND _container_summary.job_id = _container_quantity.job_id
            INNER JOIN _mail_piece_unit 
                ON _container_quantity.mpu_id = _mail_piece_unit.mpu_id 
                AND _container_quantity.job_id = _mail_piece_unit.job_id
            INNER JOIN _header 
                ON _header.job_id = _piece_detail.job_id
            INNER JOIN mailing_groups 
                ON _mail_piece_unit.mpu_company = mailing_groups.mpu_company
            INNER JOIN _scan
                ON _scan.zip = _piece_detail.zip 
                AND _scan.zip_4 = _piece_detail.zip_4 
                AND _scan.zip_delivery_point = _piece_detail.zip_delivery_point 
                AND _scan.serial_number = _piece_detail.serial_number 
        GROUP BY _piece_detail.job_id, _piece_detail.piece_id, _scan.zip, _scan.zip_4, _scan.zip_delivery_point, _scan.serial_number
    ) as t1 ON _piece_detail.job_id = t1.job_id AND _piece_detail.piece_id = t1.piece_id 
SET _piece_detail.first_scan_date_time = t1.first_scan_date_time, _piece_detail.latest_scan_date_time = t1.latest_scan_date_time
WHERE _piece_detail.first_scan_date_time < t1.first_scan_date_time 
    OR _piece_detail.latest_scan_date_time > t1.latest_scan_date_time;

我认为这可能试图一次将太多内容加载到内存中，并且可能没有正确使用索引。

然后我想我也许可以避免做那个巨大的连接子查询并添加两个左连接子查询来获得最小值/最大值，如下所示：

UPDATE _piece_detail
    INNER JOIN _container_quantity 
        ON _piece_detail.cqt_database_id = _container_quantity.cqt_database_id 
        AND _piece_detail.job_id = _container_quantity.job_id
    INNER JOIN _container_summary 
        ON _container_quantity.container_id = _container_summary.container_id 
        AND _container_summary.job_id = _container_quantity.job_id
    INNER JOIN _mail_piece_unit 
        ON _container_quantity.mpu_id = _mail_piece_unit.mpu_id 
        AND _container_quantity.job_id = _mail_piece_unit.job_id
    INNER JOIN _header 
        ON _header.job_id = _piece_detail.job_id
    INNER JOIN mailing_groups 
        ON _mail_piece_unit.mpu_company = mailing_groups.mpu_company
    LEFT JOIN _scan fs ON (fs.zip, fs.zip_4, fs.zip_delivery_point, fs.serial_number) = (
        SELECT zip, zip_4, zip_delivery_point, serial_number
        FROM _scan
        WHERE zip = _piece_detail.zip 
            AND zip_4 = _piece_detail.zip_4 
            AND zip_delivery_point = _piece_detail.zip_delivery_point 
            AND serial_number = _piece_detail.serial_number
        ORDER BY scan_date_time ASC
        LIMIT 1
        )
    LEFT JOIN _scan ls ON (ls.zip, ls.zip_4, ls.zip_delivery_point, ls.serial_number) = (
        SELECT zip, zip_4, zip_delivery_point, serial_number
        FROM _scan
        WHERE zip = _piece_detail.zip 
            AND zip_4 = _piece_detail.zip_4 
            AND zip_delivery_point = _piece_detail.zip_delivery_point 
            AND serial_number = _piece_detail.serial_number
        ORDER BY scan_date_time DESC
        LIMIT 1
        )
SET _piece_detail.first_scan_date_time = fs.scan_date_time, _piece_detail.latest_scan_date_time = ls.scan_date_time
WHERE _piece_detail.first_scan_date_time < fs.scan_date_time 
    OR _piece_detail.latest_scan_date_time > ls.scan_date_time

这些是我将它们转换为 SELECT 语句时的解释：

+----+-------------+---------------------+--------+----------------------------------------------------+---------------+---------+------------------------------------------------------------------------------------------------------------------------+--------+----------------------------------------------+
| id | select_type | table               | type   | possible_keys                                      | key           | key_len | ref                                                                                                                    | rows   | Extra                                        |
+----+-------------+---------------------+--------+----------------------------------------------------+---------------+---------+------------------------------------------------------------------------------------------------------------------------+--------+----------------------------------------------+
|  1 | PRIMARY     | <derived2>          | ALL    | NULL                                               | NULL          | NULL    | NULL                                                                                                                   | 844161 | NULL                                         |
|  1 | PRIMARY     | _piece_detail       | eq_ref | PRIMARY,first_scan_date_time,latest_scan_date_time | PRIMARY       | 18      | t1.job_id,t1.piece_id                                                                                                  |      1 | Using where                                  |
|  2 | DERIVED     | _header             | index  | PRIMARY                                            | date_prepared | 3       | NULL                                                                                                                   |     87 | Using index; Using temporary; Using filesort |
|  2 | DERIVED     | _piece_detail       | ref    | PRIMARY,cqt_database_id,zip                        | PRIMARY       | 10      | odms._header.job_id                                                                                                    |   9703 | NULL                                         |
|  2 | DERIVED     | _container_quantity | eq_ref | unique,mpu_id,job_id,job_id_container_quantity     | unique        | 14      | odms._header.job_id,odms._piece_detail.cqt_database_id                                                                 |      1 | NULL                                         |
|  2 | DERIVED     | _mail_piece_unit    | eq_ref | PRIMARY,company,job_id_mail_piece_unit             | PRIMARY       | 14      | odms._container_quantity.mpu_id,odms._header.job_id                                                                    |      1 | Using where                                  |
|  2 | DERIVED     | mailing_groups      | eq_ref | PRIMARY                                            | PRIMARY       | 27      | odms._mail_piece_unit.mpu_company                                                                                      |      1 | Using index                                  |
|  2 | DERIVED     | _container_summary  | eq_ref | unique,container_id,job_id_container_summary       | unique        | 14      | odms._header.job_id,odms._container_quantity.container_id                                                              |      1 | Using index                                  |
|  2 | DERIVED     | _scan               | ref    | PRIMARY                                            | PRIMARY       | 28      | odms._piece_detail.zip,odms._piece_detail.zip_4,odms._piece_detail.zip_delivery_point,odms._piece_detail.serial_number |      1 | Using index                                  |
+----+-------------+---------------------+--------+----------------------------------------------------+---------------+---------+------------------------------------------------------------------------------------------------------------------------+--------+----------------------------------------------+

+----+--------------------+---------------------+--------+--------------------------------------------------------------------+---------------+---------+------------------------------------------------------------------------------------------------------------------------+-----------+-----------------------------------------------------------------+
| id | select_type        | table               | type   | possible_keys                                                      | key           | key_len | ref                                                                                                                    | rows      | Extra                                                           |
+----+--------------------+---------------------+--------+--------------------------------------------------------------------+---------------+---------+------------------------------------------------------------------------------------------------------------------------+-----------+-----------------------------------------------------------------+
|  1 | PRIMARY            | _header             | index  | PRIMARY                                                            | date_prepared | 3       | NULL                                                                                                                   |        87 | Using index                                                     |
|  1 | PRIMARY            | _piece_detail       | ref    | PRIMARY,cqt_database_id,first_scan_date_time,latest_scan_date_time | PRIMARY       | 10      | odms._header.job_id                                                                                                    |      9703 | NULL                                                            |
|  1 | PRIMARY            | _container_quantity | eq_ref | unique,mpu_id,job_id,job_id_container_quantity                     | unique        | 14      | odms._header.job_id,odms._piece_detail.cqt_database_id                                                                 |         1 | NULL                                                            |
|  1 | PRIMARY            | _mail_piece_unit    | eq_ref | PRIMARY,company,job_id_mail_piece_unit                             | PRIMARY       | 14      | odms._container_quantity.mpu_id,odms._header.job_id                                                                    |         1 | Using where                                                     |
|  1 | PRIMARY            | mailing_groups      | eq_ref | PRIMARY                                                            | PRIMARY       | 27      | odms._mail_piece_unit.mpu_company                                                                                      |         1 | Using index                                                     |
|  1 | PRIMARY            | _container_summary  | eq_ref | unique,container_id,job_id_container_summary                       | unique        | 14      | odms._header.job_id,odms._container_quantity.container_id                                                              |         1 | Using index                                                     |
|  1 | PRIMARY            | fs                  | index  | NULL                                                               | updated       | 1       | NULL                                                                                                                   | 102462928 | Using where; Using index; Using join buffer (Block Nested Loop) |
|  1 | PRIMARY            | ls                  | index  | NULL                                                               | updated       | 1       | NULL                                                                                                                   | 102462928 | Using where; Using index; Using join buffer (Block Nested Loop) |
|  3 | DEPENDENT SUBQUERY | _scan               | ref    | PRIMARY                                                            | PRIMARY       | 28      | odms._piece_detail.zip,odms._piece_detail.zip_4,odms._piece_detail.zip_delivery_point,odms._piece_detail.serial_number |         1 | Using where; Using index; Using filesort                        |
|  2 | DEPENDENT SUBQUERY | _scan               | ref    | PRIMARY                                                            | PRIMARY       | 28      | odms._piece_detail.zip,odms._piece_detail.zip_4,odms._piece_detail.zip_delivery_point,odms._piece_detail.serial_number |         1 | Using where; Using index; Using filesort                        |
+----+--------------------+---------------------+--------+--------------------------------------------------------------------+---------------+---------+------------------------------------------------------------------------------------------------------------------------+-----------+-----------------------------------------------------------------+

现在，看看每个人生成的解释，我真的不知道哪个是最划算的。第一个在乘以 rows 列时显示的总行数较少，但第二个似乎执行得更快一些。

在通过修改查询结构提高性能的同时，我可以做些什么来获得相同的结果？

4

4 回答 4

1

我被教导了一些东西，直到今天我都严格遵守 - 创建尽可能多的临时表，同时避免使用派生表。特别是在更新/删除/插入的情况下

  1. 您无法预测派生表上的索引
  2. 如果结果集很大，派生表可能不会保存在内存中
  3. 每次派生查询运行时，表（MyIsam）/行（Innodb）可能会被锁定更长的时间。我更喜欢与父表具有主键连接的临时表。

最重要的是，它使您的代码看起来整洁易读。

我的方法是

CREATE table temp xxx(...)
INSERT INTO xxx select q from y inner join z....;
UPDATE _piece_detail INNER JOIN xxx on (...) SET ...;

始终减少您的停机时间！！

于 2014-02-01T17:33:13.330 回答
1

在进行批量更新时禁用索引更新

ALTER TABLE _piece_detail DISABLE KEYS;

UPDATE ....;

ALTER TABLE _piece_detail ENABLE KEYS;

请参阅 mysql 文档：http ://dev.mysql.com/doc/refman/5.0/en/alter-table.html

编辑：查看我指向的 mysql 文档后，我看到文档为 MyISAM 表指定了这个，而对于其他表类型则不清楚。更多解决方案：如何在 innodb 中禁用索引

于 2014-01-12T22:07:29.640 回答
0

从您的解释结果看来，子查询似乎两次遍历所有行，那么您如何保留第一个的 MIN/MAX 并仅使用一个左连接而不是两个？

于 2014-01-12T20:39:02.777 回答
0

为什么不为每个连接使用子查询？包括内连接？

INNER JOIN (SELECT field1, field2, field 3 from _container_quantity order by 1,2,3) 
    ON _piece_detail.cqt_database_id = _container_quantity.cqt_database_id 
    AND _piece_detail.job_id = _container_quantity.job_id
INNER JOIN (SELECT field1, field2, field3 from _container_summary order by 1,2,3)
    ON _container_quantity.container_id = _container_summary.container_id 
    AND _container_summary.job_id = _container_quantity.job_id

通过不限制您对这些内部连接的选择，您肯定会在内存中提取很多内容。通过在每个子查询的末尾使用 1,2,3 的顺序，您可以在每个子查询上创建一个索引。您唯一的索引在标题上，而您没有加入 _headers....

一些优化此查询的建议。在每个表上创建所需的索引，或使用子查询连接子句动态手动创建所需的索引。

还要记住，当您在充满聚合的“临时”表上执行左连接时，您只是在要求性能问题。

在 (zip, zip_4, zip_delivery_point, serial_number) 上包含至少一个匹配的 _scan 记录

嗯...这是您想要做的第一点，但是这些字段都没有被索引？

于 2013-12-18T19:50:57.957 回答