mysql - 随着用户活动的增加，查询增长缓慢

Question

下面的查询执行需要 0.6748 秒，但是当有许多用户同时尝试执行相同的活动时，进程会变得非常慢，执行大约需要 30 秒。

服务器带宽：7TB

我曾多次尝试优化查询并提高性能，但随着数据库变大，性能再次下降，我别无选择。

请协助帮助进一步优化我的查询

SELECT * FROM (SELECT * FROM
                    (SELECT
                         c.loan_id ,c.channel tu_channel, c.user_unique_id tu_user_unique_id ,c.ippis tu_ippis ,c.top_up_approved, c.ret_customer_bal ,c.loan_type tu_loan_type ,c.dse ,c.status ,c.current_loan tu_current_loan ,c.record_category ,c.date_created tu_date_created ,c.date_updated tu_date_updated , a.id ,a.user_unique_id ,a.loan_location ,a.ippis ,a.tel_no ,a.organisation ,a.branch ,a.loan_agree ,a.loan_type ,a.appr ,a.sold ,a.loan_status , a.channel, a.top_up ,a.current_loan, a.retrieved_customer_bal ,a.date_created ,a.date_updated 
                    FROM
                        loan_applications_tbl a 
                    LEFT JOIN
                        topup_or_reapplication_tbl AS c 
                            ON a.ippis = c.ippis 
                    WHERE
                        (
                            (
                                (
                                    (
                                        a.current_loan = '0' 
                                        AND a.loan_status IN (
                                            'Approved', 'Closed'
                                        )
                                    )
                                ) 
                                AND (
                                    ((a.loan_status='pending' 
                            AND a.appr < 500000) OR ( a.loan_status IN ( 'corrected', 'Approved By Group Head', 'NIL', 
                    'NIL', 'NIL', 'NIL' )  )) 
                                ) 
                                AND (
                                    (
                                        MONTH(a.date_updated) = MONTH(CURRENT_DATE) 
                                        AND YEAR(a.date_updated) = YEAR(CURRENT_DATE)
                                    )
                                )
                            )
                        ) 
                    UNION ALL 
                    SELECT
                         c.loan_id ,c.channel tu_channel, c.user_unique_id tu_user_unique_id ,c.ippis tu_ippis ,c.top_up_approved, c.ret_customer_bal ,c.loan_type tu_loan_type ,c.dse ,c.status ,c.current_loan tu_current_loan ,c.record_category ,c.date_created tu_date_created ,c.date_updated tu_date_updated, b.loan_id ,b.user_unique_id ,b.loan_location ,b.ippis ,b.tel_no ,b.organisation ,b.branch ,b.loan_agree ,b.loan_type ,b.appr ,b.sold ,b.loan_status ,b.channel, b.top_up ,b.current_loan, b.retrieved_customer_bal ,b.date_created ,b.date_updated 
                    FROM
                        loan_applications_tbl_dump b 
                    LEFT JOIN
                        topup_or_reapplication_tbl c 
                            ON b.ippis = c.ippis 
                    WHERE
                        (
                            (
                                (
                                    (
                                        b.current_loan = '0' 
                                        AND b.loan_status IN (
                                            'Approved', 'Closed'
                                        )
                                    )
                                ) 
                                AND (
                                    ((b.loan_status='pending' 
                            AND b.appr < 500000) OR (b.loan_status IN ( 'corrected', 'Approved By Group Head', 'NIL', 
                        'NIL', 'NIL', 'NIL' ) )) 
                                ) 
                                AND (
                                    (
                                        MONTH(b.date_updated) = MONTH(CURRENT_DATE) 
                                        AND YEAR(b.date_updated) = YEAR(CURRENT_DATE)
                                    )
                                )
                            )
                        ) 
                    UNION DISTINCT 
                    SELECT
                         c.loan_id ,c.channel tu_channel, c.user_unique_id tu_user_unique_id ,c.ippis tu_ippis ,c.top_up_approved, c.ret_customer_bal ,c.loan_type tu_loan_type ,c.dse ,c.status ,c.current_loan tu_current_loan ,c.record_category ,c.date_created tu_date_created ,c.date_updated tu_date_updated , a.id ,a.user_unique_id ,a.loan_location ,a.ippis ,a.tel_no ,a.organisation ,a.branch ,a.loan_agree ,a.loan_type ,a.appr ,a.sold ,a.loan_status , a.channel, a.top_up ,a.current_loan, a.retrieved_customer_bal ,a.date_created ,a.date_updated 
                    FROM
                        loan_applications_tbl a 
                    INNER JOIN
                        topup_or_reapplication_tbl AS c 
                            ON a.ippis = c.ippis 
                    WHERE
                        (
                            (
                                c.current_loan = '1' 
                                AND (
                                    ((c.status='pending' 
                            AND c.top_up_approved < 500000)  OR (c.status IN ('corrected', 'Approved By Group Head', 'NIL', 
                        'NIL', 'NIL', 'NIL') )) 
                                ) 
                                AND (
                                    (
                                        MONTH(c.date_updated) = MONTH(CURRENT_DATE) 
                                        AND YEAR(c.date_updated) = YEAR(CURRENT_DATE)
                                    )
                                )
                            )
                        ) 
                    UNION
                    DISTINCT SELECT
                         c.loan_id ,c.channel tu_channel, c.user_unique_id tu_user_unique_id ,c.ippis tu_ippis ,c.top_up_approved, c.ret_customer_bal ,c.loan_type tu_loan_type ,c.dse ,c.status ,c.current_loan tu_current_loan ,c.record_category ,c.date_created tu_date_created ,c.date_updated tu_date_updated, b.loan_id ,b.user_unique_id ,b.loan_location ,b.ippis ,b.tel_no ,b.organisation ,b.branch ,b.loan_agree ,b.loan_type ,b.appr ,b.sold ,b.loan_status ,b.channel, b.top_up ,b.current_loan, b.retrieved_customer_bal ,b.date_created ,b.date_updated 
                    FROM
                        loan_applications_tbl_dump b 
                    INNER JOIN
                        topup_or_reapplication_tbl c 
                            ON b.ippis = c.ippis 
                    WHERE
                        (
                            (
                                c.current_loan = '1' 
                                AND (
                                    ((c.status='pending' 
                            AND c.top_up_approved < 500000)  OR (c.status IN ('corrected', 'Approved By Group Head', 'NIL', 
                        'NIL', 'NIL', 'NIL') )) 
                                ) 
                                AND (
                                    (
                                        MONTH(c.date_updated) = MONTH(CURRENT_DATE) 
                                        AND YEAR(c.date_updated) = YEAR(CURRENT_DATE)
                                    )
                                )
                            )
                        ) 
                    UNION
                    DISTINCT SELECT
                         c.loan_id ,c.channel tu_channel, c.user_unique_id tu_user_unique_id ,c.ippis tu_ippis ,c.top_up_approved, c.ret_customer_bal ,c.loan_type tu_loan_type ,c.dse ,c.status ,c.current_loan tu_current_loan ,c.record_category ,c.date_created tu_date_created ,c.date_updated tu_date_updated , a.id ,a.user_unique_id ,a.loan_location ,a.ippis ,a.tel_no ,a.organisation ,a.branch ,a.loan_agree ,a.loan_type ,a.appr ,a.sold ,a.loan_status , a.channel, a.top_up ,a.current_loan, a.retrieved_customer_bal ,a.date_created ,a.date_updated 
                    FROM
                        loan_applications_tbl a 
                    LEFT JOIN
                        topup_or_reapplication_tbl AS c 
                            ON a.ippis = c.ippis 
                    WHERE
                        (
                            (
                                (
                                    a.current_loan = '1'
                                ) 
                                AND (
                                    ((a.loan_status='pending' 
                            AND a.appr < 500000) OR ( a.loan_status IN ( 'corrected', 'Approved By Group Head', 'NIL', 
                    'NIL', 'NIL', 'NIL' )  )) 
                                ) 
                                AND (
                                    (
                                        MONTH(a.date_updated) = MONTH(CURRENT_DATE) 
                                        AND YEAR(a.date_updated) = YEAR(CURRENT_DATE)
                                    )
                                )
                            )
                        ) 
                    UNION
                    DISTINCT SELECT
                         c.loan_id ,c.channel tu_channel, c.user_unique_id tu_user_unique_id ,c.ippis tu_ippis ,c.top_up_approved, c.ret_customer_bal ,c.loan_type tu_loan_type ,c.dse ,c.status ,c.current_loan tu_current_loan ,c.record_category ,c.date_created tu_date_created ,c.date_updated tu_date_updated, b.loan_id ,b.user_unique_id ,b.loan_location ,b.ippis ,b.tel_no ,b.organisation ,b.branch ,b.loan_agree ,b.loan_type ,b.appr ,b.sold ,b.loan_status ,b.channel, b.top_up ,b.current_loan, b.retrieved_customer_bal ,b.date_created ,b.date_updated 
                    FROM
                        loan_applications_tbl_dump b 
                    LEFT JOIN
                        topup_or_reapplication_tbl c 
                            ON b.ippis = c.ippis 
                    WHERE
                        (
                            (
                                (
                                    b.current_loan = '1'
                                ) 
                                AND (
                                    ((b.loan_status='pending' 
                            AND b.appr < 500000) OR (b.loan_status IN ( 'corrected', 'Approved By Group Head', 'NIL', 
                        'NIL', 'NIL', 'NIL' ) )) 
                                ) 
                                AND (
                                    (
                                        MONTH(b.date_updated) = MONTH(CURRENT_DATE) 
                                        AND YEAR(b.date_updated) = YEAR(CURRENT_DATE)
                                    )
                                )
                            )
                        ) 
                    UNION
                    DISTINCT SELECT
                         c.loan_id ,c.channel tu_channel, c.user_unique_id tu_user_unique_id ,c.ippis tu_ippis ,c.top_up_approved, c.ret_customer_bal ,c.loan_type tu_loan_type ,c.dse ,c.status ,c.current_loan tu_current_loan ,c.record_category ,c.date_created tu_date_created ,c.date_updated tu_date_updated , a.id ,a.user_unique_id ,a.loan_location ,a.ippis ,a.tel_no ,a.organisation ,a.branch ,a.loan_agree ,a.loan_type ,a.appr ,a.sold ,a.loan_status , a.channel, a.top_up ,a.current_loan, a.retrieved_customer_bal ,a.date_created ,a.date_updated 
                    FROM
                        loan_applications_tbl a 
                    INNER JOIN
                        topup_or_reapplication_tbl AS c 
                            ON a.ippis = c.ippis 
                    WHERE
                        (
                            (
                                c.current_loan = '1' 
                                AND (
                                    ((c.status='pending' 
                            AND c.top_up_approved < 500000)  OR (c.status IN ('corrected', 'Approved By Group Head', 'NIL', 
                        'NIL', 'NIL', 'NIL') )) 
                                ) 
                                AND (
                                    (
                                        MONTH(c.date_created) = MONTH(CURRENT_DATE) 
                                        AND YEAR(c.date_created) = YEAR(CURRENT_DATE)
                                    )
                                )
                            )
                        ) 
                    UNION
                    DISTINCT SELECT
                         c.loan_id ,c.channel tu_channel, c.user_unique_id tu_user_unique_id ,c.ippis tu_ippis ,c.top_up_approved, c.ret_customer_bal ,c.loan_type tu_loan_type ,c.dse ,c.status ,c.current_loan tu_current_loan ,c.record_category ,c.date_created tu_date_created ,c.date_updated tu_date_updated, b.loan_id ,b.user_unique_id ,b.loan_location ,b.ippis ,b.tel_no ,b.organisation ,b.branch ,b.loan_agree ,b.loan_type ,b.appr ,b.sold ,b.loan_status ,b.channel, b.top_up ,b.current_loan, b.retrieved_customer_bal ,b.date_created ,b.date_updated 
                    FROM
                        loan_applications_tbl_dump b 
                    INNER JOIN
                        topup_or_reapplication_tbl c 
                            ON b.ippis = c.ippis 
                    WHERE
                        (
                            (
                                c.current_loan = '1' 
                                AND (
                                    ((c.status='pending' 
                            AND c.top_up_approved < 500000)  OR (c.status IN ('corrected', 'Approved By Group Head', 'NIL', 
                        'NIL', 'NIL', 'NIL') )) 
                                ) 
                                AND (
                                    (
                                        MONTH(c.date_created) = MONTH(CURRENT_DATE) 
                                        AND YEAR(c.date_created) = YEAR(CURRENT_DATE)
                                    )
                                )
                            )
                        ) 
                    UNION
                    DISTINCT SELECT
                         c.loan_id ,c.channel tu_channel, c.user_unique_id tu_user_unique_id ,c.ippis tu_ippis ,c.top_up_approved, c.ret_customer_bal ,c.loan_type tu_loan_type ,c.dse ,c.status ,c.current_loan tu_current_loan ,c.record_category ,c.date_created tu_date_created ,c.date_updated tu_date_updated , a.id ,a.user_unique_id ,a.loan_location ,a.ippis ,a.tel_no ,a.organisation ,a.branch ,a.loan_agree ,a.loan_type ,a.appr ,a.sold ,a.loan_status , a.channel, a.top_up ,a.current_loan, a.retrieved_customer_bal ,a.date_created ,a.date_updated 
                    FROM
                        loan_applications_tbl a 
                    LEFT JOIN
                        topup_or_reapplication_tbl AS c 
                            ON a.ippis = c.ippis 
                    WHERE
                        (
                            (
                                (
                                    (
                                        a.current_loan = '0' 
                                        AND a.loan_status IN (
                                            'Approved', 'Closed'
                                        )
                                    )
                                ) 
                                AND (
                                    ((a.loan_status='pending' 
                            AND a.appr < 500000) OR ( a.loan_status IN ( 'corrected', 'Approved By Group Head', 'NIL', 
                    'NIL', 'NIL', 'NIL' )  )) 
                                ) 
                                AND (
                                    (
                                        MONTH(a.date_created) = MONTH(CURRENT_DATE) 
                                        AND YEAR(a.date_created) = YEAR(CURRENT_DATE)
                                    )
                                )
                            )
                        ) 
                    UNION
                    DISTINCT SELECT
                         c.loan_id ,c.channel tu_channel, c.user_unique_id tu_user_unique_id ,c.ippis tu_ippis ,c.top_up_approved, c.ret_customer_bal ,c.loan_type tu_loan_type ,c.dse ,c.status ,c.current_loan tu_current_loan ,c.record_category ,c.date_created tu_date_created ,c.date_updated tu_date_updated, b.loan_id ,b.user_unique_id ,b.loan_location ,b.ippis ,b.tel_no ,b.organisation ,b.branch ,b.loan_agree ,b.loan_type ,b.appr ,b.sold ,b.loan_status ,b.channel, b.top_up ,b.current_loan, b.retrieved_customer_bal ,b.date_created ,b.date_updated 
                    FROM
                        loan_applications_tbl_dump b 
                    LEFT JOIN
                        topup_or_reapplication_tbl c 
                            ON b.ippis = c.ippis 
                    WHERE
                        (
                            (
                                (
                                    (
                                        b.current_loan = '0' 
                                        AND b.loan_status IN (
                                            'Approved', 'Closed'
                                        )
                                    )
                                ) 
                                AND (
                                    ((b.loan_status='pending' 
                            AND b.appr < 500000) OR (b.loan_status IN ( 'corrected', 'Approved By Group Head', 'NIL', 
                        'NIL', 'NIL', 'NIL' ) )) 
                                ) 
                                AND (
                                    (
                                        MONTH(b.date_created) = MONTH(CURRENT_DATE) 
                                        AND YEAR(b.date_created) = YEAR(CURRENT_DATE)
                                    )
                                )
                            )
                        ) 
                    UNION
                    DISTINCT SELECT
                         c.loan_id ,c.channel tu_channel, c.user_unique_id tu_user_unique_id ,c.ippis tu_ippis ,c.top_up_approved, c.ret_customer_bal ,c.loan_type tu_loan_type ,c.dse ,c.status ,c.current_loan tu_current_loan ,c.record_category ,c.date_created tu_date_created ,c.date_updated tu_date_updated , a.id ,a.user_unique_id ,a.loan_location ,a.ippis ,a.tel_no ,a.organisation ,a.branch ,a.loan_agree ,a.loan_type ,a.appr ,a.sold ,a.loan_status , a.channel, a.top_up ,a.current_loan, a.retrieved_customer_bal ,a.date_created ,a.date_updated 
                    FROM
                        loan_applications_tbl a 
                    LEFT JOIN
                        topup_or_reapplication_tbl AS c 
                            ON a.ippis = c.ippis 
                    WHERE
                        (
                            (
                                (
                                    a.current_loan = '1'
                                ) 
                                AND (
                                    ((a.loan_status='pending' 
                            AND a.appr < 500000) OR ( a.loan_status IN ( 'corrected', 'Approved By Group Head', 'NIL', 
                    'NIL', 'NIL', 'NIL' )  )) 
                                ) 
                                AND (
                                    (
                                        MONTH(a.date_created) = MONTH(CURRENT_DATE) 
                                        AND YEAR(a.date_created) = YEAR(CURRENT_DATE)
                                    )
                                )
                            )
                        ) 
                    UNION
                    DISTINCT SELECT
                         c.loan_id ,c.channel tu_channel, c.user_unique_id tu_user_unique_id ,c.ippis tu_ippis ,c.top_up_approved, c.ret_customer_bal ,c.loan_type tu_loan_type ,c.dse ,c.status ,c.current_loan tu_current_loan ,c.record_category ,c.date_created tu_date_created ,c.date_updated tu_date_updated, b.loan_id ,b.user_unique_id ,b.loan_location ,b.ippis ,b.tel_no ,b.organisation ,b.branch ,b.loan_agree ,b.loan_type ,b.appr ,b.sold ,b.loan_status ,b.channel, b.top_up ,b.current_loan, b.retrieved_customer_bal ,b.date_created ,b.date_updated 
                    FROM
                        loan_applications_tbl_dump b 
                    LEFT JOIN
                        topup_or_reapplication_tbl c 
                            ON b.ippis = c.ippis 
                    WHERE
                        (
                            (
                                (
                                    b.current_loan = '1'
                                ) 
                                AND (
                                    ((b.loan_status='pending' 
                            AND b.appr < 500000) OR (b.loan_status IN ( 'corrected', 'Approved By Group Head', 'NIL', 
                        'NIL', 'NIL', 'NIL' ) )) 
                                ) 
                                AND (
                                    (
                                        MONTH(b.date_created) = MONTH(CURRENT_DATE) 
                                        AND YEAR(b.date_created) = YEAR(CURRENT_DATE)
                                    )
                                )
                            )
                        )
                    ) t3 ORDER BY t3.date_updated, t3.tu_date_updated DESC LIMIT 18446744073709551615) AS t4 GROUP BY t4.ippis

EXPLAIN 语句给了我下表：

Answer 1

这是一个非常大的查询。对于 Stack Overflow 问题来说，甚至可能太大了。

1. 我建议您尝试一次优化其中一个子查询（您的 UNION 操作的一个分支）。这样你就不必一次考虑整个混乱。

2. 如果可以，请使用UNION ALL而不是 UNION DISTINCT；UNION DISTINCT 对其结果集进行重复数据删除，这会占用 CPU 时间和 RAM。

3. 你有这个 WHERE 子句模式重复多次。

       MONTH(a.date_updated) = MONTH(CURRENT_DATE)
   AND YEAR(a.date_updated) = YEAR(CURRENT_DATE)
   

   这不是sargable。也就是说，它的编写是为了避免在loan_applications_tbl.date_updated. 请改用此等效项。如果你这样做 MySQL 将能够对索引进行范围扫描。

       a.date_updated >= LAST_DAY(CURRENT_DATE) + 1 DAY - 1 MONTH
   AND a.date_updated <  LAST_DAY(CURRENT_DATE) + 1 DAY
   

   LAST_DAY(CURRENT_DATE)得到当月最后一天的午夜，其余的日期算术得到本月第一天的午夜和下个月第一天的午夜。

4. 你没有告诉我们任何关于你的索引的事情。您的查询有 OR，这会使它们变慢，因此您必须对此进行试验。如果您还没有，请尝试创建此索引。

   ALTER TABLE loan_applications_tbl 
    ADD INDEX cur_stat_date 
              (current_loan, loan_status, date_updated, appr)
   

   这可能会有所帮助，因为它允许 MySQL 随机访问我建议的索引，然后按顺序读取它。

   你需要一个类似的索引loan_applications_tbl_dump。

5. 当您有很多用户时，您可能会看到争用。如果您的应用程序可以容忍检索对于当前正在插入和更新的行可能不完全正确的数据，请在查询之前给出此命令。

   SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;
   

   如果您的应用程序使用大量事务，这对于数据完整性可能有点风险。但这会减少争吵。

6. 您可以重构查询以减少重复性吗？

Answer 2

这些可能会有所帮助：

c:  INDEX(current_loan, status, top_up_approved, ippis)
a, b:  INDEX(current_loan, loan_status, appr, ippis)

拆分日期通常效率低下：

AND  MONTH(b.date_created) = MONTH(CURRENT_DATE)
AND  YEAR(b.date_created) = YEAR(CURRENT_DATE)

-->

AND b.date_created >= LEFT(CURDATE(), 7)

这应该允许这些索引有用：

b, c:  INDEX(current_loan, date_created, ippis)

从c开始可能会加快速度