我可以确认该问题仍然是 Oracle 12.1.0.2.0 上的问题。
甚至硬编码的分区消除界限也是不够的。
这是我的测试表:
CREATE TABLE FR_MESSAGE_PART (
ID NUMBER(38) NOT NULL CONSTRAINT PK_FR_MESSAGE_PART PRIMARY KEY USING INDEX LOCAL,
TRX_ID NUMBER(38) NOT NULL, TS TIMESTAMP NOT NULL, TEXT CLOB)
PARTITION BY RANGE (ID) (PARTITION PART_0 VALUES LESS THAN (0));
CREATE INDEX IX_FR_MESSAGE_PART_TRX_ID ON FR_MESSAGE_PART(TRX_ID) LOCAL;
CREATE INDEX IX_FR_MESSAGE_PART_TS ON FR_MESSAGE_PART(TS) LOCAL;
该表包含数百万条 OLTP 生产数据记录,这些记录持续了几个月。每个月都属于一个单独的分区。
此表的主键值始终包含较高位的时间部分,允许ID
用于按日历期间进行范围分区。所有消息都继承更高的时间位TRX_ID
。这确保了属于同一个业务操作的所有消息总是落在同一个分区中。
让我们从硬编码查询开始,用于选择给定时间段内应用分区消除边界的最新消息页面:
select * from (select * from FR_MESSAGE_PART
where TS >= DATE '2017-11-30' and TS < DATE '2017-12-02'
and ID >= 376894993815568384 and ID < 411234940974268416
order by TS DESC) where ROWNUM <= 40;
但是,在新收集的表统计信息之后,Oracle 优化器仍然错误地估计对两个整个月分区进行排序会比通过现有本地索引进行两天范围扫描要快:
-----------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | Pstart| Pstop |
-----------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 40 | 26200 | | 103K (1)| 00:00:05 | | |
|* 1 | COUNT STOPKEY | | | | | | | | |
| 2 | VIEW | | 803K| 501M| | 103K (1)| 00:00:05 | | |
|* 3 | SORT ORDER BY STOPKEY | | 803K| 70M| 92M| 103K (1)| 00:00:05 | | |
| 4 | PARTITION RANGE ITERATOR| | 803K| 70M| | 86382 (1)| 00:00:04 | 2 | 3 |
|* 5 | TABLE ACCESS FULL | FR_MESSAGE_PART | 803K| 70M| | 86382 (1)| 00:00:04 | 2 | 3 |
-----------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
1 - filter(ROWNUM<=40)
3 - filter(ROWNUM<=40)
5 - filter("TS"<TIMESTAMP' 2017-12-01 00:00:00' AND "TS">=TIMESTAMP' 2017-11-29 00:00:00' AND
"ID">=376894993815568384)
实际执行时间比计划中的估计要长一个数量级。
所以我们必须应用一个提示来强制使用索引:
select * from (select /*+ FIRST_ROWS(40) INDEX(FR_MESSAGE_PART (TS)) */ * from FR_MESSAGE_PART
where TS >= DATE '2017-11-30' and TS < DATE '2017-12-02'
and ID >= 376894993815568384 and ID < 411234940974268416
order by TS DESC) where ROWNUM <= 40;
现在该计划使用索引,但仍然涉及对两个整个分区的慢速排序:
-----------------------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | Pstart| Pstop |
-----------------------------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 40 | 26200 | | 615K (1)| 00:00:25 | | |
|* 1 | COUNT STOPKEY | | | | | | | | |
| 2 | VIEW | | 803K| 501M| | 615K (1)| 00:00:25 | | |
|* 3 | SORT ORDER BY STOPKEY | | 803K| 70M| 92M| 615K (1)| 00:00:25 | | |
| 4 | PARTITION RANGE ITERATOR | | 803K| 70M| | 598K (1)| 00:00:24 | 2 | 3 |
|* 5 | TABLE ACCESS BY LOCAL INDEX ROWID BATCHED| FR_MESSAGE_PART | 803K| 70M| | 598K (1)| 00:00:24 | 2 | 3 |
|* 6 | INDEX RANGE SCAN | IX_FR_MESSAGE_PART_TS | 576K| | | 2269 (1)| 00:00:01 | 2 | 3 |
-----------------------------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
1 - filter(ROWNUM<=40)
3 - filter(ROWNUM<=40)
5 - filter("ID">=376894993815568384)
6 - access("TS">=TIMESTAMP' 2017-11-30 00:00:00' AND "TS"<TIMESTAMP' 2017-12-02 00:00:00')
经过Oracle 提示参考和 google 的一番挣扎后,我们发现我们还必须使用INDEX_DESC或INDEX_RS_DESC提示明确指定索引范围扫描的下降方向:
select * from (select /*+ FIRST_ROWS(40) INDEX_RS_DESC(FR_MESSAGE_PART (TS)) */ * from FR_MESSAGE_PART
where TS >= DATE '2017-11-30' and TS < DATE '2017-12-02'
and ID >= 376894993815568384 and ID < 411234940974268416
order by TS DESC) where ROWNUM <= 40;
这最终给出了每个分区的快速计划,COUNT STOPKEY
它按降序扫描分区,并且每个分区最多只对 40 行进行排序:
------------------------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | Pstart| Pstop |
------------------------------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 40 | 26200 | | 615K (1)| 00:00:25 | | |
|* 1 | COUNT STOPKEY | | | | | | | | |
| 2 | VIEW | | 803K| 501M| | 615K (1)| 00:00:25 | | |
|* 3 | SORT ORDER BY STOPKEY | | 803K| 70M| 92M| 615K (1)| 00:00:25 | | |
| 4 | PARTITION RANGE ITERATOR | | 803K| 70M| | 598K (1)| 00:00:24 | 3 | 2 |
|* 5 | COUNT STOPKEY | | | | | | | | |
|* 6 | TABLE ACCESS BY LOCAL INDEX ROWID BATCHED| FR_MESSAGE_PART | 803K| 70M| | 598K (1)| 00:00:24 | 3 | 2 |
|* 7 | INDEX RANGE SCAN DESCENDING | IX_FR_MESSAGE_PART_TS | 576K| | | 2269 (1)| 00:00:01 | 3 | 2 |
------------------------------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
1 - filter(ROWNUM<=40)
3 - filter(ROWNUM<=40)
5 - filter(ROWNUM<=40)
6 - filter("ID">=376894993815568384)
7 - access("TS">=TIMESTAMP' 2017-11-30 00:00:00' AND "TS"<TIMESTAMP' 2017-12-02 00:00:00')
filter("TS">=TIMESTAMP' 2017-11-30 00:00:00' AND "TS"<TIMESTAMP' 2017-12-02 00:00:00')
这运行得非常快,但估计的计划成本仍然过高。
到目前为止,一切都很好。现在让我们尝试将查询参数化以在我们的自定义 ORM 框架中使用:
select * from (select /*+ FIRST_ROWS(40) INDEX_RS_DESC(FR_MESSAGE_PART (TS)) */ * from FR_MESSAGE_PART
where TS >= :1 and TS < :2
and ID >= :3 and ID < :4
order by TS DESC) where ROWNUM <= 40;
但随后COUNT STOPKEY
每个分区从问题中所述的计划中消失,并在另一个答案中确认:
----------------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop |
----------------------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 40 | 26200 | 82349 (1)| 00:00:04 | | |
|* 1 | COUNT STOPKEY | | | | | | | |
| 2 | VIEW | | 153 | 97K| 82349 (1)| 00:00:04 | | |
|* 3 | SORT ORDER BY STOPKEY | | 153 | 14076 | 82349 (1)| 00:00:04 | | |
|* 4 | FILTER | | | | | | | |
| 5 | PARTITION RANGE ITERATOR | | 153 | 14076 | 82348 (1)| 00:00:04 | KEY | KEY |
|* 6 | TABLE ACCESS BY LOCAL INDEX ROWID BATCHED| FR_MESSAGE_PART | 153 | 14076 | 82348 (1)| 00:00:04 | KEY | KEY |
|* 7 | INDEX RANGE SCAN DESCENDING | IX_FR_MESSAGE_PART_TS | 110K| | 450 (1)| 00:00:01 | KEY | KEY |
----------------------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
1 - filter(ROWNUM<=40)
3 - filter(ROWNUM<=40)
4 - filter(TO_NUMBER(:4)>TO_NUMBER(:3) AND TO_TIMESTAMP(:2)>TO_TIMESTAMP(:1))
6 - filter("ID">=TO_NUMBER(:3) AND "ID"<TO_NUMBER(:4))
7 - access("TS">=TO_TIMESTAMP(:1) AND "TS"<TO_TIMESTAMP(:2))
filter("TS">=TO_TIMESTAMP(:1) AND "TS"<TO_TIMESTAMP(:2))
然后我尝试退回到硬编码的每月对齐的分区消除边界,但仍保留参数化的时间戳边界以最大程度地减少计划缓存破坏。
select * from (select /*+ FIRST_ROWS(40) INDEX_RS_DESC(FR_MESSAGE_PART (TS)) */ * from FR_MESSAGE_PART
where TS >= :1 and TS < :2
and ID >= 376894993815568384 and ID < 411234940974268416
order by TS DESC) where ROWNUM <= 40;
但仍然有缓慢的计划:
------------------------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | Pstart| Pstop |
------------------------------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 40 | 26200 | | 83512 (1)| 00:00:04 | | |
|* 1 | COUNT STOPKEY | | | | | | | | |
| 2 | VIEW | | 61238 | 38M| | 83512 (1)| 00:00:04 | | |
|* 3 | SORT ORDER BY STOPKEY | | 61238 | 5501K| 7216K| 83512 (1)| 00:00:04 | | |
|* 4 | FILTER | | | | | | | | |
| 5 | PARTITION RANGE ITERATOR | | 61238 | 5501K| | 82214 (1)| 00:00:04 | 3 | 2 |
|* 6 | TABLE ACCESS BY LOCAL INDEX ROWID BATCHED| FR_MESSAGE_PART | 61238 | 5501K| | 82214 (1)| 00:00:04 | 3 | 2 |
|* 7 | INDEX RANGE SCAN DESCENDING | IX_FR_MESSAGE_PART_TS | 79076 | | | 316 (1)| 00:00:01 | 3 | 2 |
------------------------------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
1 - filter(ROWNUM<=40)
3 - filter(ROWNUM<=40)
4 - filter(TO_TIMESTAMP(:2)>TO_TIMESTAMP(:1))
6 - filter("ID">=376894993815568384)
7 - access("TS">=TO_TIMESTAMP(:1) AND "TS"<TO_TIMESTAMP(:2))
filter("TS">=TO_TIMESTAMP(:1) AND "TS"<TO_TIMESTAMP(:2))
@ChrisSaxon 在他的回答中提到,缺少嵌套STOPKEY COUNT
与filter(TO_TIMESTAMP(:2)>TO_TIMESTAMP(:1))
验证上限确实大于下限的操作有关。
考虑到这一点,我试图通过转换TS between :a and :b
为等价物来欺骗 oprimizer :b between TS and TS + (:b - :a)
。这行得通!
在对该更改的根本原因进行了一些额外调查后,我发现仅替换TS >= :1 and TS < :2
为TS + 0 >= :1 and TS < :2
有助于实现最佳执行计划。
select * from (select /*+ FIRST_ROWS(40) INDEX_RS_DESC(FR_MESSAGE_PART (TS)) */ * from FR_MESSAGE_PART
where TS + 0 >= :1 and TS < :2
and ID >= 376894993815568384 and ID < 411234940974268416
order by TS DESC) where ROWNUM <= 40;
该计划现在具有适当的COUNT STOPKEY
每个分区,并且INTERNAL_FUNCTION("TS")+0
我猜想防止有毒的额外边界检查过滤器的概念。
------------------------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | Pstart| Pstop |
------------------------------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 40 | 26200 | | 10120 (1)| 00:00:01 | | |
|* 1 | COUNT STOPKEY | | | | | | | | |
| 2 | VIEW | | 61238 | 38M| | 10120 (1)| 00:00:01 | | |
|* 3 | SORT ORDER BY STOPKEY | | 61238 | 5501K| 7216K| 10120 (1)| 00:00:01 | | |
| 4 | PARTITION RANGE ITERATOR | | 61238 | 5501K| | 8822 (1)| 00:00:01 | 3 | 2 |
|* 5 | COUNT STOPKEY | | | | | | | | |
|* 6 | TABLE ACCESS BY LOCAL INDEX ROWID BATCHED| FR_MESSAGE_PART | 61238 | 5501K| | 8822 (1)| 00:00:01 | 3 | 2 |
|* 7 | INDEX RANGE SCAN DESCENDING | IX_FR_MESSAGE_PART_TS | 7908 | | | 631 (1)| 00:00:01 | 3 | 2 |
------------------------------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
1 - filter(ROWNUM<=40)
3 - filter(ROWNUM<=40)
5 - filter(ROWNUM<=40)
6 - filter("ID">=376894993815568384)
7 - access("TS"<TO_TIMESTAMP(:2))
filter(INTERNAL_FUNCTION("TS")+0>=:1 AND "TS"<TO_TIMESTAMP(:2))
+ 0
我们必须在我们的自定义 ORM 框架中实现上述特定于 Oracle 的解决方法和分区消除边界硬编码。它允许在切换到具有本地索引的分区表后保持相同的快速分页性能。
但我希望那些冒险在没有完全控制 sql 构建代码的情况下进行相同切换的人保持耐心和理智。
当分区和分页混合在一起时,Oracle 似乎有太多的陷阱。例如,我们发现 Oracle 12 的新OFFSET ROWS / FETCH NEXT ROWS ONLY
语法糖几乎不能用于本地索引分区表,因为它所基于的大多数分析窗口函数。
获取第一个页面后面的一些页面的最短工作查询是
select * from (select * from (
select /*+ FIRST_ROWS(200) INDEX_RS_DESC(FR_MESSAGE_PART (TS)) */* from FR_MESSAGE_PART
where TS + 0 >= :1 and TS < :2
and ID >= 376894993815568384 and ID < 411234940974268416
order by TS DESC) where ROWNUM <= 200) offset 180 rows;
以下是运行此类查询后的实际执行计划示例:
SQL_ID c67mmq4wg49sx, child number 0
-------------------------------------
select * from (select * from (select /*+ FIRST_ROWS(200)
INDEX_RS_DESC("FR_MESSAGE_PART" ("TS")) GATHER_PLAN_STATISTICS */ "ID",
"MESSAGE_TYPE_ID", "TS", "REMOTE_ADDRESS", "TRX_ID",
"PROTOCOL_MESSAGE_ID", "MESSAGE_DATA_ID", "TEXT_OFFSET", "TEXT_SIZE",
"BODY_OFFSET", "BODY_SIZE", "INCOMING" from "FR_MESSAGE_PART" where
"TS" + 0 >= :1 and "TS" < :2 and "ID" >= 376894993815568384 and "ID" <
411234940974268416 order by "TS" DESC) where ROWNUM <= 200) offset 180
rows
Plan hash value: 2499404919
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Starts | E-Rows |E-Bytes|E-Temp | Cost (%CPU)| E-Time | Pstart| Pstop | A-Rows | A-Time | Buffers | OMem | 1Mem | Used-Mem |
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | | | 640K(100)| | | | 20 |00:00:00.01 | 322 | | | |
|* 1 | VIEW | | 1 | 200 | 130K| | 640K (1)| 00:00:26 | | | 20 |00:00:00.01 | 322 | | | |
| 2 | WINDOW NOSORT | | 1 | 200 | 127K| | 640K (1)| 00:00:26 | | | 200 |00:00:00.01 | 322 | 142K| 142K| |
| 3 | VIEW | | 1 | 200 | 127K| | 640K (1)| 00:00:26 | | | 200 |00:00:00.01 | 322 | | | |
|* 4 | COUNT STOPKEY | | 1 | | | | | | | | 200 |00:00:00.01 | 322 | | | |
| 5 | VIEW | | 1 | 780K| 487M| | 640K (1)| 00:00:26 | | | 200 |00:00:00.01 | 322 | | | |
|* 6 | SORT ORDER BY STOPKEY | | 1 | 780K| 68M| 89M| 640K (1)| 00:00:26 | | | 200 |00:00:00.01 | 322 | 29696 | 29696 |26624 (0)|
| 7 | PARTITION RANGE ITERATOR | | 1 | 780K| 68M| | 624K (1)| 00:00:25 | 3 | 2 | 400 |00:00:00.01 | 322 | | | |
|* 8 | COUNT STOPKEY | | 2 | | | | | | | | 400 |00:00:00.01 | 322 | | | |
|* 9 | TABLE ACCESS BY LOCAL INDEX ROWID| FR_MESSAGE_PART | 2 | 780K| 68M| | 624K (1)| 00:00:25 | 3 | 2 | 400 |00:00:00.01 | 322 | | | |
|* 10 | INDEX RANGE SCAN DESCENDING | IX_FR_MESSAGE_PART_TS | 2 | 559K| | | 44368 (1)| 00:00:02 | 3 | 2 | 400 |00:00:00.01 | 8 | | | |
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Outline Data
-------------
/*+
BEGIN_OUTLINE_DATA
IGNORE_OPTIM_EMBEDDED_HINTS
OPTIMIZER_FEATURES_ENABLE('12.1.0.2')
DB_VERSION('12.1.0.2')
OPT_PARAM('optimizer_dynamic_sampling' 0)
OPT_PARAM('_optimizer_dsdir_usage_control' 0)
FIRST_ROWS(200)
OUTLINE_LEAF(@"SEL$3")
OUTLINE_LEAF(@"SEL$2")
OUTLINE_LEAF(@"SEL$1")
OUTLINE_LEAF(@"SEL$4")
NO_ACCESS(@"SEL$4" "from$_subquery$_004"@"SEL$4")
NO_ACCESS(@"SEL$1" "from$_subquery$_001"@"SEL$1")
NO_ACCESS(@"SEL$2" "from$_subquery$_002"@"SEL$2")
INDEX_RS_DESC(@"SEL$3" "FR_MESSAGE_PART"@"SEL$3" ("FR_MESSAGE_PART"."TS"))
END_OUTLINE_DATA
*/
Predicate Information (identified by operation id):
---------------------------------------------------
1 - filter("from$_subquery$_004"."rowlimit_$$_rownumber">180)
4 - filter(ROWNUM<=200)
6 - filter(ROWNUM<=200)
8 - filter(ROWNUM<=200)
9 - filter("ID">=376894993815568384)
10 - access("TS"<:2)
filter((INTERNAL_FUNCTION("TS")+0>=:1 AND "TS"<:2))
请注意实际获取的行和时间比优化器估计要好多少。
更新
请注意,即使这个最佳计划也可能会失败,从而降低本地索引全扫描速度,以防分区消除下限被猜测得太低,以至于最低分区不包含足够的记录来匹配查询过滤器。
rleishman 的调优“BETWEEN”查询指出:
问题是索引只能扫描具有范围谓词(<、>、LIKE、BETWEEN)的一列。所以即使一个索引同时包含lower_bound 和upper_bound 列,索引扫描也会返回所有匹配lower_bound <= :b 的行,然后过滤不匹配upper_bound >= :b 的行。
在寻找的值位于中间某处的情况下,范围扫描将返回表中的一半行以找到单行。在最常见的搜索行位于顶部(最高值)的最坏情况下,索引扫描将在每次查找时处理表中的几乎每一行。
这意味着,不幸的是,Oracle 在达到 STOPKEY COUNT 条件或扫描整个分区之前不会考虑范围扫描过滤器的下限!
因此,我们必须将下分区消除边界启发式限制在时间戳下限所在的同一个月。这以不显示列表中的一些延迟事务消息的风险为代价来防御全索引扫描。但是,如果需要,可以通过延长提供的时间段来轻松解决这个问题。
我还尝试应用相同的+ 0
技巧来强制具有动态分区消除边界绑定的最佳计划:
select * from (select /*+ FIRST_ROWS(40) INDEX_RS_DESC(FR_MESSAGE_PART (TS)) */ * from FR_MESSAGE_PART
where TS+0 >= :1 and TS < :2
and ID >= :3 and ID+0 < :4
order by TS DESC) where ROWNUM <= 40;
然后,该计划仍然保留STOPKEY COUNT
每个分区的正确性,但分区消除丢失了上限,正如Pstart
计划表的列所注意到的那样:
----------------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop |
----------------------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 40 | 26200 | 9083 (1)| 00:00:01 | | |
|* 1 | COUNT STOPKEY | | | | | | | |
| 2 | VIEW | | 153 | 97K| 9083 (1)| 00:00:01 | | |
|* 3 | SORT ORDER BY STOPKEY | | 153 | 14076 | 9083 (1)| 00:00:01 | | |
| 4 | PARTITION RANGE ITERATOR | | 153 | 14076 | 9082 (1)| 00:00:01 | 10 | KEY |
|* 5 | COUNT STOPKEY | | | | | | | |
|* 6 | TABLE ACCESS BY LOCAL INDEX ROWID BATCHED| FR_MESSAGE_PART | 153 | 14076 | 9082 (1)| 00:00:01 | 10 | KEY |
|* 7 | INDEX RANGE SCAN DESCENDING | IX_FR_MESSAGE_PART_TS | 11023 | | 891 (1)| 00:00:01 | 10 | KEY |
----------------------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
1 - filter(ROWNUM<=40)
3 - filter(ROWNUM<=40)
5 - filter(ROWNUM<=40)
6 - filter("ID">=TO_NUMBER(:3) AND "ID"+0<TO_NUMBER(:4))
7 - access("TS"<TO_TIMESTAMP(:2))
filter(INTERNAL_FUNCTION("TS")+0>=:1 AND "TS"<TO_TIMESTAMP(:2))