我用带有 20M 记录的 csv 构建了一个真实的测试用例。为了我的概念证明，我将比较 Oracle 中的两个选项。Oracle 内存选项与 Oracle 并行查询。我们的想法是查看结果是否符合您的期望。
- In Memory Option 是许可费，因此您必须支付额外费用
- 并行查询是 Oracle 数据库企业版中包含的一项功能。
实验室：Linux Red Hat 7 服务器：16 个 CPU 和 32 GB RAM 内存部分 = VMware 上使用 IBM ESX 硬件系列的 4GB VM 虚拟服务器
测试用例的元素
SQL> create tablespace tbtest datafile '/bbdd_odcgrc1r/datos/test.dbf' size 2g autoextend on next 100m maxsize 10g ;
Tablespace created.
SQL> create user test_perf identified by "Oracle_1" ;
User created.
SQL> grant connect to test_perf ;
Grant succeeded.
SQL> grant create table to test_perf ;
Grant succeeded.
SQL> alter user test_perf quota unlimited on tbtest ;
User altered.
SQL>
没有内存选项的 Oracle 并行查询
我使用直接路径 sql loader 加载了 csv 文件：
Table TEST_PERFORMANCE:
20000000 Rows successfully loaded.
0 Rows not loaded due to data errors.
0 Rows not loaded because all WHEN clauses were failed.
0 Rows not loaded because all fields were null.
Bind array size not used in direct path.
Column array rows : 5000
Stream buffer bytes: 256000
Read buffer bytes: 1048576
Total logical records skipped: 1
Total logical records read: 20000000
Total logical records rejected: 0
Total logical records discarded: 0
Total stream buffers loaded by SQL*Loader main thread: 4112
Total stream buffers loaded by SQL*Loader load thread: 0
Run began on Sat Jul 25 00:57:23 2020
Run ended on Sat Jul 25 00:57:34 2020
显然，将文件加载到数据库中与在 panda 中加载文件（根本不一样）。因为 Panda 不需要将数据加载到任何东西（在本例中为数据库数据文件）
SQL> desc test_perf.test_performance
Name Null? Type
----------------------------------------- -------- ----------------------------
ID NUMBER
DATE_ID DATE
INSTANCE_ID NUMBER
TERRITORY_ID VARCHAR2(10 CHAR)
CODE VARCHAR2(10 CHAR)
PRICE NUMBER
CURRENCY_CODE_ID VARCHAR2(10 CHAR)
PRICE_IN_USD NUMBER
我收集表的统计信息，现在让我们看看它的行为
SQL> exec dbms_stats.gather_table_stats ( 'TEST_PERF' , 'TEST_PERFORMANCE' , block_sample => true );
PL/SQL procedure successfully completed.
按currency_code_id分组（不排序）
SQL> select count(*) , currency_code_id from test_perf.test_performance group by currency_code_id
90 rows selected.
Elapsed: 00:00:00.35
按 currency_code_id 分组，但按计数器排序
SQL> select count(*) , currency_code_id from test_perf.test_performance group by currency_code_id order by count(*) desc ;
90 rows selected.
Elapsed: 00:00:00.70
对所有记录进行排序并通过 sqlplus 显示它们仅用于创建输出就消耗大量时间，但查询本身非常快
SQL> select * from test_perf.test_performance order by PRICE_IN_USD desc ;
20000000 rows selected.
Elapsed: 00:01:31.48
Execution Plan
----------------------------------------------------------
Plan hash value: 1897103579
---------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | TQ |IN-OUT| PQ Distrib |
---------------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 20M| 972M| | 5940 (1)| 00:00:01 | | | |
| 1 | PX COORDINATOR | | | | | | | | | |
| 2 | PX SEND QC (ORDER) | :TQ10001 | 20M| 972M| | 5940 (1)| 00:00:01 | Q1,01 | P->S | QC (ORDER) |
| 3 | SORT ORDER BY | | 20M| 972M| 1376M| 5940 (1)| 00:00:01 | Q1,01 | PCWP | |
| 4 | PX RECEIVE | | 20M| 972M| | 435 (1)| 00:00:01 | Q1,01 | PCWP | |
| 5 | PX SEND RANGE | :TQ10000 | 20M| 972M| | 435 (1)| 00:00:01 | Q1,00 | P->P | RANGE |
| 6 | PX BLOCK ITERATOR | | 20M| 972M| | 435 (1)| 00:00:01 | Q1,00 | PCWC | |
| 7 | TABLE ACCESS FULL| TEST_PERFORMANCE | 20M| 972M| | 435 (1)| 00:00:01 | Q1,00 | PCWP | |
---------------------------------------------------------------------------------------------------------------------------------
Note
-----
- Degree of Parallelism is 32 because of table property
Statistics
----------------------------------------------------------
364 recursive calls
3 db block gets
36963 consistent gets
45558 physical reads
2860 redo size
703698256 bytes sent via SQL*Net to client
14667271 bytes received via SQL*Net from client
1333335 SQL*Net roundtrips to/from client
64 sorts (memory)
1 sorts (disk)
20000000 rows processed
查询本身消耗的实时时间是
SQL> select * from test_perf.test_performance order by PRICE_IN_USD desc ;
Elapsed: 00:00:02.03
Execution Plan
----------------------------------------------------------
Plan hash value: 1897103579
---------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | TQ |IN-OUT| PQ Distrib |
---------------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 20M| 972M| | 5940 (1)| 00:00:01 | | | |
| 1 | PX COORDINATOR | | | | | | | | | |
| 2 | PX SEND QC (ORDER) | :TQ10001 | 20M| 972M| | 5940 (1)| 00:00:01 | Q1,01 | P->S | QC (ORDER) |
| 3 | SORT ORDER BY | | 20M| 972M| 1376M| 5940 (1)| 00:00:01 | Q1,01 | PCWP | |
| 4 | PX RECEIVE | | 20M| 972M| | 435 (1)| 00:00:01 | Q1,01 | PCWP | |
| 5 | PX SEND RANGE | :TQ10000 | 20M| 972M| | 435 (1)| 00:00:01 | Q1,00 | P->P | RANGE |
| 6 | PX BLOCK ITERATOR | | 20M| 972M| | 435 (1)| 00:00:01 | Q1,00 | PCWC | |
| 7 | TABLE ACCESS FULL| TEST_PERFORMANCE | 20M| 972M| | 435 (1)| 00:00:01 | Q1,00 | PCWP | |
---------------------------------------------------------------------------------------------------------------------------------
Note
-----
- Degree of Parallelism is 32 because of table property
选择表的不同区域id
SQL> select distinct territory_id from test_perf.test_performance order by territory_id desc ;
136 rows selected.
Elapsed: 00:00:00.58
对于支点，我选择了这个简单的例子
SQL> select * from test_perf.test_performance
pivot ( count(*) for TERRITORY_ID in
(
'ZW',
'ZM',
'ZA',
'VN',
'VG',
'VE',
'UZ',
'UY',
'US',
'UG',
'UA',
'TZ',
'TW',
'TT',
'TR',
'TM',
'TJ',
'TH',
'TG',
'SZ',
'SV',
'SN',
'SK',
'SI',
'SG',
'SE',
'SA',
'RW',
'RU',
'QA',
'PY',
'PT',
'PL',
'PH',
'PG',
'PE',
'PA',
'OM',
'NZ',
'NP',
'NO',
'NL',
'NI',
'NE',
'NA',
'MZ',
'MY',
'MX',
'MU',
'MT',
'MO',
'MN',
'ML',
'MK',
'MD',
'LV',
'LU',
'LT',
'LK',
'LB',
'LA',
'KZ',
'KY',
'KW',
'KR',
'KN',
'KH',
'KG',
'JP',
'JO',
'JM',
'IT',
'IS',
'IN',
'IL',
'IE',
'ID',
'HU',
'HT',
'HR',
'HN',
'HK',
'GW',
'GT',
'GR',
'GM',
'GH',
'GD',
'GB',
'GA',
'FR',
'FM',
'FJ',
'FI',
'ES',
'EG',
'EE',
'EC',
'DO',
'DM',
'DK',
'DE',
'CZ',
'CY',
'CV',
'CR',
'CO',
'CL',
'CI',
'CH',
'CA',
'BZ',
'BY',
'BW',
'BS',
'BR',
'BO',
'BN',
'BM',
'BJ',
'BH',
'BG',
'BF',
'BE',
'BA',
'AZ',
'AW',
'AU',
'AT',
'AR',
'AO',
'AM',
'AL',
'AI',
'AG',
'AE'
) )
order by id
Elapsed: 00:00:04.74
Oracle 内存选项
我配置了一个 4GB 的 In memory 区域，并没有那么多。
Total System Global Area 1.2885E+10 bytes
Fixed Size 12192520 bytes
Variable Size 5184161016 bytes
Database Buffers 3388997632 bytes
Redo Buffers 4583424 bytes
In-Memory Area 4294967296 bytes
Database mounted.
Database opened.
SQL> SHOW PARAMETER INMEMORY
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
inmemory_adg_enabled boolean TRUE
inmemory_clause_default string
inmemory_expressions_usage string ENABLE
inmemory_force string DEFAULT
inmemory_max_populate_servers integer 8
inmemory_query string ENABLE
inmemory_size big integer 4G
inmemory_trickle_repopulate_servers_ integer 1
percent
inmemory_virtual_columns string MANUAL
optimizer_inmemory_aware boolean TRUE
SQL> ALTER TABLE TEST_PERF.TEST_PERFORMANCE INMEMORY PRIORITY HIGH;
Table altered.
SQL> select segment_name
, partition_name
2 3 , inmemory_size / 1024 / 1024 as inmemory_size_mb
, bytes / 1024 / 1024 as bytes_mb
, populate_status
4 5 6 , trunc(bytes / inmemory_size, 1) * 100 as compression_ratio
from v$im_segments
7 8 order by segment_name, partition_name;
SEGMENT_NAME
--------------------------------------------------------------------------------
PARTITION_NAME
--------------------------------------------------------------------------------
INMEMORY_SIZE_MB BYTES_MB POPULATE_STAT COMPRESSION_RATIO
---------------- ---------- ------------- -----------------
TEST_PERFORMANCE
362.25 514.046875 COMPLETED 140
SQL> select count(*),length(date_id) from test_perf.test_performance group by length(date_id)
Execution Plan
----------------------------------------------------------
Plan hash value: 3227171220
-----------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | TQ |IN-OUT| PQ Distrib |
-----------------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 11 | 121 | 29 (59)| 00:00:01 | | | |
| 1 | PX COORDINATOR | | | | | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10001 | 11 | 121 | 29 (59)| 00:00:01 | Q1,01 | P->S | QC (RAND) |
| 3 | HASH GROUP BY | | 11 | 121 | 29 (59)| 00:00:01 | Q1,01 | PCWP | |
| 4 | PX RECEIVE | | 11 | 121 | 29 (59)| 00:00:01 | Q1,01 | PCWP | |
| 5 | PX SEND HASH | :TQ10000 | 11 | 121 | 29 (59)| 00:00:01 | Q1,00 | P->P | HASH |
| 6 | HASH GROUP BY | | 11 | 121 | 29 (59)| 00:00:01 | Q1,00 | PCWP | |
| 7 | PX BLOCK ITERATOR | | 20M| 209M| 14 (15)| 00:00:01 | Q1,00 | PCWC | |
| 8 | TABLE ACCESS INMEMORY FULL| TEST_PERFORMANCE | 20M| 209M| 14 (15)| 00:00:01 | Q1,00 | PCWP | |
-----------------------------------------------------------------------------------------------------------------------------------
Note
-----
- Degree of Parallelism is 32 because of table property
让我们测试一些查询。
按地区分组
SQL> select count(*),TERRITORY_ID from test_perf.test_performance group by TERRITORY_ID ;
136 rows selected.
Elapsed: 00:00:00.24
按 instance_id 分组
SQL> select count(*) , INSTANCE_ID from test_perf.test_performance group by INSTANCE_ID
11251 rows selected.
Elapsed: 00:00:00.27
按 2 个字段分组
SQL> select count(*), instance_id, territory_id from test_perf.test_performance group by instance_id, territory_id ;
278269 rows selected.
Elapsed: 00:00:00.84
如您所见，显然当我在内存中有一张表时，这些操作的性能会提高，但如果您开始对查询应用分析，您会注意到更多改进。
几个统计功能和分组依据
SQL> select territory_id,sum(to_number(price)),avg(to_number(price)),max(to_number(price)),min(to_number(price))
2 from test_perf.test_performance group by territory_id ;
Elapsed: 00:00:00.57
但是，PIVOT 对于内存来说通常不是一个好主意，因为表使用了列存储。
The same query with pivot executed before takes
Elapsed: 00:00:15.93
正如您在上面的示例中所看到的，我没有更改表的 PARALLEL 属性，所以让我向您展示当没有为具有 INMEMORY 选项的对象启用并行时查询的行为
SQL> alter table test_perf.TEST_PERFORMANCE noparallel ;
Table altered.
SQL> select count(*) , currency_code_id from test_perf.test_performance group by currency_code_id
2 ;
90 rows selected.
Elapsed: 00:00:02.14
Execution Plan
----------------------------------------------------------
Plan hash value: 151279035
------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 90 | 450 | 909 (62)| 00:00:01 |
| 1 | HASH GROUP BY | | 90 | 450 | 909 (62)| 00:00:01 |
| 2 | TABLE ACCESS INMEMORY FULL| TEST_PERFORMANCE | 20M| 95M| 411 (16)| 00:00:01 |
------------------------------------------------------------------------------------------------
IMPQ
对于最后一部分，我保留了所有选项中最好的一个，即并行和内存的组合，称为 IMPQ 或内存并行查询。此功能提供了两全其美的效果，为了启动，您必须为您的表启用并行，将表放在内存区域并将参数 PARALLEL_DEGREE_POLICY 定义为 AUTO。
这方面的一个例子是
SQL> alter session set parallel_degree_policy=auto ;
Session altered.
SQL> set autotrace traceonly explain
SQL> select count(*),territory_id from test_perf.test_performance group by territory_id
Execution Plan
----------------------------------------------------------
Plan hash value: 3227171220
-----------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | TQ |IN-OUT| PQ Distrib |
-----------------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 136 | 408 | 78 (59)| 00:00:01 | | | |
| 1 | PX COORDINATOR | | | | | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10001 | 136 | 408 | 78 (59)| 00:00:01 | Q1,01 | P->S | QC (RAND) |
| 3 | HASH GROUP BY | | 136 | 408 | 78 (59)| 00:00:01 | Q1,01 | PCWP | |
| 4 | PX RECEIVE | | 136 | 408 | 78 (59)| 00:00:01 | Q1,01 | PCWP | |
| 5 | PX SEND HASH | :TQ10000 | 136 | 408 | 78 (59)| 00:00:01 | Q1,00 | P->P | HASH |
| 6 | HASH GROUP BY | | 136 | 408 | 78 (59)| 00:00:01 | Q1,00 | PCWP | |
| 7 | PX BLOCK ITERATOR | | 20M| 57M| 38 (16)| 00:00:01 | Q1,00 | PCWC | |
| 8 | TABLE ACCESS INMEMORY FULL| TEST_PERFORMANCE | 20M| 57M| 38 (16)| 00:00:01 | Q1,00 | PCWP | |
-----------------------------------------------------------------------------------------------------------------------------------
Note
-----
- automatic DOP: Computed Degree of Parallelism is 12
在上面的语句中检查 DOP 从 32 到 12 的变化。这表明 IMPQ 已经确定该语句的最佳度数是 12，而不是由 CPU_COUNT * 2 派生的 32（因为该表是使用默认度数创建的自动的 ）。
下面的这个查询现在使用 IMPQ 只需 0.14 秒，而不是 0.35 秒。
SQL> select count(*),territory_id from test_perf.test_performance group by territory_id
2 ;
136 rows selected.
Elapsed: 00:00:00.14
这个其他查询现在需要 0.14 秒，而之前需要 0.70 秒
SQL> select count(*) , currency_code_id from test_perf.test_performance group by currency_code_id order by count(*) desc ;
90 rows selected.
Elapsed: 00:00:00.14
概括
如果您需要的时间少于一秒，并且您主要使用分析，那么 Oracle 内存选项可能是一个好主意。如果是这种情况，您可能还希望启用 IMPQ 以获得最佳结果。关于应该或不包含哪些列、如何将表填充到内存区域等方面需要做一些工作。分析的复杂性越大，您会注意到的性能就越好。
不过，如果您可以忍受 1 到 5 秒之间的时间，那么没有索引的并行查询可能是一种免费且易于配置的解决方案。
随意评论任何内容或要求对测试用例进行任何澄清。
