AWS 中的 PostgreSQL 9.5.2 RDS
select name,setting from pg_settings
where name like '%vacuum%'
order by name;
name | setting
-------------------------------------+-----------
autovacuum | on
autovacuum_analyze_scale_factor | 0.05
autovacuum_analyze_threshold | 50
autovacuum_freeze_max_age | 450000000
autovacuum_max_workers | 3
autovacuum_multixact_freeze_max_age | 400000000
autovacuum_naptime | 30
autovacuum_vacuum_cost_delay | 20
autovacuum_vacuum_cost_limit | -1
autovacuum_vacuum_scale_factor | 0.1
autovacuum_vacuum_threshold | 50
autovacuum_work_mem | -1
log_autovacuum_min_duration | 0
rds.force_autovacuum_logging_level | log
vacuum_cost_delay | 0
vacuum_cost_limit | 300
vacuum_cost_page_dirty | 20
vacuum_cost_page_hit | 1
vacuum_cost_page_miss | 10
vacuum_defer_cleanup_age | 0
vacuum_freeze_min_age | 50000000
vacuum_freeze_table_age | 250000000
vacuum_multixact_freeze_min_age | 5000000
vacuum_multixact_freeze_table_age | 150000000
我一直试图弄清楚自动吸尘如何在两个 Postgres 数据库中工作。数据库的大小、参数和结构相同。(这是同一个应用程序的两个数据仓库——不同的位置和不同的数据模式)。
我们正在为一些非常大的表使用分区。我注意到较旧的(静态)分区会定期自动清空。我知道 XID 已被冻结,但该关系确实需要定期清理以查找新的 XID。
我一直在使用此查询来查找需要清理以避免 XID 环绕的关系:
SELECT 'Relation Name',age(c.relfrozenxid) c_age, age(t.relfrozenxid) t_age,
greatest(age(c.relfrozenxid),age(t.relfrozenxid)) as age
FROM pg_class c
LEFT JOIN pg_class t ON c.reltoastrelid = t.oid
WHERE c.relkind IN ('r', 'm')
order by age desc limit 5;
?column? | c_age | t_age | age
---------------+-----------+-----------+-----------
Relation Name | 461544753 | | 461544753
Relation Name | 461544753 | | 461544753
Relation Name | 461544753 | | 461544753
Relation Name | 461544753 | | 461544753
Relation Name | 461544753 | 310800517 | 461544753
列出的所有关系都是旧的稳定分区。列 relfrozenxid 定义为:“在此之前的所有事务 ID 都已替换为此表中的永久(“冻结”)事务 ID。这用于跟踪是否需要对表进行清理以防止事务 ID 回绕或允许 pg_clog 缩小。”
出于好奇,我查看了特定表的所有分区的 relfrozenxid:
SELECT c.oid::regclass as table_name,age(c.relfrozenxid) as age , c.reltuples::int, n_live_tup, n_dead_tup,
date_trunc('day',last_autovacuum)
FROM pg_class c
JOIN pg_stat_user_tables u on c.relname = u.relname
WHERE c.relkind IN ('r', 'm')
and c.relname like 'tablename%'
table_name | age | reltuples | n_live_tup | n_dead_tup | date_trunc
-------------------------------------+-----------+-----------+------------+------------+------------------------
schema_partition.tablename_201202 | 460250527 | 0 | 0 | 0 | 2018-09-23 00:00:00+00
schema_partition.tablename_201306 | 460250527 | 0 | 0 | 0 | 2018-09-23 00:00:00+00
schema_partition.tablename_201204 | 460250527 | 0 | 0 | 0 | 2018-09-23 00:00:00+00
schema_partition.tablename_201110 | 460250527 | 0 | 0 | 0 | 2018-09-23 00:00:00+00
schema_partition.tablename_201111 | 460250527 | 0 | 0 | 0 | 2018-09-23 00:00:00+00
schema_partition.tablename_201112 | 460250527 | 0 | 0 | 0 | 2018-09-23 00:00:00+00
schema_partition.tablename_201201 | 460250527 | 0 | 0 | 0 | 2018-09-23 00:00:00+00
schema_partition.tablename_201203 | 460250527 | 0 | 0 | 0 | 2018-09-23 00:00:00+00
schema_partition.tablename_201109 | 460250527 | 0 | 0 | 0 | 2018-09-23 00:00:00+00
schema_partition.tablename_201801 | 435086084 | 37970232 | 37970230 | 0 | 2018-09-25 00:00:00+00
schema_partition.tablename_201307 | 433975635 | 0 | 0 | 0 | 2018-09-25 00:00:00+00
schema_partition.tablename_201107 | 433975635 | 0 | 0 | 0 | 2018-09-25 00:00:00+00
schema_partition.tablename_201312 | 433975635 | 0 | 0 | 0 | 2018-09-25 00:00:00+00
schema_partition.tablename_201311 | 433975635 | 0 | 0 | 0 | 2018-09-25 00:00:00+00
schema_partition.tablename_201401 | 433975635 | 0 | 0 | 0 | 2018-09-25 00:00:00+00
schema_partition.tablename_201310 | 423675180 | 0 | 0 | 0 | 2018-09-25 00:00:00+00
schema_partition.tablename_201704 | 423222113 | 43842668 | 43842669 | 0 | 2018-09-25 00:00:00+00
schema_partition.tablename_201612 | 423222113 | 65700844 | 65700845 | 0 | 2018-09-25 00:00:00+00
schema_partition.tablename_201705 | 423221655 | 46847336 | 46847338 | 0 | 2018-09-25 00:00:00+00
schema_partition.tablename_201702 | 423171142 | 50701032 | 50701031 | 0 | 2018-09-25 00:00:00+00
schema_partition.tablename_overflow | 423171142 | 754 | 769 | 0 | 2018-09-25 00:00:00+00
schema_partition.tablename_201106 | 421207271 | 1 | 1 | 0 | 2018-09-25 00:00:00+00
schema_partition.tablename_201309 | 421207271 | 0 | 0 | 0 | 2018-09-25 00:00:00+00
schema_partition.tablename_201108 | 421207271 | 0 | 0 | 0 | 2018-09-25 00:00:00+00
schema_partition.tablename_201308 | 421207271 | 0 | 0 | 0 | 2018-09-25 00:00:00+00
schema_partition.tablename_201806 | 374122782 | 44626756 | 44626757 | 0 | 2018-09-26 00:00:00+00
schema.tablename | 360135561 | 0 | 0 | 0 | 2018-09-27 00:00:00+00
我很确定我并不真正了解 relfrozenxid 的工作原理,但似乎分区表受到父表的影响(这会影响分区表的 relfrozenxid 值)。我找不到任何有关此的文档。我认为对于静态表,relfrozenxid 将保持静态,直到出现真空。
此外,我有一些具有静态数据的关系,这些数据显然从未被自动清理过(last_autovacuum 为空)。这可能是 VACUUM FREEZE 操作的结果吗?
我是 Postgres 的新手,我很乐意承认自己没有完全理解自动真空过程。
我没有看到我可以识别的性能问题。
编辑:
我设置了一个查询,每 4 小时针对一个分区表运行一次:
SELECT c.oid::regclass as table_name,age(c.relfrozenxid) as age , c.reltuples::int, n_live_tup, n_dead_tup,
date_trunc('day',last_autovacuum)
FROM pg_class c
JOIN pg_stat_user_tables u on c.relname = u.relname
WHERE c.relkind IN ('r', 'm')
and c.relname like 'sometable%'
order by age desc;
在这里查看两个不同的分区是过去 20 小时的输出:
schemaname.sometable_201812 | 206286536 | 0 | 0 | 0 |
schemaname.sometable_201812 | 206286537 | 0 | 0 | 0 |
schemaname.sometable_201812 | 225465100 | 0 | 0 | 0 |
schemaname.sometable_201812 | 225465162 | 0 | 0 | 0 |
schemaname.sometable_201812 | 225465342 | 0 | 0 | 0 |
schemaname.sometable_201812 | 236408374 | 0 | 0 | 0 |
-bash-4.2$ grep 201610 test1.out
schemaname.sometable_201610 | 449974426 | 31348368 | 31348369 | 0 | 2018-09-22 00:00:00+00
schemaname.sometable_201610 | 449974427 | 31348368 | 31348369 | 0 | 2018-09-22 00:00:00+00
schemaname.sometable_201610 | 469152990 | 31348368 | 31348369 | 0 | 2018-09-22 00:00:00+00
schemaname.sometable_201610 | 50000051 | 31348368 | 31348369 | 0 | 2018-10-10 00:00:00+00
schemaname.sometable_201610 | 50000231 | 31348368 | 31348369 | 0 | 2018-10-10 00:00:00+00
schemaname.sometable_201610 | 60943263 | 31348368 | 31348369 | 0 | 2018-10-10 00:00:00+00
即使分区没有直接的 DML,分区的 relfrozenxid 也会被修改。我会假设对基表的插入以某种方式修改了分区的 relfrozenxid。
分区 sometable_201610 有 3100 万行,但是是静态的。当我查看日志文件时,这种分区的 autvacuum 需要 20-30 分钟。我不知道这是否是性能问题,但它看起来确实很昂贵。查看日志文件中的 autovacuum 表明,通常每晚都会自动清理多个此类大型分区。(也有很多零元组的分区是自动清空的,但这些分区只需要很少的时间)。