1

我发现自己越来越频繁地进行这样的统计——对于 tbl1 中的每条记录，使用它作为条件来扫描 tbl2 并在 tbl2 上进行聚合，返回聚合结果。使用子查询很容易。

但是我需要在同一个子查询中一次性进行多次聚合，而不是多次扫描。

例如，这里有一些用于分析的样本数据：

SELECT  to_timestamp((random()*999999)) evt_ts, 
    evt_id  
    INTO temp.test_so_44410354
FROM
    (
        SELECT *, (random()*2000)::int AS idx
            FROM generate_series( 1, 1000) AS evt_id
            ORDER BY idx 
    ) AS t

evt_ts              |evt_id |
--------------------|-------|
1970-01-01 08:13:15 |109    |
1970-01-01 08:17:22 |762    |
1970-01-01 08:19:28 |630    |
1970-01-01 08:29:34 |429    |
1970-01-01 08:48:28 |70     |

要查看事件在给定时间范围内的分布情况、当前方法和查询计划以及所需的输出：

With wcal AS (
    SELECT ts, ts::date as dt 
    FROM generate_series( '1970-01-01 0:0:0', '1970-01-12 0:0:0', INTERVAL '1 day' ) AS ts
)
SELECT ts, dt
,(
    SELECT count(evt_ts) 
    FROM temp.test_so_44410354 as el
    WHERE ( el.evt_ts < wcal.ts AND el.evt_ts > wcal.ts - INTERVAL '1 day'  )
) as ecnt
,(
    SELECT sum(evt_id) -- should be array_agg, sum is locate to the difference in the results.
    FROM temp.test_so_44410354 as el
    WHERE ( el.evt_ts < wcal.ts AND el.evt_ts > wcal.ts - INTERVAL '1 day'  )
) as eids
FROM wcal

QUERY PLAN                                                                                          |
----------------------------------------------------------------------------------------------------|
CTE Scan on wcal  (cost=12.50..21314.08 rows=1000 width=28)                                         |
  CTE wcal                                                                                          |
    ->  Function Scan on generate_series ts  (cost=0.00..12.50 rows=1000 width=12)                  |
  SubPlan 2                                                                                         |
    ->  Aggregate  (cost=10.63..10.64 rows=1 width=8)                                               |
          ->  Bitmap Heap Scan on test_so_44410354 el  (cost=4.33..10.62 rows=5 width=8)            |
                Recheck Cond: ((evt_ts < wcal.ts) AND (evt_ts > (wcal.ts - '1 day'::interval)))     |
                ->  Bitmap Index Scan on test_so_44410354_idx  (cost=0.00..4.33 rows=5 width=0)     |
                      Index Cond: ((evt_ts < wcal.ts) AND (evt_ts > (wcal.ts - '1 day'::interval))) |
  SubPlan 3                                                                                         |
    ->  Aggregate  (cost=10.63..10.64 rows=1 width=8)                                               |
    ...

ts                  |dt         |ecnt |eids  |
--------------------|-----------|-----|------|
1970-01-01 00:00:00 |1970-01-01 |0    |      |
1970-01-02 00:00:00 |1970-01-02 |68   |36156 |
1970-01-03 00:00:00 |1970-01-03 |85   |42103 |
1970-01-04 00:00:00 |1970-01-04 |94   |47092 |
1970-01-05 00:00:00 |1970-01-05 |74   |36276 |
....

有 2 个子计划，似乎pg 必须扫描表evt_log两次才能进行两次相同的聚合。我想避免这种情况。因为聚合条件是完全一样的，如果它可以一次完成许多统计聚合，它可能会快得多。日志可以变得非常大，处理需要相当长的时间。

对我嵌套join的建议解决方案的反馈未按预期工作：

With wcal AS (
SELECT ts, ts::date as dt 
FROM generate_series( '1970-01-01 0:0:0', '1970-01-12 0:0:0', INTERVAL '1 day' ) AS ts
)
, ag AS (
    SELECT 
        count(evt_ts) AS ecnt
      , sum(evt_id) AS eids
    FROM temp.test_so_44410354 as el
    JOIN wcal ON ( el.evt_ts < wcal.ts AND el.evt_ts > wcal.ts - INTERVAL '1 day'  )
)
SELECT ts, dt, ag.ecnt, eids
FROM wcal
JOIN ag ON true

QUERY PLAN                                                                                                            |
----------------------------------------------------------------------------------------------------------------------|
Nested Loop  (cost=4239.57..4269.59 rows=1000 width=28)                                                               |
  CTE wcal                                                                                                            |
    ->  Function Scan on generate_series ts  (cost=0.00..12.50 rows=1000 width=12)                                    |
  CTE ag                                                                                                              |
    ->  Aggregate  (cost=4227.06..4227.07 rows=1 width=16)                                                            |
          ->  Nested Loop  (cost=0.28..3671.50 rows=111111 width=12)                                                  |
                ->  CTE Scan on wcal wcal_1  (cost=0.00..20.00 rows=1000 width=8)                                     |
                ->  Index Scan using test_so_44410354_idx on test_so_44410354 el  (cost=0.28..2.54 rows=111 width=12) |
                      Index Cond: ((evt_ts < wcal_1.ts) AND (evt_ts > (wcal_1.ts - '1 day'::interval)))               |
  ->  CTE Scan on ag  (cost=0.00..0.02 rows=1 width=16)                                                               |
  ->  CTE Scan on wcal  (cost=0.00..20.00 rows=1000 width=12)                                                         |

ts                  |dt         |ecnt |eids   |
--------------------|-----------|-----|-------|
1970-01-01 00:00:00 |1970-01-01 |920  |464436 |
1970-01-04 00:00:00 |1970-01-04 |920  |464436 |
1970-01-05 00:00:00 |1970-01-05 |920  |464436 |
1970-01-06 00:00:00 |1970-01-06 |920  |464436 |
1970-01-07 00:00:00 |1970-01-07 |920  |464436 |
....

建议解决方案二的反馈：写作不同，但查询计划基本相同，有2个子计划。

DROP TYPE temp.my_agg ;
CREATE TYPE temp.my_agg AS ( ecnt int, eids int );
EXPLAIN
With wcal AS (
    SELECT ts, ts::date as dt 
    FROM generate_series( '1970-01-01 0:0:0', '1970-01-12 0:0:0', INTERVAL '1 day' ) AS ts
)
SELECT ts, dt
,(
    SELECT (count(evt_ts),sum(evt_id))::temp.my_agg
    FROM temp.test_so_44410354 as el
    WHERE ( el.evt_ts < wcal.ts AND el.evt_ts > wcal.ts - INTERVAL '1 day'  )
).*
FROM wcal

QUERY PLAN                                                                                                                                  |
--------------------------------------------------------------------------------------------------------------------------------------------|
CTE Scan on wcal  (cost=12.50..21349.08 rows=1000 width=20) (actual time=0.182..3.518 rows=12 loops=1)                                      |
  CTE wcal                                                                                                                                  |
    ->  Function Scan on generate_series ts  (cost=0.00..12.50 rows=1000 width=12) (actual time=0.057..0.083 rows=12 loops=1)               |
  SubPlan 2                                                                                                                                 |
    ->  Aggregate  (cost=10.64..10.66 rows=1 width=32) (actual time=0.140..0.140 rows=1 loops=12)                                           |
          ->  Bitmap Heap Scan on test_so_44410354 el  (cost=4.33..10.62 rows=5 width=12) (actual time=0.049..0.083 rows=77 loops=12)       |
                Recheck Cond: ((evt_ts < wcal.ts) AND (evt_ts > (wcal.ts - '1 day'::interval)))                                             |
                Heap Blocks: exact=66                                                                                                       |
                ->  Bitmap Index Scan on test_so_44410354_idx  (cost=0.00..4.33 rows=5 width=0) (actual time=0.037..0.037 rows=77 loops=12) |
                      Index Cond: ((evt_ts < wcal.ts) AND (evt_ts > (wcal.ts - '1 day'::interval)))                                         |
  SubPlan 3                                                                                                                                 |
    ...
Planning time: 0.548 ms                                                                                                                     |
Execution time: 3.748 ms                                                                                                                    |

对于建议的解决方案 II，如果聚合作为记录返回，查询计划将如下所示： 与之前的查询相比，速度大致提高了一倍。（但最终结果不需要记录类型元素。）

QUERY PLAN                                                                                                                                  |
--------------------------------------------------------------------------------------------------------------------------------------------|
CTE Scan on wcal  (cost=12.50..10690.79 rows=1000 width=44) (actual time=0.147..1.979 rows=12 loops=1)                                      |
  CTE wcal                                                                                                                                  |
    ->  Function Scan on generate_series ts  (cost=0.00..12.50 rows=1000 width=12) (actual time=0.057..0.082 rows=12 loops=1)               |
  SubPlan 2                                                                                                                                 |
    ->  Aggregate  (cost=10.64..10.66 rows=1 width=32) (actual time=0.151..0.151 rows=1 loops=12)                                           |
          ->  Bitmap Heap Scan on test_so_44410354 el  (cost=4.33..10.62 rows=5 width=12) (actual time=0.054..0.089 rows=77 loops=12)       |
                Recheck Cond: ((evt_ts < wcal.ts) AND (evt_ts > (wcal.ts - '1 day'::interval)))                                             |
                Heap Blocks: exact=66                                                                                                       |
                ->  Bitmap Index Scan on test_so_44410354_idx  (cost=0.00..4.33 rows=5 width=0) (actual time=0.040..0.040 rows=77 loops=12) |
                      Index Cond: ((evt_ts < wcal.ts) AND (evt_ts > (wcal.ts - '1 day'::interval)))                                         |
Planning time: 0.381 ms                                                                                                                     |
Execution time: 2.147 ms                                                                                                                    |

LATERAL加入解决方案按预期工作。谢谢。

EXPLAIN ANALYZE
With wcal AS (
    SELECT ts, ts::date as dt 
    FROM generate_series( '1970-01-01 0:0:0', '1970-01-12 0:0:0', INTERVAL '1 day' ) AS ts
)
SELECT ts, dt, p.*
FROM wcal
     CROSS JOIN LATERAL (
        SELECT count(evt_ts), sum(evt_id)
        FROM temp.test_so_44410354 as el
        WHERE ( el.evt_ts < wcal.ts AND el.evt_ts > wcal.ts - INTERVAL '1 day'  )
    ) p

QUERY PLAN                                                                                                                                |
------------------------------------------------------------------------------------------------------------------------------------------|
Nested Loop  (cost=23.15..10705.79 rows=1000 width=28) (actual time=0.130..1.611 rows=12 loops=1)                                         |
  CTE wcal                                                                                                                                |
    ->  Function Scan on generate_series ts  (cost=0.00..12.50 rows=1000 width=12) (actual time=0.054..0.075 rows=12 loops=1)             |
  ->  CTE Scan on wcal  (cost=0.00..20.00 rows=1000 width=12) (actual time=0.061..0.099 rows=12 loops=1)                                  |
  ->  Aggregate  (cost=10.64..10.65 rows=1 width=16) (actual time=0.123..0.123 rows=1 loops=12)                                           |
        ->  Bitmap Heap Scan on test_so_44410354 el  (cost=4.33..10.62 rows=5 width=12) (actual time=0.044..0.074 rows=77 loops=12)       |
              Recheck Cond: ((evt_ts < wcal.ts) AND (evt_ts > (wcal.ts - '1 day'::interval)))                                             |
              Heap Blocks: exact=66                                                                                                       |
              ->  Bitmap Index Scan on test_so_44410354_idx  (cost=0.00..4.33 rows=5 width=0) (actual time=0.033..0.033 rows=77 loops=12) |
                    Index Cond: ((evt_ts < wcal.ts) AND (evt_ts > (wcal.ts - '1 day'::interval)))                                         |
Planning time: 0.399 ms                                                                                                                   |
Execution time: 1.778 ms                                                                                                                  |

ts                  |dt         |count |sum   |
--------------------|-----------|------|------|
1970-01-01 00:00:00 |1970-01-01 |0     |      |
1970-01-02 00:00:00 |1970-01-02 |68    |36156 |
1970-01-03 00:00:00 |1970-01-03 |85    |42103 |
1970-01-04 00:00:00 |1970-01-04 |94    |47092 |
1970-01-05 00:00:00 |1970-01-05 |74    |36276 |
1970-01-06 00:00:00 |1970-01-06 |96    |43797 |
4

3 回答 3

2

喜欢吗？

With wcal AS (
    SELECT ts, ts::date as dt, 
        extract( week from ts) as wk, 
        extract( month from ts ) as mo,
        extract( quater from ts ) as qtr,
    FROM generate_series( t1, t2, tspan )
)
, ag as (
    SELECT count(evt_ts) ecnt,  array_aggr(evt_id) eids
    FROM ev_log as el
    JOIN wcal on (el.ts < wcal.ts AND el.ts < wcal.ts - tspan)
)
SELECT ts, dt, wk, mo, qtr, ecnt, eids
FROM wcal 
JOIN ag on true

我不确定连接条件 - 您需要将整个查询放到 OP 才能猜测

更新

根据 Abelisto 的通知 - 谢谢

于 2017-06-07T10:48:09.370 回答
1

这有点棘手，但是：

create type t_my_aggs as (ecnt bigint, eids bigint[]);

With wcal AS (
    SELECT ts, ts::date as dt, 
        extract( week from ts) as wk, 
        extract( month from ts ) as mo,
        extract( quater from ts ) as qtr,
    FROM generate_series( t1, t2, tspan )
)
SELECT ts, dt, wk, mo, qtr
,(
    SELECT row(count(evt_ts), array_agg(evt_id))::t_my_aggs 
    FROM ev_log as el
    WHERE ( el.ts < wcal.ts AND el.ts < wcal.ts - tspan )
).* from wcal;
于 2017-06-07T11:49:15.377 回答
0

使用横向连接：

With wcal AS (
    SELECT ts, ts::date as dt, 
        extract( week from ts) as wk, 
        extract( month from ts ) as mo,
        extract( quater from ts ) as qtr,
    FROM generate_series( t1, t2, tspan )
)
SELECT ts, dt, wk, mo, qtr, ecnt, eids
from
  wcal cross join lateral (
    SELECT count(evt_ts) as ecnt, array_agg(evt_id) as eids
    FROM ev_log as el
    WHERE (el.ts < wcal.ts AND el.ts < wcal.ts - tspan)) as el;

<table> cross join lateral (<query>)是 的同义词<table>, lateral (<query>)。

于 2017-06-10T07:37:24.737 回答