1

我们有一个简单的通用表结构，在 PostgreSQL 中实现（8.3；9.1 即将推出）。这似乎是一个非常简单和常见的实现。归结为：

events_event_types
(
    # this table holds some 50 rows
    id bigserial # PK
    "name" character varying(255)
)

events_events
(
    # this table holds some 15M rows
    id bigserial # PK
    datetime timestamp with time zone
    eventtype_id bigint # FK to events_event_types.id
)

CREATE TABLE events_eventdetails
(
    # this table holds some 65M rows
    id bigserial # PK
    keyname character varying(255)
    "value" text
    event_id bigint # FK to events_events.id
)

events_events 和 events_eventdetails 表中的某些行如下所示：

events_events               | events_eventdetails
  id  datetime eventtype_id |   id   keyname       value        event_id
----------------------------|-------------------------------------------
  100 ...      10           |   1000 transactionId 9774ae16-... 100
                            |   1001 someKey       some value   100
  200 ...      20           |   2000 transactionId 9774ae16-... 200
                            |   2001 reductionId   123          200
                            |   2002 reductionId   456          200
  300 ...      30           |   3000 transactionId 9774ae16-... 300
                            |   2001 customerId    234          300
                            |   2001 companyId     345          300

我们迫切需要一个“解决方案”，它可以在单个结果集中同时返回 events_events 第 100 行、第 200 行和第 300 行，而且速度很快！当被问到 reductionId=123 或当被问到 customerId=234 或当被问到 companyId=345 时。（可能对这些标准的 AND 组合感兴趣，但这不是本质上的目标。）目前不确定它是否重要，但结果集应该可以根据日期时间范围和 eventtype_id（IN 列表）进行过滤，并给出一个限制。

我要求一个“解决方案”，因为这可能是：

  • 单个查询
  • 两个较小的查询（只要它们的中间结果总是足够小。我遵循这种方法并被困在具有大量（~20k）关联交易（transactionId）的公司（companyId））
  • 微妙的重新设计（例如非规范化）

这不是一个新问题，因为我们在数月内尝试了所有三种方法（这些查询不会打扰您），但性能都失败了。解决方案应在 <<<1s 内返回。以前的尝试大约花了。最好10s。

我真的很感激一些帮助 - 我现在不知所措......

两个较小的查询方法看起来很像这样：

查询一：

SELECT Substring(details2_transvalue.VALUE, 0, 32)
    FROM events_eventdetails details2_transvalue
    JOIN events_eventdetails compdetails ON details2_transvalue.event_id = compdetails.event_id
    AND compdetails.keyname = 'companyId'
    AND Substring(compdetails.VALUE, 0, 32) = '4'
    AND details2_transvalue.keyname = 'transactionId'

查询 2：

SELECT events1.*
    FROM events_events events1
    JOIN events_eventdetails compDetails ON events1.id = compDetails.event_id
    AND compDetails.keyname='companyId'
    AND substring(compDetails.value,0,32)='4'
    WHERE events1.eventtype_id IN (...)
UNION
SELECT events2.*
    FROM events_events events2
    JOIN events_eventdetails details2_transKey ON events2.id = details2_transKey.event_id
    AND details2_transKey.keyname='transactionId'
    AND substring(details2_transKey.value,0,32) IN ( -- result of query 1 goes here -- )
    WHERE events2.eventtype_id IN (...)
    ORDER BY dateTime DESC LIMIT 50

由于查询 1 返回的集合很大，因此性能变差。

如您所见，events_eventdetails 表中的值始终表示为长度为 32 的子字符串，我们已经对其进行了索引。keyname、event_id、event_id + keyname、keyname + 长度为 32 的子字符串的进一步索引。

这是一种 PostgreSQL 9.1 方法——尽管我没有正式拥有该平台供我使用：

WITH companyevents AS (
SELECT events1.*
FROM events_events events1
JOIN events_eventdetails compDetails
ON events1.id = compDetails.event_id
AND compDetails.keyname='companyId'
AND substring(compDetails.value,0,32)=' -- my desired companyId -- '
WHERE events1.eventtype_id in (...)
ORDER BY dateTime DESC
LIMIT 50
)
SELECT * from events_events
WHERE transaction_id IN (SELECT transaction_id FROM companyevents)
OR id IN (SELECT id FROM companyevents)
AND eventtype_id IN (...)
ORDER BY dateTime DESC
LIMIT 250;

companyId 28228 个 transactionId 的查询计划如下：

 Limit  (cost=7545.99..7664.33 rows=250 width=130) (actual time=210.100..3026.267 rows=50 loops=1)
   CTE companyevents
     ->  Limit  (cost=7543.62..7543.74 rows=50 width=130) (actual time=206.994..207.020 rows=50 loops=1)
           ->  Sort  (cost=7543.62..7544.69 rows=429 width=130) (actual time=206.993..207.005 rows=50 loops=1)
                 Sort Key: events1.datetime
                 Sort Method: top-N heapsort  Memory: 23kB
                 ->  Nested Loop  (cost=10.02..7529.37 rows=429 width=130) (actual time=0.093..178.719 rows=28228 loops=1)
                       ->  Append  (cost=10.02..1140.62 rows=657 width=8) (actual time=0.082..27.594 rows=28228 loops=1)
                             ->  Bitmap Heap Scan on events_eventdetails compdetails  (cost=10.02..394.47 rows=97 width=8) (actual time=0.021..0.021 rows=0 loops=1)
                                   Recheck Cond: (((keyname)::text = 'companyId'::text) AND ("substring"(value, 0, 32) = '4'::text))
                                   ->  Bitmap Index Scan on events_eventdetails_substring_ind  (cost=0.00..10.00 rows=97 width=0) (actual time=0.019..0.019 rows=0 loops=1)
                                         Index Cond: (((keyname)::text = 'companyId'::text) AND ("substring"(value, 0, 32) = '4'::text))
                             ->  Index Scan using events_eventdetails_companyid_substring_ind on events_eventdetails_companyid compdetails  (cost=0.00..746.15 rows=560 width=8) (actual time=0.061..18.655 rows=28228 loops=1)
                                   Index Cond: (((keyname)::text = 'companyId'::text) AND ("substring"(value, 0, 32) = '4'::text))
                       ->  Index Scan using events_events_pkey on events_events events1  (cost=0.00..9.71 rows=1 width=130) (actual time=0.004..0.004 rows=1 loops=28228)
                             Index Cond: (id = compdetails.event_id)
                             Filter: (eventtype_id = ANY ('{103,106,107,110,45,34,14,87,58,78,7,76,42,11,25,57,98,37,30,35,33,49,52,29,74,28,85,59,51,65,66,18,13,86,75,6,44,38,43,94,56,95,96,71,50,81,90,89,16,17,4,88,79,77,68,97,92,67,72,53,2,10,31,32,80,111,104,93,26,8,61,5,73,70,63,20,60,40,41,23,22,48,36,108,99,64,62,55,69,19,46,47,15,54,100,101,27,21,12,102,105,109,112,113,114,115,116,119,120,121,122,123,124,9,127,24,130,132,129,125,131,118,117,133,134}'::bigint[]))
   ->  Index Scan Backward using events_events_datetime_ind on events_events  (cost=2.25..1337132.75 rows=2824764 width=130) (actual time=210.100..3026.255 rows=50 loops=1)
         Filter: ((hashed SubPlan 2) OR ((hashed SubPlan 3) AND (eventtype_id = ANY ('{103,106,107,110,45,34,14,87,58,78,7,76,42,11,25,57,98,37,30,35,33,49,52,29,74,28,85,59,51,65,66,18,13,86,75,6,44,38,43,94,56,95,96,71,50,81,90,89,16,17,4,88,79,77,68,97,92,67,72,53,2,10,31,32,80,111,104,93,26,8,61,5,73,70,63,20,60,40,41,23,22,48,36,108,99,64,62,55,69,19,46,47,15,54,100,101,27,21,12,102,105,109,112,113,114,115,116,119,120,121,122,123,124,9,127,24,130,132,129,125,131,118,117,133,134}'::bigint[]))))
         SubPlan 2
           ->  CTE Scan on companyevents  (cost=0.00..1.00 rows=50 width=90) (actual time=206.998..207.071 rows=50 loops=1)
         SubPlan 3
           ->  CTE Scan on companyevents  (cost=0.00..1.00 rows=50 width=8) (actual time=0.001..0.026 rows=50 loops=1)
 Total runtime: 3026.410 ms

companyId 288 个 transactionId 的查询计划如下：

 Limit  (cost=7545.99..7664.33 rows=250 width=130) (actual time=30.976..3790.362 rows=54 loops=1)
   CTE companyevents
     ->  Limit  (cost=7543.62..7543.74 rows=50 width=130) (actual time=9.263..9.290 rows=50 loops=1)
           ->  Sort  (cost=7543.62..7544.69 rows=429 width=130) (actual time=9.263..9.272 rows=50 loops=1)
                 Sort Key: events1.datetime
                 Sort Method: top-N heapsort  Memory: 24kB
                 ->  Nested Loop  (cost=10.02..7529.37 rows=429 width=130) (actual time=0.071..8.195 rows=1025 loops=1)
                       ->  Append  (cost=10.02..1140.62 rows=657 width=8) (actual time=0.060..1.348 rows=1025 loops=1)
                             ->  Bitmap Heap Scan on events_eventdetails compdetails  (cost=10.02..394.47 rows=97 width=8) (actual time=0.021..0.021 rows=0 loops=1)
                                   Recheck Cond: (((keyname)::text = 'companyId'::text) AND ("substring"(value, 0, 32) = '5'::text))
                                   ->  Bitmap Index Scan on events_eventdetails_substring_ind  (cost=0.00..10.00 rows=97 width=0) (actual time=0.019..0.019 rows=0 loops=1)
                                         Index Cond: (((keyname)::text = 'companyId'::text) AND ("substring"(value, 0, 32) = '5'::text))
                             ->  Index Scan using events_eventdetails_companyid_substring_ind on events_eventdetails_companyid compdetails  (cost=0.00..746.15 rows=560 width=8) (actual time=0.039..1.006 rows=1025 loops=1)
                                   Index Cond: (((keyname)::text = 'companyId'::text) AND ("substring"(value, 0, 32) = '5'::text))
                       ->  Index Scan using events_events_pkey on events_events events1  (cost=0.00..9.71 rows=1 width=130) (actual time=0.005..0.006 rows=1 loops=1025)
                             Index Cond: (id = compdetails.event_id)
                             Filter: (eventtype_id = ANY ('{103,106,107,110,45,34,14,87,58,78,7,76,42,11,25,57,98,37,30,35,33,49,52,29,74,28,85,59,51,65,66,18,13,86,75,6,44,38,43,94,56,95,96,71,50,81,90,89,16,17,4,88,79,77,68,97,92,67,72,53,2,10,31,32,80,111,104,93,26,8,61,5,73,70,63,20,60,40,41,23,22,48,36,108,99,64,62,55,69,19,46,47,15,54,100,101,27,21,12,102,105,109,112,113,114,115,116,119,120,121,122,123,124,9,127,24,130,132,129,125,131,118,117,133,134}'::bigint[]))
   ->  Index Scan Backward using events_events_datetime_ind on events_events  (cost=2.25..1337132.75 rows=2824764 width=130) (actual time=30.975..3790.332 rows=54 loops=1)
         Filter: ((hashed SubPlan 2) OR ((hashed SubPlan 3) AND (eventtype_id = ANY ('{103,106,107,110,45,34,14,87,58,78,7,76,42,11,25,57,98,37,30,35,33,49,52,29,74,28,85,59,51,65,66,18,13,86,75,6,44,38,43,94,56,95,96,71,50,81,90,89,16,17,4,88,79,77,68,97,92,67,72,53,2,10,31,32,80,111,104,93,26,8,61,5,73,70,63,20,60,40,41,23,22,48,36,108,99,64,62,55,69,19,46,47,15,54,100,101,27,21,12,102,105,109,112,113,114,115,116,119,120,121,122,123,124,9,127,24,130,132,129,125,131,118,117,133,134}'::bigint[]))))
         SubPlan 2
           ->  CTE Scan on companyevents  (cost=0.00..1.00 rows=50 width=90) (actual time=9.266..9.327 rows=50 loops=1)
         SubPlan 3
           ->  CTE Scan on companyevents  (cost=0.00..1.00 rows=50 width=8) (actual time=0.001..0.019 rows=50 loops=1)
 Total runtime: 3796.736 ms

对于 3s/4s，这一点也不差，但仍然太慢了 100+。此外，这不在相关硬件上。尽管如此，它应该显示疼痛在哪里。

这里有一些可能会变成解决方案的东西：

添加了一个表：

events_transaction_helper
(
    event_id bigint not null
    transactionid character varying(36) not null
    keyname character varying(255) not null
    value bigint not null
    # index on keyname, value
)

我现在“手动”填写了这张表，但是物化视图实现可以解决问题。它将遵循以下查询：

SELECT tr.event_id, tr.value AS transactionid, det.keyname, det.value AS value
   FROM events_eventdetails tr
   JOIN events_eventdetails det ON det.event_id = tr.event_id
  WHERE tr.keyname = 'transactionId'
     AND det.keyname
     IN ('companyId', 'reduction_id', 'customer_id');

在 events_events 表中添加了一列：

transaction_id character varying(36) null

这个新列的填充如下：

update events_events
set transaction_id =
    (select value from events_eventdetails
        where keyname='transactionId'
        and event_id=events_events.id);

现在，以下查询始终在 <15ms 内返回：

explain analyze select * from events_events
    where transactionId in
    (select distinct transactionid
        from events_transaction_helper
        WHERE keyname='companyId' and value=5)
    and eventtype_id in (...)
    order by datetime desc limit 250;

 Limit  (cost=5075.23..5075.85 rows=250 width=130) (actual time=8.901..9.028 rows=250 loops=1)
   ->  Sort  (cost=5075.23..5077.19 rows=785 width=130) (actual time=8.900..8.953 rows=250 loops=1)
         Sort Key: events_events.datetime
         Sort Method: top-N heapsort  Memory: 81kB
         ->  Nested Loop  (cost=57.95..5040.04 rows=785 width=130) (actual time=0.928..8.268 rows=524 loops=1)
               ->  HashAggregate  (cost=52.30..52.42 rows=12 width=37) (actual time=0.895..0.991 rows=276 loops=1)
                     ->  Subquery Scan on "ANY_subquery"  (cost=52.03..52.27 rows=12 width=37) (actual time=0.558..0.757 rows=276 loops=1)
                           ->  HashAggregate  (cost=52.03..52.15 rows=12 width=37) (actual time=0.556..0.638 rows=276 loops=1)
                                 ->  Index Scan using testmaterializedviewkeynamevalue on events_transaction_helper  (cost=0.00..51.98 rows=22 width=37) (actual time=0.068..0.404 rows=288 loops=1)
                                       Index Cond: (((keyname)::text = 'companyId'::text) AND (value = 5))
               ->  Bitmap Heap Scan on events_events  (cost=5.65..414.38 rows=100 width=130) (actual time=0.023..0.024 rows=2 loops=276)
                     Recheck Cond: ((transactionid)::text = ("ANY_subquery".transactionid)::text)
                     Filter: (eventtype_id = ANY ('{103,106,107,110,45,34,14,87,58,78,7,76,42,11,25,57,98,37,30,35,33,49,52,29,74,28,85,59,51,65,66,18,13,86,75,6,44,38,43,94,56,95,96,71,50,81,90,89,16,17,4,88,79,77,68,97,92,67,72,53,2,10,31,32,80,111,104,93,26,8,61,5,73,70,63,20,60,40,41,23,22,48,36,108,99,64,62,55,69,19,46,47,15,54,100,101,27,21,12,102,105,109,112,113,114,115,116,119,120,121,122,123,124,9,127,24,130,132,129,125,131,118,117,133,134}'::bigint[]))
                     ->  Bitmap Index Scan on testtransactionid  (cost=0.00..5.63 rows=100 width=0) (actual time=0.020..0.020 rows=2 loops=276)
                           Index Cond: ((transactionid)::text = ("ANY_subquery".transactionid)::text)
 Total runtime: 9.122 ms

我稍后会回来告诉你这是否真的是一个可行的解决方案:)

4

3 回答 3

1

这个想法不是去规范化，而是规范化。events_details() 表可以替换为两张表：一张是 event_detail_types，一张是实际值（参考 {even_id,detail_types}。这将使查询的执行更容易，因为只有数字 id 的detail_types 必须被提取和选择。好处是减少了 DBMS 必须获取的页面数量，因为所有的键名只需要存储+检索+比较一次。

注意：我稍微改变了命名。主要是出于理智和安全的原因。

SET search_path='cav';
/**** ***/
DROP SCHEMA cav CASCADE;
CREATE SCHEMA cav;
SET search_path='cav';

CREATE TABLE event_types
(
    -- this table holds some 50 rows
    id bigserial PRIMARY KEY
    , zname varchar(255)
);
INSERT INTO event_types(zname)
SELECT 'event_'::text || gs::text
FROM generate_series (1,100) gs
        ;

CREATE TABLE events
(
    -- this table holds some 15M rows
    id bigserial PRIMARY KEY
    , zdatetime timestamp with time zone
    , eventtype_id bigint REFERENCES event_types(id)
);
INSERT INTO events(zdatetime,eventtype_id)
SELECT gs, et.id
FROM generate_series ('2012-04-11 00:00:00'::timestamp
                     , '2012-04-12 12:00:00'::timestamp  ,' 1 hour'::interval ) gs
        , event_types et
        ;

-- SELECT * FROM event_types;
-- SELECT * FROM events;

CREATE TABLE event_details
(
    -- this table holds some 65M rows
    id bigserial PRIMARY KEY
    , event_id bigint REFERENCES events(id)
    , keyname varchar(255)
    , zvalue text
);

INSERT INTO event_details(event_id, keyname)
SELECT ev.id,im.*
FROM events ev
        , (VALUES ('transactionId'::text),('someKey'::text)
           ,('reductionId'::text),('customerId'::text),('companyId'::text)
          ) im
        ;
UPDATE event_details
SET zvalue = 'Some_value'::text || (random() * 1000)::int::text
        ;
        --
        -- Domain table with all valid detail_types
        --
CREATE TABLE detail_types(
        id bigserial PRIMARY KEY
        , keyname varchar(255)
        );
INSERT INTO detail_types(keyname)
SELECT DISTINCT keyname
        FROM event_details
        ;

        --
        -- Context-attribute-value table, referencing {event_id, type_id}
        --
CREATE TABLE event_detail_values
        ( event_id BIGINT
        , detail_type_id BIGINT
        , zvalue text
        , PRIMARY KEY(event_id , detail_type_id)
        , FOREIGN KEY(event_id ) REFERENCES events(id)
        , FOREIGN KEY(detail_type_id)REFERENCES detail_types(id)
        );

        --
        -- For the sake of joining we create some natural keys
        --
CREATE INDEX events_details_keyname ON event_details (keyname) ;
CREATE INDEX detail_types_keyname ON detail_types(keyname) ;

INSERT INTO event_detail_values (event_id,detail_type_id, zvalue)
        SELECT ed.event_id, dt.id
                , ed.zvalue
        FROM event_details ed
        , detail_types dt
        WHERE ed.keyname = dt.keyname
        ;
        --
        -- Now we can drop the original table, and use the view instead
        --
DROP TABLE event_details;
CREATE VIEW event_details AS (
        SELECT dv.event_id AS event_id
                , dt.keyname AS keyname
                , dv.zvalue AS zvalue
        FROM event_detail_values dv
        JOIN detail_types dt ON dt.id = dv.detail_type_id
        );
EXPLAIN ANALYZE
SELECT ev.id AS event_id
        , ev.zdatetime AS zdatetime
        , ed.keyname AS keyname
        , ed.zvalue AS zevalue
        FROM events ev
        JOIN event_details ed ON ed.event_id = ev.id
        WHERE ed.keyname IN ('transactionId','customerId','companyId')
        ORDER BY event_id,keyname
        ;

结果查询计划：

                                                                 QUERY PLAN                                                                  
----------------------------------------------------------------------------------------------------------------------------------------------
 Sort  (cost=1178.79..1197.29 rows=7400 width=40) (actual time=159.902..177.379 rows=11100 loops=1)
   Sort Key: ev.id, dt.keyname
   Sort Method: external sort  Disk: 560kB
   ->  Hash Join  (cost=108.34..703.22 rows=7400 width=40) (actual time=12.225..122.231 rows=11100 loops=1)
         Hash Cond: (dv.event_id = ev.id)
         ->  Hash Join  (cost=1.09..466.47 rows=7400 width=32) (actual time=0.047..74.183 rows=11100 loops=1)
               Hash Cond: (dv.detail_type_id = dt.id)
               ->  Seq Scan on event_detail_values dv  (cost=0.00..322.00 rows=18500 width=29) (actual time=0.006..26.543 rows=18500 loops=1)
               ->  Hash  (cost=1.07..1.07 rows=2 width=19) (actual time=0.025..0.025 rows=3 loops=1)
                     Buckets: 1024  Batches: 1  Memory Usage: 1kB
                     ->  Seq Scan on detail_types dt  (cost=0.00..1.07 rows=2 width=19) (actual time=0.009..0.014 rows=3 loops=1)
                           Filter: ((keyname)::text = ANY ('{transactionId,customerId,companyId}'::text[]))
         ->  Hash  (cost=61.00..61.00 rows=3700 width=16) (actual time=12.161..12.161 rows=3700 loops=1)
               Buckets: 1024  Batches: 1  Memory Usage: 131kB
               ->  Seq Scan on events ev  (cost=0.00..61.00 rows=3700 width=16) (actual time=0.004..5.926 rows=3700 loops=1)
 Total runtime: 192.724 ms
 (16 rows)

如您所见，查询的“最深”部分是在给定字符串列表的情况下检索 detail_type_ids。这被放入哈希表中，然后与 detail_values 的相应哈希集组合。（注意：这是 pg-9.1）

YMMV。

于 2012-04-12T11:14:14.760 回答
0

看，如果您的键（reductionId在这种情况下）在表中所有行的 7-10% 以上都满足events_eventdetails，那么 PostgreSQL 将更喜欢 SeqScan。没有什么可以做的，这是最快的方法。

我有一个使用 ISO8583 数据包的类似案例。每个数据包包含 128 个字段（按设计），因此第一个数据库设计遵循您的方法，使用 2 个表：

  • field_id和一张表中的描述（events_events在你的情况下），
  • field_id+field_value在另一个 ( events_eventdetails) 中。

虽然这样的布局遵循 3NF，但我们马上遇到了同样的问题：

  • 表现不佳，
  • 高度复杂的查询。

在您的情况下，您应该进行重新设计。一种选择（更简单的一种）是events_eventdetails.keyname成为 a smallint，这将使比较操作更快。虽然不是很大的胜利。

另一种选择是将 2 个表减少为一个，例如：

CREATE TABLE events_events (
    id            bigserial,
    datetime      timestamp with time zone,
    eventtype_id  bigint,
    transactionId text,   -- value for transactionId 
    reductionId   text,   --   -"-     reductionId
    companyId     text,   -- etc.
    customerId    text,
    anyotherId    text,
    ...
);

这将打破 3NF，但另一方面：

  • 您有更多的自由来索引您的数据；
  • 您的查询将更短且更易于维护；
  • 性能会好很多。

可能的缺点：

  • 您将为未使用的字段浪费更多空间：unused fields / 8每行字节数
  • 您可能仍然需要一个额外的表格来处理那些太靠后而无法保留单独列的事件。

编辑：

我不太明白你在这里实现的意思。

在您提到的问题中，您想要：

“解决方案”在单个结果集中同时返回 events_events 第 100 行、第 200 行和第 300 行，而且速度很快！当被问到 reductionId=123 或当被问到 customerId=234 或当被问到 companyId=345 时。

建议的重新设计从您的events_eventdetails. 要获取满足您条件的所有 events_events 行，您可以使用：

SELECT *
  FROM events_events
 WHERE id IN (100, 200, 300)
   AND reductionId = 123
-- AND customerId = 234
-- AND companyId = 345;
于 2012-04-11T06:13:42.887 回答
0

如果您必须按照这些思路使用设计，您应该从 events_eventdetails 中删除 id 列，并将主键声明为 (event_id, keyname)。这将为您提供一个非常有用的索引，而无需为合成键维护一个无用的索引。

更好的方法是完全消除 events_eventdetails 表，并为该数据使用 hstore 列，并使用 GIN 索引。这可能会让您达到您的性能目标，而无需预先定义存储的事件详细信息。

更好的是，如果您可以预测或指定可能的事件细节，那就是不要尝试在数据库中实现数据库。将每个“keyname”值放入 events_eventdetails 中的列中，并使用适合该数据性质的数据类型。这可能会允许更快的访问，但代价是ALTER TABLE随着细节的性质发生变化而需要发布声明。

于 2012-04-10T23:41:31.863 回答