2

我有一个非常简单的数据库结构，其中包含“日终”股票价格，类似于：

finalyzer_pricedata=> \d pdEndOfDayPricEentity
              Table "public.pdendofdaypriceentity"
    Column     |     Type      | Collation | Nullable | Default 
---------------+---------------+-----------+----------+---------
 id            | uuid          |           | not null | 
 close         | numeric(19,2) |           | not null | 
 day           | date          |           | not null | 
 instrument_id | uuid          |           | not null |

（instrument_id 是股票的唯一 ID）

我现在想选择所有instrument_id在本周达到 52 周高点的股票。（即close最近 7 天的列高于前 52 周的所有股票）

我尝试了许多不同的方法：group by 和 max()、select distinct on、窗口函数（row_number），但我没有设法让它低于 150 秒。我目前最好的（也是最简单的）方法是：

select CAST(weekHigh.instrument_id AS VARCHAR) instrumentId,
                       weekHigh.maxClose                       weekHighValue,
                       yearHigh.maxClose                       yearHighValue,
                       yearHigh.maxDay                         yearHighDay
                from 
                     (select distinct on (eod.instrument_id) instrument_id,
                                         eod.close  maxClose,
                                         eod.day as maxDay
                                  from pdendofdaypriceentity eod
                                  where eod.day BETWEEN (CAST('2018-11-12' AS date) - interval '52 weeks') AND (CAST('2018-11-12' AS date) - interval '1 day')
                                  order by eod.instrument_id, close desc) yearHigh
                       inner join (select eod.instrument_id instrument_id, max(eod.close) maxClose
                                   from pdendofdaypriceentity eod
                                   where eod.day BETWEEN CAST('2018-11-12' AS date) AND CAST('2018-11-18' AS date)
                                   group by eod.instrument_id) weekHigh
                         on weekHigh.instrument_id = yearHigh.instrument_id
                where weekHigh.maxClose > yearHigh.maxClose;

我很清楚有许多类似的问题，但这些方法让我找到了一个可行的解决方案，但没有一个能帮助我提高性能。该表包含来自不同 28000 支股票的 1000 万行。这只会变得更大。有没有办法在不进行非规范化的情况下通过不到 2 秒的查询来实现这个要求？显然，任何类型的索引等都可以。

上述方法的查询计划：

                                                                  QUERY PLAN                                                                              
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Hash Join  (cost=148153.45..1136087.99 rows=6112 width=74) (actual time=3056.748..144632.288 rows=411 loops=1)
   Hash Cond: (eod.instrument_id = eod_1.instrument_id)
   Join Filter: ((max(eod_1.close)) > eod.close)
   Rows Removed by Join Filter: 27317
   ->  Unique  (cost=0.56..987672.73 rows=18361 width=26) (actual time=2.139..141494.533 rows=28216 loops=1)
         ->  Index Scan using test3 on pdendofdaypriceentity eod  (cost=0.56..967290.80 rows=8152771 width=26) (actual time=2.117..79396.893 rows=8181608 loops=1)
               Filter: ((day >= '2017-11-13 00:00:00'::timestamp without time zone) AND (day <= '2018-11-11 00:00:00'::timestamp without time zone))
               Rows Removed by Filter: 1867687
   ->  Hash  (cost=147923.68..147923.68 rows=18337 width=48) (actual time=2793.633..2793.639 rows=27917 loops=1)
         Buckets: 32768  Batches: 1  Memory Usage: 1739kB
         ->  HashAggregate  (cost=147556.94..147740.31 rows=18337 width=48) (actual time=2301.968..2550.387 rows=27917 loops=1)
               Group Key: eod_1.instrument_id
               ->  Bitmap Heap Scan on pdendofdaypriceentity eod_1  (cost=2577.01..146949.83 rows=121422 width=22) (actual time=14.264..1146.610 rows=115887 loops=1)
                     Recheck Cond: ((day >= '2018-11-12'::date) AND (day <= '2018-11-18'::date))
                     Heap Blocks: exact=11992
                     ->  Bitmap Index Scan on idx5784y3l3mqprlmeyyrmwnkt3n  (cost=0.00..2546.66 rows=121422 width=0) (actual time=12.784..12.791 rows=115887 loops=1)
                           Index Cond: ((day >= '2018-11-12'::date) AND (day <= '2018-11-18'::date))
 Planning time: 13.758 ms
 Execution time: 144635.973 ms
(19 rows)

我当前的（基本上是随机的）索引：

Indexes:
    "pdendofdaypriceentity_pkey" PRIMARY KEY, btree (id)
    "ukcaddwp8kcx2uox18vss7o5oly" UNIQUE CONSTRAINT, btree (instrument_id, day)
    "idx5784y3l3mqprlmeyyrmwnkt3n" btree (day)
    "idx5vqqjfube2j1qkstc741ll19u" btree (close)
    "idxcaddwp8kcx2uox18vss7o5oly" btree (instrument_id, day)
    "test1" btree (close DESC, instrument_id, day)
    "test2" btree (instrument_id, day, close DESC)
    "test3" btree (instrument_id, close DESC)
4

3 回答 3

2

如果您在固定窗口而不是动态范围上查询，查询可能会变得更快。

例如

  • 今年（日历）
  • 本季度
  • 这个月

这是因为它允许您使用可以找到工具的最高收盘价的索引，而无需读取一系列值。

例如，如果我们想找到 2017 年的最高值

首日指数

|day|instrument|close|
|31-12-2016|1|12.00|
|01-01-2017|1|19.00|    <-- start scan here
...
|06-01-2017|1|31.00|    <-- highest
...
|31-12-2017|1|11.00|    <-- end scan here
|01-01-2018|1|13.00|

注意：close实际上是无序的，因此扫描

年首指数

|extract(year from day)|instrument|close|day|
|2016|1|12.00|31-12-2016|
|2017|1|31.00|06-01-2017|  <-- highest close for that year at the top
...
|2017|1|19.00|01-01-2017|
...
|2017|1|11.00|31-12-2017|
|2018|1|13.00|01-01-2018|

因此，您扫描的记录可能会减少 365 倍。你仍然可以请求这一天。

注意：您的过滤器需要使用与索引相同的日期函数

于 2018-11-20T16:50:09.823 回答
2

尝试以下查询

select weekHigh.instrument_id,
       weekHigh.maxClose                       weekLowValue,
       yearHigh.maxClose                       yearLowValue
from (
    select instrument_id,
         max(eod.close)  maxClose
    from pdendofdaypriceentity eod
    where eod.day BETWEEN (CAST('2018-11-12' AS date) - interval '52 weeks') AND (CAST('2018-11-12' AS date) - interval '1 day')
    group by eod.instrument_id
) yearHigh
inner join (
    select eod.instrument_id instrument_id, max(eod.close) maxClose
    from pdendofdaypriceentity eod
    where eod.day BETWEEN CAST('2018-11-12' AS date) AND CAST('2018-11-18' AS date)
    group by eod.instrument_id
) weekHigh on weekHigh.instrument_id = yearHigh.instrument_id
where weekHigh.maxClose > yearHigh.maxClose;

与索引pdendofdaypriceentity(day, instrument_id, close)。请注意，它缺少maxDay您在查询中的内容。

可以maxDay通过另一个连接来添加pdendofdaypriceentity，但是，我将从上面的查询开始，而没有distinct on和order by在第一个子查询中。

于 2018-11-20T14:29:51.747 回答
0

您会考虑在其他地方进行数据分析吗？您提到的尺寸对于明智的 R 或 Python 方法来说并没有那么多。下面的示例适用于 2000 万行的表（28,000 个仪器 ID，每个具有 720 个观测值），在当前的 Macbook Pro 上需要 ±1 秒。我制作了一些模拟数据来配合它。

用 R 和data.table：

# Filter data for the past 52 weeks
result <-
    data[day >= max(day) - 52*7 & day <= max(day)]

# Get all instances where close was at max
result_52max <- 
    result[result[, .I[close == max(close)], by = instrument_id]$V1]

# Get all instances where this happened last week
result_7max <- 
    result_52max[day >= max(day) - 7 & day <= max(day)]

平均运行时间：< 1 秒。

再生产

数据

# Attention: takes a couple of minutes
library(data.table)
set.seed(1701)
data <- NULL
id <- 1
n <- 1000
obs <- 720
for(j in 1:28){
  dt <- NULL
  dates <- seq.Date(from = as.Date("2017-01-01"), by = "day", length.out = obs)
  for(i in (1+(j-1)*n):(n*j)){
    start <- sample(1:200, 1)
    dt <- rbindlist(list(dt,
                         data.table(id = id:(id+obs-1),
                                    close = abs(start + cumsum(sample(c(-1, 1), obs, TRUE))),
                                    day = dates,
                                    instrument_id = rep(i, obs))))
    id <- id+obs
  }
  data <- rbindlist(list(data, dt))
}

结构

> str(data)
Classes ‘data.table’ and 'data.frame':  20160000 obs. of  4 variables:
 $ id           : int  1 2 3 4 5 6 7 8 9 10 ...
 $ close        : num  27 26 27 28 27 28 29 28 29 30 ...
 $ day          : Date, format: "2017-01-01" "2017-01-02" "2017-01-03" ...
 $ instrument_id: int  1 1 1 1 1 1 1 1 1 1 ...
 - attr(*, ".internal.selfref")=<externalptr>

绘制前五个仪器 ID

在此处输入图像描述

library(ggplot2)
ggplot(subset(data, data$instrument_id <= 5), 
       aes(day, close, color = as.factor(instrument_id), 
           group = as.factor(instrument_id))) +
    geom_line()
于 2018-11-22T02:56:07.263 回答