0

我在分区表上有一个大型数据库（每分钟插入 6000 行），当数据库很小时我工作得很好，现在我有一个大型数据库。我使用这个解决方案以前的解决方案SQL 按日期连接，但它使用 250MB 的硬盘并且随着我的表的增长而增长，然后我决定将其更改为简单查询的迭代，它适用于 10 行，但是得到超过 10 辆汽车（响应时间为 15 秒）很慢，并且使用超过 200MB 的硬盘。

我的问题是如何更快地构建一个好的查询来解决这个问题

额外信息

  1. ajax 在 django 应用程序上调用查询
  2. 我正在考虑迭代 ajax 调用，而不是一个带有完整项目列表响应的调用
  3. 表按天分区

我的实际查询是

CREATE OR REPLACE FUNCTION gps_get_last_positions (
    _plates varchar(8)
)
RETURNS TABLE (
    plate varchar,
    device_id integer,
    date_time_process timestamp with time zone, 
    latitude double precision, 
    longitude double precision, 
    course smallint, 
    speed smallint, 
    mileage integer,
    gps_signal smallint,
    gsm_signal smallint,
    alarm_status boolean,
    gsm_status boolean,
    vehicle_status boolean,
    alarm_over_speed boolean,
    other text,
    realtime  double precision
) AS $func$
DECLARE 
    arr varchar[];

BEGIN
    arr := regexp_split_to_array(_plates, E'\\s+');
    FOR i IN 1..array_length(arr, 1) LOOP
        RETURN QUERY SELECT 
        gpstracking_vehicles.registration,
        gpstracking_device_tracks.device_id, 
        gpstracking_device_tracks.date_time_process,
        gpstracking_device_tracks.latitude,
        gpstracking_device_tracks.longitude,
        gpstracking_device_tracks.course,
        gpstracking_device_tracks.speed,
        gpstracking_device_tracks.mileage,
        gpstracking_device_tracks.gps_signal,
        gpstracking_device_tracks.gsm_signal,
        gpstracking_device_tracks.alarm_status,
        gpstracking_device_tracks.gps_status,
        gpstracking_device_tracks.vehicle_status,
        gpstracking_device_tracks.alarm_over_speed,
        gpstracking_device_tracks.other,
        EXTRACT(EPOCH FROM current_timestamp - gpstracking_device_tracks.date_time_process)/60 AS realtime
        FROM (
        gpstracking_devices INNER JOIN (
        gpstracking_vehicles INNER JOIN gpstracking_vehicles_devices ON ( gpstracking_vehicles.id = gpstracking_vehicles_devices.vehicle_id AND gpstracking_vehicles_devices.is_joined = TRUE )
        ) ON ( gpstracking_devices.id = gpstracking_vehicles_devices.device_id AND gpstracking_vehicles_devices.is_joined = TRUE )
        ) INNER JOIN gpstracking_device_tracks ON ( gpstracking_devices.id = gpstracking_device_tracks.device_id )
        WHERE gpstracking_vehicles.registration = arr[i]::VARCHAR
        ORDER BY gpstracking_device_tracks.date_time_process DESC
        LIMIT 1;
    END LOOP;
    RETURN;
END;
$func$ 
LANGUAGE plpgsql VOLATILE SECURITY DEFINER;

配置参数

application_name            phpPgAdmin_5.0.4                    client
constraint_exclusion        on                                  configuration file
DateStyle                   ISO, MDY                            session
default_text_search_config  pg_catalog.english                  configuration file
external_pid_file           /var/run/postgresql/9.1-main.pid    configuration file
lc_messages                 en_US.UTF-8                         configuration file
lc_monetary                 en_US.UTF-8                         configuration file
lc_numeric                  en_US.UTF-8                         configuration file
lc_time                     en_US.UTF-8                         configuration file
log_line_prefix             %t                                  configuration file
log_timezone                localtime                           environment variable
max_connections             100                                 configuration file
max_stack_depth             2MB                                 environment variable
port                        5432                                configuration file
shared_buffers              24MB                                configuration file
ssl                         on                                  configuration file
TimeZone                    localtime                           environment variable
unix_socket_directory       /var/run/postgresql                 configuration file

我的第一个慢查询是：

CREATE OR REPLACE VIEW view_vehicle_devices AS
SELECT 
gpstracking_vehicles_devices.id AS id,
gpstracking_devices.id AS device_id,
gpstracking_vehicles.id AS vehicle_id,
gpstracking_drivers.id AS driver_id,
gpstracking_device_protocols.name AS protocol,
gpstracking_vehicles.registration AS plate,
gpstracking_drivers.firstname as first_name,
gpstracking_drivers.lastname as last_name,
gpstracking_devices.imei,
gpstracking_devices.simcard, 
gpstracking_device_tracks.date_time_process,
gpstracking_device_tracks.latitude,
gpstracking_device_tracks.longitude,
gpstracking_device_tracks.course,
gpstracking_device_tracks.speed,
gpstracking_device_tracks.mileage,
gpstracking_device_tracks.gps_signal,
gpstracking_device_tracks.gsm_signal,
gpstracking_device_tracks.alarm_status,
gpstracking_device_tracks.gps_status,
gpstracking_device_tracks.vehicle_status,
gpstracking_device_tracks.alarm_over_speed,
gpstracking_device_tracks.other,
gpstracking_device_tracks.point,
EXTRACT(EPOCH FROM current_timestamp - gpstracking_device_tracks.date_time_process)/60 realtime,
gpstracking_devices.created,
gpstracking_devices.updated,
gpstracking_devices.is_connected

FROM (
gpstracking_vehicles LEFT JOIN (
gpstracking_drivers  LEFT JOIN gpstracking_vehicles_drivers ON gpstracking_drivers.id = gpstracking_vehicles_drivers.driver_id AND gpstracking_vehicles_drivers.is_joined = TRUE
) ON gpstracking_vehicles.id = gpstracking_vehicles_drivers.vehicle_id AND gpstracking_vehicles_drivers.is_joined = TRUE
) LEFT JOIN (((
gpstracking_device_protocols RIGHT JOIN gpstracking_devices ON gpstracking_device_protocols.id = gpstracking_devices.device_protocol_id
) LEFT JOIN (
SELECT DISTINCT ON (gpstracking_device_tracks.device_id) gpstracking_device_tracks.device_id, 
gpstracking_device_tracks.date_time_process,
gpstracking_device_tracks.latitude,
gpstracking_device_tracks.longitude,
gpstracking_device_tracks.course,
gpstracking_device_tracks.speed,
gpstracking_device_tracks.mileage,
gpstracking_device_tracks.gps_signal,
gpstracking_device_tracks.gsm_signal,
gpstracking_device_tracks.alarm_status,
gpstracking_device_tracks.gps_status,
gpstracking_device_tracks.vehicle_status,
gpstracking_device_tracks.alarm_over_speed,
gpstracking_device_tracks.other,
gpstracking_device_tracks.point
FROM gpstracking_device_tracks 
ORDER BY gpstracking_device_tracks.device_id, gpstracking_device_tracks.date_time_process DESC
) AS gpstracking_device_tracks ON gpstracking_devices.id = gpstracking_device_tracks.device_id  
) LEFT JOIN gpstracking_vehicles_devices ON ( gpstracking_devices.id = gpstracking_vehicles_devices.device_id AND gpstracking_vehicles_devices.is_joined = TRUE )
) ON ( gpstracking_vehicles.id = gpstracking_vehicles_devices.vehicle_id AND gpstracking_vehicles_devices.is_joined = TRUE )

我已经为开始我的帖子的循环更改了它，我的循环 rujns 更快，但是还不够快，因为我需要

4

1 回答 1

2

您的问题是计划者无法知道哪个分区是您查询的答案。它只有统计数据。因此，您根本不会从按一天对数据进行分区中受益。

要从中受益，您可以修改查询，以便查找当天的最新坐标，如果未找到则从昨天开始，如果在前一天未找到，依此类推。我想 99% 的答案只能在今天的分区中找到。

或者您可以device_id % 256改为分区。

但更好的是创建一个仅包含几个最近设备坐标的附加表。它将通过一个触发器来维护，gpstracking_device_tracks它只会执行（伪代码）：

if random()*64 < 1.0 then
  -- statistically once per 64 runs do a cleanup
  with todelete as (
    -- lock rows in particular order to avoid possible deadlocks
    -- if run concurrently
    select id from gpstracking_device_tracks_recent where device_id=?
      order by id for share;
  )
  delete from gpstracking_device_tracks_recent
    where id in (select id from todelete)
end if;
insert into gpstracking_device_tracks_recent (...) values (...);

然后在这个小得多的表中查找最新坐标。

于 2013-09-01T22:48:07.950 回答