mysql - InnoDB 集群内存问题与 LOAD DATA

Question

我需要创建一个大约 8M 行的新表，而不会对使用集群的其他服务造成太多噪音。我将数据分成 100k 行的块并使用LOAD DATA LOCAL FILE, sleep 5s 并重复。Mysql 开始消耗所有内存，直到 OOM 杀死集群节点上的 mysqld。

我究竟做错了什么？？

欢迎任何关于如何进步的帮助或想法，谢谢！

InnoDB Cluster 8.0.24，三个虚拟主机上的 Ubuntu 20.04（8.0.23 的行为相同）。6 个 CPU 内核，每个 12GB 内存。与默认配置相比，唯一的配置更改是local_infile = ON.

@Wilson 请求的附加信息（链接到 pastebin）：

我正在将数据加载到我的表中：

CREATE TABLE IF NOT EXISTS `RefdataSnapshotM_20210421140303` (
  `SERVICE_INFO`             VARCHAR(64) NOT NULL,
  `PORTING_TIME`             TIMESTAMP   DEFAULT NULL,
  `IDN`                      VARCHAR(64) NOT NULL,
  `PORTING_ID`               VARCHAR(64) NOT NULL,
  `RECIPIENT_SP_ID`          CHAR(3)     NOT NULL,
  `DONOR_SP_ID`              CHAR(3)     NOT NULL,
  `RECIPIENT_ROUTING_NUMBER` CHAR(3)     NOT NULL,
  `DONOR_ROUTING_NUMBER`     CHAR(3)     NOT NULL,
  `PORTING_INDICATOR`        VARCHAR(64) NOT NULL,
  `CREATION_DATE`            TIMESTAMP   DEFAULT NULL,
  PRIMARY KEY  (`IDN`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4

我如何加载数据：

LOAD DATA
  LOCAL INFILE '/tmp/datasource.dat'
  INTO TABLE `RefdataSnapshotM_20210421140303`
  CHARACTER SET utf8mb4
  (
    `SERVICE_INFO`,
    @var_PORTING_TIME,
    @var_IDN,
    `PORTING_ID`,
    `RECIPIENT_SP_ID`,
    `DONOR_SP_ID`,
    `RECIPIENT_ROUTING_NUMBER`,
    `DONOR_ROUTING_NUMBER`,
    `PORTING_INDICATOR`,
    @var_CREATION_DATE
  )
  SET  `PORTING_TIME` = STR_TO_DATE(@var_PORTING_TIME, GET_FORMAT(DATETIME, 'INTERNAL')),
                `IDN` = REGEXP_REPLACE(@var_IDN, '^46', '', 1),
      `CREATION_DATE` = STR_TO_DATE(@var_CREATION_DATE, '%d-%m-%Y %H:%i:%s')

一些内存相关信息：

SELECT total_allocated DIV total_allocated 1024 AS FROM sys.x$memory_global_total
+-----------------+
| total_allocated |
+-----------------+
| 11366374 kB     |
+-----------------+
cat /proc/<pid_of_mysql_here>/smaps_rollup
Rss:            11784296 kB
Pss:            11783545 kB
Pss_Anon:       11767256 kB
Pss_File:          16289 kB
Pss_Shmem:             0 kB
Shared_Clean:        952 kB
Shared_Dirty:          0 kB
Private_Clean:     16196 kB
Private_Dirty:  11767148 kB
Referenced:     11123016 kB
Anonymous:      11767256 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
FilePmdMapped:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:             371376 kB
SwapPss:          371376 kB
Locked:                0 kB

SELECT SUM(current_alloc) AS total_allocated
  FROM sys.x$memory_global_by_current_bytes
UNION
SELECT total_allocated
  FROM sys.x$memory_global_total;
+-----------------+
| total_allocated |
+-----------------+
|      1344314202 |
|      1344251938 |
+-----------------+

score 0 · Accepted Answer

每秒速率 = RPS

避免在您的辅助实例上出现 OOM 的建议，

HTOP附加讨论实现睡眠任务减少

显示完整进程列表指示 event_scheduler 为“开启”并且似乎始终处于空闲状态。请发布 SHOW GLOBAL STATUS LIKE 'com%event%' 的结果；确认。

ulimit -a 报告指示打开文件限制为 1024，但 MySQL 实例要求 open_files_limit 为 10,000。需要额外讨论。

在您的 SECONDARY my.cnf [mysqld] 部分中，考虑这些可能的值以提高性能，

read_rnd_buffer_size=128K  # from 256K to reduce handler_read_rnd_next RPS of 139
max_connections=64  # from default of 151 since max_used_connections was 19 in 2+ days
innodb_lru_scan_depth=100  # from 1024 to conserve 90% of cpu cycles used for function
innodb_log_buffer_size=96M  # from 16M for ~ 30 minutes bufffering before write to log
innodb_change_buffer_max_size=50  # from 25 (percent) to reduce time required to LOAD data to tables
innodb_concurrency_tickets=20000  # from 5000 to reduce reque by extending time allowed for inserts

这只是改善您的响应时间和数据吞吐量的开始。

mysql - InnoDB 集群内存问题与 LOAD DATA

1 回答 1

Related

Reference