1

好的,主题行的明显答案是:呃!这是70GB,请耐心等待。但听我说完。

我用来mysqlimport快速加载数据文件(mysql标准制表符分隔格式)。完成 mysqlimport 命令大约需要 4 个小时。在此期间,磁盘以容量运行(约 50MB/秒磁盘活动)

但是它完成后又过了 5 个小时,我仍然无法运行简单的查询,例如

 select * from my_big_table limit 1

当我运行该查询时,它只会(似乎)无限期地阻塞。在没有运行查询的情况下,我看到了磁盘活动(一个 mysql 进程运行 3MB/秒的读写,另一个运行几 MB/秒的读取活动),自从我运行mysqlimport. 我还看到了一个 mysql 线程从那以后运行得接近 100% mysqlimport

我的猜测是 mysql 正在后台处理索引等(150M 记录,InnoDB),这就是阻碍我查询表的能力的原因。但我不知道如何验证或查看正在进行的活动。或者对它可能完成的时间进行任何估计


多余的细节:

我敢肯定有人会质疑为什么一个 70GB 的表会进入 mysql。它是 Web 应用程序中使用的只读数据表。它仅在 上建立索引,id并且查询只会在一组有限的 ID 上(没有对该表的范围查询),只是在id列上连接并直接在id列上查询。该表在一个大型 Hadoop 作业中更新,并且将使用 mysqlimport 每晚更新以提高效率。


更新: MySQL 在 70GB 导入后最终崩溃,我在错误日志中看不到太多。我已将表更改为 myisam 引擎并再次尝试导入。

错误日志:

Version: '5.5.28-0ubuntu0.12.04.2-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)
121209 18:41:07 [Warning] IP address '10.0.4.160' could not be resolved: Name or service not known
121217  1:54:16 [Note] Plugin 'FEDERATED' is disabled.
121217  1:54:16 InnoDB: The InnoDB memory heap is disabled
121217  1:54:16 InnoDB: Mutexes and rw_locks use GCC atomic builtins
121217  1:54:16 InnoDB: Compressed tables use zlib 1.2.3.4
121217  1:54:16 InnoDB: Initializing buffer pool, size = 11.7G
121217  1:54:18 InnoDB: Completed initialization of buffer pool
121217  1:54:18 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 217624804600
121217  1:54:18  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 217630047232
InnoDB: Doing recovery: scanned up to log sequence number 217635290112
InnoDB: Doing recovery: scanned up to log sequence number 217640532992
InnoDB: Doing recovery: scanned up to log sequence number 217645775872
InnoDB: Doing recovery: scanned up to log sequence number 217651018752
InnoDB: Doing recovery: scanned up to log sequence number 217656261632
InnoDB: Doing recovery: scanned up to log sequence number 217661504512
InnoDB: Doing recovery: scanned up to log sequence number 217666747392
InnoDB: Doing recovery: scanned up to log sequence number 217671990272
InnoDB: Doing recovery: scanned up to log sequence number 217677233152
InnoDB: Doing recovery: scanned up to log sequence number 217682476032
InnoDB: Doing recovery: scanned up to log sequence number 217687718912
InnoDB: Doing recovery: scanned up to log sequence number 217692961792
InnoDB: Doing recovery: scanned up to log sequence number 217698204672
InnoDB: Doing recovery: scanned up to log sequence number 217703447552
InnoDB: Doing recovery: scanned up to log sequence number 217708690432
InnoDB: Doing recovery: scanned up to log sequence number 217713933312
InnoDB: Doing recovery: scanned up to log sequence number 217719176192
InnoDB: Doing recovery: scanned up to log sequence number 217724419072
InnoDB: Doing recovery: scanned up to log sequence number 217729661952
InnoDB: Doing recovery: scanned up to log sequence number 217734904832
InnoDB: Doing recovery: scanned up to log sequence number 217740147712
InnoDB: Doing recovery: scanned up to log sequence number 217745390592
InnoDB: Doing recovery: scanned up to log sequence number 217750633472
InnoDB: Doing recovery: scanned up to log sequence number 217755876352
InnoDB: Doing recovery: scanned up to log sequence number 217761119232
InnoDB: Doing recovery: scanned up to log sequence number 217766362112
InnoDB: Doing recovery: scanned up to log sequence number 217771604992
InnoDB: Doing recovery: scanned up to log sequence number 217776847872
InnoDB: Doing recovery: scanned up to log sequence number 217782090752
InnoDB: Doing recovery: scanned up to log sequence number 217787333632
InnoDB: Doing recovery: scanned up to log sequence number 217792576512
InnoDB: Doing recovery: scanned up to log sequence number 217797819392
InnoDB: Doing recovery: scanned up to log sequence number 217803062272
InnoDB: Doing recovery: scanned up to log sequence number 217808305152
InnoDB: Doing recovery: scanned up to log sequence number 217813548032
InnoDB: Doing recovery: scanned up to log sequence number 217818790912
InnoDB: Doing recovery: scanned up to log sequence number 217824033792
InnoDB: Doing recovery: scanned up to log sequence number 217829276672
InnoDB: Doing recovery: scanned up to log sequence number 217834519552
InnoDB: Doing recovery: scanned up to log sequence number 217839762432
InnoDB: Doing recovery: scanned up to log sequence number 217845005312
InnoDB: Doing recovery: scanned up to log sequence number 217850248192
InnoDB: Doing recovery: scanned up to log sequence number 217855491072
InnoDB: Doing recovery: scanned up to log sequence number 217860733952
InnoDB: Doing recovery: scanned up to log sequence number 217865976832
InnoDB: Doing recovery: scanned up to log sequence number 217871219712
InnoDB: Doing recovery: scanned up to log sequence number 217876462592
InnoDB: Doing recovery: scanned up to log sequence number 217881705472
InnoDB: Doing recovery: scanned up to log sequence number 217886948352
InnoDB: Doing recovery: scanned up to log sequence number 217892191232
InnoDB: Doing recovery: scanned up to log sequence number 217897434112
InnoDB: Doing recovery: scanned up to log sequence number 217902676992
InnoDB: Doing recovery: scanned up to log sequence number 217907919872
InnoDB: Doing recovery: scanned up to log sequence number 217913162752
InnoDB: Doing recovery: scanned up to log sequence number 217918405632
InnoDB: Doing recovery: scanned up to log sequence number 217923648512
InnoDB: Doing recovery: scanned up to log sequence number 217928891392
InnoDB: Doing recovery: scanned up to log sequence number 217934134272
InnoDB: Doing recovery: scanned up to log sequence number 217939377152
InnoDB: Doing recovery: scanned up to log sequence number 217944620032
InnoDB: Doing recovery: scanned up to log sequence number 217949862912
InnoDB: Doing recovery: scanned up to log sequence number 217955105792
InnoDB: Doing recovery: scanned up to log sequence number 217960348672
InnoDB: Doing recovery: scanned up to log sequence number 217965591552
InnoDB: Doing recovery: scanned up to log sequence number 217970834432
InnoDB: Doing recovery: scanned up to log sequence number 217976077312
InnoDB: Doing recovery: scanned up to log sequence number 217981320192
InnoDB: Doing recovery: scanned up to log sequence number 217986563072
InnoDB: Doing recovery: scanned up to log sequence number 217991805952
InnoDB: Doing recovery: scanned up to log sequence number 217997048832
InnoDB: Doing recovery: scanned up to log sequence number 218002291712
InnoDB: Doing recovery: scanned up to log sequence number 218007534592
InnoDB: Doing recovery: scanned up to log sequence number 218012777472
InnoDB: Doing recovery: scanned up to log sequence number 218018020352
InnoDB: Doing recovery: scanned up to log sequence number 218023263232
InnoDB: Doing recovery: scanned up to log sequence number 218028506112
InnoDB: Doing recovery: scanned up to log sequence number 218033748992
InnoDB: Doing recovery: scanned up to log sequence number 218038991872
InnoDB: Doing recovery: scanned up to log sequence number 218044234752
InnoDB: Doing recovery: scanned up to log sequence number 218047052128
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 17071570 row operations to undo
InnoDB: Trx id counter is 3D0D200
121217  1:54:32  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position 0 1211503, file name ./frugg-bin-backups.000389
InnoDB: Starting in background the rollback of uncommitted transactions
121217  2:01:34  InnoDB: Rolling back trx with id 3D02544, 17071570 rows to undo

InnoDB: Progress in percents: 1121217  2:01:34  InnoDB: Waiting for the background threads to start
121217  2:01:35 InnoDB: 1.1.8 started; log sequence number 218047052128
121217  2:01:35 [Note] Recovering after a crash using frugg-bin-backups
121217  2:01:35 [Note] Starting crash recovery...
121217  2:01:35 [Note] Crash recovery finished.
121217  2:01:35 [Note] Server hostname (bind-address): '10.0.1.227'; port: 3306
121217  2:01:35 [Note]   - '10.0.1.227' resolves to '10.0.1.227';
121217  2:01:35 [Note] Server socket created on IP: '10.0.1.227'.
121217  2:01:35 [Note] Event Scheduler: Loaded 0 events
121217  2:01:35 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.28-0ubuntu0.12.04.2-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)
121217  2:01:38 [Warning] IP address '10.0.4.160' could not be resolved: Name or service not known
 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100
InnoDB: Rolling back of trx id 3D02544 completed
121217  9:36:40  InnoDB: Rollback of non-prepared transactions completed
121217 16:12:45 [Note] /usr/sbin/mysqld: Normal shutdown

121217 16:12:45 [Note] Event Scheduler: Purging the queue. 0 events
121217 16:12:45  InnoDB: Starting shutdown...
121217 16:12:54  InnoDB: Shutdown completed; log sequence number 230916413935
121217 16:12:54 [Note] /usr/sbin/mysqld: Shutdown complete
4

2 回答 2

0

您只需做一件事,而不是编写 select * from.. 尝试使用

将表的数据分成几个块(暂时在查询中)后联合。由于 union 是集合运算,所以它是并行运算。因此,如果您将表格分为 5 个部分,如我对问题的回答中所示,那么您的数据加载速度至少会快 5 倍。请参见此处

于 2012-12-17T12:39:16.933 回答
0

切换到 MyISAM 后,我能够更快地加载表格。它加载了表,然后在加载后又花了好几个小时才可以运行查询。这主要是由于启用了二进制日志记录,它两次写入 70GB 文件,一次写入表,另一次写入二进制日志。

我还启用了:

[mysqld]
innodb_file_per_table

这使得了解 innodb 发生了什么变得容易得多。

于 2012-12-18T13:17:55.497 回答