1

故事是这样的，我们有一个名为 syslog 的 MySQL 数据库，有以下几个表：

Tables_in_syslog            |
+-----------------------------+
| plugin_camm_syslog          |
| plugin_camm_syslog_MonthDec |
| plugin_camm_syslog_MonthJan |
| plugin_camm_syslog_incoming

表plugin_camm_syslog_incoming有存储引擎 MEMORY，其他的是MyISAM.

这是每月第一天从 cron 运行的脚本：

<?php
# This script rotates the mysql syslog-ng database
# on a monthly basis.   Place in "/etc/cron.monthly".
# To change to weekly/daily, adjust "$date" and place
# in the appropriate cron folder.

$link = mysql_connect("localhost", "syslogadmin", "sysadmin");
if (!$link) {
    die("Could not connect: " . mysql_error());
}
echo "Connected successfully\n";

// make syslog the current db
$db_selected = mysql_select_db("syslog", $link);
if (!$db_selected) {
    die ("Can\"t use syslog : " . mysql_error());
}

// Rename the table, appending short month
$last= mktime(0, 0, 0, date("m")-1, date("d"), date("Y"));
$drop= mktime(0, 0, 0, date("m")-3, date("d"), date("Y"));
$d=date("M", $last); $dd=date("M", $drop);
$date="Month$d"; $dropdate="Month$dd";

// drop old table
// drop 2 month old table
$result=mysql_query("DROP TABLE `plugin_camm_syslog_$dropdate`");
//$result=mysql_query("DROP TABLE `plugin_camm_syslog_$date`");

//make new date
$last= mktime(0, 0, 0, date("m")-1, date("d"), date("Y")); $d=date("M", $last);
$date="Month$d";
#echo "XX $date\n";
#die ("test $result:".mysql_error());

$result = mysql_query("ALTER TABLE `plugin_camm_syslog` RENAME `plugin_camm_syslog_$date`");
if (!$result) { die("Invalid query: " . mysql_error()); }


// Recreate the original table
$result="
    CREATE TABLE `plugin_camm_syslog` (
        `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
        `host` varchar(128) DEFAULT NULL,
        `sourceip` varchar(45) NOT NULL,
        `facility` varchar(10) DEFAULT NULL,  
        `priority` varchar(10) DEFAULT NULL,
        `sys_date` datetime DEFAULT NULL,
        `message` text,
        `status` tinyint(4) NOT NULL DEFAULT '0', 
        `alert` smallint(3) unsigned NOT NULL DEFAULT '0',
        PRIMARY KEY (`id`,`sys_date` ),
        KEY `facility` (`facility`),
        KEY `priority` (`priority`),
        KEY `sourceip` (`sourceip`),
        KEY `status` (`status`),
        KEY `alert` (`alert`),
        KEY `status_date` (`status`,`sys_date`),
        KEY `sys_date` (`sys_date`)
    ) ENGINE=MyISAM 
    DEFAULT CHARSET=latin1
    COMMENT='camm plugin SYSLOG Data' 
    PARTITION BY RANGE ( dayofyear(sys_date)) (
        PARTITION syslog_day0 VALUES LESS THAN (0) ENGINE = MyISAM,
        PARTITION syslog_day14 VALUES LESS THAN (14) ENGINE = MyISAM,
        PARTITION syslog_day28 VALUES LESS THAN (28) ENGINE = MyISAM,
        PARTITION syslog_day42 VALUES LESS THAN (42) ENGINE = MyISAM,
        PARTITION syslog_day56 VALUES LESS THAN (56) ENGINE = MyISAM,
        PARTITION syslog_day70 VALUES LESS THAN (70) ENGINE = MyISAM, 
        PARTITION syslog_day84 VALUES LESS THAN (84) ENGINE = MyISAM, 
        PARTITION syslog_day98 VALUES LESS THAN (98) ENGINE = MyISAM, 
        PARTITION syslog_day112 VALUES LESS THAN (112) ENGINE = MyISAM, 
        PARTITION syslog_day126 VALUES LESS THAN (126) ENGINE = MyISAM, 
        PARTITION syslog_day140 VALUES LESS THAN (140) ENGINE = MyISAM, 
        PARTITION syslog_day154 VALUES LESS THAN (154) ENGINE = MyISAM, 
        PARTITION syslog_day168 VALUES LESS THAN (168) ENGINE = MyISAM, 
        PARTITION syslog_day182 VALUES LESS THAN (182) ENGINE = MyISAM, 
        PARTITION syslog_day196 VALUES LESS THAN (196) ENGINE = MyISAM, 
        PARTITION syslog_day210 VALUES LESS THAN (210) ENGINE = MyISAM, 
        PARTITION syslog_day224 VALUES LESS THAN (224) ENGINE = MyISAM, 
        PARTITION syslog_day238 VALUES LESS THAN (238) ENGINE = MyISAM, 
        PARTITION syslog_day252 VALUES LESS THAN (252) ENGINE = MyISAM, 
        PARTITION syslog_day266 VALUES LESS THAN (266) ENGINE = MyISAM, 
        PARTITION syslog_day280 VALUES LESS THAN (280) ENGINE = MyISAM, 
        PARTITION syslog_day294 VALUES LESS THAN (294) ENGINE = MyISAM, 
        PARTITION syslog_day308 VALUES LESS THAN (308) ENGINE = MyISAM, 
        PARTITION syslog_day322 VALUES LESS THAN (322) ENGINE = MyISAM, 
        PARTITION syslog_day336 VALUES LESS THAN (336) ENGINE = MyISAM, 
        PARTITION syslog_day350 VALUES LESS THAN (350) ENGINE = MyISAM, 
        PARTITION syslog_day364 VALUES LESS THAN (364) ENGINE = MyISAM, 
        PARTITION syslog_day369 VALUES LESS THAN (369) ENGINE = MyISAM
    )
";

if (mysql_query($result)) {
    echo "success in plugin-camm table creation.\n";
}
else {
    echo "no table created.\n";
}

if ( mysql_query($result)) {
    echo "success in table creation.\n";
} else {
    echo "no table created.\n";
}

mysql_close($link);


$link = mysql_connect("localhost", "cacxt", "cacxt");
if (!$link) {
    die("Could not connect: " . mysql_error());
}
echo "Connected successfully\n";

$db_selected = mysql_select_db("cacti", $link);
if (!$db_selected) {
    die ("Can\"t use cacti : " . mysql_error());
}

$result=mysql_query("TRUNCATE TABLE `plugin_camm_tree2`");
$result=mysql_query("TRUNCATE TABLE `plugin_camm_temp`");
$result=mysql_query("TRUNCATE TABLE `plugin_camm_keys`");
$result=mysql_query("UPDATE `plugin_camm_rule` SET `actual_triggered` = '0';");
mysql_close($link);
?>

问题是服务器无法再处理这么多的数据，并且每个月大约有 20 天服务器面临性能问题。我不是 MySQL 忍者，但试图了解发生了什么，似乎索引填充太大 [服务器可以处理的更大]

看：

:/var/lib/mysql/syslog# ls -lh | grep G total 21G
-rw-rw---- 1 mysql mysql 2,8G 2012-12-15 00:50 plugin_camm_syslog_MonthDec#P#syslog_day350.MYD
-rw-rw---- 1 mysql mysql 1,5G 2013-01-31 14:12 plugin_camm_syslog_MonthDec#P#syslog_day350.MYI
-rw-rw---- 1 mysql mysql 1,6G 2012-12-29 00:46 plugin_camm_syslog_MonthDec#P#syslog_day364.MYD
-rw-rw---- 1 mysql mysql 2,0G 2013-01-14 00:46 plugin_camm_syslog_MonthJan#P#syslog_day14.MYD
-rw-rw---- 1 mysql mysql 1,1G 2013-01-28 08:01 plugin_camm_syslog_MonthJan#P#syslog_day14.MYI
-rw-rw---- 1 mysql mysql 2,1G 2013-01-28 05:41 plugin_camm_syslog_MonthJan#P#syslog_day28.MYD
-rw-rw---- 1 mysql mysql 1,1G 2013-01-28 08:01 plugin_camm_syslog_MonthJan#P#syslog_day28.MYI
-rw-rw---- 1 mysql mysql 1,7G 2013-02-11 00:46 plugin_camm_syslog#P#syslog_day42.MYD
-rw-rw---- 1 mysql mysql 1,7G 2013-02-21 07:29 plugin_camm_syslog#P#syslog_day56.MYD

今天我需要运行脚本来轮换系统日志，我的问题是它会保留我 1 月 / 2 月的数据吗？

或者有人可以建议如何减少索引使用？

4

1 回答 1

0

也不是 mysql 忍者，但您的代码为我提出了一些标志：

  • 您使用所有这些索引的频率如何，您可以减少索引的数量吗？

    • sourceip、facility 和priority 可以放在一个单独的表中，并且只将ids 存储在plugin_camm_syslog 表中吗？（我相信这会节省很多空间）

    • 使用如何在 MySQL 中处理大表的建议 怎么样？, 将数据拆分到多个表中，例如每日表

关于操作的日志记录请看这里：https ://serverfault.com/questions/71071/how-to-enable-mysql-logging

于 2013-02-21T09:40:11.930 回答