php - rsyslog 性能优化

Question

应该如何设置rsyslog以获得最佳性能？

我们可以允许某些项目在服务器崩溃时丢失或只是丢失。
我们将把日志保存到 MySQL DB。
我们希望能够以 0.001 - 0.005 秒的延迟处理每秒至少 100 次日志写入。
我们正在从 PHP 应用程序写入日志。

谢谢你的帮助。

score 7 · Accepted Answer

我们刚刚使用 MongoDB 作为数据库进行了类似的练习，因此我将记录我们所做的事情并希望它对您有所帮助。

这是我们第一次使用 rsyslog，因此需要花费一些精力来找到正确的文档并将所有内容拼凑在一起。最后，我们的测试驱动程序（我们使用 SoapUI）能够通过使用 rsyslog 编写事务摘要记录的 php Web 服务获得 1000 TPS。

我们发现以下文章让我们开始：

概述是您将启用 rsyslog 的队列基础结构，以便在守护进程的内存队列已满时将传入消息写入磁盘。在我们的例子中，我们启用了 $ActionQueueSaveOnShutdown，听起来您不需要。然后，您将配置 rsyslog 规则集以解析传入消息并将它们传递给 MySQL 的输出处理程序。最后，您的 php 脚本将使用 openlog() 和 syslog() 来写入您想要记录的任何数据。哦，我们还必须从源代码编译 rsyslog，以启用 json/mongo 插件，这本身就是一个练习。我们在 Ubuntu 12.04 上使用 rsyslog 7.4.5。

我当然不是 rsyslog 方面的专家，但是可以为您提供我们的配置文件和代码作为起点。同样，它们适用于 MongoDB，希望它能让您了解该做什么以及在哪里为您的实现进行更改。

祝你好运！

/etc/rsyslog.conf：

$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)

# Load modules for MongoDB integration: json parser and MongoDB output driver
module(load="mmjsonparse")
module(load="ommongodb")

# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Filter duplicated messages
$RepeatedMsgReduction on

# Set the default permissions for all log files.
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog

# Where to place spool files
$WorkDirectory /var/spool/rsyslog

# use queue to decouple the db writes from default message handling
# From http://www.rsyslog.com/doc/rsyslog_high_database_rate.html
$MainMsgQueueFileName mainq     # set file name for main queue, also enables disk mode
$ActionQueueType LinkedList     # use asynchronous processing
$ActionQueueFileName mongodbq   # set file name for mongo db queue, enables disk mode
$ActionResumeRetryCount -1      # infinite retries on insert failure
$ActionQueueSaveOnShutdown on   # write all queue data to disk when rsyslogd is
                                #   terminated (default is off)

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

/etc/rsyslog.d/10-mongo.conf：

input(type="imuxsock" socket="/dev/log")

template(name="mongodblocal" type="subtree" subtree="$!")

# use json parser for all "local0" facility messages, 
# if parsed successfully run the template to load the
# message into the MongoDB database.
if $syslogfacility-text == 'local0' then {
        action(type="mmjsonparse")
        if $parsesuccess == "OK" then {
                # set some local vars that are appended onto the 
                # document that's written to MongoDB
                set $!time = $timestamp;
                set $!sys = $hostname;
                set $!procid = $syslogtag;
                set $!syslog_fac = $syslogfacility;
                set $!syslog_sever = $syslogpriority;
                set $!pid = $procid;
                action(type="ommongodb" server="127.0.0.1" db="test" collection="syslog" template="mongodblocal")
        }
}

/etc/rsyslog.d/50-default.conf：注意：这会禁用“local0”消息以进行默认处理。

# First some standard log files.  Log by facility.
auth,authpriv.*         /var/log/auth.log

# don't write "local0" messages to syslog, 
# as they're processed using ommongodb (see 10-mongo.conf)
*.*;local0,auth,authpriv.none   -/var/log/syslog

kern.*              -/var/log/kern.log
mail.*              -/var/log/mail.log

# Logging for the mail system.  Split it up so that
# it is easy to write scripts to parse these files.
mail.err            /var/log/mail.err

# Logging for INN news system.
news.crit           /var/log/news/news.crit
news.err            /var/log/news/news.err
news.notice         -/var/log/news/news.notice

# Emergencies are sent to everybody logged in.
*.emerg                                :omusrmsg:*

php web 服务相关调用：

// open syslog, include the process ID and open the connection to the logger 
// immediately, and use a user defined logging mechanism Local0
openlog($SCRIPT_NAME, LOG_PID | LOG_NDELAY, LOG_LOCAL0);
// note: calling closelog() is optional, and we don't use it

...
// construct $doc, which is what will be logged, change this as appropriate
// for your implementation; here $ary_headers is the request's HTTP headers,
// and $request/$response are what was posted/returned
$doc = array("headers" => $ary_headers
            ,"request" => $request
            ,"response" => $response
            );
...

// write the log entry to syslog, where it queues it and writes it to MongoDB
// NOTE: need the '@cee: ' prefix so the rsyslog json parser will process it
// See:  http://www.rsyslog.com/doc/rsyslog_conf_modules.html/mmjsonparse.html

// JSON_BIGINT_AS_STRING = Encodes large integers as their original string value.
// JSON_NUMERIC_CHECK = Encodes numeric strings as numbers.
// JSON_UNESCAPED_SLASHES = Don't escape "/".

syslog(LOG_INFO, '@cee: ' . json_encode($doc, JSON_BIGINT_AS_STRING | JSON_NUMERIC_CHECK | JSON_UNESCAPED_SLASHES));

score 1 · Accepted Answer

这可能会有所帮助，即使它是关于 Elasticsearch，而不是 MySQL：http: //blog.sematext.com/2014/01/20/rsyslog-8-1-elasticsearch-output-performance/

但它解释了 rsyslog 如何与队列和线程一起工作的基础知识，并且您可以开始使用一种新型配置。加上将您发送到 rsyslog 有关队列、规则集等的文档的链接。

有关新型配置格式的更多信息，这是一个很好的资源：http ://www.rsyslog.com/doc/rainerscript.html

php - rsyslog 性能优化

2 回答 2

Related

Reference