我正在使用 2 个进程池来并行解析多个日志文件,
po = Pool(processes=2)
pool_object = po.apply_async(log_parse, (hostgroup_sender_dir, hostname, host_depot_dir, synced_log, prev_last_pos, get_report_rate), )
(curr_last_pos, remote_report_datetime, report_gen_rate) = pool_object.get()
然而,它在初始运行时相当慢,大约 12 个 ~20Mb 文件约 16 分钟。
考虑到我将每 2 或 3 分钟解析一次日志新字节,在下一次迭代中不会有太大问题,但在第一次运行时肯定还有改进的余地。将日志预先分割成几个较小的拼接(这样 pyparse 就不必将整个日志分配到内存中)会加快速度吗?
我仍在双核开发 VM 上运行它,但很快将不得不迁移到四核物理服务器(我将尝试获得额外的四核 CPU),它可能需要能够管理 ~50日志。
原木的拼接,
log_splice = """
# XX_MAIN (23143) Report at 2011-08-30 20:00:00.003 Type: Periodic #
# Report number 1790 State: Active #
################################################################################
# Running since : 2011-08-12 04:40:06.153 #
# Total execution time : 18 day(s) 15:19:53.850 #
# Last report date : 2011-08-30 19:45:00.002 #
# Time since last periodic report: 0 day(s) 00:15:00.000 #
################################################################################
----------------------------------------------------
| Periodic | Global |
----------------------------|-----------------------|--------------------------|
Simultaneous Accesses | Curr Max Cumulative | Max Cumulative |
--------------------------- | ---- ---- ---------- | ---- ------------- |
Accesses | 1 5 - | 180 - |
- in start/stop state | 1 5 12736 | 180 16314223 |
-------------------------------------------------------------------------------|
Accesses per Second | Max Occurr. Date | Max Occurrence Date |
--------------------------- | ------ -------------- | ------ --------------- |
Accesses per second | 21.00 08-30 19:52:33 | 40.04 08-16 20:19:18 |
-------------------------------------------------------------------------------|
Service Statistics | Success Total % | Success Total % |
--------------------------- | -------- -------- --- | --------- ---------- --- |
Services accepted accesses | 17926 17927 99 | 21635954 21637230 -98 |
- 98: NF | 7546 7546 100 | 10992492 10992492 100 |
- 99: XFC | 10380 10380 100 | 10643462 10643462 100 |
----------------------------------------------------------------------------- |
Services succ. terminations | 12736 12736 100 | 16311566 16314222 99 |
- 98: NF | 7547 7547 100 | 10991401 10992492 99 |
- 99: XFC | 5189 5189 100 | 5320165 5321730 99 |
----------------------------------------------------------------------------- |
"""
使用 pyparse,
unparsed_log_data = input_log.read()
#------------------------------------------------------------------------
# Define Grammars
#------------------------------------------------------------------------
integer = Word( nums )
# XX_MAIN ( 4801) Report at 2010-01-25 06:55:00
binary_name = "# XX_MAIN"
pid = "(" + Word(nums) + ")"
report_id = Suppress(binary_name) + Suppress(pid)
# Word as a contiguous set of characters found in the string nums
year = Word(nums, max=4)
month = Word(nums, max=2)
day = Word(nums, max=2)
# 2010-01-25 grammar
yearly_day_bnf = Combine(year + "-" + month + "-" + day)
# 06:55:00. grammar
clock24h_bnf = Combine(Word(nums, max=2) + ":" + Word(nums, max=2) + ":" + Word(nums, max=2) + Suppress("."))
timestamp_bnf = Combine(yearly_day_bnf + White(' ') + clock24h_bnf)("timestamp")
report_bnf = report_id + Suppress("Report at ") + timestamp_bnf
# Service Statistics | Success Total % |
# Services succ. terminations | 40 40 100 | 3494775 3497059 99 |
partial_report_ignore = Suppress(SkipTo("Services succ. terminations", include=True))
succ_term_bnf = Suppress("|") + integer("succTerms") + integer("totalTerms")
terminations_report_bnf = report_bnf + partial_report_ignore + succ_term_bnf
# Apply the BNF to the unparsed data
terms_parsing = terminations_report_bnf.searchString(unparsed_log_data)