我正在使用具有 3 个节点的 RMQ 高可用集群,版本:3.8.3
规格:
- 内存:4GB
- CPU : 2CPU
间歇性地出现以下错误,并且一些节点因内存警报而崩溃。
Application mnesia exited with reason: stopped
wal: encountered error during recovery: badarg
完整的日志条目:
**********************************************************
*** Publishers will be blocked until this alarm clears ***
**********************************************************
2020-07-14 01:13:00.914 [warning] <0.328.0> rabbit_sysmon_handler busy_dist_port <0.456.0> [{name,rabbit_alarm},{initial_call,{gen_event,init_it,6}},{erlang,bif_return_trap,2},{message_queue_len,0}] {#Port<0.968>,unknown}
2020-07-14 01:13:02.838 [warning] <0.328.0> rabbit_sysmon_handler busy_dist_port <0.684.0> [{initial_call,{rabbit_prequeue,init,1}},{erts_internal,dsend_continue_trap,1},{message_queue_len,0}] {#Port<0.968>,unknown}
2020-07-14 01:31:34.457 [info] <0.8.0> Log file opened with Lager
2020-07-14 01:31:37.799 [info] <0.8.0> Feature flags: list of feature flags found:
2020-07-14 01:31:37.799 [info] <0.8.0> Feature flags: [x] drop_unroutable_metric
2020-07-14 01:31:37.799 [info] <0.8.0> Feature flags: [x] empty_basic_get_metric
2020-07-14 01:31:37.799 [info] <0.8.0> Feature flags: [x] implicit_default_bindings
2020-07-14 01:31:37.799 [info] <0.8.0> Feature flags: [x] quorum_queue
2020-07-14 01:31:37.800 [info] <0.8.0> Feature flags: [x] virtual_host_metadata
2020-07-14 01:31:37.800 [info] <0.8.0> Feature flags: feature flag states written to disk: yes
2020-07-14 01:31:37.910 [info] <0.43.0> Application mnesia exited with reason: stopped
2020-07-14 01:31:38.072 [info] <0.395.0> ra: meta data store initialised. 0 record(s) recovered
2020-07-14 01:31:38.072 [info] <0.402.0> WAL: recovering ["/var/lib/rabbitmq/mnesia/rabbit@rmq-3/quorum/rabbit@rmq-3/00000058.wal"]
2020-07-14 01:31:38.518 [warning] <0.402.0> wal: encountered error during recovery: badarg
在这个时候我能够看到系统iowait很高,
而且我能够看到高 TCP 错误
这可能是什么原因?
任何帮助将不胜感激。
谢谢。