i have a problem using Graylog, i've been trying to ingest plain log file which contains lines of messages; However, i found out that Graylog not consuming log files content completely; I.e. Graylog only consume 800 from 1000 lines of messages in a log file;
I took a look on every component's log and found no error message;
I checked via web console, everything is fine with green condition (Elasticsearch shard)
Below is my setup in a single node machine
Machine Info
- CentOS release 6.7 (Final)
- CPU core : 3
- Memory Total : 12GB
However, there are more than one application running in this machine, so i have limited resource here, which is probably fine, since i was trying to setup Graylog for POC purpose;
Application workflow
There will be about 60 remote machine clients and each of them will generate log files (about 24MB / file / hour), this log file contains message that need to be ingested by Graylog for Support service needs.
As for production mode, i thought about custom collector using Nifi, fetching log files via ftp to our Graylog server. For POC, we will be using single machine for deploying Graylog core and web UI, MongoDB and ElasticSearch and we only need to simulate ingesting log files that already reside in Graylog server's local directory; I am using NX Log to consume the file and throw GELF as the output to be processed later by Graylog core.
The log files suppose to be ingested quickly (real time) by Graylog, however, as i stated previously, Graylog seems like stopped in the middle of task, this issue happened before and i just restart everything and Graylog resumed its task, unfortunately i didn't check the completeness.
Graylog's configuration files
elasticsearch.yml : http://pastebin.com/eerR9LNb
nxlog.conf : http://pastebin.com/cN20rFnr
server.conf : http://pastebin.com/Nqp44wBH
collector-sidecar.yml : http://pastebin.com/YkbX4Mh9
Graylog's configuration setup via web console
Input
Raw / Plain GELF UDP Receive buffer size 1048576 Decompressed size limit 8388608
Collector sidecar configuration
I added Input and Output for NX Logs here, and only config dependent variables like URL, port, input file format (regex), etc, and i didn't touch any checklist box
Decorator
My partner used decorator with format string +${ORIG_ADDRESS_ADDRESS} and target field ORIG_ADDRESS_ADDRESS
Extractor
Again, my partner used an extractor, and when he realized about this current issue, he removed the extractor, and found that this issue still happened