elasticsearch - packetbeat is not catching large http POST packets

Question

i'm using packetbeat along with ELK (Elasticsearch, Logstash, Kibana) to monitor my server http packets and it is working great however i noticed that packetbeat is not capturing large HTTP POST requests below is my packetbeat.yml file

interfaces:
  device: any

############################# Protocols #######################################
protocols:
  dns:
    ports: [53]
    include_authorities: true
    include_additionals: true
  http:
    ports: [80, 8080, 8000, 5000, 8002]
  memcache:
    ports: [11211]
  mysql:
    ports: [3306]
  pgsql:
    ports: [5432]
  redis:
    ports: [6379]
  thrift:
    ports: [9090]
  mongodb:
    ports: [27017]

output:
  ### Logstash as output
  logstash:
    # The Logstash hosts
    hosts: ["localhost:5044"]


############################# Shipper #########################################

shipper:
############################# Logging #########################################
logging:

  files:
    rotateeverybytes: 10485760 # = 10MB

Answer 1

If the HTTP stream becomes larger than ~10MB then Packetbeat drops it.

The max size is defined in a the TCP_MAX_DATA_IN_STREAM constant which cannot be changed at runtime.

If you enable debug for Packetbeat you should be able to confirm that the stream is being dropped by looking for "Stream data too large, dropping TCP stream" in your logs. Run Packetbeat with -d "http" to enable this debug output.