Use of ELK - Filebeat collects logs for Logstash

Why use Filebeat to collect logs for Logstash

Logstash itself can also collect logs, but only take up more cups and memory, not secure transmission, is the framework level . Filter function filtering analysis, strong filtering ability, and a wide range of uses.

Filebeat collecting logs is a lot of CPU and memory for Logstash, performance is much better for Logstash, collecting logs faster than Logstash.

Even though Logstash can both filter and collect logs, we consider resource consumption and performance issues, and decisively choose to use Filebeat to collect logs from Logstash.

That, Filebeat can replace Logstash? It must be impossible. Filebeat's filtering function is weaker than Logstash. Although Filebeat is used to collect logs, filtering is still more powerful.

Specific operation:

1, first install Filebeat.

2, configure the logstash.conf file

input {
     beats {
    port => 5041  // port number entered by the configuration file. }
}

Output {
  Stdout {
   Codec => rubydebug
  }

  Elasticsearch {
    Hosts =>"elasticsearch ip:9200"index =>"log-%{+YYYY.MM.dd}"


  }
}

3, configure the filebeat.yml file

at the beginning to configure the location of the *.log.

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled andfetched. Glob based paths.
  Paths:
    - /var/log/*.log
    / / The specific location of the log. 
output.logstash:
  # The Logstash hosts
  hosts: ["Logstash IP:5041"]//5041 The port number should be the same as the port number entered by logstash. 

4, execute the command to test Execute

sudo ./filebeat -e -c filebeat.yml -d "publish"

in the directory where the filebeat is installed. Execute

bin/logstash -f logstash.conf

A in the logstash installation directory, create a new T.log file in the /var/log directory and write "tw".

a, filebeat output information is

2018-09-16T16:11:02.533+0800
INFO
log/harvester.go:251    
Harvester started for file: /var/log/1.log

b, logstash output information is

这里写图片描述

through the above configuration, Filebeat collection log is thrown to Logstash to filter, and then thrown to Elasticsearch.

ELK uses Filebeat to collect logs from Logstash, success!