ElasticSearch, LogStash, and Kibana - Beginners guide | Beaver

This is part 1 of my ElasticSearch, LogStash, and Kibana - Beginners guide.

What is Beaver? Beaver is very simple and straightforward. You tell it files to look at and it send off any new data added to those files to Redis.

Why Beaver? Yes, LogStash can do this as well. It can monitor files and send them to redis. But from my experience, this doesn't work the best. It can cause duplicates or even miss some items from my testing with a large amount of files (100+). With Beaver, you get a more linear flow of data as well (beaver -> redis -> logstash -> elasticsearch)


Installing:
The beaver install uses the python pip method.

If you don't have python-pip installed, do this first: apt-get install python-pip
Else do this to install Beaver: pip install Beaver

And that's it. If this doesn't work for some reason, the install method could have changed. Check their link out for the current install instructions.



Configure:

There are a lot of options to beaver you can see HERE at the projects page. Below is just a basix set up to get it working though. Paste all of the below in to a beaver.conf file somewhere. I do most of my work in the /home/logstash directory.

[beaver]
transport: redis # Tell it you are using redis
redis_url: redis://10.60.0.82:6379 # Give it the IP to your redis server (can be local too; 6379 is standard redis port)
redis_namespace: logstash-data # This is an arbitrary name for your data in redis.
logstash_version: 1

[/home/logstash/LOGS/LogstashQueue/LOG1*] # This is what or where to look for text files, wildcard use is standard
type: log_line # The type is arbitrary
tags: es1_redis,Inbound # Optional tags to add to your data, can come in handy in ElasticSearch later on

[/home/logstash/LOGS/LogstashQueue/LOG2*] # This is just another set of data to look for
type: log_line
tags: es1_redis,Outbound



And that's the end of the config file. To start up beaver, it's simple. Simply run the command "beaver -c /path/to/beaver.conf".

Now beaver does run in the foreground. If you want it to run in the background even after you logout, do the below steps.
beaver -c /path/to/beaver.conf
[Press ctrl+z ]
disown -h %1 # %1 if it was job 1, it will tell you otherwise
bg 1 # Use whatever number you did above

And that's it for beaver.