ElasticSearch, LogStash, and Kibana - Beginners guide | Redis

Part 2 of ElasticSearch, LogStash, and Kibana - Beginners guide 

What is Redis and what is it's part in the ELK life flow? I won't go deep in to this but it's best to think of Redis as a queue manager. Beaver sends log data to redis. Redis queues it up in a....queue. From there, logstash can read lines out of redis at it's own pace. And many different logstash's running on other servers can all do this at once.

Installing:
Just this: apt-get install redis-server

Configuring:

There is a lot you can do with Redis, but again here are the bsics to get it working for ELK.

This is what is in my /etc/redis/redis.conf file:

daemonize yes
pidfile /var/run/redis/redis-server.pid
port 6379
bind 10.100.10.14  # Ip address of the server
timeout 0
loglevel notice
logfile /var/log/redis/redis-server.log
databases 16
rdbcompression yes
dbfilename dump.rdb
dir /var/lib/redis
slave-serve-stale-data yes
appendonly no
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
slowlog-log-slower-than 10000
slowlog-max-len 128
vm-enabled no
vm-swap-file /var/lib/redis/redis.swap
vm-max-memory 0
vm-page-size 32
vm-pages 134217728
vm-max-threads 4
hash-max-zipmap-entries 512
hash-max-zipmap-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes

The main thing to note is make sure you have the right IP address in there. I also commented out the "save" lines. These options save the redis database after a certain amount of time or certain amount of changes. This was causing some delays for me due to the amount of data. The world woudn't end if I lost the redis database (since logstash could keep up for the most part anyways) so I disabled them all.


Some useful redis commands:
Start and stop it like normal: service redis-server [start, stop, restart, status]

Redis also has a cli you can use interactively or via commands. Here two useful commands..

This one lets you see how many items are queued in the redis-namespace you configured back when setting up Beaver. It returns the number of items queued.
redis-cli -h 10.100.10.14 llen logstash-data



This one lets you see what's happening live with the nodes communicating with redis.
redis-cli -h 10.100.10.14 monitor