Databases

How to build Redis Cluster on CentOS 7 using Sentinels

redis

Redis (a.k.a. Remote Directory Server) is in-memory database. It stores data in key-value pairs. As it lies in memory, it is blazing fast and an excellent choice for caching. Compare to Memcached, Redis supports many advanced datatypes. It can also be used as the distributed message broker between different components of system.

In this article, we will see how to build Redis cluster for high availability using master-slave setup. Setup will be finished in 2 steps.

  1. Master-Slave Setup – replicate content of master on slaves.
  2. Failover Setup using Sentinels – when master goes down, promote automatically one of the slave as master

Tutorial Environment

For this tutorial, let’s assume 3 virtual servers having below IPv4 addresses.

  • 192.168.10.10 master
  • 192.168.10.11 slave-1
  • 192.168.10.12 slave-2

Standard Redis port is 6379. To differentiate easily between master and slaves, we will run master on 6379 port while slave -1 on 6380 and slave-2 on 6381.

This tutorial is tested on CentOS-7 .

Redis master-slave setup

Redis Installation

First install Redis from source using below commands on all hosts. At this moment, 5.0.5 is the latest stable version. Visit to see the latest versions and change download URL accordingly.

yum update -y
yum groupinstall -y 'Development Tools'
yum install -y wget
cd /opt
wget download.redis.io/releases/redis-5.0.5.tar.gz
tar -xf redis-5.0.5.tar.gz
cd redis-5.0.5
make
make install

Now lets create required directories.

mkdir -p /etc/redis /var/run/redis /var/log/redis /var/redis/
cp redis.conf redis.conf.bak
cp redis.conf /etc/redis/redis.conf

Also add less-privileged user to run Redis daemon.

adduser redis -M -g daemon
passwd -l redis
chown -R redis:daemon /opt/redis-5.0.5
chown -R redis:daemon /var/run/redis
chown -R redis:daemon /var/log/redis
chown -R redis:daemon /var/redis/

Repeat these steps on all hosts.

Master Instance Configuration

Open /etc/redis/redis.conf in your favorite editor and change values of below options as mentioned.

bind 192.168.10.10
port 6379
daemonize yes
pidfile "/var/run/redis/redis.pid"
logfile "/var/log/redis/redis.log"
dir "/var/redis/"

We will add same config on slaves too except few changes.

Slave-1 Instance Configuration

Open /etc/redis/redis.conf in your favorite editor and change values of below options as mentioned.

bind 192.168.10.11
port 6380
daemonize yes
pidfile "/var/run/redis/redis.pid"
logfile "/var/log/redis/redis.log"
dir "/var/redis/"
slaveof 192.168.10.10 6379

Slave-2 Instance Configuration

Open /etc/redis/redis.conf in your favorite editor and change values of below options as mentioned.

bind 192.168.10.12
port 6381
daemonize yes
pidfile "/var/run/redis/redis.pid"
logfile "/var/log/redis/redis.log"
dir "/var/redis/"
slaveof 192.168.10.10 6379

Systemd setup

Systemd is default init system for CentOS 7. Lets add unit file to manage redis-service using systemd. Open /etc/systemd/system/redis.service in your favorite editor and add following content.

[Unit]
Description=Redis 
After=network.target

[Service]
Type=forking
User=redis
Group=daemon
ExecStart=/usr/local/bin/redis-server /etc/redis/redis.conf
ExecStop=/usr/local/bin/redis-cli shutdown
Restart=always

[Install]
WantedBy=multi-user.target

Now start redis-service and make it runnable on reboot automatically.

systemctl start redis.service
systemctl enable redis.service

Setup unit file on all hosts and start service.

Test replication setup

To test replication, first connect to master instance using redis-cli and create a new key using below command.

192.168.10.10:6379>  set foo bar

Now connect to slave-1 using below command and see if newly added key is present or not. Here we are specifying port as it is not standard.

# redis-cli -p 6380
192.168.10.11:6380>  get foo
"bar"

If you don’t get “bar” in output, replication is not setup properly. Repeat similar test on slave-2.

# redis-cli -p 6381
192.168.10.12:6381>  get foo
"bar"

Awesome ! Our master-slave setup is ready. Data is replicating. Now lets move on to Failover Setup.

Failover Setup using Sentinels

Redis provides a feature called Sentinel to monitor redis instances. These sentinels are deployed across cluster. If master goes down, sentinels promote one of the slave as master. Standard port for sentinel service is 16379. Similar to main service, we will run slave-sentinel services on non-standard ports to differentiate them easily. So master-sentinel service will listen on port 16379 while slave-1-sentinel service on port 16380 and slave-2-sentinel service on port 16381.

Master-Sentinel Configuration

Open /etc/redis/sentinel.conf file in your favorite editor and add following content.

bind 192.168.10.10
port 16379
sentinel monitor redis-cluster 192.167.10.10 6379 2
sentinel down-after-milliseconds redis-cluster 5000
sentinel parallel-syncs redis-cluster 1
sentinel failover-timeout redis-cluster 10000
daemonize yes
pidfile "/var/run/redis/sentinel.pid"
dir "/var/redis/"

Slave-1-Sentinel Configuration

Add following content in /etc/redis/sentinel.conf.

bind 192.168.10.11
port 16380
sentinel monitor redis-cluster 192.167.10.10 6379 2
sentinel down-after-milliseconds redis-cluster 5000
sentinel parallel-syncs redis-cluster 1
sentinel failover-timeout redis-cluster 10000
daemonize yes
pidfile "/var/run/redis/sentinel.pid"
dir "/var/redis/"

Slave-2-Sentinel Configuration

Add following content in /etc/redis/sentinel.conf.

bind 192.168.10.12
port 16381
sentinel monitor redis-cluster 192.167.10.10 6379 2
sentinel down-after-milliseconds redis-cluster 5000
sentinel parallel-syncs redis-cluster 1
sentinel failover-timeout redis-cluster 10000
daemonize yes
pidfile "/var/run/redis/sentinel.pid"
dir "/var/redis/"

Systemd Setup

Now lets add unit file to manage sentinel service using systemd. Open /etc/systemd/systemd/sentinel.service file and add following content.

[Unit]
Description=Redis-Sentinel
After=network.target

[Service]
Type=forking
User=redis
Group=daemon
PIDFile=/var/run/redis/sentinel.pid
ExecStart=/usr/local/bin/redis-server /etc/redis/sentinel.conf --sentinel
ExecStop=/bin/kill -s TERM $MAINPID
Restart=always

[Install]
WantedBy=multi-user.target

Now start sentinel-service on all hosts and make it runnable on boot automatically.

systemctl start sentinel.service
systemctl enable sentinel.service

Test failover setup

To test failover, we will force failure of master-instance. First connect to master using redis-cli and create segmentation fault using below command.

192.168.10.10:6379> DEBUG SEGFAULT

Alternatively, without segmentation fault we can initiate failover using below command.

192.168.10.10:6379> SENTINEL failover redis-cluster

After few seconds, you can check for new master.

192.168.10.10:6379> SENTINEL masters

Notes

  • During failover, sentinel service rewrite configuration files. So when you try to up old-master instance again, check configuration file carefully and adjust accordingly.
  • By default redis authentication is not setup. And even it is setup, it uses weak algorithm, so not much ideal for security. So make sure to NOT run redis services on public interface.
  • In same context, cloud providers like AWS provide ephemeral IP addresses. Even you bind redis service to private interface, it can be still accessed over public interface. So make sure to set proper firewall rules.

Leave a Reply