Redis哨兵模式

哨兵模式 官方文档
基于Docker搭建Redis一主两从三哨兵的实现

自动从变主,上面【主从复制】都是为了演示,真正的生产不使用,只有哨兵才是真正要使用的。

因为主从复制,是主写从读模式,主挂了,从就不能进行写入,此时整个集群无主状态。哨兵模式则可以排哨兵节点进行监控,主挂了,就会选出一个从机当做主机,此时整个集群仍可以主写从读,正常使用。

提取说明:碰到一个坑,不知道是不是,整了一天,发现 redis.conf 和 sentinel.conf 不能配置daemonize yes,设置为yes,就启动错误。

官网说明

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
# 运行
redis-sentinel /path/to/sentinel.conf
redis-server /path/to/sentinel.conf --sentinel

两种方式都相同。

但是,在运行Sentinel时必须使用配置文件,因为系统将使用此文件来保存当前状态,以便在重启时重新加载。如果未提供配置文件或配置文件路径不可写,Sentinel只会拒绝启动。

Sentinels默认情况下会运行以侦听与TCP端口26379的连接,因此,为了使Sentinels正常工作,必须打开服务器的端口26379 才能从其他Sentinel实例的IP地址接收连接。否则,Sentinels无法讨论也不能就该做什么达成共识,因此将永远不会执行故障转移。

# 部署前有关Sentinel的基本知识
1. 一个健壮的部署至少需要三个Sentinel实例。
2. 应该将三个Sentinel实例放置到被认为以独立方式失败的计算机或虚拟机中。因此,例如在不同的可用区域上执行的不同物理服务器或虚拟机。
3. 由于Redis使用异步复制,因此Sentinel + Redis分布式系统不能保证在故障期间保留已确认的写入。但是,有一些部署Sentinel的方法使窗口丢失写入仅限于某些时刻,而还有其他一些不太安全的方法来部署它。
4. 您的客户需要Sentinel支持。流行的客户端库具有Sentinel支持,但不是全部。
5. 如果您不在开发环境中不时进行测试,则没有安全的HA设置,如果可以,则在生产环境中甚至可以更好地进行测试。您可能有一个错误的配置,只有在为时已晚时(主服务器停止工作的凌晨3点),该配置才会变得明显。
6. Sentinel,Docker或其他形式的网络地址转换或端口映射应格外小心:Docker执行端口重新映射,破坏Sentinel对其他Sentinel进程的自动发现以及主副本的列表。有关更多信息,请参阅本文档后面有关Sentinel和Docker的部分。

# 配置
port 5000
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 60000
sentinel parallel-syncs mymaster 1

# 启动看到
+monitor master mymaster 127.0.0.1 6379 quorum 2

# 查看主人
$ redis-cli -p 5000
127.0.0.1:5000> sentinel master mymaster
1) "name"
2) "mymaster"
3) "ip"
4) "127.0.0.1"

# 模拟挂掉
redis-cli -p 6379 DEBUG sleep 30

127.0.0.1:5000> SENTINEL get-master-addr-by-name mymaster
1) "127.0.0.1"
2) "6380"

# Sentinel API
* PING # 此命令仅返回PONG。
* SENTINEL masters # 显示受监视的master及其状态的列表。
* SENTINEL master <master name> # 显示指定master的状态和信息。
* SENTINEL slaves <master name> # 显示此master的slave列表及其状态。
* SENTINEL sentinels <master name> # 显示此主节点的sentinel实例及其状态的列表。
* SENTINEL get-master-addr-by-name <master name> # 返回具有该名称的master的ip和端口号。如果此master正在进行故障转移或成功终止,它将返回升级slave的地址和端口。
* SENTINEL reset <pattern> # 此命令将重置所有具有匹配名称的master。pattern参数是glob样式的模式。重置过程将清除master中的所有先前状态(包括正在进行的故障转移),并删除已发现并与master关联的每个slave和sentinel。
* SENTINEL failover <master name> # 强制进行故障转移,就好像master不可访问一样,并且无需征得其他Sentinels的同意(但是将发布新版本的配置,以便其他Sentinels可以更新其配置)。
* SENTINEL ckquorum <master name> # 检查当前的Sentinel配置是否能够达到故障转移主机所需的quorum,以及授权故障转移所需的大多数。在监视系统中应使用此命令来检查Sentinel部署是否正常。
* SENTINEL flushconfig # 强制Sentinel重写其在磁盘上的配置,包括当前Sentinel状态。通常,每当状态更改时(在重新启动后一直保留在磁盘上的状态子集的上下文中),Sentinel都会重写配置。但是,有时由于操作错误,磁盘故障,程序包升级脚本或配置管理器,配置文件可能会丢失。在这些情况下,强制Sentinel重写配置文件的方法很方便。即使以前的配置文件完全丢失,此命令也有效。

# 在运行时重新配置Sentinel
请注意,如果您有多个哨兵,则应将所有更改应用于您的实例,以使Redis Sentinel正常工作。这意味着更改单个Sentinel的配置不会自动将更改传播到网络中的其他Sentinel。
以下是SENTINEL用于更新Sentinel实例的配置的子命令列表。
SENTINEL MONITOR<name> <ip> <port> <quorum> # 此命令告诉Sentinel开始监视具有指定名称,ip,端口和仲裁的新主服务器。它与配置文件中的sentinel monitor配置指令相同sentinel.conf,不同之处在于您不能在as中使用主机名ip,但需要提供IPv4或IPv6地址。
SENTINEL REMOVE<name> # 用于删除指定的主机:主机将不再受到监视,并且将从Sentinel的内部状态中完全删除,因此将不再由SENTINEL masters等等列出。
SENTINEL SET <name> <option> <value> # SET命令与Redis 的CONFIG SET命令非常相似,用于更改特定主机的配置参数。可以指定多个选项/值对(或完全不指定)。可以通过sentinel.confSET命令配置所有可以通过配置的配置参数。

配置文件解释

Redis学习笔记–Redis配置文件Sentinel.conf参数配置详解

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
# Example sentinel.conf

# 哨兵sentinel实例运行的端口 默认26379
port 26379

# 哨兵sentinel的工作目录
dir /tmp

# 哨兵sentinel监控的redis主节点的 ip port
# master-name 可以自己命名的主节点名字 只能由字母A-z、数字0-9 、这三个字符".-_"组成。
# quorum 配置多少个sentinel哨兵统一认为master主节点失联 那么这时客观上认为主节点失联了
# sentinel monitor <master-name> <ip> <redis-port> <quorum>
sentinel monitor mymaster 127.0.0.1 6379 2

# 当在Redis实例中开启了requirepass foobared 授权密码 这样所有连接Redis实例的客户端都要提供 密码
# 设置哨兵sentinel 连接主从的密码 注意必须为主从设置一样的验证密码
# sentinel auth-pass <master-name> <password>
sentinel auth-pass mymaster MySUPER--secret-0123passw0rd

# 指定多少毫秒之后 主节点没有应答哨兵sentinel 此时 哨兵主观上认为主节点下线 默认30秒
# sentinel down-after-milliseconds <master-name> <milliseconds>
sentinel down-after-milliseconds mymaster 30000

# 这个配置项指定了在发生failover主备切换时多可以有多少个slave同时对新的master进行 同步, 这个数字越小,完成failover所需的时间就越长, 但是如果这个数字越大,就意味着越 多的slave因为replication而不可用。 可以通过将这个值设为 1 来保证每次只有一个slave 处于不能处理命令请求的状态。 # sentinel parallel-syncs <master-name> <numslaves> sentinel parallel-syncs mymaster 1
# 故障转移的超时时间 failover-timeout 可以用在以下这些方面:
#1. 同一个sentinel对同一个master两次failover之间的间隔时间。
#2. 当一个slave从一个错误的master那里同步数据开始计算时间。直到slave被纠正为向正确的master那 里同步数据时。
#3.当想要取消一个正在进行的failover所需要的时间。  
#4.当进行failover时,配置所有slaves指向新的master所需的大时间。不过,即使过了这个超时,
slaves依然会被正确配置为指向master,但是就不按parallel-syncs所配置的规则来了
# 默认三分钟 # sentinel failover-timeout <master-name> <milliseconds>
sentinel failover-timeout mymaster 180000


# SCRIPTS EXECUTION

#配置当某一事件发生时所需要执行的脚本,可以通过脚本来通知管理员,例如当系统运行不正常时发邮件通知 相关人员。
#对于脚本的运行结果有以下规则:
#若脚本执行后返回1,那么该脚本稍后将会被再次执行,重复次数目前默认为10
#若脚本执行后返回2,或者比2更高的一个返回值,脚本将不会重复执行。
#如果脚本在执行过程中由于收到系统中断信号被终止了,则同返回值为1时的行为相同。
#一个脚本的大执行时间为60s,如果超过这个时间,脚本将会被一个SIGKILL信号终止,之后重新执行。

#通知型脚本:当sentinel有任何警告级别的事件发生时(比如说redis实例的主观失效和客观失效等等),
将会去调用这个脚本,这时这个脚本应该通过邮件,SMS等方式去通知系统管理员关于系统不正常运行的信 息。
调用该脚本时,将传给脚本两个参数,一个是事件的类型,一个是事件的描述。如果sentinel.conf配 置文件中配置了这个脚本路径,
那么必须保证这个脚本存在于这个路径,并且是可执行的,否则sentinel无 法正常启动成功。
# 通知脚本
# shell编程
# sentinel notification-script <master-name> <script-path>
sentinel notification-script mymaster /var/redis/notify.sh

# 客户端重新配置主节点参数脚本
# 当一个master由于failover而发生改变时,这个脚本将会被调用,通知相关的客户端关于master地址已 经发生改变的信息。
# 以下参数将会在调用脚本时传给脚本: # <master-name> <role> <state> <from-ip> <from-port> <to-ip> <to-port>
# 目前<state>总是“failover”,
# <role>是“leader”或者“observer”中的一个。
# 参数 from-ip, from-port, to-ip, to-port是用来和旧的master和新的master(即旧的slave)通 信的
# 这个脚本应该是通用的,能被多次调用,不是针对性的。
# sentinel client-reconfig-script <master-name> <script-path>
sentinel client-reconfig-script mymaster /var/redis/reconfig.sh # 一般都是由运维来配 置!

看懂了配置文件,如果不想修改,可以直接在配置文件只加入一行代码即可

1
sentinel monitor mymaster 172.22.0.79 6379 2

》》》如果想配置多个主机,可写

1
2
3
4
# 主机1
sentinel monitor mymaster 172.22.0.79 6379 2
# 主机2
sentinel monitor mymaster2 172.22.0.89 6379 2

部分命令

Linux命令 sed

1)、循环启动79 80 81

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
for n in $(seq 79 81); \
do \
cp redis.conf redis${n}.conf
sed -i 's/bind 127.0.0.1/bind 0.0.0.0/g' redis${n}.conf

if [ ${n} -ne 79 ];then
sed -i 's/# replicaof <masterip> <masterport>/replicaof 172.22.0.79 6379/g' redis${n}.conf
fi

docker run --name redis${n} -p 63${n}:6379 \
-v /app/docker/redis/sentinel-test/redis${n}.conf:/usr/local/etc/redis/redis.conf \
--net redis --ip 172.22.0.${n} -d redis \
redis-server /usr/local/etc/redis/redis.conf

done

2)、循环启动5001,5002,5003

1
2
3
4
5
6
7
8
9
10
11
for n in $(seq 1 3); \
do \
cp sentinel.conf sentinel${n}.conf
sed -i 's/sentinel monitor mymaster 127.0.0.1 6379 2/sentinel monitor mymaster 172.22.0.79 6379 2/g' sentinel${n}.conf

docker run --name redis-sentinel${n} -p 500${n}:26379 \
-v /app/docker/redis/sentinel-test/sentinel${n}.conf:/usr/local/etc/redis/sentinel.conf \
--net redis --ip 172.22.0.1${n} -d redis \
redis-sentinel /usr/local/etc/redis/sentinel.conf

done

3)、删除代码

1
2
3
docker rm -f redis79 redis80 redis81 redis-sentinel1 redis-sentinel2 redis-sentinel3

rm rf /app/docker/redis/sentinel-test/

Docker搭建一主二从三哨

配置一主二从

1)、配置一主二从

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
# 1. 创建自定义网络,进行测试
[root@localhost ~]# docker network create redis --subnet 172.22.0.0/16 --gateway 172.22.0.1
c7c9666ef7102a886fafd4b353022f755af17f11de69490caa13633d7b2b9c92
# 2. 创建目录,放置配置文件
[root@localhost ~]# mkdir -p /app/docker/redis/sentinel-test/
# 3. 进入目录
[root@localhost ~]# cd /app/docker/redis/sentinel-test/
# 4. 下载稳定版,配置文件
[root@localhost sentinel-test]# wget http://download.redis.io/redis-stable/redis.conf
--2020-06-15 23:59:35-- http://download.redis.io/redis-stable/redis.conf
Resolving download.redis.io (download.redis.io)... 109.74.203.151
Connecting to download.redis.io (download.redis.io)|109.74.203.151|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 82645 (81K) [text/plain]
Saving to: ‘redis.conf’

100%[=================================================================>] 82,645 43.9KB/s in 1.8s

2020-06-15 23:59:38 (43.9 KB/s) - ‘redis.conf’ saved [82645/82645]
# 5. 循环启动三个容器,主79,从80,81
[root@localhost sentinel-test]# for n in $(seq 79 81); \
> do \
> cp redis.conf redis${n}.conf
> sed -i 's/bind 127.0.0.1/bind 0.0.0.0/g' redis${n}.conf # 把文件bind 127.0.0.1 改为 bind 0.0.0.0 ,所有IP可访问
>
> if [ ${n} -ne 79 ];then
> sed -i 's/# replicaof <masterip> <masterport>/replicaof 172.22.0.79 6379/g' redis${n}.conf # 如果不是79,就设置为从机,认79当主机
> fi
>
> docker run --name redis${n} -p 63${n}:6379 \
> -v /app/docker/redis/sentinel-test/redis${n}.conf:/usr/local/etc/redis/redis.conf \ # 挂载
> --net redis --ip 172.22.0.${n} -d redis \ # 指定网络启动
> redis-server /usr/local/etc/redis/redis.conf # 启动命令
>
> done
7c53664247be1b219b6efec7716f2536eea5df8e91822267c5c8802d38b5c63d
3806856f61b3e66ae7e4d06b6202b02fe8be845d7abc52189cb8085645eeb518
ab00a9306e6bea1ead4e7cc825170541a0f5ff0878d6d622f3678d69a41bde2a
# 6. 查看容器,全部启动成功
[root@localhost sentinel-test]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ab00a9306e6b redis "docker-entrypoint..." 2 seconds ago Up 2 seconds 0.0.0.0:6381->6379/tcp redis81
3806856f61b3 redis "docker-entrypoint..." 3 seconds ago Up 2 seconds 0.0.0.0:6380->6379/tcp redis80
7c53664247be redis "docker-entrypoint..." 3 seconds ago Up 3 seconds 0.0.0.0:6379->6379/tcp redis79

查看一主二从

2)、查看79, 80, 81状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
# 79
[root@localhost ~]# docker exec -it redis79 redis-cli
127.0.0.1:6379> INFO replication
# Replication
role:master # 主机
connected_slaves:2 # 两个从机
slave0:ip=172.22.0.80,port=6379,state=online,offset=294,lag=0 # 从机80
slave1:ip=172.22.0.81,port=6379,state=online,offset=294,lag=0 # 从机81
master_replid:1df935573b333ea2de67d1cc4282d34b3eb2079a
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:294
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:294
127.0.0.1:6379>



# 80
[root@localhost ~]# docker exec -it redis80 redis-cli
127.0.0.1:6379> INFO replication
# Replication
role:slave # 从机
master_host:172.22.0.79 # 主机79
master_port:6379
master_link_status:up # 连接正常
master_last_io_seconds_ago:7
master_sync_in_progress:0
slave_repl_offset:294
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:1df935573b333ea2de67d1cc4282d34b3eb2079a
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:294
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:294
127.0.0.1:6379>


# 81
[root@localhost ~]# docker exec -it redis81 redis-cli
127.0.0.1:6379> INFO replication
# Replication
role:slave # 从机
master_host:172.22.0.79 # 主机79
master_port:6379
master_link_status:up # 连接正常
master_last_io_seconds_ago:6
master_sync_in_progress:0
slave_repl_offset:294
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:1df935573b333ea2de67d1cc4282d34b3eb2079a
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:294
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:294
127.0.0.1:6379>

启动三个哨兵

3)、启动三个哨兵

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# 1. 进入目录
[root@localhost ~]# cd /app/docker/redis/sentinel-test/
# 2. 下载哨兵的稳定版配置文件,注意:当前在sentinel-test目录
[root@localhost sentinel-test]# wget http://download.redis.io/redis-stable/sentinel.conf
--2020-06-16 00:05:11-- http://download.redis.io/redis-stable/sentinel.conf
Resolving download.redis.io (download.redis.io)... 109.74.203.151
Connecting to download.redis.io (download.redis.io)|109.74.203.151|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10743 (10K) [text/plain]
Saving to: ‘sentinel.conf’

100%[=================================================================>] 10,743 --.-K/s in 0s

2020-06-16 00:05:12 (38.2 MB/s) - ‘sentinel.conf’ saved [10743/10743]

# 3. 循环启动三个哨兵,端口分别本地映射为5001, 5002, 5003
[root@localhost sentinel-test]# for n in $(seq 1 3); \
> do \
> cp sentinel.conf sentinel${n}.conf
> sed -i 's/sentinel monitor mymaster 127.0.0.1 6379 2/sentinel monitor mymaster 172.22.0.79 6379 2/g' sentinel${n}.conf
>
> docker run --name redis-sentinel${n} -p 500${n}:26379 \
> -v /app/docker/redis/sentinel-test/sentinel${n}.conf:/usr/local/etc/redis/sentinel.conf \
> --net redis --ip 172.22.0.1${n} -d redis \
> redis-sentinel /usr/local/etc/redis/sentinel.conf
>
> done
96f2b7f56672884279458fecc79e08568b5b163f809e9521d58ffef643f6fb0e
62c6e9d5286da27176556d26d0ba5d5f2367bd95f835291f86c27c4b250e672c
51f75b10fdcd48163db81a61034ec37b78e2b2297b0cf55694cb3fff88542230
# 4. 查看启动状态
[root@localhost sentinel-test]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
51f75b10fdcd redis "docker-entrypoint..." 3 seconds ago Up 1 second 6379/tcp, 0.0.0.0:5003->26379/tcp redis-sentinel3
62c6e9d5286d redis "docker-entrypoint..." 3 seconds ago Up 2 seconds 6379/tcp, 0.0.0.0:5002->26379/tcp redis-sentinel2
96f2b7f56672 redis "docker-entrypoint..." 4 seconds ago Up 2 seconds 6379/tcp, 0.0.0.0:5001->26379/tcp redis-sentinel1
ab00a9306e6b redis "docker-entrypoint..." 5 minutes ago Up 5 minutes 0.0.0.0:6381->6379/tcp redis81
3806856f61b3 redis "docker-entrypoint..." 5 minutes ago Up 5 minutes 0.0.0.0:6380->6379/tcp redis80
7c53664247be redis "docker-entrypoint..." 5 minutes ago Up 5 minutes 0.0.0.0:6379->6379/tcp redis79

查看哨兵状态

4)、查看哨兵状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
# 5001
[root@localhost ~]# docker logs -f redis-sentinel1
1:X 16 Jun 2020 07:05:23.291 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:X 16 Jun 2020 07:05:23.291 # Redis version=6.0.4, bits=64, commit=00000000, modified=0, pid=1, just started
1:X 16 Jun 2020 07:05:23.292 # Configuration loaded
1:X 16 Jun 2020 07:05:23.295 * Running mode=sentinel, port=26379.
1:X 16 Jun 2020 07:05:23.295 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:X 16 Jun 2020 07:05:23.307 # Sentinel ID is b6ff3e16c7c6df1ba927e25ff3d5ef0a307181b7
1:X 16 Jun 2020 07:05:23.307 # +monitor master mymaster 172.22.0.79 6379 quorum 2
1:X 16 Jun 2020 07:05:23.308 * +slave slave 172.22.0.80:6379 172.22.0.80 6379 @ mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:05:23.314 * +slave slave 172.22.0.81:6379 172.22.0.81 6379 @ mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:05:25.919 * +sentinel sentinel 8531e2800fe4c2f09d59ed9bf4eaef41e86ba705 172.22.0.12 26379 @ mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:05:26.468 * +sentinel sentinel b0cad35c402d81ced4cd6601c9f06d3b2ccfebd4 172.22.0.13 26379 @ mymaster 172.22.0.79 6379

# 5002
[root@localhost ~]# docker logs -f redis-sentinel2
1:X 16 Jun 2020 07:05:23.870 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:X 16 Jun 2020 07:05:23.870 # Redis version=6.0.4, bits=64, commit=00000000, modified=0, pid=1, just started
1:X 16 Jun 2020 07:05:23.870 # Configuration loaded
1:X 16 Jun 2020 07:05:23.871 * Running mode=sentinel, port=26379.
1:X 16 Jun 2020 07:05:23.874 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:X 16 Jun 2020 07:05:23.878 # Sentinel ID is 8531e2800fe4c2f09d59ed9bf4eaef41e86ba705
1:X 16 Jun 2020 07:05:23.878 # +monitor master mymaster 172.22.0.79 6379 quorum 2
1:X 16 Jun 2020 07:05:23.881 * +slave slave 172.22.0.80:6379 172.22.0.80 6379 @ mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:05:23.891 * +slave slave 172.22.0.81:6379 172.22.0.81 6379 @ mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:05:25.326 * +sentinel sentinel b6ff3e16c7c6df1ba927e25ff3d5ef0a307181b7 172.22.0.11 26379 @ mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:05:26.469 * +sentinel sentinel b0cad35c402d81ced4cd6601c9f06d3b2ccfebd4 172.22.0.13 26379 @ mymaster 172.22.0.79 6379


# 5003
[root@localhost ~]# docker logs -f redis-sentinel3
1:X 16 Jun 2020 07:05:24.407 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:X 16 Jun 2020 07:05:24.407 # Redis version=6.0.4, bits=64, commit=00000000, modified=0, pid=1, just started
1:X 16 Jun 2020 07:05:24.407 # Configuration loaded
1:X 16 Jun 2020 07:05:24.412 * Running mode=sentinel, port=26379.
1:X 16 Jun 2020 07:05:24.412 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:X 16 Jun 2020 07:05:24.418 # Sentinel ID is b0cad35c402d81ced4cd6601c9f06d3b2ccfebd4
1:X 16 Jun 2020 07:05:24.418 # +monitor master mymaster 172.22.0.79 6379 quorum 2
1:X 16 Jun 2020 07:05:24.420 * +slave slave 172.22.0.80:6379 172.22.0.80 6379 @ mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:05:24.422 * +slave slave 172.22.0.81:6379 172.22.0.81 6379 @ mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:05:25.328 * +sentinel sentinel b6ff3e16c7c6df1ba927e25ff3d5ef0a307181b7 172.22.0.11 26379 @ mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:05:25.922 * +sentinel sentinel 8531e2800fe4c2f09d59ed9bf4eaef41e86ba705 172.22.0.12 26379 @ mymaster 172.22.0.79 6379

主机79挂了

5)、挂掉主机79,查看哨兵状态

1
2
[root@localhost ~]# docker stop redis79
redis79

哨兵选举主机

6)、稍等一会,等待哨兵检测主机挂了,然后,哨兵进行投票,从80, 81中选取一个当做新的主机
查看哨兵日志,这里只列出了新增的日志信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
# 5001
1:X 16 Jun 2020 07:07:46.067 # +sdown master mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:07:46.155 # +new-epoch 1
1:X 16 Jun 2020 07:07:46.159 # +vote-for-leader b0cad35c402d81ced4cd6601c9f06d3b2ccfebd4 1
1:X 16 Jun 2020 07:07:46.652 # +config-update-from sentinel b0cad35c402d81ced4cd6601c9f06d3b2ccfebd4 172.22.0.13 26379 @ mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:07:46.652 # +switch-master mymaster 172.22.0.79 6379 172.22.0.80 6379
1:X 16 Jun 2020 07:07:46.652 * +slave slave 172.22.0.81:6379 172.22.0.81 6379 @ mymaster 172.22.0.80 6379
1:X 16 Jun 2020 07:07:46.652 * +slave slave 172.22.0.79:6379 172.22.0.79 6379 @ mymaster 172.22.0.80 6379
1:X 16 Jun 2020 07:08:16.694 # +sdown slave 172.22.0.79:6379 172.22.0.79 6379 @ mymaster 172.22.0.80 6379


# 5002
1:X 16 Jun 2020 07:07:46.068 # +sdown master mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:07:46.154 # +new-epoch 1
1:X 16 Jun 2020 07:07:46.158 # +vote-for-leader b0cad35c402d81ced4cd6601c9f06d3b2ccfebd4 1
1:X 16 Jun 2020 07:07:46.169 # +odown master mymaster 172.22.0.79 6379 #quorum 2/2
1:X 16 Jun 2020 07:07:46.169 # Next failover delay: I will not start a failover before Tue Jun 16 07:13:462020
1:X 16 Jun 2020 07:07:46.651 # +config-update-from sentinel b0cad35c402d81ced4cd6601c9f06d3b2ccfebd4 172.22.0.13 26379 @ mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:07:46.651 # +switch-master mymaster 172.22.0.79 6379 172.22.0.80 6379
1:X 16 Jun 2020 07:07:46.651 * +slave slave 172.22.0.81:6379 172.22.0.81 6379 @ mymaster 172.22.0.80 6379
1:X 16 Jun 2020 07:07:46.651 * +slave slave 172.22.0.79:6379 172.22.0.79 6379 @ mymaster 172.22.0.80 6379
1:X 16 Jun 2020 07:08:16.677 # +sdown slave 172.22.0.79:6379 172.22.0.79 6379 @ mymaster 172.22.0.80 6379


# 5003
1:X 16 Jun 2020 07:07:46.075 # +sdown master mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:07:46.147 # +odown master mymaster 172.22.0.79 6379 #quorum 3/2
1:X 16 Jun 2020 07:07:46.147 # +new-epoch 1
1:X 16 Jun 2020 07:07:46.147 # +try-failover master mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:07:46.151 # +vote-for-leader b0cad35c402d81ced4cd6601c9f06d3b2ccfebd4 1
1:X 16 Jun 2020 07:07:46.159 # 8531e2800fe4c2f09d59ed9bf4eaef41e86ba705 voted for b0cad35c402d81ced4cd6601c9f06d3b2ccfebd4 1
1:X 16 Jun 2020 07:07:46.159 # b6ff3e16c7c6df1ba927e25ff3d5ef0a307181b7 voted for b0cad35c402d81ced4cd6601c9f06d3b2ccfebd4 1
1:X 16 Jun 2020 07:07:46.252 # +elected-leader master mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:07:46.252 # +failover-state-select-slave master mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:07:46.311 # +selected-slave slave 172.22.0.80:6379 172.22.0.80 6379 @ mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:07:46.311 * +failover-state-send-slaveof-noone slave 172.22.0.80:6379 172.22.0.80 6379 @ mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:07:46.388 * +failover-state-wait-promotion slave 172.22.0.80:6379 172.22.0.80 6379 @ mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:07:46.590 # +promoted-slave slave 172.22.0.80:6379 172.22.0.80 6379 @ mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:07:46.590 # +failover-state-reconf-slaves master mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:07:46.648 * +slave-reconf-sent slave 172.22.0.81:6379 172.22.0.81 6379 @ mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:07:47.310 # -odown master mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:07:47.635 * +slave-reconf-inprog slave 172.22.0.81:6379 172.22.0.81 6379 @ mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:07:47.635 * +slave-reconf-done slave 172.22.0.81:6379 172.22.0.81 6379 @ mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:07:47.734 # +failover-end master mymaster 172.22.0.79 6379
# 切换master 79变80
1:X 16 Jun 2020 07:07:47.734 # +switch-master mymaster 172.22.0.79 6379 172.22.0.80 6379
# 81认80当主人
1:X 16 Jun 2020 07:07:47.735 * +slave slave 172.22.0.81:6379 172.22.0.81 6379 @ mymaster 172.22.0.80 6379
# 79认80当主人
1:X 16 Jun 2020 07:07:47.735 * +slave slave 172.22.0.79:6379 172.22.0.79 6379 @ mymaster 172.22.0.80 6379
# sdown:79挂了
1:X 16 Jun 2020 07:08:17.765 # +sdown slave 172.22.0.79:6379 172.22.0.79 6379 @ mymaster 172.22.0.80 6379

查看80,81状态

7)、再次查看80, 81状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# 80
127.0.0.1:6379> INFO replication
# Replication
role:master # 从机变主机
connected_slaves:1
slave0:ip=172.22.0.81,port=6379,state=online,offset=27254,lag=1 # 从机81
master_replid:cd6870dd9c075bcc69f208c1bc6ee8809b0a9b2c
master_replid2:1df935573b333ea2de67d1cc4282d34b3eb2079a
master_repl_offset:27542
second_repl_offset:22957
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:27542
127.0.0.1:6379>


# 81
127.0.0.1:6379> INFO replication
# Replication
role:slave # 仍是从机
master_host:172.22.0.80 #主机79变80
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:27953
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:cd6870dd9c075bcc69f208c1bc6ee8809b0a9b2c
master_replid2:1df935573b333ea2de67d1cc4282d34b3eb2079a
master_repl_offset:27953
second_repl_offset:22957
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:27953
127.0.0.1:6379>

再启动79

8)、再启动79,看看还会不会成为主机?答:不会,它会以从机方式启动,认80当主机
我愿把这种方式称之为:谋朝篡位

在上面哨兵选举主机的注释中已经解释了,79,81都认80当master

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
# 79
[root@localhost ~]# docker start redis79
redis79
[root@localhost ~]# docker exec -it redis79 redis-cli
127.0.0.1:6379> INFO replication
# Replication
role:slave # 启动变从机
master_host:172.22.0.80 # 主机80
master_port:6379
master_link_status:up # 连接正常
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:98322
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:cd6870dd9c075bcc69f208c1bc6ee8809b0a9b2c
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:98322
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:97889
repl_backlog_histlen:434
127.0.0.1:6379>

# 80
127.0.0.1:6379> INFO replication
# Replication
role:master # 仍然是主机
connected_slaves:2 # 两台从机
slave0:ip=172.22.0.81,port=6379,state=online,offset=99007,lag=0 # 从机81
slave1:ip=172.22.0.79,port=6379,state=online,offset=99007,lag=1 # 从机79
master_replid:cd6870dd9c075bcc69f208c1bc6ee8809b0a9b2c
master_replid2:1df935573b333ea2de67d1cc4282d34b3eb2079a
master_repl_offset:99158
second_repl_offset:22957
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:99158
127.0.0.1:6379>

查看哨兵日志

9)、查看哨兵日志,这里只列出了新增的日志信息

1
2
3
4
5
6
7
8
9
# 5001
1:X 16 Jun 2020 07:13:44.703 # -sdown slave 172.22.0.79:6379 172.22.0.79 6379 @ mymaster 172.22.0.80 6379

# 5002
1:X 16 Jun 2020 07:13:44.537 # -sdown slave 172.22.0.79:6379 172.22.0.79 6379 @ mymaster 172.22.0.80 6379
1:X 16 Jun 2020 07:13:54.492 * +convert-to-slave slave 172.22.0.79:6379 172.22.0.79 6379 @ mymaster 172.22.0.80 6379

# 5003
1:X 16 Jun 2020 07:13:44.723 # -sdown slave 172.22.0.79:6379 172.22.0.79 6379 @ mymaster 172.22.0.80 6379

再停止80

10)、如果再进行停止80,道理同上,三个哨兵会继续在当前的从机 79, 81中进行投票选举一个作为主机

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
[root@localhost ~]# docker stop redis80
redis80

# 5001
1:X 16 Jun 2020 07:18:24.557 # +sdown master mymaster 172.22.0.80 6379
1:X 16 Jun 2020 07:18:24.620 # +new-epoch 2
1:X 16 Jun 2020 07:18:24.622 # +vote-for-leader b0cad35c402d81ced4cd6601c9f06d3b2ccfebd4 2
1:X 16 Jun 2020 07:18:24.622 # +odown master mymaster 172.22.0.80 6379 #quorum 2/2
1:X 16 Jun 2020 07:18:24.622 # Next failover delay: I will not start a failover before Tue Jun 16 07:24:242020
1:X 16 Jun 2020 07:18:25.815 # +config-update-from sentinel b0cad35c402d81ced4cd6601c9f06d3b2ccfebd4 172.22.0.13 26379 @ mymaster 172.22.0.80 6379
1:X 16 Jun 2020 07:18:25.815 # +switch-master mymaster 172.22.0.80 6379 172.22.0.79 6379
1:X 16 Jun 2020 07:18:25.816 * +slave slave 172.22.0.81:6379 172.22.0.81 6379 @ mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:18:25.816 * +slave slave 172.22.0.80:6379 172.22.0.80 6379 @ mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:18:55.858 # +sdown slave 172.22.0.80:6379 172.22.0.80 6379 @ mymaster 172.22.0.79 6379

# 5002
1:X 16 Jun 2020 07:18:24.524 # +sdown master mymaster 172.22.0.80 6379
1:X 16 Jun 2020 07:18:24.619 # +new-epoch 2
1:X 16 Jun 2020 07:18:24.622 # +vote-for-leader b0cad35c402d81ced4cd6601c9f06d3b2ccfebd4 2
1:X 16 Jun 2020 07:18:25.630 # +odown master mymaster 172.22.0.80 6379 #quorum 3/2
1:X 16 Jun 2020 07:18:25.630 # Next failover delay: I will not start a failover before Tue Jun 16 07:24:252020
1:X 16 Jun 2020 07:18:25.814 # +config-update-from sentinel b0cad35c402d81ced4cd6601c9f06d3b2ccfebd4 172.22.0.13 26379 @ mymaster 172.22.0.80 6379
1:X 16 Jun 2020 07:18:25.814 # +switch-master mymaster 172.22.0.80 6379 172.22.0.79 6379
1:X 16 Jun 2020 07:18:25.814 * +slave slave 172.22.0.81:6379 172.22.0.81 6379 @ mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:18:25.814 * +slave slave 172.22.0.80:6379 172.22.0.80 6379 @ mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:18:55.818 # +sdown slave 172.22.0.80:6379 172.22.0.80 6379 @ mymaster 172.22.0.79 6379


# 5003
1:X 16 Jun 2020 07:18:24.558 # +sdown master mymaster 172.22.0.80 6379
1:X 16 Jun 2020 07:18:24.611 # +odown master mymaster 172.22.0.80 6379 #quorum 3/2
1:X 16 Jun 2020 07:18:24.611 # +new-epoch 2
1:X 16 Jun 2020 07:18:24.611 # +try-failover master mymaster 172.22.0.80 6379
1:X 16 Jun 2020 07:18:24.615 # +vote-for-leader b0cad35c402d81ced4cd6601c9f06d3b2ccfebd4 2
1:X 16 Jun 2020 07:18:24.622 # 8531e2800fe4c2f09d59ed9bf4eaef41e86ba705 voted for b0cad35c402d81ced4cd6601c9f06d3b2ccfebd4 2
1:X 16 Jun 2020 07:18:24.622 # b6ff3e16c7c6df1ba927e25ff3d5ef0a307181b7 voted for b0cad35c402d81ced4cd6601c9f06d3b2ccfebd4 2
1:X 16 Jun 2020 07:18:24.706 # +elected-leader master mymaster 172.22.0.80 6379
1:X 16 Jun 2020 07:18:24.706 # +failover-state-select-slave master mymaster 172.22.0.80 6379
1:X 16 Jun 2020 07:18:24.790 # +selected-slave slave 172.22.0.79:6379 172.22.0.79 6379 @ mymaster 172.22.0.80 6379
1:X 16 Jun 2020 07:18:24.790 * +failover-state-send-slaveof-noone slave 172.22.0.79:6379 172.22.0.79 6379 @ mymaster 172.22.0.80 6379
1:X 16 Jun 2020 07:18:24.881 * +failover-state-wait-promotion slave 172.22.0.79:6379 172.22.0.79 6379 @ mymaster 172.22.0.80 6379
1:X 16 Jun 2020 07:18:25.715 # +promoted-slave slave 172.22.0.79:6379 172.22.0.79 6379 @ mymaster 172.22.0.80 6379
1:X 16 Jun 2020 07:18:25.715 # +failover-state-reconf-slaves master mymaster 172.22.0.80 6379
1:X 16 Jun 2020 07:18:25.812 * +slave-reconf-sent slave 172.22.0.81:6379 172.22.0.81 6379 @ mymaster 172.22.0.80 6379
1:X 16 Jun 2020 07:18:26.490 * +slave-reconf-inprog slave 172.22.0.81:6379 172.22.0.81 6379 @ mymaster 172.22.0.80 6379
1:X 16 Jun 2020 07:18:26.490 * +slave-reconf-done slave 172.22.0.81:6379 172.22.0.81 6379 @ mymaster 172.22.0.80 6379
1:X 16 Jun 2020 07:18:26.580 # +failover-end master mymaster 172.22.0.80 6379
1:X 16 Jun 2020 07:18:26.580 # +switch-master mymaster 172.22.0.80 6379 172.22.0.79 6379
1:X 16 Jun 2020 07:18:26.580 * +slave slave 172.22.0.81:6379 172.22.0.81 6379 @ mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:18:26.580 * +slave slave 172.22.0.80:6379 172.22.0.80 6379 @ mymaster 172.22.0.79 6379
1:X 16 Jun 2020 07:18:56.619 # +sdown slave 172.22.0.80:6379 172.22.0.80 6379 @ mymaster 172.22.0.79 6379

# 79
127.0.0.1:6379> INFO replication
# Replication
role:master # 从机变主机
connected_slaves:1
slave0:ip=172.22.0.81,port=6379,state=online,offset=159456,lag=1 # 从机81
master_replid:fcc8eaecc47c0597f69d82acbe6c5153757cc133
master_replid2:cd6870dd9c075bcc69f208c1bc6ee8809b0a9b2c
master_repl_offset:159593
second_repl_offset:146609
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:97889
repl_backlog_histlen:61705
127.0.0.1:6379>

# 81
127.0.0.1:6379> INFO replication
# Replication
role:slave # 从机
master_host:172.22.0.79 # 主机变79
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:160004
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:fcc8eaecc47c0597f69d82acbe6c5153757cc133
master_replid2:cd6870dd9c075bcc69f208c1bc6ee8809b0a9b2c
master_repl_offset:160004
second_repl_offset:146609
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:160004
127.0.0.1:6379>