Redis cluster 集群配置

redis集群
搭建61-66的环境:安装redis 端口号6351-6356 删除127.0.0.1 添加192.168.4.61-66
完成后确认服务已启动,且有两个端口,其中一个10000+的就是集群时用的端口
IP 规划
• redis 服务器 ip 地址及端口规划
– redisA 192.168.4.51 6351
– redisB 192.168.4.52 6352
– redisC 192.168.4.53 6353
– redisD 192.168.4.54 6354
– redisE 192.168.4.55 6355
– redisF 192.168.4.56 6356

[root@61 ~]# /etc/init.d/redis_6351 restart
Stopping …
Redis stopped
Starting Redis server…
[root@61 ~]# netstat -antup | grep redis
tcp 0 0 192.168.4.61:6351 0.0.0.0:* LISTEN 2807/redis-server 1
tcp 0 0 192.168.4.61:16351 0.0.0.0:* LISTEN 2807/redis-server 1

全部登陆一遍,确认远程登陆正常
[root@66 ~]# redis-cli -h 192.168.4.61 -p 6351
192.168.4.61:6351> ping
PONG
192.168.4.61:6351> exit
not connected> exit
[root@66 ~]# redis-cli -h 192.168.4.62 -p 6352
192.168.4.62:6352> ping
PONG
192.168.4.62:6352> exit
[root@66 ~]# redis-cli -h 192.168.4.63 -p 6353
192.168.4.63:6353> ping
PONG
192.168.4.63:6353> exit
[root@66 ~]# redis-cli -h 192.168.4.64 -p 6354
192.168.4.64:6354> ping
PONG
192.168.4.64:6354> exit
[root@66 ~]# redis-cli -h 192.168.4.65 -p 6355
192.168.4.65:6355> ping
PONG
192.168.4.65:6355> exit
[root@66 ~]# redis-cli -h 192.168.4.66 -p 6356
192.168.4.66:6356> ping
PONG
192.168.4.66:6356> exit

安装集群环境
安装ruby执行环境环境,为了调用ridis下面的集群环境的脚本
[root@61 redis-cluster]# yum install -y ruby rubygems
已安装:
ruby.x86_64 0:2.0.0.648-30.el7 rubygems.noarch 0:2.0.14.1-30.el7
作为依赖被安装:
libyaml.x86_64 0:0.1.4-11.el7_0 ruby-irb.noarch 0:2.0.0.648-30.el7
ruby-libs.x86_64 0:2.0.0.648-30.el7 rubygem-bigdecimal.x86_64 0:1.2.0-30.el7
rubygem-io-console.x86_64 0:0.4.2-30.el7 rubygem-json.x86_64 0:1.7.7-30.el7
rubygem-psych.x86_64 0:2.0.0-30.el7 rubygem-rdoc.noarch 0:4.0.0-30.el7
完毕!

安装gem包,必须先装gem程序,默认已安装
[root@61 redis-cluster]# rpm -qf /usr/bin/gem
rubygems-2.0.14.1-30.el7.noarch

[root@61 redis-cluster]# gem install redis-3.2.1.gem
Successfully installed redis-3.2.1
Parsing documentation for redis-3.2.1
Installing ri documentation for redis-3.2.1
1 gem installed

安装ruby-devel
[root@61 redis-cluster]# yum install -y ruby-devel-2.0.0.648-30.el7.x86_64.rpm
已安装:
ruby-devel.x86_64 0:2.0.0.648-30.el7
完毕!

上面的操作,为的是部署ruby环境,因为执行redis-cluster脚本时,需要ruby环境,脚本就在安装包SRC文件夹里面
[root@61 src]# pwd
/root/soft/redis/redis-4.0.8/src
[root@61 src]# ls
adlist.c cluster.o geo.h memtest.o rdb.c scripting.c syncio.o
adlist.h config.c geohash.c mkreleasehdr.sh rdb.h scripting.o testhelp.h
adlist.o config.h geohash.h module.c rdb.o sdsalloc.h t_hash.c
ae.c config.o geohash_helper.c module.o redisassert.h sds.c t_hash.o
ae_epoll.c crc16.c geohash_helper.h modules redis-benchmark sds.h t_list.c
ae_evport.c crc16.o geohash_helper.o multi.c redis-benchmark.c sds.o t_list.o
ae.h crc64.c geohash.o multi.o redis-benchmark.o sentinel.c t_set.c
ae_kqueue.c crc64.h geo.o networking.c redis-check-aof sentinel.o t_set.o
ae.o crc64.o help.h networking.o redis-check-aof.c server.c t_string.c
ae_select.c db.c hyperloglog.c notify.c redis-check-aof.o server.h t_string.o
anet.c db.o hyperloglog.o notify.o redis-check-rdb server.o t_zset.c
anet.h debug.c intset.c object.c redis-check-rdb.c setproctitle.c t_zset.o
anet.o debugmacro.h intset.h object.o redis-check-rdb.o setproctitle.o util.c
aof.c debug.o intset.o pqsort.c redis-cli sha1.c util.h
aof.o defrag.c latency.c pqsort.h redis-cli.c sha1.h util.o
asciilogo.h defrag.o latency.h pqsort.o redis-cli.o sha1.o valgrind.sup
atomicvar.h dict.c latency.o pubsub.c redismodule.h siphash.c version.h
bio.c dict.h lazyfree.c pubsub.o redis-sentinel siphash.o ziplist.c
bio.h dict.o lazyfree.o quicklist.c redis-server slowlog.c ziplist.h
bio.o endianconv.c lzf_c.c quicklist.h redis-trib.rb slowlog.h ziplist.o
bitops.c endianconv.h lzf_c.o quicklist.o release.c slowlog.o zipmap.c
bitops.o endianconv.o lzf_d.c rand.c release.h solarisfixes.h zipmap.h
blocked.c evict.c lzf_d.o rand.h release.o sort.c zipmap.o
blocked.o evict.o lzf.h rand.o replication.c sort.o zmalloc.c
childinfo.c expire.c lzfP.h rax.c replication.o sparkline.c zmalloc.h
childinfo.o expire.o Makefile rax.h rio.c sparkline.h zmalloc.o
cluster.c fmacros.h Makefile.dep rax_malloc.h rio.h sparkline.o
cluster.h geo.c memtest.c rax.o rio.o syncio.c

执行 redis-trib.rb create 创建集群,后面跟上节点IP地址和端口号
[root@61 src]# ./redis-trib.rb create –replicas 1 192.168.4.61:6351 192.168.4.62:6352 192.168.4.63:6353 192.168.4.64:6354 192.168.4.65:6355 192.168.4.66:6356
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes…
Using 3 masters:
192.168.4.61:6351
192.168.4.62:6352
192.168.4.63:6353
Adding replica 192.168.4.65:6355 to 192.168.4.61:6351
Adding replica 192.168.4.66:6356 to 192.168.4.62:6352
Adding replica 192.168.4.64:6354 to 192.168.4.63:6353
M: fb2dc0d120eea527ccf2aedaa61503d58c5c7f80 192.168.4.61:6351
slots:0-5460 (5461 slots) master
M: 6fa6d44af120d0a4be5f0ba2d083b800b59be41a 192.168.4.62:6352
slots:5461-10922 (5462 slots) master
M: 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8 192.168.4.63:6353
slots:10923-16383 (5461 slots) master
S: 4d672007b9e27119049efe94c798ca1b0ab7c7b3 192.168.4.64:6354
replicates 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8
S: 56c7c875ec0ace4902f70c0dabeb401b0e60eeee 192.168.4.65:6355
replicates fb2dc0d120eea527ccf2aedaa61503d58c5c7f80
S: fbe8bb6550867ef98feb2e06238a96e1df85f1d7 192.168.4.66:6356
replicates 6fa6d44af120d0a4be5f0ba2d083b800b59be41a
Can I set the above configuration? (type ‘yes’ to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join…
>>> Performing Cluster Check (using node 192.168.4.61:6351)
M: fb2dc0d120eea527ccf2aedaa61503d58c5c7f80 192.168.4.61:6351
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: 56c7c875ec0ace4902f70c0dabeb401b0e60eeee 192.168.4.65:6355
slots: (0 slots) slave
replicates fb2dc0d120eea527ccf2aedaa61503d58c5c7f80
S: 4d672007b9e27119049efe94c798ca1b0ab7c7b3 192.168.4.64:6354
slots: (0 slots) slave
replicates 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8
M: 6fa6d44af120d0a4be5f0ba2d083b800b59be41a 192.168.4.62:6352
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: fbe8bb6550867ef98feb2e06238a96e1df85f1d7 192.168.4.66:6356
slots: (0 slots) slave
replicates 6fa6d44af120d0a4be5f0ba2d083b800b59be41a
M: 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8 192.168.4.63:6353
slots:10923-16383 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots…
>>> Check slots coverage…
[OK] All 16384 slots covered.

登陆查看状态

[root@61 src]# redis-cli -c -h 192.168.4.61 -p 6351
192.168.4.61:6351> ping
PONG
192.168.4.61:6351> cluster nodes
56c7c875ec0ace4902f70c0dabeb401b0e60eeee 192.168.4.65:6355@16355 slave fb2dc0d120eea527ccf2aedaa61503d58c5c7f80 0 1528428354808 5 connected
4d672007b9e27119049efe94c798ca1b0ab7c7b3 192.168.4.64:6354@16354 slave 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8 0 1528428354808 4 connected
6fa6d44af120d0a4be5f0ba2d083b800b59be41a 192.168.4.62:6352@16352 master – 0 1528428354306 2 connected 5461-10922
fb2dc0d120eea527ccf2aedaa61503d58c5c7f80 192.168.4.61:6351@16351 myself,master – 0 1528428346000 1 connected 0-5460
fbe8bb6550867ef98feb2e06238a96e1df85f1d7 192.168.4.66:6356@16356 slave 6fa6d44af120d0a4be5f0ba2d083b800b59be41a 0 1528428352800 6 connected
7b27e1369fb56a16f4b090d011ba291cf6a1dcf8 192.168.4.63:6353@16353 master – 0 1528428353804 3 connected 10923-16383

192.168.4.61:6351> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:6569
cluster_stats_messages_pong_sent:5928
cluster_stats_messages_sent:12497
cluster_stats_messages_ping_received:5923
cluster_stats_messages_pong_received:6569
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:12497
192.168.4.61:6351> exit

查看有对应的集群配置文件
[root@61 src]# ls /var/lib/redis/6351/
dump.rdb nodes-6379.conf

[root@61 src]# cat /var/lib/redis/6351/nodes-6379.conf
56c7c875ec0ace4902f70c0dabeb401b0e60eeee 192.168.4.65:6355@16355 slave fb2dc0d120eea527ccf2aedaa61503d58c5c7f80 0 1528425332116 5 connected
4d672007b9e27119049efe94c798ca1b0ab7c7b3 192.168.4.64:6354@16354 slave 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8 0 1528425330101 4 connected
6fa6d44af120d0a4be5f0ba2d083b800b59be41a 192.168.4.62:6352@16352 master – 0 1528425331814 2 connected 5461-10922
fb2dc0d120eea527ccf2aedaa61503d58c5c7f80 192.168.4.61:6351@16351 myself,master – 0 1528425326000 1 connected 0-5460
fbe8bb6550867ef98feb2e06238a96e1df85f1d7 192.168.4.66:6356@16356 slave 6fa6d44af120d0a4be5f0ba2d083b800b59be41a 0 1528425330614 6 connected
7b27e1369fb56a16f4b090d011ba291cf6a1dcf8 192.168.4.63:6353@16353 master – 0 1528425331612 3 connected 10923-16383
vars currentEpoch 6 lastVoteEpoch 0
解读以上信息:该集群共3对1主1从,且1个主对应一个从
fbe8bb6550867ef98feb2e06238a96e1df85f1d7 192.168.4.66:6356@16356 slave 6fa6d44af120d0a4be5f0ba2d083b800b59be41a 0 1528425330614 6 connected
表示该节点是主机*1a的从
6fa6d44af120d0a4be5f0ba2d083b800b59be41a 192.168.4.62:6352@16352 master – 0 1528425331814 2 connected 5461-10922

以此类推

计算方法
把key用CRC16算法做计算 然后计算结果和16384做求模 确定在哪个槽
结果%16384 得到 497,可以发现对应这个槽的范围,因此放在这个槽(对应192.168.4.61)
fb2dc0d120eea527ccf2aedaa61503d58c5c7f80 192.168.4.61:6351@16351 myself,master – 0 1528425326000 1 connected 0-5460
然后它的从自动复制信息(对应192.168.4.65)
56c7c875ec0ace4902f70c0dabeb401b0e60eeee 192.168.4.65:6355@16355 slave fb2dc0d120eea527ccf2aedaa61503d58c5c7f80 0 1528425332116 5 connected

然后对应的从去自动赋值主的信息

测试集群
随便插入一个值试一下
192.168.4.61:6351> set name jerry
-> Redirected to slot [5798] located at 192.168.4.62:6352
OK
192.168.4.62:6352> get name
“jerry”
192.168.4.62:6352> exit

我们挨个登陆检查一下看看在哪个库,由上面存储跳动的IP可以发现,值存在了62这个库,然后因为66是62的从库,所有66库里面也有一份。
然后这是分布式存储,所以值是分布在各个主库里面的,3个主库里只有1个库有这个值name,其它库是没有的
[root@61 src]# redis-cli -c -h 192.168.4.61 -p 6351
192.168.4.61:6351> keys *
(empty list or set)
192.168.4.61:6351> exit
[root@61 src]# redis-cli -c -h 192.168.4.62 -p 6352
192.168.4.62:6352> keys *
1) “name”
192.168.4.62:6352> exit
[root@61 src]# redis-cli -c -h 192.168.4.63 -p 6353
192.168.4.63:6353> keys *
(empty list or set)
192.168.4.63:6353> exit
[root@61 src]# redis-cli -c -h 192.168.4.64 -p 6354
192.168.4.64:6354> keys *
(empty list or set)
192.168.4.64:6354> exit
[root@61 src]# redis-cli -c -h 192.168.4.65 -p 6355
192.168.4.65:6355> keys *
(empty list or set)
192.168.4.65:6355> exit
[root@61 src]# redis-cli -c -h 192.168.4.66 -p 6356
192.168.4.66:6356> keys *
1) “name”
192.168.4.66:6356> exit

注意,当值在存储时,是根据算法来决定存在哪个库,所以实际访问时,IP会改变是正常的,也是根据算法来决定从哪个库上去找

管理集群
新准备2台机器57,58,运行redis做集群配置
同上面的配置集群一样,配置好57,58两台redis服务器
(省略具体的配置)
[root@localhost redis-4.0.8]# pwd
/root/soft/redis/redis-4.0.8
[root@localhost redis-4.0.8]# make
[root@localhost redis-4.0.8]# make install
[root@localhost redis-4.0.8]# ./utils/install_server.sh
[root@localhost redis-4.0.8]# vim /etc/redis/6358.conf
[root@localhost redis-4.0.8]# /etc/init.d/redis_6358 restart
[root@localhost redis-4.0.8]# netstat -antup | grep redis
tcp 0 0 192.168.4.68:6358 0.0.0.0:* LISTEN 4941/redis-server 1
tcp 0 0 192.168.4.68:16358 0.0.0.0:* LISTEN 4941/redis-server 1
[root@localhost redis-4.0.8]#

在没有添加集群时也是可以看状态的
192.168.4.68:6358> cluster nodes
f6d60b431050daac64c0a5c020dcf49a399d9cc6 :6358@16358 myself,master – 0 0 0 connected
192.168.4.68:6358> cluster info
cluster_state:fail
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:1
cluster_size:0
cluster_current_epoch:0
cluster_my_epoch:0
cluster_stats_messages_sent:0
cluster_stats_messages_received:0

规划哈希槽位,一共16384个槽位,原来3个,现在加1个,4个,那每个槽位刚好4096个槽位
执行该步骤的命令为reshard重新为节点主机分片

add-node 添加新节点
check 对节点主机做检查
reshard 对节点主机重新分片
add-node –slave 添加从节点主机,如果不指定,则默认为主
del-node 删除节点主机

从节点都可以随意增删,但是主节点因为手里有哈希槽,所以不论时增加或者删除,都需要对哈希槽进行配置

进入部署了luster的主机去操作
进入源码目录包
[root@61 src]# pwd
/root/soft/redis/redis-4.0.8/src
添加第一台主机,没有加选项则默认为主
[root@61 src]# ./redis-trib.rb add-node 192.168.4.67:6357 192.168.4.61:6351
>>> Adding node 192.168.4.67:6357 to cluster 192.168.4.61:6351
>>> Performing Cluster Check (using node 192.168.4.61:6351)
M: fb2dc0d120eea527ccf2aedaa61503d58c5c7f80 192.168.4.61:6351
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: 56c7c875ec0ace4902f70c0dabeb401b0e60eeee 192.168.4.65:6355
slots: (0 slots) slave
replicates fb2dc0d120eea527ccf2aedaa61503d58c5c7f80
S: 4d672007b9e27119049efe94c798ca1b0ab7c7b3 192.168.4.64:6354
slots: (0 slots) slave
replicates 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8
M: 6fa6d44af120d0a4be5f0ba2d083b800b59be41a 192.168.4.62:6352
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: fbe8bb6550867ef98feb2e06238a96e1df85f1d7 192.168.4.66:6356
slots: (0 slots) slave
replicates 6fa6d44af120d0a4be5f0ba2d083b800b59be41a
M: 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8 192.168.4.63:6353
slots:10923-16383 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots…
>>> Check slots coverage…
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.4.67:6357 to make it join the cluster.
[OK] New node added correctly.

查一下刚刚新添加的主机的状态,可以看到有4台M(主机)了,但是还没有分配槽位
[root@61 src]# ./redis-trib.rb check 192.168.4.61:6351
>>> Performing Cluster Check (using node 192.168.4.61:6351)
M: fb2dc0d120eea527ccf2aedaa61503d58c5c7f80 192.168.4.61:6351
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: 56c7c875ec0ace4902f70c0dabeb401b0e60eeee 192.168.4.65:6355
slots: (0 slots) slave
replicates fb2dc0d120eea527ccf2aedaa61503d58c5c7f80
S: 4d672007b9e27119049efe94c798ca1b0ab7c7b3 192.168.4.64:6354
slots: (0 slots) slave
replicates 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8
M: 6fa6d44af120d0a4be5f0ba2d083b800b59be41a 192.168.4.62:6352
slots:5461-10922 (5462 slots) master
1 additional replica(s)
M: cd07e05296288cb1e5f8e901015783100624b91a 192.168.4.67:6357
slots: (0 slots) master
0 additional replica(s)
S: fbe8bb6550867ef98feb2e06238a96e1df85f1d7 192.168.4.66:6356
slots: (0 slots) slave
replicates 6fa6d44af120d0a4be5f0ba2d083b800b59be41a
M: 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8 192.168.4.63:6353
slots:10923-16383 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots…
>>> Check slots coverage…
[OK] All 16384 slots covered.

重新进行分片,分配新的哈希槽

[root@61 src]# ./redis-trib.rb reshard 192.168.4.61:6351
>>> Performing Cluster Check (using node 192.168.4.61:6351)
M: fb2dc0d120eea527ccf2aedaa61503d58c5c7f80 192.168.4.61:6351
slots:0-7509,10923-12969 (9557 slots) master
1 additional replica(s)
S: 56c7c875ec0ace4902f70c0dabeb401b0e60eeee 192.168.4.65:6355
slots: (0 slots) slave
replicates fb2dc0d120eea527ccf2aedaa61503d58c5c7f80
S: 4d672007b9e27119049efe94c798ca1b0ab7c7b3 192.168.4.64:6354
slots: (0 slots) slave
replicates 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8
M: 6fa6d44af120d0a4be5f0ba2d083b800b59be41a 192.168.4.62:6352
slots:7510-10922 (3413 slots) master
1 additional replica(s)
M: cd07e05296288cb1e5f8e901015783100624b91a 192.168.4.67:6357
slots: (0 slots) master
0 additional replica(s)
S: fbe8bb6550867ef98feb2e06238a96e1df85f1d7 192.168.4.66:6356
slots: (0 slots) slave
replicates 6fa6d44af120d0a4be5f0ba2d083b800b59be41a
M: 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8 192.168.4.63:6353
slots:12970-16383 (3414 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots…
>>> Check slots coverage…
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 4096
//问你需要挪多少哈希槽出来,挪4096个(上面算了的,一共16384,分4台主机,刚好一台4096个)
What is the receiving node ID? cd07e05296288cb1e5f8e901015783100624b91a
Please enter all the source node IDs.
Type ‘all’ to use all the nodes as source nodes for the hash slots.
Type ‘done’ once you entered all the source nodes IDs.
Source node #1:all
//问你从哪里挪,ALL就是所有的节点挪
Ready to move 4096 slots.
Source nodes:
M: fb2dc0d120eea527ccf2aedaa61503d58c5c7f80 192.168.4.61:6351
slots:0-7509,10923-12969 (9557 slots) master
1 additional replica(s)
M: 6fa6d44af120d0a4be5f0ba2d083b800b59be41a 192.168.4.62:6352
slots:7510-10922 (3413 slots) master
1 additional replica(s)
M: 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8 192.168.4.63:6353
slots:12970-16383 (3414 slots) master
1 additional replica(s)
Destination node:
M: cd07e05296288cb1e5f8e901015783100624b91a 192.168.4.67:6357
slots: (0 slots) master
0 additional replica(s)
Resharding plan:
Moving slot 0 from fb2dc0d120eea527ccf2aedaa61503d58c5c7f80
*********************************
Moving slot 8361 from 6fa6d44af120d0a4be5f0ba2d083b800b59be41a
Moving slot 8362 from 6fa6d44af120d0a4be5f0ba2d083b800b59be41a
Do you want to proceed with the proposed reshard plan (yes/no)? yes
问你上面这个挪的计划怎么样,可以就回答YES
Moving slot 0 from 192.168.4.61:6351 to 192.168.4.67:6357:
Moving slot 1 from 192.168.4.61:6351 to 192.168.4.67:6357:
****************************************
完成了检查一下

[root@61 src]# ./redis-trib.rb check 192.168.4.61:6351
>>> Performing Cluster Check (using node 192.168.4.61:6351)
M: fb2dc0d120eea527ccf2aedaa61503d58c5c7f80 192.168.4.61:6351
slots:4514-7509,10923-12969 (5043 slots) master
1 additional replica(s)
S: 56c7c875ec0ace4902f70c0dabeb401b0e60eeee 192.168.4.65:6355
slots: (0 slots) slave
replicates fb2dc0d120eea527ccf2aedaa61503d58c5c7f80
S: 4d672007b9e27119049efe94c798ca1b0ab7c7b3 192.168.4.64:6354
slots: (0 slots) slave
replicates 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8
M: 6fa6d44af120d0a4be5f0ba2d083b800b59be41a 192.168.4.62:6352
slots:0-1212,2390-4513,8363-10922,13823-14580 (6655 slots) master
1 additional replica(s)
M: cd07e05296288cb1e5f8e901015783100624b91a 192.168.4.67:6357
slots:1213-2389,7510-8362,12970-13822 (2883 slots) master
0 additional replica(s)
S: fbe8bb6550867ef98feb2e06238a96e1df85f1d7 192.168.4.66:6356
slots: (0 slots) slave
replicates 6fa6d44af120d0a4be5f0ba2d083b800b59be41a
M: 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8 192.168.4.63:6353
slots:14581-16383 (1803 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots…
>>> Check slots coverage…
[OK] All 16384 slots covered.

*************************************************************
可以发现改乱了,重新分配一下:
(其实就是第1步,规划一下,哪个节点要分多少哈希槽;
第2步,查看一下各个节点的哈希槽情况,其实就是看看哪台的少,然后算一下要挪多少过去补上;
第3步,就是挪哈希槽)
[root@61 src]# ./redis-trib.rb check 192.168.4.61:6351
>>> Performing Cluster Check (using node 192.168.4.61:6351)
M: fb2dc0d120eea527ccf2aedaa61503d58c5c7f80 192.168.4.61:6351
slots:5461-7509,10923-12969 (4096 slots) master
1 additional replica(s)
S: 56c7c875ec0ace4902f70c0dabeb401b0e60eeee 192.168.4.65:6355
slots: (0 slots) slave
replicates fb2dc0d120eea527ccf2aedaa61503d58c5c7f80
S: 4d672007b9e27119049efe94c798ca1b0ab7c7b3 192.168.4.64:6354
slots: (0 slots) slave
replicates 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8
M: 6fa6d44af120d0a4be5f0ba2d083b800b59be41a 192.168.4.62:6352
slots:3736-4513,8363-10922,13823-14580 (4096 slots) master
1 additional replica(s)
M: cd07e05296288cb1e5f8e901015783100624b91a 192.168.4.67:6357
slots:0-597,1213-2389,4514-4966,7510-8362,12970-13822,14581-14741 (4095 slots) master
0 additional replica(s)
S: fbe8bb6550867ef98feb2e06238a96e1df85f1d7 192.168.4.66:6356
slots: (0 slots) slave
replicates 6fa6d44af120d0a4be5f0ba2d083b800b59be41a
M: 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8 192.168.4.63:6353
slots:598-1212,2390-3735,4967-5460,14742-16383 (4097 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots…
>>> Check slots coverage…
[OK] All 16384 slots covered.
//经过不停的来回挪动,终于改得差不多了,最后2个,差别就1个,不改了,就这样吧

添加从节点
[root@61 src]# ./redis-trib.rb add-node –slave 192.168.4.68:6358 192.168.4.61:6351
>>> Adding node 192.168.4.68:6358 to cluster 192.168.4.61:6351
>>> Performing Cluster Check (using node 192.168.4.61:6351)
M: fb2dc0d120eea527ccf2aedaa61503d58c5c7f80 192.168.4.61:6351
slots:5461-7509,10923-12969 (4096 slots) master
1 additional replica(s)
S: 56c7c875ec0ace4902f70c0dabeb401b0e60eeee 192.168.4.65:6355
slots: (0 slots) slave
replicates fb2dc0d120eea527ccf2aedaa61503d58c5c7f80
S: 4d672007b9e27119049efe94c798ca1b0ab7c7b3 192.168.4.64:6354
slots: (0 slots) slave
replicates 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8
M: 6fa6d44af120d0a4be5f0ba2d083b800b59be41a 192.168.4.62:6352
slots:3736-4513,8363-10922,13823-14580 (4096 slots) master
1 additional replica(s)
M: cd07e05296288cb1e5f8e901015783100624b91a 192.168.4.67:6357
slots:0-597,1213-2389,4514-4966,7510-8362,12970-13822,14581-14741 (4095 slots) master
0 additional replica(s)
S: fbe8bb6550867ef98feb2e06238a96e1df85f1d7 192.168.4.66:6356
slots: (0 slots) slave
replicates 6fa6d44af120d0a4be5f0ba2d083b800b59be41a
M: 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8 192.168.4.63:6353
slots:598-1212,2390-3735,4967-5460,14742-16383 (4097 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots…
>>> Check slots coverage…
[OK] All 16384 slots covered.
Automatically selected master 192.168.4.67:6357
>>> Send CLUSTER MEET to node 192.168.4.68:6358 to make it join the cluster.
Waiting for the cluster to join.
>>> Configure node as replica of 192.168.4.67:6357.
[OK] New node added correctly.
//从上面的提示可以看到自动添加成为了67的从

最后检查一下添加的情况,验证68确实成为了67的从
[root@61 src]# ./redis-trib.rb check 192.168.4.61:6351
>>> Performing Cluster Check (using node 192.168.4.61:6351)
M: fb2dc0d120eea527ccf2aedaa61503d58c5c7f80 192.168.4.61:6351
slots:5461-7509,10923-12969 (4096 slots) master
1 additional replica(s)
S: 56c7c875ec0ace4902f70c0dabeb401b0e60eeee 192.168.4.65:6355
slots: (0 slots) slave
replicates fb2dc0d120eea527ccf2aedaa61503d58c5c7f80
S: 4d672007b9e27119049efe94c798ca1b0ab7c7b3 192.168.4.64:6354
slots: (0 slots) slave
replicates 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8
M: 6fa6d44af120d0a4be5f0ba2d083b800b59be41a 192.168.4.62:6352
slots:3736-4513,8363-10922,13823-14580 (4096 slots) master
1 additional replica(s)
M: cd07e05296288cb1e5f8e901015783100624b91a 192.168.4.67:6357
slots:0-597,1213-2389,4514-4966,7510-8362,12970-13822,14581-14741 (4095 slots) master
1 additional replica(s)
S: fbe8bb6550867ef98feb2e06238a96e1df85f1d7 192.168.4.66:6356
slots: (0 slots) slave
replicates 6fa6d44af120d0a4be5f0ba2d083b800b59be41a
M: 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8 192.168.4.63:6353
slots:598-1212,2390-3735,4967-5460,14742-16383 (4097 slots) master
1 additional replica(s)
S: f6d60b431050daac64c0a5c020dcf49a399d9cc6 192.168.4.68:6358
slots: (0 slots) slave
replicates cd07e05296288cb1e5f8e901015783100624b91a
[OK] All nodes agree about slots configuration.
>>> Check for open slots…
>>> Check slots coverage…
[OK] All 16384 slots covered.

移除节点:移除主节点和从节点
删除主节点很简单,不存在释放哈希槽的问题,所以直接del-node就行了

先登陆68,查看一下对应的ID是多少
[root@61 src]# redis-cli -c -h 192.168.4.68 -p 6358
192.168.4.68:6358> cluster node
(error) ERR Wrong CLUSTER subcommand or number of arguments
192.168.4.68:6358> cluster nodes
7b27e1369fb56a16f4b090d011ba291cf6a1dcf8 192.168.4.63:6353@16353 master – 0 1528446557000 11 connected 598-1212 2390-3735 4967-5460 14742-16383
fbe8bb6550867ef98feb2e06238a96e1df85f1d7 192.168.4.66:6356@16356 slave 6fa6d44af120d0a4be5f0ba2d083b800b59be41a 0 1528446559000 9 connected
6fa6d44af120d0a4be5f0ba2d083b800b59be41a 192.168.4.62:6352@16352 master – 0 1528446557000 9 connected 3736-4513 8363-10922 13823-14580
4d672007b9e27119049efe94c798ca1b0ab7c7b3 192.168.4.64:6354@16354 slave 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8 0 1528446558000 11 connected
f6d60b431050daac64c0a5c020dcf49a399d9cc6 192.168.4.68:6358@16358 myself,slave cd07e05296288cb1e5f8e901015783100624b91a 0 1528446553000 0 connected
cd07e05296288cb1e5f8e901015783100624b91a 192.168.4.67:6357@16357 master – 0 1528446559098 10 connected 0-597 1213-2389 4514-4966 7510-8362 12970-13822 14581-14741
56c7c875ec0ace4902f70c0dabeb401b0e60eeee 192.168.4.65:6355@16355 slave fb2dc0d120eea527ccf2aedaa61503d58c5c7f80 0 1528446559000 7 connected
fb2dc0d120eea527ccf2aedaa61503d58c5c7f80 192.168.4.61:6351@16351 master – 0 1528446559298 7 connected 5461-7509 10923-12969

********
//因为上面分片问题,暂时忽略,一会再重新分配一下

开始删除该从节点,然后check一下,确定删除成功
[root@61 src]# ./redis-trib.rb del-node 192.168.4.61:6351 f6d60b431050daac64c0a5c020dcf49a399d9cc6
>>> Removing node f6d60b431050daac64c0a5c020dcf49a399d9cc6 from cluster 192.168.4.61:6351
>>> Sending CLUSTER FORGET messages to the cluster…
>>> SHUTDOWN the node.
[root@61 src]# ./redis-trib.rb check 192.168.4.61:6351
>>> Performing Cluster Check (using node 192.168.4.61:6351)
M: fb2dc0d120eea527ccf2aedaa61503d58c5c7f80 192.168.4.61:6351
slots:5461-7509,10923-12969 (4096 slots) master
1 additional replica(s)
S: 56c7c875ec0ace4902f70c0dabeb401b0e60eeee 192.168.4.65:6355
slots: (0 slots) slave
replicates fb2dc0d120eea527ccf2aedaa61503d58c5c7f80
S: 4d672007b9e27119049efe94c798ca1b0ab7c7b3 192.168.4.64:6354
slots: (0 slots) slave
replicates 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8
M: 6fa6d44af120d0a4be5f0ba2d083b800b59be41a 192.168.4.62:6352
slots:3736-4513,8363-10922,13823-14580 (4096 slots) master
1 additional replica(s)
M: cd07e05296288cb1e5f8e901015783100624b91a 192.168.4.67:6357
slots:0-597,1213-2389,4514-4966,7510-8362,12970-13822,14581-14741 (4095 slots) master
0 additional replica(s)
S: fbe8bb6550867ef98feb2e06238a96e1df85f1d7 192.168.4.66:6356
slots: (0 slots) slave
replicates 6fa6d44af120d0a4be5f0ba2d083b800b59be41a
M: 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8 192.168.4.63:6353
slots:598-1212,2390-3735,4967-5460,14742-16383 (4097 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots…
>>> Check slots coverage…
[OK] All 16384 slots covered.

删除主节点,其实思路同上面重新分配哈希槽一样,先把哈希槽统统分给其它主节点,然后再删除
[root@61 src]# ./redis-trib.rb reshard 192.168.4.61:6351
>>> Performing Cluster Check (using node 192.168.4.61:6351)
M: fb2dc0d120eea527ccf2aedaa61503d58c5c7f80 192.168.4.61:6351
slots:6827-7509,10923-12969 (2730 slots) master
1 additional replica(s)
S: 56c7c875ec0ace4902f70c0dabeb401b0e60eeee 192.168.4.65:6355
slots: (0 slots) slave
replicates fb2dc0d120eea527ccf2aedaa61503d58c5c7f80
S: 4d672007b9e27119049efe94c798ca1b0ab7c7b3 192.168.4.64:6354
slots: (0 slots) slave
replicates 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8
M: 6fa6d44af120d0a4be5f0ba2d083b800b59be41a 192.168.4.62:6352
slots:8950-10922,13823-14580 (2731 slots) master
1 additional replica(s)
M: cd07e05296288cb1e5f8e901015783100624b91a 192.168.4.67:6357
slots:1979-2389,4514-4966,7510-8362,12970-13822,14581-14741 (2731 slots) master
0 additional replica(s)
S: fbe8bb6550867ef98feb2e06238a96e1df85f1d7 192.168.4.66:6356
slots: (0 slots) slave
replicates 6fa6d44af120d0a4be5f0ba2d083b800b59be41a
M: 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8 192.168.4.63:6353
slots:0-1978,2390-4513,4967-6826,8363-8949,14742-16383 (8192 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots…
>>> Check slots coverage…
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 2731
//注意,要看自己的案例,本次移动之前,我又做了多次移动,所以 67 这台主机,之剩下2731个哈希槽了,
所以只移动2731个
What is the receiving node ID? fb2dc0d120eea527ccf2aedaa61503d58c5c7f80
Please enter all the source node IDs.
Type ‘all’ to use all the nodes as source nodes for the hash slots.
Type ‘done’ once you entered all the source nodes IDs.
Source node #1:cd07e05296288cb1e5f8e901015783100624b91a
Source node #2:done

Ready to move 2731 slots.
Source nodes:
M: cd07e05296288cb1e5f8e901015783100624b91a 192.168.4.67:6357
slots:1979-2389,4514-4966,7510-8362,12970-13822,14581-14741 (2731 slots) master
0 additional replica(s)
Destination node:
M: fb2dc0d120eea527ccf2aedaa61503d58c5c7f80 192.168.4.61:6351
slots:6827-7509,10923-12969 (2730 slots) master
1 additional replica(s)
Resharding plan:
Moving slot 1979 from cd07e05296288cb1e5f8e901015783100624b91a
Moving slot 1980 from cd07e05296288cb1e5f8e901015783100624b91a
********************************************************

完成了以后重新确认一下,可以看到67没有槽位了,可以放心删除了
[root@61 src]# ./redis-trib.rb check 192.168.4.61:6351
>>> Performing Cluster Check (using node 192.168.4.61:6351)
M: fb2dc0d120eea527ccf2aedaa61503d58c5c7f80 192.168.4.61:6351
slots:1979-2389,4514-4966,6827-8362,10923-13822,14581-14741 (5461 slots) master
1 additional replica(s)
S: 56c7c875ec0ace4902f70c0dabeb401b0e60eeee 192.168.4.65:6355
slots: (0 slots) slave
replicates fb2dc0d120eea527ccf2aedaa61503d58c5c7f80
S: 4d672007b9e27119049efe94c798ca1b0ab7c7b3 192.168.4.64:6354
slots: (0 slots) slave
replicates 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8
M: 6fa6d44af120d0a4be5f0ba2d083b800b59be41a 192.168.4.62:6352
slots:8950-10922,13823-14580 (2731 slots) master
1 additional replica(s)
M: cd07e05296288cb1e5f8e901015783100624b91a 192.168.4.67:6357
slots: (0 slots) master
0 additional replica(s)
S: fbe8bb6550867ef98feb2e06238a96e1df85f1d7 192.168.4.66:6356
slots: (0 slots) slave
replicates 6fa6d44af120d0a4be5f0ba2d083b800b59be41a
M: 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8 192.168.4.63:6353
slots:0-1978,2390-4513,4967-6826,8363-8949,14742-16383 (8192 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots…
>>> Check slots coverage…
[OK] All 16384 slots covered.

开始移除主节点57
[root@61 src]# ./redis-trib.rb del-node 192.168.4.61:6351 cd07e05296288cb1e5f8e901015783100624b91a
>>> Removing node cd07e05296288cb1e5f8e901015783100624b91a from cluster 192.168.4.61:6351
>>> Sending CLUSTER FORGET messages to the cluster…
>>> SHUTDOWN the node.

确认一下移除操作,可以看到57已经没了
[root@61 src]# ./redis-trib.rb check 192.168.4.61:6351
>>> Performing Cluster Check (using node 192.168.4.61:6351)
M: fb2dc0d120eea527ccf2aedaa61503d58c5c7f80 192.168.4.61:6351
slots:1979-2389,4514-4966,6827-8362,10923-13822,14581-14741 (5461 slots) master
1 additional replica(s)
S: 56c7c875ec0ace4902f70c0dabeb401b0e60eeee 192.168.4.65:6355
slots: (0 slots) slave
replicates fb2dc0d120eea527ccf2aedaa61503d58c5c7f80
S: 4d672007b9e27119049efe94c798ca1b0ab7c7b3 192.168.4.64:6354
slots: (0 slots) slave
replicates 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8
M: 6fa6d44af120d0a4be5f0ba2d083b800b59be41a 192.168.4.62:6352
slots:8950-10922,13823-14580 (2731 slots) master
1 additional replica(s)
S: fbe8bb6550867ef98feb2e06238a96e1df85f1d7 192.168.4.66:6356
slots: (0 slots) slave
replicates 6fa6d44af120d0a4be5f0ba2d083b800b59be41a
M: 7b27e1369fb56a16f4b090d011ba291cf6a1dcf8 192.168.4.63:6353
slots:0-1978,2390-4513,4967-6826,8363-8949,14742-16383 (8192 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots…
>>> Check slots coverage…
[OK] All 16384 slots covered.

思考题

此条目发表在reids分类目录,贴了标签。将固定链接加入收藏夹。

发表回复

您的电子邮箱地址不会被公开。 必填项已用*标注