到了弃用Redis-sentinel架构的时候了

vlambda
2020-04-24

到了弃用Redis-sentinel架构的时候了

redis5.0新特性

新的流数据类型(Stream data type)
新的 Redis 模块 API：定时器、集群和字典 API(Timers, Cluster and Dictionary APIs)
RDB 增加 LFU 和 LRU 信息
从 Ruby (redis-trib.rb)中将集群管理的C语言代码移植到了redis-cli 中
新的有序集合(sorted set)命令：ZPOPMIN/MAX 和阻塞变体(blocking variants)
升级 Active defragmentation 至 v2 版本
增强 HyperLogLog 的实现更好的内存统计报告许多包含子命令的命令现在都有一个 HELP 子命令
客户端频繁连接和断开连接时，性能表现更好许多
错误修复和其他方面的改进升级
Jemalloc 至 5.1 版本
引入 CLIENT UNBLOCK 和 CLIENT ID
新增 LOLWUT 命令 antirez.com/news/123
在不存在需要保持向后兼容性的地方，弃用 "slave" 术语
网络层中的差异优化
Lua 相关的改进
引入动态的 HZ(Dynamic HZ) 以平衡空闲 CPU 使用率和响应性
对 Redis 核心代码进行了重构并在许多方面进行了改进

服务的可用性

在集群分片的过程中，依旧可以提供部分的可用性，当集群一部分节点故障或者无法与其他节点通信,集群依旧可以运行
当集群中大部分主节点出现问题,集群将不可用
因为将哈希槽从一个节点移动到另一个节点不需要停止操作，所以添加和删除节点或更改节点持有的哈希槽的百分比都不需要停机
redis cluster可以通过增加N个redis slave提高集群的可用性以及健壮性

Redis集群中节点端口

服务客户端常规tcp端口6379
大端口,为tcp端口加上10000,是一个数据端口，作为集群总线，主要用于节点与节点之间通信通道的二进制协议，它的功用还有:故障检测，配置更新，故障转移授权

Redis集群与Docker

当前的Redis集群不支持NAT网络模式以及被重新映射的IP和端口。如果使用docker部署redis,要使用 --network=host

Redis-cluster 5.0 部署篇

编译安装Redis二进制文件

 
   
   
 
  
    
    
  mkdir redisworkdir && cd $_
  
    
    
  wget http://download.redis.io/releases/redis-5.0.8.tar.gz
  
    
    
  tar xf redis-5.0.8.tar.gz
  
    
    
  cd redis-5.0.8 && make # [option]此步之后可以选择make test.比较耗时,测试成功后会遇到以下信息。
  
    
    
  #######
  
    
    
  # 124 seconds - integration/replication-psync
  
    
    
  # 108 seconds - unit/obuf-limits
  
    
    
  # 98 seconds - unit/memefficiency
  
    
    
  # 162 seconds - integration/replication
  
    
    
  #
  
    
    
  #\o/ All tests passed without errors!
  
    
    
  #
  
    
    
  #Cleanup: may take some time... OK
  
    
    
  #make[1]: Leaving directory `/root/redis-5.0.8/src'
  
    
    
  cp src/redis-cli src/redis-server /usr/local/bin/

准备Redis实例的基础配置文件

 
   
   
 
  
    
    
  cat << EOF > redis-7000.conf
  
    
    
  port 7000
  
    
    
  daemonize yes
  
    
    
  dir "/data/redis/data"
  
    
    
  dbfilename "dump-7000.rdb"
  
    
    
  logfile "redis-7000.log"
  
    
    
  cluster-enabled yes
  
    
    
  cluster-config-file nodes-7000.conf
  
    
    
  EOF

生成多实例配置文件

我们使用Redis Cluster方案，建议需要准备最少6个实例.

 
   
   
 
  
    
    
  redis-server redis-7000.conf
  
    
    
  for i in `seq 7001 7005`
  
    
    
  do
  
    
    
   sed "s/7000/$i/g" redis-7000.conf > redis-"$i".conf
  
    
    
   redis-server redis-"$i".conf
  
    
    
  done
  
    
    
  

  
    
    
  [root@qiniu redis-5.0.8]# ps -ef |grep redis-server
  
    
    
  

  
    
    
  root 32284 1 0 21:24 ? 00:00:00 redis-server *:7000 [cluster]
  
    
    
  root 32291 1 0 21:24 ? 00:00:00 redis-server *:7001 [cluster]
  
    
    
  root 32297 1 0 21:24 ? 00:00:00 redis-server *:7002 [cluster]
  
    
    
  root 32303 1 0 21:24 ? 00:00:00 redis-server *:7003 [cluster]
  
    
    
  root 32309 1 0 21:24 ? 00:00:00 redis-server *:7004 [cluster]
  
    
    
  root 32312 1 0 21:24 ? 00:00:00 redis-server *:7005 [cluster]

此时进去某一个Redis节点中会发现集群下线不可用的状态

 
   
   
 
  
    
    
  [root@qiniu redis-5.0.8]# redis-cli -p 7000
  
    
    
  127.0.0.1:7000> set hello world
  
    
    
  (error) CLUSTERDOWN Hash slot not served
  
    
    
  127.0.0.1:7000> cluster info
  
    
    
  cluster_state:fail
  
    
    
  cluster_slots_assigned:0
  
    
    
  cluster_slots_ok:0
  
    
    
  cluster_slots_pfail:0
  
    
    
  cluster_slots_fail:0
  
    
    
  cluster_known_nodes:1
  
    
    
  cluster_size:0
  
    
    
  cluster_current_epoch:0
  
    
    
  cluster_my_epoch:0
  
    
    
  cluster_stats_messages_sent:0
  
    
    
  cluster_stats_messages_received:0
  
    
    
  127.0.0.1:7000> cluster nodes
  
    
    
  b1019e6e716125f14def5f35fac2c7024324a8ec :7000@17000 myself,master - 0 0 0 connected

Cluster节点主要的配置

 
   
   
 
  
    
    
  cluster-enabled yes
  
    
    
  cluster-node-timeout 15000
  
    
    
  cluster-config-file "nodes.conf"
  
    
    
  clutser-require-full-coverage yes # 当集群中出现一个节点不合理,是否就标识集群不可用,一般配置为no

meet集群节点

 
   
   
 
  
    
    
  for i in `seq 7001 7005`
  
    
    
  do
  
    
    
   redis-cli -h 127.0.0.1 -p 7000 cluster meet 127.0.0.1 "$i"
  
    
    
  done

当连接到7002端口时候，就会发现redis-7002发现了7000和7001

 
   
   
 
  
    
    
  redis-cli -h 127.0.0.1 -p 7002 cluster nodes #这里会有6个节点
  
    
    
  redis-cli -h 127.0.0.1 -p 7005 cluster info # 这里的cluster_know_nodes:6

分配槽位

 
   
   
 
  
    
    
  #!/bin/sbash
  
    
    
  start=$1
  
    
    
  end=$2
  
    
    
  port=$3
  
    
    
  for slot in `seq ${start} ${end}`
  
    
    
  do
  
    
    
   echo "slot:${slot}"
  
    
    
   redis-cli -p ${port} cluster addslots ${slot}
  
    
    
  done

 
   
   
 
  
    
    
  chmod +x addslots.sh
  
    
    
  ./addslots.sh 0 5460 7000
  
    
    
  ./addslots.sh 5461 10922 7001
  
    
    
  ./addslots.sh 10923 16383 7002
  
    
    
  

  
    
    
  ...

通过NodeID设置主从关系

 
   
   
 
  
    
    
  [root@qiniu redis-5.0.8]# redis-cli -h 127.0.0.1 -p 7000 cluster nodes
  
    
    
  c28ad47247c7172faeddeebbfbbbcb7bf9f151b0 127.0.0.1:7002@17002 master - 0 1585835343000 4 connected 10923-16383
  
    
    
  5260ee3591f0a43698d71735dd57a65efb16dd2f 127.0.0.1:7004@17004 master - 0 1585835342000 3 connected
  
    
    
  f68ef79fbed0b2faa59227037f99d09ac32295dc 127.0.0.1:7005@17005 master - 0 1585835342615 5 connected
  
    
    
  df55c705abff3cc905eefadbc5332a9a59af9597 127.0.0.1:7003@17003 master - 0 1585835343617 2 connected
  
    
    
  fb1026c49a5cccd2d18310a5bafd67433538bc78 127.0.0.1:7001@17001 master - 0 1585835341000 1 connected 5461-10922
  
    
    
  fdd63d2dddddbca505b824685f7a03e08d88558b 127.0.0.1:7000@17000 myself,master - 0 1585835342000 0 connected 0-5460
  
    
    
  redis-cli -h 127.0.0.1 -p 7003 cluster replicate fdd63d2dddddbca505b824685f7a03e08d88558b
  
    
    
  redis-cli -h 127.0.0.1 -p 7004 cluster replicate fb1026c49a5cccd2d18310a5bafd67433538bc78
  
    
    
  redis-cli -h 127.0.0.1 -p 7005 cluster replicate c28ad47247c7172faeddeebbfbbbcb7bf9f151b0
  
    
    
  

  
    
    
  [root@qiniu redis-5.0.8]# redis-cli -h 127.0.0.1 -p 7000 cluster info
  
    
    
  cluster_state:ok
  
    
    
  cluster_slots_assigned:16384
  
    
    
  cluster_slots_ok:16384
  
    
    
  cluster_slots_pfail:0
  
    
    
  cluster_slots_fail:0
  
    
    
  cluster_known_nodes:6
  
    
    
  cluster_size:3
  
    
    
  cluster_current_epoch:5
  
    
    
  cluster_my_epoch:0
  
    
    
  cluster_stats_messages_ping_sent:577
  
    
    
  cluster_stats_messages_pong_sent:558
  
    
    
  cluster_stats_messages_meet_sent:5
  
    
    
  cluster_stats_messages_sent:1140
  
    
    
  cluster_stats_messages_ping_received:558
  
    
    
  cluster_stats_messages_pong_received:582
  
    
    
  cluster_stats_messages_received:1140
  
    
    
  

  
    
    
  [root@qiniu redis-5.0.8]# redis-cli -h 127.0.0.1 -p 7000 cluster nodes
  
    
    
  c28ad47247c7172faeddeebbfbbbcb7bf9f151b0 127.0.0.1:7002@17002 master - 0 1585835417764 4 connected 10923-16383
  
    
    
  5260ee3591f0a43698d71735dd57a65efb16dd2f 127.0.0.1:7004@17004 slave fb1026c49a5cccd2d18310a5bafd67433538bc78 0 1585835415762 3 connected
  
    
    
  f68ef79fbed0b2faa59227037f99d09ac32295dc 127.0.0.1:7005@17005 slave c28ad47247c7172faeddeebbfbbbcb7bf9f151b0 0 1585835418765 5 connected
  
    
    
  df55c705abff3cc905eefadbc5332a9a59af9597 127.0.0.1:7003@17003 slave fdd63d2dddddbca505b824685f7a03e08d88558b 0 1585835417000 2 connected
  
    
    
  fb1026c49a5cccd2d18310a5bafd67433538bc78 127.0.0.1:7001@17001 master - 0 1585835419767 1 connected 5461-10922
  
    
    
  fdd63d2dddddbca505b824685f7a03e08d88558b 127.0.0.1:7000@17000 myself,master - 0 1585835418000 0 connected 0-5460
  
    
    
  

  
    
    
  [root@qiniu redis-5.0.8]# redis-cli -h 127.0.0.1 -p 7000 cluster slots
  
    
    
  1) 1) (integer) 10923
  
    
    
   1) (integer) 16383
  
    
    
   2) 1) "127.0.0.1"
  
    
    
   1) (integer) 7002
  
    
    
   2) "c28ad47247c7172faeddeebbfbbbcb7bf9f151b0"
  
    
    
   3) 1) "127.0.0.1"
  
    
    
   1) (integer) 7005
  
    
    
   2) "f68ef79fbed0b2faa59227037f99d09ac32295dc"
  
    
    
  2) 1) (integer) 5461
  
    
    
   1) (integer) 10922
  
    
    
   2) 1) "127.0.0.1"
  
    
    
   1) (integer) 7001
  
    
    
   2) "fb1026c49a5cccd2d18310a5bafd67433538bc78"
  
    
    
   3) 1) "127.0.0.1"
  
    
    
   1) (integer) 7004
  
    
    
   2) "5260ee3591f0a43698d71735dd57a65efb16dd2f"
  
    
    
  3) 1) (integer) 0
  
    
    
   1) (integer) 5460
  
    
    
   2) 1) "127.0.0.1"
  
    
    
   1) (integer) 7000
  
    
    
   2) "fdd63d2dddddbca505b824685f7a03e08d88558b"
  
    
    
   3) 1) "127.0.0.1"
  
    
    
   1) (integer) 7003
  
    
    
   2) "df55c705abff3cc905eefadbc5332a9a59af9597"

以上就完成了三主三从的Redis集群配置

redis cluster测试

通过go-redis简单地测试

 
   
   
 
  
    
    
  package main
  
    
    
  

  
    
    
  import (
  
    
    
   "fmt"
  
    
    
   "github.com/go-redis/redis"
  
    
    
   "strconv"
  
    
    
  )
  
    
    
  

  
    
    
  func main() {
  
    
    
   //redis cluster
  
    
    
   rdb := redis.NewClusterClient(&redis.ClusterOptions{
  
    
    
   Addrs: []string{"192.168.99.128:7000", "192.168.99.129:7001", "192.168.99.130:7002", "192.168.99.130:7003", "192.168.99.129:7004", "192.168.99.128:7005"},
  
    
    
   Password: "passwd!s",
  
    
    
   MaxRetries: 4,
  
    
    
   })
  
    
    
   for i := 0; i < 1000; i++ {
  
    
    
   key := "test" + strconv.Itoa(i)
  
    
    
   err := rdb.Set(key, "set key from golang", 0).Err()
  
    
    
   //rdb.Process()
  
    
    
   if err != nil {
  
    
    
   panic(err)
  
    
    
   }
  
    
    
   }
  
    
    
   fmt.Println("insert data done.")
  
    
    
   for i := 500; i < 1000; i++ {
  
    
    
   keys := "test" + strconv.Itoa(i)
  
    
    
   fmt.Println("keys: ", rdb.ClusterKeySlot(keys))
  
    
    
   }
  
    
    
  }

官方方式部署redis-cluster

在Redis 5.0 之后可以通过Redis-cli实现集群的部署与安装

 
   
   
 
  
    
    
  # 首先我们需要提前准备好这些redis实例
  
    
    
  [root@qiniu ~]# redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 --cluster-replicas 1
  
    
    
  >>> Performing hash slots allocation on 6 nodes...
  
    
    
  Master[0] -> Slots 0 - 5460
  
    
    
  Master[1] -> Slots 5461 - 10922
  
    
    
  Master[2] -> Slots 10923 - 16383
  
    
    
  Adding replica 127.0.0.1:7004 to 127.0.0.1:7000
  
    
    
  Adding replica 127.0.0.1:7005 to 127.0.0.1:7001
  
    
    
  Adding replica 127.0.0.1:7003 to 127.0.0.1:7002
  
    
    
  >>> Trying to optimize slaves allocation for anti-affinity
  
    
    
  [WARNING] Some slaves are in the same host as their master
  
    
    
  M: ab3f9f295905d3ff113c0cfebc3000ae8d0eb717 127.0.0.1:7000
  
    
    
   slots:[0-5460] (5461 slots) master
  
    
    
  M: a22ca7f0c97d86a47897694a8e1f48a29702ae72 127.0.0.1:7001
  
    
    
   slots:[5461-10922] (5462 slots) master
  
    
    
  M: 5f08b2ea3c26ae39843b0bf964bac5a8596e8cda 127.0.0.1:7002
  
    
    
   slots:[10923-16383] (5461 slots) master
  
    
    
  S: 3712bc46f0fcc14ef4fa8044ad319eee1ddc56c9 127.0.0.1:7003
  
    
    
   replicates 5f08b2ea3c26ae39843b0bf964bac5a8596e8cda
  
    
    
  S: a34cf5c793e3e606f6fe0c2834d9660cd4b7dcf0 127.0.0.1:7004
  
    
    
   replicates ab3f9f295905d3ff113c0cfebc3000ae8d0eb717
  
    
    
  S: 88ecb048427276a287504e124ef78dd5b7a6e183 127.0.0.1:7005
  
    
    
   replicates a22ca7f0c97d86a47897694a8e1f48a29702ae72
  
    
    
  Can I set the above configuration? (type 'yes' to accept): yes
  
    
    
  >>> Nodes configuration updated
  
    
    
  >>> Assign a different config epoch to each node
  
    
    
  >>> Sending CLUSTER MEET messages to join the cluster
  
    
    
  Waiting for the cluster to join
  
    
    
  ...
  
    
    
  >>> Performing Cluster Check (using node 127.0.0.1:7000)
  
    
    
  M: ab3f9f295905d3ff113c0cfebc3000ae8d0eb717 127.0.0.1:7000
  
    
    
   slots:[0-5460] (5461 slots) master
  
    
    
   1 additional replica(s)
  
    
    
  S: 88ecb048427276a287504e124ef78dd5b7a6e183 127.0.0.1:7005
  
    
    
   slots: (0 slots) slave
  
    
    
   replicates a22ca7f0c97d86a47897694a8e1f48a29702ae72
  
    
    
  S: a34cf5c793e3e606f6fe0c2834d9660cd4b7dcf0 127.0.0.1:7004
  
    
    
   slots: (0 slots) slave
  
    
    
   replicates ab3f9f295905d3ff113c0cfebc3000ae8d0eb717
  
    
    
  S: 3712bc46f0fcc14ef4fa8044ad319eee1ddc56c9 127.0.0.1:7003
  
    
    
   slots: (0 slots) slave
  
    
    
   replicates 5f08b2ea3c26ae39843b0bf964bac5a8596e8cda
  
    
    
  M: 5f08b2ea3c26ae39843b0bf964bac5a8596e8cda 127.0.0.1:7002
  
    
    
   slots:[10923-16383] (5461 slots) master
  
    
    
   1 additional replica(s)
  
    
    
  M: a22ca7f0c97d86a47897694a8e1f48a29702ae72 127.0.0.1:7001
  
    
    
   slots:[5461-10922] (5462 slots) master
  
    
    
   1 additional replica(s)
  
    
    
  [OK] All nodes agree about slots configuration.
  
    
    
  >>> Check for open slots...
  
    
    
  >>> Check slots coverage...
  
    
    
  [OK] All 16384 slots covered.
  
    
    
  [root@qiniu ~]# redis-cli -c -p 7000
  
    
    
  127.0.0.1:7000> cluster info
  
    
    
  cluster_state:ok
  
    
    
  cluster_slots_assigned:16384
  
    
    
  cluster_slots_ok:16384
  
    
    
  cluster_slots_pfail:0
  
    
    
  cluster_slots_fail:0
  
    
    
  cluster_known_nodes:6
  
    
    
  cluster_size:3
  
    
    
  cluster_current_epoch:6
  
    
    
  cluster_my_epoch:1
  
    
    
  cluster_stats_messages_ping_sent:68
  
    
    
  cluster_stats_messages_pong_sent:69
  
    
    
  cluster_stats_messages_sent:137
  
    
    
  cluster_stats_messages_ping_received:64
  
    
    
  cluster_stats_messages_pong_received:68
  
    
    
  cluster_stats_messages_meet_received:5
  
    
    
  cluster_stats_messages_received:137
  
    
    
  127.0.0.1:7000> cluster nodes
  
    
    
  88ecb048427276a287504e124ef78dd5b7a6e183 127.0.0.1:7005@17005 slave a22ca7f0c97d86a47897694a8e1f48a29702ae72 0 1585820315000 6 connected
  
    
    
  a34cf5c793e3e606f6fe0c2834d9660cd4b7dcf0 127.0.0.1:7004@17004 slave ab3f9f295905d3ff113c0cfebc3000ae8d0eb717 0 1585820312475 5 connected
  
    
    
  3712bc46f0fcc14ef4fa8044ad319eee1ddc56c9 127.0.0.1:7003@17003 slave 5f08b2ea3c26ae39843b0bf964bac5a8596e8cda 0 1585820316484 4 connected
  
    
    
  5f08b2ea3c26ae39843b0bf964bac5a8596e8cda 127.0.0.1:7002@17002 master - 0 1585820315000 3 connected 10923-16383
  
    
    
  a22ca7f0c97d86a47897694a8e1f48a29702ae72 127.0.0.1:7001@17001 master - 0 1585820315481 2 connected 5461-10922
  
    
    
  ab3f9f295905d3ff113c0cfebc3000ae8d0eb717 127.0.0.1:7000@17000 myself,master - 0 1585820314000 1 connected 0-5460
  
    
    
  127.0.0.1:7000> 
  
    
    
  127.0.0.1:7000> set hello redis
  
    
    
  OK
  
    
    
  127.0.0.1:7000> exit
  
    
    
  [root@qiniu ~]# redis-cli -c -p 7001
  
    
    
  127.0.0.1:7001> get hello
  
    
    
  -> Redirected to slot [866] located at 127.0.0.1:7000
  
    
    
  "redis"
  
    
    
  127.0.0.1:7000>

vlambda博客
学习文章列表

到了弃用Redis-sentinel架构的时候了

redis5.0新特性

服务的可用性

Redis集群中节点端口

Redis集群与Docker

Redis-cluster 5.0 部署篇

编译安装Redis二进制文件

准备Redis实例的基础配置文件

生成多实例配置文件

meet集群节点

分配槽位

通过NodeID设置主从关系

redis cluster测试

通过go-redis简单地测试

官方方式部署redis-cluster

标签:

推荐阅读

相关文章

vlambda博客 学习文章列表

到了弃用Redis-sentinel架构的时候了

redis5.0新特性

服务的可用性

Redis集群中节点端口

Redis集群与Docker

Redis-cluster 5.0 部署篇

编译安装Redis二进制文件

准备Redis实例的基础配置文件

生成多实例配置文件

meet集群节点

分配槽位

通过NodeID设置主从关系

redis cluster测试

通过go-redis简单地测试

官方方式部署redis-cluster

标签:

推荐阅读

相关文章

vlambda博客
学习文章列表