巨杉 Tech | SequoiaDB SQL实例高可用负载均衡实践
为了充分合理的利用服务器资源，提高数据服务的性能和稳定性，在较低成本的前提下，保证在部分服务器宕机或发生故障的情况下不影响业务的正常运作。本文主要介绍 Nginx+Keepalived 连接 SequoiaDB -MySQL 实例的高可用方案与实践。
SequoiaDB 巨杉数据库是一款完全自研的金融级分布式数据库产品，采用计算与存储分离架构，由数据库实例层和数据库存储引擎层组成。数据库实例层负责解析请求并转发至数据库存储引擎层处理，同时会将数据库存储引擎层的响应结果反馈给应用层，数据库实例层支持包括针对结构化数据的 MySQL 实例、PostgreSQL 实例、SparkSQL 实例，以及针对非结构化数据的 S3 和 PosixFS 文件系统的对象存储实例实例，而数据库存储引擎层是由 SequoiaDB 巨杉数据库的协调节点、编目节点和数据节点组成。该数据库集群架构能方便用户实现由传统数据库到巨杉数据库的无缝迁移，减少应用开发者的开发和学习成本。
如图所示是 SequoiaDB 巨杉数据库集群的逻辑架构图，用户由图可知，应用程序实际是通过计算层的实例节点连接、访问数据库。例如图中所示，应用程序可以直接通过 MySQL 的 JDBC 驱动连接、访问 SequoiaDB 数据库集群。
SequoiaDB 数据库的存储层是一个能够支持分布式事务、多中心容灾的分布式存储集群。
SequoiaDB 的分布式存储层，其中包括：协调节点、编目节点和数据库节点三类角色节点。分布式存储层中的所有节点均支持多节点部署，支持分布式水平扩展和高可用容灾功能。
在过去企业级解决方案中，为了解决数据库的高可用问题，一般会在数据库服务器与外部网络之间安装负载均衡器（如F5），从而避免因为某台服务器宕机而对业务造成影响。
虽然这种硬件负载均衡技术较为成熟，在过去的企业重要系统中均有部署使用，但该种方案需要额外购买价格高昂的负载均衡设备，成本比较高。
VRRP 技术可以将两台或者多台物理路由器设备虚拟成一个虚拟路由器，这个虚拟路由器通过虚拟IP（192.168.81.100）对外提供服务。
在虚拟路由器内部，同一时间只有一台物理路由器在对外提供服务，这台物理路由器被称为主路由器（Master），一般而言Master通过选举算法产生，它拥有对外服务的虚拟IP，提供各种网络功能。而其他物理路由器不拥有对外的虚拟IP，仅仅接收 Master 的 VRRP 状态通告信息，这部分路由器叫做备份路由器（Slave）。
当主路由器失效的时候，备份路由器重新进行选举，产生一个新的路由器成为 Master。如下图所示，当主路由器（Master）宕机后，备份路由器（Slave）进行选举成为Master，然后接管虚拟IP，继续对外提供服务，对应用程序服务器而言，整个切换过程是对应用程序透明，无任何影响。
Nginx (engine x) 是一个高性能的 HTTP 和反向代理 web 服务器，同时也提供了 IMAP/POP3/SMTP 服务。Nginx是一款轻量级的 Web 服务器/反向代理服务器及电子邮件（IMAP/POP3）代理服务器，在 BSD-like 协议下发行。其特点是占有内存少，并发能力强，事实上Nginx的并发能力在同类型的网页服务器中表现较好。
Nginx 有很强的代理功能，但是仅在一台服务器部署 Nginx 服务，依然存在 Nginx 单点故障的问题，因此在高可用负载均衡解决方案中，Nginx 需要结合 Keepalived 软件解决单点故障问题。
Nginx+Keepalived 双机实现 Nginx 反向代理服务的高可用，保证任意服务器宕机，或者是任何 Nginx 服务被意外停止，也会不影响业务的正常运作。
Keepalived 软件的作用是检测服务器的状态。
在部署了 Keepalived 服务的软件中，如果一台服务器意外宕机，或工作出现故障， Keepalived 软件将会检测到该问题服务器，并将有故障的服务器从系统中剔除。
同时，Keepalived 软件会自动使用其他服务器代替该问题服务器的工作。当问题服务器重新正常工作后，Keepalived 软件又会自动将原来问题服务器加入到服务器列表中。整个高可用切换工作全部由 Keepalived 软件自动感知，自动切换，无任何需人工干涉。
假设存在以下的服务器环境环境，后续内容将介绍如何通过 Keepalived + Nginx 软件实现 SequoiaDB 数据库的 MySQL 实例高可用部署：
检查 selinux 状态是否为 disabled
sestatus使用 root 权限，打开 /etc/selinux/config 文件
vi /etc/selinux/config将 SELINUX 调整成 disabled
# This file controls the state of SELinux on the system.# SELINUX= can take one of these three values:# enforcing - SELinux security policy is enforced.# permissive - SELinux prints warnings instead of enforcing.# disabled - No SELinux policy is loaded.SELINUX=disabled# SELINUXTYPE= can take one of three two values:# targeted - Targeted processes are protected,# minimum - Modification of targeted policy. Only selected processes are protected.# mls - Multi Level Security protection.SELINUXTYPE=targeted
rebootyum install -y gcc pcre pcre-devel openssl openssl-devel gd gd-devel net-snmp-agent-libs4.2.1 Nginx 安装
wget https://nginx.org/download/nginx-1.16.0.tar.gztar -zxvf nginx-1.16.0.tar.gzcd nginx-1.16.0./configure --prefix=/usr/local/nginx --with-streammake && make install
4.2.2 Nginx部署
vi /usr/local/nginx/conf/nginx.conf修改 Nginx 配置文件，新增以下内容，stream 段的配置要与 http 段在同级目录。新增内容表示将134、135和136 服务器的 3306 服务加入到 Nginx 的负载均衡中，并且 Nginx 对外发布 3307 的服务端口。
stream {upstream sequoiadb-mysql {server 192.168.81.134:3306 weight=1;server 192.168.81.135:3306 weight=1;server 192.168.81.136:3306 weight=1;}server {listen 3307;proxy_connect_timeout 5s;proxy_timeout 300s;proxy_pass sequoiadb-mysql;}}
ln -s /usr/local/nginx/sbin/nginx /usr/bin/nginxnginx -Vnginx version: nginx/1.16.0built by gcc 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC)configure arguments: --prefix=/usr/local/nginx
vi /etc/init.d/nginx#!/bin/sh# nginx - this script starts and stops the nginx daemin## chkconfig: - 85 15# description: Nginx is an HTTP(S) server, HTTP(S) reverse \# proxy and IMAP/POP3 proxy server# processname: nginx# config: /usr/local/nginx/conf/nginx.conf# pidfile: /usr/local/nginx/logs/nginx.pid# Source function library.. /etc/rc.d/init.d/functions# Source networking configuration.. /etc/sysconfig/network# Check that networking is up.[ "$NETWORKING" = "no" ] && exit 0nginx="/usr/local/nginx/sbin/nginx"prog=$(basename $nginx)NGINX_CONF_FILE="/usr/local/nginx/conf/nginx.conf"lockfile=/var/lock/subsys/nginxstart() {[ -x $nginx ] || exit 5[ -f $NGINX_CONF_FILE ] || exit 6echo -n $"Starting $prog: "daemon $nginx -c $NGINX_CONF_FILEretval=$?echo[ $retval -eq 0 ] && touch $lockfilereturn $retval}stop() {echo -n $"Stopping $prog: "killproc $prog -QUITretval=$?echo[ $retval -eq 0 ] && rm -f $lockfilereturn $retval}restart() {configtest || return $?stopstart}reload() {configtest || return $?echo -n $"Reloading $prog: "killproc $nginx -HUPRETVAL=$?echo}force_reload() {restart}configtest() {$nginx -t -c $NGINX_CONF_FILE}rh_status() {status $prog}rh_status_q() {rh_status >/dev/null 2>&1}case "$1" instart)rh_status_q && exit 0$1;;stop)rh_status_q || exit 0$1;;restart|configtest)$1;;reload)rh_status_q || exit 7$1;;force-reload)force_reload;;status)rh_status;;condrestart|try-restart)rh_status_q || exit 0;;*)echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"exit 2esac
chmod 755 /etc/init.d/nginxchkconfig --add nginxsystemctl start nginx.servicesystemctl enable nginx.servicesystemctl status nginx.service● nginx.service - SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy serverLoaded: loaded (/etc/rc.d/init.d/nginx; bad; vendor preset: disabled)Active: active (running) since Fri 2020-01-17 01:03:12 PST; 36s agoDocs: man:systemd-sysv-generator(8)Main PID: 407 (nginx)CGroup: /system.slice/nginx.service├─407 nginx: master process /usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf└─409 nginx: worker processJan 17 01:03:12 sdb2 systemd[1]: Starting SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server...Jan 17 01:03:12 sdb2 nginx[394]: Starting nginx: [ OK ]Jan 17 01:03:12 sdb2 systemd[1]: PID file /usr/local/nginx/logs/nginx.pid not readable (yet?) after start.Jan 17 01:03:12 sdb2 systemd[1]: Started SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server.
mysql -h 192.168.81.134 -uroot -P 3307 -prootmysql: [Warning] Using a password on the command line interface can be insecure.Welcome to the MySQL monitor. Commands end with ; or \g.Your MySQL connection id is 5Server version: 5.7.25 Source distributionCopyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql>
4.3.1 Keepalived 安装
wget https://www.keepalived.org/software/keepalived-2.0.19.tar.gztar -zxvf keepalived-2.0.19.tar.gzcd keepalived-2.0.19./configure --prefix=/usr/local/keepalivedmake && make install
4.3.2 Keepalived 部署
ln -s /usr/local/keepalived/sbin/keepalived /usr/bin/keepalivedkeepalived -vKeepalived v2.0.19 (10/19,2019)Copyright(C) 2001-2019 Alexandre Cassen, <[email protected]>Built with kernel headers for Linux 3.10.0Running on Linux 3.10.0-693.el7.x86_64 #1 SMP Thu Jul 6 19:56:57 EDT 2017configure options: --prefix=/usr/local/keepalivedConfig options: LVS VRRP VRRP_AUTH OLD_CHKSUM_COMPAT FIB_ROUTINGSystem options: PIPE2 SIGNALFD INOTIFY_INIT1 VSYSLOG EPOLL_CREATE1 IPV6_ADVANCED_API RTA_ENCAP RTA_EXPIRES FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK FRA_OIFNAME IFA_FLAGS IP_MULTICAST_ALL NET_LINUX_IF_H_COLLISION LIBIPTC_LINUX_NET_IF_H_COLLISION VRRP_VMAC IFLA_LINK_NETNSID CN_PROC SOCK_NONBLOCK SOCK_CLOEXEC O_PATH GLOB_BRACE INET6_ADDR_GEN_MODE SO_MARK SCHED_RT SCHED_RESET_ON_FORK
vi /usr/local/keepalived/nginx_check.sh#!/bin/bashA=`ps -C nginx --no-header |wc -l`if [ $A -eq 0 ];then/usr/sbin/nginxsleep 2if [ `ps -C nginx --no-header |wc -l` -eq 0 ];thenkillall keepalivedfifi
chmod +x /usr/local/keepalived/nginx_check.shvi /usr/local/keepalived/etc/keepalived/keepalived.conf! Configuration File for keepalivedglobal_defs {router_id sdb1}vrrp_script chk_nginx {script "/usr/local/keepalived/nginx_check.sh"interval 3weight -20}vrrp_instance VI_1 {state MASTERinterface ens33virtual_router_id 66priority 100advert_int 1authentication {auth_type PASSauth_pass 1111}track_script {chk_nginx}virtual_ipaddress {192.168.81.100}}
! Configuration File for keepalivedglobal_defs {router_id sdb2}vrrp_script chk_nginx {script "/usr/local/keepalived/nginx_check.sh"interval 3weight -20}vrrp_instance VI_1 {state BACKUPinterface ens33virtual_router_id 66priority 90advert_int 1authentication {auth_type PASSauth_pass 1111}track_script {chk_nginx}virtual_ipaddress {192.168.81.100}}
vi /usr/local/keepalived/etc/sysconfig/keepalivedKEEPALIVED_OPTIONS="-D -f /usr/local/keepalived/etc/keepalived/keepalived.conf"
systemctl start keepalived.servicesystemctl enable keepalived.servicesystemctl status keepalived.servicekeepalived.service - LVS and VRRP High Availability MonitorLoaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)Active: active (running) since Fri 2020-01-17 07:52:10 PST; 1min 45s agoProcess: 76416 ExecStart=/usr/local/keepalived/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)Main PID: 76417 (keepalived)CGroup: /system.slice/keepalived.service├─76417 /usr/local/keepalived/sbin/keepalived -D -f /usr/local/keepalived/etc/keepalived/keepalived.conf└─76418 /usr/local/keepalived/sbin/keepalived -D -f /usr/local/keepalived/etc/keepalived/keepalived.conf
ip addr show ens33inet 192.168.81.134/24 brd 192.168.81.255 scope global ens33valid_lft forever preferred_lft foreverinet 192.168.81.100/32 scope global ens33valid_lft forever preferred_lft forever
inet 192.168.81.135/24 brd 192.168.81.255 scope global ens33valid_lft forever preferred_lft forever
mysql -h 192.168.81.100 -uroot -P 3307 -prootmysql: [Warning] Using a password on the command line interface can be insecure.Welcome to the MySQL monitor. Commands end with ; or \g.Your MySQL connection id is 5Server version: 5.7.25 Source distributionCopyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql>
因为 sdb1 服务器部署的 Keepalived 为 Master，所以 Navicat 连接的是这台机器，通过执行 netstat 命令可以确认
[root@sdb1 ~]# netstat -nap|grep 3307tcp 0 0 0.0.0.0:3307 0.0.0.0:* LISTEN 1393/nginx: mastertcp 0 0 192.168.81.100:3307 192.168.81.1:53385 ESTABLISHED 1395/nginx: workertcp 0 0 192.168.81.100:3307 192.168.81.1:53380 ESTABLISHED 1395/nginx: worker
shutdown -h nowip addr show ens33inet 192.168.81.135/24 brd 192.168.81.255 scope global ens33valid_lft forever preferred_lft foreverinet 192.168.81.100/32 scope global ens33valid_lft forever preferred_lft forever
service sequoiasql-mysql stop根据上述步骤，目前部署了 Nginx+Keepalived 的为 sdb1 和 sdb2 两台服务器，其中 sdb1 服务器已经关机，而 sdb2 的 MySQL 实例服务已经停止，从而推断出 Navicat 是通过 sdb2 的 Nginx 连接到 sdb3 的 MySQL 实例，通过执行 netstat 命令可以确认
[root@sdb3 ~]# netstat -nap|grep 3306tcp6 0 0 :::3306 :::* LISTEN 1388/mysqldtcp6 0 0 192.168.81.136:3306 192.168.81.135:49644 ESTABLISHED 1388/mysqldtcp6 0 0 192.168.81.136:3306 192.168.81.135:49650 ESTABLISHED 1388/mysqldtcp6 0 0 192.168.81.136:3306 192.168.81.135:49646 ESTABLISHED 1388/mysqldunix 2 [ ACC ] STREAM LISTENING 22202 1388/mysqld /opt/sequoiasql/mysql/database/3306/mysqld.sock