Fork me on GitHub

LVS高可用实战之DR模式(CentOS 7)

发表于 |
字数统计: 3,134

环境

VirtualBox虚拟机3台，使用的网络有桥接网卡、网络地址转换(NAT)

桥接网卡网段是10.1.80.0/24，网络地址转换(NAT)是为了能访问internet以便安装软件

节点 IP VIP
ds1 10.1.80.91 enp0s3
ds2 10.1.80.92 enp0s3
rs1 10.1.80.93 lo:0
rs2 10.1.80.95 lo:0

VIP地址为10.1.80.90

RS的RIP可以使用私网或公网IP， 文中DS与RS用的是同一网段。实际场景，一般情况下RS的RIP与VIP是不同网段的。

LVS常用名词

  • Load Balancer：负载均衡器，运行LVS负责负载均衡的服务器
  • DS：Director Server，指的是前端负载均衡器节点，也就是运行LVS的服务器；
  • RS：Real Server，后端真实的工作服务器；
  • VIP：Virtual Server IP，向外部直接面向用户请求，作为用户请求的目标的IP地址，一般也是DS的外部IP地址；
  • DIP：Director Server IP，主要用于和内部主机通讯的IP地址，一般也是DS的内部IP地址；
  • RIP：Real Server IP，后端服务器的IP地址；
  • CIP：Client IP，访问客户端的IP地址；

Director Server

ds1

安装ipvsadm

1
sudo yum install -y ipvsadm

安装keepalived

1
sudo yum install -y keepalived

备份keepalived.conf配置文件

1
sudo cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak

1
sudo vi /etc/keepalived/keepalived.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

global_defs {
   router_id LVS_DEVEL
}

vrrp_instance VI_1 {
    state MASTER
    interface enp0s3
    virtual_router_id 51
    priority 101
    advert_int 1
    
    # 如果两节点的上联交换机禁用了组播，则采用 vrrp 单播通告的方式
    # unicast_src_ip 10.1.80.91
    # unicast_peer {
    #    10.1.80.92
    # }
    
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.1.80.90
    }
}

virtual_server 10.1.80.90 80 {
    delay_loop 6
    lb_algo wrr
    lb_kind DR
    persistence_timeout 5
    protocol TCP

    real_server 10.1.80.93 80 {
        weight 100
        TCP_CHECK {
           connect_timeout 3
        }
    }


    real_server 10.1.80.95 80 {
        weight 100
        TCP_CHECK {
           connect_timeout 3
        }
    }

}

ds1作为MASTER优先级比ds2高，ds1设置priority 101，ds2设置priority 100

重启keepalived

1
sudo systemctl restart keepalived

防火墙开放80端口

1
sudo firewall-cmd --zone=public --add-port=80/tcp --permanent

防火墙允许vrrp的组播

1
2
3
sudo firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --in-interface enp0s3 --destination 224.0.0.18 --protocol vrrp -j ACCEPT

sudo firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0 --out-interface enp0s3 --destination 224.0.0.18 --protocol vrrp -j ACCEPT

防火墙重新加载

1
sudo firewall-cmd --reload

ds2

安装ipvsadm

1
sudo yum install -y ipvsadm

安装keepalived

1
sudo yum install -y keepalived

备份keepalived.conf配置文件

1
sudo cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak

1
sudo vi /etc/keepalived/keepalived.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

global_defs {
   router_id LVS_DEVEL
}

vrrp_instance VI_1 {
    state BACKUP
    interface enp0s3
    virtual_router_id 51
    priority 100
    advert_int 1
    
    # 如果两节点的上联交换机禁用了组播，则采用 vrrp 单播通告的方式
    # unicast_src_ip 10.1.80.92
    # unicast_peer {
    #    10.1.80.91
    # }
    
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.1.80.90
    }
}

virtual_server 10.1.80.90 80 {
    delay_loop 6
    lb_algo wrr
    lb_kind DR
    persistence_timeout 5
    protocol TCP

    real_server 10.1.80.93 80 {
        weight 100
        TCP_CHECK {
           connect_timeout 3
        }
    }


    real_server 10.1.80.95 80 {
        weight 100
        TCP_CHECK {
           connect_timeout 3
        }
    }

}

重启keepalived

1
sudo systemctl restart keepalived

防火墙开放80端口

1
sudo firewall-cmd --zone=public --add-port=80/tcp --permanent

防火墙允许vrrp的组播

1
2
3
sudo firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --in-interface enp0s3 --destination 224.0.0.18 --protocol vrrp -j ACCEPT

sudo firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0 --out-interface enp0s3 --destination 224.0.0.18 --protocol vrrp -j ACCEPT

防火墙重新加载

1
sudo firewall-cmd --reload

Real Server

rs1

1
sudo mkdir -p /opt/scripts
1
sudo vi /opt/scripts/lvs_rs.sh

脚本如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
#!/bin/bash 
VIP=10.1.80.90
case "$1" in
start)
       ifconfig lo:0 $VIP netmask 255.255.255.255 broadcast $VIP
       /sbin/route add -host $VIP dev lo:0
       echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
       echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
       echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
       echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
       sysctl -p >/dev/null 2>&1
       echo "RealServer Start OK"
       ;;
stop)
       ifconfig lo:0 down
       route del $VIP >/dev/null 2>&1
       echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
       echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
       echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
       echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
       echo "RealServer Stoped"
       ;;
*)
       echo "Usage: $0 {start|stop}"
       exit 1
esac
exit 0

1
2
sudo chmod +x /opt/scripts/lvs_rs.sh 
sudo /opt/scripts/lvs_rs.sh start

安装tengine

1
2
3
4
5
6
7
sudo mkdir -p /tmp/software
sudo cd /tmp/software
sudo curl -O https://tengine.taobao.org/download/tengine-2.3.3.tar.gz
sudo tar -xzvf tengine-2.3.3.tar.gz
sudo cd tengine-2.3.3
sudo yum install -y gcc make pcre-devel openssl-devel
sudo ./configure --prefix=/usr/local/tengine

修改index.html

1
sudo vi /usr/local/tengine/html/index.html

1
2
3
其余省略...
<h1>Welcome to tengine! I'm rs1</h1>
其余省略...

启动tengine

1
sudo /usr/local/tengine/sbin/nginx

防火墙设置开放80端口

1
2
firewall-cmd --zone=public --add-port=80/tcp --permanent
firewall-cmd --reload

rs2

1
sudo mkdir -p /opt/scripts
1
sudo vi /opt/scripts/lvs_rs.sh

脚本如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
#!/bin/bash 
VIP=10.1.80.90
case "$1" in
start)
       ifconfig lo:0 $VIP netmask 255.255.255.255 broadcast $VIP
       /sbin/route add -host $VIP dev lo:0
       echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
       echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
       echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
       echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
       sysctl -p >/dev/null 2>&1
       echo "RealServer Start OK"
       ;;
stop)
       ifconfig lo:0 down
       route del $VIP >/dev/null 2>&1
       echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
       echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
       echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
       echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
       echo "RealServer Stoped"
       ;;
*)
       echo "Usage: $0 {start|stop}"
       exit 1
esac
exit 0

1
2
sudo chmod +x /opt/scripts/lvs_rs.sh 
sudo /opt/scripts/lvs_rs.sh start

安装tengine

1
2
3
4
5
6
7
sudo mkdir -p /tmp/software
sudo cd /tmp/software
sudo curl -O https://tengine.taobao.org/download/tengine-2.3.3.tar.gz
sudo tar -xzvf tengine-2.3.3.tar.gz
sudo cd tengine-2.3.3
sudo yum install -y gcc make pcre-devel openssl-devel
sudo ./configure --prefix=/usr/local/tengine

修改index.html

1
sudo vi /usr/local/tengine/html/index.html

1
2
3
其余省略...
<h1>Welcome to tengine! I'm rs2</h1>
其余省略...

启动tengine

1
sudo /usr/local/tengine/sbin/nginx

防火墙设置开放80端口

1
2
firewall-cmd --zone=public --add-port=80/tcp --permanent
firewall-cmd --reload

测试keepalived vrrp报文

1
sudo tcpdump -vvv -n -i enp0s3 host 224.0.0.18

在ds1上执行(ds2,rs1,rs2都可以的)

1
2
3
4
14:43:04.752665 IP (tos 0xc0, ttl 255, id 130, offset 0, flags [none], proto VRRP (112), length 40)
    10.1.80.91 > 224.0.0.18: vrrp 10.1.80.91 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 101, authtype simple, intvl 1s, length 20, addrs: 10.1.80.90 auth "1111^@^@^@^@"
14:43:05.753059 IP (tos 0xc0, ttl 255, id 131, offset 0, flags [none], proto VRRP (112), length 40)
    10.1.80.91 > 224.0.0.18: vrrp 10.1.80.91 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 101, authtype simple, intvl 1s, length 20, addrs: 10.1.80.90 auth "1111^@^@^@^@"

可以看到10.1.80.91在发组播消息，10.1.80.91证明我是MASTER，vip(10.1.80.90)在ds1的enp0s3上。
然后把ds1的keepalived关了,sudo systemctl stop keepalived

观察报文变化

1
2
3
4
14:45:08.488190 IP (tos 0xc0, ttl 255, id 253, offset 0, flags [none], proto VRRP (112), length 40)
    10.1.80.91 > 224.0.0.18: vrrp 10.1.80.91 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 0, authtype simple, intvl 1s, length 20, addrs: 10.1.80.90 auth "1111^@^@^@^@"
14:45:09.120876 IP (tos 0xc0, ttl 255, id 8772, offset 0, flags [none], proto VRRP (112), length 40)
    10.1.80.92 > 224.0.0.18: vrrp 10.1.80.92 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20, addrs: 10.1.80.90 auth "1111^@^@^@^@"

发现已经由10.1.80.91改为10.1.80.92发组播消息,vip(10.1.80.90)飘到了ds2的enp0s3上。

重新启动ds1上的keepalived,再次观察报文

1
2
3
4
14:46:59.177236 IP (tos 0xc0, ttl 255, id 8880, offset 0, flags [none], proto VRRP (112), length 40)
    10.1.80.92 > 224.0.0.18: vrrp 10.1.80.92 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20, addrs: 10.1.80.90 auth "1111^@^@^@^@"
14:46:59.177350 IP (tos 0xc0, ttl 255, id 1, offset 0, flags [none], proto VRRP (112), length 40)
    10.1.80.91 > 224.0.0.18: vrrp 10.1.80.91 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 101, authtype simple, intvl 1s, length 20, addrs: 10.1.80.90 auth "1111^@^@^@^@"

发现由10.1.80.92改为了10.1.80.91发组播消息了，vip(10.1.80.90)飘回了ds1的enp0s3上。

验证

页面展示 Welcome to tengine! I’m rs1

ds1上查看sudo ipvsadm -ln

1
2
3
4
5
6
7
[jaychang@ds01 ~]# sudo ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.1.80.90:80 wrr persistent 50
  -> 10.1.80.93:80              Route   100    0          0
  -> 10.1.80.95:80              Route   100    2          0
  • 停掉rs1的nginx后

ds1上查看/var/log/keepalived.log(如果未设置过，默认是在/var/log/message)

1
2
3
4
Oct  8 15:50:14 ds01 Keepalived_healthcheckers[15444]: TCP connection to [10.1.80.93]:80 failed.
Oct  8 15:50:15 ds01 Keepalived_healthcheckers[15444]: TCP connection to [10.1.80.93]:80 failed.
Oct  8 15:50:15 ds01 Keepalived_healthcheckers[15444]: Check on service [10.1.80.93]:80 failed after 1 retry.
Oct  8 15:50:15 ds01 Keepalived_healthcheckers[15444]: Removing service [10.1.80.93]:80 from VS [10.1.80.90]:80

从日志内容可以看到10.1.80.93已被剔除了

ds1上查看sudo ipvsadm -ln

1
2
3
4
5
6
[jaychang@ds01 ~]$ sudo ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.1.80.90:80 wrr persistent 50
  -> 10.1.80.95:80              Route   100    4          0

刷新浏览器页面显示Welcome to tengine! I’m rs2

  • 重新开启rs1的nginx

观察/var/log/keepalived.log(如果未设置过，默认是在/var/log/message)

1
2
Oct  8 16:11:41 ds01 Keepalived_healthcheckers[15444]: TCP connection to [10.1.80.93]:80 success.
Oct  8 16:11:41 ds01 Keepalived_healthcheckers[15444]: Adding service [10.1.80.93]:80 to VS [10.1.80.90]:80

ds1上查看sudo ipvsadm -ln

1
2
3
4
5
6
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.1.80.90:80 wrr persistent 50
  -> 10.1.80.93:80              Route   100    0          0
  -> 10.1.80.95:80              Route   100    2          0

发现10.1.80.93已经重新被加入到服务列表中了

Q&A

  • ds1,ds2上的LVS规则如何删除？
1
sudo ipvsadm --clear
  • 如何删掉ds1,ds2上的VIP？
1
sudo ifconfig enp0s3:0 down
  • 如何查看LVS统计信息？
1
sudo ipvsadm -ln --stats
  • 如何查看LVS连接条目？
1
sudo ipvsadm -lnc
  • ds1,ds2上用tcpdump抓包观察组播消息？
1
sudo tcpdump -vvv -n -i enp0s3 host 224.0.0.18

或

1
sudo tcpdump -vvv -n -i enp0s3 vrrp

  • 指定访问来源主机，指定抓包端口号，网卡名称
    1
    sudo tcpdump -i enp0s3 tcp port 80 and host 10.1.80.62

查看详细一点信息

1
sudo tcpdump -vvv -n -i enp0s3 tcp port 80 and host 10.1.80.62
  • 用iptables如何允许组播?
1
2
sudo iptables -I INPUT -i enp0s3 -d 224.0.0.18 -p vrrp -j ACCEPT
sudo iptables -I OUTPUT -o enp0s3 -d 224.0.0.18 -p vrrp -j ACCEPT
  • firewall防火墙如何开放端口，关闭端口？

开放80端口访问

1
sudo firewall-cmd --zone=public --add-port=80/tcp --permanent

关闭80端口访问

1
sudo firewall-cmd --zone=public --remove-port=80/tcp --permanent

记得要reload或restart下，防火墙才会生效

1
sudo firewall-cmd --reload

或

1
sudo systemctl reload firewalld

或

1
sudo systemctl restart firewalld

推荐使用第一种方式重新加载防火墙

  • 查看防火墙开放的端口
1
sudo firewall-cmd --list-port
  • 脑裂的原因有哪些？
  1. 主备的virtual_router_id 不一致
  2. 交换机不允许组播(可以改成单播方式)
  3. 防火墙未允许组播

注意事项

  • 注意要使得rs1,rs2上的配置重启生效的话，可以将/opt/scripts/lvs_rs.sh作为启动脚本（推荐）
1
sudo echo "/opt/scripts/lvs_rs.sh start" >> /etc/rc.d/rc.local

在centos7中，/etc/rc.d/rc.local文件的权限被降低了

1
sudo chmod +x /etc/rc.d/rc.local

  • 或使用以下方法

要使得rs1,rs2上的lo:0重启后仍生效的话，要创建一个ifcfg-lo:0文件

1
sudo cp /etc/sysconfig/network-scripts/ifcfg-lo /etc/sysconfig/network-scripts/ifcfg-lo:0
1
sudo vi /etc/sysconfig/network-scripts/ifcfg-lo:0

内容如下

1
2
3
4
5
DEVICE=lo:0
IPADDR=10.1.80.90
NETMASK=255.255.255.255
BROADCAST=10.1.80.90
ONBOOT=yes

重启网络

1
sudo service network restart

1
sudo vi /etc/sysctl.conf

添加以下配置

1
2
3
4
5
6
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2

刷新文件

1
sudo sysctl -p

路由设置添加开启自启动
echo “route add -host 10.1.80.90 dev lo:0” >> /etc/rc.local

参考

LVS三种模式的工作原理

CentOS7搭建LVS负载均衡器

LVS High Availability - The Keepalived Solution

keepalived组播故障排查

centos7 keepalived 主备通信 防火墙vrrp 协议

CentOS 7 配置 Keepalived 实现双机热备

VRRP技术原理与注意点

LVS+Keepalived+Nginx高可用实现

分享到： 微信 新浪微博 Twitter Facebook 复制网址 邮件 收藏夹 一键分享 更多