上篇文章介绍了nginx作为反向代理/负载均衡服务器，假如nginx出现宕机的话，那么将无法转发请求到我们后端的网站服务器，现在介绍nginx+keepalived实现前端反向代理/负载均衡高可用架构的搭建！
实验环境如下需要四台服务器，其实严格的讲只需要三台就可以了，后端的网站服务器可以是单台也可以是多台，说一下我这里四台机器的软件包都是安装操作系统时全部安装的。这里环境如下：
IP地址 用途 系统版本 nginx版本 keepalived版本
192.168.2.73 nginx+keepalived（MASTER） RedHat 4.8(64位) 1.3.5 1.1.15
192.168.5.55 nginx+keepalived（BACKUP） RedHat 4.8(64位) 1.3.5 1.1.15
192.168.5.54 apache（系统自带） RedHat 4.8(64位) N/A N/A
192.168.5.57 apache（系统自带） RedHat 4.8(64位) N/A N/A
192.168.2.100 VIP（用于切换）
1、MASTER上安装nginx
groupadd www
useradd -g www www
tar zxvf nginx-1.3.5.tar.gz
cd nginx-1.3.5
./configure --user=www --group=www --prefix=/usr/local/nginx --with-http_stub_status_module
make
make install
2、修改/usr/local/nginx/conf/nginx.conf配置文件如下：
user www www;
worker_processes 1;
pid logs/nginx.pid;
worker_rlimit_nofile 1024;
events
{
use epoll;
worker_connections 1024;
}
http
{
include mime.types;
default_type application/octet-stream;
keepalive_timeout 120;
server_tokens off;
send_timeout 60;
tcp_nodelay on;
upstream https {
server 192.168.5.54:8080;
server 192.168.5.57:8080;
}
log_format access_log '$remote_addr - $remote_user [$time_local] $request'
'"$status" $body_bytes_sent "$http_referer"'
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /usr/local/nginx/logs/access.log access_log;
server
{
listen 80;
server_name 192.168.2.73;
location / {
proxy_pass ;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
}
3、检查配置文件是否有错误，出现如下两行则说明没问题！
worker_processes 1;
pid logs/nginx.pid;
worker_rlimit_nofile 1024;
events
{
use epoll;
worker_connections 1024;
}
http
{
include mime.types;
default_type application/octet-stream;
keepalive_timeout 120;
server_tokens off;
send_timeout 60;
tcp_nodelay on;
upstream https {
server 192.168.5.54:8080;
server 192.168.5.57:8080;
}
log_format access_log '$remote_addr - $remote_user [$time_local] $request'
'"$status" $body_bytes_sent "$http_referer"'
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /usr/local/nginx/logs/access.log access_log;
server
{
listen 80;
server_name 192.168.2.73;
location / {
proxy_pass ;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
}
3、检查配置文件是否有错误，出现如下两行则说明没问题！
/usr/local/nginx/sbin/nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
4、安装keepalived
tar zxvf keepalived-1.1.15.tar.gz
vi /usr/src/kernels/2.6.9-89.EL-smp-x86_64/include/linux/types.h
将如下两行注释掉，否则编译会出错，跟我这个版本的系统有关系，你的也许不要！
/*
typedef __u16 __bitwise __sum16;
typedef __u32 __bitwise __wsum;
*/
typedef __u16 __bitwise __sum16;
typedef __u32 __bitwise __wsum;
*/
cd keepalived-1.1.15
./configure
make
make install
将keepalived作为系统服务启动
cp /usr/local/etc/rc.d/init.d/keepalived /etc/init.d/
cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/
mkdir /etc/keepalived/
cp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/
cp /usr/local/sbin/keepalived /usr/sbin/
5、修改/etc/keepalived/keepalived.conf配置文件如下：
cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/
mkdir /etc/keepalived/
cp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/
cp /usr/local/sbin/keepalived /usr/sbin/
5、修改/etc/keepalived/keepalived.conf配置文件如下：
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
}
vrrp_script Monitor_Nginx {
script "/root/scripts/monitor_nginx.sh" #根据自己的实际路径放置monitor_nginx.sh
interval 2
weight 2
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
track_script {
Monitor_Nginx
}
virtual_ipaddress {
192.168.2.100
}
}
global_defs {
router_id LVS_DEVEL
}
vrrp_script Monitor_Nginx {
script "/root/scripts/monitor_nginx.sh" #根据自己的实际路径放置monitor_nginx.sh
interval 2
weight 2
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
track_script {
Monitor_Nginx
}
virtual_ipaddress {
192.168.2.100
}
}
6、从keepalived配置文件里面看到了有一处调用了一个脚本，脚本内容如下：
#!/bin/bash
if [ "$(ps -ef | grep "nginx: master process"| grep -v grep )" == "" ]
then
/usr/local/nginx/sbin/nginx
sleep 5
if [ "$(ps -ef | grep "nginx: master process"| grep -v grep )" == "" ]
then
killall keepalived
fi
fi
if [ "$(ps -ef | grep "nginx: master process"| grep -v grep )" == "" ]
then
/usr/local/nginx/sbin/nginx
sleep 5
if [ "$(ps -ef | grep "nginx: master process"| grep -v grep )" == "" ]
then
killall keepalived
fi
fi
7、增加可执行权限
chmod +x /root/scripts/monitor_nginx.sh
注：备机的Nginx、keepalived和以上安装步骤一样，只是个别的地方要修改！
注：备机的Nginx、keepalived和以上安装步骤一样，只是个别的地方要修改！
例如nginx的配置文件里面的server_name 192.168.2.73的IP地址改为server_name 192.168.5.55
例如keepalived的配置文件里面修改两处
state MASTER修改为state BACKUP
priority 100修改为priority 99
priority 100修改为priority 99
至此MASTER和BACKUP就配置完毕了！！！
7、配置两台apache服务器
登录192.168.5.54上操作：
[root@hadoop5 ~]# echo 'this is 192.168.5.54!' > /var/www/html/index.html
修改/etc/httpd/conf/httpd.conf文件的监听端口为8080
[root@hadoop5 ~]# sed -i 's/Listen 80/Listen 8080/g' /etc/httpd/conf/httpd.conf
[root@hadoop5 ~]# /etc/init.d/httpd start
登录192.168.5.57上操作：
[root@service ~]# echo 'Hello,This is 192.168.5.57!' > /var/www/html/index.html
修改/etc/httpd/conf/httpd.conf文件的监听端口为8080
[root@service ~]# sed -i 's/Listen 80/Listen 8080/g' /etc/httpd/conf/httpd.conf
[root@service ~]# /etc/init.d/httpd start
8、测试
启动MASTER的keepalived服务
/etc/init.d/keepalived start
执行ip a命令看是否有192.168.2.100的VIP出现，再查看nginx是否已经启动？
ps -ef | grep nginx
[root@hadoop3 ~]# for i in $(seq 20); do curl ; done
this is 192.168.5.54!
Hello,This is 192.168.5.57!
this is 192.168.5.54!
Hello,This is 192.168.5.57!
this is 192.168.5.54!
Hello,This is 192.168.5.57!
this is 192.168.5.54!
Hello,This is 192.168.5.57!
this is 192.168.5.54!
Hello,This is 192.168.5.57!
this is 192.168.5.54!
Hello,This is 192.168.5.57!
this is 192.168.5.54!
Hello,This is 192.168.5.57!
this is 192.168.5.54!
Hello,This is 192.168.5.57!
this is 192.168.5.54!
Hello,This is 192.168.5.57!
this is 192.168.5.54!
Hello,This is 192.168.5.57!
启动BACKUP的keepalived服务
this is 192.168.5.54!
Hello,This is 192.168.5.57!
this is 192.168.5.54!
Hello,This is 192.168.5.57!
this is 192.168.5.54!
Hello,This is 192.168.5.57!
this is 192.168.5.54!
Hello,This is 192.168.5.57!
this is 192.168.5.54!
Hello,This is 192.168.5.57!
this is 192.168.5.54!
Hello,This is 192.168.5.57!
this is 192.168.5.54!
Hello,This is 192.168.5.57!
this is 192.168.5.54!
Hello,This is 192.168.5.57!
this is 192.168.5.54!
Hello,This is 192.168.5.57!
this is 192.168.5.54!
Hello,This is 192.168.5.57!
启动BACKUP的keepalived服务
/etc/init.d/keepalived start
查看nginx服务也随之启动了
停止MASTER的keepalived服务，查看BACKUP是否已接替了VIP地址？
/etc/init.d/keepalived stop
[root@nagios-server scripts]# ip a
1: lo: mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:14:22:4a:ec:39 brd ff:ff:ff:ff:ff:ff
inet 192.168.5.55/21 brd 192.168.7.255 scope global eth0
inet 192.168.2.100/32 scope global eth0
inet6 fe80::214:22ff:fe4a:ec39/64 scope link
valid_lft forever preferred_lft forever
3: sit0: mtu 1480 qdisc noop
link/sit 0.0.0.0 brd 0.0.0.0
查看BACKUP的/var/log/messages日志是否接管VIP？
1: lo: mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:14:22:4a:ec:39 brd ff:ff:ff:ff:ff:ff
inet 192.168.5.55/21 brd 192.168.7.255 scope global eth0
inet 192.168.2.100/32 scope global eth0
inet6 fe80::214:22ff:fe4a:ec39/64 scope link
valid_lft forever preferred_lft forever
3: sit0: mtu 1480 qdisc noop
link/sit 0.0.0.0 brd 0.0.0.0
查看BACKUP的/var/log/messages日志是否接管VIP？
Oct 11 12:27:18 nagios-server Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Oct 11 12:27:18 nagios-server Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.100
Oct 11 12:27:22 nagios-server Keepalived_vrrp: ip address associated with VRID not present in received packet : 1677895872
Oct 11 12:27:22 nagios-server Keepalived_vrrp: one or more VIP associated with VRID mismatch actual MASTER advert
然后再启动MASTER的keepalived服务，看是否接管VIP？
Oct 11 12:27:18 nagios-server Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.100
Oct 11 12:27:22 nagios-server Keepalived_vrrp: ip address associated with VRID not present in received packet : 1677895872
Oct 11 12:27:22 nagios-server Keepalived_vrrp: one or more VIP associated with VRID mismatch actual MASTER advert
然后再启动MASTER的keepalived服务，看是否接管VIP？
/etc/init.d/keepalived start
执行ip a命令查看是否有192.168.2.100地址？
查看messages日志
Oct 11 13:06:27 hadoop3 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.100
Oct 11 13:06:28 hadoop3 Keepalived_vrrp: ip address associated with VRID not present in received packet : 1677895872
Oct 11 13:06:28 hadoop3 Keepalived_vrrp: one or more VIP associated with VRID mismatch actual MASTER advert
Oct 11 13:06:28 hadoop3 Keepalived_vrrp: bogus VRRP packet received on eth0 !!!
这样说明就OK了！！！
Oct 11 13:06:28 hadoop3 Keepalived_vrrp: ip address associated with VRID not present in received packet : 1677895872
Oct 11 13:06:28 hadoop3 Keepalived_vrrp: one or more VIP associated with VRID mismatch actual MASTER advert
Oct 11 13:06:28 hadoop3 Keepalived_vrrp: bogus VRRP packet received on eth0 !!!
这样说明就OK了！！！