As more and more critical commercial applications move on the Internet, providing highly available services becomes increasingly important. One of the advantages of a clustered system is that it has hardware and software redundancy. High availability can be provided by detecting node or daemon failures and reconfiguring the system appropriately so that the workload can be taken over by the remaining nodes in the cluster.
In fact, high availability is a big field. An elegant highly available system may have a reliable group communication sub-system, membership management, quoram sub-systems, concurrent control sub-system and so on. There is a lot of works. However, we can use some existing software to construct highly available LVS systems now. There must be lots of methods to build highly available LVS systems, please drop me a message if you have your elegant methods. The following two solutions is only for reference.
The high availability of virtual server can be provided by using of "mon", "heartbeat" , "fake" and "coda" software. The "mon" is a general-purpose resource monitoring system, which can be used to monitor network service availability and server nodes. The "heartbeat" code currently provides the heartbeats among two node computers through serial line and UDP heartbeats. Fake is IP take-over software by using of ARP spoofing. The high availability of Linux Virtual Server is illustrated in the following figure.
The server failover is handle as follows: The "mon" daemon is running on the load balancer to monitor service daemons and server nodes in the cluster. The fping.monitor is configured to detect whether the server nodes is alive every t seconds, and the relative service monitor is also configured to detect the service daemons on all the nodes every m seconds. For example, http.monitor can be used to check the http services; ftp.monitor is for the ftp services, and so on. An alert was written to remove/add a rule in the linux virtual server table while detecting the server node or daemon is down/up. Therefore, the load balancer can automatically mask service daemons or servers failure and put them into service when they are back.
Now, the load balancer becomes a single failure point of the whole system. In order to mask the failure of the primary load balancer, we need setup a backup server of the load balancer. The "fake" software is used for the backup to takeover the IP addresses of the load balancer when the load balancer fails, and the "heartbeat" code is used to detect the status of the load balancer to activate/deactivate the "fake" on the backup server. Two heartbeat daemons run on the primary and the backup, they heartbeat the message like "I'm alive" each other through the serial line periodically. When the heartcode daemon of the backup cannot hear the "I'm alive" message from the primary in the defined time, it activates the fake to take over the Virtual IP address to provide the load-balancing service; when it receives the "I'm alive" message from the primary later, it deactivate the fake to release the Virtual IP address, and the primary comes back to work again.
However, the failover or the takeover of the primary load balancer will cause the established connection in the hash table lost in the current implementation, which will require the clients to send their requests again.
Coda is a fault-tolerant distributed file systems, a descendant of Andrew file system. The contents of servers can be stored in Coda, so that files can be highly available and easy to manage.
The failover of real servers
The "mon" is used to monitor service daemons and server nodes in the cluster. For example, the fping.monitor can be used to monitor the server nodes, http.monitor can be used to check the http services, ftp.monitor is for the ftp services, and so on. So, we just need to write an alert to remove/add a rule in the virtual server table while detecting the server node or daemon is down/up. Here is an example calleded lvs.alert, which takes virtual service(IP:Port) and the service port of real servers as parameters.
#!/usr/bin/perl
#
# lvs.alert - Linux Virtual Server alert for mon
#
# It can be activated by mon to remove a real server when the
# service is down, or add the server when the service is up.
#
#
use Getopt::Std;
getopts ("s:g:h:t:l:P:V:R:W:F:u");
$ipvsadm = "/sbin/ipvsadm";
$protocol = $opt_P;
$virtual_service = $opt_V;
$remote = $opt_R;
if ($opt_u) {
$weight = $opt_W;
if ($opt_F eq "nat") {
$forwarding = "-m";
} elsif ($opt_F eq "tun") {
$forwarding = "-i";
} else {
$forwarding = "-g";
}
if ($protocol eq "tcp") {
system("$ipvsadm -a -t $virtual_service -r $remote -w $weight $forwarding");
} else {
system("$ipvsadm -a -u $virtual_service -r $remote -w $weight $forwarding");
}
} else {
if ($protocol eq "tcp") {
system("$ipvsadm -d -t $virtual_service -r $remote");
} else {
system("$ipvsadm -d -u $virtual_service -r $remote");
}
};
The lvs.alert is put under the /usr/lib/mon/alert.d directory. The mon configuration file (/etc/mon/mon.cf or /etc/mon.cf) can be configured to monitor the http services and servers in the cluster as follows.
#
# The mon.cf file
#
#
# global options
#
cfbasedir = /etc/mon
alertdir = /usr/lib/mon/alert.d
mondir = /usr/lib/mon/mon.d
maxprocs = 20
histlength = 100
randstart = 30s
#
# group definitions (hostnames or IP addresses)
#
hostgroup www1 www1.domain.com
hostgroup www2 www2.domain.com
#
# Web server 1
#
watch www1
service http
interval 10s
monitor http.monitor
period wd {Sun-Sat}
alert mail.alert wensong
upalert mail.alert wensong
alert lvs.alert -P tcp -V 10.0.0.3:80 -R 192.168.0.1 -W 5 -F dr
upalert lvs.alert -P tcp -V 10.0.0.3:80 -R 192.168.0.1 -W 5 -F dr
#
# Web server 2
#
watch www2
service http
interval 10s
monitor http.monitor
period wd {Sun-Sat}
alert mail.alert wensong
upalert mail.alert wensong
alert lvs.alert -P tcp -V 10.0.0.3:80 -R 192.168.0.2 -W 5 -F dr
upalert lvs.alert -P tcp -V 10.0.0.3:80 -R 192.168.0.2 -W 5 -F dr
Note that you need to set the paramter of lvs.alert like "lvs.alert -V 10.0.0.3:80 -R 192.168.0.3:8080" if the destination port is different in LVS/NAT.
Now the load balancer can automatically mask service daemons or servers failure and put them into service when they are back.
The failover of the load balancer
In order to prevent the load balancer becoming a single failure point of the whole system, we need setup a backup of the load balancer and let them heartbeat each other periodically. Please read the GettingStarted document include the heartbeat package, it is simple to setup 2-node heartbeating system.
For an example, we assume that the two load balancers have the following addresses:
| lvs1.domain.com (primary) | 10.0.0.1 |
| lvs2.domain.com (backup) | 10.0.0.2 |
| www.domain.com (VIP) | 10.0.0.3 |
After install the heartbeat on both lvs1.domain.com and lvs2.domain.com, simply create the /etc/ha.d/ha.conf as follows:
# # keepalive: how many seconds between heartbeats # keepalive 2 # # deadtime: seconds-to-declare-host-dead # deadtime 10 # hopfudge maximum hop count minus number of nodes in config hopfudge 1 # # What UDP port to use for udp or ppp-udp communication? # udpport 1001 # What interfaces to heartbeat over? udp eth0 # # Facility to use for syslog()/logger (alternative to log/debugfile) # logfacility local0 # # Tell what machines are in the cluster # node nodename ... -- must match uname -n node lvs1.domain.com node lvs2.domain.com
The /etc/ha.d/haresources file is as follows:
lvs1.domain.com 10.0.0.3 lvs mon
The /etc/rc.d/init.d/lvs is as follows:
#!/bin/sh
#
# You probably want to set the path to include
# nothing but local filesystems.
#
PATH=/bin:/usr/bin:/sbin:/usr/sbin
export PATH
IPVSADM=/sbin/ipvsadm
case "$1" in
start)
if [ -x $IPVSADM ]
then
echo 1 > /proc/sys/net/ipv4/ip_forward
$IPVSADM -A -t 10.0.0.3:80
$IPVSADM -a -t 10.0.0.3:80 -r 192.168.0.1 -w 5 -g
$IPVSADM -a -t 10.0.0.3:80 -r 192.168.0.2 -w 5 -g
fi
;;
stop)
if [ -x $IPVSADM ]
then
$IPVSADM -C
fi
;;
*)
echo "Usage: lvs {start|stop}"
exit 1
esac
exit 0
Finally, make sure that all the files are created on both the lvs1 and lvs2 nodes and alter them for your own configuration, then start the heartbeat daemon on the two nodes.
Note that "fake" (IP takeover by Gratuitous Arp) is already included in the heartbeat package, so there is no need to setup "fake" separately. When the lvs1.domain.com node fails, the lvs2.domain.com will take over all the haresources of the lvs1.domain.com, i.e. taking over the 10.0.0.3 address by Gratuitous ARP, start the /etc/rc.d/init.d/lvs and /etc/rc.d/init.d/mon scripts. When the lvs1.domain.com come back, the lvs2 releases the HA resources and the lvs1 takes them back.
The ldirectord (Linux Director Daemon) written by Jacob Rief is a stand-alone daemon to monitor services of real servers, currently http and https service. It is simple to install and works with the heartbeat code. The ldirectord program can be found under the contrib directory inside the ipvs tar ball, or you can check the CVS repository of heartbeat for the latest version. See 'perldoc ldirectord' for all the information about ldirectord. Thank Jacob Rief for writing this great program!
The advantages of ldirectord over mon are as follows:
node1 IPaddr::10.0.0.3 ldirectord::www ldirectord::mail
Anyway, the ldirectord can also be started and stopped manually. You can use it in your LVS cluster without the backup of load balancer.
For the example introduced in the mon+heartbeat+fake+coda solution, you can configure the /etc/ha.d/www.cf as follows:
#
# The /etc/ha.d/www.cf for ldirectord
#
# the number of second until a real server is declared dead
timeout = 10
# the number of second between server checks
checkinterval = 10
#
# virtual = x.y.z.w:p
# protocol = tcp|udp
# scheduler = rr|wrr|lc|wlc
# real = x.y.z.w:p gate|masq|ipip [weight]
# ...
#
virtual = 10.0.0.3:80
protocol = tcp
scheduler = wlc
real = 192.168.0.1:80 gate 5
real = 192.168.0.2:80 gate 5
request = "/.testpage"
receive = "test page"
The /etc/ha.d/haresources file is simple as follows:
lvs1.domain.com IPaddr::10.0.0.3 ldirectord::www
You need to create the .testpage file at the DocumentRoot directory of each web server.
echo "test page" > .testpage
Start the heartbeat daemons on the primary and the backup. If there is anything wrong, you may check the /var/log/ha-log and /var/log/ldirector.log respectively.
$Id: HighAvailability.html,v 1.3 2000/01/11 14:00:52 wensong Exp $
Created on: 1998/12/5