The Linux Virtual Server is a highly scalable and highly available server built on a cluster of real servers, with the load balancer running on the Linux operating system. The architecture of the server cluster is fully transparent to end users, and the users interact as if it were a single high-performance virtual server.
We use LVS extensively at work to provide a scalable and highly available website which gets around 300 hits per second. Setting up and managing LVS can be made a lot easier using a tool that our ex staff wrote called LVSadmin. Written in perl it is easily configurable and provides a curses based front end to manage the servers. Setting up a new LVS cluster is really easy.
For our new cluster we have 2 servers that I want to load balance with LVS:
dev-blobdirector0 10.0.2.4:8889
dev-blobdirector1 10.0.2.17:8889
And I want them presented with the following hostname:
lvs-dev-blobdirector 10.0.2.23
We want 2 LVS instances, for redundancy in case one dies which will run on the following servers:
lvs0 10.0.2.18
lvs1 10.0.2.19
The target platform is Ubuntu Dapper, which is our platform of choice at the moment until Hardy is out!
On lvs0/lvs1 Grab the source code (lvsadmin, LVS.pm) for LVS from SourceForge and place it in /usr/local/bin, lvsadmin should be +x.
Then install the following packages
apt-get install perl-modules libcurses-perl libcurses-widgets-perl keepalived
A few variable changes need to be done in the LVS.pm:
* Change the $MASTER to the hostname of the master server, in our case lvs0
* Change $IF to the interface that packets will be coming from, in our case eth0
* There is a br0 further down the script that needs to reflect the $IF change so again change that to eth0
* Change $PASSWORD to the keepalived password you want
* Find the lvs_id and change that to a new unique instance for this LVS cluster.
LVS.pm on both servers should be identical.
The following files need to then be made in /etc/keepalived
portlist (this is a list of all the realserver ports LVS will manage)
8889
serverlist (these hostnames are resolved to create a meaningful display)
lvs-dev-blobdirector.btn.dbplc.com
dev-blobdirector0
dev-blobdirector1
serverstate (the default state of the servers, lvsadmin will read and write it’s state to this file when you change things)
10.0.2.23:10.0.2.4:8889:up
10.0.2.23:10.0.2.17:8889:up
services (list of virtual server ports)
8889
viplist (list of virtual server IPs)
10.0.2.23 eth0
Then on each of the real servers we need to create the virtual IP for them to listen on in /etc/network/interfaces add:
auto lo:23
iface lo:23 inet static
address 10.0.2.23
netmask 255.255.255.255
broadcast 10.0.255.255
pre-up echo 1 >/proc/sys/net/ipv4/conf/all/arp_ignore; echo 2 >/proc/sys/net/ipv4/conf/all/arp_announce
then on each real server start the interface:
root@dev-blobdirector0:~# ifup lo:23
That’s all the configuration done, we just now have to start the LVS system, the first time is a little flaky but from them on in it will work smoothly.
On each lvs server start lvsadmin, then press go to info and press Shift-S to save, it will then create /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
lvs_id LVS_XEN
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 150
advert_int 1
smtp_alert
authentication {
auth_type PASS
auth_pass eggsandham
}
virtual_ipaddress {
10.0.2.23 dev eth0
}
}
# Virtualserver: 10.0.2.23
virtual_server 10.0.2.23 8889 {
delay_loop 60
lb_algo wrr
lb_kind DR
protocol TCP
virtualhost www.digitalbrain.com
# Realserver: 10.0.2.4
real_server 10.0.2.4 8889 {
weight 30
inhibit_on_failure
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
# Realserver: 10.0.2.17
real_server 10.0.2.17 8889 {
weight 30
inhibit_on_failure
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
Then start keepalived
/etc/init.d/keepalived start
You should now be able to telnet to the correct port on the virtual server if it’s working!
rus@absinthe:~$ telnet lvs-dev-blobdirector 8889
Trying 10.0.2.23...
Connected to lvs-dev-blobdirector.btn.dbplc.com.
Escape character is '^]'.
To test LVS redundancy take down the master (lvs0) and see if you can still connect to the virtual server.
root@lvs0:~# /etc/init.d/keepalived stop
Stopping keepalived: keepalived.
root@lvs0:~# ps aux | grep keep
root 3859 0.0 0.1 3940 900 pts/1 R+ 14:04 0:00 grep keep
rus@absinthe:~$ telnet lvs-dev-blobdirector 8889
Trying 10.0.2.23...
Connected to lvs-dev-blobdirector.btn.dbplc.com.
Escape character is '^]'.
And there we have it, an easy way to create a scalable, highly available server platform!