Sysadmin has gone simple

WSO2 recently announced the 2.0.0 release of it’s own Elastic Loadbalancer (ELB). As you may already know, you can front a single or multiple service clusters with the ELB to implement high availability at the service level. But still the ELB is a SPOF and a failure could be critical for your organization. This post explains how you can have a fail-proof ELB deployment with high availability using few free and opensource tools and packages.

I’ll be using a Debian based operating system for the deployment and you can use the same tools and packages on any other Linux system. Before we get in to details, let’s try to understand how this setup is going to work.

As the above diagram shows, there will be 2 identical ELB setups running on active and passive modes. Those will be fronting a single or several service clusters and the Keepalived package is installed on both hosts running ELBs. A health check service is running to check the availability of each ELB and both ELB installed hosts are fronted with a virtual IP address. I hope the story is clear now and you can understand how this is going to work at a higher level. Let’s dive into details now.

I will continue assuming you have some sort of an idea of how to setup WSO2 ELB. Fist let’s think of a way to check the availability of the ELB. You cloud;

•       Check the pid of the running java process
•       Send a request to a hosted service through the ELB
•       Connect to an open socket exposed by the ELB
•       Host a proxy service inside the ELB and ping the service, etc.

There can be many other ways and you can use any method you are familiar and comfortable with. For my convenience, I’ll go for the last option, that is hosting a proxy service inside the ELB and ping the service to check whether the ELB is functioning properly or not. Main reasons for leaving other options are;

•      The process could be still running even though the ELB is not functioning as expected (cloud be frozen)
•      Sending a request to the backend service could fail due to a problem in the service, not in the ELB
•      You could still connect to a socket even though the ELB isn’t functioning properly

Let’s see how you can host a proxy service inside the ELB. You can browse to the extracted directory, probably with the name wso2elb-x.x.x and create a file, let’s say as EchoProxy.xml at repository/deployment/server/synapse-configs/default/proxy-services/ with following contents.

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse" name="EchoProxy" transports="https http" startOnLoad="true" trace="disable">
 <target>
 <inSequence>
 <header name="To" action="remove"/>
 <property name="NO_ENTITY_BODY" scope="axis2" action="remove"/>
 <property name="RESPONSE" value="true" scope="default" type="STRING"/>
 <send/>
 </inSequence>
 </target>
</proxy>

Now you can start the ELB and send a request to the service and see whether you get a HTTP 200 as the response code. For that you could use curl, if you don’t have curl installed, you can install it from your package repositories. To ping your hosted service, you could run the following command and it should return the following output.

~$ curl -v http://localhost:8280/services/EchoProxy
* About to connect() to localhost port 8280 (#0)
* Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 8280 (#0)
> GET /services/EchoProxy HTTP/1.1
> User-Agent: curl/7.21.0 (x86_64-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.15 libssh2/1.2.6
> Host: localhost:8280
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: application/x-www-form-urlencoded
< Accept: */*
< Host: localhost:8280
< Date: Mon, 24 Sep 2012 16:10:43 GMT
< Server: Synapse-HttpComponents-NIO
< Transfer-Encoding: chunked
< 
* Connection #0 to host localhost left intact
* Closing connection #0

If you can notice the HTTP/1.1 200 OK section, you can say that the ping is successful and the ELB is functioning properly.  Now let’s put that into a script so that you can use it when configuring keepalived. For my convenience I’m placing the script inside /opt/bin/.

#!/bin/bash
ENDPOINT="http://localhost:8280/services/EchoProxy"
CURL=`which curl`
CODE=$(${CURL} -sL -w "%{http_code}" ${ENDPOINT} -o /dev/null)
[ ${CODE} == "200" ] && exit 0 || exit 1

This will return 0 when the HTTP response code is 200 otherwise it’ll return 1. You need to have this script on both active and passive hosts. Well, now you have all tools ready to setup keepalived. Inside keepalived we’ll be using above script for the health check. You can install keepalived from your packages repositories. When configuring, the idea is to have a virtual IP address assigned to the active ELB and switch it to the passive ELB when the health check script at the active host returns 1, which means the active ELB has failed.

Once keepalived is installed, you can create the keepalived.conf at /etc/keepalived  with following contents.

global_defs {
     notification_email {
          admin@domain.com
          user@domain.com
     }
     notification_email_from automatic-failover@domain.com
     smtp_server xx.xx.xx.xx
     smtp_connect_timeout 30
     router_id VRRP-director1
}
vrrp_script check_elb {
     script "/opt/bin/check_elb.sh"
     interval 2
     weight 2
}
vrrp_instance VRRP-director1 {
     virtual_router_id 51
     advert_int 1
     priority 101
     interface eth0
     state MASTER
     smtp_alert
     track_script {
          check_elb
     }
     virtual_ipaddress {
          192.168.4.250/24
     }
}

The configuration is pretty much neat right? As it says, you have defined SMTP details at the global_defs level so that you can get alerts when the state of host changes from MASTER to BACKUP. Inside the vrrp_script block you have the health check script, the interval it should run and the weight for load balancing decisions. Inside the vrrp_instance block you have the VRRP router id the instance belongs, details advertising interval from this router instance to other keepalived nodes, instance priority of the VRRP router, network interface for the instance to run on, instance state, to advertise whether this is a MASTER or a BACKUP host, activate SMTP alerts, the track script and the virtual IP address for the router instance.

AS mentioned above, you need to setup keepalived on the other node in the same manner by only changing sections in red. There you have to set a lesser value for the priority as it’s going to be the BACKUP host. So the vrrp_instance would look like this.

vrrp_instance VRRP-director1 {
     virtual_router_id 51
     advert_int 1
     priority 100
     interface eth0
     state BACKUP
     smtp_alert
     track_script {
          check_elb
     }
     virtual_ipaddress {
          192.168.4.250/24
     }
}

Once everything is ready you can start keepalived as any other Linux based service (/etc/init.d/keepalived start) and you can see that there are 2 IP addresses bound to the eth0 interface.

~# ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
 link/ether 00:50:56:b7:2f:fc brd ff:ff:ff:ff:ff:ff
 inet 192.168.4.102/24 brd 192.168.7.255 scope global eth0
 inet 192.168.4.250/24 scope global secondary eth0
 inet6 fe80::250:56ff:feb7:2ffc/64 scope link 
 valid_lft forever preferred_lft forever

One IP address is the one which is bound to the physical network interface and the other one is the virtual IP address you mentioned in  keepalived.conf. If you stop the ELB in the MASTER host the health check script fails and keepalived will advertise it to the BACKUP host and you’ll notice that the virtual IP address gets assigned to the BACKUP host. And when the MASTER comes back online, it’ll get assigned to the MASTER host again.

You can extend this method and implement several  other features on top of this, such as using the same script to monitor and alert if there’s something not right, etc.

I hope this post explains everything necessary to have a fail-proof, high available WSO2 ELB setup and if you have any further queries, feel free to drop a comment and I’ll try to reply  when time permits.

Enjoy!