This is part 2 of the two-part article on configuring Elastix with Redfone Fonebridge2 devices. In this part, we will cover all the details of setting up a robust high availability cluster with Elastix and a T1/E1 interface. At the end of this article, you will be able to implement high availability in any environment so if either server were to fail for any reason, the other will take all future calls.

SCHEMATIC DIAGRAM

Our objective for this article is to assemble a cluster of two (2) Elastix servers with a Redfone Fonebridge2 device. The following is the diagram of the solution that we are trying to accomplish.

Basically, the call originates in extension 509 in the Elastix cluster (actually registered with the primary Elastix appliance) and then gets routed to the Fonebridge2 device and out the span 1. The call then loopsback from span 1 to span 2 of the Fonebridge2. After this, the call is received by the primary Elastix appliance and terminated on extension 503. The secondary Elastix appliance does not intervene up until this point. When we unplug the primary Elastix appliance from the network, we can try the call again and the call path is exactly the same but it's routed through the cluster by the secondary Elastix appliance (which would actually now be the primary). In either case, the call should complete without problems demonstrating the reliability of this environment.

EQUIPMENT REQUIRED

For the purposes of this article, we will be making use of the following equipment:

1) One (1) Redfone Fonebridge2 Dual Port T1/E1:

2) Two (2) Elastix Appliances ELX-3000 loaded with Elastix 1.5.2-2 Stable, with two (2) network interfaces each:

Please ensure to connect the devices according to the diagram above before continuing. You could review the "CONNECTING THE DEVICES" section of Part 1 for more information here.

INSTALLING AND PREPARING ELASTIX

If you purchased an Elastix appliance there is no need to reinstall your Elastix software since a portion of your hard drive has already been reserved for use with high availability applications. If you are setting up an Elastix server of your own please install Elastix ON BOTH SERVERS according to the following specifications.

1. Boot Elastix from Installation CD.

2. In Boot menu, select 'advanced' and press<Enter>.

3. Partition the disk space manually based on the following example of a 120GB hard drive:

• root (/), ext3, 60% of total hard disk capacity (sda1)
• SWAP, twice the size of available RAM
• Boot (/boot) 100 MB
• Free unformatted space, fill remaining space

4. All other choices are standard installation options so just follow on-screen instructions.

5. After installation is finished, please log into the system and update any new packages from the Elastix repository. We do this by executing the following command:

[root@artemisa ~]# yum –y update

6. Check the /etc/grub/grub.conf file and ensure that the first kernel scheduled to boot is not the Xen kernel which could cause problems.

NOTE: At some point during installation or soon after, please set the IP address and hostname for each server. In our example we have assigned 192.168.50.14 to the primary server with hostname "artemisa.elastixdepot.com" and 192.168.50.15 to the secondary server with hostname "apolo.elastixdepot.com".

Now that you have both Elastix servers installed, we can proceed to set up the DRDB service to replicate and keep both servers synchronized.

DRDB AND HEARTBEAT PREPARATIONS

Whether you are using Elastix appliances or you installed your own Elastix server you will need to create a new partition in the unused space that you left in your hard drive. This must be done on both servers.

1. Create a partition that will contain the data to be replicated between both servers as follows:

[root@artemisa ~]# fdisk /dev/hda

• Command to add a new partition (n)
• Primary (p)
• Partition number (4)

IMPORTANT NOTE: If the two servers have hard drives of different sizes, it is imperative that the size of the replication partition is identical in both servers, otherwise, they will never be able to synchronize over DRBD. Due to this requirement, please select the desired size of the partition exactly the same in the continuing options of the fdisk command. In our case, we chose 3GB (3072M).

• First cylinder (<enter> to choose default)
• Last cylinder (+3072M)
• Command to select a partition (t)
• Choose the correct partition (4)
• Select type of the partition (83)
• Command to save all changes (w)
• Reboot both servers

Verify that the partition was created correctly with the "fdisk -l" command.

2. Format the newly created partition (in our case with the "mke2fs -j /dev/hda4" command).

3. Now, you must erase some file system structure information from the partition that we just created in the previous step. If we don't do this, this file system structure information will also be replicated between the servers and may cause problems.

4. Install DRBD, HEARTBEAT and all dependencies with yum.

[root@artemisa ~]# rpm -ivh vault.centos.org/5.2/extras/i386/RPMS/drbd82-8.2.6-1.el5.centos.i386.rpm
[root@artemisa ~]# yum install kmod-drbd82

[root@artemisa ~]# yum install heartbeat

[root@artemisa ~]# yum install OpenIPMI-libs openhpi heartbeat-libs heartbeat-stonith

5. Make sure that hostnames are correctly configured for IP resolution in the file (/etc/hosts) IN BOTH SERVERS. The /etc/hosts files in both servers should contain the same entries. Please note that the IP address for the primary server (artemisa) is 192.168.50.14 and for the secondary server (apolo) is 192.168.50.15. As you will see later, the cluster will share a floating IP address of 192.168.50.16.

CONFIGURING DRDB

1. DRBD allows us to write in the partition declared in both cluster members, in our case /dev/hda4. We must create a virtual partition called /dev/drdb0 IN BOTH SERVERS. Why?

2. Edit /etc/drbd.conf IN BOTH SERVERS. The file drbd.conf must be identical in both servers. Modify this sample file according to your particular requirements:

3. Before continuing, change the name for the Elastix server from web interface. Why? Delete?

4. Now, we create the metadata for /dev/drdb0 in the two servers. This is a required step to setup the environment for DRDB. We do this with the following command:

[root@artemisa ~]# drbdadm create-md r0

5. Start DRDB services IN BOTH SERVERS to initiate sync process.

[root@artemisa ~]# service drbd start

6. Verify that the synchronization process is ready with the following command:

[root@artemisa ~]# service drbd status

7. Initially, both servers will be secondary (as seen in the previous screen as "Secondary/Secondary"). We need to further designate which one will be the primary server.

Run the following command IN THE PRIMARY SERVER ONLY:

[root@artemisa ~]# drbdsetup /dev/drbd0 primary -o

And this command IN THE SECONDARY SERVER ONLY:

[root@artemisa ~]# drbdadm secondary r0

8. Check DRDB status again with the command "service drbd status". The result of this command should be similar to the one below for the primary server.

And the following for the secondary server.

9. We can determine a server's role by executing this command ON BOTH SERVERS:

[root@artemisa ~]# drbdadm state r0

The primary server should reply something like this:

The secondary server should reply something like this:

10. We will mount the virtual partition /dev/drbd0 as "/replica" in the primary server, move the contents of relevant files to the /replica directory and the create symbolic links in the main file system to the new location of the relevant files as follows. In the secondary server we will create symbolic links in the same fashion and remove the contents of the relevant files as they will be replicated into the /replica directory by DRDB and the directory mounted by HEARTBEAT when the secondary server is awakened.

The relevant files are located in:

• /etc/asterisk
• /var/lib/asterisk
• /usr/lib/asterisk
• /var/spool/asterisk
• /var/lib/mysql
• /var/log/asterisk
• /var/www

Perform the following command ON BOTH SERVERS:

[root@artemisa ~]# mkdir /replica

Perform the following commands IN THE PRIMARY SERVER ONLY.

[root@artemisa ~]# mount /dev/drbd0 /replica

[root@artemisa ~]# cd /replica
[root@artemisa replica]# tar -zcvf etc-asterisk.tgz /etc/asterisk
[root@artemisa replica]# tar -zxvf etc-asterisk.tgz
[root@artemisa replica]# tar -zcvf var-lib-asterisk.tgz /var/lib/asterisk
[root@artemisa replica]# tar -zxvf var-lib-asterisk.tgz
[root@artemisa replica]# tar -zcvf usr-lib-asterisk.tgz /usr/lib/asterisk/
[root@artemisa replica]# tar -zxvf usr-lib-asterisk.tgz
[root@artemisa replica]# tar -zcvf var-spool-asterisk.tgz /var/spool/asterisk/
[root@artemisa replica]# tar -zxvf var-spool-asterisk.tgz
[root@artemisa replica]# tar -zcvf var-lib-mysql.tgz /var/lib/mysql/
[root@artemisa replica]# tar -zxvf var-lib-mysql.tgz
[root@artemisa replica]# tar -zcvf var-log-asterisk.tgz /var/log/asterisk/
[root@artemisa replica]# tar -zxvf var-log-asterisk.tgz
[root@artemisa replica]# tar -zcvf var-www.tgz /var/www/
[root@artemisa replica]# tar -zxvf var-www.tgz

Perform the following commands ON BOTH SERVERS:

[root@artemisa replica]# rm -rf /etc/asterisk
[root@artemisa replica]# rm -rf /var/lib/asterisk
[root@artemisa replica]# rm -rf /usr/lib/asterisk/
[root@artemisa replica]# rm -rf /var/spool/asterisk
[root@artemisa replica]# rm -rf /var/lib/mysql/
[root@artemisa replica]# rm -rf /var/log/asterisk/
[root@artemisa replica]# ln -s /replica/etc/asterisk/ /etc/asterisk
[root@artemisa replica]# ln -s /replica/var/lib/asterisk/ /var/lib/asterisk
[root@artemisa replica]# ln -s /replica/usr/lib/asterisk/ /usr/lib/asterisk
[root@artemisa replica]# ln -s /replica/var/spool/asterisk/ /var/spool/asterisk
[root@artemisa replica]# ln -s /replica/var/lib/mysql/ /var/lib/mysql
[root@artemisa replica]# ln -s /replica/var/log/asterisk/ /var/log/asterisk
[root@artemisa replica]# ln -s /replica/var/www /var/www

11. Finally, we must restart mysql ON BOTH SERVERS:

[root@artemisa replica]# service mysqld restart

CONFIGURING HEARTBEAT

We will be making use of the HEARTBEAT service to awaken the secondary service in the case it's detected that the primary server has stopped responding. Both servers must have HEARTBEAT enabled. Please follow the following steps.

1. Remember to stop all services that will be controlled by heartbeat IN BOTH SERVERS (and make sure that they don't start during boot time). These services will be managed by HEARTBEAT on the server that is in control in that moment. You can accomplish all this with the following commands:

[root@artemisa ~]# chkconfig asterisk off
[root@artemisa ~]# chkconfig mysqld off
[root@artemisa ~]# chkconfig httpd off

2. Edit file/etc/ha.d/ha.cf. This is a basic configuration of the file that provides all the functionalities we require. Please make sure both servers have the same exact contents in this file. In the following picture we show the contents in the primary server only.

3. Edit /etc/ha.d/haresources. In this file we will specify which is our primary server initially, which partition will be mounted when the server is brought up as primary, our floating IP (192.168.50.16 in this example), network interface (eth2 in this example) and the init scripts from init.d that will be awakened in case of a server failure (fonulator,asterisk, mysqld y httpd). Please note that the order of this list is important. Also note in the pictures below that /etc/ha.d/haresources on both servers contain the primary server name (artemisa.elastixdepot.com); in fact, both files are exactly the same.

NOTE: Ensure that all the parameters are specified in a single line in this file.

4. Edit /etc/ha.d/authkeys and change the key to one of your preference. Please make sure that both servers contain the same key. This provides security and authentication capabilities between nodes.

5. Change permissions to file/etc/ha.d/authkeys.

[root@artemisa ~]# chmod 600 /etc/ha.d/authkeys

6. Configure HEARTBEAT to initiate at server boot time.

[root@artemisa ~]# chkconfig --add heartbeat

7. Unmount /replica IN THE PRIMARY SERVER ONLY. We do this because we are going to restart HEARTBEAT and it will automatically mount /replica when it comes up.

[root@artemisa ~]# umount /replica

8. Restart DRDB IN BOTH SERVERS.

[root@artemisa ~]# service drbd restart

9. Verify that BOTH SERVERS are secondary initially.

[root@artemisa ~]# drbdadm state r0

The result should be ‘Secondary/Secondary’ in both servers.

10. Restart heartbeat ON BOTH SERVERS.

[root@artemisa ~]# service heartbeat restart

11. Wait about a minute and recheck the server status ON BOTH SERVERS.

[root@artemisa ~]# drbdadm state r0

This time, the result should be ‘Primary/Secondary’ on primary server and 'Secondary/Secondary' in the secondary server.

12. Execute ‘df –h’ ON THE PRIMARY SERVER and confirm that the /dev/drbd0 partition is mounted and in use.

[root@artemisa ~]# df -h

Filesystem Size Used Avail Use% Mounted on
/dev/sda1 5.9G 1.9G 3.8G33% /
tmpfs 1.5G 0 1.5G 0%/dev/shm
/dev/drbd0 138G 305M 131G1% /replica

13. You can monitor the synchronization process with following command:

[root@artemisa ~]# cat /proc/drbd

INSTALLING THE FONEBRIDGE

We must now install the actual Redfone Fonebridge2 device. You may find more detailed instructions on how to do this in Part 1 of this article here. Please follow the following steps ON BOTH SERVERS:

1. Install the fonulator utility from the online repository.

[root@artemisa ~]# rpm -ivh support.red-fone.com/downloads/fonulator/fonulator-2.0.0-36.i386.rpm

2. Download a demo redfone.conf from the online repository.

[root@artemisa ~]# cd /etc/
[root@artemisa etc]# wget support.red-fone.com/downloads/fonulator/redfone.conf

3. Edit redfone.conf to meet your needs. Please see Part 1 for more information on this here.

4. Download the fonulator initd script from the online repository.

[root@artemisa etc]# cd /etc/init.d
[root@artemisa init.d]# wget support.red-fone.com/downloads/fonulator/old/fonulator_initd_script
[root@artemisa init.d]# mv fonulator_initd_script fonulator
[root@artemisa init.d]# chmod +x fonulator

5. Reboot BOTH SERVERS and confirm proper functionality in both. Under regular conditions, your primary server must boot, mount the DRDB partition and initiate the scripts specified in /etc/ha.d/haresources. The secondary server must boot, NOT mount the DRDB partition and NOT initiate the scripts specified in /etc/ha.d/haresources. Why? Any particular order of booting?

6. Verify that the DRDB partition is mounted IN THE PRIMARY SERVER by executing ‘df –h’. It should show something similar to:

[root@artemisa init.d]# df -h

Filesystem Size Used Avail Use% Mounted on
/dev/sda1 5.9G 1.8G 3.8G32% /
tmpfs 125M 0 125M 0%/dev/shm
/dev/drbd0 5.7G 409M 5.0G8% /replica

7. It is recommended that you use the dahdi init script provided by Redfone due to some earlier issues with the default. We will backup the default dahdi script and then download the one from Redfone. Please do the following ON BOTH SERVERS.

[root@artemisa init.d]# mv /etc/init.d/dahdi ~/dahdi.backup
[root@artemisa init.d]# wget support.red-fone.com/downloads/elastix/dahdi
[root@artemisa init.d]# chmod 755 /etc/init.d/dahdi

8. Sometimes we run into problems with an erroneous detection of another Asterisk instance running. To prevent this, we'll implement a work-around by changing the exit value of the 'exit' from 1 to 0 in the /etc/init.d/asterisk file ON BOTH SERVERS. Please see the before and after pictures below.

[root@artemisa ~]# vim /etc/init.d/asterisk

9. In this version of Elastix (1.5.2-2 Stable) there is an issue with a kernel panic occurring when we shutdown the network service before dahdi. Therefore, we must change the order for shutting down these services as follows:

[root@artemisa ~]# cd /etc/rc0.d/
[root@artemisa rc0.d]# mv K92dahdi K90dahdi
[root@artemisa rc0.d]# mv K90network K92network
[root@artemisa rc0.d]# cd ../rc6.d/
[root@artemisa rc0.d]# mv K92dahdi K90dahdi
[root@artemisa rc0.d]# mv K90network Kk92network

Congratulations! You have successfully finished setting up your high availability environment. Now we must make sure it's working well.

TESTING OUR HIGH AVAILABILITY CLUSTER

After having performed all configurations properly and made the connections properly you should have something similar to this:

Now we need to make sure all this hard work will pay off. We should be able to define our extensions 503 and 509 (please see Diagram at the top of this article) in our high availability cluster. In order to do this, we need to direct our browser to the floating IP we defined for the cluster. In our case this would be 192.168.50.16. This request from your browser is actually being handled by the primary server unless it were to fail. In case of failure of the primary server, the secondary server will respond any request to 192.168.50.16.

In either case we will be presented with an Elastix GUI where we can specify the required extensions. Please refer to Part 1 of this article (TESTING THE ELASTIX / FONEBRIDGE2 SETUP section) for more information and screenshots of how to do this here. After the extensions are setup, we can initiate a call from extension 509 and dial 1503 (we defined an outgoing route of '1|XXX' in Part 1 of this article). The call should be completed and the phone in extension 503 should ring.

In order to test our high availability setup you must now unplug the primary server's ethernet cable from the switch (to simulate a failure of this server). After a couple of minutes try making the same call from 509 to 1503. The call should complete without any problems although the call is now being routed through the secondary server.

Voila! There you have it. You have just completed your very own high availability clustered PBX with Elastix and Redfone.

Alfredo Zambrano