Setting up IEP SE clustering

Introduction

IEP SE, like any other JBI component in the Open ESB community, relies upon and uses the GlassFish clustering feature. In this setup, a GlassFish cluster is created with one or more instances; and the JBI components are installed on this cluster. The installation of the component is propagated "homogeniously" to all the instances of the cluster. That is, the component gets installed on all the instances. This is called a homogenious cluster. Similarly, when a Service Assembly (SA) is deployed, its gets propagated and deployed onto all the instances.

The sample application

We will use a simple IEP application to test the cluster setup - TupleBasedAvgCalculator. The application is fed events via HTTP. The event contains a simple number. The process has a tuple based aggregator which takes 10 events and calculates its average. It then writes this average to the file system. We will deploy this application on a cluster with two instances. So, we will have an HTTP BC, IEP SE and File BC installed on each instance. The SA will also be deployed on each of these instances. This means that there are two HTTP ports on which the application will listen for events. Events can be sent to both these ports directly or by configuring an HTTP Load Balancer. Each of the HTTP BC will forward the event to the IEP SE in the same cluster instance. Each such IEP SE instance will insert the event into the IEP database. From this point onwards, only one IEP SE instance will "own" and process the event. It also follows that the output will only be written by the IEP SE which owns the instance. In the event that the IEP SE that currently owns the process fails, then other live IEP SE instances will compete to take the ownership of the process. Only one of such live instances will succeed in taking the ownership. This will be clearer as we walk through the example.

The steps

1. Install GlassFish cluster and setting up the HTTP Load Balancer
2. Set up IEP database
3. Install components and deploy the application
4. Verify that the application is deployed properly
5. Test the application in clustered mode
6. How to test the application in clustered mode without the HTTP Load Balancer

Install GlassFish cluster and setting up the HTTP Load Balancer

Follow the instructions from here. Only the first three steps need to be done, which are

1. Installing and Configuring GlassFish and the Web Server
2. Setting up a One-machine Cluster using GlassFish V2
3. Configuring the Load Balancer Plugin

Refer to these helpful tips for configuring the HTTP Load Balancer.

Set up IEP database

IEP SE requires setting up of database connection pools and JDBC resources. When IEP SE is installed and started on a non clustered instance, for example on the domain administartive server (DAS), then IEP SE automatically creates the connection pools and JDBC resources. When installing IEP SE to a cluster, IEP SE does not create the connection pools and JDBC Resources. These must be created using one of the two methods described below. Note that you must have the connection pools and JDBC Resources configured as descibed below before you install IEP SE to the cluster.

1. Re-use the connection pools and JDBC Resources that were created when IEP SE was installed and started on the DAS. This is an option if you are an advanced user and familiar with the IEP SE clustering setup. When you installed and started IEP SE on the DAS, the target for the connection pools as well as the JDBC Resource is "server". For IEP SE clustering, set the target for the two JDBC Resources ("jdbc/iepseDerbyNonXA" and "jdbc/iepseDerbyXA") to the cluster that you want to use. No change is required to the JDBC connection pools.
2. Create the connection pools and JDBC Resources manually. This option is good if you are new to IEP SE clustering. Follow these steps to create the connection pools and JDBC resources required for the clustering setup.

Install components and deploy the application

Follow these steps to install the required components and to deploy the application. If you don't aready have the components, the latest version can be downloaded from here. The application can be downloaded from here. The source of the application can be downloaded from here

Verify that the application is deployed properly

Download the project files and open the projects TupleBasedAvgCalculator and TupleBasedAvgCalculatorApp in Netbeans. Open the file TupleBasedAvgCalculatorProcess.wsdl. Locate the following line

<soap:address location="http://localhost:${HttpDefaultPort}/service/TupleBasedAvgCalculatorProcess_iep"/>

This is the address to which the events (SOAP messages) will be sent. However, the token ${HttpDefaultPort} will need to be replaced by actual port numbers. Since the application is deployed on both instances of the cluster, this token will be replaced by the values of "HTTP Default Port Number" that we noted for the two instances in the setp "Install components and deploy the application". Usually these values are 2110 and 2111. So the addresses will be

http://localhost:2110/service/TupleBasedAvgCalculatorProcess_iep OR
http://localhost:2111/service/TupleBasedAvgCalculatorProcess_iep

To make sure that the application is deployed correctly on both the instances, try to access the WSDL by entering the above addresses and appending "?WSDL" to it. For example, enter in a browser

http://localhost:2110/service/TupleBasedAvgCalculatorProcess_iep?wsdl OR  
http://localhost:2111/service/TupleBasedAvgCalculatorProcess_iep?wsdl

For both the cases, the browser should display the correct WSDL with the port number replaced. Since you know the port numbers and the URLs to access you application, you can configure the loadbalance.xml now. Refer to the tip on configuring loadbalancer.xml here.

Test the application in clustered mode

You will need a SOAP client like SOAP UI to send messages to the application. We will assume that you are using SOAP UI (the steps will be similar for other clients). Create a new project and import the WSDL in the SOAP UI. Usually a sample XML message is generated. Specify an integer value for the element named number - for example 10. Change the URL to "http://localhost/service/TupleBasedAvgCalculatorProcess_iep". Note that this will send the message to the HTTP Load Balancer. Send five messages. Change the value to 20. Send another five messages. The load balancer should have directed the requests to both instances. However, the processing would be done only by one instance and consequently, the output will be written to the file by that instance. Open TupleBasedAvgCalculatorProcess.wsdl, and locate the lines

    <port name="OutputPort_StreamOutput0" binding="tns:OutputBinding_StreamOutput0">
        <file:address fileDirectory="TupleBasedAvgCalculatorProcess_iep"/>
    </port>

Here the file path is relative, so the output file will be generated relative to the instance which is doing the processing. Search for a directory under name "TupleBasedAvgCalculatorProcess_iep" under {glassfish}\nodeagents\cluster1-nodeagent. You should see a file named StreamOutput0.txt. The value of the average should be 15. If you find the directory under instance-TWO then it means that instance-TWO owns the IEP process. Note: if you do not see the diectory or the file, try sending another message from the SOAP UI.

Now, let us stop the instance that owns the process, say instance-TWO, to see whether the other live instance (instance-ONE) takes over the processing. First send five messages from the SOAP UI. Then open a command prompt and go to {glassfish}\bin. Enter asadmin stop-instance instance-TWO. Send another 5-6 messages from the SOAP UI. Search for a diectory named "TupleBasedAvgCalculatorProcess_iep" under {glassfish}\nodeagents\cluster1-nodeagent\instance-ONE. You should see the file StreamOutput0.txt inside it with the correct average value. The StreamOutput0.txt file under instance-TWO should be unchanged. This means that instance-ONE has taken over the ownership of the IEP process. This shows that the IEP clustering and failover works!

How to test the application in clustered mode without the HTTP Load Balancer

In case you want to test without the LB, you can do so by simply sending the SOAP messages to

http://localhost:2110/service/TupleBasedAvgCalculatorProcess_iep OR 
http://localhost:2111/service/TupleBasedAvgCalculatorProcess_iep

Similar to previous case, import the project in SOAP UI. In the address, enter one of the values above to send messages directly to the instances of the cluster.

This page (revision-28) was last changed on 14-Apr-09 23:40 PM, -0700 by Murali.
This page was created on 06-Jul-08 11:45 AM, -0700 by Prashant Bhagat.

More info...