Clustering Railo with Tomcat and Apache

Mittwoch, 30. September 2009

We are going to set up a "High-Availability" cluster with load balancing using Apache to perform the proxying, Tomcat as our servlet container and Railo to serve up our CFML. You notice that I placed high availability in quotation marks; this is because while automatic failover will be enabled our sessions won't be replicated to all nodes in the cluster. The reason we won't be replicating the sessions is mainly because of performance reasons. Though session replication using Tomcats Delta Manager for small clusters is considered to be a possibility.

First Steps

  1. Latest Apache version http://httpd.apache.org/download.cgi
  2. Tomcat 6 http://tomcat.apache.org/download-60.cgi
  3. Railo Custom (jar files) http://www.getrailo.org/index.cfm/download/
  4. Latest Java JDK (Installation assumed to happen prior to tomcat install)
And of course you will need two or more Servers.

Installing Apache

Do a basic install of Apache and set up a standard virtual host to point to where the CFML files lets say c:/inetpub/wwwroot/ will be, the vhosts.conf file looks something like this.

<VirtualHost *:80>
DocumentRoot "c:/inetpub/wwwroot/"
ServerName railo.local
<Directory />
Options FollowSymLinks
AllowOverride None
Order deny,allow
allow from all
</Directory>
DirectoryIndex index.html index.htm index.cfm index.cfml
</VirtualHost>

In the wwwroot add an index.html file with "hello world", add an entry for railo.local in the hosts file and test the install http://railo.local/index.html to make sure that Apache install is working before going onto the next step.

Installing Tomcat

There is one major point to remember when installing Tomcat as a service on Windows 64Bit, it won't work with the MSI installer since the tomcat6.exe is 32 bit. You will need to get a 64 bit version of tomcat from http://svn.apache.org/viewvc/tomcat/tc6.0.x/tags/TOMCAT_6_0_16/res/procrun/amd64/. Once you have replaced the tomcat6.exe file with the one from the site it should work.

If you are installing on a 32 bit machine you can install it under port 8080, we won't be using the connector later anyway. Once you have it installed it test to make sure it actually works. http://localhost:8080 should give you the tomcat welcome screen. Now that you have confirmed that it works you can shut it down and edit a few things.

Web.xml

In the web.xml (/conf/) you need to add the railo servlet code. This has been covered in several blog entries but I will paste the snippets here again. Add the following after the last servlet tag:

<!-- servlet mapping for Railo -->
<servlet-name>CFMLServlet</servlet-name>
<servlet-class>railo.loader.servlet.CFMLServlet</servlet-class>
<init-param>
<param-name>railo-server-directory</param-name>
<param-value>d:/tomcat/railo-server-config/</param-value>
<description>Configuraton directory</description>
</init-param>
<init-param>
<param-name>configuration</param-name>
<param-value>d:/tomcat/railo-web-config/{web-context-hash}</param-value>
<description>Railo Web Directory</description>
</init-param>
<load-on-startup>2</load-on-startup></servlet>
<servlet-mapping>
<servlet-name>CFMLServlet</servlet-name>
<url-pattern>*.cfm</url-pattern>
</servlet-mapping>
<servlet-mapping>
<servlet-name>CFMLServlet</servlet-name>
<url-pattern>*.cfml</url-pattern>
</servlet-mapping>
<servlet-mapping>
<servlet-name>CFMLServlet</servlet-name>
<url-pattern>*.cfc</url-pattern>
</servlet-mapping>
<!-- servlet mapping for AMF -->
<servlet>
<servlet-name>AMFServlet</servlet-name>
<servlet-class>railo.loader.servlet.AMFServlet</servlet-class>
<load-on-startup>3</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>AMFServlet</servlet-name>
<url-pattern>/flashservices/gateway/*</url-pattern>
</servlet-mapping>

The initialization parameters at the top of the CFMLServlet specify where the web and sever context information will be saved. Instead of the default which is {web-root-directory}/WEB-INF/railo

Then add the following to the welcome files list at the bottom of the file.

<welcome-file>index.cfm</welcome-file>
<welcome-file>index.cfml</welcome-file>

We are now finished with the web.xml file

Server.xml

The server.xml file is also located in the conf directory. Here we will be adding a Host definition to mirror our virtual host, yes you need to have it in both Apache/conf and Tomcat/conf! We will also be preparing for clustering in the server.xml file too.

On approximately like 71 is the HTTP connector for port 8080 which we no longer need so go ahead and comment the connector block out. The next thing we will do is make sure that our AJP connector is NOT commented out.

<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />

We then add a new attribute to the Engine tag, namely we will be adding the jvmRoute attribute to the Engine tag to support load-balancing. I have called this node of the cluster "node1"


<!-- You should set jvmRoute to support load-balancing via AJP ie :
<Engine name="Standalone" defaultHost="localhost" jvmRoute="jvm1">
-->
<Engine name="Catalina" defaultHost="localhost" jvmRoute="node1">

The last thing we need to do is set up our Host.

<Host name="railo.local" appBase="c:/tomcat/webapps">
<Context path="" docBase="c:/inetpub/wwwroot"/>
</Host>

If you don't want railo to put WEB-INF folders in every subfolder in the tomcat/webapps directory you should delete the default , we don't need it anyway. We are now finished with tomcat set up.

Installing Railo

Create a folder "railolib" in your tomcat install directory c:\tomcat\railolib in this case. Now extract the Railo Jar files from the zip file into this directory. This is done to keep the tomcat specific Jar files separate from the Railo Jars instead of packing them all in the same folder. In order to make them visible on start-up you need to edit the catalina.properties file located in the tomcat/conf directory. Find the common.loader line (line 47), it should look something like the following but without the bold part:

common.loader=${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/railolib,${catalina.home}/railolib/*.jar

The part in bold needs to be added and is telling Tomcat where to look for additional jar files.

If you start Tomcat now from the command prompt C:\tomcat\bin>startup you should see that Tomcat automatically builds the WEB-INF folder under c:/inetpub/wwwroot. The railo-server-config folder will get filled; additionally the railo-web-config folder will be populated with a folder with a hash name representing your Railo web.

Load Balancing

We will be setting up our load-balancing for a single server at the moment so we can at least see our cfml pages being processed before complicating things with load-balancing. In your httpd.conf file uncomment the following:

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so

In your Vhosts.conf file add the following to your VirtualHost:

<Proxy balancer://mycluster>BalancerMember ajp://<your server1 ip>:8009/ route=node1 loadfactor=1
ProxySet lbmethod=byrequests
</Proxy>
ProxyPreserveHost On
ProxyPass / balancer://mycluster/ nofailover=On


Save your vhosts.conf and httpd.conf file and restart apache. You should now be able to brows to http://railo.local:8080/railo-context/admin/server.cfm and get your Administrator. Please log in and set the session type to J2EE. Create an application.cfm file and place the following:

<cfapplication name="myapplication" sessionTimeout = #CreateTimeSpan(0, 0, 0, 60)# SessionManagement = "Yes">

In your index.cfm file add the following (thanks to coldshen for the script)

<cfscript>
mySession = getPageContext().getRequest().getSession(true);
ival = mySession.getAttribute("simplesession.counter");
if (NOT isDefined("ival")){
ival = 1;
} else {
ival = ival + 1;
}
mySession.setAttribute("simplesession.counter", ival);
</cfscript>
<cfoutput>
CFSESSIONID:
<br /><strong>#session.URLToken#</strong>
<br />
JSESSIONID:
<cfdump var="#mySession.getID()#">
Number of refreshes:
<cfdump var="#ival#">
Cookies:
<cfdump var="#cookie#">
<cfif NOT structKeyExists(cookie,"JSESSIONID")>
NO JSESSIONID! Are you proxy/rewriting? Did you remember to change the cookie path (ProxyPassReverseCookiePath)?<br />
</cfif>
Session:
<cfdump var="#session#">
</cfoutput>

When you now go to http://railo.local/index.cfm you should see something like:

CFSESSIONID:
CFID=967dbb52-286d-4099-86f3-4702624b3f95&CFTOKEN=0&jsessionid=68C7222F15AFD1CA4B8A4DA9F06836D0.node1

Notice the node number way at the end? That's telling us what node we are currently on.

Naturally at this point we only have one node. Now would be a good time to set up your second server following the above steps. Before enabling the load balancing you may want to verify that second node works. Once you have verified that it works you just need to make sure that your second machine has jvmRoute="node2" and that you have both nodes in both your vhosts files.

<Proxy balancer://mycluster>
BalancerMember ajp://<your server1 ip>:8009/ route=node1 loadfactor=1
BalancerMember ajp://<your server2 ip>:8009/ route=node2 loadfactor=1
ProxySet lbmethod=byrequests
</Proxy>

Restart both apache machines and hit your index.cfm page, hit refresh a couple of times and you should see in the cfsessionid that node1 turns to node2 back and forth for almost every request.

Problem: Node Number Doesn't Change

Check the error log file for apache if you see something like this "(OS 10061)No connection could be made because the target machine actively refused it. : proxy: AJP: attempt to connect to 192.168.1.34:8009 (192.168.1.34) failed" it means that the windows firewall is blocking port 8009. Even if you have the windows firewall off you still need to add the port to the exceptions, for some reason windows will continue to block anything that doesn't show up in the exceptions list even if you have the firewall turned off! Thanks Microsoft for wasting my time looking in the wrong place for the problem, I should have known better.

Double Negative: nofailover

This could be a bit confusing but don't make the mistake and set nofailover=Off that means if one of your nodes goes down it will NOT failover. This value needs to be On. It would make more sense if it was simply called "failover", then we could use true and false values instead. You can test this by leaving it as "On" and kill the tomcat process on the server you are currently on, you should notice that the node number stays on the node that is still up. Now set the nofailover=off on the server you are currently on and restart Apache, you should receive a "503 Service Not Available" error.

Sticky Sessions

Now that we have proven that our load is divided equally between the two servers we can set up "sticky sessions". You need sticky sessions since we aren't replicating the sessions across all the nodes.

Once a session is established the user will stay on that node until the session expires or the machine crashes whichever comes first (hopefully the session). We enable sticky sessions by adding ProxySet stickysession=JSESSIONID in the tag so that it looks something like:


<Proxy balancer://mycluster>
BalancerMember ajp://192.168.1.25:8009/ route=node1 loadfactor=1
BalancerMember ajp://192.168.1.34:8009/ route=node2 loadfactor=1
ProxySet lbmethod=byrequests
ProxySet stickysession=JSESSIONID
</Proxy>


Congratulations you now have two load-balanced Railo servers with automatic fail-over! zurück