Monitoring ActiveMQ Using JMX Over SSH

2009/05/23 by 9 Comments

Frustration If you want to know true confusion and frustration I suggest you attempt to monitor ActiveMQ using JMX over SSH port forwarding.

For those who use ActiveMQ, the JMX monitoring is some pretty impressive stuff.  You can dive into connections, queues, topics, subscriptions, etc and get stats about the current state of the system.  However, this nirvana of information is practically unreachable when AMQ is in a data center.  For the first six months or so working on AMQ I was able to get around this using the web console which gives some level of detail on topics and queues but once we moved into trying to debug bigger and bigger problems the console wasn’t cutting it any longer.

The Investigation Begins

As with anything AMQ, I started looking into the JMX documentaion.  Step one was to enable JMX in the activemq.xml file.

<broker useJmx="true" brokerName="BROKER1">

Fire it up, forward port 1099 (the default JMX port), give it a try, and, no go.  Come to find out, JMX uses two ports, one which is defaults to 1099 and one which is negotiated at runtime.  No worries, there is an “Advanced JMX Configuration” seciton in the documentation which lead me to this xml configuration.

<managementContext>
    <managementContext connectorPort="2011" jmxDomainName="test.domain"/>
</managementContext>

Awesome, I can change the default 1099 port but nothing else.  The AMQ documentation was failing me, as usual.  Google, here I come.  After some searching around I was able to dig up AMQ-892.  (Power user tip, search AMQ’s Jira site once the documentation has inevitably failed.)  It turns out you can configure the RMI server port and have been able to do so since version 4.1.

<managementContext connectorPort="11099" rmiServerPort="11119" jmxDomainName="org.apache.activemq"/>

I particularly liked the “This necessitates a documentation update in the wiki.” comment from 2007.  Way to jump on that.  Confident that this is bound to work I forward port 11099 and 11109, give it a whirl, and, failure.  It was at this point that I started to despair.

Deeper Down The Rabbit Hole

Quickly running out of options I decided to try running the same configuration on a dev box in the office, and of course I could connect to it flawlessly.  To add insult to injury I tried another dev box and this one failed.  The only difference I could easily see was one was running a VM image and the other was native but this didn’t point at any easily understandable target.  I was done.  I conceded defeat.  I would soldier on without JMX.

Meanwhile, my co-worker had been pressuring me to start packet sniffing with wireshark.  I had resisted, mostly because I dislike getting that low level, but also because RMI is a binary protocol and I didn’t think I could learn much.  However, I was bloody and beaten at this point and gave in. I downloaded the package, fired it up and started sniffing away.  Sure enough the binary was a bunch of gibberish, but tucked away in there was one string, the hostname of the amq server I was trying to connect to.  What the hell is this?!?  That question is left to the reader but at least the issue was clear, the hostname was inside the data center and not something I could connect to from my machine.  It also jived with the test on the local dev boxes.  The box with the VM image had an internal hostname that was not in the office dns, where as the native machine had an addressable hostname.

I attacked the problem with renewed vigor.  I found a JConsole FAQ on the AMQ website that I had previously dismissed for its cryptic message.  However, this time it made sense.  I quickly added

-Djava.rmi.server.hostname=127.0.0.1

to the Java Service Wrapper configuration for AMQ so communication would be forced back through the SSH tunnel, rebooted AMQ, and, wait for it, VICTORY.

The Morale of the Story

So the short answer is add the broker configuration, the management context (the second on up there), and the rmi hostname configuration, port forward 11099 and 11109, and fire up jconsole

jconsole service:jmx:rmi://127.0.0.1:11119/jndi/rmi://127.0.0.1:11099/jmxrmi

As with most of these posts, so easy once you know, but so hard to get there.

Filed under Software Engineering Tagged with , , ,

Unit Testing Spring Jersey Integration

2009/04/20 by 3 Comments

Recently, I’ve been working on a side project which has given me the opportunity to play with an interesting Java technology stack. After a few weekends of working on the build and moving my way up the stack from database to service layer I was at the point of writing a full integration test. In the past I’ve done integration testing with an iBatis/Spring stack, which, with the help of DBUnit is a fairly simple exercise. However, this project is heavily REST based and the addition of Jersey (JAX-RS) was a serious curveball.

To really test the whole stack an embedded webserver is required.  Looking through the Jersey/Spring unit tests they make use of an exploded WAR using Glassfish.  Assuming this was the optimal solution I added Glassfish to my ivy.xml config and ran an ivy update.  However, once I realized the overwhelming volume of jars being downloaded I decided it was much too heavy weight of a solution.

My next choice was Jetty.  I have had experience with using Jetty in the past but I quickly ran into issues trying to get it working right with Jersey, Spring, and an exploded WAR configuration.  It was then that I saw the GrizzlyServerFactory referenced in another Jersey unit test.  After some prodding and reading into the source code I ended up with the following:

final URI baseUri = UriBuilder.fromUri( "http://localhost/" ).port( 9998 ).build();
final ServletAdapter adapter = new ServletAdapter();
adapter.addInitParameter( "com.sun.jersey.config.property.packages", <your-package-name> );
adapter.addContextParameter( "contextConfigLocation","classpath:applicationContext.xml" );
adapter.addServletListener( "org.springframework.web.context.ContextLoaderListener" );
adapter.setServletInstance( new SpringServlet() );
adapter.setContextPath( baseUri.getPath() );
SelectorThread threadSelector = GrizzlyServerFactory.create( baseUri, adapter );

We start by making a base Uri for localhost at a high numbered port.  Then we build the Grizzly ServletAdaptor.  The first parameter is the package name where your Jersey enabled are located.  The next two lines are taken directly from the web.xml file that is used for the actual WAR.  The first is the config location for the Spring Context and the second is the ContextLoaderListener servlet.  Finally we add the Jersey SpringServlet and the context path.  With these two objects created we can build a SelectorThread and begin using the server using the Jersey Client api that I talked about in my previous post.  When you are done, don’t forget to call stopEndpoint() on the SelectorThread when you are done.

EDIT:

I’ve recently come across one additional helpful tip regarding this setup. If you need to get access to your ApplicationContext, for instance to get access to the spring managed DataSource for database testing, there is a simple but non obvious way of doing this. The simple part is using Spring’s built in utilities for getting the application from a ServletContext.


WebApplicationContextUtils.getWebApplicationContext( adapter.getServletInstance().getServletConfig().getServletContext() );

However, when I first tried this I got an unexplained null pointer exception. After a long while trying to google around for the answer I gave up and dug into the source code. Turns out you need to add one important property to the ServletAdaptor for this to all work.


adapter.setProperty( "load-on-startup", 1 );

This will force your adaptor to initialize the servlet immediately and you populate the ServletConfig and ServletContext so you can get the ApplicationContext from them.

Filed under Software Engineering Tagged with , , , , ,

REST calls and JSON results in Java

2009/04/20 by 1 Comment

I recently had the need interact with Twittervision’s RESTful api from Java. As usual, I started googling around for a solution but nothing obviously stood out. There are a handful of RESTful java frameworks, but these focus on providing a RESTful api and not consuming one. Finally, after an annoyingly difficult search I found an impressively simple solution based on Jersey.

The point to realize is that Jersey provides both a sever and a client api. To make use of the client api requires three simple lines:

Client client = Client.create();
WebResource webResource = 
client.resource("http://twittervision.com/user/current_status/bdarfler.json");
String response = webResource.get(String.class);

The result is a string of JSON, but now, how to parse it. After searching around I ended up settling on json-simple. A few simple lines of code and we can get the address for any twitter user.
final JSONObject jsonObj = (JSONObject) JSON_PARSER.parse( response );
if ( jsonObj != null &amp;&amp; jsonObj.containsKey( "location" ) )
{
final JSONObject location = (JSONObject) jsonObj.get( "location" );
return location.get( "address" ).toString();
}

So, once again, simple stuff but not as obvious as I was hoping it would be.

Filed under Software Engineering Tagged with , , , , ,

Integrate ant with dbdeploy not in the classpath

2009/01/26 by 5 Comments

FrustrationThe need for a schema versioning system at LocaModa finally became a priority this sprint so I’ve spent the past few days researching and diving into implementation.  After comparing LiquiBase, dbdeploy, MIGRATEdb, and dbmigrate I settled on the simple SQL driven solution that dbdeploy provides and started integrating it with our Ant build.  Pramod Sadalage over at Agile DBA put up a nice post on the basics but I quickly hit a wall, the MySQL driver was not in my classpath.

As I mentioned earlier we are using Ivy for our dependancy management which makes it difficult to throw jars into the Ant classpath since they are automatically downloaded at build time.  So far this hasn’t been an issue since most Ant tasks allow you to specify the classpath as a parameter within the task.  Dbdeploy does not.  Goggling around I quickly found a posting that suggested patching the dbdeploy ant task as the only way to go, so off I went.  However, a few hours later, after wandering through the Java classloader abyss I was stymied.

I backed out of the changes and decided a different tactic was needed; back to Goggle I went.  This time I came across a post which suggested all I needed was to add the jar to the taskdef classpath  Could it really be that easy?  Switching back to my build.xml I doctored up the dbdeploy taskdef, kicked off a build and sure enough the damn thing worked.

So, if you want to centrally manage your ant dependencies, just remember to add the necessary classpath references to your taskdefs, like so:

<taskdef name="dbdeploy"
        classname="net.sf.dbdeploy.AntTarget"
        classpath="${dbdeploy.jar}:${mysql-connector.jar}"/>


Filed under Software Engineering Tagged with , , , ,

Controlling Log4j Logs with Contexts and Filters

2008/11/21 by 5 Comments

Its a problem we’ve all had. Something has gone awry, you jack up the log level to debug and all of the sudden you are inundated with everything under the sun. While one option could be changing some of the logs to trace level there is another option, using Log4j contexts. Log4j contains two different contexts mapped and nested. In this post I’ll focus on nested.

The first thing to do is to isolate the section of your code that you want to create a context for. Once you have accomplished this you make a static call to NDC.push(). The parameter passed in is the string name for the context you would like create. I recommend pulling these strings out into a constants file ease of use and additionally calling .intern() on them so you can use pointer equality checks later when we code up the filters. When you leave the context you call NDC.pop() and life goes back to normal. Finally, remember to call NDC.remove() when you leave the thread.
The code ends up looking like this:

NDC.push(Constants.HEARTBEAT_CONTEXT);
try{
  if (log.isDebugEnabled()) {
    log.debug("[MASTER/SLAVE] Publishing heartbeat");
  }
  doStuff();
}
finally {
  NDC.pop();
  NDC.remove();
}

Now that we have the logs tagged with the correct context we need to create a log4j filter by extending the filter class and overriding the decide(LoggingEvent event) method. The important concept to remember here is that filters can be chained which means your decision results in either a DENY, ACCEPT, or NEUTRAL decision where deny forces the log to be rejected, accept force the log to be written and neutral passes the decision down the chain. Additionally, you access the context by calling getNDC() on the LoggingEvent object passed in to the decide method.
The code then looks like this:
public class HeartbeatFilter extends Filter
{
  @Override
  public int decide(final LoggingEvent event) {
    if (event.getNDC() == Constants.HEARTBEAT_CONTEXT) {
      return DENY;
    }
    return NEUTRAL;
  }
}
Finally we have to add this filter to our log4j configuration. For this we have to make sure we are using the xml configuration file type as it is more expressive and allows us to define filters to use where as the property configuration file type does not. All we have to do is add a filter tag to an appender in the xml and point it at our class.
It looks something like this:

<appender name="console" class="org.apache.log4j.ConsoleAppender">
  <param name="Target" value="System.out"/>
  <layout class="org.apache.log4j.PatternLayout">
    <param name="ConversionPattern" value="%-5p %c{1} - %m%n"/>
  </layout>
  <filter class="my.package.HeartbeatFilter" />
</appender>

Filed under Software Engineering Tagged with , , , ,

Pimpin' your Ant build with Ivy

2008/11/11 by 3 Comments

As of late Maven has been making headway in the Java build world.  However, there are some who are either in the position of working with an existing system build with Ant or who don’t want to get locked into the Maven file structure.  For those folks there is Ivy to the rescue.

Ivy is an “agile dependancy manager” which recently released 2.0.0-rc1.  Thought jam packed with lovely features the first one that I have put to use is the ability to manage 3rd party dependancies, automagically downloading them, dealing with version conflicts and ensuring that any dependancies they in turn have are dealt with.

Ivy allows for two configuration files, one for describing where to look for jars and one to describe what jars to look for.  Beyond that, the rest of the work is done within your ant build.xml file.  Lets start with the ivysettings.xml file which describes where to look for jars.

<?xml version="1.0" encoding="UTF-8"?>
<ivysettings>  
    <settings defaultResolver="chained"/>  
    <resolvers>  
        <chain name="chained" returnFirst="true">
            <url name="jboss">  
                <artifact pattern="http://repository.jboss.com/maven2/javax/jms/[module]/[revision]/[module]-[revision].jar" />
            </url> 
            <ibiblio name="ibiblio" m2compatible="true"/>
            <filesystem name="local-filesystem">
                <artifact pattern="/local/file/system/[organization]-[revision]/lib/[module].jar" />
                <artifact pattern="/local/file/system/[organization]-[revision]/[module].jar" />
            </filesystem>
        </chain>  
    </resolvers>  
</ivysettings>  

There are a few things to point out here.  First is the resolvers tag. There are a few different type of resolvers and I have chosen to use the chain resolver here.  If you are lucky, everything you need with be in ibiblio and all you would need would be the middle ibiblio tag.  However, if you are using any jars that are not mainstream you will have to end up specifying the url here.  The url tag I’m using here is to deal with the unfortunate issue of jboss not having the latest javax.jms jar up on ibiblio.  Here we specify the url pattern to down load the jar directly as long as the url points at a valid maven style repo.  Notice you can insert the module revision and organization as parameters.  These are passed in from the ivy.xml file which we will discuss next. Also note that in this case there is already a javax.jms jar in ibiblio, just not the right version so we put the url tag first.  Additionally, you might have to deal with some jars that are proprietary or not in a valid maven style repo.  In this case you will have to download these using ant (show later) or store them in source control.  However you can still leverage Ivy by having it pick them up from the local filesystem.

Now onto the ivy.xml file.

<?xml version="1.0" encoding="ISO-8859-1"?>
<ivy-module version="2.0">
    <info organisation="locamoda" module="lms" status="integration" />
    <configurations>
        <conf name="build"  description="build dependancies"/>
        <conf name="runtime" description="runtime dependancies"/>
        <conf name="junit" extends="build, runtime" description="junit dependances"/>
        <conf name="mysql" description="mysql dependances"/>
        <conf name="cobertura" description="cobertura dependances"/>
        <conf name="findbugs" description="findbugs dependances"/>
    </configurations>
    <dependencies defaultconfmapping="*->default">
        <dependency org="com.google.collections" name="google-collections" rev="0.8" conf="build, runtime" />
        <dependency org="log4j" name="log4j" rev="1.2.14" conf="build, runtime" />
        <dependency org="org.apache.activemq" name="activemq-all" rev="5.1.0" conf="build, runtime" />
        <dependency org="commons-lang" name="commons-lang" rev="2.4" conf="build, runtime" />
        <dependency org="commons-dbcp" name="commons-dbcp" rev="1.2.2" conf="build, runtime" />
        <dependency org="org.mortbay.jetty" name="jetty" rev="6.1.11" conf="build, runtime" />
        <dependency org="junit" name="junit" rev="4.5" conf="build" />
        <dependency org="mysql" name="mysql-connector-java" rev="5.1.6" conf="mysql, runtime"/>
        <dependency org="xerces" name="xercesImpl" rev="2.8.1" conf="runtime"/>
        <dependency org="commons-logging" name="commons-logging" rev="1.1.1" conf="runtime" />
        <dependency org="javax.jms" name="jms" rev="1.1" conf="runtime" />
        <dependency org="org.easymock" name="easymock" rev="2.4" conf="junit" />
        <dependency org="org.easymock" name="easymockclassextension" rev="2.4" conf="junit" />
        <dependency org="org.dbunit" name="dbunit" rev="2.3.0" conf="junit" />
        <dependency org="net.sourceforge.cobertura" name="cobertura" rev="1.9rc1" conf="cobertura, junit" />
        <dependency org="findbugs" name="findbugs" rev="1.3.5" conf="findbugs, build" />
    </dependencies>
</ivy-module>
Lets start with the dependencies first.  Each dependancy has an org, name, rev and an optional conf.  The org, name and rev correspond to the organization, module, and revision in the ivysettings.xml as you might assume.  For your common jars you can find the necessary values by looking up the package at mvnrepository.com of course they call org group, module artifact, and revision version, just to keep you on your toes.  Additionally you should notice the defaultconfmapping=”*->default”, if you forget this things blow up quick and in weird ways. The dependancy thing is pretty sweet but the real power comes when you start in on the configurations.  As you can see I have defined six configurations that correspond to different sets of jars that I need to reference in my build.xml.  Configurations can inherit from eachother using the extends so junit is a superset of build, runtime, and the junit specific jars.  Assigning jars to configurations is as easy as listing the right configurations in the conf attribute of the dependancy tag.
Next we tie this all together in the build.xml.
    <target name="dl-binaries" depends="checkfiles, dl-findbugs, dl-ivy" description="download runtime binaries to external"/>

    <target name="checkfiles" description="checks if files have already been downloaded">
        <available file="${findbugs.dir}" type="dir" property="findbugs.exists" />
        <available file="${ivy.jar.file}" property="ivy.exists" />
    </target>

    <target name="dl-findbugs" description="download findbugs" depends="checkfiles" unless="findbugs.exists">
        <get src="${findbugs.url}" dest="${findbugs.zip}" usetimestamp="true" />
        <unzip src="${findbugs.zip}" dest="${external_home}" />
        <delete file="${findbugs.zip}"/>
    </target>

    <target name="dl-ivy" description="download ivy" depends="checkfiles" unless="ivy.exists">
	<mkdir dir="${ivy.jar.dir}"/>
        <get src="http://repo1.maven.org/maven2/org/apache/ivy/ivy/${ivy.install.version}/ivy-${ivy.install.version}.jar"
            dest="${ivy.jar.file}" usetimestamp="true"/>
    </target>

    <target name="ivy" depends="dl-ivy" description="configure ivy" >
    	<path id="ivy.lib.path">
            <fileset dir="${ivy.jar.dir}" includes="*.jar"/>
        </path>
        <taskdef resource="org/apache/ivy/ant/antlib.xml"
              uri="antlib:org.apache.ivy.ant" classpathref="ivy.lib.path"/>
        <ivy:settings file="ivysettings.xml" />
    	<ivy:retrieve />

        <!-- Define Paths -->
        <ivy:cachepath pathid="build.path" conf="build" />
        <ivy:cachepath pathid="runtime.path" conf="runtime" />
        <ivy:cachepath pathid="junit.path" conf="junit" />
        <ivy:cachepath pathid="cobertura.path" conf="cobertura" />
        <ivy:cachepath pathid="mysql.path" conf="mysql" />
    	<ivy:cachepath pathid="findbugs.path" conf="findbugs" />
    </target>
To start off we have the ivy target which takes care of a few things.  First it depends on dl-ivy for bootstrap downloading of ivy.  Once Ivy is downloaded it sets up the ivy path, defines the ivy ant tasks, and reads the settings file.  At this point all the magic happens with the <ivy:retrieve /> tag.  This goes out, reads the ivy.xml and grabs all the necessary jars.  Once that is done we can create ant paths for all the different configurations we defined in ivy.xml.  This will become super useful later on.  Another thing to notice here is downloading of findbugs.  This is done for two reasons.  First this is a good example of downloading 3rd party libs using ant that are not in a maven style repo.  This target goes out and gets findbugs, unzips it and then ivy can use it through the local filesystem resolver in the ivysettings.xml. The other reason it is here is that the default way of downloading findbugs through the maven style repo conflicts horribly with the findbugs ant plugin.  The plugin expects the files in a very specific layout which is broken when you download from maven vs downloading it from their site.  I use this idiom for downloading some other picky 3rd party jars as well as some binaries that are useful for development but are not actually needed for java.
Finally we get to see the power of these configurations.
     <target name="compile" depends="init,messages,dbobjs" description="compile all the java source">
       <javac target="${TARGET}" source="${SOURCE}" debug="${DEBUG}" destdir="${build}" includes="**/*.java">
            <src path="${src}"/>
            <classpath>
                   <path refid="build.path" />
            </classpath>
            <!-- <compilerarg value="-Xlint"/> -->
        </javac>
    </target>
All we have to do is add the build.path reference and away we go.  We can do the same for findbugs, mysql, junit, adding runtime dependancies into a jar that we create in our build and so on.  Need to add a new jar?  Look it up in the maven repo, add it to ivy.xml and define it as a build, runtime, junit, etc dependancy and away you go.  Pretty slick!

Filed under Software Engineering Tagged with , , ,

Groovin' Up Your Database Tests

2008/09/04 by Leave a comment

While TDD proclaims the gospel of fine grained tests and a healthy dose of mock objects, at some point, interaction with a database becomes inevitable.  Luckily, there is DbUnit to assist us.  DbUnit is a fantastic framework which allows for painless setup of database data sets, ensuring repeatable, self contained unit tests. However, there are already plenty of tutorials on the subject.  Instead, what I want to talk about is the dark side of dbunit… xml.

Using dbunit on any reasonably sized project ends up with an explosion of xml data set files, even if care is taken to factor out common data sets and reuse them when possible.  For the longest time this seemed an unfortunate but necessary evil of using such an amazingly useful library, but that was before I attended the Northeast No Fluff Just Stuff in April.  Before that point I had been interested in Groovy and was looking for places to add it to our code base (actually snuck it into a script that ran in the build) but I hadn’t really pondered the possibilities it provided.  However, NFJS was over flowing with talks and general love for Groovy.  Somewhere between the Groovy unit testing and the Groovy MOP talks it dawned on me.  Why not inline the xml using a Groovy builder?!

Groovy has this fantastic feature called builders which allows for very compact syntax to generate any sort of markup (html, xml, java swing, etc).  Of course the one we want to focus on is the xml builder.  By simply changing the file extension from .java to .groovy on the unit test we are already half way there.  After using Groovy’s syntax for creating xml we simply call .toString() on the underlying writer, wrap it in a new StringReader() and then pass create a new FlatXmlDataSet() with the reader as the only constructor argument.

If you are using Ant for your build you can simply wire up the groovyc task to compile all of your groovy tests and then call junit on the .class files.  If you want to get this working in Eclipse, best of luck, I was able to kinda get it working under windows xp but now that I have switched to os x the groovy plugin screams bloody murder with 64-bit java under os x in Eclipse.

Filed under Software Engineering Tagged with , , , , ,

Newer posts →

Follow

Get every new post delivered to your Inbox.