Jan Van Besien's blog

A Java library for IPv6

2011-10-12T21:18:00.001+02:00

A while ago, I wrote a library to work with IPv4 concepts (IPv4 addresses, networks, network masks, prefix lengths, ranges of addresses etc) in Scala. Along those lines, I recently created a similar library to work with IPv6 concepts. This time though, I used good old Java.

If you simply need to make IPv6 connections, java.net.Inet6Address will be sufficient. However when doing things a little bit more advanced with the addresses themselves, I find java.net.InetAddress (both Inet4Address and Inet6Address) lacking a lot of functionality.

Here is a short overview of what can be done with my java-ipv6 library.

IPv6Address

The class IPv6Address represents an IPv6 address.

final IPv6Address iPv6Address = IPv6Address.fromString("fe80::226:2dff:fefa:cd1f");

Internally, the IPv6Address class uses two long values to store the IPv6 address. This makes for an optimized implementation, and a lot of bit twiddling fun for me while writing it...

IPv6Address can be used to make simple calculations on IPv6 addresses, such as addition and subtraction.

final IPv6Address iPv6Address = IPv6Address.fromString("fe80::226:2dff:fefa:cd1f");
final IPv6Address next = iPv6Address.add(1);
final IPv6Address previous = iPv6Address.subtract(1);
System.out.println(next.toString()); // prints fe80::226:2dff:fefa:cd20
System.out.println(previous.toString()); // prints fe80::226:2dff:fefa:cd1e

IPv6AddressRange

The class IPv6AddressRange represents a continuous range of consecutive IPv6 addresses.

final IPv6AddressRange range = new IPv6AddressRange(IPv6Address.fromString("fe80::226:2dff:fefa:cd1f"),
                                                    IPv6Address.fromString("fe80::226:2dff:fefa:ffff"));
System.out.println(range.contains(IPv6Address.fromString("fe80::226:2dff:fefa:dcba"))); // prints true

IPv6AddressRange contains methods to iterate over all the addresses in the range. Ranges can be compared with other ranges by checking if they overlap or if one range contains the other range.

IPv6Network

An IPv6Network is a range (extends IPv6AddressRange) that can be expressed as a network address and a prefix length.

final IPv6Network range = new IPv6Network(IPv6Address.fromString("fe80::226:2dff:fefa:0"),
                                          IPv6Address.fromString("fe80::226:2dff:fefa:ffff"));
final IPv6Network network = IPv6Network.fromString("fe80::226:2dff:fefa:0/112");
System.out.println(range.equals(network)); // prints true

Note that every IPv6Network is also an IPv6AddressRange, but not all IPv6AddressRanges are valid IPv6Networks. That is why, when constructing an IPv6Network from a range in between a first address and a last address, the smallest possible IPv6Network (i.e. the one with the longest prefix length) will be constructed.

IPv6AddressPool

An IPv6AddressPool is like a range (extends IPv6AddressRange) of which certain subnets are "allocated" and other are "free".

final IPv6AddressPool pool = new IPv6AddressPool(IPv6Address.fromString("fe80::226:2dff:fefa:0"),
                                                 IPv6Address.fromString("fe80::226:2dff:fefa:ffff"), 120);
System.out.println(pool.isFree(IPv6Network.fromString("fe80::226:2dff:fefa:5ff/120"))); // prints true
final IPv6AddressPool newPool = pool.allocate(IPv6Network.fromString("fe80::226:2dff:fefa:5ff/120"));
System.out.println(newPool.isFree(IPv6Network.fromString("fe80::226:2dff:fefa:5ff/120"))); // prints false

This was only a short introduction. Much more can be done with these types. I invite you to have a look at the javadoc and of course the actual source code.

Immutable

I decided to make all types immutable. For things like IPv6Address and IPv6Network, this obviously makes sense because they represent immutable concepts. For IPv6AddressPool, I was in doubt whether immutability was the right choice. When allocating an address in a pool, immutability means I have to return a new IPv6AddressPool instance. Internally the IPv6AddressPool maintains a SortedSet of all ranges of addresses that are still available in the pool. This SortedSet (TreeSet) thus has to be copied each time a new IPv6AddressPool is to be created. It is not a deep copy (the IPv6AddressRanges in the SortedSet are immutable themselves), but still constructing new TreeSet instances each time seems sub optimal. It would be nice to investigate if I can improve the situation using some kind of persistent data structure to replace the TreeSet. I had a quick look at pcollections, but it doesn't seem to provide an alternative to TreeSet with SortedSet semantics. Other suggestions are much appreciated!

Load Balancing Actors with Work Stealing Techniques

2010-03-31T19:59:00.005+02:00

Actor concurrency is well known to provide a more convenient model to concurrency than traditional thread based approaches. In contrast with threads, you can have as many actors as you want (if memory permits) and the actor implementation will take care of processing actors on threads whenever necessary behind the scenes.

In Akka, which among other things provides actors for the JVM (in Scala), the component which schedules actors for processing on a set of threads is called the "dispatcher". The default dispatcher uses a fixed size thread pool to process actors on. When a message is sent to an actor, a task is scheduled on the thread pool to process all the messages in the mailbox of the actor.

What actor concurrency does not provide out of the box is load balancing among equal actors ("equal" meaning they are both capable of processing the same types of messages).

You can implement load balancing on top of actor frameworks (example in Akka), by sending messages to a set of actors in some clever way (round robin, always choosing the one with the smallest mailbox, ...) but this has a few limitations. This kind of load balancing is rather "static": once a message is sent to an actor, it stays there until that actor processes it. If another actor is already done, it can never help the slow actor for messages that have already been distributed.

Here comes work stealing.

Work stealing is an idea used for example in the Fork/Join framework. Applied to actors, it implies that idle actors can "steal work" from the tail of the mailbox of busy actors. Of course actors are no threads, so "idle actors" can not really do anything because they are not assigned to a thread as long as they stay idle. This is why the implementation is a bit more subtle than the conceptual idea.

I have implemented a work stealing dispatcher for Akka actors. Although its called "work stealing" the implementation actually behaves more as "work donating" because the victim actor takes the initiative. I.e. it actually donates work to its thief, rather than having the thief steal work from the victim.

There is some cleverness in the dispatcher implementation using ReentrantLock.tryLock to check if an actor which is being scheduled on the dispatcher is actually capable of processing its mailbox (another thread might already be busy). If an actor can not process messages for this reason, it will try selecting a thief (round robin), donate a message from its mailbox to the thief (a ConcurrentLinkedDeque, using Deque.pollLast) and process the message in the thief.

Using the victim thread to do the "work donating" while this thread can not be used to process the victims mailbox itself because another thread is already doing that, implies that no threads are "waisted" on the work stealing process. Furthermore, when selecting a thief, the algorithm will make sure that the thief is idle, such that no messages are being redistributed to an actor which can not process them immediately. This also prevents stolen messages from being stolen again by another actor later on.

To use it, you simply declare to use the work stealing dispatcher in your Akka actors:

val dispatcher = Dispatchers.newExecutorBasedEventDrivenWorkStealingDispatcher("pooled-dispatcher")

class WorkstealingActor extends Actor {
  messageDispatcher = dispatcher
  def receive = {
    case test => {/* process message */}
  }
}

When you create multiple instances of this actor, they'll all share the same work stealing dispatcher.

Automatically reload log4j configuration in tomcat

2010-02-15T20:10:00.005+01:00

I was looking for a way to have Log4j reload its configuration file automatically when it changes, in Tomcat. Log4j can do this with the configureAndWatch method, but the default initialization procedure (simply putting a log4j.properties file in the classpath) doesn't use configureAndWatch. You have to write at least a little bit of code to get Log4j to do this. I found the easiest solution for integration with Tomcat to be to implement a Tomcat LifecycleListener.

import org.apache.catalina.Lifecycle;
import org.apache.catalina.LifecycleEvent;
import org.apache.catalina.LifecycleListener;
import org.apache.log4j.LogManager;
import org.apache.log4j.PropertyConfigurator;

public class Log4JInitializer implements LifecycleListener
{
    private String propertiesFile;

    public String getPropertiesFile()
    {
        return this.propertiesFile;
    }

    public void setPropertiesFile(String propertiesFile)
    {
        this.propertiesFile = propertiesFile;
    }

    @Override
    public void lifecycleEvent(LifecycleEvent event)
    {
        if (Lifecycle.BEFORE_START_EVENT.equals(event.getType()))
            initializeLog4j();
    }

    private void initializeLog4j()
    {
        // configure from file, and let log4j monitor the file for changes
        PropertyConfigurator.configureAndWatch(propertiesFile);

        // shutdown log4j (and its monitor thread) on shutdown
        Runtime.getRuntime().addShutdownHook(new Thread()
        {
            @Override
            public void run()
            {
                LogManager.shutdown();
            }
        });
    }
}

I simply listen for the "BEFORE_START_EVENT", and if that happens (which is once per Tomcat startup) I initialize Log4j using the configureAndWatch method. I also don't forget to install a shutdown hook to cleanup the thread Log4j creates to poll the configuration file for changes (I could also have chosen to listen to the "AFTER_STOP_EVENT" from Tomcat in stead).

Package this in a jar, put it on the Tomcat classpath, and now you can configure it in your Tomcat serverl.xml.

<Server>
  ...
  <Listener className="Log4JInitializer" propertiesFile="/path/to/log4j.properties"/>
</Server>

Can't be much easier, and it does what it has to do.

Performance tuning tools for a multi core world

2010-01-20T19:55:00.013+01:00

Recently I was performance tuning a Java application on a quad core Intel Xeon X5560. This processor has hyper threading, so it shows up in the operating system as if it were 8 cores.

My application was slower than I wanted it to be, and I was faced with this vmstat output:

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  0 1048660 209840 199608 4123568    0    0   150    24 1577 2599 13  9 77  1  0
 2  0 1048660 210220 199612 4123712    0    0   104    82 1593 2891 13 10 75  2  0
 1  1 1048660 209260 199628 4124596    0    0   484   210 1537 2567 10  7 77  6  0
 1  1 1048660 213940 199612 4120184    0    0  1232    32 1548 2694  9  7 76  8  0

At first sight, I thought it looked ok. That's between 9 and 13 % CPU time in user space, between 7 and 10 % in system space, and 75 to 77 % is idle. So it looked like my application is not using much of the CPU at all...

My application was still slower than I wanted it to be, so I went on looking for reasons why it was not using more of the CPU. I took some thread dumps and used the YourKit profiler to measure the time threads were waiting for locks. I couldn't find anything.

Then I had a look at what "top" had to say, in stead of using vmstat. It showed comparable numbers:

Cpu(s): 14.6%us, 10.5%sy,  0.0%ni, 73.2%id,  1.2%wa,  0.0%hi,  0.5%si,  0.0%st

But it says "Cpu(s)", so I hit "1" to get a view per CPU... now things started to get interesting:

Cpu0  : 12.7%us,  8.7%sy,  0.0%ni, 73.3%id,  3.3%wa,  0.0%hi,  2.0%si,  0.0%st
Cpu1  : 13.2%us,  8.3%sy,  0.0%ni, 77.5%id,  0.7%wa,  0.0%hi,  0.3%si,  0.0%st
Cpu2  : 14.3%us,  8.3%sy,  0.0%ni, 77.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  : 18.0%us,  9.7%sy,  0.0%ni, 70.7%id,  0.7%wa,  0.0%hi,  1.0%si,  0.0%st
Cpu4  : 18.0%us, 17.3%sy,  0.0%ni, 64.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu5  : 15.3%us, 12.0%sy,  0.0%ni, 72.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu6  : 11.6%us,  8.9%sy,  0.0%ni, 79.5%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  : 13.3%us,  9.0%sy,  0.0%ni, 77.3%id,  0.3%wa,  0.0%hi,  0.0%si,  0.0%st

Now of course, the problem was staring me in the face. The CPU times of vmstat and top (and many other tools) are relative to the total amount of CPUs. With 8 cores, this means that (for a single thread) 12.5% represents a CPU usage of 100% on only one CPU. With this in mind, the numbers of vmstat start to make sense. They mean that more or less 1 CPU is fully busy in user space, and one other CPU is almost fully busy in system space. In other words, my application would probably be faster if it did less I/O and more concurrent processing.

It also means that, with more and more CPUs, tools like vmstat are becoming less usefull. With only one CPU, you get a scale from 0 to 100. With 8 cores, the scale is reduced from 0 to 12. The granularity of the output of vmstat is 1 percentage, so with 100 cores the difference between some random "noise" on an otherwise idle machine and an idle machine except for 1 core which is fully used, will become invisible.

Its not only vmstat of course, the output of the CPU profiling in YourKit for example, has the same problem. The graph stays around 12.5%:

So to conclude:

I think we might need to update our profiling tools. We can also add new tools to our profiling toolbox. I experimented a little with mpstat for example, which gives much of the same information as vmstat, but does it per CPU.

And of course, as we all already know, we also also have to update our applications. Concurrency is not going away any time soon.

A Scala library for IPv4 related concepts

2010-01-03T09:55:00.012+01:00

I've been using the Xmas holidays to write lots of Scala. One of the things I wrote, partially just to learn some more Scala, partially because I believe it can be useful, is a library to work with IPv4 related concepts: IP addresses, IP networks, ranges of IP addresses etc.

In this post, I'll give a short overview of the library, and share some thoughts about my Scala experience while writing it.

The library exposes a few core concepts which can be useful when doing calculations with IP addresses etc.

IpAddress

IpAddress represents an IPv4 address.

val address = new IpAddress("192.168.0.1")

IpAddress can be used to make calculations on IP addresses, such as finding the next address.

val address = new IpAddress("192.168.0.1")
val next = address + 1
println(next == new IpAddress("192.168.0.2")) // prints true

IpAddressRange

IpAddressRange represents a continuous range of consecutive addresses.

val range = new IpAddressRange(new IpAddress("192.168.0.1"), new IpAddress("192.168.0.5"))
println(range.contains(new IpAddress("192.168.0.3")) // prints true

IpNetworkMask

IpNetworkMask represents a network mask, to be used in an IpNetwork.

val mask1 = new IpNetworkMask("255.255.255.128")
val mask2 = IpNetworkMask.fromPrefixLength(25)
println(mask1 == mask2) // prints true

val invalid = new IpNetworkMask("255.255.255.100") // throws exception

IpNetwork

An IpNetwork is a range (extends IpAddressRange) that can be expressed as a network address and a network mask.

val network1 = new IpNetwork(new IpAddress("192.168.0.0"), new IpNetworkMask("255.255.255.0"))
val network2 = new IpNetwork("192.168.0.0/24")
println(network1 == network2) // prints true

IpAddressPool

An IpAddressPool is like a range (extends IpAddressRange) of which certain addresses are "allocated" and others are "free". It could for example be used as part of a DHCP server implementation.

var pool = new IpAddressPool(new IpAddress("1.2.3.4"), new IpAddress("1.2.3.10"))
println(pool.isFree(new IpAddress("1.2.3.6"))) // prints true
pool.allocate(new IpAddress("1.2.3.6")) match {
    case (newPool, allocated) => {
            println(newPool.isFree(new IpAddress("1.2.3.6"))) // prints false
    }
}

And Much More

This was only a short introduction. Much more can be done with these types. Have a look at the scaladoc to get an idea of the possibilities.

What I liked about Scala

I became a fan of Scala some time ago already, but there were a few things which struck me as especially nice or elegant while writing this library:

Lazy lists

The IpAddressRange class has a method to list all addresses in the range. Also, the IpAddressPool class has methods to list all the free, and all the allocated addresses. With big ranges, it becomes virtually impossible to return an array or so containing all the addresses. Scala has the Stream concept to deal with this. A Stream is effectively a "lazy list". In other words, a List of which the elements are only evaluated when they are needed. With a recursive implementation, this results in very elegant code:

def addresses(): Stream[IpAddress] = {
    if (first < last) {
        Stream.cons(first, new IpAddressRange(first + 1, last).addresses)
    } else {
        Stream.cons(first, Stream.empty)
    }
}

Functional or Procedural

Although I try to write most of my scala code as "functional" as possible, sometimes it is nice to be able to fall back on a procedural implementation. To do calculations on IP addresses and network masks, a lot of "bit twiddling" is required. I found it usually easier to write that in a procedural style than purely functional (although that could be due to my limited functional experience as well), e.g.:

def fromLongToPrefixLength(value: Long): Int = {
    val lsb: Long = value & 0xFFFFFFFFL
    var result: Int = 0
    var bit: Long = 1L << 31

    while (((lsb & bit) != 0) && (result & 32)) {
      bit = bit >> 1
      result += 1
    }
    result
}

Option

The Option type needs no introduction, but it's so much more fun than using null values all over the place.

Immutability

Scala favours immutable types. This naturally steered me towards making all types in my library immutable as well. Because you can rely on immutable data structures in the Scala library (List, SortedSet, ...), it is much easier than in Java to get it right.

What I might dislike about Scala

h I encountered a few things which seemed a bit awkward or strange. Most of the issues are due to my lack of experience, so I'll give myself some more time to figure out whether they are really things I don't like or not.

Tuples

Tuples for example, are a concept that I am in doubt about. I used them extensively in the IpAddressPool class. For example, when allocating an address in a pool, the method returns an Option of the allocated address (None if the address was not free) and the new pool (because the IpAddressPool is immutable):

def allocate(toAllocate: IpAddress): (IpAddressPool, Option[IpAddress]) = {...}

This is really easy on the implementation side, but I'm not sure whether I like the client code so much. You either have to access the values in the tuple with "._1" and "._2", which doesn't read easy, or you have to pattern match, which seems a bit verbose. You can also assign the result to a new tuple directly

val pool = new IpAddressPool(new IpAddress("1.2.3.1"), new IpAddress("1.2.3.10"))
val (newPool, allocatedAddress) = pool.allocate(new IpAddress("1.2.3.6"))

but it doesn't seem to be possible to assign the result of a subsequent invocation again to the same tuple (using a var)

val pool = new IpAddressPool(new IpAddress("1.2.3.1"), new IpAddress("1.2.3.10"))
var (newPool, allocatedAddress) = pool.allocate(new IpAddress("1.2.3.6"))

// this won't compile
(newPool, allocatedAddress) = pool.allocate(new IpAddress("1.2.3.7")

Collections

I've had some usability issues with the collections library. The collections library is being worked on for the upcoming scala 2.8 version, so things will definitely improve.
I also found a bug in the TreeSet implementation of SortedSet, causing the tree to be heavily unbalanced which results in StackOverflowErrors when recursively traversing it (I hope this can get fixed for 2.8 as well).

Conclusion

Besides these minor issues, it was a very positive experience. I happen to have a Java version of this library as well, and I certainly like the scala version much better (both in using and in writing it).

The Dreadful JMX Tunneling Problem

2009-12-09T13:22:00.003+01:00

JMX is notoriously difficult to access remotely. It relies on RMI which means that you need a connection to an RMI registry and a separate connection to an RMI server. The port (and host name) to be used for the second connection are assigned by the RMI registry, and can thus not be "guessed" in advanced. This makes it almost impossible to setup the necessary port forwarding or ssh tunnels to access JMX on a server which is not directly in your network.

We used to use JBoss, which provides a html wrapper around the MBean server ("jmx-console"), thus allowing easy access to our MBeans. It's not as nice and graphical as JConsole, but at least http is easy enough to tunnel through ssh or firewalls.

After switching to tomcat (for various other reasons), I was looking for a solution to this "access our MBeans through ssh tunnels"-problem. There are all kinds of workarounds available to make RMI "tunnelable', but everything I tried came with its own set of problems (for example having to append stuff on the class path of JConsole).

So if tunneling turned out not to be a viable option, I started looking for html based alternatives like the "jmx-console" from JBoss. I found a couple of them:
• JManage looks very nice and feature complete but it has the downside of being a standalone application running an embedded jetty rather than something I can simply deploy in tomcat.
• On some mailing list I found somebody who once created an application called "jmx-console" which does what I want. I tried it and it seemed to work OK, but it's an old and unmaintained project.
• Coldbeans has an application written entirely in JSP. When testing it, it found it somewhat cumbersome to work with though, and I couldn't seem to access all my MBean operations (as if some of them where "read only").

During my search, I learned that the original reference implementation for JMX (or now in http://opendmk.dev.java.net/) contains a class named HtmlAdaptorServer, which didn't make it into JDK 5 when JMX was first introduced. The reference implementation is still usable in itself though (in maven2 repository as com.sun.jdmk:jmxtools:1.2.1). This HtmlAdaptorServer does exactly what I need: expose MBeans in a straightforward html view. I simply instantiate it in a ServletContextListener like this:

final HtmlAdaptorServer adapter = new HtmlAdaptorServer(port);
ManagementFactory.getPlatformMBeanServer().registerMBean(adapter, new ObjectName("my.domain:name=htmladapter,port=8000"));
adapter.start();

Although this is a good enough solution for me right now, I still think some configuration options to make JMX/RMI easily 'tunnelable' (being able to fix the port and host name would probably be enough) are something we should consider adding to the JDK...

A performance tuning story

2009-10-29T19:29:00.010+01:00

Recently at work, we were performance testing a few use cases of an application which are completely bottlenecked by the database. Actually, we were trying to figure out how much data we could input in the system, while sustaining the required response times. The "input into system" use cases obviously involves a few "insert" operations on the database, but also a few "select" queries on what is already in the database for verification of the inserted data etc. Initially we simply wanted to see how the performance of this "input into system" use case degrades when the database gets filled more and more.

A colleague of mine performed the tests (thanks Bart, if you are reading this), and created some nice graphs of the results. One of these graphs looks like this:

On the vertical axis you can see the response times. On the horizontal axis the number of "input into system" operations that have already been performed.

You notice that, as expected, the performance degrades as the database is filled with more data. But something strange is visible: the response times bounce up and down between some kind of minimum and maximum. Response times for individual requests seem to be either "slow", either "fast". We were convinced that if we could find the cause of the response times sometimes being slow, we could fix it, and all response times would be on the lowest line in the graph.

This is where the discussion started to diverge. We started thinking about all kinds of possible explanations as to why this could be the case. Many graphs where produced and many tests conducted, to try and figure out what the underlying cause could be.

Although this kind of reasoning is highly enjoyable, and watching all these graphs is very intriguing indeed, I proposed a more pragmatic approach: lets just find the slowest query in the system and try to improve that one... without bothering about trying to explain the strange graphs. Or as Kirk Pepperdine would say: "big gains first".

A quick check with vmstat confirmed our initial idea that the database was indeed the bottleneck. The MySQL process was using all of the CPU (user time). This also meant that I/O was not the problem. MySQL probably had all the data cached in memory, but some query (or queries) demanded lots of CPU time to be executed.

Next we added logging of the database queries being executed using P6Spy. We let the test run for a few minutes and then opened the log in IroneEye. This gives a nice overview of all the queries that have been executed, with their execution times. One query came out as being about 5 times slower as the other ones. We had a look at the actual query, and it turned out not only to be slow but also incorrect. The corrected version of the query had to do less joins, and was much faster. Running the test again resulted in a constant linear degradation of the response times, as expected.

Actually, when running the test for a longer time, the bouncing up and down pattern came back. This time it was much less severe though, and it happened only when much more data was already in the system. We'll probably have to hunt down the next slowest query now, in order to try and improve the situation even more.

Taken all together, it took us only 30 minutes or so to perform the test with P6Spy, analyze the logs with IroneEye, find the slowest query, fix the query and run the test again. Compared to endless discussions about possible explanations for the strange behaviour, the pragmatic approach clearly won in this case. Agreed, the pragmatic approach didn't help us to explain why the response times are sometimes slow, sometimes fast... but who cares, it's much faster now anyway, and we fixed an incorrect query in the process.

On a side note... P6Spy and IroneEye seem both rather dead. IroneEye can not even be downloaded anymore. Does anybody know a more alive (and free) alternative? I could probably look into the tools of the database vendors (MySQL in our case) themselves, but they are typically not free and maybe also not so easy to setup?

Mysterious JVM crashes explained

2009-08-11T21:30:00.005+02:00

We've recently suffered from mysterious crashes of the JVM running our application server. The JVM (Sun JDK 1.5.0_06 but it was later reproduced with JDK 1.6.0_14 as well) simply crashed with all sorts of memory related errors. A few examples:

java.lang.OutOfMemoryError: requested 8388608 bytes for uint64_t in /BUILD_AREA/jdk1.5.0_06/hotspot/src/share/vm/utilities/growableArray.cpp. Out of swap space?
java.lang.OutOfMemoryError: requested 157286476 bytes for JvmtiTagHashmapEntry* in /BUILD_AREA/jdk1.5.0_06/hotspot/src/share/vm/prims/jvmtiTagMap.cpp. Out of swap space?
java.lang.OutOfMemoryError: requested 16 bytes for CHeapObj-new. Out of swap space?

The server was not out of swap space and it had plenty of free memory left, so that was not the problem.

I started googling around and found lots of different mentions of these kinds of problems, but there turned out to be many possible causes.

In our case, the problem turned out to be that we allowed the JVM to use too much memory... The JVM was configured with a 2GB heap (and 256MB of perm size), and was running on a 32bit machine with 4GB of memory (linux 2.6.x kernel without hugemem or anything like that). As it turns out, in this kind of configuration, linux allows each process to allocate at most a little over 3GB. When our application was requiring a lot of memory, the JVM was trying to (perhaps temporary, to perform garbage collection) allocate more memory than it is allowed to by the linux kernel, resulting in one of the above types of crashes.

Configuring the JVM with a smaller heap (1.5GB) caused our application to simple go out of memory with a regular "java.lang.OutOfMemoryError: Java heap space". But now at least, it didn't just crash, and after a few hours with a memory profiler (YourKit), the cause of the excessive memory usage in our application was found as well.

I still wonder if the JVM or the linux kernel could arrage for a more meaningful error message though.

Consistent SVN branch and tag names for maven projects

2009-06-27T22:10:00.001+02:00

When using maven together with subversion (SVN), it is good to have a naming strategy for your branches and tags which is consistent with the maven versions of your projects (in the pom.xml files). A consistent naming strategy makes it trivial to know which version of a maven project can be found where in the SVN repository (and vice versa), and it prevents possible version conflicts between maven projects when for example branching twice from the same starting point.

We designed a simple scheme at work which suits us well. I thought I might as well share it, you might find it useful.

In principle, we have a X.Y.Z numbering scheme for all our maven projects (but having only one or two numbers would be possible as well); for example someproject-3.4.4.
All main development happens on trunk (for the "current" release). A maven project on trunk thus has a version number X.Y.Z-SNAPSHOT; for example someproject-3.4.5-SNAPSHOT.
When releasing a maven project, the word "SNAPSHOT" is removed from the version; for example someproject-3.4.5. On trunk, development continues with the next SNAPSHOT version; for example someproject-3.4.6-SNAPSHOT (or perhaps 3.5.0-SNAPSHOT or even 4.0.0-SNAPSHOT).
A released project is tagged in SVN as ${artifactId}-${version}; for example http://svnserver/svnrepo/tags/someproject-3.4.5.
A branch (for fixes in "old" releases) is always taken starting from a tag. Branching a project tagged with version X.Y.Z results in a branch SVN directory called X.Y.Z.1.x; for example http://svnserver/svnrepo/branches/someproject-3.4.5.1.x (the 1 means "first branch", the x indicates that that is the only part of the version which changes in this branch during the lifetime of the branched project).
If another branch is taken from the same tag, it becomes X.Y.Z.2.x etc (This actually corresponds to how CVS does it: branching introduces two extra numbers); for example http://svnserver/svnrepo/branches/someproject-3.4.5.2.x.
The version of the maven project in a branch follows this naming strategy; for example the maven version of the project in .../branches/someproject-3.4.5.1.x is initially 3.4.5.1.0-SNAPSHOT.
Releasing from a branch results in a tag with 5 numbers; for example the first release would be someproject-3.4.5.1.0 and the branch would continue with 3.4.5.1.1-SNAPSHOT.
Branching on a branch is also possible, just add another two numbers; for example X.Y.Z.1.2.1.x (first branch of the second release from that first branch of project with version X.Y.Z). In our case, this has been necessary only once though.

I guess we aren't the first team to invent this, but I have never seen it written down for the SVN/maven combination like this.

How to provide hibernate persistence support along with class libraries

2009-06-23T20:54:00.002+02:00

Suppose you want to encapsulate some kind of functionality in a class library, and you want to provide an easy way for the clients of your library to persist certain entities from your library with hibernate. I think the most straight forward way to do this, is to provide hibernate UserType implementations along with the entities in the class library that the client might want to persist. Joda Time for example, uses this approach. Joda Time is a class library to work with dates and times, and the Joda Time Hibernate Support project provides hibernate UserType implementations for many of the entities from Joda Time.

This approach works perfectly well, but it is limited to what one can express in a hibernate UserType. One of those limitations is that you can not (or almost not) use a single UserType to persist an entity that has a one2many relationship with something else (probably a value type). I encountered this problem when writing a class library to manage IP address pools (an IP address pool is a continuous range of IP addresses of which some are "free" and some are "allocated"). I represent an IP address pool as a class (IpAddressPool) which contains a sorted set of free sub-ranges in the pool. Persisting this would naturally result in a one2many mapping (using hibernate to map the SortedSet). Writing a single UserType which is capable of persisting IpAddressPool instances (including the one2many mapping) is (almost) not possible. I say almost, because it might be possible to access the JDBC connection directly and do everything by hand, but obviously I would like hibernate to manage the one2many relation.

So, when UserType implementations are not an option, what other options do we have?

we can simply do nothing and leave it up to the client to create the necessary hibernate mappings;
we can provide a mapping and a perhaps even a DAO and let the client use that by "injecting" its session factory into it;
we can do the same with annotations;
we can prevent the one2many mapping by serializing the sorted set into a single string representation or something like that, and use a simple UserType after all;
... ?

The first solution is not really a solution at all. Apart from not helping the client of your library with the persistence aspect, it also means that we need to expose the internals of our classes such that clients can write correct hibernate mapping files for them.

The second solution is a bit better, but still not as good as the UserType approach. It typically results in having two mapping files (one from the class library, one from the client application) and thus complicates things like schema generation. It also makes it more cumbersome to integrate it with your client application. The provided DAO for example, might not be aligned with the DAO style of the client application, or the DAO might not provide certain queries the user wants to perform. Also, the client application might use annotations to express the hibernate mappings, while the class library uses an XML mapping file. Querying on properties of the entity in the class library is possible, but it means the internal representation has to be exposed. The UserType (and more specifically CompositeUserType) API has a nicer way to expose properties on which HQL queries can be performed without having to expose the internals of the entities in your class library, which can unfortunately not be used in this solution. It also becomes a bit problematic if you have multiple entities in your class library of which a client might only use a few.

The third solution is basically the same concept as the previous, except we could use annotations in stead of mapping files in the class library. It has all the typical advantages of annotations, but also means that the class library now needs a dependency on hibernate, even if the client doesn't care about persistence.

The fourth solution is more a workaround than a real solution. I think it might be appropriate in some situations though. In the IP address pool example this actually works very well, although it could become a problem for very large pools with lots of fragmentation (many entries in the sorted set of free sub-ranges) and it also complicates manual SQL database queries.

If anybody has better suggestions, or if anybody can prove me wrong on the claim that this is not feasible with a UserType, I'd be happy to hear about it.

Modernizing MultithreadedTC: JUnit 4 integration

2009-06-08T19:40:00.019+02:00

MultithreadedTC is a nice library to help writing tests for multi threaded applications (go check out their website for some examples, I can 't explain it better). Writing and testing multi threaded applications is not an easy task, so having a decent test suite for to help with that task is no luxury at all. I think MultithreadedTC does a nice job, it certainly helped me to write better and more elegant tests than I would have been able to without the library.

One thing I find it a bit frustrating though, is that it doesn't integrate with JUnit (4) very nicely. Basically, with MultithreadedTC, you have to write a class which extends MultithreadedTestCase, with some methods that start with "thread" (a naming convention for MultithreadedTC to know what to run in which threads)

class MyMultithreadedTest extends MultithreadedTestCase {
  public void threadFoo() {
    ...
  }
}

To execute this "multi threaded test case", you have to instantiate it and pass it on to one of the static methods in the TestFramework

TestFramework.runOnce(new MyMultithreadedTest());

Although the name MultithreadedTestCase suggests it is a JUnit test, it is not. To get it all bootstrapped in a JUnit test, you typically end up with an inner class (extending MultithreadedTestCase) inside a JUnit test, and a single JUnit test method to call the TestFramework.runOnce() method

public class SomeTest {
  class MyMultithreadedTest extends MultithreadedTestCase {
    public void threadFoo() {
      ...
    }

    ...
  }
  
  @Test
  public void test() {
    TestFramework.runOnce(new MyMultithreadedTest());
  }
}

This obviously works, but I think we can do better.

I updated the MultithreadedTC library such that multi threaded tests are real JUnit tests (including all the benefits that has, such as using the @After and @Before annotations), in stead of merely something which can be bootstrapped from within JUnit. I also added some annotations for the "thread" methods, in stead of the naming conventions.

The above now looks like this

public class SomeTest {

  @Thread("Foo")
  public void threadFoo() {
    ...
  }
  
  @Test
  public void test() {
    ...
  }
}

Consider the first example on the authors website for a more complete example. This can now be written like this:

public class MTCCompareAndSetTest extends MultithreadedTestCase
{

    AtomicInteger ai;

    @Before
    public void initialize()
    {
        ai = new AtomicInteger(1);
    }

    @Threaded
    public void getTwoSetThree()
    {
        while (!ai.compareAndSet(2, 3)) Thread.yield();
    }

    @Threaded
    public void getOneSetTwo()
    {
        assertTrue(ai.compareAndSet(1, 2));
    }

    @Test
    public void resultShouldBeThree()
    {
        assertEquals(ai.get(), 3);
    }
}

Advantages of this approach include: no inner classes, no explicit TestFramework.runOnce() method calls, usage of standard JUnit annotations, test class is a JUnit class, annotations in stead of naming conventions, ...

Let me know if you are interested in a tgz file which contains the code of this modified MultithreadedTC library. It also contains all the tests of the original sourcecode, adapted to the more modern version.

I have tried to contact the authors of MultithreadedTC, asking them if they are interested. I am willing to share my modifications with them to get them included in the MultithreadedTC library, but I'm still awaiting their reply. Maybe they are deadlocked ;-)

High frequency task scheduling with inaccurate system clock in Java

2009-04-10T21:33:00.004+02:00

Java has two general purpose scheduling implementations: java.util.Timer (further referenced as "Timer") and java.util.concurrent.ScheduledThreadPoolExecutor (further referenced as "STPE"). STPE is generally concieved as the successor of Timer and their should normally not be a reason to use Timer any longer. However, both Timer and STPE rely on the underlying system clock, and can thus not be more accurate then the accuracy of that clock.

I happen to have access to a machine where the accuracy of the clock is 20 milliseconds. This means that when you try to make a thread sleep for a single millisecond

public class SleepTest
{
    public static void main(String[] args) throws InterruptedException
    {
        while (true)
        {
            long start = System.nanoTime();
            Thread.sleep(1);
            System.out.println((System.nanoTime() - start) / 1000000);
        }
    }
}

the thread will actually sleep for about 20 milliseconds every time

This also has an impact on the scheduling behaviour with Timer and STPE. Note that the Timer javadoc explicitely mentions this, while the STPE javadoc doesn't.

For example timing a task to execute every 10ms "with fixed delays" using Timer or STPE results in the task being executed every 20ms in stead:

running task [elapsed nanos since previous = 20157000]
running task [elapsed nanos since previous = 19999000]
running task [elapsed nanos since previous = 19895000]
running task [elapsed nanos since previous = 19940000]
running task [elapsed nanos since previous = 19987000]
running task [elapsed nanos since previous = 19991000]
running task [elapsed nanos since previous = 20013000]
running task [elapsed nanos since previous = 20011000]
running task [elapsed nanos since previous = 19980000]
...

When using Timer or STPE to schedule "at fixed rate", two tasks are being executed in immidiate succession every 20ms. This is also consistent with the documentation: it tries to "catch up".

running task [elapsed nanos since previous = 19808000]
running task [elapsed nanos since previous = 187000]
running task [elapsed nanos since previous = 19813000]
running task [elapsed nanos since previous = 185000]
running task [elapsed nanos since previous = 19842000]
running task [elapsed nanos since previous = 185000]
running task [elapsed nanos since previous = 19803000]
running task [elapsed nanos since previous = 214000]
...

Note that the "fixed delay" of Timer doesn't take the execution time of the task itself into account, while the STPE "fixed delay" does, but that is not something I was trying to measure here. The task I am executing is just an empty Runnable.run() method.

Now imagine you are in a situation where you actually need to schedule something at a higher frequency then what the underlying system clock can handle (sending out a UDP stream at a fixed rate for example). Scheduling with "fixed delay" clearly is no option; both Timer and STPE will only give you a frequency equal to the accuracy of the underlying clock, nothing higher. The only option is to schedule "at fixed rate". Both Timer and STPE will "catch up" in this mode, so the average throughput corresponds to the requested frequency to execute the tasks with. The tasks will be executed in bursts though: one burst with a few tasks every time the underlying clock reports a new time value (every 20ms in my example). The problem with "fixed rate" scheduling is that you don't really have a way to control these bursts. If for example your application is blocked for a longer time for some reasons (garbage collection for example), there will be a gap in the task execution, and then a "big" burst to catch up all the tasks it missed.

This is where Timer offers an advantage over STPE. With STPE you schedule runnables, so inside the task there is not really an API to find out information about the scheduling of the task. With Timer, you schedule TimerTask instances. TimerTask has a method scheduledExecutionTime which returns the time the task "should have run". By comparing this to the actual time, you can discover "big bursts" and for example decide not to execute the task. I haven't found a practical way to do that with STPE.

I discovered this when trying to send out a UDP data stream at a fixed rate. The requirement was that a few missed packets were not so bad (a little drop in the speed), but big bursts with many packets (more then about 10 in a row) caused a problematically high peak in the data rate (because then the stream was "clipped" further down the path resulting in dropped packets) and should thus never happen.

Note that in my tests I used System.nanoTime() to measure the time between task executions. This gives me a higher accuracy then the System.currentTimeMillis() which makes me wonder why there is no scheduler with the same accuracy in Java. Something I should look into...

Runtime dispatch between different versions of the same library in Java

2008-12-23T17:51:00.013+01:00

Java can (at least not yet?) dispatch between different versions of the same library at runtime. In fact, Java doesn't even know about the concept of library versions.

I was recently faced with a problem where it was required to dispatch between different versions of the same library at runtime. We have this architecture where network management software (lets call it the "master") has to communicate with different other software components somewhere on (remote) hardware devices in the network. This communication goes over various protocols, some Java specific (for example RMI using springs http invoker), some not (for example telnet). We also have the requirement to support different versions of these remote devices from within this single "master".

This UML diagram (click to enlarge) shows an example possible situation where the "master" needs to communicate with two different versions of two devices. There are two instances of device A in the network, one with version X and one with version Y. There are also two instances of device B, one with version Z and one with version Q. A driver implementation exists for every version of the device, but ultimately each device type (A or B) has a single interface to be used by the "master". It is the drivers job to abstract away the differences between different versions of the same device type. If the protocol is telnet or something like that, we wouldn't really need different driver implementations. A naive implementation of the "master" could simply switch around the different possible versions with "if" statements (if this version send telnet command X, if that version send telnet command Y, etc). If the protocol is Java, this becomes a problem, because you then typically depend on a library which specifies the interfaces and types used for the communication. This is technically not required of course (you could manually write RMI to a raw byte output stream), but it is what you generally want. Event for the non Java protocols, you might want to abstract away the different commands required for different versions behind a single interface in different implementation libraries.

We discussed (and experimented with) different alternative solutions to this problem. I'll go through each of them, and explain the pros and cons.

idea 1: custom class loading magic

The first idea was to package the driver implementation libraries in jar files (together with their dependencies), and class load the appropriate version of such a driver at runtime. This also requires the usage of reflection to actually instantiate such a driver. The usage of reflection prevents compile time dependencies on the driver implementation libraries (this is impossible because we would have to depend on different implementations of the same driver interface at the same time). The compile time dependencies are from the "master" to the driver interfaces, and from the driver implementation libraries to the remote device interfaces.

To make this idea work, the class loaders of the "master" and the driver implementation libraries have to be separated. On the other hand, the driver interfaces (used to communicate between the two) have to be known by both. In Java, this is only possible if they both use the driver interface classes from the same class loader. This means that the driver class loader has to have a dependency on the "master" class loader, such that driver interfaces are loaded from there. Having the "master" class loader as parent from the driver library class loader solves this problem.

As a consequence, the driver will find all its libraries in the parent class loader first, unless they are not available there. For libraries on which the driver and the "master" both depend (in our situation, spring was such a library), this is a problem, because from within these classes, the classes loaded by the class loader from the driver itself are not visible. We found a solution to this problem by writing a custom class loader which inverses the regular pattern for finding classes (first in own class loader, then in parent, in stead of first in the parent). This solves the problem, but we have to make sure that the driver interfaces are loaded from the parent class loader (by not packaging them together with the driver implementation libraries).

In the end, this approach does work, but the implementation is scattered around the different components. The "class loading magic" happens in the "master", but it only works if the driver implementation libraries are correctly packaged. This correct packaging depends on whether certain libraries are used by the master and the driver together, or only by the driver interface, etc.

idea 2: deploy the drivers as WAR files

The second idea was to package the driver implementation libraries as WAR files, and to give them a remote interface (RMI or something like that). This would imply that the container would take care of all the separation of class loaders, which is nice.

To understand the downside of this approach, you should know that the "master" doesn't simply uses these drivers once when it has to communicate with a remote device, and then forgets about them. The "master" does lots of interesting things with the drivers: it caches them (for example to reuse the same TCP connection to the device the next time), it manages concurrent access to the drivers (allowed on some, disallowed on others), etc... This means the master needs to have control over the lifetime of the drivers.

If we deploy the drivers as WAR files, we loose this functionality. We would effectively have to make the drivers stateless, or we would have to move a lot of the functionality from the "master" into the individual drivers.

idea 3: OSGi to the rescue

Our next idea was to use OSGi. If we make every driver implementation an OSGi bundle, we could decide at runtime which service of which bundle to invoke, and OSGi would do all the class loading separation for us. I did some experiments which proved that this would be a feasible solution (although I didn't look into the possibilities for lifetime management of the drivers in detail), but apparently OSGi is a bit of an all or nothing approach. There are of course many more components in our architecture than what is shown in the above UML diagram, and at the time it was not feasible to introduce OSGi in all of them. It seems that we would have to deal with the same class loading problems as in the first idea on the boundaries between non-OSGi and OSGi components.

A colleague of mine found a library (transloader) which was created to solve this problem. The transloader library is not tight to OSGi though, it is a general solution to bridge the gap between different class loaders within Java. This led us to our fourth idea.

idea 4: class loading magic revisited

If this transloader library can bridge the gap between different class loaders, why not use it to implement our first idea? We now have the same architecture as in the first idea, but without custom class loaders and without weird packaging constraints on the driver implementation libraries. All we have to do is package the drivers together with all their dependencies (no exceptions) and use the transloader library for the communication between the "master" and the drivers.

Check out the transloader tutorial to learn more, it was really easy to set up and use.

Conclusion

Built in support for modules and library versions in Java would probably still be the most desirable solution. I think until then, something like OSGi can be a nice solution too, if you can introduce it in your architecture. If that is not an option, the above described approach offers a valid alternative (at least in our use case) with a fairly simple architecture. We will be using this idea in production in the coming months, lets see if it lives up to our expectations.

analyzing maven dependencies with UML tools

2008-04-21T15:09:00.015+02:00

Maven is very good in managing dependencies in large projects with many different artifacts, but unfortunately maven doesn't provide an easy way to visualize and analyze these dependencies. The dependency report that comes by default with a maven generated web site provides some information, but it is textual only and it only provides information about outgoing dependencies: it tells you what artifacts project X depends on, but not what other projects depend on project X.

Some maven dependency visualization tools exist. The best I could find is the dependency analyzer from jfrog. It's a standalone java application that allows you to open a pom.xml file and graphically explore its dependencies in a swing GUI. It does however not provide advanced analyzing tools, its merely a visualization tool.

Existing UML modeling tools (e.g. magicdraw) typically allow you to do much more advanced kinds of analyzes on dependency and other UML diagrams. They also allow you to represent dependencies in different ways (as a diagram, as a matrix, ...). Why not use the power of these tools to analyze maven dependencies?

maven xmi plugin

In this blog entry, I present a maven plug in that generates an XMI model from a pom.xml file and it's dependencies. XMI is an XML standard most commonly used to represent UML models. Most professional modeling tools can read XMI files. The XMI itself is not directly a visualization of the dependencies, but with a decent modeling tool, all kinds of fancy visualizations are possible.

To give you an idea, this screen shot (click to enlarge) shows a diagram created with magicdraw for the dependencies of the xmi maven plugin itself.

using the maven xmi plugin

First, you'll have to check out the code of the xmi-plugin from subversion:

svn checkout http://xmi-plugin.googlecode.com/svn/trunk/ xmi-plugin

Then you can install the xmi-plugin in your local maven repository, by invoking "mvn install" in the checkout directory. Now you are ready to use the plugin on your favorite maven project. Simply run

mvn xmi:xmi

in the root directory of your project (where the pom.xml file is located). The resulting XMI model will be generated in the target directory of your project, with the name ${artifactid}-${version}.xmi. You can now start analyzing the XMI model in an UML tool.

more information

The xmi-plugin is hosted on google code, for more information go to http://code.google.com/p/xmi-plugin/.

SOAP faults with spring ws and JAXB 2.0

2008-03-27T18:25:00.016+01:00

When migrating existing web services from Axis2 to spring web services (1.0.3) and JAXB (2.0), I ran into three problems related to SOAP faults. I found some hints towards a solution on the net, but none of them were really exhaustive. In this blog post, I give a step by step overview of how we solved these three issues.

Let's start by sketching the context in which we encountered the problems. We already had a contract first approach (actually, our WSDL was generated using an AndroMDA cartridge), so we could start from an existing WSDL. We only had to move the types into a separate XSD file (for JAXB, see further). Our WSDL defines a lot of operations with input, output and fault messages. With axis2, these faults were represented as java exceptions (inheriting from java.lang.Exception). When serializing these java classes back to XML, they were represented as SOAP faults with as detail element of the soap fault a serialized version of the java exception (as defined in the XML schema). This is an example of a SOAP fault message sent to the client:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
  <soapenv:Body>
     <soapenv:Fault>
        <faultcode>soapenv:Client</faultcode>
        <faultstring>FooException: null</faultstring>
        <detail>
           <ns:FooException xsi:type="ns:FooException" xmlns:ns="http://our.namespace" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
              <ns:customField>0</ns:customField>
           </ns:FooException>
        </detail>
     </soapenv:Fault>
  </soapenv:Body>
</soapenv:Envelope>

The first problem when migrating to spring web services and JAXB, is that JAXB generates java types from the XSD rather then from the WSDL. The types of the fault messages are off course known as an XSD element, but the XSD has no information to distinguish them from other types (input and output messages). Typically all input messages end with "Request", all output messages with "Response" and all exceptions with "Exception", but JAXB doesn't do anything special with these naming conventions. Apart from these names, there is thus nothing special about the java types that JAXB generates for them. For exceptions, this leads to the confusing situation of having something named FooException that doesn't inherit from java.lang.Exception (I'll call them "fake" exceptions from now on).

This means that in our AbstractMarshallingPayloadEndpoint implementation (which is what you use if you want to implement web services with spring and a marshalling technology like JAXB), we can't throw these exceptions (as they are not throwable).

The second problem is that these fake exceptions don't have the JAXB @XmlRootElement annotation on them. This means that they can not directly be serialized: you first have to create a valid JAXBElement for them using the ObjectFactory that is generated by JAXB, something like

final JAXBElement element = new ObjectFactory().createFooException((FooException) exception);

The third problem is that spring's built in support for dealing with exceptions doesn't serialize the exception in the detail part of a SOAP fault. By default, spring only renders a fault code and error message in the SOAP fault.

Serializing exceptions as SOAP fault detail messages

Let's forget about the first and second problems for now and assume all exceptions properly inherit from java.lang.Exception and have the JAXB @XmlRootElement annotation on them. By writing a custom SoapFaultMappingExceptionResolver (or directly implementing EndpointExceptionResolver if you want to), you can customize the way the SOAP fault is generated. This is how the customizeFault method could be implemented:

@Override
protected void customizeFault(
 final Object endpoint, 
 final Exception exception, 
 final SoapFault fault)
{
 super.customizeFault(endpoint, exception, fault);

 // get the marshaller
 AbstractMarshallingPayloadEndpoint marshallingendEndpoint = (AbstractMarshallingPayloadEndpoint) endpoint;

 // get the result inside the fault detail to marshal to
 Result result = fault.addFaultDetail().getResult();

 // marshal
 try
 {
   marshallingendEndpoint.getMarshaller().marshal(exception, result);
 } catch (IOException e)
 {
   throw new RuntimeException(e);
 }
}

Note how we use the marshaller from the given endpoint to marshal the exception into the detail of the SOAP fault (represented using the Result interface).

Using JAXB's ObjectFactory to solve the second problem

As explained already, the second of the three problems forces us to use JAXB's ObjectFactory for the exceptions. This means that, in the code example above, we can't directly pass the exception to the marshaller. We first have to create a JAXBElement from the exception. This forces us to known the exact type of the exception, and thus requires ugly casting, for example:

@Override
protected void customizeFault(
 final Object endpoint,
 final Exception exception,
 final SoapFault fault)
{
 super.customizeFault(endpoint, exception, fault);

 // get the marshaller
 AbstractMarshallingPayloadEndpoint marshallingendEndpoint = (AbstractMarshallingPayloadEndpoint) endpoint;

 // get the result inside the fault detail to marshal to
 Result result = fault.addFaultDetail().getResult();

 // create the corresponding jaxb element
 final JAXBElement element;
 if (exception instanceof FooException)
 {
  element = new ObjectFactory().createFooException((FooException) exception);
 } else if (...)
 {
  // else if required for all possible exceptions
 }

 // marshal
 try
 {
   marshallingendEndpoint.getMarshaller().marshal(exception, result);
 } catch (IOException e)
 {
  throw new RuntimeException(e);
 }
}

As you can see, this code is no longer scalable. For every new exception, you have to add code.

Wrapping fake exceptions in a real exception

The last problem to solve is that the fake exceptions are not really java exceptions. We've looked into possibilities to customize the JAXB generation of java types in such a way that everything that ends with "Exception" is actually a real exception, but that didn't seem to be possible.

We ended up writing a solution that works, although I must say it is not very elegant. We've created a custom exception to wrap the "fake" exceptions generated by JAXB (called it UnmarshalledExceptionWrapperException). The wrapper wraps instances of type Object and verifies that their class names end with "Exception" (that really seems to be the best thing we can do). The custom SoapFaultMappingExceptionResolver now has to unwrap these fake exceptions and build a valid JAXB element from them:

@Override
protected void customizeFault(
 final Object endpoint,
 final Exception exception,
 final SoapFault fault)
{
 super.customizeFault(endpoint, exception, fault);

 // get the wrapper
 UnmarshalledExceptionWrapperException exception = (UnmarshalledExceptionWrapperException) ex;

 // unwrap the exception
 Object unwrappedException = exception.getWrappedException();

 // get the marshaller
 AbstractMarshallingPayloadEndpoint 
  marshallingendEndpoint = (AbstractMarshallingPayloadEndpoint) endpoint;

 // get the result inside the fault detail to marshal to
 Result result = fault.addFaultDetail().getResult();

 // create the corresponding jaxb element
 final JAXBElement element;
 if (unwrappedException instanceof FooException)
 {
  element = new ObjectFactory().createFooException((FooException) unwrappedException);
 } else if (...)
 {
  // else if required for all possible exceptions
 }

 // marshal
 try
 {
   marshallingendEndpoint.getMarshaller().marshal(exception, result);
 } catch (IOException e)
 {
  throw new RuntimeException(e);
 }
}

Conclusion

We ended up with a working spring web services implementation to mimic the behavior of Axis2 in respect to SOAP faults, although it is not the kind of java code one can be very proud of. The major problem seems to be that code modification is required when adding, removing or changing the available fault messages in the WSDL. We would have a far more elegant solution if we could get JAXB 2.0 to generate real exceptions with an @XmlRootElement annotation on them.

building the latest scala eclipse plugin from source

2008-03-05T10:08:00.010+01:00

Rumors have been around for some time now that someone in the scala team is working on a new version of the eclipse plugin for scala. From the scala website you can download a version of the eclipse plugin that is up to date with the latest development on scala itself, but that plugin doesn't seem to get any new features.

The new rewritten plugin can not be downloaded yet (or at least I didn't find it anywhere), so this is what I did to build it from source. You'll need java and ant (including the optional tasks) installed to get this working. The examples are for building on linux, so there might be some differences for other platforms.

First you'll need to checkout the scala and plugin sources from subversion:

svn co http://lampsvn.epfl.ch/svn-repos/scala/scala/trunk scala
svn co http://lampsvn.epfl.ch/svn-repos/scala/plugin plugin

You can build scala, or if you already have an up to date scala distribution on your system somewhere, you can also point the plugin build to that existing scala distribution (see build.properties.SAMPLE in the plugin directory).

Building scala is as simply as running ant with enough memory (in the scala directory):

export ANT_OPTS='-Xms512M -Xmx1024M'; ant dist

The same is true for building the plugin (in the plugin directory):

export ANT_OPTS='-Xms512M -Xmx1024M'; ant dist

Now start eclipse and uninstall the existing scala plugin if you happen to have that one installed. To install the new plugin, you can create a new local update site and point it to the dist/scala.update directory in the plugin directory you've just built.

To get existing scala projects to work with the new plugin, you have to modify the .project files a bit. This is an example:

<?xml version="1.0" encoding="UTF-8"?>
<projectDescription>
 <name>myproject</name>
 <comment></comment>
 <projects>
 </projects>
 <buildSpec>
  <buildCommand>
   <name>scala.plugin.scalabuilder</name>
   <arguments>
   </arguments>
  </buildCommand>
 </buildSpec>
 <natures>
  <nature>scala.plugin.scalanature</nature>
  <nature>org.eclipse.jdt.core.javanature</nature>
 </natures>
</projectDescription>

That's all. Now you've got code completion in scala ;-)

Why paging database queries doesn't work

2008-02-26T18:42:00.020+01:00

At work, we have several (Java) applications where there is a need to 'page' the data from the database (using hibernate and MySQL). With paging I mean that from all the interesting records, only a few have to be processed or displayed at a time. This requirement often comes from GUI design, but equally often it is required for reasons of performance and memory usage. For example we have this application that generates all kinds of reports at regular intervals based on information it obtains over some remote interface from different remote applications. The amounts of data we have, prevent us from not paging these queries.

The question is how this paging can be implemented. We also require every page to be of the same size (except obviously the last one).

We use hibernate, but the problem is the same when using plain SQL, so I'll stick to SQL in the examples. As a running example, let's assume a one-to-many mapping from a table 'a' to 'b'.

create table a (
   id integer auto_increment, 
   primary key (id));
create table b (
   id integer auto_increment, 
   a_id integer, 
   primary key(id));

A first naive attempt

select * from a 
    inner join b on b.a_id = a.id 
    order by a.id limit x,y;

This query uses the MySQL limit x,y construct, meaning "return y results, starting with result number x".

Unfortunately it doesn't work as expected. Due to the join, the above query returns a result for every b, not for every a. This means the paging will be plain wrong.

A sub query approach

select * from a 
    inner join b on b.a_id = a.id 
    where a.id in
(
    select id from a order by id limit x,y
);

In this query, we try to circumvent the problem by paging on a sub query without joins, and then joining in another query for all the results obtained from the paged query. Unfortunately this doesn't work either, because MySQL doesn't allow the limit construct inside a sub query inside an in clause.

Variation on the same idea

The query and sub query can be split into two distinct queries. First we query the primary keys of all a's in a certain page:

select id from a 
    order by id limit x,y;

Then we use that list of primary keys in a second query:

select * from a 
    inner join b on b.a_id = a.id 
    where a.id in 
    (list of ID's from the previous query);

This actually works, but now the size of the second query is proportional to the page size. This has crashed our MySQL clustered database if the pages grow over 1000 records.

A no-join solution

Another obviously working solution is to not join at all:

select * from a 
    order by id limit x,y;

If necessary, the b records can be fetched from the database with subsequent queries. If, in the context of a certain application, all these b records are going to be needed anyway, this has a tremendous performance impact because in stead of only 1 (fairly complex) query, we now have to do n + 1 (simple) queries.

Dropping the fixed page size requirement

If we drop the requirement that every page (except the last one) should be of the same size, we can page the query without relying on the MySQL limit construct. This allows us to page on actual well chosen constraints, rather than on the number of results that happens to be the consequence of a certain set of joins in a query. For example, we could page on the primary key:

select * from a 
    inner join b on b.a_id = a.id 
    where a.id > x and a.id <= x+y;

With an auto increment primary key on a table where deletes are not very frequent, this might yield good results. However, if from time to time many records are being deleted from the database, this might result in smaller or even empty pages.

Conclusion

This leads me to a rather pessimistic conclusion: paged database queries are not generally possible with a single well performing query, at least not with MySQL 5.

DataInput.skipBytes(int n)

2008-02-17T21:29:00.010+01:00

In Java, DataInput defines an interface to reconstruct primitives from an InputStream (usualy written earlier with a DataOutput). Besides methods like readInt, readLong, readDouble etc, it also provides a method skipBytes(int n). What do you expect this method to do? Do you spot the problem in this piece of code?

DataInputStream input = new DataInputStream(
   new BufferedInputStream(
       new FileInputStream(file)));
input.skipBytes(2);

I wasn't aware of any possible problem until I enabled the maven findBugs plugin on our continuous build server. It warned me about not checking the return value of the skipBytes(int n) method. What? What can it possibly return? If you read the javadoc of the method (what I probably should have done in the first place), you'll see that it returns the number of bytes actually skipped. That is because the method skipBytes doesn't actually skip bytes... it only attempts to skip bytes.

So here's my question: what am I supposed to do when the return value is less then the number of bytes I wanted to skip? Do I need to loop until it is OK? Do I need a fall back implementation? Do I need to raise a runtime exception? Do I need to Thread.yield() and hope another thread will get the chance to fill the buffer of my underlying BufferedInputStream? The only thing the javadoc has to say about this, is that there may be many different reasons why the actual number of skipped bytes differs from the number of bytes you wanted to skip. It seems to me that, depending on the reason, another strategy might be appropriate... but of course there is no way to know what the reason was if it would happen.

Although I could probably have looked into a completely different solution using Java NIO, I ended up writing this:

// skip 2 bytes
input.readByte();
input.readByte();