(Non-technical readers of my blog, you’ll want to glaze past this entry. This is a case where I encountered a problem, scoured the net for any clues and, finding none, slogged on forever until finally solving it. I’m writing this entry because I’ve found that the last couple of times I’ve solved an obscure computer problem, if I posted my solution in my blog then the entry would accumulate lots of comments saying “Thanks so much. I was really stuck on this.”)
So, to state the problem we encountered…
We’ve got a JBoss application that makes pretty heavy use of MDB (message drive beans) to manage tasks. Our JBoss server talks to another proprietary server (via HTTP mostly) and the status of many tasks depends on whether that other server reports it is ready or finished or whatever.
Essentially, we have a fair number of MDBs that kick off other MDBs using JMS messages.
The problem was that we discovered the JBoss server was “accumulating” hundreds if not thousands of threads until eventually the server would die from resource starvation. (OutOfMemory error in this case.)
We looked at the stack traces of these threads to get an idea of what was going on. This has been made possible as of JBoss 4.0.2 via the jmx-console and the ServerInfo JMX MBean. (Note: you have to be running JBoss under a Java 1.5 JVM. That doesn’t mean your code has to be Java 1.5, as ours isn’t)
What we found was hundreds of pairs of threads running
UIL2.SocketManager.WriteTask. Where the ReadTasks were blocking in
java.net.SocketInputStream.socketRead0 and the WriteTasks were waiting via
So here’s the magic clue I’m going to give that I sure wish someone else had posted on the web earlier. This behavior happens if your JBoss code doesn’t close its QueueSession and QueueConnection objects. Actually, it may be one or the other; I found in our code that we hadn’t called
close() on either and the garbage collector was obviously not closing them for me.
All these zombie threads had apparently been created by JBoss’s JMS/MQ system to facilitate reading messages from a JMS sender, and these threads were never knowing to terminate themselves, even when we’d been finished consuming the messages on the server-side.
Hope this helps anyone else in a similar situation! Write a comment if it has.