What are these warnings in catalina.out?



Answers

In a nutshell, some ThreadLocals have not been cleaned properly. Most (if not all) J2EE-servers / Application Containers use thread pools for incoming requests and other tasks, to prevent the overhead of starting new threads all the time. The problem is that some libraries (and you yourself, if you don't know better) don't clean up their ThreadLocals after the execution of the task/request ends.

Normally, as the thread dies on end of execution, the objects stored in ThreadLocals are no longer referenced and the garbage collector takes care of removing such objects:

Each thread holds an implicit reference to its copy of a thread-local variable as long as the thread is alive and the ThreadLocal instance is accessible; after a thread goes away, all of its copies of thread-local instances are subject to garbage collection (unless other references to these copies exist).

But when the thread has been fetched from a thread pool, it does not die, but instead is returned to the pool. Since the thread is still alive, so are the referenced ThreadLocals. This manifests both as memory leaks and "leaking" of values from one request to another when the same ThreadLocal is used and the thread handling the request/task was used before.

Question

This question already has an answer here:

I have a web application in Tomcat 7.
When I shut down Tomcat I see these warning (but not always)

SEVERE: The web application [/MyApplication] created a ThreadLocal
with key of type
[org.apache.xml.security.algorithms.MessageDigestAlgorithm$1] (value
[org.apache.xml.security.algorithms.MessageDigestAlgorithm$1@2e2c2e2c])
and a value of type [java.util.HashMap] (value
[{http://www.w3.org/2000/09/xmldsig#sha1=MESSAGE DIGEST SHA-1}]) but
failed to remove it when the web application was stopped. Threads are
going to be renewed over time to try and avoid a probable memory leak.
Apr 3, 2012 1:56:19 PM org.apache.catalina.loader.WebappClassLoader
checkThreadLocalMapForLeaks   SEVERE: The web application
[/MyApplication] created a ThreadLocal with key of type
[com.sun.xml.bind.v2.ClassFactory$1] (value
[com.sun.xml.bind.v2.ClassFactory$1@25442544]) and a value of type
[java.util.WeakHashMap] (value [{class
com.classes.internal.ContactType=java.lang.ref.WeakReference@17eb17eb,
class
javax.xml.bind.annotation.adapters.HexBinaryAdapter=java.lang.ref.WeakReference@178a178a,
class
com.classes.internal.xjc.ListType=java.lang.ref.WeakReference@181c181c,
class
com.classes.internal.xjc.MessageType=java.lang.ref.WeakReference@17711771,
class
com.classes.internal.xjc.MessageType=java.lang.ref.WeakReference@17011701}])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.   Apr 3, 2012 1:56:19 PM
org.apache.catalina.loader.WebappClassLoader
checkThreadLocalMapForLeaks   SEVERE: The web application
[/MyApplication] created a ThreadLocal with key of type
[org.apache.xml.security.utils.UnsyncBufferedOutputStream$1] (value
[org.apache.xml.security.utils.UnsyncBufferedOutputStream$1@4a904a90])
and a value of type [byte[]] (value [[B@67486748]) but failed to
remove it when the web application was stopped. Threads are going to
be renewed over time to try and avoid a probable memory leak.  

What do these warnings mean in catalina.out on shutdown?
I see some of my JAXB classes mentioned, but can't tell what the issue is here.

Any help please?




Solution depends on the library that created these Threads/ThreadLocal-s. Basically you need to call library's cleanup code from your CustomServletContextListener.contextDestroyed() method.

So find what is the library and how to shut it down properly.




Creating Threads without cleaning them up correctly will eventually run you out of memory - been there, done that.

Those who are still wondering for quick solution/workaround, can go for below:

  • If running the standalone tomcat, kill javaw.exe or the process bearing it.
  • If running from eclipse, kill eclipse.exe and java.exe or enclosing process.
  • Still not resolved, Check for the task manager, it is likely that the process which is causing this will be shown with highest memory usage - Do your analysis and kill that.

You should be good to redeploy the stuff and proceed without memory issues.




Memory leak when redeploying application in Tomcat

When you redeploy your application, Tomcat creates a new class loader. The old class loader must be garbage collected, otherwise you get a permgen memory leak.

Tomcat cannot check if the garbage collection will work or not, but it knows about several common points of failures. If the webapp class loader sets a ThreadLocal with an instance whose class was loaded by the webapp class loader itself, the servlet thread holds a reference to that instance. This means that the class loader will not be garbage collected.

Tomcat does a number of such detections, see here for more information. Cleaning thread locals is difficult, you would have to call remove() on the ThreadLocal in each of the threads that is was accessed from. In practice this is only important during development when you redeploy your web app multiple times. In production, you probably do not redeploy, so this can be ignored.

To really find out which instances define the thread locals, you have to use a profiler. For example the heap walker in JProfiler (disclaimer: my company develops JProfiler) will help you to find those thread locals. Select the reported value class (com.sun.xml.bind.v2.runtime.property.SingleElementLeafProperty or com.sun.xml.bind.v2.ClassFactory) and show the cumulated incoming references. One of those will be a java.lang.ThreadLocal$ThreadLocalMap$Entry. Select the referenced objects for that incoming reference type and switch to the allocations view. You will see where the instance has been allocated. With that information you can decide whether you can do something about it or not.




Links



Tags