Last days I was working in analyzing some reported OutOfMemoryError in a JVM application, that’s why I decided to share some information about how to deal with it. The first thing many people have in mind when they see an OutOfMemoryError is that we might have in front of a Memory Leak, a quick answer to that is that maybe yes but maybe not.
So, let’s try to determine the root cause of this error.
Memory Types in Java (JVM applications)
Before Java 8, we had 2 types of memory in Java: Heap and Permgen.
PermGen Space: Is the internal space where the representation of the Java classes is stored. PermGen contains meta-data of the classes and the object.
Heap Space: is the place where the memory allocates Objects. Whenever we create an object, it’s always created in the Heap space. The Garbage Collection runs on the heap memory to free the memory used by objects that don’t have any reference.
If you are getting an java.lang.OutOfMemoryError: PermGen space it’s probable that you don’t have a memory leak. Your application is just loading more meta-data than the available space can handle.
To fix that, just increment the available PermGen Space with the JVM argument -XX:MaxPermSize=[size] and the initial space with –XX:PermSize=[size]
With Java 8 and later versions, the Permgen space has been replaced by the Meta-Space. It basically does the same, but one key difference is that Meta-Space grows automatically. Therefore, it’s less possible to get an OutOfMemoryError because of the meta-space.
Heap Space remains the same after with changes in Java 8.
Analyzing the Memory Usage
We are getting an OutOfMemoryError: Heap space, then we need to find out the root cause of the issue. In the best case, we might realise that the application has become bigger and it runs complex logic that requires more memory than the available. Therefore the simple solution is to increase the heap memory with the JVM arguments -Xms=[size] and -Xmx=[size].
In the other hand, we might realise increasing the memory heap is not the solution, maybe it only delays the application to crash, but it finally crashes no matter how much memory the application has. Therefore, we are facing a memory leak.
How to know which scenario we are facing is the key to get a real solution. That’s why there are tools to help us to analyze the memory usage of your application, which objects are being created, how much space those objects are wasting, and other interesting metrics, and have the big picture of our application and take the correct solution.
Check the Memory remotely with a JMX Client
If we have the possibility to get remote access to your environment (production, staging) where the application is presenting the OutOfMemory errors I would recommend this approach. By adding the following JVM entries we enable your application to expose the memory usage via JMX.
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.rmi.port=9999
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.local.only=false
-Djava.rmi.server.hostname=127.0.0.1"The values of port and hostname might change as you needed.
Then we can use a JMX client like JVisualVM to connect remotely and look at real-time the memory usage of your application. We might want to perform and execute the processes in the system that we think might trigger the memory leak, and monitor the objects created and the consumed memory, and when the processes finish we can check whether everything gets back to normal or not.
Generate a Heap Dump when needed
Maybe we don’t permission to expose ports or get access to your application as the previous approach needs, but we can get SSH access to it. In this case, we can use the JMap tool (included in your JDK installation) and generate a memory Heap dump.
jmap --dump:live,format=b,file=<filename of dump> <app process id>
It will require to know the process id (PID) of your application and indicate the filename or location where the Heap Dump will be saved.
Then we can copy that file to your PC, and use a tool like JVisualVM to see the content of the file. We will see the instances and classes created, and how much memory are they using.
We would be able to generate many heap-dumps when the application is working and analyze its behaviour, and determine if the returns to normal stay after heavy work or if the memory increases even though the heavy work has finished.
Generate a Heap Dump when an OOM occurs
For sure an OutOfMemoryError is very hard to reproduce in some applications, and we won’t be able to keep tracking the memory usage 24/7 for many days until the error is thrown.
What about generating a Heap Dump in the exact moment when your application crashes with a OutOfMemoryError? That would be nice and very helpful. That’s exactly the result of adding the JVM argument -XX:+HeapDumpOnOutOfMemoryError to the application.
-XX:+HeapDumpOnOutOfMemoryErrorI recommend adding this JVM flag in the applications always. It won’t generate overhead in your system, just be sure that we have enough space in your hard drive to save the file.
Then we can copy that file to your PC, and use a tool like JVisualVM to see the content of the file. We will see the objects created and with lucky determine what was consuming so much memory to trigger that error.
How do I fix the memory leak?
Let’s say we already analyzed the memory usage of your application. We find out it has a memory leak and it’s required to fix it. The billion-dollar question, How do I fix the memory leak?
Unfortunately, there is no answer to that. It depends on what you see during your analysis, which process triggers the leak. However, I can give some tips that you can try.
- Check the objects that are consuming more memory have the equals() method implemented. In many cases collections like Map, can have bad performance because the equals() method is not well implemented.
- If you found some logic where the application triggers a memory leak, look for unclosed connections (File, Sockets, InputStream, OutputStream) in that code. In other words, check that you always close streams and/or connections correctly.
- Avoid overriding the finalize method. Even though that method is executed when the garbage collector tries to release an object, and you can try to release resources there. Overriding that method can cause a healthy object to become a prone to memory leak object.
- It might be required to apply huge changes in your application (structures, patterns, existing logic), so be prepared to that. Do not think the solution will require 5 minutes and the work is done.
References
- https://www.baeldung.com/java-permgen-metaspace
- https://www.journaldev.com/4098/java-heap-space-vs-stack-memory
- https://plumbr.io/outofmemoryerror/java-heap-space