I’ve a springRest web service endpoint that returns a string text of size 4MB. As we do load test of this endpoint we constantly see heap spikes and ultimately the system crashes. I’m thinking – as we make requests each request is serviced by a thread separately. My hypothesis is: Because the string is saved in a global static variable, each thread takes a copy of the 4MB and after around 3000 requests the heap is all consumed and the system crashes because 3000 threads taking each 4MB is around 12GB and hence the system goes out of memory. But this is my hypothesis.
My question: doesn’t tomcat reclaim the memory after each thread that processes a request has done it’s job?
Is this related to GC (garbage collection)?
In the request life cycle – as a request comes, a thread is created (per that request) does the thread get it’s own copy of the response or it just references the response? if that huge string response is copied to each thread then may be that’s why the heap spike is showing.
When the response is given back to the client how does tomcat reclaim the resources of that thread? when does it do it? is claiming request threads related to GC?
Another aspect that i observed is: delay on the method socketWrite0() – this takes from 70-95% of the response time. It is a bottle neck i think. So in the flow of request response – who writes to the socket? the thread? or the thread hands the response to tomcat and tomcat writes it?
If any of you could give me a hint or an aspect to look at that relates memory spikes with huge string responses, i’d really appreciate it. thanks guys!
rose
Thanks guys for the comments given.
The problem was very cryptic because it comes only when the system is stressed with multiple users on performance test. We tried different changes to cache client, caching strategies and cache time to live, connection time out.
We tried changing the algorithm. Caching more data to avoid expensive calls to legacy systems.
We tried changing the data structure from JSONArray to String.
All didn’t help!!
Finally we figured out it was the load balancer that was creating the issue. We disabled it and it all worked fine.
If i’m given a chance to say one thing: performance/scalability of an enterprise system is bottle necked by the less performant component, so do a careful empirical testing and attack the problem after spending time figuring out the correct pain point. DO NOT RUN TO CHANGE YOUR CODE.