I am using the embedded jetty. I have it deployed on a unix machine, I am monitoring the memory using the top command. I wrote a class that would send 10 requests per second to the Server. When the number of requests reaches 16000, the engine start throwing an exception: Memory allocation failed during query processing. The engine performs sql queries fro every request that is sent.
I used the verbose option and I noticed the increase in memory consumption:
0 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
1401.581: [GC [PSYoungGen: 640K->128K(768K)] 126729K->126249K(130816K), 0.0004840 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
1401.585: [GC [PSYoungGen: 640K->96K(768K)] 126761K->126305K(130816K), 0.0004510 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
1401.590: [GC [PSYoungGen: 605K->128K(768K)] 126814K->126379K(130816K), 0.0004500 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
1401.595: [GC [PSYoungGen: 640K->80K(768K)] 126891K->126419K(130816K), 0.0004270 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
I use jetty.xml to configure the embedded jetty server, I am using a queuedthreadpool, with the following values:
minThreads:10
maxThreads:20
Can anyone help me identify the memory leak? Is it related to the way I am accessing the database? Is it related to Jetty?
Below is the main class (Took a sample from eclipsepedia :(http://wiki.eclipse.org/Jetty/Tutorial/Embedding_Jetty)
public class PPTEST {
private static String fileName = "*************/properties.conf";
private static Properties p = new Properties();
public static void main(String[] args) throws Exception {
try {
loadProperties(fileName);
Resource fileserver_xml = Resource.newResource("********/jetty.xml");
XmlConfiguration configuration = new XmlConfiguration(fileserver_xml.getInputStream());
Server server = (Server) configuration.configure();
server.setHandler(new HandlerApple());
server.start();
server.join()
} catch (Exception e) {
e.printStackTrace();
}
}
private static void loadProperties(String fileName)
throws IOException {
FileInputStream propsFile = new FileInputStream(fileName);
p.load(propsFile);
propsFile.close();
}
}
And this is the HandlerApple Class:
public class HandlerApple extends AbstractHandler {
//Properties p = new Properties();
private String CHARACTERS = "0123456789ABCDEF";
private String OK = "HTTP/1.1 200 OK";
Properties config;
public org.apache.log4j.Logger logger;
private String fileName = "****************/gprs-properties.conf";
private String smsformat = "*****************/smsformat.prop";
private Connection dbConn;
private HashMap hpSystem;
private HashMap hpConf;
private String prefix;
private String mobfiltering;
private String resp600;
private String resp601;
private String resp602;
private String resp603;
private String resp604;
private String resp605;
private String resp607;
private String resp608;
private String resp612;
private String resp613;
private String resp611;
private String resp606;
private String resp666;
private String ipH;
private String portH;
private String timeoutH;
private String trpmC;
private String ttpipC;
private String lpdC;
private String igprsC;
private String first_timeout;
private String second_timeout;
private PreparedStatement pstmt;
private ResultSet rs;
Random randomGenerator = new Random();
HandlerApple() {
DatabaseSingleton d = new DatabaseSingleton();
try {
org.apache.log4j.PropertyConfigurator.configure("************/log4j.properties");
logger = org.apache.log4j.Logger.getLogger(HandlerApple.class.getName());
logger.info("Contructor Level");
dbConn = d.getConnection();
hpSystem = loadConfigFile(fileName);
hpConf = loadConfigFile(smsformat);
prefix = (String) hpSystem.get("prefixes");
mobfiltering = (String) hpSystem.get("mobile-filtering");
resp600 = (String) hpConf.get("600");
resp601 = (String) hpConf.get("601");
resp602 = (String) hpConf.get("602");
resp603 = (String) hpConf.get("603");
resp604 = (String) hpConf.get("604");
resp605 = (String) hpConf.get("605");
resp607 = (String) hpConf.get("607");
resp608 = (String) hpConf.get("608");
resp612 = (String) hpConf.get("612");
resp613 = (String) hpConf.get("613");
resp611 = (String) hpConf.get("611");
resp606 = (String) hpConf.get("606");
resp666 = (String) hpConf.get("666");
ipH = (String) hpSystem.get("module-1.ip");
portH = (String) hpSystem.get("module-1.port");
timeoutH = (String) hpSystem.get("module-1.timeout");
trpmC = (String) hpSystem.get("trpm");
ttpipC = (String) hpSystem.get("ttpip");
lpdC = (String) hpSystem.get("lpd");
igprsC = (String) hpSystem.get("igprs.trpm");
first_timeout = (String) hpSystem.get("timeout.1");
second_timeout = (String) hpSystem.get("timeout.2");
logger.info("==================================================================");
logger.info("Clearing maps");
logger.info("==================================================================");
hpSystem.clear();
hpConf.clear();
pstmt = null;
rs = null;
} catch (Exception e) {
e.printStackTrace();
}
}
public void handle(String target, Request baseRequest, HttpServletRequest request, HttpServletResponse response)
throws IOException, ServletException {
try{
//Some code
}catch (Exception e){
//
}
finally {
response.flushBuffer();
if (pstmt != null) {
try {
pstmt.close();
} catch (SQLException ex) {
Logger.getLogger(HandlerApple.class.getName()).log(Level.SEVERE, null, ex);
}
}
if (rs != null) {
try {
rs.close();
} catch (SQLException ex) {
Logger.getLogger(HandlerApple.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
}
I am using the DatabaseSingleton class:
public class DatabaseSingleton {
//Static instance of connection, only one will ever exist
private Connection connection = null;
//Returns single instance of connection
public Connection getConnection() {
//If instance has not been created yet, create it
if (connection == null) {
initConnection();
}
return connection;
}
private void initConnection() {
try {
Class.forName("com.informix.jdbc.IfxDriver");
Properties p=new Properties();
loadProperties("************/gprs-properties.conf", p);
String dburl = p.getProperty("gprs.informix.url");
String dbuser = p.getProperty("gprs.informix.user");
String dbpass = p.getProperty("gprs.informix.password");
connection =
DriverManager.getConnection(dburl, dbuser, dbpass);
} catch (ClassNotFoundException e) {
System.out.println(e.getMessage());
System.exit(0);
} catch (SQLException e) {
System.out.println(e.getMessage());
System.exit(0);
} catch (Exception e) {
}
}
private void loadProperties(String fileName, Properties p)
throws IOException {
FileInputStream propsFile = new FileInputStream(fileName);
p.load(propsFile);
propsFile.close();
}
After the exception is thrown and I stop the engine, the following gets printed (using the verbose option):
Heap
PSYoungGen total 832K, used 288K [0xb4d60000, 0xb4e60000, 0xb4e60000)
eden space 640K, 32% used [0xb4d60000,0xb4d94060,0xb4e00000)
from space 192K, 41% used [0xb4e30000,0xb4e44010,0xb4e60000)
to space 192K, 0% used [0xb4e00000,0xb4e00000,0xb4e30000)
PSOldGen total 219520K, used 212479K [0x94e60000, 0xa24c0000, 0xb4d60000)
object space 219520K, 96% used [0x94e60000,0xa1ddfc50,0xa24c0000)
PSPermGen total 16384K, used 8145K [0x90e60000, 0x91e60000, 0x94e60000)
object space 16384K, 49% used [0x90e60000,0x91654420,0x91e60000)
The exception trace is the following:
java.sql.SQLException: Memory allocation failed during query processing.
at com.informix.util.IfxErrMsg.getSQLException(IfxErrMsg.java:373)
at com.informix.jdbc.IfxSqli.a(IfxSqli.java:3208)
at com.informix.jdbc.IfxSqli.E(IfxSqli.java:3518)
at com.informix.jdbc.IfxSqli.dispatchMsg(IfxSqli.java:2353)
at com.informix.jdbc.IfxSqli.receiveMessage(IfxSqli.java:2269)
at com.informix.jdbc.IfxSqli.executePrepare(IfxSqli.java:1153)
at com.informix.jdbc.IfxPreparedStatement.e(IfxPreparedStatement.java:318)
at com.informix.jdbc.IfxPreparedStatement.a(IfxPreparedStatement.java:298)
at com.informix.jdbc.IfxPreparedStatement.<init>(IfxPreparedStatement.java:168)
at com.informix.jdbc.IfxSqliConnect.h(IfxSqliConnect.java:5918)
at com.informix.jdbc.IfxSqliConnect.prepareStatement(IfxSqliConnect.java:1999)
at pinpayhttp.HandlerApple.checkIP(HandlerApple.java:520)
at pinpayhttp.HandlerApple.handle(HandlerApple.java:227)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:346)
at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596)
at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:1068)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:807)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:220)
at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:426)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:520)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:40)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:528)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.sql.SQLException
at com.informix.util.IfxErrMsg.getSQLException(IfxErrMsg.java:373)
at com.informix.jdbc.IfxSqli.E(IfxSqli.java:3523)
… 21 more
jmap -histo:live 12469
After the Memory allocation failed exception:
1: 209324 76966096 [Ljava.lang.Object;
2: 12971 45337576 [B
3: 1537794 24604704 java.lang.Integer
4: 172537 13265160 [C
5: 32768 10223616 com.informix.jdbc.IfxSqli
6: 32767 6291264 com.informix.jdbc.IfxPreparedStatement
7: 207565 4981560 java.util.Vector
8: 39807 4056792 [I
9: 160921 3862104 java.lang.String
10: 32767 3145632 com.informix.jdbc.IfxResultSet
11: 19329 2202664 <constMethodKlass>
12: 21846 2097216 com.informix.jdbc.IfxColumnInfo
13: 19329 1550760 <methodKlass>
14: 32767 1310680 com.informix.jdbc.IfxResultSetMetaData
15: 30807 1118264 <symbolKlass>
16: 1400 843192 <constantPoolKlass>
17: 10901 784872 com.informix.jdbc.IfxChar
18: 1400 597000 <instanceKlassKlass>
19: 1276 542848 <constantPoolCacheKlass>
20: 32774 524384 java.lang.StringBuffer
21: 10923 524304 com.informix.jdbc.IfxRowColumn
22: 1274 326536 <methodDataKlass>
23: 2258 181760 [S
24: 1563 150048 java.lang.Class
25: 2147 117904 [[I
26: 2820 67680 java.util.HashMap$Entry
27: 1936 46464 java.lang.StackTraceElement
28: 134 42880 <objArrayKlassKlass>
29: 319 42456 [Ljava.util.HashMap$Entry;
30: 1478 35472 java.util.Hashtable$Entry
31: 293 18752 org.eclipse.jetty.io.BufferCache$CachedBuffer
32: 212 16008 [Ljava.lang.String;
33: 445 14240 org.eclipse.jetty.util.StringMap$Node
34: 76 14000 [Ljava.util.Hashtable$Entry;
35: 163 13040 [Lorg.eclipse.jetty.util.StringMap$Node;
36: 301 12040 java.util.HashMap
37: 358 11456 java.util.LinkedHashMap$Entry
38: 308 9856 java.lang.ref.SoftReference
39: 89 8816 [Ljava.lang.StackTraceElement;
40: 140 7840 org.eclipse.jetty.io.ByteArrayBuffer
41: 195 7800 java.sql.SQLException
42: 192 6144 java.util.concurrent.ConcurrentHashMap$Segment
43: 81 5184 java.lang.reflect.Constructor
44: 10 4896 [[Ljava.lang.Object;
45: 294 4704 java.util.jar.Attributes$Name
46: 195 4680 java.util.concurrent.locks.ReentrantLock$NonfairSync
47: 78 4368 java.net.URL
48: 161 3864 java.lang.ref.WeakReference
49: 192 3440 [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
50: 143 3432 sun.security.util.ObjectIdentifier
51: 85 3400 java.math.BigInteger
52: 138 3312 java.util.ArrayList
53: 66 3168 java.beans.MethodDescriptor
54: 98 3136 java.lang.ref.Finalizer
55: 56 3136 java.security.Provider$Service
56: 124 2976 com.sun.org.apache.xerces.internal.xni.QName
57: 24 2880 java.net.SocksSocketImpl
58: 118 2832 java.security.Provider$ServiceKey
59: 8 2560 <typeArrayKlassKlass>
60: 6 2464 [Lcom.sun.org.apache.xerces.internal.xni.QName;
61: 61 2440 java.util.Hashtable
62: 22 2288 sun.nio.ch.SocketChannelImpl
63: 25 2248 [Z
64: 2 2248 [Lorg.eclipse.jetty.http.HttpStatus$Code;
65: 140 2240 java.util.jar.Attributes
66: 70 2240 java.lang.ThreadLocal$ThreadLocalMap$Entry
67: 7 2120 [J
68: 1 2048 [Lorg.eclipse.jetty.http.HttpGenerator$Status;
69: 36 2016 java.nio.DirectByteBuffer
70: 251 2008 java.lang.Object
71: 27 1944 java.util.jar.JarFile$JarFileEntry
72: 21 1680 java.lang.reflect.Method
73: 16 1664 java.lang.Thread
74: 65 1560 java.util.concurrent.ConcurrentHashMap$HashEntry
75: 19 1520 [Ljava.lang.ThreadLocal$ThreadLocalMap$Entry;
76: 14 1456 [[Ljava.lang.String;
77: 36 1440 sun.misc.Cleaner
78: 30 1440 java.util.ResourceBundle$CacheKey
79: 22 1408 java.beans.PropertyDescriptor
80: 35 1400 sun.misc.URLClassPath$JarLoader
81: 73 1360 [Ljava.lang.Class;
82: 27 1296 java.lang.Package
83: 81 1296 java.util.HashMap$EntrySet
84: 27 1296 java.util.jar.JarFile
85: 50 1200 sun.security.util.DerInputBuffer
86: 50 1200 sun.security.util.DerValue
87: 30 1200 java.util.ResourceBundle$BundleReference
88: 50 1200 sun.security.x509.RDN
89: 47 1128 java.net.InetSocketAddress
90: 47 1128 org.eclipse.jetty.http.HttpStatus$Code
91: 47 1128 java.io.ExpiringCache$Entry
92: 10 1120 <klassKlass>
93: 46 1104 org.eclipse.jetty.http.HttpGenerator$Status
94: 8 1088 java.text.DecimalFormat
95: 45 1080 java.net.Inet4Address
96: 2 1056 [Ljava.lang.Integer;
97: 22 1056 sun.nio.ch.SocketAdaptor
98: 1 1040 [Lcom.sun.org.apache.xerces.internal.impl.dtd.models.ContentModelValidator;
99: 1 1040 [Lsun.text.normalizer.UnicodeSet;
100: 1 1040 [Lcom.sun.org.apache.xerces.internal.impl.dv.DatatypeValidator;
101: 8 1000 [Ljava.beans.MethodDescriptor;
102: 41 984 com.sun.org.apache.xerces.internal.util.SymbolTable$Entry
103: 12 960 [Ljava.util.concurrent.ConcurrentHashMap$Segment;
104: 30 960 java.util.ResourceBundle$LoaderReference
105: 40 960 com.informix.util.IfxMessage$bundleMap
106: 58 928 <compiledICHolderKlass>
107: 28 896 java.util.zip.Inflater
108: 8 896 java.util.GregorianCalendar
109: 36 864 java.nio.DirectByteBuffer$Deallocator
110: 9 864 sun.util.calendar.Gregorian$Date
111: 52 832 java.security.Provider$UString
112: 50 800 sun.security.util.DerInputStream
113: 50 800 sun.security.x509.AVA
114: 50 800 [Lsun.security.x509.AVA;
115: 33 792 sun.reflect.NativeConstructorAccessorImpl
116: 16 768 java.util.LinkedHashMap
117: 23 736 java.util.Locale
118: 30 720 java.security.Provider$EngineDescription
119: 18 720 com.sun.org.apache.xerces.internal.impl.dtd.XMLSimpleType
120: 1 704 [Lcom.sun.org.apache.xerces.internal.util.SymbolTable$Entry;
121: 2 640 <arrayKlassKlass>
122: 16 640 java.util.WeakHashMap$Entry
123: 39 624 java.util.regex.Pattern$CharPropertyNames$1
124: 19 608 java.io.ObjectStreamField
125: 15 600 sun.nio.cs.ISO_8859_1$Decoder
126: 25 600 com.sun.org.apache.xerces.internal.util.XMLStringBuffer
127: 12 576 java.nio.HeapByteBuffer
128: 36 576 sun.reflect.DelegatingConstructorAccessorImpl
129: 12 576 sun.security.x509.X500Name
130: 11 528 java.util.Properties
131: 22 528 java.util.Date
132: 22 528 com.sun.org.apache.xerces.internal.xni.XMLString
133: 22 528 sun.security.x509.OIDMap$OIDInfo
134: 22 528 [Ljava.nio.channels.SelectionKey;
135: 8 512 java.text.DecimalFormatSymbols
136: 21 504 java.security.AccessControlContext
137: 21 504 sun.security.x509.AVAKeyword
138: 7 504 org.eclipse.jetty.io.nio.DirectNIOBuffer
139: 31 496 java.io.FileDescriptor
140: 12 480 java.util.concurrent.ConcurrentHashMap
141: 15 480 com.sun.org.apache.xerces.internal.impl.dtd.XMLElementDecl
142: 19 456 java.lang.ThreadLocal$ThreadLocalMap
143: 7 448 java.text.SimpleDateFormat
144: 8 448 org.eclipse.jetty.io.nio.IndirectNIOBuffer
145: 9 432 sun.util.calendar.ZoneInfo
146: 17 408 com.informix.msg.bundleMap
147: 5 400 java.net.URI
Total 2644614 206176808
Can anyone help? I just need some guidance…
Your memory limit appears to be very low for a high load. I would try
-mx1gas a start, more if you have more memory (and a 64-bit JVM).Can you print the exact exception as the message can be important?