Home/ Questions/Q 9212723
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-18T01:35:24+00:00

I have a multithreaded crawler. In this program, if I load a lot of

  • 0

I have a multithreaded crawler. In this program, if I load a lot of seeds, I get an error. I saw the java.lang.OutOfMemoryError and thought maybe the memory is not enough. I tried running the crawler.jar file with these arguments: java -Xms512m -Xmx3G -jar crawler.jar but so far, no luck.

This is the StackTrace of the program:

Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:597)
        at com.sleepycat.je.utilint.DaemonThread.runOrPause(DaemonThread.java:99)
        at com.sleepycat.je.dbi.EnvironmentImpl.runOrPauseDaemons(EnvironmentImpl.java:772)
        at com.sleepycat.je.dbi.EnvironmentImpl.envConfigUpdate(EnvironmentImpl.java:717)
        at com.sleepycat.je.dbi.EnvironmentImpl.finishInit(EnvironmentImpl.java:579)
        at com.sleepycat.je.dbi.DbEnvPool.getEnvironment(DbEnvPool.java:204)
        at com.sleepycat.je.Environment.makeEnvironmentImpl(Environment.java:230)
        at com.sleepycat.je.Environment.<init>(Environment.java:212)
        at com.sleepycat.je.Environment.<init>(Environment.java:166)
        ...

Is this related to memory as I suspected? Does adding the -Xms512m -Xmx3G work when I’m running the jar file using java -jar?

I ran task manager (I’m running on Windows Server) but after running the app, the memory doesn’t go that far higher! Am I wrong?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    The -Xms512m -Xmx3G option only effect the heap size of JVM and would not solve your problem.

    However, the default thread number limits should be enough in most scenarios. You can increase the limit by tweaking JVM/System options, but no matter how many thread you create, the capacity of your system is bounded to your computer resource. e.g. cpu, memory, network, etc.

    I suggest to solve this problem from a different view:

    • Try to share the Environment instances among the worker threads.
    • Try to control the concurrent level to a rate in which your system is busy and thread number limits is not touched.
    • 0