Home/ Questions/Q 8131953
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-06T09:13:28+00:00

FILE: I’m working with a refined csv version of a searchlog file which contains

  • 0

FILE:
I’m working with a refined csv version of a searchlog file which contains 3.3mio lines of data, with each line resembling a single query and containing various data about that query.
The entries in the file are sorted ascending by the session / userid.

GOAL:
Coupling entries that submitted the same queryterm while belonging to the same userid

APPROACH:
I’m reading the csv file line by line, saving the data in selfmade ‘Entry’-object and adding these objects to an arraylist. When this is done, I’ll sort the list by two criteria with a custom comparator

PROBLEM:

While reading the lines and adding the Entry-objects to the list (which takes very long) the program terminates with a OutOfMemoryException “Java heap”

So it seems that my approach is too hard on memory (and runtime).
Any ideas for a better approach?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Your approach itself may be valid, and perhaps the simplest solution is to simply boost the memory available to the JVM.

    The JVM will only allocate itself a maximum amount of system memory, and you can increase this value via the -Xmx command line attribute. See here for more details.

    Obviously this solution doesn’t scale, and if (in the future) you want to read much bigger files, then you’ll likely need a better solution to reading these files.

    • 0