Home/ Questions/Q 608025
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-13T17:25:13+00:00

I have a relatively strange question. I have a file that is 6 gigabytes

  • 0

I have a relatively strange question.

I have a file that is 6 gigabytes long. What I need to do, is scan the entire file, line by line, and determine all rows that match an id number of any other row in the file. Essentially, its like analyzing a web log file where there are many session ids that are organized by the time of each click rather than by userID.

I tried to do the simple (dumb) thing, which was to create 2 file readers. One that scans the file line by line getting the userID, and the next to 1. verify that the userID has not been processed already and 2. If it hasn’t been processed, read every line that begins with the userID that is contained in the file and store (some value X, related to the rows)

Any advice or tips on how I can make this process work more efficiently?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Easiest: create a datamodel and import the file in a database and take benefit of JDBC and SQL powers. You can if necessary (when the file format is pretty specific) write a some Java which does import line by line with help of under each BufferedReader#readLine() and PreparedStatement#addBatch().

    Hardest: write your Java code so that it doesn’t unnecessarily keep large amounts of data in the memory. You’re then basically reinventing what the average database already does.

    • 0