Home/ Questions/Q 6106573
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-23T14:05:39+00:00

I am looking some java implementation of sorting algorithm. The file could be HUGE,

  • 0

I am looking some java implementation of sorting algorithm. The file could be HUGE, say 20000*600=12,000,000 lines of records. The line is comma delimited with 37 fields and we use 5 fields as keys. Is it possible to sort it quickly, say 30 minutes?

If you got other approach other than java, it is welcome if it can be easily integrated into java system. For example, unix utility.

Thanks.

Edit: The lines need to be sort is dispersed into 600 files, with 20000 lines each, 4mb for each file. Finally I would like them to be 1 big sorted file.

I am trying to time a unix sort, would update that afterwards.

Edit:

I appended all the files into a big one, and tried the unix sort function, it is pretty good. The time to sort a 2gb file is 12-13 minutes. The append action require 4 minutes for 600 files.

sort -t ',' -k 1,1 -k 4,7 -k 23,23 -k 2,2r big.txt -o sorted.txt
Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    How does the data get in the CSV format? Does it come from a relational database? You can make it such that whatever process creates the file writes its entries in the right order so you don’t have to solve this problem down the line.

    If you are doing a simple lexicographic order you can try the unix sort, but I am not sure how that will perform on a file with that size.

    • 0