Question

I’m trying to use Sphinx Search Server to index a really huge file (around 14gb).
The file is whitespace separated, one entry per line.

To be able to use it with Sphinx, I need to provide a xml file to the Sphinx server.
How can I do it without killing my computer ?

What is the best strategy? Should I try to split the main file in several little files? What’s the best way to do it?

Note: I’m doing it in Ruby, but I’m totally open to other hints.

Thanks for your time.

Answer 1

I hate guys who doesn’t write solution after a question. So I’ll try to don’t be one of them, hopefully it will help somebody.

I added a simple reader method to the File class then used it to loop on the file based on a chunk size of my choice. Quite simple actually, working like a charm with Sphinx.

class File

  # New static method
  def self.seq_read(file_path,chunk_size=nil)
    open(file_path,"rb") do |f|
      f.each_chunk(chunk_size) do |chunk|
        yield chunk
      end
    end
  end

  # New instance method
  def each_chunk(chunk_size=1.kilobyte)
    yield read(chunk_size) until eof?
  end

end

Then just use it like this:

source_path = "./my_very_big_file.txt"
CHUNK_SIZE  = 10.megabytes
File.seq_read(source_path, CHUNK_SIZE) do |chunk|
  chunk.each_line do |line|
    ...
  end
end