Question

Amazon S3 API for Ruby only streams objects when passing a block to the read method.
I’m developing a driver for em-ftpd, and It needs an IOish object to stream the S3 data to the client.
If simply pass myS3Object.read, whole file is loaded into the memory, as stated by S3 API.
Is there any way of encapsulating that into something like a custom IO class, so I can pass the stream to em-ftpd?

Here is my code:

def get_file(path)
  #loading whole file into memory - not efficient
  yield bucket.objects[path].read
end

Here is the code inside em-ftpd that gets the result of my code, preferably as an IOish object, using a block to get its data chunks:

# send a file to the client
def cmd_retr(param)
  send_unauthorised and return unless logged_in?
  send_param_required and return if param.nil?

  path = build_path(param)

  @driver.get_file(path) do |data|
    if data
      send_response "150 Data transfer starting #{data.size} bytes"
      send_outofband_data(data)
    else
      send_response "551 file not available"
    end
  end
end

Thank you.

Answer 1

Turns out that I needed to create a class with a buffer of the S3 data and the methods read and eof?.