Question

I have a 1.6gb xml file, and when I parse it with Sax Machine it does not seem to be streaming or eating the file in chunks – rather it appears to be loading the whole file into memory (or maybe there is a memory leak somewhere?) because my ruby process climbs upwards of 2.5gb of ram. I don’t know where it stops growing because I ran out of memory.

On a smaller file (50mb) it also appears to be loading the whole file. My task iterates over the records in the xml file and saves each record to a database. It takes about 30 seconds of “idling” and then all of a sudden the database queries start executing.

I thought SAX was supposed to allow you to work with large files like this without loading the whole thing in memory.

Is there something I am overlooking?

Many thanks

Update to add code sample

class FeedImporter

  class FeedListing
    include ::SAXMachine

    element :id
    element :title
    element :description
    element :url

    def to_hash
      {}.tap do |hash|
        self.class.column_names.each do |key|
          hash[key] = send(key)
        end
      end
    end
  end

  class Feed
    include ::SAXMachine
    elements :listing, :as => :listings, :class => FeedListing
  end

  def perform
    open('~/feeds/large_feed.xml') do |file|

      # I think that SAXMachine is trying to load All of the listing elements into this one ruby object.
      puts 'Parsing'
      feed = Feed.parse(file)

      # We are now iterating over each of the listing elements, but they have been "parsed" from the feed already.
      puts 'Importing'
      feed.listings.each do |listing|
        Listing.import(listing.to_hash)
      end

    end
  end

end

As you can see, I don’t care about the <listings> element in the feed. I just want the attributes of each <listing> element.

The output looks like this:

Parsing
... wait forever
Importing (actually, I don't ever see this on the big file (1.6gb) because too much memory is used :(

Answer 1

I forked sax-machine so that it uses constant memory: https://github.com/gregwebs/sax-machine

Good news: there is a new maintainer that is planning on merging my changes.
Myself and the new maintainer have been using my fork without issue for a year now.