Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
We need to import large amount of data(about 5 millions records) to the postgresql db under rails application.
Data will be provided in xml format with images inside it encoded with Base64.
Estimated size of the xml file is 40GB.
What xml parser can handle such amount of data in ruby?
Thanks.
You’ll want to use some kind of SAX parser. SAX parsers do not load everything to memory at once.
I don’t know about Ruby parsers but quick googling gave this blog post. You could start digging from there.
You could also try to split the XML file to smaller pieces to make it more manageable.