Home/ Questions/Q 8149177
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-06T14:47:47+00:00

I have a number of data files to process from a data warehouse that

  • 0

I have a number of data files to process from a data warehouse that have the following format:

:header 1 ...
:header n
# remarks 1 ...
# remarks n
# column header 1
# column header 2
DATA ROWS
(Example: "#### ## ## #####   ########  ####### ###afp##      ##e###")

The data is separated by white spaces and has both numbers and other ASCII chars. Some of those pieces of data will be split up and made more meaningful.

All of the data will go into a database, initially an SQLite db for development, and then pushed up to another, more permanent, storage.

These files will be pulled in actually via HTTP from the remote server and I will have to crawl a bit to get some of it as they span folders and many files.

I was hopeful to get some input what the best tools and methods may be to accomplish this the “Ruby way”, as well as to abstract out some of this. Otherwise, I’ll tackle it probably similar to how I would in Perl or other such approaches I’ve taken before.

I was thinking along the lines of using OpenURI to open each url, then if input is HTML collect links to crawl, otherwise process the data. I would use String.scan to break apart the file appropriately each time into a multi-dimensional array parsing each component based on the established formatting by the data provider. Upon completion, push the data into the database. Move on to next input file/uri. Rinse and repeat.

I figure I must be missing some libs that those with more experience would use to clean/quicken this process up dramatically and make the script much more flexible for reuse on other data sets.

Additionally, I will be graphing and visualizing this data as well as generating reports, so perhaps that should too be considered.

Any input as to perhaps a better approach or libs to simply this?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Your question focuses on a lot on “low level” details — parsing URL’s and so on. One key aspect of the “Ruby Way” is “Don’t reinvent the wheel.” Leverage existing libraries. 🙂

    My recommendation? First, leverage a crawler such as spider or anemone. Second, use Nokogiri for HTML/XML parsing. Third, store the results. I recommend this because you might do different analyses later and you don’t want to throw away the hard work of your spidering.

    Without knowing too much about your constraints, I would look at storing your results in MongoDB. After thinking this, I did a quick search and found a nice tutorial Scraping a blog with Anemone and MongoDB.

    • 0