Home/ Questions/Q 4265632
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-21T06:39:03+00:00

I’ve got a relatively small (<100K) numerical CSV dataset that I want to process

  • 0

I’ve got a relatively small (<100K) numerical CSV dataset that I want to process and graph with some numpy and pylab utilities, and it occurred to me that there’s probably a better way of processing the data than ridiculous custom if-ladders for siphoning out the relevent experimental scenarios and comparisons.

If this data were in a DB rather than a CSV this wouldn’t be a problem, but throwing together a ‘real’ db instance for the sake of this seems to be overkill. Is there a pythonic solution to what I’m looking for?

TL;DR Want to query CSV files like a DB / move CSV’s into a mini-db.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Without knowing any specific details (at all) of your case, I’ll expect that you’ll find eventually one of the following ladders as a dominant one for your case:

    • Simply just use the built in Python sqlite3.
      • However, if the relational model is not a necessity then pytables may be the way to go on.
        • Perhaps still, structured arrays can provide the necessary functionality.
          • But plenty can be still achieved with just proper usage of plain logic functions.
            • After all, get acquainted to live with your ridiculous custom if-ladders.

    Obviously, any of the ladders sketched above will posses its specific pros and cons, depending on the actual case. Thus a really careful mix of them may eventually yield to best ‘overall’ result.

    • 0