Home/ Questions/Q 8009261
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-04T18:23:37+00:00

I’m currently rewriting some python code to make it more efficient and I have

  • 0

I’m currently rewriting some python code to make it more efficient and I have a question about saving python arrays so that they can be re-used / manipulated later.

I have a large number of data, saved in CSV files. Each file contains time-stamped values of the data that I am interested in and I have reached the point where I have to deal with tens of millions of data points. The data has got so large now that the processing time is excessive and inefficient—the way the current code is written the entire data set has to be reprocessed every time some new data is added.

What I want to do is this:

  1. Read in all of the existing data to python arrays
  2. Save the variable arrays to some kind of database/file
  3. Then, the next time more data is added I load my database, append the new data, and resave it. This way only a small number of data need to be processed at any one time.
  4. I would like the saved data to be accessible to further python scripts but also to be fairly “human readable” so that it can be handled in programs like OriginPro or perhaps even Excel.

My question is: whats the best format to save the data in? HDF5 seems like it might have all the features I need—but would something like SQLite make more sense?

EDIT: My data is single dimensional. I essentially have 30 arrays which are (millions, 1) in size. If it wasn’t for the fact that there are so many points then CSV would be an ideal format! I am unlikely to want to do lookups of single entries—more likely is that I might want to plot small subsets of data (eg the last 100 hours, or the last 1000 hours, etc).

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    HDF5 is an excellent choice! It has a nice interface, is widely used (in the scientific community at least), many programs have support for it (matlab for example), there are libraries for C,C++,fortran,python,… It has a complete toolset to display the contents of a HDF5 file. If you later want to do complex MPI calculation on your data, HDF5 has support for concurrently read/writes. It’s very well suited to handle very large datasets.

    • 0