I need to store some data in a file. FOR THE MOMENT each record (data set) consists in:
- a string (variable length),
- an array of integers (variable length),
- an array of bytes (variable length),
- some integer values.
It wouldn’t be difficult at all to save all these stuff in a binary file. However, I know for sure that (unfortunately) my data format will change in time, and I want to have the possibility to add more fields to each “record”. So, obviously my file format cannot be fixed. I suppose that the best solution will be to save my data in (DB) table but I don’t want to mess up with the big guns (SQL, ADO, BDE, Nexus…).
I need a rudimentary library (if possible single PAS file) that can do that.
Since the purpose of this is rather storing data than working with data, can it be done without a DB table?
Requirements for this library:
- it needs to easily support more than 1 million rows
- really lightweight
- single PAS file if possible
- MANDATORY: easy to install in a new machine (together with the project in which it compiles)
- MANDATORY: in order to use it I don’t need to redistribute anything
- MANDATORY: in order to use it the user doesn’t have to install/setup stuff
- can be freeware/shareware
- it doesn’t have to support SQL queries or similar advanced stuff
I use D7
I don’t think you need a database for this. If you use a database I don’t see how it solves the problem of your data structure changing.
I personally would store to YAML format which is very easily extensible. That requires quite a bit of work linking to some LIBYAML so a very lightweight alternative would be to store to INI files. These are easily extensible whilst maintaining compatibility with old files.
You can quite easily roll your own binary format that is extensible. What you do is you write each record to a block. Each block has a short header which includes its length.
When you read the data you read up to the end of the block and then if you are expecting more data you simply stop reading and use default values for the data. If you have read all the data you know about but are not at the end of the block, the file must have come from a later version of your program and you just skip to the end of the block. Perhaps you warn that the file contained data which you didn’t know about.
Extensibility is achieved by always writing data out in the same order as previous versions. Any new data goes at the end of each block.