Question

Problem description: In my application, I have to present the contents of data packets with a certain format. An example:

An example Any packed binary data, for example: 4 byte header, 4 byte type (type codes having pre-defined meanings), then source address, destination address, and so on.

Previously, I made home cooked implementations that stored the data in a binary file (fixed record length allowed fast lookup), but with time I’m realized I’m inventing some kind of a database. For example, I’m implementing my own efficient binary storage format for very large data files. I’m also implementing my own indexing to rapidly run searches on some fields. I think a real DB (even the simple SQLite) can make this stuff transparently simple.

Question #1: are DBs useful for storing such data, and how should it be done? Note that there are no 1-to-many, many-to-many mappings here and other advanced things, it’s just a plain sequence of packets with a certain internal structure I want to display to the user and let him interact with (i.e. search by a certain field).

Question #2: Now suppose the user himself can specify the format of his packets, i.e. in a configuration file: the length of each field, its type, what its values mean (in case of an enumeration) and so on. How do I extend a DB-backed implementation for this? Should the user define DB schemas? Should the configuration file be auto-translated into this schemas? ORM?

Question #3: Even more advanced… Now suppose the data packages can be varying in length and contents. I.e., for type #2 packages, there are some field, for type #3, some other fields, and so on. But I’d still like my app to handle it, displaying everything nicely and also allowing users to specify the formats in config files. How is it done?

Thanks in advance.

Answer 1

Question #1: are DBs useful for storing such data, and how should it be done?

Certainly a database is useful for this application. You could implement your own special-purpose data store, and perhaps it would be more efficient for your specific application, because you can design it for that specialization. A relational database is more general-purpose, but you can avoid weeks or months of development time by employing a database.

I answered another question earlier today on the subject of how to handle extensible types, where each new sub-type has its own distinct set of attributes.

‘product table, many kind of product, each product have many parameters.’

For your application, I would choose the Concrete Table Inheritance design.

Question #2: Now suppose the user himself can specify the format of his packets, i.e. in a configuration file: the length of each field, its type, what its values mean (in case of an enumeration) and so on. How do I extend a DB-backed implementation for this?

I assume the number of packet types are relatively few, and then many packets are inserted with pretty much the same structure. So you should use the database’s ability to manage metadata. I would define an additional table for each new packet types.

I would also store the packets ‘exploded’ so each field of the packet is stored in a separate database column. That way you can index each column individually, to support efficient searching.

You can also define constraints so that some fields are mandatory (NOT NULL) or their values constrained by lookup tables. Again, leveraging the database’s capabilities to use metadata to enforce consistent structure where it’s desirable.

SQL already supports a standard, declarative language for specifying fields with data types, constraints, etc. Why develop a different language which you then have to translate to SQL?

Question #3: Even more advanced… Now suppose the data packages can be varying in length and contents.

Fields that are optional in a given packet type should permit NULL in the corresponding column.