Question

i have a binary file with following format :

[N bytes identifier & record length] [n1 bytes data] 
[N bytes identifier & record length] [n2 bytes data] 
[N bytes identifier & record length] [n3 bytes data]

as you see i have records with different lengths. in each record i have N bytes fixed which contains and id and the length of data in record.

this file is very big and can contains 3 millions records.

I want to open this file by an application and let user to browse and edit the records.
( Insert / Update / Delete records)

my initial plan is to create and index file from original file and for each record, keep next and previous record address to navigate forward and backward easily. (some sort of linked list but in file not in memory)

• is there library (java library) to help me to implement this requirement ?

• any recommendation or experience that you think is useful?

—————– EDIT ———————————————-

Thanks for guides and suggestions,

some more info:

the original file and its format is out of my control (it’s a third party file) and i can’t change the file format. but i have to read it, let user to navigate over records and edit some of them (insert new record/ update an existing record/ delete a record) and at the end save it back to original file format.

do u still recommend DataBase instead of a normal index file ?

—————– SECOND EDIT ———————————————-

record size in update mode is fixed. it means updated (edited) record has same length as original record’s, unless user delete the record and create another record with different format.

Many Thanks

Answer 1

Seriously, you should NOT be using a binary file for this. You should use a database.

The problems with trying to implement this as a regular file stem from the fact that operating systems do not allow you to insert extra bytes into the middle of an existing file. So if you need to insert a record (anywhere but the end), update a record (with a different size) or remove a record, you would need to:

• rewrite other records (after the insertion/update/deletion point) to make or reclaim space, or
• implement some kind of free space management within the file.

All of this is complicated and / or expensive.

Fortunately, there is a class of software that implements this kind of thing. It is called database software. There are a wide range of options, ranging from using a full-scale RDBMS to light-weight solutions like BerkeleyDB files.

---

In response to your 1st and 2nd edits, a database will still be simpler.

However, here’s an alternative that might perform better for this use-case than using a DB… without doing complicated free-space management.

1. Read the file and build an in-memory index that maps ids to file locations.

2. Create a second file to hold new and updated records.

3. Perform the record adds/updates/deletes:

   1. An addition is handled by writing the new record to the end of the second file, and adding an index entry for it.

   2. An update is handled by writing the updated record to the end of the second file, and changing the existing index entry to point to it.

   3. A delete is handled by deleting the index entry for the record’s key.

4. Compact the file as follows:

   1. Create a new file.

   2. Read each record in the old file in order, and check the index for the record’s key. If the entry still points to the location of the record, copy the record to the new file. Otherwise skip it.

   3. Repeat the step 4.2 for the second file.

5. If we completed all of the above successfully, delete the old file and second file.

Note this relies on being able to keep the index in memory. If that is not feasible, then the implementation is going to be more complicated … and more like a database.