Question

Possible Duplicate:
Removing the first line of a text file in C#

What would be the fastest and smartest way to remove the first line from a huge (think 2-3 GB) file?

• I think, that you probably can’t avoid rewriting the whole file chunk-by-chunk, but I might be wrong.

• Could using memory-mapped files somehow help to solve this issue?

• Is it possible to achieve this behavior by operating directly on the file system (NTFS, for example) – say, update the corresponding inode data and change the file starting sector, so that the first line is ignored? If yes, would this approach be really fragile or there are many other applications, except the OS itself that do something similiar?

Answer 1

NTFS by default on most volumes (but importantly not all!) stores data in 4096 byte chunks. These are referenced by the $MFT record, which you cannot edit directly because it’s disallowed by the Operating System (for reasons of sanity). As a result, there is no trick available to operate on the filesystem to do something approaching what you want (in other words, you cannot directly reverse truncate a file on NTFS, even in filesystem chunk sized amounts.)

Because of the way files are stored in a filesystem, the only answer is that you must rewrite the entire file directly. Or figure out a different way to store your data. a 2-3GB file is massive and crazy, especially considering you referred to lines meaning that this data is at least in part text information.

You should look into putting this data into a database perhaps? Or organizing it a bit more efficiently at the very least.