My essential problem is that I have read only access to a large data set.
This data set has about 300,000 records. Some of these records
contain errors ranging from typos to chunks of missing data. Unfortunately,
it is not possible to fix at the source and also not possible to create a
fixed local clone. As these errors are discovered they are relatively easy
to fix and can usually be accomplished in a few lines of code. The problem
I am struggling with is a sane and extensible system to allow these errors to be
documented and fixed.
Having a method with a bunch of if(primaryKey == 245664) {fix here} works but
is a terrible solution.
I am writing in Java and have looked at the ServiceLoader mechanism but these feels
a bit heavyweight for my purposes. Any suggestions would be much appreciated.
I know your pain.
It may not be exactly what you want, but I suggest you take a look at Google Refine. I prefer tutorials to screencasts, but these are quite good:
http://code.google.com/p/google-refine/wiki/Screencasts
It’s a real super-tool for processing data and might save you a lot of programming work.
I am not sure if it helps automatize the process of fixing or cleaning data, but my best guess is they thought of something like that.