I have to migrate a very large dataset from one system to another. One of the “source” column contains a date but is really a string with no constraint, while the destination system mandates a date in the format yyyy-mm-dd.
Many, but not all, of the source dates are formatted as yyyymmdd. So to coerce them to the expected format, I do (in Perl):
return "$1-$2-$3" if ($val =~ /(\d{4})[-\/]*(\d{2})[-\/]*(\d{2})/);
The problem arises when the source dates moves away from the “generic” yyyymmdd. The goal is to salvage as many dates as possible, before giving up. Example source strings include:
21/3/1998,
March 2004,
2001,
3/4/97
I can try to match as many of the examples I can find with a succession of regular expressions such as the one above.
But is there something smarter to do? Am I not reinventing the wheel? Is there a library somewhere doing something similar? I couldn’t find anything relevant googling “forgiving date parser”. (any language is OK).
I finally extracted a test set of more than 200 examples of dates that actually occur in the data set. Some are mildly misbehaved, a few are totally sick (“01010” for example).
I tried all the existing Perl modules I could find, but the success rate was too low. I eventually dived in an reinvented my wheel, achieving a more than 98% success rate.
My algorithm is a succession of increasingly fuzzier recognizers, starting with the rigidly valid dates downs to total guess territory. The first to return a “success” result wins. In the middle of that stack, I have the “main” recognizer which does something like this:
parse sets of numbers in the string, anywhere. “months names” in French and in English are recognized also.
For each of them I put them in three buckets: candidates for year, candidates for month, candidates for day. For example “13” will be in the “possible year” bucket, and in the “possible day” bucket. “February” will only go in the “months” bucket of course. In each bucket, the value is tagged with a “plausibility level”, an arbitrary number that depends on a number of things. For example, 2010 is more plausible as a year than 10.
look in each of the three buckets. If any of them has only one item in it, it’s the value for that bucket. It’s also removed from the other buckets.
look for the remaining missing values in their respective buckets in order (year, month, day), taking the one with the highest plausibility. In case of tie, take the one occurring last in the string (actually, those have slightly higher plausibility). This rules breaks 7/3/2010 as march 7, as I need here in France. Remove that value from the other buckets, should the case apply.
if any value is missing, use a default value (eg I use 8191 as the default year, the largest allowed value in my target system).
The whole thing is awfully heuristics, but fits my requirement that it’s better to have garbage than to lose information.