Question

I have a large collection of data in an excel file (and csv files). The data needs to be placed into a database (mysql). However, before it goes into the database it needs to be processed..for example if columns 1 is less than column 3 add 4 to column 2. There are quite a few rules that must be followed before the information is persisted.

What would be a good design to follow to accomplish this task? (using java)

Additional notes

The process needs to be automated. In the sense that I don’t have to manually go in and alter the data. We’re talking about thousands of lines of data with 15 columns of information per line.

Currently, I have a sort of chain of responsibility design set up. One class(Java) for each rule. When one rule is done, it calls the following rule.

More Info

Typically there are about 5000 rows per data sheet. Speed isn’t a huge concern because this large input doesn’t happen often.

I’ve considered drools, however I wasn’t sure the task was complicated enough for drols.

Example rules:

1. All currency (data in specific columns) must not contain currency symbols.

2. Category names must be uniform (e.g. book case = bookcase)

3. Entry dates can not be future dates

4. Text input can only contain [A-Z 0-9 \s]

etc..
Additionally if any column of information is invalid it needs to be reported when processing is complete (or maybe stop processing).

My current solution works. However I think there is room for improvement so I’m looking for ideals as to how it can be improved and or how other people have handled similar situations.

I’ve considered (very briefly) using drools but I wasn’t sure the work was complicated enough to take advantage of drools.

Answer 1

If I didn’t care to do this in 1 step (as Oli mentions), I’d probably use a pipe and filters design. Since your rules are relatively simple, I’d probably do a couple delegate based classes. For instance (C# code, but Java should be pretty similar…perhaps someone could translate?):

interface IFilter {    public IEnumerable<string> Filter(IEnumerable<string> file) {    } }  class PredicateFilter : IFilter {    public PredicateFilter(Predicate<string> predicate) { }     public IEnumerable<string> Filter(IEnumerable<string> file) {       foreach (string s in file) {          if (this.Predicate(s)) {             yield return s;          }       }    } }  class ActionFilter : IFilter {   public ActionFilter(Action<string> action) { }    public IEnumerable<string> Filter(IEnumerable<string> file) {       foreach (string s in file) {          this.Action(s);          yield return s;       }   } }  class ReplaceFilter : IFilter {   public ReplaceFilter(Func<string, string> replace) { }    public IEnumerable<string> Filter(IEnumerable<string> file) {      foreach (string s in file) {         yield return this.Replace(s);      }   } } 

From there, you could either use the delegate filters directly, or subclass them for the specifics. Then, register them with a Pipeline that will pass them through each filter.