Question

Problem description

I want to refactor a parser for a flexible csv like format which describes the columns in the first line. Depending on this information I want the parser to build objects which have simple attributes but also complex ones like a List<String> (space separated), for example Things:

Example data type

import java.util.List;

public class Thing {
    protected int           foo;
    protected String        bar
    protected List<String>  baz;

    public Thing(int foo, String bar, List<String> baz) {
        this.foo = foo;
        this.bar = bar;
        this.baz = baz;
    }

    public String toString() {
        return "foo: " + foo + ", bar: " + bar + ", baz: " + baz;
    }
}

The parser’s input will be text files with a column line (comma separated) in the first line and the data in the n next lines (comma separated). To simplify testing, I will use Iterator<String> for input lines. This simple test should illustrate what I want to build:

JUnit test

// prepare example string iterator
List<String> lines = new ArrayList<String>();
lines.add("bar,baz,foo");
lines.add("yay,quux quuux,17");
lines.add("hey,qaax qaaax,42");

// test parsed things
List<Thing> things = ThingBuilder.buildThings(lines.iterator());
assertNotNull(things);
assertEquals(2, things.size());
assertEquals("foo: 17, bar: yay, baz: [quux, quuux]", things.get(0).toString());
assertEquals("foo: 42, bar: hey, baz: [qaax, qaaax]", things.get(1).toString());

Easiest approach

1. read the first line and split it in column names
2. read all other lines and do the following with them:
   • split the line in tokens
   • loop over them:
     • for token i do a big switch/else if on column name i to
     • transform token i
     • store the extracted value somewhere
   • collect everything and build a Thing
3. done.

My problem with this approach is the inner switch. After processing the first line, it should be clear how lines are parsed.

What I would like

In a language with closures, I would try the following:

1. read the first line and split it in column names
2. for each column name create a closure which sets the right value for a given token and add it to an array of parser closures
3. read all other lines and do the following with them:
   • split the line in tokens
   • loop over them:
     • call parser closure i with token i
   • collect everything and build a Thing
4. done.

What I tried

I have a simple interface for all three token parsers. They are supposed to get a token and inject the generated value in the given ThingBuilder‘s cache:

public interface TokenParser {
    public void parse(String token, ThingBuilder builder);
}

public class FooParser implements TokenParser {
    @Override public void parse(String token, ThingBuilder builder) {
        builder.setFoo(Integer.parseInt(token));
    }
}

public class BarParser implements TokenParser {
    @Override public void parse(String token, ThingBuilder builder) {
        builder.setBar(token);
    }
}

import java.util.ArrayList;
import java.util.List;
public class BazParser implements TokenParser {
    @Override public void parse(String token, ThingBuilder builder) {
        List<String> baz = new ArrayList<String>();
        for (String s : token.split(" ")) baz.add(s);
        builder.setBaz(baz);
    }
}

My ThingBuilder‘s buildThings method is static and creates a ThingBuilder object internally, the constructor gets the first (columns) line. This is also the place where the token parser list is filled. After this the hidden ThingBuilder object is ready and with the following input lines the buildThing method is called repeatedly to create a list of Things:

import java.util.ArrayList;
import java.util.List;
import java.util.Iterator;

public class ThingBuilder {

    // single column parsers
    protected List<TokenParser> columnParsers;

    // thing attribute cache
    protected int           fooCache;
    protected String        barCache;
    protected List<String>  bazCache;

    // thing attribute cache setter
    public void setFoo(int          foo) { fooCache = foo; }
    public void setBar(String       bar) { barCache = bar; }
    public void setBaz(List<String> baz) { bazCache = baz; }

    // cleanup helper method
    protected void cleanup() {
        setFoo(0); setBar(null); setBaz(null);
    }

    // statically build a list of things from given lines
    public static List<Thing> buildThings(Iterator<String> lines) {

        // prepare builder with the first line
        ThingBuilder builder = new ThingBuilder(lines.next());

        // parse things
        List<Thing> things = new ArrayList<Thing>();
        while (lines.hasNext()) {
            things.add(builder.buildThing(lines.next()));
        }
        return things;
    }

    // prepares a builder to parse thing lines
    protected ThingBuilder(String columnLine) {

        // split line into columns
        String[] columns = columnLine.split(",");

        // prepare a parser for each column
        columnParsers = new ArrayList<TokenParser>();
        for (String column : columns) {
            TokenParser parser;
            if      (column.equals("foo")) parser = new FooParser();
            else if (column.equals("bar")) parser = new BarParser();
            else if (column.equals("baz")) parser = new BazParser();
            else throw new RuntimeException("unknown column: " + column);
            columnParsers.add(parser);
        }
    }

    // builds a thing from a string
    protected Thing buildThing(String line) {

        // split the line in tokens
        String[] tokens = line.split(",");

        // let the parsers do the work
        for (int i = 0; i < tokens.length; i++) {
            columnParsers.get(i).parse(tokens[i], this);
        }

        // hopefully they're done
        Thing thing = new Thing(fooCache, barCache, bazCache);
        cleanup();
        return thing;
    }
}

This works, but:

What I don’t like about my solution

• It feels complicated!
• The public cache setter thing. Only TokenParsers should be allowed to fill the builder cache.
• What if I have more than one column with int‘s? Do I have to build a single parser class for each column or is it possible to use an IntegerParser class more the once? The problem here is, that the parser has to call the right cache setter method.

Thanks in advance for your hints!

Answer 1

I agree with @btiernay’s answer, but if you want to roll your own implementation, read on …

---

The public cache setter thing. Only TokenParsers should be allowed to fill the builder cache.

Yea. That is a consequence of your TokenParser API, and the way that it “returns” a value by calling a setter on ThingBuilder. In fact, this has an even worse consequence than the one that you’ve identified. That is: your TokenParser API and all of the TokenParser classes are specific to one and only one ThingBuilder class. They are not reusable …

I think you would be better off with an API like this:

  public interface TokenParser<T> {
      public T parse(String token);
  }

What if I have more than one column with int’s? Do I have to build a single parser class for each column or is it possible to use an IntegerParser class more the once? The problem here is, that the parser has to call the right cache setter method.

Yup.

Next create a RowBuilder interface:

  public interface RowBuilder<R>
      public R buildRow(List<String> tokens);
  }

And here’s the tricky bit — create a generic RowBuilder class that looks something like this:

  public class GenericRowBuilder<R> implements RowBuilder<R> {
      public GenericRowBuilder(Class<R> clazz, TokenParser<?>[] parsers) {
          // Extract the return types of the reified parse objects' `parse` 
          // methods, and use this to locate a matching `Constructor<R>` in 
          // `clazz`.  If there isn't one, throw an exception.
          this.clazz = clazz;
          this.parsers = parsers;
      }
      public R parse(List<String> tokens) {
          // Check number of tokens matches number of parsers.
          // Parse each token with corresponding parsers.
          // Use the `Constructor<R>` found above to create the instance of `R`
      }
  }

Now that is all pretty complicated … and requires a good understanding of youw to use Java’s reflection APIs … but the end result is that you can then instantiate a RowBuilder for your class like this:

  RowBuilder<MyRow> rb = new GenericRowBuilder<MyRow>(MyRow.class,
      new TokenParser<?>[]{
          new IntTokenParser(), new FloatTokenParser(), new CustomTokenParser});

and you have something that will:

• check that the right types are used for the right fields,
• check the number of values in each row, and use the right parser for each column value, and
• work with any row class R provided that it has a suitable constructor.