Question

Hey StackOverflow Community,

So, I have this line of information from a txt file that I need to parse.

Here is an example lines:

-> date & time AC Power Insolation Temperature Wind Speed
-> mm/dd/yyyy hh:mm.ss kw W/m^2 deg F mph

Using a scanner.nextLine() gives me a String with a whole line in it, and then I pass this off into StringTokenizer, which then separates them into individual Strings using whitespace as a separator.

so for the first line it would break up into:
date
&
time
AC
Power
Insolation
etc…

I need things like “date & time” together, and “AC Power” together. Is there anyway I can specify this using a method already defined in StringTokenizer or Scanner? Or would I have to develop my own algorithm to do this?

Would you guys suggest I use some other form of parsing lines instead of Scanner? Or, is Scanner sufficient enough for my needs?

ejay

Answer 1

oh, this one was tricky, maybe you could build up some Trie structure with your tokens, i was bored and wrote a little class which solves your problem. Warning: it’s a bit hacky, but was fun to implement.

The Trie class:

class Trie extends HashMap<String, Trie> {

    private static final long serialVersionUID = 1L;
    boolean end = false;

    public void addToken(String strings) {
        addToken(strings.split("\\s+"), 0);
    }

    private void addToken(String[] strings, int begin) {
        if (begin == strings.length) {
            end = true;
            return;
        }

        String key = strings[begin];
        Trie t = get(key);
        if (t == null) {
            t = new Trie();
            put(key, t);
        }
        t.addToken(strings, begin + 1);

    }

    public List<String> tokenize(String data) {
        String[] split = data.split("\\s+");
        List<String> tokens = new ArrayList<String>();
        int pos = 0;
        while (pos < split.length) {
            int tokenLength = getToken(split, pos, 0);
            tokens.add(glue(split, pos, tokenLength));
            pos += tokenLength;
        }

        return tokens;
    }

    public String glue(String[] parts, int pos, int length) {
        StringBuilder sb = new StringBuilder();
        sb.append(parts[pos]);
        for (int i = pos + 1; i < pos + length; i++) {
            sb.append(" ");
            sb.append(parts[i]);
        }
        return sb.toString();
    }

    private int getToken(String[] tokens, int begin, int length) {
        if (end) {
            return length;
        }
        if (begin == tokens.length) {
            return 1;
        }

        String key = tokens[begin];
        Trie t = get(key);
        if (t != null) {
            return t.getToken(tokens, begin + 1, length + 1);
        }
        return 1;
    }
}

and how to use it:

Trie t = new Trie();
t.addToken("AC Power");
t.addToken("date & time");
t.addToken("date & foo");
t.addToken("Speed & fun");

String data = "date & time AC Power Insolation Temperature Wind Speed";

List<String> tokens = t.tokenize(data);
for (String s : tokens) {
    System.out.println(s);
}