Question

I believe I am not using correctly String Tokenizer. Here is my code:

  buffer = new byte[(int) (end - begin)];
                fin.seek(begin);
                fin.read(buffer, 0, (int) (end - begin));

                StringTokenizer strk = new StringTokenizer(new String(buffer),
                                DELIMS,true);

As you can see I am reading a chunk of lines from a file(end and begin are line numbers) and I am transfering the data to a string tokenizer. My delimitators are:

DELIMS = "\r\n ";

because I want to separate words that have a space between them, or are on the next line.
However this code sometimes separates whole words also. What could be the explanation?? Is my DELIMS string conceived wrong?

Also I am passing “true” as an argument to the tokenizer because I want the delimitators to be treated as tokens as well.( I want this because I want to count the line I am currently at)

Could you please help me. Thanks a lot.

Answer 1

You could always wrap your input stream in a LineNumberReader. That will keep track of the line number for you. LineNumberReader extends BufferedReader, which has a readLine() method. With that, you could use a regular StringTokenizer to get your words as tokens. You could use regular expressions or Scanner, but for this case, StringTokenizer is simpler for beginners to understand and quicker.

You must have a RandomAccessFile. You didn’t specify that, but I’m guessing based on the methods you used. Try something like:

byte [] buffer = ...; // you know how to get this.
ByteArrayInputStream stream = new ByteArrayInputStream(buffer);

// if you have java.util.Scanner
{
    int lineNumber = 0;
    Scanner s = new Scanner(stream);

    while (s.hasNextLine()) {
        lineNum++;
        String line = s.nextLine();
        System.out.format("I am on line %s%n", lineNum);
        Scanner lineScanner = new Scanner(line);

        while (lineScanner.hasNext()) {
            String word = lineScanner.next();
            // do whatever with word
        }
    }
}

// if you don't have java.util.Scanner, or want to use StringTokenizer
{
    LineNumberReader reader = new LineNumberReader(
                          new InputStreamReader(stream));
    String line = null;

    while ((line = reader.nextLine()) != null) {
        System.out.println("I am on line " + reader.getLineNumber());
        StringTokenizer tok = new StringTokenizer(line);

        while (tok.hasMoreTokens()) {
            String word = tok.nextToken();
            // do whatever with word
        }
    }
}