Question

Java Scanner has a next() method, which read the next token from a stream, where a token is something separated by delimiters (by default the delimiter is any whitespace character, including tab and new line).

How can I do this with C#/.NET? A stream in C# has Read() and ReadLine(), but they don’t have a notion of whitespace or custom delimiter. I can read the whole line and then use string’s Split() method, but that means I have to read the whole string first (as opposed to next() which only read the next token).

Answer 1

Well, you could read a character at a time, and check whether it matches the delimiter. I believe that StreamReader keeps a buffer anyway, so it shouldn’t have much performance cost (i.e. it won’t hit the disk each time you read a character). You can just keep a StringBuilder of “the token so far” and keep going until you hit a delimiter, at which point you return the token.

Alternatively, for performance reasons, you could read a chunk at a time, and scan through the array of characters looking for the delimiter, then build the string when you’ve found it. Keeping the state consistent will be significantly trickier than just using a StringBuilder and reading a character at a time, but if this is critical to performance then it might be worth it.

I wouldn’t try to emulate all of what Scanner does – just write the bits that you need. You might want to consider implementing this as an IEnumerable<string> using iterator blocks for convenience. You’d probably want to pass in a Func<TextReader> rather than the TextReader itself, so that the iterator block can close the reader appropriately.