Question

I study source codes of programs, (written in c++) that operate on file streams. But every now and then i stumble upon something called tokens. Can you please explain to me, what tokens do and why are they useful? Thank you.

Answer 1

Stream refers to a sequence of data elements made available over time. Data elements in streams could be of any kind: characters, strings, special-purpose data structures, etc. A text file could be trivially interpreted as a stream of characters.

In the context of programming languages, the word token usually refers to a sequence of one or more related characters from a stream of characters. Tokens offer a level of abstraction on top of character streams, and are often themselves grouped in streams for further processing.

The usual processing sequence in compilers of programming languages starts with lexical analyzers converting streams of characters into streams of tokens, which are then passed on to parsers. Typical representation of a token consists of a type indicator and the content of the token.

Here is an example: consider this fragment of C++ program:

class MyClass {
public:
    string name;
};

A lexer analyzer groups this program into a stream of ten tokens:

'class' (keyword)
'MyClass' (identifier)
'{' (opening brace)
'public' (keyword)
':' (colon)
'string' (identifier)
'name' (identifier)
';' (semicolon)
'}' (closing brace)
';' (semicolon)