Question

I’m not sure how am I gonna tokenize source for lexer. For now, I only can think of using regex to parse string into array with given rule (identifier, symbols such as +,-, etc).

For instance,

begin x:=1;y:=2;

then I want to tokenize word, variable (x, y in this case) and each symbol (:,=,;).

Answer 1

Using regexes is a common way of implementing a lexer. If you don’t want to use them then you’ll sort of end up implementing some regex parts yourself anyway.

Although performance-wise it can be more efficient if you do it yourself, it isn’t a must.