Question

I am making a lexer, don’t tell me to not do because I already did most of it.
Currently it makes an array of tokens and that’s it.

I would like to know, what functions the lexer needs to provide and a brief explanation of what each function needs to do.

I’ll accept the most complete list.

An example function would be:

next: Consume the current token and return it

Also, should the lexer have the expect function or should the interpreter implement it?

By the way, the lexer constructor accepts a string as argument and make the lexical analyses and store all the tokens in the “tokens” variable.

The language is javascript, so I can’t overload operators.

Answer 1

You should be able to compile a comprehensive list by writing a program that uses your lexer, and implementing the functions you end up needing.