Question

I’m writing a lexer in Prolog which will be used as a part of functional language interpreter. Language spec allows expressions like for example let \x = x + 2; to occur. What I want lexer to do for such input is to “return”:

[tokLet, tokLambda, tokVar(x), tokEq, tokVar(x), tokPlus, tokNumber(2), tokSColon] 

and the problem is, that Prolog seems to ignore the \ character and “returns” the line written above except for tokLambda.

One approach to solve this would be to somehow add second backslash before/after every occurrence of one in the program code (because everything works fine if I change the original input to let \\x = x + 2;) but I don’t really like it.

Any ideas?

EDIT:
If anyone should have similar problems, that’s how I solved it:

main(File) :-
  open(File,read,Stream),
  read_stream_to_codes(Stream, Codes),
  lexer(X,Codes,[]),
    ... invoke other methods

Answer 1

Where did you get the string let \x = x + 2; from?

• If it is in your Prolog program: yes, you have to double the backslashes.
• If it is from an external file: How do you read it from there? Maybe that predicate is interpreting the backslash specially.

I got inspired by that problem and wrote a bit of code, which should be portable to all Prolog implementations:

% readline(-Line)
%
% Reads one line from the current input. The line is then returned as a list
% of one-character atoms, excluding the newline character.
% The returned line doesn't tell you whether the end of input has been reached
% or not.
readline(Line) :-
    'readline:read'([], Reversed),
    reverse(Line, Reversed).

'readline:read'(Current, Out) :-
    get_char(C), 'readline:append'(C, Current, Out).

'readline:append'(-1, Current, Current) :- !.
'readline:append'('\n', Current, Current) :- !.
'readline:append'(C, Current, Line) :-
    'readline:read'([C | Current], Line).

I tried it, and it worked for me.

Of course, as explained in question 1846199, you can also use read_line_to_codes/2.