Question

I’m writing a tokenizer that reads code from a file and am having difficulty with the – seemingly – trivial task of getting it to branch into the case that handles string constants (any token that starts and ends with a double quote).

The conditional I wrote for the branch is

if (token.charAt(0) == '"') { // Do some stuff }

where token is the string that was read in.

Alternatively, I’ve also tried

if (token.startsWith("\"")) { // Do some stuff }

I’ve checked the debugger and it still refuses to enter the branch even for valid cases such as the string "Paris" with value field [”, P, a, r, i, s, ”]

Any suggestions would be appreciated.

Answer 1

The left ” and right “ double quotes are different from the standard double quote character ". Try doing a replace() on the string to replace all left and right double quotes with the standard double quote before tokenizing.