Question

Is there a simple (possibly with no dependencies) syntax highlighter witch tokenizes Python source code? I need it to return the token type, the offset from the beginning of the code and token length.

I wanted to use Python build-in tokenize but it does not return the offsets.

I also looked to a few of standard syntax highlighters but they all tokenize other languages as well, which is overkill for my case. All I’m looking for is simple (and fast) Python syntax highlighter.

Thanks in advance!

Answer 1

I don’t know about the offset but you can get back a stream of tokens from a Python snippet using pygments. It’s a single standalone Python library as far as I know. The stream of tokens can then be used to do whatever you want. I have an example that renders a snippet of Python onto a pygame surface with syntax highlighting using Pygments. It tokenises the input line by line.