Question

I am working on a latex document that will require typesetting significant amounts of python source code. I’m using pygments (the python module, not the online demo) to encapsulate this python in latex, which works well except in the case of long individual lines – which simply continue off the page. I could manually wrap these lines except that this just doesn’t seem that elegant a solution to me, and I prefer spending time puzzling about crazy automated solutions than on repetitive tasks.

What I would like is some way of processing the python source code to wrap the lines to a certain maximum character length, while preserving functionality. I’ve had a play around with some python and the closest I’ve come is inserting \\\n in the last whitespace before the maximum line length – but of course, if this ends up in strings and comments, things go wrong. Quite frankly, I’m not sure how to approach this problem.

So, is anyone aware of a module or tool that can process source code so that no lines exceed a certain length – or at least a good way to start to go about coding something like that?

Answer 1

You might want to extend your current approach a bit, but using the tokenize module from the standard library to determine where to put your line breaks. That way you can see the actual tokens (COMMENT, STRING, etc.) of your source code rather than just the whitespace-separated words.

Here is a short example of what tokenize can do:

>>> from cStringIO import StringIO
>>> from tokenize import tokenize
>>> 
>>> python_code = '''
... def foo(): # This is a comment
...     print 'foo'
... '''
>>> 
>>> fp = StringIO(python_code)
>>> 
>>> tokenize(fp.readline)
1,0-1,1:    NL  '\n'
2,0-2,3:    NAME    'def'
2,4-2,7:    NAME    'foo'
2,7-2,8:    OP  '('
2,8-2,9:    OP  ')'
2,9-2,10:   OP  ':'
2,11-2,30:  COMMENT '# This is a comment'
2,30-2,31:  NEWLINE '\n'
3,0-3,4:    INDENT  '    '
3,4-3,9:    NAME    'print'
3,10-3,15:  STRING  "'foo'"
3,15-3,16:  NEWLINE '\n'
4,0-4,0:    DEDENT  ''
4,0-4,0:    ENDMARKER   ''