Question

Given a very large string. I would like to process parts of the string in a loop like this:

large_string = "foobar..."
while large_string:
    process(large_string.pop(200))

What is a nice and efficient way of doing this?

Answer 1

You can wrap the string in a StringIO or BytesIO and pretend it’s a file. That should be pretty fast.

from cStringIO import StringIO
# or, in Py3/Py2.6+:
#from io import BytesIO, StringIO

s = StringIO(large_string)
while True:
    chunk = s.read(200)
    if len(chunk) > 0:
        process(chunk)
    if len(chunk) < 200:
        break