Question

I received some multi-line data via HTTP and have it in one string. I need to filter only lines containing specific keywords and write it to a file.

How do I process these individual lines without consuming excessive memory? I.e. without splitting the input string at newline and then processing the list?

Jython-specific solutions are welcome, too.

Answer 1

I now actually tested the memory requirements of using data.split(‘\n’), re.finditer(‘.*?\n’, data) and StringIO.readline() in Jython. I was surprised to find out that split() didn’t increase used memory (PS Old Gen), StringIO came second and re third.

Jython 2.5.1+:
  split()  +0 x data
  StringIO +2 x data
  re       +4 x data

Jython 2.2.1:
  split()  +0 x data
  re       +2 x data
  StringIO +7 x data

StringIO didn’t use additional memory after the .write() call, i.e. it seems to be backed by the same string in Jython.

I didn’t test speed.