Question

I want to be able to run a regular expression on an entire file, but I’d like to be able to not have to read the whole file into memory at once as I may be working with rather large files in the future. Is there a way to do this? Thanks!

Clarification: I cannot read line-by-line because it can span multiple lines.

Answer 1

You can use mmap to map the file to memory. The file contents can then be accessed like a normal string:

import re, mmap  with open('/var/log/error.log', 'r+') as f:   data = mmap.mmap(f.fileno(), 0)   mo = re.search('error: (.*)', data)   if mo:     print 'found error', mo.group(1) 

This also works for big files, the file content is internally loaded from disk as needed.