Question

Is there a way to get the re.findall, or better yet, re.finditer functionality applied to a stream (i.e. an filehandle open for reading)?

Note that I am not assuming that the pattern to be matched is fully contained within one line of input (i.e. multi-line patterns are permitted). Nor am I assuming a maximum match length.

It is true that, at this level of generality, it is possible to specify a regex that would require that the regex engine have access to the entire string (e.g. r'(?sm).*'), and, of course, this means having to read the entire file into memory, but I am not concerned with this worst-case scenario at the moment. It is, after all, perfectly possible to write multi-line-matching regular expressions that would not require reading the entire file into memory.

Is it possible to access the underlying automaton (or whatever is used internally) from a compiled regex, to feed it a stream of characters?

Thanks!

Edit: Added clarifications regarding multi-line patterns and match lengths, in response to Tim Pietzcker’s and rplnt’s answers.

Answer 1

This is possible if you know that a regex match will never span a newline.

Then you can simply do

for line in file:
    result = re.finditer(regex, line)
    # do something...

If matches can extend over multiple lines, you need to read the entire file into memory. Otherwise, how would you know if your match was done already, or if some content further up ahead would make a match impossible, or if a match is only unsuccessful because the file hasn’t been read far enough?

Edit:

Theoretically it is possible to do this. The regex engine would have to check whether at any point during the match attempt it reaches the end of the currently read portion of the stream, and if it does, read on ahead (possibly until EOF). But the Python engine doesn’t do this.

Edit 2:

I’ve taken a look at the Python stdlib’s re.py and its related modules. The actual generation of a regex object, including its .match() method and others is done in a C extension. So you can’t access and monkeypatch it to also handle streams, unless you edit the C sources directly and build your own Python version.