Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
I need to run a regex match over a file, but I’m faced with an unexpected problem: the file is too big to read() or mmap() in one call, File objects don’t support the buffer() interface, and the regex module takes only strings or buffers.
Is there an easy way to do this?
The Python
mmapmodule provides a nice Python-friendly way of memory mapping a file. On a 32-bit operating system, the maximum size of the file is will be limited to no more than a GB or maybe two, but on a 64-bit OS you will be able to memory map a file of arbitrary size (until storage sizes exceed 264, of course).I’ve done this with files of up to 30 GB (the Wikipedia XML dump file) in Python with excellent results.