Question

I’m trying to search rather big files for a certain string and return its offset. I’m new to lua and my current approach would look like this:

linenumber = 0
for line in io.lines(filepath) do
result=string.find(line,"ABC",1)
linenumber = linenumber+1

if result ~= nil then
offset=linenumber*4096+result
io.close
end
end

I realize that this way is rather primitive and certainly slow. How could I do this more efficiently?

Thanks in advance.

Answer 1

If the file is not too big, and you can spare the memory, it’s faster to just slurp in the whole file and just use string.find. If not you can search the file by block.

Your approach isn’t all that bad. I’d suggest loading the file in overlapping blocks though. The overlap avoids having the pattern split just between the blocks and going unnoticed like:

".... ...A BC.. ...."

My implementation goes like this:

size=4096 -- note, size should be bigger than the length of pat to work.
pat="ABC"
overlap=#pat
fh=io.open(filepath,'rb') -- On windows, do NOT forget the b
block=fh:read(size+overlap)
n=0
while block do
    block_offset=block:find(pat)
    if block_offset then
        print(block_offset)
        offset=block_offset+size*n
        break
    end
    fh:seek('cur',-overlap)
    cur=fh:seek'cur'
    block=fh:read(size+overlap)
    n=n+1
end

if offset then
    print('found pattern at', offset, 'after reading',n,'blocks')
else
    print('did not find pattern')
end

If your file really has lines, you can also use the trick explained here. This section in the Programming in Lua book explains some performance considerations reading files.