Python is not my best language, and so I’m not all that good at finding the most efficient solutions to some of my problems. I have a very large string (coming from a 30 MB file) and I need to check if that file contains a smaller substring (this string is only a few dozen characters). The way I am currently doing it is:
if small_string in large_string:
# logic here
But this seems to be very inefficient because it will check every possible sequence of characters in the file. I know that there will only be an exact match on a newline, so would it be better to read the file in as a list and iterate through that list to match?
EDIT: To clear up some confusion on “matching on a newline only”, here’s an example:
small_string = "This is a line"
big_string = "This is a line\nThis is another line\nThis is yet another"
If I’m not mistake, the in keyword will check all sequences, not just every line.
You can use one of these algorithms:
Rabin–Karp string search algorithm
Knuth–Morris–Pratt algorithm (aka KMP) see an implementation here
Although I believe KMP is more efficient, it’s more complicated to implement.The first link includes some pseudo-code that should make it very easy to implement in python.
you can look for alternatives here: http://en.wikipedia.org/wiki/String_searching_algorithm