Question

I’m trying to develop a Python script to examine every sentence in Barack Obama’s second inaugural address and find similar sentences in past inaugurals. I’ve developed a very crude fuzzy match, and I’m hoping to improve it.

I start by reducing all inaugurals to lists of stopword-free sentences. I then build a frequency index.

Next, I compare each sentence in Obama’s 2013 address to each sentence of every other address, and evaluate the similarity like so:

#compare two lemmatized sentences. Assumes stop words already removed. frequencies is dict of frequencies across all inaugural    
def compare(sentA, sentB, frequencies):
    intersect = [x for x in sentA if x in sentB]
    N = [frequencies[x] for x in intersect]
    #calculate sum that weights uncommon words based on frequency inaugurals
    n = sum([10.0 / (x + 1) for x in N])
    #ratio of matches to total words in both sentences. (John Adams and William Harrison both favored loooooong sentences that tend to produce matches by sheer probability.)
    c = float(len(intersect)) / (len(sentA) + len(sentB))
    return (intersect, N, n, c)

Last, I filter out results based on arbitrary cutoffs for n and c.

It works better than one might think, identifying sentences that share uncommon words in a non-negligible proportion to total words.

For example, it picked up these matches:

---

Obama, 2013:
For history tells us that while these truths may be self-evident, they have never been self-executing; that while freedom is a gift from God, it must be secured by His people here on Earth.

Kennedy, 1961:
With a good conscience our only sure reward, with history the final judge of our deeds, let us go forth to lead the land we love, asking His blessing and His help, but knowing that here on earth God’s work must truly be our own.

---

Obama, 2013
Through blood drawn by lash and blood drawn by sword, we learned that no union founded on the principles of liberty and equality could survive half-slave and half-free.

Lincoln, 1861
Yet, if God wills that it continue until all the wealth piled by the bondsman’s two hundred and fifty years of unrequited toil shall be sunk, and until every drop of blood drawn with the lash shall be paid by another drawn with the sword, as was said three thousand years ago, so still it must be said “the judgments of the Lord are true and righteous altogether.

---

Obama, 2013
This generation of Americans has been tested by crises that steeled our resolve and proved our resilience

Kennedy, 1961
Since this country was founded, each generation of Americans has been summoned to give testimony to its national loyalty.

---

But it’s very crude.

I don’t have the chops for a major machine-learning project, but I do want to apply more theory if possible. I understand bigram searching, but I’m not sure that will work here — it’s not so much exact bigrams we’re interested in as general proximity of two words that are shared between quotes. Is there a fuzzy sentence comparison that looks at probability and distribution of words without being too rigid? The nature of allusion is that it’s very approximate.

Current effort available on Cloud9IDE

UPDATE, 1/24/13
Per the accepted answer, here’s a simple Python function for bigram windows:

def bigrams(tokens, blur=1):
    grams = []
    for c in range(len(tokens) - 1):
        for i in range(c + 1, min(c + blur + 1, len(tokens))):
            grams.append((tokens[c], tokens[i]))
    return grams

Answer 1

If you are inspired to use bigrams, you could build your bigrams while allowing gaps of one, two, or even three words so as to loosen up the definition of bigram a little bit. This could work since allowing n gaps means not even n times as many “bigrams”, and your corpus is pretty small. With this, for example, a “bigram” from your first paragraph could be (similar, inaugurals).