Question

I’m trying to extract a sentence from a paragraph using regular expressions in python.
Usually the code that I’m testing extracts the sentence correctly, but in the following paragraph the sentence does not get extracted correctly.

The paragraph:

"But in the case of malaria infections and sepsis, dendritic cells throughout the body are concentrated on alerting the immune system, which prevents them from detecting and responding to any new infections."
A new type of vaccine?

The code:

def splitParagraphIntoSentences(paragraph):

import re

sentenceEnders = re.compile('[.!?][\s]{1,2}(?=[A-Z])')
sentenceList = sentenceEnders.split(paragraph)
return sentenceList
if __name__ == '__main__':
    f = open("bs.txt", 'r')
    text = f.read()
    mylist = []
    sentences = splitParagraphIntoSentences(text)
    for s in sentences:
        mylist.append(s.strip())
        for i in mylist:
            print i

When tested with the above paragraph it gives output exactly as the input paragraph but the output should look like-

But in the case of malaria infections and sepsis, dendritic cells throughout the body are concentrated on alerting the immune system, which prevents them from detecting and responding to any new infections

A new type of vaccine

Is there anything wrong with the regular expression?

Answer 1

The paragraph you’ve posted as an example has its first sentence
enclosed in double quotes ", and the closing quote comes immediately
after the full stop: infections.”

Your regexp [.!?]\s{1,2} is looking for a period followed by one or
two spaces as sentence terminator, so it won’t catch it.

It can be adjusted to cope with this case by allowing for optional
closing quotes:

sentenceEnders = re.compile(r'''[.!?]['"]?\s{1,2}(?=[A-Z])''')

However, with the above regexp you would be removing the end quote
from the sentence. Keeping it is slightly more tricky and can be done
using a look-behind assertion:

sentenceEnders = re.compile(r'''(?<=[.!?]['"\s])\s*(?=[A-Z])''')

Note, however, that there are a lot of cases where a regexp-based splitter
fails, e.g.:

• Abbreviations: “In the works of Dr. A. B. Givental …” —
  according to your regexp, this will be incorrectly split after
  “Dr.”, “A.” and “B.” (You can adjust the single-letter case,
  but you cannot detect an abbreviation unless you hard-code it.)

• Use of exclamation marks in the middle of the sentence:
  “… when, lo and behold! M. Deshayes himself appeared…”

• Use of multiple quote marks and nested quotes, etc.