Question

I’ve written a Scrapy spider that extracts text from a page. The spider parses and outputs correctly on many of the pages, but is thrown off by a few. I’m trying to maintain line breaks and formatting in the document. Pages such as http://www.state.gov/r/pa/prs/dpb/2011/04/160298.htm are formatted properly like such:

April 7, 2011

Mark C. Toner

2:03 p.m. EDT

MR. TONER: Good afternoon, everyone. A couple of things at the top,
and then I’ll take your questions. We condemn the attack on innocent
civilians in southern Israel in the strongest possible terms, as well
as ongoing rocket fire from Gaza. As we have reiterated many times,
there’s no justification for the targeting of innocent civilians,
and those responsible for these terrorist acts should be held
accountable. We are particularly concerned about reports that indicate
the use of an advanced anti-tank weapon in an attack against civilians
and reiterate that all countries have obligations under relevant
United Nations Security Council resolutions to prevent illicit
trafficking in arms and ammunition. Also just a brief statement —

QUESTION: Can we stay on that just for one second?

MR. TONER: Yeah. Go ahead, Matt.

QUESTION: Apparently, the target of that was a school bus. Does that
add to your outrage?

MR. TONER: Well, any attack on innocent civilians is abhorrent, but
certainly the nature of the attack is particularly so.

While pages like http://www.state.gov/r/pa/prs/dpb/2009/04/121223.htm have output like this with no line breaks:

April 2, 2009

Robert Wood

11:53 a.m. EDTMR. WOOD: Good morning, everyone. I think it’s just
about still morning. Welcome to the briefing. I don’t have anything,
so – sir.QUESTION: The North Koreans have moved fueling tankers, or
whatever, close to the site. They may or may not be fueling this
missile. What words of wisdom do you have for the North Koreans at
this moment?MR. WOOD: Well, Matt, I’m not going to comment on, you
know, intelligence matters. But let me just say again, we call on the
North to desist from launching any type of missile. It would be
counterproductive. It’s provocative. It further inflames tensions in
the region. We want to see the North get back to the Six-Party
framework and focus on denuclearization.Yes.QUESTION: Japan has also
said they’re going to call for an emergency meeting in the Security
Council, you know, should this launch go ahead. Is this something that
you would also be looking for?MR. WOOD: Well, let’s see if this test
happens. We certainly hope it doesn’t. Again, calling on the North
not to do it. But certainly, we will – if that test does go forward,
we will be having discussions with our allies.

The code I’m using is as follows:

def parse_item(self, response):
    self.log('Hi, this is an item page! %s' % response.url) 

    hxs = HtmlXPathSelector(response)

    speaker = hxs.select("//span[contains(@class, 'official_s_name')]") #gets the speaker
    speaker = speaker.select('string()').extract()[0] #extracts speaker text
    date = hxs.select('//*[@id="date_long"]') #gets the date
    date = date.select('string()').extract()[0] #extracts the date
    content = hxs.select('//*[@id="centerblock"]') #gets the content
    content = content.select('string()').extract()[0] #extracts the content

    texts = "%s\n\n%s\n\n%s" % (date, speaker, content) #puts everything together in a string

    filename = ("/path/StateDailyBriefing-" + '%s' ".txt") % (date) #creates a file using the date

    #opens the file defined above and writes 'texts' using utf-8
    with codecs.open(filename, 'w', encoding='utf-8') as output:
        output.write(texts)

I think they problem lies in the formatting of the HTML of the page. On the pages that output the text incorrectly, the paragraphs are separated by <br> <p></p>, while on the pages that output correctly the paragraphs are contained within <p align="left" dir="ltr">. So, while I’ve identified this, I’m not sure how to make everything output consistently in the correct form.

Answer 1

The problem is that when you getting text() or string(), <br> tags are not converted to newline.

Workaround – replace <br> tags before doing XPath requests. Code:

response = response.replace(body=response.body.replace('<br />', '\n')) 
hxs = HtmlXPathSelector(response)

And let me give some advice, if you know, that there is only one node, you can use text() instead string():

date = hxs.select('//*[@id="date_long"]/text()').extract()[0]