Question

I am trying to scrape some content (am very new to Python) and I have hit a stumbling block. The code I am trying to scrape is:

<h2><a href="/best-sellers/sj-b9822.html">Spear & Jackson Predator Universal Hardpoint Saw     - 22"</a></h2>
<p><span class="productlist_mostwanted_rrp">    
Was: <span class="strikethrough">£12.52</span></span><span class="productlist_mostwanted_save">Save: £6.57(52%)</span></p>

<div class="clear"></div>

<p class="productlist_mostwanted_price">Now: £5.95</p>

What I am trying to scrape is the link text (Spear & Jackson etc) and the price (£5.95). I have looked about on Google, the BeautifulSoup documentation and on this forum and I managed to get to extract the “Now: £5.95” using this code:

for node in soup.findAll('p', { "class" : "productlist_grid_price" }):
     print ''.join(node.findAll(text=True))

However the result I am after is just 5.95. I have also had limited success trying to get the link text (Spear & Jackson) using:

soup.h2.a.contents[0]

However of course this returns just the first result.

The ultimate result that I am aiming for is to have the results look like:

Spear & Jackson Predator Universal Hardpoint Saw - 22 5.95
etc
etc

As I am looking to export this to a csv, I need to figure out how to put the data into 2 columns. Like I say I am very new to python so I hope this makes sense.

I appreciate any help!

Many thanks

Answer 1

I think what you’re looking for is something like this:

from BeautifulSoup import BeautifulSoup
import re

soup = BeautifulSoup(open('prueba.html').read())
item = re.sub('\s+', ' ', soup.h2.a.text)
price = soup.find('p', {'class': 'productlist_mostwanted_price'}).text
price = re.search('\d+\.\d+', price).group(0)

print item, price

Example output:

Spear & Jackson Predator Universal Hardpoint Saw – 22″ 5.95

Note that for the item, the regular expression is used just to remove extra whitespace, while for the price is used to capture the number.