Question

I’m using BeautifulSoup (BS4) to build a scraper tool that will allow me to pull the product name from any TopShop.com product page, which sits between ‘h1’ tags. Can’t figure out why the code I’ve written isn’t working!

from urllib2 import urlopen
from bs4 import BeautifulSoup
import re

TopShop_URL = raw_input("Enter a TopShop Product URL")
ProductPage = urlopen(TopShop_URL).read()

soup = BeautifulSoup(ProductPage)

ProductNames = soup.find_all('h1')

print ProductNames

Answer 1

I get this working using requests (http://docs.python-requests.org/en/latest/)

from bs4 import BeautifulSoup
import requests

content = requests.get("TOPShop_URL").content
soup = BeautifulSoup(content)
product_names = soup.findAll("h1")
print product_names