Question

I’m new here, and quite new to web development in general. My background is in 3D modeling and design, but I recently started a project that I think could be a nice resource for the 3D community.

I’ve got the page mostly designed and coded here: The Top Row, but I’m just about at the end of my knowledge. The upper section, and lower (artist spotlight) section will be manually updated, so I’m not worried about those.

The portion I’m having trouble with is the middle section under the “Best of the Rest” heading. What I want to do is scrape images (and links) from seven prominent CG forums and display them in the content areas I have laid out. Each of the forums have a section at the top of their page that displays five or six featured images.

If you look at CGSociety, for example: They have a top row with six featured pieces. I want to take the three newest and display them in my CGSociety content box with links to the original threads. It’s important I get the links too, since the whole point of the website is to generate exposure for artists that deserve it.

The images are always in the same locations and always have a predictable path all the way up to the image name:

i.e: http://features.cgsociety.org/cgtalk/plugs/"featured image".jpg

I don’t know if it’s relevant, but the xpath for the images is reliable too. For CGSociety, the image is basically determined by the number contained in the final set of brackets.

/x:html/x:body/x:div[4]/x:div/x:div/x:table[1]/x:tbody/x:tr/x:td[1]/x:a/x:img

I’ve read so many different stack overflow threads, but so much of it is going over my head. I don’t have much programming experience, but I suspect what I’m trying to do isn’t really all that complicated.

So here are my main questions:

1. What’s the best (easiest) method for this kind of scraping? I keep seeing Python with Beautiful Soup or lxml mentioned, but someone else recommended PHP with cURL and xPath.

2. Is there a particular method that will put the least possible strain on the source forums? These forums all have membership in the tens (or hundreds) of thousands, so this probably isn’t a huge concern, but I’d love to do this without directly hotlinking if possible.

3. Am I even headed in the right direction?

Also: I know scraping is a legal grey-area. I plan on asking permission from each of the forums involved, but I want to have a working model to show them when I ask.

Any help will be very very much appreciated. I think this could be a cool site if I can get it working.

Answer 1

I refreshed my lxml knowledge a bit and wrote you some code that scrapes what you wanted from that page:

import lxml.html

images = []

html = lxml.html.parse("http://forums.cgsociety.org/")
table = html.xpath("//div[@class='page']/div[1]/table[1]")[0]

for cell in table.iterfind(".//td"):
    image = {}
    image['img_url'] = cell.find('a/img').get('src')
    image['link_url'] = cell.find('a').get('href')
    images.append(image)

images now contains:

[{'img_url': 'http://features.cgsociety.org/cgtalk/plugs/meind_p.jpg',
  'link_url': 'http://forums.cgsociety.org/showthread.php?s=&threadid=975814&utm_medium=plugblock&utm_source=cgtalk'},
 {'img_url': 'http://features.cgsociety.org/cgtalk/plugs/plugimg.jpg',
  'link_url': 'http://forums.cgsociety.org/showthread.php?s=&threadid=975032&utm_medium=plugblock&utm_source=cgtalk'},
 {'img_url': 'http://features.cgsociety.org/cgtalk/plugs/cg_portfolio_elmoooo.jpg',
  'link_url': 'http://elmoooo.cgsociety.org/gallery/?z=0&utm_medium=plugblock&utm_source=cgtalk'},
 {'img_url': 'http://features.cgsociety.org/cgtalk/plugs/suck_p.jpg',
  'link_url': 'http://forums.cgsociety.org/showthread.php?s=&threadid=973971&utm_medium=plugblock&utm_source=cgtalk'},
 {'img_url': 'http://features.cgsociety.org/cgtalk/plugs/cry_p.jpg',
  'link_url': 'http://forums.cgsociety.org/showthread.php?s=&threadid=972537&utm_medium=plugblock&utm_source=cgtalk'},
 {'img_url': 'http://features.cgsociety.org/cgtalk/plugs/gerrard_p.jpg',
  'link_url': 'http://forums.cgsociety.org/showthread.php?s=&threadid=972012&utm_medium=plugblock&utm_source=cgtalk'}]

Feel free to send me an email (you can find it on my profile) if you’d like some more help.