Question

I am considering starting a project so that I can learn more and keep the things I have learned thus far from getting rusty.

A lot of the project will be new things so I thought I would come here and ask for advice on what to do and how to go about doing it.

I enjoy photoshop and toying around with it, so I thought I would mix my project with something like that. So I decided my program will do something along the lines of grab new resources for photoshop put them in their own folder on my computer. (from deviantart for now)

For now I want to focus on a page like this:

http://browse.deviantart.com/resources/applications/psbrushes/?order=9    

I’m not fluent with understanding exactly what is going on in the html source so it is a bit hard to see what is going on.

But lets say I am on that page and I have the following options chosen:

Sorted by Popular
Sorted by All Time 
Sorted by 24 Items Per Page

My goal is to individually go to each thumbnail and grab the following:

The Author
The Title
The Description
Download the File (create folder based on title name)
Download the Image (place in folder with the file above)
Create text file with the author, title, and description in it

I would like to do that for each of the 24 items on the page and then go to the next page and do the same. (I am thinking of just going through the first five pages as I don’t have too much interest in trying out brushes that aren’t too popular)

So, I’m posting this for a sense of direction and perhaps some help on how to parse such a page to get what I’m looking for. I’m sure this project will keep me busy for awhile, but I’m hoping it will become useful in teaching me things.

Any help and suggestions are always appreciated.

.

.

EDIT

Each page is made up of 24 of these:

<div class="tt-a" usericon="http://a.deviantart.net/avatars/s/h/shad0w-gfx.gif" collect_rid="1:19982524">
 <span class="shad0w" style="background-image: url ("http://sh.deviantart.net/shad0w/x/107/150/logo3.png");">
  <a class="t" title="Shad0ws Blood Brush Set by ~Shad0w-GFX, Jun 28, 2005" href="http://Shad0w-GFX.deviantart.com/art/Shad0ws-Blood-Brush-Set-19982524?q=boost%3Apopular+in%3Aresources%2Fapplications%2Fpsbrushes&qo-0">Shad0ws Blood Brush Set</a>

My assumption is, I want to grab all my information from the:

<a class="t" ... >

Since it contains the title, author, and link to where the download url and large image is located.

If this sounds correct, how would one go about getting that info for each object on the page. (24 per page) I would assume by using CyberNeko. I’m just not exactly sure how to get to the proper level where is located and for each of them on the page

.

.

EDIT #2

I have some test code that looks like this:

divs = []
client = new WebClient(BrowserVersion.FIREFOX_3)
client.javaScriptEnabled = false

page = client.getPage("http://browse.deviantart.com/resources/applications/psbrushes/?order=9&offset=0")

divs = page.getByXPath("//html/body/div[2]/div/div/table/tbody/tr/td[2]/div/div[5]/div/div[2]/span/a[@class='t']")

divs.each { println it }

The XPath is correct, but it prints out:

<?xml version="1.0" encoding="UTF-8"?><a href="http://Shad0w-GFX.deviantart.com/

art/Shad0ws-Blood-Brush-Set-19982524?q=boost%3Apopular+in%3Aresources%2Fapplicat
ions%2Fpsbrushes&qo=0″ class=”t” title=”Shad0ws Blood Brush Set by ~Shad0w-G
FX, Jun 28, 2005″>Shad0ws Blood Brush Set

Can you explain what I need to do to just get the href out of there? Is there a simple way to do it with HtmlUnit?

Answer 1

Meeting the requirements you’ve listed above is actually pretty easy. You can probably do it with a simple Groovy script of about 50 lines. Here’s how I would go about it:

The URL of the first page is
http://browse.deviantart.com/resources/applications/psbrushes/?order=9&offset=0

To get the next page, simply increase the value of the offset parameter by 24:
http://browse.deviantart.com/resources/applications/psbrushes/?order=9&offset=24

So now you know how to construct the URLs for the pages you need to work with. To download the content of this page use:

def pageUrl = 'http://browse.deviantart.com/resources/applications/psbrushes/?order=9&offset=0'

// get the content as a byte array
byte[] pageContent = new URL(pageUrl).bytes

// or get the content as a String
String pageContentAsString = new URL(pageUrl).text

Now all you need to do is parse out the elements of the content that you’re interested in as save it in files. For the parsing, you should use a HTML parser like CyberNeko or Jericho.