Home/ Questions/Q 3230560
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-17T16:56:29+00:00

As mentioned in a previous question, I’m coding a crawler for the QuakeLive website.

  • 0

As mentioned in a previous question, I’m coding a crawler for the QuakeLive website.
I’ve been using WWW::Mechanize to get the web content and this worked fine for all the pages except the one with matches. The problem is that I need to get all these kind of IDs:

<div id="ffa_c14065c8-d433-11df-a920-001a6433f796_50498929" class="areaMapC">

These are used to build specific matches URLs, but I simply can’t.

I managed to see those IDs only via FireBug and no page downloader, parser, getter I tried was able to help here. All I can get is a simpler version of the page which code is the one you can see by “showing source code” in Firefox.

Since FireBug shows the IDs I can safely assume they are already loaded, but then I can’t understand why nothing else gets them. It might have something to do with JavaScript.

You can find a page example HERE

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    To get at the DOM containing those IDs you’ll probably have to execute the javascript code on that site. I’m not aware of any libraries that’d allow you to do that, and then introspect the resulting DOM within perl, so just controlling an actual browser and later asking it for the DOM, or only parts of it, seems like a good way to go about this.

    Various browsers provide ways to be controlled programatically. With a Mozilla based browser, such as Firefox, this could be as easy as loading mozrepl into the browser, opening a socket from perl space, sending a few lines of javascript code over to actually load that page, and then some more javascript code to give you the parts of the DOM you’re interested in back. The result of that you could then parse with one of the many JSON modules on CPAN.

    Alternatively, you could work through the javascript code executed on your page and figure out what it actually does, to then mimic that in your crawler.

    • 0