Home/ Questions/Q 8020699
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-04T21:45:38+00:00

I’m a little new to web crawlers and such, though I’ve been programming for

  • 0

I’m a little new to web crawlers and such, though I’ve been programming for a year already. So please bear with me as I try to explain my problem here.

I’m parsing info from Yahoo! News, and I’ve managed to get most of what I want, but there’s a little portion that has stumped me.

For example: http://news.yahoo.com/record-nm-blaze-test-forest-management-225730172.html

I want to get the numbers beside the thumbs up and thumbs down icons in the comments. When I use “Inspect Element” in my Chrome browser, I can clearly see the things that I have to look for – namely, an em tag under the div class ‘ugccmt-rate’. However, I’m not able to find this in my python program. In trying to track down the root of the problem, I clicked to view source of the page, and it seems that this tag is not there. Do you guys know how I should approach this problem? Does this have something to do with the javascript on the page that displays the info only after it runs? I’d appreciate some pointers in the right direction.

Thanks.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    The page is being generated via JavaScript.

    Check if there is a mobile version of the website first. If not, check for any APIs or RSS/Atom feeds. If there’s nothing else, you’ll either have to manually figure out what the JavaScript is loading and from where, or use Selenium to automate a browser that renders the JavaScript for you for parsing.

    • 0