Question

I am a bit puzzled by the “hidden” html code. If I use the “View Source” function, a section of the html source that I am trying to extract looks like this:

<div class="comments_with_more">
  <div id="comments_section">
  </div>
</div>

If I use the “Inspect Element” function in Firefox, it shows the additional layers that are hidden from the “View Source” option:

<div class="comments_with_more">
  <div id="comments_section">
    <div id="comments">
      (....what I am trying to get to.........)
    </div>
    <script>
  </div>
</div>

I am suspecting the script might be what’s messing this up. Does anyone know how to get to the hidden code in general or in Python?

Answer 1

you should use webkit‘s binding for this, here you can see it in action to do what you need (parse a page with javascript changes already loaded)