Question

I’m just starting out with BeautifulSoup and I’m trying to make a script that will go to a number very similar pages and then return all of the paragraphs underneath a section. Currently I have the following code

def BrightstormPageTest():
    soup = Soup(urllib.urlopen('http://brightstorm.com/science/chemistry/chemical-reaction-rates/collision-theory/').read())
    relevantTagText = ""
    for element in soup.findAll("section"):
            print element.nextSibling

Which works great for the first paragraph, but there are two sections that I want the siblings from and the first section will always have exactly one paragraph, while the second will have an undetermined number somewhere between 1 and 10 probably. Any ideas on how to do this?

Relevant html:

<section>
      <div class="page-header">
       <h2>
        Explanation
       </h2>
      </div>
     </section>
     <p>
      <strong>
       Collision theory
      </strong>
      is a model for explaining chemical reactions and reaction rates using the interactions of particles within the reactants. There are three important parts to
      <strong>
       collision theory
      </strong>
      , that reacting substances must collide, that they must collide with enough energy and that they must collide with the correct orientation. Increasing the kinetic energy of these particles or decreasing their volume increases the frequency of collisions and speeds a reaction.
     </p>
     <section>
      <div class="page-header">
       <h2>
        Transcript
       </h2>
      </div>
     </section>
     <p>
      Alright so we're going to talk about the collision theory. And the collision theory comes into play when you're talking about reactions and actually what happens in a reaction and how a reaction actually goes from the reactant all the way to the product. So the first thing we're going to have to discuss is, the fact that the reacting substances whatever we're dealing with the atoms, ions or molecules must collide in order for the reaction to occur. Okay that seems pretty obvious so we have our 2 reactants a and b and they must collide, and this is what we're going to call activated complex or a transition states that's going from, transitioning from the reactants towards the product and it's going to recreate this independent, very high energy activated complex and then yield our products, our 2ab. So the first postulate is that they must come together, okay that's easy enough.
     </p>
     <p>
      The second one says the reactant substances must collide with sufficient energy in order to form that activated complex. Because this activated complex is extremely high, very high in energy, very unstable so they must collide with a certain amount of energy to get to this point. If they don't collide with a good amount of energy then they're actually not going to react at all. So that energy is going to be called our activation energy to get to our activated complex. And you might see the symbol e with a subscript a to note that. And the last thing in the collision theory is that reacting substances must collide with the correct orientation so if they, made a collision at a range that wasn't great for them, they would actually rebound off of each other and not react at all.
     </p>
     <p>
      But if they if they did they have to make sure they line up correctly and then for the correct reaction to occur then they get their activated complex to form the products. And so these 3 things are the basis of that collision theory and how reactants go from reactants to the products.
     </p>

Would like to just get the things inside these paragraphs.

Answer 1

You need to iterate over the sections, then iterate over the paragraphs. I’ve modified your code to print the text of each paragraph for demonstration purposes.

from bs4 import BeautifulSoup as Soup

def BrightstormPageTest():
    soup = Soup(urllib.urlopen('http://brightstorm.com/science/chemistry/chemical-reaction-rates/collision-theory/').read())
    sections = soup.findAll("section")
    for section in sections:
        ps = section.findAll("p")
        for p in ps:
            print p.text

def BrightstormPageTest2():
    soup = Soup(urllib.urlopen('http://brightstorm.com/science/chemistry/chemical-reaction-rates/collision-theory/').read())
    sections = soup.findAll("section")
    for section in sections:
        while True:
             try:
                 print section.nextSibling.text
             except TypeError:
                 # .text is a valid method on a <p> element, but not a NavigableString.  
                 break