Question

I’m sure this has been answered somewhere, because it’s a very basic question – I can not, however, for the life of me, find the answer on the web. I feel like a complete idiot, but I have to ask so, here goes:

I’m writing a python code that will produce a list of all page addresses on a domain. This is done using selenium 2 – my problem occurs when I try to access the list of all links produced by selenium.

Here’s what I have so far:

from selenium import webdriver
import time

HovedDomene = 'http://www.example.com'
Listlinker = []
Domenesider = []
Domenesider.append(HovedDomene)

driver = webdriver.Firefox()

for side in Domenesider:        

        driver.get(side)
        time.sleep(10)
        Listlinker = driver.find_elements_by_xpath("//a")

        for link in Listlinker: 

            if link in Domenesider:
              pass
            elif str(HovedDomene) in str(link):
              Domenesider.append(side)

print(Domenesider)
driver.close()

the Listlinker variable does not contain the links found on the page – instead the list contains, (I’m guessing here) selenium specific objects called WebElements. I can not, however, find any WebElement attributes that will give me the links – as a matter of fact I can’t find any examples of WebElement attributes being accessed in python (at least not in a manner i can reproduce)

I would really appreciate any help you all could give me

Sincerely
Rookie

Answer 1

I’m familiar with python’s api of selenium
but you probably can receive link using get_attribute(attributename) method. So it should be something like:

linkstr = ""
for link in Listlinker: 
  linkstr = link.get_attribute("href")

  if linkstr in Domenesider:
    pass
  elif str(HovedDomene) in linkstr:
    Domenesider.append(side)