Question

I have written a spider that crawls through a folder named fid and extracts the names of all the sub-folders as a link. Now the problem is that each of these sub-folders have an html page inside them and i want to extract the names of all these html files and add to the current “start_urls”, so that i can scrape out required information from all these html pages. I have tried:

os.listdir()
glob.glob()

but none of these worked. Please help me with this.

Answer 1

One stdlib approach is using os.walk in combination with fnmatch:

import fnmatch
import os

start_urls = []

for root, dirnames, filenames in os.walk('/start/dir/'):
    for filename in fnmatch.filter(filenames, '*.html'):
        start_urls.append(os.path.join(root, filename))