Question

Hi i am working on scrapy

Below is my code

class examplespider(CrawlSpider):
    name = "example"
    domain_name = "www.example.com"
    start_urls = ["http://www.example.com/sch/mobile-/67939/i.html?_catref=1"]

def parse(self,response):
    hxs = HtmlXPathSelector(response)
    for i in xrange(1,10):
        yield Request(url="http://www.example.com/sch/mobile-/67939/i.html?_catref=1?_trkparms=65%253A12%257C66%253A1%257C39%253A1%257C72%253A3276&rt=nc&_catref=1&_dmpt=IN_Mobile_Phones&_trksid=p3286.c0.m14.l1513&_pgn=%d"%i,
                                 callback=self.parse_item)
def parse_item(self,response):
    print response,"Here it is................."

Result::

    File "/home/local/username/project/example/example/spiders/example_spider.py", line 117, in parse
    yield Request(url="http://www.example.com/sch/mobile-/67939/i.html?_catref=1?_trkparms=65%253A12%257C66%253A1%257C39%253A1%257C72%253A3276&rt=nc&_catref=1&_dmpt=IN_Mobile_Phones&_trksid=p3286.c0.m14.l1513&_pgn=%d"%i,
                                 callback=self.parse_item)
    exceptions.ValueError: unsupported format character 'A' (0x41) at index 61

Can any one tell me whats wrong is code and why unsupported format character code error is displaying, whether url is not supported here,also when we supply a a single integer without format specifier i can able to get response.
please let me know the above,

Thanks in advance.

Answer 1

Your URL contains many unescaped % symbols and you’re trying to interpolate a value into the string using the % operator. Python is looking for % characters to substitute your i into the URL. But the first likely one it sees is %A and that’s not a valid format character. The error message even tells you exactly what has happened and where.

Probably the easiest general solution is to use the .format() method of the string instead of the % operator:

("http://www.example.com/sch/mobile-/67939/i.html?_catref=1?"
 "_trkparms=65%253A12%257C66%253A1%257C39%253A1%257C72%253A3276&rt=nc"
 "&_catref=1&_dmpt=IN_Mobile_Phones&_trksid=p3286.c0.m14.l1513&_pgn={0}").format(i)

It uses different formatting placeholders that don’t conflict with your URL’s use of the % character.

Or, in this case specifically, since you just want to append your variable to the URL, you could use straight-up concatenation:

("http://www.example.com/sch/mobile-/67939/i.html?_catref=1?"
 "_trkparms=65%253A12%257C66%253A1%257C39%253A1%257C72%253A3276&rt=nc"
 "&_catref=1&_dmpt=IN_Mobile_Phones&_trksid=p3286.c0.m14.l1513&_pgn=") + str(i)