I’m scraping a website in Flash, writing in Python. I can see in Firebug that the page loads its Flash file and then some background data in an .asmx file.
The background data is what I’m interested in – so how can I get hold of the .asmx file? I already know what it’s called.
I can’t get at the .asmx file directly, but can I grab it using Mechanize?
— UPDATE —-
The page I’m scraping is http://www.citroen.co.uk/new-cars/car-range/#/configurator/1C58AF/pop/pre-configuration/
The .asmx file is https://sfg-bpf.servicesgp.mpsa.com/uk/services/ServicePSAGF_Dealer.asmx – I can view it in Firebug.
I don’t believe so. The .asmx extension says that the resource you are accessing is a (SOAP-based) .NET Web service, written in a language such as C# or VB.NET. Normally the .asmx code would return a SOAP response, perhaps to be parsed by the Flash application. But it’s hard to see what’s going on without a little more detail – for example, whether the .asmx request is a separate Ajax request.
Update: The link to the Flash page doesn’t work for me now; it worked once, then subsequent requests get redirected to an error page.
The .asmx page you linked to just shows the entry point to the Web service; you’d have to make a request to a specific entry point with appropriate parameters to get the actual XML data (assuming of course that you’re authorised).