Question

I am creating a google scraper in Adobe AIR using the Flex 4 framework.
I have run into a brick wall: Google forces a captcha after around 10 pages are read.

Can anyone tell me how to get the page through a proxy server?

I am using HTTPService
Here’s my code:

service=new HTTPService();
service.addEventListener(ResultEvent.RESULT, googleResult);
service.addEventListener(FaultEvent.FAULT, googleFault);
service.resultFormat="text";
service.url=_googleURL+keyPhrase.text
service.send();

Cheers,

Answer 1

Solution:

I created a ProxyHTTPService class which extends HTTPService

package com.pageone.proxyserv {

    import mx.rpc.AsyncToken;
    import mx.rpc.http.mxml.HTTPService;
    import mx.utils.URLUtil;

    public class ProxyHTTPService extends HTTPService {
        private var _finalURL:String;

        private var _tempURL:String;

        private var _proxy:Object;

        private var phpProxyURL:String="http://myserver/proxy.php";

        public function ProxyHTTPService(rootURL:String="") {
            super();

        }

        public function get proxy():Object
        {
            return _proxy;
        }

        public function set proxy(value:Object):void
        {
            _proxy = value;
        }


        public function get finalURL():String {
            return _finalURL;
        }

        public function set finalURL(value:String):void {
            _finalURL=value;
        }

        override public function send(parameters:Object=null):AsyncToken {
            this.url=phpProxyURL;

            var proxyargs:Object=new Object();
            proxyargs.proxy=_proxy.ip + ":" + _proxy.port;

            _tempURL=_finalURL;
            var params:String=URLUtil.objectToString(parameters, "&");;
            if(_finalURL.indexOf("?") > 0) {
                _tempURL += "&" + params;
            } else {
                _tempURL += "?" + params;
            }
            _tempURL=encodeURI(_tempURL);
            _tempURL=replaceAll(_tempURL, "%253A", ":");
            _tempURL=replaceAll(_tempURL, "%252F", "/");

            proxyargs.url=_tempURL;

            return super.send(proxyargs);
        }

        private function replaceAll(string:String, find:String, replace:String):String {
            return string.split(find).join(replace);
        }
    }
}

Then I created a php page on the server

<?php

$url = $_GET["url"] or die("require url parameter");
$proxyuri = $_GET["proxy"] or die("require proxy parameter");
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 0);
curl_setopt($ch, CURLOPT_PROXY, $proxyuri);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 GTB7.1');

curl_setopt($ch, CURLOPT_RETURNTRANSFER, 0);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 2);
curl_setopt($ch, CURLOPT_TIMEOUT, 60);
curl_setopt ($ch, CURLOPT_HEADER, 0);
$exec=curl_exec ($ch);

curl_close($ch);


?>

Now in ActionScript, you can call the ProxyHTTPService like this:

var p:ProxyHTTPService=new ProxyHTTPService;
p.addEventListenet(ResultEvent.RESULT, resultListener);
p.addEventListenet(FaultEvent.FAULT, faultListener);
p.finalURL="http://www.google.com/search";
p.proxy={ip: "xxx.xxx.xxx.xxx", port:8080};
p.send({q: "StackOverflow"});