Question

I’m trying to create one transparent HTTP proxy. It’s purpose is to stay between the browser and the web server and be invisible. Here is the code I’m using. Unfortunately it’s not working very well. When I open the web page (referenced by $base_uri) there are different results depending on that whether I’ve opened it using the browser only or the browser and the proxy. I’m trying it on a web site which is returning all kinds of responses including “transfer-encoding: chunked” (so I guess may be the problem could be there?!). I think that there could be also problems with the cookies but I don’t know how to solve them (if any…).

#!/usr/bin/perl

use strict;

use HTTP::Daemon;
use LWP::UserAgent;
use HTTP::Cookies;

my $cookie_jar = HTTP::Cookies->new();

my $ua = LWP::UserAgent->new( max_redirect => 0, env_proxy => 0,keep_alive => 1, timeout => 30, agent => "Mozilla/4.76 [en] (Win98; U)");
my $d = HTTP::Daemon->new( 
    LocalHost => "localhost",  # remove this to listen from other machines 
                   # (i.e. open-relay... be careful of spammers!)
    LocalPort => 33331
) || die;
print "[Proxy URL:", $d->url, "]\n";

fork(); fork(); fork();  # 2^3 = 8 processes

$ua->cookie_jar($cookie_jar);

my $base_uri = 'http://example.com/';

while (my $c = $d->accept) {
        while (my $request = $c->get_request) {
            my $uri = $base_uri . $request->uri->as_string();
            my $method = $request->method;
            my $req = HTTP::Request->new($method, $uri);

            $request->uri($uri);

            print "[[ $method >> $uri ]]\n";
            my $response = $ua->simple_request($request);
            $c->send_response( $response );
        }
    $c->close;
    undef($c);
}

Thank you in advance!

Answer 1

It is not clear, what you really want. You should describe it much better. If you describe what and why you need that proxy and what features it needs, any help can be much better. Nevertheless I’ll try.

What you currently do is to take an incoming connection, extract the URI and the call method and pass it to your source. Well, HTTP is much more complex – you strip everything like the transported data (e.g. for POST requests) as well as all the header lines (cookies, login data, browser identification, language specs, …), which usually carry important information. Also you modify the timing behavior a lot. Then you sent it to your proxy target.

Now you take the server answer and again strip everything relevant. Also you only reply the answer after it is finished. For streaming data this will not work (you already mentioned the chunked transfer mode). Also your method requires a lot of memory for large files (e.g. a DVD image – 4GB).

Without further details about your application no real suggestion is possible, but some ideas:

a) As told in the comments there are Perl modules available. You may test them and see if they fit your needs.

b) You can go down a level. Use IO::Socket::INET or INET6 and directly work on the socket level. Send each packet as it comes in directly to the output. This is nearly 100% transparent (except for IP address and probably the TCP packet sizes). Thought if you want to change or inspect data, you directly need to care for the HTTP yourself, which can be really complicated nowadays (especially due to transfer encoding).

c) Maybe don’t code yourself, but use an existing proxy (e.g. the Apache webserver or specific proxy programs).