Question

How can I create a non-http proxy with Twisted. Instead I would like to do it for the Terraria protocol which is made entirely of binary data. I see that they have a built-in proxy for HTTP connections, but this application needs to act more like an entry point which is forwarded to a set server (almost like a BNC on IRC).
I can’t figure out how to read the data off of one connection and send it to the other connection.

I have already tried using a socket for this task, but the blocking recv and send methods do not work well as two connections need to be live at the same time.

Answer 1

There are several different ways to create proxies in Twisted. The basic technique is built on peering, by taking two different protocols, on two different ports, and somehow gluing them together so that they can exchange data with each other.

The simplest proxy is a port-forwarder. Twisted ships with a port-forwarder implementation, see http://twistedmatrix.com/documents/current/api/twisted.protocols.portforward.html for the (underdocumented) classes ProxyClient and ProxyServer, although the actual source at http://twistedmatrix.com/trac/browser/tags/releases/twisted-11.0.0/twisted/protocols/portforward.py might be more useful to read through. From there, we can see the basic technique of proxying in Twisted:

def dataReceived(self, data):
    self.peer.transport.write(data)

When a proxying protocol receives data, it puts it out to the peer on the other side. That’s it! Quite simple. Of course, you’ll usually need some extra setup… Let’s look at a couple of proxies I’ve written before.

This is a proxy for Darklight, a little peer-to-peer system I wrote. It is talking to a backend server, and it wants to only proxy data if the data doesn’t match a predefined header. You can see that it uses ProxyClientFactory and endpoints (fancy ClientCreator, basically) to start proxying, and when it receives data, it has an opportunity to examine it before continuing, either to keep proxying or to switch protocols.

class DarkServerProtocol(Protocol):
    """
    Shim protocol for servers.
    """

    peer = None
    buf = ""

    def __init__(self, endpoint):
        self.endpoint = endpoint
        print "Protocol created..."

    def challenge(self, challenge):
        log.msg("Challenged: %s" % challenge)
        # ...omitted for brevity...
        return is_valid(challenge)

    def connectionMade(self):
        pcf = ProxyClientFactory()
        pcf.setServer(self)
        d = self.endpoint.connect(pcf)
        d.addErrback(lambda failure: self.transport.loseConnection())

        self.transport.pauseProducing()

    def setPeer(self, peer):
        # Our proxy passthrough has succeeded, so we will be seeing data
        # coming through shortly.
        log.msg("Established passthrough")
        self.peer = peer

    def dataReceived(self, data):
        self.buf += data

        # Examine whether we have received a challenge.
        if self.challenge(self.buf):
            # Excellent; change protocol.
            p = DarkAMP()
            p.factory = self.factory
            self.transport.protocol = p
            p.makeConnection(self.transport)
        elif self.peer:
            # Well, go ahead and send it through.
            self.peer.transport.write(data)

This is a rather complex chunk of code which takes two StatefulProtocols and glues them together rather forcefully. This is from a VNC proxy (https://code.osuosl.org/projects/twisted-vncauthproxy to be precise), which needs its protocols to do a lot of pre-authentication stuff before they are ready to be joined. This kind of proxy is the worst case; for speed, you don’t want to interact with the data going over the proxy, but you need to do some setup beforehand.

def start_proxying(result):
    """
    Callback to start proxies.
    """

    log.msg("Starting proxy")
    client_result, server_result = result
    success = True
    client_success, client = client_result
    server_success, server = server_result

    if not client_success:
        success = False
        log.err("Had issues on client side...")
        log.err(client)

    if not server_success:
        success = False
        log.err("Had issues on server side...")
        log.err(server)

    if not success:
        log.err("Had issues connecting, disconnecting both sides")
        if not isinstance(client, Failure):
            client.transport.loseConnection()
        if not isinstance(server, Failure):
            server.transport.loseConnection()
        return

    server.dataReceived = client.transport.write
    client.dataReceived = server.transport.write
    # Replay last bits of stuff in the pipe, if there's anything left.
    data = server._sful_data[1].read()
    if data:
        client.transport.write(data)
    data = client._sful_data[1].read()
    if data:
        server.transport.write(data)

    server.transport.resumeProducing()
    client.transport.resumeProducing()
    log.msg("Proxying started!")

So, now that I’ve explained that…

I also wrote Bravo. As in, http://www.bravoserver.org/. So I know a bit about Minecraft, and thus about Terraria. You will probably want to parse the packets coming through your proxy on both sides, so your actual proxying might start out looking like this, but it will quickly evolve as you begin to understand the data you’re proxying. Hopefully this is enough to get you started!