Question

I have a threaded python socket server that opens a new thread for each connection.

The thread is a very simple communication based on question and answer.
Basically client sends initial data transmission, server takes it run an external app that does stuff to the transmission and returns a reply that the server will send back and the loop will begin again until client disconnects.

Now because the client will be on a mobile phone thus an unstable connection I get left with open threads no longer connected and because the loop starts with recv it is rather difficult to break on lost connectivity this way.

I was thinking on adding a send before the recv to test if connection is still alive but this might not help at all if the client disconnects after my failsafe send as the client sends a data stream every 5 seconds only.

I noticed the recv will break sometimes but not always and in those cases I am left with zombie threads using resources.

Also this could be a solid vulnerability for my system to be DOSed.
I have looked through the python manual and Googled since thursday trying to find something for this but most things I find are related to client and non blocking mode.

Can anyone point me in the right direction towards a good way on fixing this issue?

Code samples:

Listener:

serversocket = socket(AF_INET, SOCK_STREAM)
serversocket.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
serversocket.bind(addr)
serversocket.listen(2)
logg("Binded to port: " + str(port))

# Listening Loop
while 1:
  clientsocket, clientaddr = serversocket.accept()
  threading.Thread(target=handler, args=(clientsocket, clientaddr,port,)).start()

# This is useless as it will never get here
serversocket.close()

Handler:

  # Socket connection handler (Threaded)
  def handler(clientsocket, clientaddr, port):
    clientsocket.settimeout(15)

    # Loop till client closes connection or connection drops
    while 1:
      stream = ''
      while 1:
        ending = stream[-6:] # get stream ending
        if ending == '.$$$$.':
          break

        try:
          data = clientsocket.recv(1)
        except:
          sys.exit()

        if not data:
          sys.exit()
          # this is the usual point where thread is closed when a client closes connection normally

        stream += data

      # Clear the line ending
      stream = base64.b64encode(stream[:-6])

      # Send data to be processed
      re = getreply(stream)

      # Send response to client
      try:
        clientsocket.send(re + str('.$$$$.'))
      except:
        sys.exit()

As you can see there are three conditions that at least one should trigger exit if connection fails but sometimes they do not.

Answer 1

Sorry, but I think that threaded idea in this case is not good. As you do not need to process/do a lot of stuff in these threads (workers?) and most of the time these threads are waiting for socket (is the blocking operation, isn’t it?) I would advice to read about event-driven programming. According to sockets this pattern is extremly useful, becouse you can do all stuff in one thread. You are communicate with one socket at a time, but the rest of connections are just waiting to data so there is almost no loss. When you send several bytes you just check that maybe another connection requires carrying. You can read about select
and epoll.

In python there is several libraries to play with this nicly:

• libev (c library wrapper) – pyev
• tornado
• twisted

I used tornado in some projects and it is done this task very good. Libev is nice also, but is a c-wrapper so it is a little bit low-level (but very nice for some tasks).