Question

I have a circumstance where my server may close TCPServer and restart, saving all the users to a file, and immediately reloading them; their connections do not sever.

The problem is I can’t seem to reinitialize their streams.

When we restart (and attempt to maintain connections), I reinitialize TCPServer, and load my array of connected users – Since these each have an existing socket address, stored as <TCPSocket:0x00000000000000>, can I reinitialize these addresses with TCPServer?

Normally, each user connects and is accepted:

$nCS = TCPServer.new(HOST, PORT)

begin
  while socket = $nCS.accept
    Thread.new( socket ) do |sock|
      begin
        d = User.new(sock)
        while sock.gets
          szIn = $_.chomp
          DBG( "Received '" + szIn + "' from Client " + sock.to_s )
          d.parseInput( szIn )
        end
      rescue => e
        $stdout.puts "ERROR: Caught error in Client Thread: #{e} \r\n #{e.backtrace.to_s.gsub(",", ",\r\n")}"
        sock.write("Sorry, an error has occurred, and you have been disconnected."+EOL+"Please try again later."+EOL)
        d.closeConnection
      end
    end
  end
rescue => e
  $stdout.puts "ERROR: Caught error in Server Thread: #{e} \r\n #{e.backtrace.to_s.gsub(",", ",\r\n")}"
  exit
end

To give it a command to hot reboot, we use exec('./main --copyover') to flag that a copy over is occurring.

If $connected holds an array of all users, and each user has a socket, how do I reinitialize the socket that was open before the restart (assuming the other end is still connected)?

I suspect that using exec("./main", "--copyover", *$nCS, *$connected) is getting me closer, since this simply replaces the process, and should maintain the files (not close them).

Answer 1

How to Hot-Reboot a TCPServer in Ruby

Hot-Rebooting (aka Copyover) is a process by which an administrator can reload the application (along with any new changes made since last boot) without losing the client connections. This is useful in managing customer expectations as the application does not need to suffer severe downtime and disruption if in use.

What I propose below may not be the best practice, but it’s functioning and perhaps will guide others to a similar solution.

The Command

I use a particular style of coding that makes use of command tables to find functions and their accessibility. All command functions are prefixed with cmd. I’ll clean up the miscellany to improve readability:

def cmdCopyover
  #$nCS is the TCPServer object
  #$connected holds an array of all users sockets
  #--copyover flags that this is a hot reboot.
  connected_args = $connected.map do |sock|
    sock.close_on_exec = false if sock.respond_to?(:close_on_exec=)
    sock.fileno.to_s
  end.join(",")
  exec('./main.rb', '--copyover', $nCS.fileno.to_s, connected_args)
end

What we’re passing are strings; $nCS.fileno.to_s provides us the file descriptor of the main TCPServer object, while connected_args is a comma-delineated list of file descriptors for each user connected. When we restart, ARGV will be an array holding each argument:

• ARGV[0] == "--copyover"
• ARGV[1] == "5" (Or whatever the file descriptor for TCPServer was)
• ARGV[2] == "6,7,8,9" (Example, assuming 4 connected users)

What To Expect When You’re Expecting (a Copyover)

Under normal circumstances, we may have a basic server (in main.rb that looks something like this:

puts "Starting Server"
$connected = Array.new
$nCS = TCPServer.new("127.0.0.1",9999)

begin
  while socket = $nCS.accept
    # NB: Move this loop to its own function, threadLoop()
    Thread.new( socket ) do |sock|
      begin
        while sock.gets
          szIn = $_.chomp
          #do something with input.
        end
      rescue => e
        puts "ERROR: Caught error in Client Thread: #{e}"
        puts #{e.backtrace.to_s.gsub(",", ",\r\n")}"
        sock.write("Sorry, an error has occurred, and you have been disconnected."+EOL+"Please try again later."+EOL)
        sock.close
      end
    end
  end
rescue => e
  puts "Error: Caught Error in Server Thread: #{e}"
  puts "#{e.backtrace.to_s.gsub(",", ",\r\n")}"
  exit
end

We want to move that main loop to its own function to make it accessible — our reconnecting users will need to be reinserted in the loop.

So let’s get main.rb ready for accepting a hot reboot:

def threadLoop( socket )
  Thread.new( socket ) do |sock|
    begin
      while sock.gets
        szIn = $_.chomp
        #do something with input.
      end
    rescue => e
      puts "ERROR: Caught error in Client Thread: #{e}"
      puts #{e.backtrace.to_s.gsub(",", ",\r\n")}"
      sock.write("Sorry, an error has occurred, and you have been disconnected."+EOL+"Please try again later."+EOL)
      sock.close
    end
  end
end

puts "Starting Server"
$connected = Array.new
if ARGV[0] == '--copyover'
  $nCS = TCPServer.for_fd( ARGV[1].to_i )
  $nCS.close_on_exec = false if $nCS.respond_to?(:close_on_exec=)
  connected_args = ARGV[2]
  connected_args.split(/,/).map do |sockfd|
  $connected << sockfd

  $connected.each {|c| threadLoop( c ) }
else
  $nCS = TCPServer.new("127.0.0.1",9999)
  $nCS.close_on_exec = false if $nCS.respond_to?(:close_on_exec=)
end

begin
  while socket = $nCS.accept
    threadLoop( socket )
  end
rescue => e
  puts "Error: Caught Error in Server Thread: #{e}"
  puts "#{e.backtrace.to_s.gsub(",", ",\r\n")}"
  exit
end

Caveat

My actual usage was a lot more ridiculously complicated, so I did my best to strip out all the garbage; however, I was realizing when I got the end here that you could probably do without $connected (it’s a part of a larger system for me). There may be some errors, so please comment if you find them and I’ll correct.

Hope this helps anyone who finds it.