Question

I’m not sure if this is a known issue that I am running into, but I couldn’t find a good search string that would give me any useful results.
Anyway, here’s the basic rundown:

we’ve got a relatively simple application that takes data from a source (DB or file) and streams that data over TCP to connected clients as new data comes in. its a relatively low number of clients; i would say at max 10 clients per server, so we have the following rough design:

client: connect to server, set to read (with timeout set to higher than the server heartbeat message frequency). It blocks on read.

server: one listening thread that accepts connections and then spawns a writer thread to read from the data source and write to the client. The writer thread is also detached(using boost::thread so just call the .detach() function). It blocks on writes indefinetly, but does check errno for errors before writing. We start the servers using a single perl script and calling “fork” for each server process.

The problem(s):
at seemingly random times, the client will shutdown with a “connection terminated (SUCCESFUL)” indicating that the remote server shutdown the socket on purpose. However, when this happens the SERVER application ALSO closes, without any errors or anything. it just crashes.

Now, to further the problem, we have multiple instances of the server app being started by a startup script running different files and different ports. When ONE of the servers crashes like this, ALL the servers crash out.

Both the server and client using the same “Connection” library created in-house. It’s mostly a C++ wrapper for the C socket calls.

here’s some rough code for the write and read function in the Connection libary:

int connectionTimeout_read = 60 * 60 * 1000; 
int Socket::readUntil(char* buf, int amount) const
    {
        int readyFds = epoll_wait(epfd,epEvents,1,connectionTimeout_read);
        if(readyFds < 0)
        {
            status = convertFlagToStatus(errno);
            return 0;
        }
        if(readyFds == 0)
        {
            status = CONNECTION_TIMEOUT;
            return 0;
        }
        int fd = epEvents[0].data.fd;
        if( fd != socket)
        {
            status = CONNECTION_INCORRECT_SOCKET;
            return 0;
        }
        int rec = recv(fd,buf,amount,MSG_WAITALL);

        if(rec == 0)
            status = CONNECTION_CLOSED;
        else if(rec < 0)
            status = convertFlagToStatus(errno);
        else
            status = CONNECTION_NORMAL;
        lastReadBytes = rec;
        return rec;

    }



int Socket::write(const void* buf, int size) const
    {

        int readyFds = epoll_wait(epfd,epEvents,1,-1);
        if(readyFds < 0)
        {
            status = convertFlagToStatus(errno);
            return 0;
        }
        if(readyFds == 0)
        {
            status = CONNECTION_TERMINATED;
            return 0;
        }
        int fd = epEvents[0].data.fd;
        if(fd != socket)
        {
            status = CONNECTION_INCORRECT_SOCKET;
            return 0;
        }
        if(epEvents[0].events != EPOLLOUT)
        {
            status = CONNECTION_CLOSED;
            return 0;
        }
        int bytesWrote = ::send(socket, buf, size,0);
        if(bytesWrote < 0)
            status = convertFlagToStatus(errno);
        lastWriteBytes = bytesWrote;
        return bytesWrote;

    }

Any help solving this mystery bug would be great! at the VERY least, I would like it to NOT crash out the server even if the client crashes (which is really strange for me, since there is no two-way communication).

Also, for reference, here is the server listening code:

while(server.getStatus() == connection::CONNECTION_NORMAL)
        {
            connection::Socket s = server.listen();
                if(s.getStatus() != connection::CONNECTION_NORMAL)
                {
                    fprintf(stdout,"failed to accept a socket. error: %s\n",connection::getStatusString(s.getStatus()));
                }

                DATASOURCE* dataSource;
                dataSource = open_datasource(XXXX);  /* edited */               if(dataSource == NULL)
                {
                    fprintf(stdout,"FATAL ERROR. DATASOURCE NOT FOUND\n");
                    return;
                }
                    boost::thread fileSender(Sender(s,dataSource));
                    fileSender.detach();


        }

…And also here is the spawned child sending thread:

::signal(SIGPIPE,SIG_IGN);

    //const int headerNeeds = 29;
    const int BUFFERSIZE = 2000;
    char buf[BUFFERSIZE];

    bool running = true;
    while(running)
    {
             memset(buf,'\0',BUFFERSIZE*sizeof(char));
        unsigned int readBytes = 0;
        while((readBytes = read_datasource(buf,sizeof(unsigned char),BUFFERSIZE,dataSource)) == 0)
        {
            boost::this_thread::sleep(boost::posix_time::milliseconds(1000));
        }
        socket.write(buf,readBytes);
        if(socket.getStatus() != connection::CONNECTION_NORMAL)
            running = false;

    }
    fprintf(stdout,"socket error: %s\n",connection::getStatusString(socket.getStatus()));
    socket.close();
    fprintf(stdout,"sender exiting...\n");

Any insights would be welcome! Thanks in advance.

Answer 1

You’ve probably got everything backwards… when the server crashes, the OS will close all sockets. So the server crash happens first and causes the client to get the disconnect message (FIN flag in a TCP segment, actually), the crash is not a result of the socket closing.

Since you have multiple server processes crashing at the same time, I’d look at resources they share, and also any scheduled tasks that all servers would try to execute at the same time.

EDIT: You don’t have a single client connecting to multiple servers, do you? Note that TCP connections are always bidirectional, so the server process does get feedback if a client disconnects. Some internet providers have even been caught generating RST packets on connections that fail some test for suspicious traffic.

Write a signal handler. Make sure it uses only raw I/O functions to log problems (open, write, close, not fwrite, not printf).

Check return values. Check for negative return value from write on a socket, but check all return values.