I am creating a networking library in C# that I can use in any application, and as part of this library I have a TCP client/server setup. This setup works perfectly in almost every situation; it connects, sends/receives data, and disconnects flawlessly when under minimal and medium stress loads. However, when I send large amounts of data from the client to the server, the client socket works for a varied amount of time (sometimes short, sometimes long) and then just refuses to send data for a while. Specifically, my data rate goes from the 550-750 KBps range to 0 KBps, and sits there for again a varied amount of time. Then the socket will start sending again for a very short time, and get “throttled” again. During the throttling, i was assuming that the socket was disconnected because I couldn’t send anything, but Polling returns that the socket IS connected using this code:
public bool IsConnected(Socket socket)
{
try
{
return !(socket.Poll(1, SelectMode.SelectRead) && socket.Available == 0);
}
catch (SocketException) { return false; }
}
I just took a networking class at my college, so I started thinking about the congestion control and flow control mechanisms in TCP, but it seems to me that neither would cause this problem; congestion control only slows the data rate, and a full buffer on the receiver’s side wouldn’t last nearly the length of time I am getting a 0 KBps data rate. The symptom seems to point towards either some type of heavy data throttling or mass scale dropping of packets.
My question is this: does anyone have any idea what might be causing this data “throttling”, for lack of a better term? Also, is it possible that the packets I send are going further than just my router even though they are addressed to a host in the same subnet?
Edit: Just so it is clear, the reason I am trying to fix this problem is because I want to send files over TCP at the highest possible data rate. I understand that UDP can be used as well, and I will also be making a solution using it, but I want TCP to work first.
Specific Information:
I am using blocking read/write operations, and the server is multi-threaded. The client runs on its own thread as well. I am testing on my local subnet, bouncing all packets through my router, which should have a throughput of 54 Mbps. The packets are 8 KB each in size, and at maximum would be sent 1000 times a second (sending thread sleeps 1 ms), but obviously are not reaching that rate. Reducing the size of the packets so the data rate is lower causes the throttling to disappear. Windows 7 machines, 1 server, 1 client. The send operation always completes, it is the receive operation that errors out.
The send operation is below:
//get a copy of all the packets currently in the queue
IPacket[] toSend;
lock (packetQueues[c])
{
if (packetQueues[c].Count > SEND_MAX)
{
toSend = packetQueues[c].GetRange(0, SEND_MAX).ToArray();
packetQueues[c].RemoveRange(0, SEND_MAX);
}
else
{
toSend = packetQueues[c].ToArray();
packetQueues[c].RemoveRange(0, toSend.Length);
}
}
if (toSend != null && toSend.Length > 0)
{ //write the packets to the network stream
try
{
writer.Write(toSend.Length);
}
catch (Exception e)
{
Logger.Log(e);
if (showErrorMessages)
MessageBox.Show("Client " + (int)c + ": " + e, "Error", MessageBoxButtons.OK);
}
for (int i = 0; i < toSend.Length; i++)
{
try
{
toSend[i].Write(writer);
if (onSend != null)
{
object[] args = new object[2];
args[0] = c;
args[1] = toSend[i];
onSend(args);
}
}
catch (Exception e)
{
Logger.Log(e);
if (showErrorMessages)
MessageBox.Show("Client " + (int)c + ": " + e, "Error", MessageBoxButtons.OK);
}
}
}
And this is the receive code:
try
{
//default buffer size of a TcpClient is 8192 bytes, or 2048 characters
if (client.Available > 0)
{
int numPackets = reader.ReadInt32();
for (int i = 0; i < numPackets; i++)
{
readPacket.Clear();
readPacket.Read(reader);
if (owner != null)
{
owner.AcceptPacket(readPacket, c); //application handles null packets itself.
if (onReceive != null)
{
object[] args = new object[2];
args[0] = c;
args[1] = readPacket;
onReceive(args);
}
}
}
timestamps[c] = TimeManager.GetCurrentMilliseconds();
}
else
{
double now = TimeManager.GetCurrentMilliseconds();
if (now - timestamps[c] >= timeToDisconnect)
{ //if timestamp is old enough, check for connection.
connected[c] = IsConnected(client.Client);
if (!connected[c])
{
netStream.Close();
clients[c].Close();
numConnections--;
if (onTimeout != null) onTimeout(c);
}
else
{
timestamps[c] = now;
}
}
}
}
catch (Exception s)
{
Logger.Log(s);
if (showErrorMessages)
MessageBox.Show("Client " + (int)c + ": " + s, "Error", MessageBoxButtons.OK);
}
Packet send/receive:
public void Write(BinaryWriter w)
{
w.Write(command); //byte
w.Write(data.Type); //short
w.Write(data.Data.Length); //int
w.Write(data.Data); //byte array
w.Flush();
}
/// <summary>
/// Reads a command packet from data off a network stream.
/// </summary>
/// <param name="r">The stream reader.</param>
public void Read(BinaryReader r)
{
command = r.ReadByte();
short dataType = r.ReadInt16();
int dataSize = r.ReadInt32();
byte[] bytes = r.ReadBytes(dataSize);
data = new PortableObject(dataType, bytes);
}
Full Server Communication Loop:
public void Communicate(object cl)
{
int c = (int)cl;
timestamps[c] = TimeManager.GetCurrentMilliseconds();
try
{
//Console.Out.WriteLine("Thread " + Thread.CurrentThread.ManagedThreadId + " has started up. c = " + (int)c);
TcpClient client = clients[c];
client.ReceiveTimeout = 100;
NetworkStream netStream = client.GetStream();
BinaryReader reader = new BinaryReader(netStream);
BinaryWriter writer = new BinaryWriter(netStream);
while (client != null && connected[c])
{
#region Receive
try
{
//default buffer size of a TcpClient is 8192 bytes, or 2048 characters
if (client.Available > 0)
{
int numPackets = reader.ReadInt32();
for (int i = 0; i < numPackets; i++)
{
readPacket.Clear();
readPacket.Read(reader);
if (owner != null)
{
owner.AcceptPacket(readPacket, c); //application handles null packets itself.
if (onReceive != null)
{
object[] args = new object[2];
args[0] = c;
args[1] = readPacket;
onReceive(args);
}
}
}
timestamps[c] = TimeManager.GetCurrentMilliseconds();
}
else
{
double now = TimeManager.GetCurrentMilliseconds();
if (now - timestamps[c] >= timeToDisconnect)
{ //if timestamp is old enough, check for connection.
connected[c] = IsConnected(client.Client);
if (!connected[c])
{
netStream.Close();
clients[c].Close();
numConnections--;
if (onTimeout != null) onTimeout(c);
}
else
{
timestamps[c] = now;
}
}
}
}
catch (Exception s)
{
Logger.Log(s);
if (showErrorMessages)
MessageBox.Show("Client " + (int)c + ": " + s, "Error", MessageBoxButtons.OK);
}
#endregion
Thread.Sleep(threadLatency);
#region Send
//get a copy of all the packets currently in the queue
IPacket[] toSend;
lock (packetQueues[c])
{
if (packetQueues[c].Count > SEND_MAX)
{
toSend = packetQueues[c].GetRange(0, SEND_MAX).ToArray();
packetQueues[c].RemoveRange(0, SEND_MAX);
}
else
{
toSend = packetQueues[c].ToArray();
packetQueues[c].RemoveRange(0, toSend.Length);
}
}
if (toSend != null && toSend.Length > 0)
{ //write the packets to the network stream
try
{
writer.Write(toSend.Length);
}
catch (Exception e)
{
Logger.Log(e);
if (showErrorMessages)
MessageBox.Show("Client " + (int)c + ": " + e, "Error", MessageBoxButtons.OK);
}
for (int i = 0; i < toSend.Length; i++)
{
try
{
toSend[i].Write(writer);
if (onSend != null)
{
object[] args = new object[2];
args[0] = c;
args[1] = toSend[i];
onSend(args);
}
}
catch (Exception e)
{
Logger.Log(e);
if (showErrorMessages)
MessageBox.Show("Client " + (int)c + ": " + e, "Error", MessageBoxButtons.OK);
}
}
}
#endregion
}
}
catch (ThreadAbortException tae)
{
Logger.Log(tae);
MessageBox.Show("Thread " + (int)cl + " was aborted.", "Error", MessageBoxButtons.OK);
}
}
It is probably your code, but it’s difficult for us to say as it’s incomplete.
I wrote up my own set of best practices in a .NET TCP/IP FAQ – after many, many years of TCP/IP experience. I recommend you start with that.
P.S. I reserve the term “packet” for packets on-the-wire. A TCP app has no control over packets. I use the term “message” for application-protocol-level messages. I think this reduces confusion, especially for newcomers.