I am currently building a fast paced multi-player game where security is a big concern.
I have implemented stop-and-wait (a temporary measure, for simplicity’s sake) over UDP in Java.
However, I’m concerned that a malicious party could send false acks or messages to the sender or receiver respectively, making the sequence numbers go out of sync — essentially causing either messages to be lost (crashing the game!) or the sender to resend indefinitely.
Further up in my application, each participant is identified by an RSA public key, so I could add a signed hash to each message for authentication purposes… but this greatly increases header size and muddles my architecture layers somewhat.
Is this a common approach? Are there good known solutions to this problem?
What Jonathan said. Generally speaking, you need to provide your own means of defending against out of order or dropped packets while using UDP. There isn’t a “standard” way of dealing with this problem — other than using TCP instead. You can, as you said, validate the client’s identity on every packet — but keep in mind validating the client’s identity, even with something like RSA — doesn’t help you against malicious clients. They can just connect to the game server, validate themselves, and start sending garbage. After all, they are who they say they are.
For most game server designs, to my (albeit limited) understanding, usually the server has the notion of a “tick” — whereby clients can register that some events happen within the tick, and the server enforces the rules against what the clients send. But when the tick ends, any events received by the server corresponding to that tick are discarded. This adds a race element which does not increase actual security of the system, but does add an element which makes it more difficult.
Of course making any system like this “secure” in that a client couldn’t spoof their origin would essentially require doing a TLS tunnel, which I would assume won’t meet your performance requirements.