Question

I’m using RSA to encrypt communication between a server and a client. Lets say we have 2 Asymetric keys, key 1 and key2.

The server has key1 (Private) from the start and the client has the key1(public)

So here is the scenario:

1. the client generates key2
2. client connects to the server
3. sending key2(public) encrypted with key1(public)
4. from now on the server will send all data encrypted with the key2(public)
5. the client sends some random data to the server
6. the server sends back the same data hashed
7. the client verifies that the data is right

As far as I can see this should prevent a man-in-the-middle attack, or am I missing something? At point 7 the client should know if someone is trying to give the server the wrong key to encrypt with, as no one else but the server can decrypt key2(public).

If there is anything that can be done to improve the security please tell me.

Answer 1

The best thing you can do to improve the security is to use an existing design and not try to reinvent the wheel. I’m not saying that what you’ve done is necessarily wrong, but just that many people much smarter than you and me have spent a lot of time thinking about this problem. Use TLS instead.