Question

I need to establish two channels between client and server, first is a UDP channel for data transmission and second is TCP channel for sending key and iv for AES-128 in UDP channel.

The TCP socket is created at server as follows:

listen_fd = socket (AF_INET, SOCK_STREAM, 0);
// sa_serv contains TCP port
error = bind(listen_fd, (struct sockaddr*) &sa_serv, sizeof (sa_serv));

The UDP socket is created at server as follows:

sock_fd = socket(AF_INET, SOCK_DGRAM, 0);
// local contains UDP port
error = bind(sock_fd, (struct sockaddr*) &local, sizeof(local));

Server needs to be able to connect to multiple clients, the TCP and UDP socket are used in select() as follows:

max = (listen_fd > sock_fd) : listen_fd : sock_fd;
fd_set set;
FD_ZERO(&set);
FD_SET(listen_fd, &set); FD_SET(sock_fd, &set);

while(1)
{
      select(max + 1, &set, NULL, NULL, NULL);

      if(FD_ISSET(listen_fd, &set){
          // server accepts connection
          // server receives key and IV over TCP connection
      }
      if(FD_ISSET(sock_fd, &set){
          // server receives encrypted data from client using UDP socket
      }
}

When the server receives the data in UDP socket, the server decrypts it using the key and IV received using TCP connection; the decryption code is as follows:

 int decrypt(unsigned char *plain, unsigned char *key, unsigned char *iv, unsigned char *cipher, int len)
{
   int i;

    unsigned char buf[3000];
    int outlen, tmplen;
    EVP_CIPHER_CTX ctx;
    EVP_CIPHER_CTX_init(&ctx);
    EVP_DecryptInit_ex(&ctx, EVP_aes_128_cbc(), NULL, key, iv);

    if(!EVP_DecryptUpdate(&ctx, buf, &outlen, cipher, len))
    {
            EVP_CIPHER_CTX_cleanup(&ctx);
            return 0;
    }

    if(!EVP_DecryptFinal_ex(&ctx, buf + outlen, &tmplen))
    {
            EVP_CIPHER_CTX_cleanup(&ctx);
            return 0;
    }

    outlen += tmplen;
    EVP_CIPHER_CTX_cleanup(&ctx);

    printf("\nLength decrypted :%d\n",outlen);

    printf("\nBuf: ");
    for(i=0; i<outlen; i++){
           plain[i] = outbuf[i];    
       printf(" %02x ",buf[i]);
    }
     printf("\n");

     return outlen;
 }

When a cipher received from client is passed to this function along with key and IV, the result plain text does not turn out to be correct (around 8 bytes are wrong). Now, one may argue that the cipher may be wrong, or the key or iv may have a problem; I verified all of them.

But the strange situation exists that the decryption code above deciphers correctly when I consider my server to be connected to only one client; when I don’t use my TCP socket in select() and use it outside (before) select() to accept connection & get key/iv from just one client (the code for accepting connection and receiving key/iv from client is exactly the same as when used inside select()), and in select() just use UDP socket to send/receive data; encrypted data received is correctly deciphered.

What I am not able to understand is that by putting the TCP socket in the select() fd_set, why the same decryption code is creating a problem, inspite of the fact that I get the correct cipher, key and IV.

Does anyone have an explanation for this?

Thanks.

Answer 1

Assuming exactly the first 8 bytes are wrong and the following bytes are correct, then you are using a different IV for decryption than you have used for encryption. When decrypting, the IV only affects the first decrypted block (the first 128 bits of plaintext).

Assuming bytes at the end are wrong: are you correctly taking message expansion into account? I.e. are you sending the full ciphertext to the other end, or are you passing only len(plaintext) bytes of ciphertext?

Additional points:

• In case you are re-using the same IV for multiple UDP packets: you should use a different IV for each message (each separately encrypted UDP packet).
• You might want to use a random IV and prepend it to the ciphertext you send over UDP instead of passing it via an out-of-band channel.
• TCP is not a good out-of-band channel because it is insecure.

Essentially, what you are doing is reinventing DTLS.