Question

We have a communication protocol that requires us to Base64 Encoded a SHA1 hash of a UTF-16 encoded password. We have been given Java, javascript, and visual basic examples however we are running under Linux (redhat)

the provided test string: TESTED@8691
the final output: rBbBKqbJodT5awZal/CSCYF/sFo=

I have tried

iconv_t conv = iconv_open("UTF-16LE","ASCII"); // open succeeds
char *from_string=strdup("TESTED@8691");
size_t from_length=strlen(from_string);
size_t to_length=from_length*3;
size_t original_to_length=to_length;

char *to_string=(char*)calloc(1,to_length);
int convert_return=iconv(conv,&from_string,&from_length,&to_string,&to_length);
// convert_return is 0 indicating success, to_length is 11, from_length is 0

run sha1 and base64 encoding on to_string with a length of 22
resulting output: GCXe7HMDoq/NRqo1WWYJDDYZzP0=

If I loop through to_string I get:

for (int i=0; i<original_to_length-to_length; ++i) {
   printf("to_string %d = %x",i,to_string[i]);  
}

output:
to_string 0 = 0
to_string 1 = 0
to_string 2 = 0
to_string 3 = 0
to_string 4 = 0
to_string 5 = 0
to_string 6 = 0
to_string 7 = 0
to_string 8 = 0
to_string 9 = 0
to_string 10 = 0
to_string 11 = 0
to_string 12 = 0
to_string 13 = 0
to_string 14 = 21
to_string 15 = 0
to_string 16 = 0
to_string 17 = 0
to_string 18 = 4
to_string 19 = 7e
to_string 20 = 13
to_string 21 = e

Here is the javascript conversion:

function str2rstr_utf16le(input)
{
   var output = "";
   for(var i = 0; i < input.length; i++)
    output += String.fromCharCode( input.charCodeAt(i) & 0xFF,
                              (input.charCodeAt(i) >>> 8) & 0xFF);

   return output;
}

What am I missing?
Thank You

Answer 1

I checked using a shell script and it seems the result you were given is indeed correct, as long as you assume UTF-16 to be UTF-16LE (Little Endian):

$ echo -e $(echo -n 'TESTED@8691' | iconv -f utf-8 -t utf-16le | sha1sum - | egrep -o '[0-9a-f]+' | sed -r 's/(..)/\\x\1/g') | tr -d '\n'  | base64
rBbBKqbJodT5awZal/CSCYF/sFo=

For Big-Endian, I get YrAwH9v3d88gjvsg0Hypu2Cfjc8= which is not your result, so I think endianness isn’t the issue here.

The man page for iconv(3) states:

The  iconv  function converts one multibyte character at a
time, and for  each  character  conversion  it  increments
*inbuf  and  decrements *inbytesleft by the number of con­
verted input bytes, it increments *outbuf  and  decrements
*outbytesleft by the number of converted output bytes, and
it updates the conversion state contained in cd.

This suggests that iconv modifies your target buffer pointer (to_string) – that’s why you pass it &to_string, not to_string itself. So, probably you need to subtract the number of bytes that were processed from to_string after iconv and before the further operations (SHA1 and BASE64).