Question

If I run the openssl command line in hmac mode (as below), is the key used for the hmac used directly or is it hashed before using it as the key?

echo "foo" | openssl dgst -sha256 -binary -hmac "test" | openssl base64

Similarly, when encrypting a file with openssl (as below)is the pass phrase hashed with the salt? (If so how is it done? A pointer to the right source file would be even better.)

openssl enc -salt

Answer 1

The hmac option does not use salting or hashing; it just uses the passphrase directly as the key. See apps/dgst.c in the source distribution:

            else if (!strcmp(*argv,"-hmac"))
                    {
                    if (--argc < 1)
                            break;
                    hmac_key=*++argv;
                    }
    ...

    if (hmac_key)
            {
            sigkey = EVP_PKEY_new_mac_key(EVP_PKEY_HMAC, e,
                                    (unsigned char *)hmac_key, -1);
            if (!sigkey)
                    goto end;
            }

The enc command does seem to use some form of salting, at least in some cases. The relevant source file is apps/enc.c, but seems to come with some caveats:

            /* Note that str is NULL if a key was passed on the command
             * line, so we get no salt in that case. Is this a bug?
             */
            if (str != NULL)
                    {
                    /* Salt handling: if encrypting generate a salt and
                     * write to output BIO. If decrypting read salt from
                     * input BIO.
                     */

It then uses the function EVP_BytesToKey (in crypto/evp/evp_key.c) to generate a random key. This function seems to be a non-standard algorithm, which looked perhaps plausibly OK at a very brief glance but I couldn’t attest to it beyond that.

Source snippets and comments are all from the OpenSSL 1.0.0 release.