Today I was doing some leisurely reading and stumbled upon Section 5.8 (on page 45) of Recommendation for Pair-Wise Key Establishment Schemes Using Discrete Logarithm Cryptography (Revised) (NIST Special Publication 800-56A). I was very confused by this:
An Approved key derivation function
(KDF) shall be used to derive secret
keying material from a shared secret.
The output from a KDF shall only be
used for secret keying material, such
as a symmetric key used for data
encryption or message integrity, a
secret initialization vector, or a
master key that will be used to
generate other keys (possibly using a
different process). Nonsecret keying
material (such as a non-secret
initialization vector) shall not be
generated using the shared secret.
Now I’m no Alan Turing, but I thought that initialization vectors need not be kept secret. Under what circumstances would one want a “secret initialization vector?”
Thomas Pornin says that IVs are public and he seems well-versed in cryptography. Likewise with caf.
An initialization vector needs not be secret (it is not a key) but it needs not be public either (sender and receiver must know it, but it is not necessary that the Queen of England also knows it).
A typical key establishment protocol will result in both involve parties computing a piece of data which they, but only they, both know. With Diffie-Hellman (or any Elliptic Curve variant thereof), the said shared piece of data has a fixed length and they have no control over its value (they just both get the same seemingly random sequence of bits). In order to use that shared secret for symmetric encryption, they must derive that shared data into a sequence of bits of the appropriate length for whatever symmetric encryption algorithm they are about to use.
In a protocol in which you use a key establishment algorithm to obtain a shared secret between the sender and the receiver, and will use that secret to symmetrically encrypt a message (possibly a very long streamed message), it is possible to use the KDF to produce the key and the IV in one go. This is how it goes in, for instance, SSL: from the shared secret (called "pre-master secret" in the SSL spec) is computed a big block of derived secret data, which is then split into symmetric keys and initialization vectors for both directions of encryption. You could do otherwise, and, for instance, generate random IV and send them along with the encrypted data, instead of using an IV obtained through the KDF (that’s how it goes in recent versions of TLS, the successor to SSL). Both strategies are equally valid (TLS uses external random IV because they want a fresh random IV for each "record" — a packet of data within a TLS connection — which is why using the KDF was not deemed appropriate anymore).