Question

Given a stream of bytes, how can I tell if this stream contains a PDF document or something else?

I am using .NET and C# but it does not matter.

Answer 1

It all depends of how well/reliable you want the detection working.

Here my selection of most important bits+pieces from the 756 page long official definition, straight from the horse’s mouth (PDF 32000:1-2008):

A basic conforming PDF file shall be constructed of following four elements (see Figure 2):

• A one-line header identifying the version of the PDF specification to which the file conforms
• A body containing the objects that make up the document contained in the file
• A cross-reference table containing information about the indirect objects in the file
• A trailer giving the location of the cross-reference table and of certain special objects within the body of the file
  [….]

The first line of a PDF file shall be a header consisting of the 5 characters %PDF– followed by a version number of the form 1.N, where N is a digit between 0 and 7.
A conforming reader shall accept files with any of the following headers:
%PDF–1.0
%PDF–1.1
%PDF–1.2
%PDF–1.3
%PDF–1.4
%PDF–1.5
%PDF–1.6
%PDF–1.7
[…]

If a PDF file contains binary data, as most do (see 7.2, “Lexical Conventions”), the header line shall be immediately followed by a comment line containing at least four binary characters—that is, characters whose codes are 128 or greater. This ensures proper behaviour of file transfer applications that inspect data near the beginning of a file to determine whether to treat the file’s contents as text or as binary.

Trailer
[….] The last line of the file shall contain only the end-of-file marker, %%EOF. The two preceding lines shall contain, one per line and in order, the keyword startxref and the byte offset in the decoded stream from the beginning of the file to the beginning of the xref keyword in the last cross-reference section.

Summary

Two of the most important thing to remember:

(a) The first ‘header line’

   %PDF-1.X

[where X in 0..7] must appear on a line of its own be followed by a newline. This line must appear within the first 4096 Bytes, not necessarily on the very first line. The preceding lines may contain non-PDF content, but printer job languange commands (PJL) or comments.

(b) The very next line must be four binary bytes if the PDF contains binary data.

Just parsing for ‘%PDF-1.’ has bitten a lot of people already….