Question

The background:
I’ve written a python script to inspect IP packets, specifically the payload/data of a packet in order to detect if it could be used in a buffer (stack) overflow. Now as I understand it a NOP sled is used to pad the stack so that the instruction pointer will eventually run into your exploit code, this I can easily detect by looking for repeating occurrences of 0x90. I’ve seen code with a lot of NOP commands to as few as 8 in the case of SQL slammer so I could perhaps use 8 as a minimum.

Now my question, are NOP sleds often used in legitimate code? If the answer is yes, are there a few specific cases (which means I can look for these cases and then rule out the packet as potentially harmless) or is this approach just not practical for identifying malicious code?

Answer 1

The compiler will generate NOPs to align code — for instance, on some iterations of the x86, jumps execute faster if the jump destination is aligned to a 4-, 8-, or even 16-byte boundary.

Some compilers try to use “long NOPs” when possible — single instructions that take up more than one byte of space, and may formally do something, but have no effect on processor state — as on some iterations of the x86 architecture this is faster. For instance, 66 90 is a two-byte NOP, and 8d 74 26 00 is a four-byte NOP (technically lea 0(%esi,%eiz,1),%esi, but as you can see that just copies the value in %esi to itself, so there’s no effect). However, these can’t be used in all cases, and the sequences that are fastest on some x86es are depressingly often really slow on others. I haven’t read the current micro-optimization guidelines but I wouldn’t be surprised if Intel and AMD were working to make a string of 90s the fastest way to do a long NOP, and had their compilers match.