Question

I am currently trying to write a polymorphism engine in C++ to toy-around with a neat anti-hacking stay-alive checking idea I have. However, writing the polymorphism engine is proving rather difficult – I haven’t even established how I should go about doing it. The idea is to stream executable code to the user (that is the application I am protecting) and occasionally send them some code that runs some checksums on the memory image and returns it to the server. The problem is that I don’t want someone to simply hijack or programatically crack the stay-alive check; instead each one would be generated on the server working off of a simple stub of code and running it through a polymorphism engine. Each stay-alive check would return a value dependant on the check-sum of the data and a random algorithm sneaked inside of the stay-alive check. If the stub returns incorrectly, I know that the stay-alive check has been tampered with.

What I have to work with:

*The executable images PDB file.
*Assembler & disassembler engine of which I have implemented a interface between them which allows to to relocate code etc…

Here are the steps I was thinking of doing and how I might do them. I am using the x86 instruction set on a windows PE executable

Steps I plan on taking(My problem is with step 2):

1. Expand instructions

   • Find simple instructions like mov, or push and replace them with a couple instructions which achieve the same end though with more instructions. In this step I’ll also add loads of junk-code.
   • I plan on doing this just by using a series of translation tables in a database. This shouldn’t be very difficult to do.

2. Shuffling

   • This is the part I have the most trouble with. I need to isolate the code in to functions. Then I need to establish a series of instruction dependancies trees, and then I need to relocate them based upon which one depend on the other. I can find functions by parsing the pdb files, but creating instruction dependancy trees is the tricky part I am totally lost on.

3. Compress instructions

   • Compress instructions and implement a series of uncommon & obscure instructions in the process. And, like the first step, do this by using a database of code signatures.

To clarify again, I need help performing step number 2 and am unsure how I should even begin. I have tried making some diagrams but they become very confusing to follow.

OH: And obviously the protected code is not going to be very optimal – but this is just a security project I wanted to play with for school.

Answer 1

I think what you are after for “instruction dependency trees” is data flow analysis. This is classic compiler technology that determines for each code element (primitive operations in a programming language), what information is delivered to it from other code elements. When you are done, you end up with what amounts to a graph with nodes being code elements (in your case, individual instructions) and directed arcs between them showing what information has to flow so that the later elements can execute on results produced by “earlier” elements in the graph.

You can see some examples of such flow analysis at my website (focused on tool that does program analysis; this tool is probably not appropriate for binary analysis but the examples should be helpful).

There’s a ton of literature in the compiler books on doing data flow analysis. See any compiler text book.

There are a number of issues you need to handle:

• Parsing the code to extract the code elements. You sound like you have access to all the instructions already.

• Determining the operands required by, and the values produced, by each code element. This is pretty simple for “ADD register,register”, but you may find this daunting for production x86 CPU which has an astonishingly big and crazy instruction set. You have to collect this for every instruction the CPU might execute, and that means all of them pretty much. Nontheless, this is just sweat and a lot of time spent looking at the instruction reference manuals.

• Loops. Values can flow from an instruction, through other instructions, back to that same instruction, so the dataflows can form cycles (lots of cycles for complex loops). The dataflow literature will tell you how to handle this in terms of computing the dataflow arcs in the graph. What these mean for your protection scheme I don’t know.

• Conservative Analysis: You can’t get the ideal data flow, because in fact you are analyzing arbitrary algorithms (e.g., a Turing machine); pointers aggravate this problem pretty severely and machine code is full of pointers. So what the data flow analysis engines often do when unable to decide if “x feeds y”, is to simply assume “x (may) feed y”. The dataflow graph turns conceptually from “x (must) feed y” into the pragmatic “x (may) feed y” type arcs; the literature in fact is full of “must” and “may” algorithms because of this.
  Again, the literature tells you many ways to do [conservative] flow analysis (mostly having different degrees of conservatism; in fact the most conservatinve data flow analysis simply says “every x feeds every y”!). What this means in practice for your scheme, I don’t know.

There are a lot of people interested in binary code analysis (e.g., the NSA), and they do data flow analysis on machines instructions complete with pointer analysis. You might find this presentation interesting: http://research.cs.wisc.edu/wisa/presentations/2002/0114/gogul/gogul.1.14.02.pdf