Question

I am looking for a way to obfuscate (in the object code) a test – something like what might be done to check that a license key is valid. What I am trying to prevent is someone searching through an image binary for the code that processes the response.

bool checkError = foo();
if ( checkError ) // I'd like to avoid making a simple check like this one.
{
   // process response
}

This is a simplistic example, but not a recommended approach:

int check = 71 * 13;
check += 35 * isValid(); // will only return 0 or 1

//later (delayed execution of response)
if ( check % 71 )
{
   //process response
}

EDIT:
Just to clarify, the actual test is already finished and I’m getting a pass/fail return. My response processing will be a basic jmp and would be interested in pointers on how to obfuscate the location of the jmp.

Answer 1

One approach would be to put the code that does the license check into a separate DLL. In the main application, load the DLL at runtime and calculate the checksum of the DLL itself. The app stores the checksum that was calculated with the DLL was built. If the checksums don’t match, you have several options, show a wrong-version message – a bit obvious; Do not call the license check – less obvious but will be noticed when the attacker wonders why the license check doesn’t get called; call a function with a similar name to the real license-check function.

Think of it as using Public Key Encryption. Use a public key as part of the config and have a private key built into the app. If they mess with the public key, the digital signature of the app will be compromised in a detectable way.

I agree with @camccann that it would help to understand the kind of attack you expect. As a last resort, split the license-check into as many parts as is feasible to make it harder to bypass by changing a single branch point.

[EDIT]

Another thought would be to use a State Machine. See the command structure example in the top answer to this question. Put the evaluation of the license check into the form of a hash lookup and a set of dummy function calls into an array along with the proper one. The decision code that evaluates the license check into a table/hash lookup for the appropriate function will not look like your typical

if(){ pass;} else { fail; } 

construct.

Two benefits,
1) there isn’t a boolean condition to bypass and
2) they can’t do a simple JMP instruction without knowing the address/name of the function to pass control to.

SO thread on a state machine turorial.
SO thread on state machine implementations