I want to protect some algorithms, from being reversed engineered. I know there is always a risk, but I want to make the work as complicated as possible. I know in Java there are ProGuard and other obfuscator. But the most knowledge isn’t in the structure of the application, but in the numerical details of the algorithm. And reading about it, made me doubt on the protection of the algorithm.
Simple renaming some variables, wouldn’t make it hard enough to reverse engineer the algorithms. Perhaps you can tell me, which methods would be more appropriate for algorithms and which of obfuscator may do the best work on algorithms.
At the moment I’m thinking about a bit handwork and to combine it with a tool.
Assuming that your algorithm should be executed as Java bytecode, on arbitrary JVMs. Then people can hack their JVM to dump the bytecode somewhere, no matter how much you obfuscate the class loading process. Once you have the bytecode, you can do control flow analysis, i.e. decide what information gets passed from where to where.
You can confuse the order of the individual instructions, but that won’t change the computation. For someone who simply wants to run your algorithm unmodified, this doesn’t change anything. How much a reordering will prevent people from modifying your algorithm very much depends on the algorithm and the complexity of the control flow.
You might be able to confuse the control flow using reflection in some obscure way, or by implementing your own interpreter and using that to run the algorithm. But both these approaches will likely come at a severe penalty to the performance of the algorithm.
In other languages (like native x86 code) you might be able to confuse the disassembler by introducing ambiguity about how the bytes should be split into instructions, using some bytes as tail part of an instruction in one case, but as a distinct instruction in other cases. But in Java there is no such option, the meaning of bytecode is too well defined.
One way you might be able to obfuscate things somewhat is by closely intermixing the algorithm with other steps of the program. For a straight-line program, this might make things a wee bit harder to track, in particular if you pass numbers through invisible GUI objects or similar bizarre stuff. But once you require loops or similar, getting the loop bounds lined up seems very hard, so I doubt that this approach has much potential either. And I doubt there is a ready-to-use obfuscator for this, so you’d have to do things by hand.