My software is using AES Rijndael.
I am using a SHA-256 hash to generate a key from a string with an arbitrary length, and then passing this as both the private and public key since in this instance I do not need to differentiate between the two.
How do I protect my key from being hacked out of the executable?
I know not to use a literal but instead generate the key at runtime with some predetermined steps, but all the same the key will still be in memory right before its sent on to the AES initialization function and so can quite easily be retrieved then.
AES is obviously very secure, but what good does that do me if someone breaks the executable instead?
Is there some common practise when solving this problem?
This can’t be done. This is the basic problem with e.g. DRM scheme’s on PC’s: they need to have the key in memory, so it can be extracted. You can maybe obscure it while it is not in use, but that’s about it. And if your application is popular and distributed, then somebody will crack you delicious scheme. That’s why some companies use dongles or TPM chips for high value applications.