I have a DLL that I need to load (I have written it and compiled it), and I would like to insert instructions between existing instructions of the assembly code before loading the DLL into memory. Of course, you can’t just read every byte and insert them between there because instructions sometimes are multiple bytes.
I was thinking of using something like Udis86 and reading instructions one by one and then writing them to memory, and between them writing my other instructions. Is this a good approach or is there something better?
You have (at least) two choices:
Both are possible, though with the latter it will be necessary to circumvent or disable security protection: code (as opposed to data) is loaded into memory which is not writable in most (if not all—not sure about CE) Windows environments.
With the code loaded, there is direct o/s support to find symbol addresses. Modifying a file will require either advanced knowledge about decoding the symbol information and interpreting the file offset, or searching for patterns placed in the file specifically for the purpose. This can be logic in the DllMain entry point which is called to initialize the DLL, or any other function within the DLL which is known to execute early enough.
There are also the techniques of DLL injection which could accomplish the same ends.
But, Instead of all this, what about arranging for the DLL to use some callback function pass to it once you’ve composed the code to execute? It’s hard to tell what’s useful without knowing more about what you are trying to accomplish.
As for how to do it, an easy way is to write the instructions in an assembly module, assemble it, and then examine the generated bytes. Naturally, you should thoroughly understand the target assembly language so that branch and relative jump offsets—as well as data references—are computed or fixed up correctly. While is it possible and sometimes practical to disassemble instructions at runtime, it’s usually easier to have already identified the instructions through some other means (like a debugger) and have the program look for the byte sequence and then perform whatever transformations are needed.