As far as I know, there are two possible calling conventions for the x64 code – Microsoft x64 and AMD64.
Now, gcc can be launched with the -mregparm=0 parameter, which doesn’t work if we are working using the AMD64 calling convention. This happens because the AMD64 convention mandates the usage of the registers for the first 6 variables (I’m not really sure why this is done, but I suspect it’s implemented due do possibly stack security issues).
So, here is the question:
Are there some strict rules like this (forced register usage) when compiling using gcc under Microsoft x64 convention? And, if yes, how can they be bypassed without breaking the ABI compatibility?
I don’t know Microsoft Windows (and never used it), so I probably cannot answer your question about it.
However, the AMD64 Application Binary Interface calling conventions (On Linux and other Unixes) are documented in the AMD64 ABI spec (maybe you should also find and read the equivalent document for Microsoft calling conventions). I believe they are using registers for the 6 first arguments because of performance considerations (passing values thru register is faster than passing them on the stack), not because of security considerations.
And whatever C++ compiler you use, you want it to follow some calling conventions, and these are practically dictated by the system (because you want to be able to call system libraries from your code). So if you break them, you will break the ABI compatibility.
But I cannot guess why are asking such a question. Are you developing a compiler with its own calling conventions? If yes, you still should have some means to call C libraries, and this required that for call to external C libraries, you follow the ABI conventions governing them. Look into the Ocaml compiler for an example.