Has there been any work done on any hypothetical hardware + OS architecture or overall software design which is provably not possible to hack? In other words, an architecture which allows for only limited code execution such that exploits are not at all feasible within some particular mathematical definition of an “exploit”. Therefore the potential damage that exploits could do under this framework would be provably limited.
Generally speaking, I’m looking for a mathematically motivated theory of “hackable” versus “unhackable” architectures based on some reasonable underlying assumptions.
If you restrict that to ‘hacking’ and leave all other security aspects aside (say, broadcasting sensitive information or whatever), you will find many answers at LANGSEC. (Be sure to look at the short intro.)
They offer a definition for hacking (unexpected input-driven computation), describe the theory behind the problems (turing complete file formats and protocols, ad-hoc parsing, differing grammars) and explain how to avoid that. You just need to actually do this in your code.
(If you meant to include more general security aspects (undefined behavior, program crashes, access restrictions): Yes, tons of stuff. I can write a loong explanation, if you like.)