I’ve heard a bit about using automated theorem provers in attempts to show that security vulnerabilities don’t exist in a software system. In general this is fiendishly hard to do.
My question is has anyone done work on using similar tools to find vulnerabilities in existing or proposed systems?
Eidt: I’m NOT asking about proving that a software system is secure. I’m asking about finding (ideally previously unknown) vulnerabilities (or even classes of them). I’m thinking like (but an not) a black hat here: describe the formal semantics of the system, describe what I want to attack and then let the computer figure out what chain of actions I need to use to take over your system.
So, at least in some meaningful sense, the opposite of proving something is secure is finding code paths for which it isn’t.
Try Byron Cook’s TERMINATOR project.
And at least two videos on Channel9. Here’s one of them
His research is likely to be a good starting point for you to learn about this extremely interesting area of research.
Projects such as Spec# and Typed-Assembly-Language are related too. In their quest to move the possibility of safety checks from runtime back to compile-time, they allow the compiler to detect many bad code paths as compilation errors. Strictly, they don’t help your stated intent, but the theory they exploit might be useful to you.