I mean in operating systems or their applications. The only way I can think of is examine binaries for the use of dangerous functions like strcpy(), and then try to exploit those. Though with compiler improvements like Visual Studio’s /GS switch this possibility should mostly be a thing of the past. Or am I mistaken?
What other ways do people use to find vulnerabilities? Just load your target in a debugger, then send unexpected input and see what happens? This seems like a long and tedious process.
Could anyone recommend some good books or websites on this subject?
Thanks in advance.
There are two major issues involved with “Client Side Security”.
The most common client exploited today is the browser in the form of “Drive By Downloads“. Most often memory corruption vulnerabilities are to blame. ActiveX com objects have been a common path on windows systems and AxMan is a good ActiveX fuzzer.
In terms of memory protection systems the /GS is a canary and it isn’t the be all end all for stopping buffer overflows. It only aims to protect stack based overflows that are attempting to overwrite the return address and control the EIP. NX Zones and canaries are a good things, but ASLR can be a whole lot better at stopping memory corruption exploits and not all ASLR implementations are made equally secure. Even with all three of these systems you’re still going to get hacked. IE 8 Running on Windows 7 had all of this and it was one of the first to be hacked at the pwn2own and here is how they did it. It involved chaining together a Heap Overflow and a Dangling Pointer vulnerability.
The problem with “client side security” is CWE-602: Client-Side Enforcement of Server-Side Security are created when the server side is trusting the client with secret resources (like passwords) or to send report on sensitive information such as the Players Score in a flash game.
The best way to look for client side issues is by looking at the traffic. WireShark is the best for non-browser client/server protocols. However TamperData is by far the best tool you can use for browser based platforms such as Flash and JavaScript. Each case is going to be different, unlike buffer overflows where its easy to see the process crash, client side trust issues are all about context and it takes a skilled human to look at the network traffic to figure out the problem.
Sometimes foolish programmers will hardcode a password into their application. Its trivial to decompile the app to obtain the data. Flash decompiling is very clean, and you’ll even get full variable names and code comments. Another option is using a debugger like OllyDBG to try and find the data in memory. IDA-Pro is the best decompiler for C/C++ applications.