Analyze binary for holes

[Peter Bondra <olafandjasper () hushmail com>]

Hello
I am interested in how you may go about analyzing a binary file to 
determine potential format string or buffer overflow holes.

The platforms I am testing are: SunOs Solaris 2.7/8/9(SPARC) and Windows 
NT/2000/XP.

This is my process, maybe you could direct and fill in the massive blanks:

UNIX:
In the unix world my first step is to list out the SUID-root files.
My next step is to identify which files have potential vulnerabilities.  
On the Unix side I have used strings, but what does that tell me about.
I have seen a few mallocs, callocs, and things that look like a format 
string for a printf... But not sure what to do next...SO I was thinking
of brute forcing the binary command line args and/or environmental vars
to see if I can dump core..

Can you identify potential format string vulnerabilities from binary?
Can you identify potential buffer overflow vulns. from binary?

WINDOWS:
I have no idea how to recognize a vulnerable program in the Windows 
word.Is there anything like SUID-roor, etc??

Thanks