I Am calling Captcha page as popup winow in the page using below syntax
this.Page.ClientScript.RegisterStartupScript(this.Page.GetType(),"Test","window.open('CaptchaLogin.aspx',null,'height=200,width=500,status= no,resizable= no,scrollbars=no,toolbar=no,toolbar=no,location=centre,menubar=no');", true);
Here is a simple article on the subject, with a demo:
http://www.nigraphic.com/blog/java-script/how-open-new-window-popup-center-screen
BTW Google is your best friend 🙂 There are hundreds of implementations available to you:
http://www.google.com/search?q=javascript+center+popup