It’s generally disallowed to install and use unmanaged applications and/or libraries on Windows Phone.
Even though there are builds of OpenCV targeting ARM, is there an effort to build a Computer Vision library to run under a managed environment provided by Microsoft?
You could try out the Silverlight Augmented Reality Toolkit:
http://slartoolkit.codeplex.com/
This works for WP7.