Question

Problem

Problem shaping

Image sequence position and size are fixed and known beforehand (it’s not scaled). It will be quite short, maximum of 20 frames and in a closed loop. I want to verify (event driven by button click), that I have seen it before.

Lets say I have some image sequence, like:

http://img514.imageshack.us/img514/5440/60372aeba8595eda.gif

If seen, I want to see the ID associated with it, if not – it will be analyzed and added as new instance of image sequence, that has been seen. I have though about this quite a while, and I admit, this might be a hard problem. I seem to be having hard time of putting this all together, can someone assist (in C#)?

Limitations and uses

I am not trying to recreate copyright detection system, like content id system Youtube has implemented (Margaret Gould Stewart at TED ( link )). The image sequence can be thought about like a (.gif) file, but it is not and there is no direct way to get binary. Similar method could be used, to avoid duplicates in “image sharing database”, but it is not what I am trying to do.

My effort

Gaussian blur

Mathematica function to generate Gaussian blur kernels:

getKernel[L_] := Transpose[{L}].{L}/(Total[Total[Transpose[{L}].{L}]])
getVKernel[L_] := L/Total[L]

Turns out, that it is much more efficient to use 2 passes of vector kernel, then matrix kernel. Thy are based on Pascal triangle uneven rows:

{1d/4, 1d/2, 1d/4}
{1d/16, 1d/4, 3d/8, 1d/4, 1d/16}
{1d/64, 3d/32, 15d/64, 5d/16, 15d/64, 3d/32, 1d/64}

Data input, hashing, grayscaleing and lightboxing

Example of source bits, that might be useful:

• Lightbox around the known rectangle: FrameX
• Using MD5CryptoServiceProvider to get md5 hash of the content inside known rectangle atm.
• Using ColorMatrix to grayscale image

Source example

Source example (GUI; code):

Get current content inside defined rectangle.

        private Bitmap getContentBitmap() {
            Rectangle r = f.r;
            Bitmap hc = new Bitmap(r.Width, r.Height);
            using (Graphics gf = Graphics.FromImage(hc)) {
                gf.CopyFromScreen(r.Left, r.Top, 0, 0, //
                    new Size(r.Width, r.Height), CopyPixelOperation.SourceCopy);
            }
            return hc;
        }

Get md5 hash of bitmap.

        private byte[] getBitmapHash(Bitmap hc) {
            return md5.ComputeHash(c.ConvertTo(hc, typeof(byte[])) as byte[]);
        }

Get grayscale of the image.

        public static Bitmap getGrayscale(Bitmap hc){
            Bitmap result = new Bitmap(hc.Width, hc.Height);
            ColorMatrix colorMatrix = new ColorMatrix(new float[][]{   
                new float[]{0.5f,0.5f,0.5f,0,0}, new float[]{0.5f,0.5f,0.5f,0,0},
                new float[]{0.5f,0.5f,0.5f,0,0}, new float[]{0,0,0,1,0,0},
                new float[]{0,0,0,0,1,0}, new float[]{0,0,0,0,0,1}});

            using (Graphics g = Graphics.FromImage(result)) {
                ImageAttributes attributes = new ImageAttributes();
                attributes.SetColorMatrix(colorMatrix);
                g.DrawImage(hc, new Rectangle(0, 0, hc.Width, hc.Height),
                   0, 0, hc.Width, hc.Height, GraphicsUnit.Pixel, attributes);
            }
            return result;
        }

Answer 1

I think you have a few issues with this:

1. Not all image sequences [videos] are equal [but many are similar]
2. Where is your data coming from?
3. How will you repesent the data related to your viewings?
4. Size of the data

Issue #1:

Many images can differ slightly by compression, water marking, missing frames, and adding clips. I would suggest sampling the video. For example you may want to consider sub-sampling small sections of the images in the video. Additionally, to avoid noisy images and issues with lossely compression algorithms. You may want to consider grayscaling the frames sampled, and doing a gaussian blur. [Guassian because its “more natural” (short answer)] Once you have enough sub samples to where you have a good confidence of similarity to the video then store it in a database. With the samples you can hash them, or store them to do a % similarity later.

Issue #2

Your datasource is going to influence the tool kits, and libraries that you use.
I would suggest keeping this simple [keep it with gifs and create a custom viewer, dont’ try to write a browser plugin while developing your logic]

Issue #3

Using something like Postgres [if there are a lot of large sized objects] or SQLLite is highly suggested for indexing, storing, and recalling past meta data.

Issue #4

The size of the data will have a huge determination on recall, sampling, querying the database, etc.

Overall advice: Don’t bite off more than you can handle at this stage. Start small and then grow.

Also take a look at Computer Vision algorithms for more help on the object representation/recall.