I need to write and algorithm that can detect which state an application (used to fill out forms) is in based on screenshots.
It has 2 inputs:
A: Approximately 2-10 screenshots from an application with different tabs selected. These are made by the user, so i can instruct him to things like “select the upper area of the program” or “select the whole window”, but i can not expect pixel-perfect precision.
B: a screenshot of one of these states. The forms are filled with different data.
The goal is to determine which screenshot from “A” is from the same state as “B”.
An example screenshot:
An example based on this screenshot:
A input: 10 screenshots from this program with “Menu”,”Sale Order”,”Purchase order”,… tabs selected
B input: the screenshot above.
The task is to determine which of the 10 screenshots matches this image.
I have tried to use an image descriptor algorithm, (SURF) but it has a really high error ratio, since its not made for such tasks.
Anyone has an idea how to make such classification? Should i use some filter (e.g median or blur) on the screenshots, and then run trought some classification algorithm? Or extract some other feature to classify (FFT,histogram,..)?
I guess you can use the tab width instead of the tab label, which is much easier to calculate. For example, {“Menu”, “Sale Order”, “Purchase Order”} all have different widths.
If you have to look inside the tab, you can attempt some template matching.