Question

I am working on extracting information from figures in scientific papers, combining image processing on the figure itself and natural language processing on its caption.

I have got to a stage where I have isolated objects within the image, and got an “average” colour for each (in both RGB and Lab colorspaces, not sure what’s best yet). From the caption, I have parsed a list of objects, along with the color name used to describe them.

So I have two lists:

names = ['Red', 'Brown', 'Yellow', 'Magenta'];
rgbs = [ (249,0,252), (253,0,1), (250,248,60), (140,70,20)];

I’m trying to figure out an automated method of determining the best pairings between the names and values. Thinking about it, I think it might be best to start by using a lookup table for all common names to convert the names into their “accepted” rgb values. Then I can work out the “distance” (Euclidean?) between each of the rgb values and each of the name rgbs. At this stage somehow I should be able to use those distances to find the optimum pairings, but I’m not sure exactly how.

Does anyone have any ideas, or know of any libraries that might provide useful tools for this?

Answer 1

Try reading this work it looks like it solves quite a similar problem.

Can you obtain such list pairs ( names – RGB-values ) from different figures?
If so, by intersecting these list you may be isolated a color name that is common to a bunch of pairs (only this color) and then try and find the RGB-triplet that is “as-common-as-possible” (up to a little distortion).
You may use this elimination process till you isolate all colors.

For example: suppose you have

{ ['Red','Green'], {[1 0 0], [0 1 0]} }
{ ['Red','blue'] , {[.9 .1. 1], [ .2 .3 .9] }

You have ‘Red’ in the intersection, and [1 0 0], [.9 .1 .1] the closest colors.