I’m working on a project that is trying to use context-free grammars for parsing images. We are trying to construct trees of image segments, then use machine learning to parse images using these visual grammars.
I have found SVM-CFG which looks ideal, the trouble is that it is designed for string parsing, where each terminal in the string has at most two neighbors (the words before and after). In our visual grammar, each segment can be next to an arbitrary number of other segments.
What is the best way to parse these visual grammars? Specifically, can I encode my data to use SVM-CFG? Or am I going to have to write my own Kernel/parsing library?
SVM-CFG is a specific implementation of the cutting plane optimization algorithm used in SVM-struct (described here http://www.cs.cornell.edu/People/tj/publications/tsochantaridis_etal_04a.pdf, Section 4).
At each step, the cutting plane algorithm calls a function to find the highest scoring structured output assignment (in SVM-CFG this is the highest scoring parse).
For one-dimensional strings, SVM-CFG runs a dynamic programming algorithm to find the highest scoring parse in polynomial time.
You could extend SVM-struct to return the highest scoring parse for an image, but no polynomial-time algorithm exists to do this!
Here is a reference for a state-of-the-art technique that parses images: http://www.socher.org/uploads/Main/SocherLinNgManning_ICML2011.pdf. They run into the same problem for finding the highest scoring parse of an image segmentation, so they use a greedy algorithm to find an approximate solution (see section 4.2). You might be able to incorporate a similar greedy algorithm into SVM-struct.