Question

I’m working on implementing a histogram, and one of the key points is the quick merging of histogram bins. Because I don’t have a priori knowledge of the data set being approximated by the histogram, I need to come up with a way that quickly merges neighboring bins, after I’ve exceeded the maximum number of bins.

So, as an example, if you’re approximating a data stream 23, 19, 10, 16, 36, 2, 9, 32, 30, 45 with five histogram bins, you’d read in the first five elements, obtaining:

(23, 1), (19,1), (10,1), (16,1), (36,1)

Adding the bin (2,1) causes an issue, since we’ve exceeded the maximum number of bins. So, we add (2,1) and merge the two closest bins — (16,1) and (19,1) — to get a new bin (17.5,2) that replaces those two.

Repeating this method for the rest of the histogram gives us the final output:

(2,1), (9.5,2), (19.33,3), (32.67,3), (45,1).

Implementing this without respect for complexity issues is trivial. However, I’m really concerned about optimizing this for large data sets, because my “trivial” implementation ends up taking 15 seconds to run on a stream of 100,000 gaussian distributed values.

My current thought is to use boost::multi_index to keep track of my HistogramBin struct, which is defined as:

struct HistogramBin
{
    double bin;
    unsigned long count;
    bool isNull;

    HistogramBin(double x, bool n = false)
    : bin(x), count(1), isNull(n) {}

    bool operator<(const HistogramBin &other) const
    { return (bin < other.bin); }

    // Merges other with this histogram bin
    // E.g., if you have (2.0,1) and (3.0,2), you'd merge them into (2.67,3)
    void merge(const HistogramBin &other)
    {
        unsigned long old_count = count;
        count += other.count;
        bin = (bin*old_count + other.bin*other.count)/count;
    }

    // Gets the difference between two histogram bins
    const double getDifference(const HistogramBin &other) const
    { return (double)abs(bin - other.bin); }
};

So, the multi_index would use ordered_unique<> to sort on HistogramBin::bin.

Now, this doesn’t resolve the issue of sorting the bins by the differences between neighboring bins. Indexing by HistogramBin::bin gives us an ordered list of HistogramBin objects, but then the next step is to calculate the difference between the current bin and the next one, and then to sort on those values as well.

Is there any way to sort on those values, while maintaining the integrity of the list, and without introducing a new container (such as a multimap of difference/iterator key/value pairs)?

Maintaining this list is my current idea on a near-optimal solution to the complexity problem, because it only needs to be changed when there’s a merge, and a merge only happens when a new value is added.

Any thoughts or insight would be appreciated.

Answer 1

Posting this as an answer:

Take a look at the answer to this question: Online k-means clustering. If I understand your problem correctly, that’s pretty much what your looking for, where the initial k guesses are your first k values.

If you keep your bin centers ordered, you can binary search into the list and the closest value is either the one before or the one after it, with an overall complexity of O(n*log(m)) where m is the number of bins and n is the amount of data.