Question

I’m wondering how people test artificial intelligence algorithms in an automated fashion.

One example would be for the Turing Test – say there were a number of submissions for a contest. Is there any conceivable way to score candidates in an automated fashion – other than just having humans test them out.

I’ve also seen some data sets (obscured images of numbers/letters, groups of photos, etc) that can be fed in and learned over time. What good resources are out there for this.

One challenge I see: you don’t want an algorithm that tailors itself to the test data over time, since you are trying to see how well it does in the general case. Are there any techniques to ensure it doesn’t do this? Such as giving it a random test each time, or averaging its results over a bunch of random tests.

Basically, given a bunch of algorithms, I want some automated process to feed it data and see how well it “learned” it or can predict new stuff it hasn’t seen yet.

Answer 1

Generally when it comes to this sort of thing you have two datasets – one large “training set” which you use to build and tune the algorithm, and a separate smaller “probe set” that you use to evaluate its performance.