Question

Let’s assume someone has to write a solution to a problem and I have to test his solution with some tests. Is it possible (maybe with reflections or something) his program to pass all my tests, but to have nothing in common with the real solution to the problem?

Answer 1

General Case

In the general case, no. Since they do not even know you implemented the test correctly, nothing they do – not even correctly implementing the homework – can guarantee passing the test.

Special Cases

If the students can run your test, they could use a learning algorithm.

If the students have access to the source code of your test, they can look for tricks like using reflection to get expected results stored in variables. If they can’t find any source code tricks, they can still compile and run your tests.

If the students have access to the byte code of your test, they can probably disassemble it and use source code tricks. Even if you effectively obsfuscated it, they can still run it.

If the students know that you are testing student code against other student code, all they have to do is be consistent.

Recommendations

1. Do not make the test available to students. They should be independently testing their homework. After they have submitted their homework to you for formal evaluation, run the tests.
2. Use randomization to make your tests more robust.
3. If your test compares student code against other student code, then you should manually investigate every failure. Maybe one student implemented correctly and the others copied an incorrect implementation. In this case, majority rules rewards incompetence.
4. You may collect their tests and test their tests.