Question

Mathematica’s Entropy function is order-dependent when using the SameTest option.

That is:

Entropy[RandomSample[Range[11]], SameTest->(Abs[#1-#2]>1&) ]

will give different results many times.

I assume that this is because Entropy[] is in fact Union-izing the list, but, unlike Union, it is actually replacing one of the SameTest values with the other, and this replacement is order sensitive.

Is this a bug or is it the expected behaviour?

Answer 1

You can see using Trace[ ] that the Entropy[ ] function ends up using Tally[ ] for counting the frequency of each state (numbers in this case).

So for example

 Entropy[{1,2,3,4}, SameTest->(Abs[#1-#2]>1&)]  

calls

 Tally[{1,2,3,4}, SameTest->(Abs[#1-#2]>1&)]  

which gives

 -> {{1, 3}, {2, 1}}

because it groups {1,3,4} and {2}

But if you ask for

 Tally[{2,1,3,4}, SameTest->(Abs[#1-#2]>1&)]  

you get

  -> {{2, 2}, {1, 2}}

because it groups {2,4} and {1,3}

Resulting in a different states distribution (2,2) vs (3,1) before, and hence in a different entropy value.

I think the problem arises because your SameTest is not partitioning the domain in two equivalence classes, as it should.

Edit

Just reformulating the last sentence:

Mma assumes that

a === b && b === c  Implies a === c  

which is not true in your case. For example

2 === 4 && 4 === 1  but  2 !=== 1