Question

I have an array which contains error values as a function of two different quantities (alpha and eigRange).

I fill my array like this :

   for j in range(n): 
        for i in range(alphaLen):
            alpha = alpha_list[i]
            c = train.eig(xt_, yt_,m-j, m,alpha, "cpu")
            costListTrain[j, i] = cost.err(xt_, xt_, yt_, c)

   normedValues=costListTrain/np.max(costListTrain.ravel())

where

n = 20
alpha_list = [0.0001,0.0003,0.0008,0.001,0.003,0.006,0.01,0.03,0.05]

My costListTrain array contains some values that have very small differences, e.g.:

2.809458902485728 2.809458905776425 2.809458913576337 2.809459011062461
2.030326752376704 2.030329906064879 2.030337351188699 2.030428976282031
1.919840839066182 1.919846470077076 1.919859731440199 1.920021453630778
1.858436351617677 1.858444223016128 1.858462730482461 1.858687054377165
1.475871326997542 1.475901926855846 1.475973476249240 1.476822830933632
1.475775410801635 1.475806023102173 1.475877601316863 1.476727286424228
1.475774284270633 1.475804896751524 1.475876475382906 1.476726165223209
1.463578292548192 1.463611627166494 1.463689466240788 1.464609083309240
1.462859608038034 1.462893157900139 1.462971489632478 1.463896516033939
1.461912706143012 1.461954067956570 1.462047793798572 1.463079574605320
1.450581041157659 1.452770209885761 1.454835202839513 1.459676311335618
1.450581041157643 1.452770209885764 1.454835202839484 1.459676311335624
1.450581041157651 1.452770209885735 1.454835202839484 1.459676311335610
1.450581041157597 1.452770209885784 1.454835202839503 1.459676311335620
1.450581041157575 1.452770209885757 1.454835202839496 1.459676311335619
1.450581041157716 1.452770209885711 1.454835202839499 1.459676311335613
1.450581041157667 1.452770209885744 1.454835202839509 1.459676311335625
1.450581041157649 1.452770209885750 1.454835202839476 1.459676311335617
1.450581041157655 1.452770209885708 1.454835202839442 1.459676311335622
1.450581041157571 1.452770209885700 1.454835202839498 1.459676311335622

as you can here the value are very very close together!

I am trying to plotting this data in a way where I have the two quantities in the x, y axes and the error value is represented by the dot color.

This is how I’m plotting my data:

    alpha_list = np.log(alpha_list)        
    eigenvalues, alphaa  = np.meshgrid(eigRange, alpha_list) 

    vMin = np.min(costListTrain)
    vMax = np.max(costListTrain)

    plt.scatter(x, y, s=70, c=normedValues, vmin=vMin, vmax=vMax, alpha=0.50)

but the result is not correct.

• I tried to normalize my error value by dividing all values by the max, but it didn’t work !

• The only way that I could make it work (which is incorrect) is to normalize my data in two different ways. One is base on each column (which means factor1 is constant, factor 2 changing), and the other one based on row (means factor 2 is constant and factor one changing). But it doesn’t really make sense because I need a single plot to show the tradeoff between the two quantities on the error values.

UPDATE

this is what I mean by last paragraph.
normalizing values base on max on each rows which correspond to eigenvalues:

maxsEigBasedTrain= np.amax(costListTrain.T,1)[:,np.newaxis]    
maxsEigBasedTest= np.amax(costListTest.T,1)[:,np.newaxis]

normEigCostTrain=costListTrain.T/maxsEigBasedTrain
normEigCostTest=costListTest.T/maxsEigBasedTest

normalizing values base on max on each column which correspond to alphas:

maxsAlphaBasedTrain= np.amax(costListTrain,1)[:,np.newaxis]
maxsAlphaBasedTest= np.amax(costListTest,1)[:,np.newaxis]

normAlphaCostTrain=costListTrain/maxsAlphaBasedTrain
normAlphaCostTest=costListTest/maxsAlphaBasedTest

plot 1:

where no. eigenvalue = 10 and alpha changes (should correspond to column 10 of plot 1) :

where alpha = 0.0001 and eigenvalues change (should correspond to first row of plot1)

but as you can see the results are different from plot 1!

UPDATE:

just to clarify more stuff this is how I read my data:

from sklearn.datasets.samples_generator import make_regression

rng = np.random.RandomState(0)
diabetes = datasets.load_diabetes()

X_diabetes, y_diabetes = diabetes.data, diabetes.target
X_diabetes=np.c_[np.ones(len(X_diabetes)),X_diabetes]
ind = np.arange(X_diabetes.shape[0])
rng.shuffle(ind)
#===============================================================================
# Split Data 
#===============================================================================
import math
cross= math.ceil(0.7*len(X_diabetes))
ind_train = ind[:cross]
X_train, y_train = X_diabetes[ind_train], y_diabetes[ind_train]

ind_val=ind[cross:]
X_val,y_val=  X_diabetes[ind_val], y_diabetes[ind_val]

I also uploaded .csv files HERE

log.csv contain the original value before normalization for plot 1

normalizedLog.csv for plot 1

eigenConst.csv for plot 2

alphaConst.csv for plot 3

Answer 1

I think I found the answer. First of all there was one problem in my code. I was expecting the “No. of eigenvalue” correspond to rows but in my for loop they fill the columns. The currect answer is this :

for i in range(alphaLen):
    for j in range(n): 
        alpha=alpha_list[i]
        c=train.eig(xt_, yt_,m-j,m,alpha,"cpu")
        costListTrain[i,j]=cost.err(xt_,xt_,yt_,c)
        costListTest[i,j]=cost.err(xt_,xv_,yv_,c)

After asking questions from friends and colleagues I got this answer :

I would assume on default imshow and other plotting commands you
might want to use, do equally sized intervals on the values you are
plotting. if you can set that to logarithmic you should be fine.
Ideally, equally “populated bins” would proof most effective, i guess.

for plotting I just subtract the min value from the error and the add a small number and at the end take the log.

 temp=costListTrain- costListTrain.min()
 temp+=0.00000001
 extent = [0, 20,alpha_list[0], alpha_list[-1]]

 plt.imshow(np.log(temp),interpolation="nearest",cmap=plt.get_cmap('spectral'), extent =  extent, origin="lower")
 plt.colorbar()

and result is :