Question

I am trying this simple setup of variables:

In [94]: cc
Out[94]: 
                 d0         d1
class sample                    
5     66      0.128320  0.970817
      66      0.160488  0.969077
      77      0.919263  0.008597
6     77      0.811914  0.123960
      88      0.639887  0.262943
      88      0.312303  0.660786

In [101]: bb
Out[101]: 
                     d0         d1
class sample                    
2     22      0.730631  0.656266
      33      0.871292  0.942768
3     44      0.081831  0.714360
      55      0.600095  0.770108

In [102]: aa
Out[102]: 
                     d0         d1
class sample                    
0     00      0.190409  0.789750
      11      0.588001  0.250663
1     22      0.888343  0.428968
      33      0.185525  0.450020

I can perform the following command

In [103]: aa.append(bb)
Out[103]: 
                     d0         d1
class sample                    
0     00      0.190409  0.789750
      11      0.588001  0.250663
1     22      0.888343  0.428968
      33      0.185525  0.450020
2     22      0.730631  0.656266
      33      0.871292  0.942768
3     44      0.081831  0.714360
      55      0.600095  0.770108

Why I cant perform the following command in the same manner?

aa.append(cc)

[I get the following exception]

ValueError: all arrays must be same length

UPDATE:

It works fine if I did not provide column names, but if for example I have 4 columns, with names [‘d0′,’d0′,’d1′,’d1’] for 4X4 and 8X4, it does not work anymore

here is the code for reproducing the error

import pandas
y1 = [['0','0','1','1'],['00','11','22','33']]
y2 = [['2','2','3','3','4','4'],['44','55','66','77','88','99']]
x1  = np.random.rand(4,4)
x2 = np.random.rand(6,4)
cols = ['d1']*2 + ['d2']*2
names = ['class','idx']
aa = pandas.DataFrame(x1,index=y1,columns = cols)
aa.index.names = names
print aa
bb = pandas.DataFrame(x2,index=y2,columns = cols)
bb.index.names = names
print bb

aa.append(bb)

What should I do to get this running?

Thanks

Answer 1

concatenated = pd.concat([bb, cc])

concatenated

                      0         1
class  sample                    
2      22      0.730631  0.656266
       33      0.871282  0.942768
3      44      0.081831  0.714360
       55      0.600095  0.770108
5      66      0.128320  0.970817
       66      0.160488  0.969077
       77      0.919263  0.008597
6      77      0.811914  0.123960
       88      0.639887  0.262943
       88      0.312303  0.660786

Answer To Your Edited Question

So to answer your edited question, the problem lies with your column names having duplicates.

 cols = ['d1']*2 + ['d2']*2  # <-- this creates ['d1', 'd1', 'd2', 'd2']

and your dataframes end up having what-is-considered duplicated columns, i.e.

In [62]: aa
Out[62]: 
                 d1        d1        d2        d2
class idx                                        
0     00   0.805445  0.442059  0.296162  0.041271
      11   0.384600  0.723297  0.997918  0.006661
1     22   0.685997  0.794470  0.541922  0.326008
      33   0.117422  0.667745  0.662031  0.634429

and

In [64]: bb
Out[64]: 
                 d1        d1        d2        d2
class idx                                        
2     44   0.465559  0.496039  0.044766  0.649145
      55   0.560626  0.684286  0.929473  0.607542
3     66   0.526605  0.836667  0.608098  0.159471
      77   0.216756  0.749625  0.096782  0.547273
4     88   0.619338  0.032676  0.218736  0.684045
      99   0.987934  0.349520  0.346036  0.926373

pandas.append() (or concat() method) can only append correctly if you have unique column names.

Try this and you will not get any error:-

cols2 = ['d1', 'd2', 'd3', 'd4']

cc = pandas.DataFrame(x1, index=y1, columns=cols2)
cc.index.names = names

dd = pandas.DataFrame(x2, index=y2, columns=cols2)
cc.index.names = names

Now…

In [70]: cc.append(dd)
Out[70]: 
                 d1        d2        d3        d4
class idx                                        
0     00   0.805445  0.442059  0.296162  0.041271
      11   0.384600  0.723297  0.997918  0.006661
1     22   0.685997  0.794470  0.541922  0.326008
      33   0.117422  0.667745  0.662031  0.634429
2     44   0.465559  0.496039  0.044766  0.649145
      55   0.560626  0.684286  0.929473  0.607542
3     66   0.526605  0.836667  0.608098  0.159471
      77   0.216756  0.749625  0.096782  0.547273
4     88   0.619338  0.032676  0.218736  0.684045
      99   0.987934  0.349520  0.346036  0.926373