Question

I’m using Pandas (0.9.1) to write a physics code. I have two dataframes:

Levels:

class 'pandas.core.frame.DataFrame'>
Int64Index: 37331 entries, 0 to 37330
Data columns:
atomic_number    37331  non-null values
ion_number       37331  non-null values
level_number     37331  non-null values
energy           37331  non-null values
g                37331  non-null values
metastable       37331  non-null values

Lines:

<class 'pandas.core.frame.DataFrame'>
Int64Index: 314338 entries, 0 to 314337
Data columns:
id                    314338  non-null values
wavelength            314338  non-null values
atomic_number         314338  non-null values
ion_number            314338  non-null values
f_ul                  314338  non-null values
f_lu                  314338  non-null values
level_number_lower    314338  non-null values
level_number_upper    314338  non-null values
dtypes: float64(3), int64(7)

There’s a couple of things I need to do:
I need to join levels with lines (atom, ion, level): at first on atom, ion, level_number_upper and then atom, ion, level_number_lower. Is there a way to precompute the join – memory is not an issue, but speed is.

I also need to group levels (on atom, ion) and do an operation on levels. I did this already (incredibly fast), but then had trouble joining the resulting series with the lines dataframe.

How do I do this?

Cheers
Wolfgang

update v1:

To show what I want to join merge here a code snippet

def calc_group_func(group):
    return np.sum(group['g']*np.exp(-group['energy'])
grouped_data = levels.group_by('atomic_number', 'ion_number')
grouped_data.apply(calc_group_func)

and then I want to join/merge grouped data with lines on atomic_number and ion_number

Answer 1

There may be a better way, but perhaps df.merge() would work here. df.merge() works on two DataFrames, so the values computed for each (atom, ion) pair, which are in a Series after apply(), need to be placed in a DataFrame first, at which time the final column name can also be specified.

In [9]: grouped_vals = grouped_data.apply(calc_group_func)

In [10]: grouped_vals
Out[10]: 
atomic_number  ion_number
0              0             0.517541
               1             0.046833
1              0             0.253188
               1             0.440194

In [11]: lines.merge(pd.DataFrame({'group_val': grouped_vals}),
   ....:             left_on=['atomic_number', 'ion_number'],
   ....:             right_index=True)
Out[11]: 
    atomic_number  ion_number  group_val
id                                      
a               0           0   0.517541
b               0           0   0.517541
c               0           1   0.046833
d               0           1   0.046833
e               1           0   0.253188
f               1           0   0.253188
g               1           1   0.440194
h               1           1   0.440194