Question

I have a dataframe which includes a column that has a list. When I write the dataframe to a file then re-open it, I end up converting the list to a string. Is there a way to safely read/write dataframes that have lists as members?

df1 = DataFrame({'a':[['john quincy', 'tom jones', 'jerry rice'],['bob smith','sally ride','little wayne'],['seven','eight','nine'],['ten','eleven','twelve']],'b':[9,2,4,5], 'c': [7,3,0,9]})

df1.to_csv('temp.csv')
df2 = read_csv('temp.csv')

#note how the list (df1) has been converted to a string (df2)
df1['a'][0]
['john quincy', 'tom jones', 'jerry rice']

df2['a'][0]
"['john quincy', 'tom jones', 'jerry rice']"

Answer 1

No need to covert the list into string in the first place, list’s will be converted to string automatically. Just write the dataframe containing a list, and use ast.literal_eval on df2

                                             a  b  c
0   ['john quincy', 'tom jones', 'jerry rice']  9  7
1  ['bob smith', 'sally ride', 'little wayne']  2  3
2                   ['seven', 'eight', 'nine']  4  0
3                  ['ten', 'eleven', 'twelve']  5  9

df1.to_csv('temp.csv')
df2 = read_csv('temp.csv')

Use ast.literal_eval to get the string back to list:

import ast
fd2['a']=df2['a'].apply(lambda x: ast.literal_eval(x))
type(df2['a'][1])

Output:

list