Question

I have a list of values – all strings. I want to convert these values to their respective datatypes. I have mapping of values to the types information available.

There are three different datatypes: int, str, datetime.
The code needs to be able to handle the error cases with the data.

I am doing something like:-

tlist =  [ 'some datetime value', '12', 'string', .... ]

#convert it to: [ datetime object, 12, 'string', ....]

error_data = ['', ' ', '?', ...]

d = { 0: lambda x: datetime.strptime(x,...) if x not in error_data else x, 
      1: lambda x: int(x) if x not in error_data else 0,
      2: lambda x: x 
      ...
     }

result = [ d[i](j) for i, j in enumerate(tlist) ]

The list to convert is very long, like 180 values and I need to do it for thousands of such lists. The performance of above code is very poor. What is the fastest way to do it?

Thank you

Answer 1

Thank you guys for all those approaches. Yeah, I tried pretty much all the approaches mentioned, but none did perform well.

I tried the following approach and it worked pretty well for my performance needs. This is what I did.

1. I handled the datetime values as suggested by utdemir

2. I’ve inserted value 0 for all the int error values with code like –

   [i] = value if value != '' else 0

3. Instead of coercing value by value using dictionary, I coerced all the value at once using a list.

   
def coerce(l):
return [ l[0], int(l[1]), int(l[2]) ... ]


My Observations:

1. handling the error cases separately and coercing at once helps a lot.
2. letting the code recover from an exception takes a lot of time.
3. Going over a list multiple times doesn’t really hurt( I did it for handling error cases)
4. To give an idea, for couple of hours of data (0.60 million records), my approach brought down the runtime from 6-7 minutes to 2m30s