Question

I have about 50 million lists of strings in Python like this one:

["1", "1.0", "", "foobar", "3.0", ...]

And I need to turn these into a list of floats and Nones like this one:

[1.0, 1.0, None, None, 3.0, ...]

Currently I use some code like:

def to_float_or_None(x):
    try:
        return float(x)
    except ValueError:
        return None

result = []
for record in database:
    result.append(map(to_float_or_None, record))

The to_float_or_None function is taking in total about 750 seconds (according to cProfile)… Is there a faster way to perform this conversion from a list of strings to a list of floats/Nones?

Update
I had identified the to_float_or_None function as the main bottleneck. I can not find a significant difference in speed between using map and using list comprehensions.
I applied Paulo Scardine’s tip to check the input, and it already saves 1/4 of the time.

def to_float_or_None(x):
    if not(x and x[0] in "0123456789."):
        return None
    try:
        return float(x)
    except:
        return None

The use of generators was new to me, so thank you for the tip Cpfohl and Lattyware! This indeed speeds up the reading of the file even more, but I was hoping to save some memory by converting the strings to floats/Nones.

Answer 1

The answers given thus far don’t really fully answer the question. try...catch vs a validating if then can result in different performance (see: https://stackoverflow.com/a/5591737/456188). To summarize that answer: depends on the ratio of failures to successes and the MEASURED time of a failure and success in both cases. Basically we can’t answer this, but we can tell you how to:

1. Look at a few representative cases to get a ratio.
2. Write an if/then that tests the same as the try/catch optimize it and then measure how long it takes both version of the to_float_or_None to fail 100 times and measure how long it takes both versions of the to_float_or_None to succeed 100 times.
3. Do a little math to figure out which will be faster.

Side note about the list comprehension issue:

Depending on whether the you want to be able to index the results of this, or whether you just want to iterate over it a generator expression would actually be even better than a list comprehension (just replace the [ ] characters with ( ) characters).

It takes essentialy no time to create, and the actual execution of to_float_or_None (which is the expensive part) can be delayed until the result it needed.

This is useful for many reasons, but won’t work if you’re going to need to index it. It will however, allow you to zip the original collection with the generator so you can still have access to the original string along with its float_or_none result.