Question

I have a 3000×300 matrix file (float). when I read and convert to float, I am getting float64, which is default in python. I tried numpy and map() to convert it to float32() but they both seem very inefficient.

my code:

x = open(readFrom, 'r').readlines()
y = [[float(i) for i in s.split()] for s in x]

time taken: 0:00:00.996000

numpy implementation:

x = open(readFrom, 'r').readlines()
y = [[np.float32(i) for i in s.split()] for s in x]

time taken: 0:00:06.093000

map()

x = open(readFrom, 'r').readlines()
y = [map(np.float32, s.split()) for s in x]

time taken: 0:00:05.474000

How can I convert to float32 very efficiently?

Thank you.

Update:

numpy.loadtxt() or numpy.genfromtxt() not working (giving memory error) for huge file. I have posted a question related to that and the approach I presented here works well for huge matrix file (50,000×5000). here is the question

Answer 1

If memory is a problem, and if you know the size of the field ahead of time, you probably don’t want to read the entire file in the first place. Something like this is probably more appropriate:

#allocate memory (np.empty would work too and be marginally faster, 
#                 but probably not worth mentioning).
a=np.zeros((3000,300),dtype=np.float32)  
with open(filename) as f:
    for i,line in enumerate(f):
        a[i,:]=map(np.float32,line.split()) 

from a couple quick (and surprising) tests on my machine, it appears that the map may not even be necessary:

a=np.zeros((3000,300),dtype=np.float32)  
with open(filename) as f:
    for i,line in enumerate(f):
        a[i,:]=line.split() 

This might not be the fastest, but certainly it’ll be the most memory efficient way to do it.

Some tests:

import numpy as np

def func1():   #No map -- And pretty speedy :-).
    a=np.zeros((3000,300),dtype=np.float32)
    with open('junk.txt') as f:
        for i,line in enumerate(f):
            a[i,:]=line.split()

def func2():
    a=np.zeros((3000,300),dtype=np.float32)
    with open('junk.txt') as f:
        for i,line in enumerate(f):
            a[i,:]=map(np.float32,line.split())

def func3():
    a=np.zeros((3000,300),dtype=np.float32)
    with open('junk.txt') as f:
        for i,line in enumerate(f):
            a[i,:]=map(float,line.split())

import timeit

print timeit.timeit('func1()',setup='from __main__ import func1',number=3)  #1.36s
print timeit.timeit('func2()',setup='from __main__ import func2',number=3)  #11.53s
print timeit.timeit('func3()',setup='from __main__ import func3',number=3)  #1.72s