Question

I am new to both stackoverflow and python, so this might look obvious:

In this procedure I want to create a new file named database out of a list of files generated by a previous procedure. The files in the list are quite big (around 13.6 MB). The goal is to have a single file with lines from all other:

database = open('current_database', 'a')

def file_apender(new):
    for line in new:
        database.write(line)

def file_join(list_of_files):
    for file in list_of_files:
        file_apender(file)

Then if I:

file_join(a_file_list)

I get the database file, but 26 lines are missing and the last one is not complete.
Here is the ending of the file:

63052300774565. 12 4 3 0 0.37 0.79 10.89 12.00 1.21 25.26 0.00 0.00 0.00 0.00
63052300774565. 12 2 0 0 0.06 0.12 2.04 2.21 0.86 5.30 0.00 0.00 0.00 0.00
63052300774565. 12 0 0 0 0.12 0.26 3.13 4.63 3.81 11.95 0.00 0.00 0.00 0.00
63052300774565. 12 2 2 0 0.06 0.15 1.35 2.39 0.00 3.94 0.00 0.00 0.00 0.00
63052300774565. 12 0 1 0 0.06 0.08 1.13 1.29 3.60 6.16 0.00 0.00 0.00 0.00
63052300774565. 12 2 0 0 0.23 0.41 4.02 6.47 8.39 19.52 0.00 0.00 0.00 0.00
63052300774565. 12 1 3 0 0.05 0.16 1.85 2.50 0.57 5.13 0

I have tried to find out if there is a memory limitation… Otherwise I got no ideas.

Answer 1

I’m going to use my psychic debugging skills, and guess that you don’t have a database.close().

If you don’t close the file when writing to it, there may still be data in the Python output buffers that hasn’t been written to the OS yet. If your program exits at that point, then the data is not written to disk and you will be missing data at the end.