Question

I’m running a test, and found that the file doesn’t actually get written until I control-C to abort the program. Can anyone explain why that would happen?

I expected it to write at the same time, so I could read the file in the middle of the process.

import os
from time import sleep

f = open("log.txt", "a+")
i = 0
while True:
  f.write(str(i))
  f.write("\n")
  i += 1
  sleep(0.1)

Answer 1

Writing to disk is slow, so many programs store up writes into large chunks which they write all-at-once. This is called buffering, and Python does it automatically when you open a file.

When you write to the file, you’re actually writing to a “buffer” in memory. When it fills up, Python will automatically write it to disk. You can tell it “write everything in the buffer to disk now” with

f.flush()

This isn’t quite the whole story, because the operating system will probably buffer writes as well. You can tell it to write the buffer of the file with

os.fsync(f.fileno())

Finally, you can tell Python not to buffer a particular file with open(f, "w", 0) or only to keep a 1-line buffer with open(f,"w", 1). Naturally, this will slow down all operations on that file, because writes are slow.