Question

I am monitoring a file in Python and triggering an action when it reaches a certain size. Right now I am sleeping and polling but I’m sure there is a more elegant way to do this:

POLLING_PERIOD = 10
SIZE_LIMIT = 1 * 1024 * 1024
while True:
    sleep(POLLING_PERIOD)
    if stat(file).st_size >= SIZE_LIMIT:
        # do something

The thing is, if I have a big POLLING_PERIOD, my file limit is not accurate if the file grows quickly, but if I have a small POLLING_PERIOD, I am wasting CPU.

Thanks!

How can I do this?

Thanks!

Answer 1

Linux Solution

You want to look at using pyinotify it is a Python binding for inotify.

Here is an example on watching for close events, it isn’t a big jump to listening for size changes.

#!/usr/bin/env python

import os, sys
from pyinotify import WatchManager, Notifier, ProcessEvent, EventsCodes

def Monitor(path):
    class PClose(ProcessEvent):
        def process_IN_CLOSE(self, event):
            f = event.name and os.path.join(event.path, event.name) or event.path
            print 'close event: ' + f

    wm = WatchManager()
    notifier = Notifier(wm, PClose())
    wm.add_watch(path, EventsCodes.IN_CLOSE_WRITE|EventsCodes.IN_CLOSE_NOWRITE)

    try:
        while 1:
            notifier.process_events()
            if notifier.check_events():
                notifier.read_events()
    except KeyboardInterrupt:
        notifier.stop()
        return


if __name__ == '__main__':
    try:
        path = sys.argv[1]
    except IndexError:
        print 'use: %s dir' % sys.argv[0]
    else:
        Monitor(path)

Windows Solution

pywin32 has bindings for file system notifications for the Windows file system.

What you want to look for is using FindFirstChangeNotification and tie into that and list for FILE_NOTIFY_CHANGE_SIZE. This example listens for File Name change it isn’t a big leap to listen for size changes.

import os

import win32file
import win32event
import win32con

path_to_watch = os.path.abspath (".")

#
# FindFirstChangeNotification sets up a handle for watching
#  file changes. The first parameter is the path to be
#  watched; the second is a boolean indicating whether the
#  directories underneath the one specified are to be watched;
#  the third is a list of flags as to what kind of changes to
#  watch for. We're just looking at file additions / deletions.
#
change_handle = win32file.FindFirstChangeNotification (
  path_to_watch,
  0,
  win32con.FILE_NOTIFY_CHANGE_FILE_NAME
)

#
# Loop forever, listing any file changes. The WaitFor... will
#  time out every half a second allowing for keyboard interrupts
#  to terminate the loop.
#
try:

  old_path_contents = dict ([(f, None) for f in os.listdir (path_to_watch)])
  while 1:
    result = win32event.WaitForSingleObject (change_handle, 500)

    #
    # If the WaitFor... returned because of a notification (as
    #  opposed to timing out or some error) then look for the
    #  changes in the directory contents.
    #
    if result == win32con.WAIT_OBJECT_0:
      new_path_contents = dict ([(f, None) for f in os.listdir (path_to_watch)])
      added = [f for f in new_path_contents if not f in old_path_contents]
      deleted = [f for f in old_path_contents if not f in new_path_contents]
      if added: print "Added: ", ", ".join (added)
      if deleted: print "Deleted: ", ", ".join (deleted)

      old_path_contents = new_path_contents
      win32file.FindNextChangeNotification (change_handle)

finally:
  win32file.FindCloseChangeNotification (change_handle)

OSX Solution

There is equivalent hooks into the OSX file system using PyKQueue as well, but if you can understand these examples you can Google for the OSX solution as well.

Here is a good article about Cross Platform File System Monitoring.