Question

Sometimes, while conducting long-running measurements in some loop in python, I have no clue when my measurements are going to finish.

So I needed a tool to print some progress along the way. See my answer below.

Answer 1

Most of the time such measurement involves some heavy processing inside the main loop, so I developed a simple wrapper function that prints loop progress for any iterable:

from datetime import datetime
def print_progress(iterable, percent_step=1):
    total = float(len(iterable))
    # or 1 means that iterable has < 100 elems
    abs_step = int((total * percent_step)/100) or 1
    for i, obj in enumerate(iterable):
        if i and not i % abs_step:
             print "{0:.2%} processed, {1:%H:%M:%S}".format(i/total, datetime.now())
        yield obj

percent_step argument defines the granularity level of printing: each time the loop finished processing the percent_step % amount of data, the total percentage of processed data is printed on the screen.

Then can apply this wrapper to the loop:

for x in print_progress(my_list):
    # processing
    ...

The same iterator could be applied for django querysets, but calculating total object count before the loop could be expensive by itself, so in case of retrieving all objects from the database like PostgreSQL it is better to replace len with direct SQL:

"select reltuples from pg_class where relname='%s'" % table_name

Another variant that prints progress after the specified amount of seconds:

def print_progress(iterable, second_step=10):
    total = float(len(iterable))
    import time
    time1 = time.time()
    for i, obj in enumerate(iterable):
        if time.time() - time1 > second_step:
             print "{0:.2%} processed".format(i/total)
             time1 = time.time()
        yield obj