Question

How should I monitor the progress of a mapped function in clojure?

When processing records in an imperative language I often print a message every so often to indicate how far things have gone, e.g. reporting every 1000 records. Essentially this is counting loop repetitions.

I was wondering what approaches I could take to this in clojure where I am mapping a function over my sequence of records. In this case printing the message (and even keeping count of the progress) seem to be essentially side-effects.

What I have come up with so far looks like:

(defn report
  [report-every val cnt]
  (if (= 0 (mod cnt report-every))
    (println "Done" cnt))
    val)

(defn report-progress
  [report-every aseq]
  (map (fn [val cnt] 
          (report report-every val cnt)) 
       aseq 
       (iterate inc 1)))

For example:

user> (doall (report-progress 2 (range 10)))
Done 2
Done 4
Done 6
Done 8
Done 10
(0 1 2 3 4 5 6 7 8 9)

Are there other (better) ways of achieving this effect?

Are there any pitfalls in what I am doing? (I think I am preserving laziness and not holding the head for example.)

Answer 1

The great thing about clojure is you can attach the reporting to the data itself instead of the code that does the computing. This allows you to separate these logically distinct parts. Here is a chunk from my misc.clj that I find I use in just about every project:

(defn seq-counter 
  "calls callback after every n'th entry in sequence is evaluated. 
  Optionally takes another callback to call once the seq is fully evaluated."
  ([sequence n callback]
     (map #(do (if (= (rem %1 n) 0) (callback)) %2) (iterate inc 1) sequence))
  ([sequence n callback finished-callback]
     (drop-last (lazy-cat (seq-counter sequence n callback) 
                  (lazy-seq (cons (finished-callback) ())))))) 

then wrap the reporter around your data and then pass the result to the processing function.

(map process-data (seq-counter inc-progress input))