Question

My application processes a potentially large text file line by line, my code is currently:

(with-open [r (clojure.java.io/reader "large_text_file")]
  (doall
    (map #(process %) (line-seq r)))

In my code, will already processed lines stay in memory? If so, how do I avoid it?

And in general, do consumed lazy sequence elements stay in memory?

Thank you!

Answer 1

The use of doall will retain the head of the seq and return it. The whole seq will reside in memory.

I assume you traverse the seq for doing side effects. Thats the purpose of doseq:

(with-open [r (clojure.java.io/reader "your_large_file.txt")]
  (doseq [l (line-seq r)]
    (println l)))