Question

How does one process large binary data files in Clojure? Let’s assume data/files are about 50MB – small enough to be processed in memory (but not with a naive implementation).

The following code correctly removes ^M from small files but it throws OutOfMemoryError for larger files (like 6MB):

(defn read-bin-file [file]
  (to-byte-array (as-file file)))

(defn remove-cr-from-file [file]
  (let [dirty-bytes (read-bin-file file)
        clean-bytes (filter #(not (= 13 %)) dirty-bytes)
        changed?    (< (count clean-bytes) (alength dirty-bytes))]    ; OutOfMemoryError
    (if changed?
      (write-bin-file file clean-bytes))))    ; writing works fine

It seems that Java byte arrays can’t be treated as seq as it is extremely inefficient.

On the other hand, solutions with aset, aget and areduce are bloated, ugly and imperative because you can’t really use Clojure sequence library.

What am I missing? How does one process large binary data files in Clojure?

Answer 1

I would probably personally use aget / aset / areduce here – they may be imperative but they are useful tools when dealing with arrays, and I don’t find them particularly ugly. If you want to wrap them in a nice function then of course you can 🙂

If you are determined to use sequences, then your problem will be in the construction and traversal of the seq since this will require creation and storage of a new seq object for every byte in the array. This is probably ~24 bytes for each array byte……

So the trick is to get it to work lazily, in which case the earlier objects will be garbage collected before you get to the end of the the array. However to make this work, you’ll have to avoid holding any reference to the head of the seq when you traverse the sequence (e.g. with count).

The following might work (untested), but will depend on write-bin-file being implemented in a lazy-friendly manner:

(defn remove-cr-from-file [file]
  (let [dirty-bytes (read-bin-file file)
        clean-bytes (filter #(not (= 13 %)) dirty-bytes)
        changed-bytes (count (filter #(not (= 13 %)) dirty-bytes))
        changed?    (< changed-bytes (alength dirty-bytes))]   
    (if changed?
      (write-bin-file file clean-bytes))))

Note this is essentially the same as your code, but constructs a separate lazy sequence to count the number of changed bytes.