Question

I’m trying to run an external program in SBCL and capture its output.
The output is binary data (a png image), while SBCL insists on interpreting it as strings.

I tried a number of ways, like

(trivial-shell:shell-command "/path/to/png-generator" :input "some input")

(with-input-from-string (input "some input")
  (with-output-to-string (output)
    (run-program "/path/to/png-generator" () :input input :output output))


(with-input-from-string (input "some input")
  (flexi-streams:with-output-to-sequence (output)
    (run-program "/path/to/png-generator" () :input input :output output))

But I get errors like

Illegal :UTF-8 character starting at byte position 0.

It seems to me that SBCL is trying to interpret the binary data as a text and decode it. How do I change this behaviour ? I’m interested only in obtaining a vector of octets.

Edit: Since it is not clear from the text above, I’d like to add that at least in the case of flexi-stream, the element-type of the stream is a flexi-streams:octect (which is a (unsigned-byte 8)).
I would expect at least in this case run-program to read the raw bytes without many issues. Instead I get a message like Don't know how to copy to stream of element-type (UNSIGNED-BYTE 8)

Answer 1

Edit: I got angry at not being able to do this very simple task and solved the problem.

Functionally, the ability to send a stream of type UNSIGNED-BYTE into run-program and have it work correctly is severely limited, for reasons I don’t understand. I tried gray streams, flexi-streams, fd streams, and a few other mechanisms, like you.

However, perusing run-program’s source (for the fifth or sixth time), I noticed that there’s an option :STREAM you can pass to output. Given that, I wondered if read-byte would work… and it did. For more performant work, one could determine how to get the length of a non-file stream and run READ-SEQUENCE on it.

(let* 
       ;; Get random bytes
      ((proc-var (sb-ext:run-program "head" '("-c" "10" "/dev/urandom")
                                     :search t
       ;; let SBCL figure out the storage type. This is what solved the problem.
                                     :output :stream))
       ;; Obtain the streams from the process object.
       (output (process-output proc-var))
       (err (process-error proc-var)))
  (values
   ;;return both stdout and stderr, just for polish.
   ;; do a byte read and turn it into a vector.
   (concatenate 'vector
                ;; A byte with value 0 is *not* value nil. Yay for Lisp!
                (loop for byte = (read-byte output nil)
                   while byte
                   collect byte))
   ;; repeat for stderr
   (concatenate 'vector
                (loop for byte = (read-byte err nil)
                   while byte
                   collect byte))))