Question

I’ve looked through On Lisp, Practical Common Lisp and the SO archives in order to answer this on my own, but those attempts were frustrated by my inability to name the concept I’m interested in. I would be grateful if anyone could just tell me the canonical term for this sort of thing.

This question is probably best explained by an example. Let’s say I want to implement Python-style list comprehensions in Common Lisp. In Python I would write:

[x*2 for x in range(1,10) if x > 3]

So I begin by writing down:

(listc (* 2 x) x (range 1 10) (> x 3))

and then defining a macro that transforms the above into the correct comprehension. So far so good.

The interpretation of that expression, however, would be opaque to a reader not already familiar with Python list comprehensions. What I’d really like to be able to write is the following:

(listc (* 2 x) for x in (range 1 10) if (> x 3)) 

but I haven’t been able to track down the Common Lisp terminology for this. It seems that the loop macro does exactly this sort of thing. What is it called, and how can I implement it? I tried macro-expanding a sample loop expression to see how it’s put together, but the resulting code was unintelligible. Could anyone guide me in the right direction?

Thanks in advance.

Answer 1

Well, what for does is essentially, that it parses the forms supplied as its body. For example:

(defmacro listc (expr &rest forms)
  ;; 
  ;;
  ;; (listc EXP for VAR in GENERATOR [if CONDITION])
  ;;
  ;;
  (labels ((keyword-p (thing name)
             (and (symbolp thing)
                  (string= name thing))))
    (destructuring-bind (for* variable in* generator &rest tail) forms
      (unless (and (keyword-p for* "FOR") (keyword-p in* "IN"))
        (error "malformed comprehension"))
      (let ((guard (if (null tail) 't
                       (destructuring-bind (if* condition) tail
                         (unless (keyword-p if* "IF") (error "malformed comprehension"))
                         condition))))
        `(loop 
            :for ,variable :in ,generator 
            :when ,guard 
            :collecting ,expr)))))


(defun range (start end &optional (by 1))
  (loop
     :for k :upfrom start :below end :by by
     :collecting k))

Apart from the hackish “parser” I used, this solution has a disadvantage, which is not easily solved in common lisp, namely the construction of the intermediate lists, if you want to chain your comprehensions:

(listc x for x in (listc ...) if (evenp x))

Since there is no moral equivalent of yield in common lisp, it is hard to create a facility, which does not require intermediate results to be fully materialized. One way out of this might be to encode the knowledge of possible “generator” forms in the expander of listc, so the expander can optimize/inline the generation of the base sequence without having to construct the entire intermediate list at run-time.

Another way might be to introduce “lazy lists” (link points to scheme, since there is no equivalent facility in common lisp — you had to build that first, though it’s not particularily hard).

Also, you can always have a look at other people’s code, in particular, if they tries to solve the same or a similar problem, for example:

• Iterate
• Loop in SBCL
• Pipes (which does the lazy list thing)