Question

I’d like to write a Clojure with-test-tags macro that wraps a bunch of forms, and adds some metadata to the name of each deftest form – specifically, add some stuff to a :tags key, so that I can play with a tool to run tests with a specific tag.

One obvious implementation for with-test-tags is to walk the entire body recursively, modifying each deftest form as I find it. But I’ve been reading Let Over Lambda recently, and he makes a good point: instead of walking the code yourself, just wrap the code in a macrolet and let the compiler walk it for you. Something like:

(defmacro with-test-tags [tags & body]
  `(macrolet [(~'deftest [name# & more#]
                `(~'~'deftest ~(vary-meta name# update-in [:tags] (fnil into []) ~tags)
                   ~@more#))]
     (do ~@body)))

(with-test-tags [:a :b] 
  (deftest x (...do tests...)))

This has the obvious problem, though, that the deftest macro continues to expand recursively forever. I could make it expand to clojure.test/deftest instead, thus avoiding any further recursive expansions, but then I can’t usefully nest instances of with-test-tags to label sub-groups of tests.

At this point, especially for something as simple as deftest, it looks like walking the code myself will be simpler. But I wonder if anyone knows a technique for writing a macro which “slightly modifies” certain subexpressions, without recursing forever.

For the curious: I considered some other approaches, such as having a compile-time binding-able var that I set as I go up and down the code, and using that var when I finally see a deftest, but since each macro only returns a single expansion its bindings won’t be in place for the next call to macroexpand.

Edit

I did the postwalk implementation just now, and while it works it doesn’t respect special forms such as quote – it expands inside of those as well.

(defmacro with-test-tags [tags & body]
  (cons `do
        (postwalk (fn [form]
                    (if (and (seq? form)
                             (symbol? (first form))
                             (= "deftest" (name (first form))))
                      (seq (update-in (vec form) [1]
                                      vary-meta update-in [:tags] (fnil into []) tags))
                      form))
                  body)))

(Also, sorry for possible noise on the common-lisp tag – I thought you might be able to help out with weirder macro stuff even with minimal Clojure experience.)

Answer 1

(This is a new approach, eval– and binding-free. As discussed in
the comments on this answer, the use of eval is problematic because
it prevents tests from closing over the lexical environments they seem
to be defined in (so (let [x 1] (deftest easy (is (= x 1)))) no
longer works). I leave the original approach in the bottom half of the
answer, below the horizontal rule.)

The macrolet approach

Implementation

Tested with Clojure 1.3.0-beta2; it should probably work with 1.2.x as
well.

(ns deftest-magic.core
  (:use [clojure.tools.macro :only [macrolet]]))

(defmacro with-test-tags [tags & body]
  (let [deftest-decl
        (list 'deftest ['name '& 'body]
              (list 'let ['n `(vary-meta ~'name update-in [:tags]
                                         (fnil into #{}) ~tags)
                          'form `(list* '~'clojure.test/deftest ~'n ~'body)]
                    'form))
        with-test-tags-decl
        (list 'with-test-tags ['tags '& 'body]
              `(list* '~'deftest-magic.core/with-test-tags
                      (into ~tags ~'tags) ~'body))]
    `(macrolet [~deftest-decl
                ~with-test-tags-decl]
       ~@body)))

Usage

…is best demonstrated with a suite of (passing) tests:

(ns deftest-magic.test.core
  (:use [deftest-magic.core :only [with-test-tags]])
  (:use clojure.test))

;; defines a test with no tags attached:
(deftest plain-deftest
  (is (= :foo :foo)))

(with-test-tags #{:foo}

  ;; this test will be tagged #{:foo}:
  (deftest foo
    (is true))

  (with-test-tags #{:bar}

    ;; this test will be tagged #{:foo :bar}:
    (deftest foo-bar
      (is true))))

;; confirming the claims made in the comments above:
(deftest test-tags
  (let [plaintest-tags (:tags (meta #'plain-deftest))]
    (is (or (nil? plaintest-tags) (empty? plaintest-tags))))
  (is (= #{:foo} (:tags (meta #'foo))))
  (is (= #{:foo :bar} (:tags (meta #'foo-bar)))))

;; tests can be closures:
(let [x 1]
  (deftest lexical-bindings-no-tags
    (is (= x 1))))

;; this works inside with-test-args as well:
(with-test-tags #{:foo}
  (let [x 1]
    (deftest easy (is true))
    (deftest lexical-bindings-with-tags
      (is (= #{:foo} (:tags (meta #'easy))))
      (is (= x 1)))))

Design notes:

1. We want to make the macrolet-based design described in the
   question text work. We care about being able to nest
   with-test-tags and preserving the possibility of defining tests
   whose bodies close over the lexical environments they are defined
   in.

2. We will be macroletting deftest to expand to a
   clojure.test/deftest form with appropriate metadata attached to
   the test’s name. The important part here is that with-test-tags
   injects the appropriate tag set right into the definition of the
   custom local deftest inside the macrolet form; once the
   compiler gets around to expanding the deftest forms, the tag sets
   will have been hardwired into the code.

3. If we left it at that, tests defined inside a nested
   with-test-tags would only get tagged with the tags passed to the
   innermost with-test-tags form. Thus we have with-test-tags also
   macrolet the symbol with-test-tags itself behaving much like
   the local deftest: it expands to a call to the top-level
   with-test-tags macro with the appropriate tags injected into the
   tagset.

4. The intention is that the inner with-test-tags form in

   (with-test-tags #{:foo}
     (with-test-tags #{:bar}
       ...))
   

   expand to (deftest-magic.core/with-test-tags #{:foo :bar} ...)
   (if indeed deftest-magic.core is the namespace with-test-tags
   is defined in). This form immediately expands into the familiar
   macrolet form, with the deftest and with-test-tags symbols
   locally bound to macros with the correct tag sets hardwired inside
   them.

---

(The original answer updated with some notes on the design, some
rephrasing and reformatting etc. The code is unchanged.)

The binding + eval approach.

(See also https://gist.github.com/1185513 for a version
additionally using macrolet to avoid a custom top-level
deftest.)

Implementation

The following is tested to work with Clojure 1.3.0-beta2; with the
^:dynamic part removed, it should work with 1.2:

(ns deftest-magic.core)

(def ^:dynamic *tags* #{})

(defmacro with-test-tags [tags & body]
  `(binding [*tags* (into *tags* ~tags)]
     ~@body))

(defmacro deftest [name & body]
  `(let [n# (vary-meta '~name update-in [:tags] (fnil into #{}) *tags*)
         form# (list* 'clojure.test/deftest n# '~body)]
     (eval form#)))

Usage

(ns example.core
  (:use [clojure.test :exclude [deftest]])
  (:use [deftest-magic.core :only [with-test-tags deftest]]))

;; defines a test with an empty set of tags:
(deftest no-tags
  (is true))

(with-test-tags #{:foo}

  ;; this test will be tagged #{:foo}:
  (deftest foo
    (is true))

  (with-test-tags #{:bar}

    ;; this test will be tagged #{:foo :bar}:
    (deftest foo-bar
      (is true))))

Design notes

I think that on this occasion a judicious use of eval leads to a
useful solution. The basic design (based on the “binding-able Var”
idea) has three components:

1. A dynamically bindable Var — *tags* — which is bound at compile
   time to a set of tags to be used by deftest forms to decorate the
   tests being defined. We add no tags by default, so its initial
   value is #{}.

2. A with-test-tags macro which installs an appropriate for
   *tags*.

3. A custom deftest macro which expands to a let form resembling
   this (the following is the expansion, slightly simplified for
   clarity):

   (let [n    (vary-meta '<NAME> update-in [:tags] (fnil into #{}) *tags*)
         form (list* 'clojure.test/deftest n '<BODY>)]
     (eval form))
   

   <NAME> and <BODY> are the arguments given to the custom
   deftest, inserted in the appropriate spots through unquoting the
   appropriate parts of the syntax-quoted expansion template.

Thus the expansion of the custom deftest is a let form in which,
first, the name of the new test is prepared by decorating the given
symbol with the :tags metadata; then a clojure.test/deftest form
using this decorated name is constructed; and finally the latter form
is handed to eval.

The key point here is that the (eval form) expressions here are
evaluated whenever the namespace their contained in is AOT-compiled or
required for the first time in the lifetime of the JVM running this
code. This is exactly the same as the (println "asdf") in a
top-level (def asdf (println "asdf")), which will print asdf
whenever the namespace is AOT-compiled or required for the first
time; in fact, a top-level (println "asdf") acts similarly.

This is explained by noting that compilation, in Clojure, is just
evaluation of all top-level forms. In (binding [...] (deftest ...),
binding is the top-level form, but it only returns when deftest
does, and our custom deftest expands to a form which returns when
eval does. (On the other hand, the way require executes top-level
code in already-compiled namespaces — so that if you have (def t
(System/currentTimeMillis)) in your code, the value of t will
depend on when you require your namespace rather than on when it was
compiled, as can be determined by experimenting with AOT-compiled code
— is just the way Clojure works. Use read-eval if you want actual
constants embedded in code.)

In effect, the custom deftest runs the compiler (through eval) at
the run-time-at-compile-time of macro expansion. Fun.

Finally, when a deftest form is put inside a with-test-tags form,
the form of (eval form) will have been prepared with the bindings
installed by with-test-tags in place. Thus the test being defined
will be decorated with the appropriate set of tags.

At the REPL

user=> (use 'deftest-magic.core '[clojure.test :exclude [deftest]])
nil
user=> (with-test-tags #{:foo}
         (deftest foo (is true))
         (with-test-tags #{:bar}
           (deftest foo-bar (is true))))
#'user/foo-bar
user=> (meta #'foo)
{:ns #<Namespace user>,
 :name foo,
 :file "NO_SOURCE_PATH",
 :line 2,
 :test #<user$fn__90 user$fn__90@50903025>,
 :tags #{:foo}}                                         ; <= note the tags
user=> (meta #'foo-bar)
{:ns #<Namespace user>,
 :name foo-bar,
 :file "NO_SOURCE_PATH",
 :line 2,
 :test #<user$fn__94 user$fn__94@368b1a4f>,
 :tags #{:foo :bar}}                                    ; <= likewise
user=> (deftest quux (is true))
#'user/quux
user=> (meta #'quux)
{:ns #<Namespace user>,
 :name quux,
 :file "NO_SOURCE_PATH",
 :line 5,
 :test #<user$fn__106 user$fn__106@b7c96a9>,
 :tags #{}}                                             ; <= no tags works too

And just to be sure working tests are being defined…

user=> (run-tests 'user)

Testing user

Ran 3 tests containing 3 assertions.
0 failures, 0 errors.
{:type :summary, :pass 3, :test 3, :error 0, :fail 0}