lucid.query intuitive search for code

Author: Chris Zheng  (z@caudate.me)
Date: 29 June 2017
Repository: https://www.github.com/zcaudate/lucidity
Version: 1.3.13

1    Introduction

lucid.query makes it easy for querying and manipulation of clojure source code through an xpath/css-inspired syntax. This library was originally developed as jai.

  • to simplify traversal and manipulation of source code
  • to provide higher level abstractions on top of rewrite-clj
  • to leverage core.match's pattern matching for a more declarative syntax

1.1    Installation

Add to project.clj dependencies:

[im.chit/lucid.query "1.3.13"]

All functionality is in the lucid.query namespace:

(use 'lucid.query)

2    API



^

select and manipulation of clojure source code

v 1.2
(defmacro $
  [context path & args]
  `($* ~context (quote ~path) ~@args))
link
($ {:string "(defn hello1) (defn hello2)"} [(defn _ ^:%+ (keyword "oeuoeuoe"))]) => '[(defn hello1 :oeuoeuoe) (defn hello2 :oeuoeuoe)] ($ {:string "(defn hello1) (defn hello2)"} [(defn _ | ^:%+ (keyword "oeuoeuoe") )]) => '[:oeuoeuoe :oeuoeuoe] (->> ($ {:string "(defn hello1) (defn hello2)"} [(defn _ | ^:%+ (keyword "oeuoeuoe"))] {:return :string}) ) => [":oeuoeuoe" ":oeuoeuoe"] ($ (source/of-string "a b c") [{:is a}]) => '[a]

match ^

matches the source code

v 1.2
(defn match
  [zloc selector]
  (let [match-fn (-> selector
                     (compile/expand-all-metas)
                     (common/prepare-deletion)
                     (match/compile-matcher))]
    (try (match-fn zloc)
         (catch Throwable t false))))
link
(match (source/of-string "(+ 1 1)") '(symbol? _ _)) => false (match (source/of-string "(+ 1 1)") '(^:% symbol? _ _)) => true (match (source/of-string "(+ 1 1)") '(^:%- symbol? _ | _)) => true (match (source/of-string "(+ 1 1)") '(^:%+ symbol? _ _)) => false

modify ^

modifies location given a function

v 1.2
(defn modify
  ([zloc selectors func] (modify zloc selectors func nil))
  ([zloc selectors func opts]
   (let [[match-map [cidx ctype cform]] (compile/prepare selectors)
         match-fn (match/compile-matcher match-map)
         walk-fn (case (:walk opts)
                   :top walk/topwalk
                   walk/matchwalk)]
     (walk-fn zloc
              [match-fn]
              (fn [zloc]
                (if (= :form ctype)
                  (let [{:keys [level source]} (traverse/traverse zloc cform)
                        nsource (func source)]
                    
                    (if (or (nil? level) (= level 0))
                      nsource
                      (nth (iterate source/up nsource) level)))
                  (func zloc)))))))
link
(source/root-string (modify (source/of-string "^:a (defn hello3) (defn hello)") ['(defn | _)] (fn [zloc] (source/insert-left zloc :hello)))) => "^:a (defn :hello hello3) (defn :hello hello)"

select ^

selects all patterns from a starting point

v 1.2
(defn select
  ([zloc selectors] (select zloc selectors nil))
  ([zloc selectors opts]
   (let [[match-map [cidx ctype cform]] (compile/prepare selectors)
         match-fn (match/compile-matcher match-map)
         walk-fn (case (:walk opts)
                   :top walk/topwalk
                   walk/matchwalk)]
     (let [atm  (atom [])]
       (walk-fn zloc
                [match-fn]
                (fn [zloc]
                  (swap! atm conj 
                         (if (= :form ctype)
                           (:source (traverse/traverse zloc cform))
                           zloc))
                  zloc))
       (if (:first opts)
         (first @atm)
         @atm)))))
link
(map source/sexpr (select (source/of-string "(defn hello [] (if (try))) (defn hello2 [] (if (try)))") '[defn if try])) => '((defn hello [] (if (try))) (defn hello2 [] (if (try))))

traverse ^

uses a pattern to traverse as well as to edit the form

v 1.2
(defn traverse
  ([zloc pattern]
   (let [pattern (compile/expand-all-metas pattern)]
     (:source (traverse/traverse zloc pattern))))
  ([zloc pattern func]
   (let [pattern (compile/expand-all-metas pattern)
         {:keys [level source]} (traverse/traverse zloc pattern)
         nsource (func source)]
     (if (or (nil? level) (= level 0))
       nsource
       (nth (iterate source/up nsource) level)))))
link
(source/sexpr (traverse (source/of-string "^:a (+ () 2 3)") '(+ () 2 3))) => '(+ () 2 3) (source/sexpr (traverse (source/of-string "()") '(^:&+ hello))) => '(hello) (source/sexpr (traverse (source/of-string "()") '(+ 1 2 3))) => throws (source/sexpr (traverse (source/of-string "(defn hello "world" {:a 1} [])") '(defn ^:% symbol? ^:?%- string? ^:?%- map? ^:% vector? & _))) => '(defn hello [])

3    Usage

We first define a code fragment to query on. The library currently works with strings and files.

(def fragment {:string "(defn hello [] (println \"hello\"))\n
                        (defn world [] (if true (prn \"world\")))"})

3.1    Basics

Find all the defn forms:

($ fragment [defn])
=> '[(defn hello [] (println "hello"))
     (defn world [] (if true (prn "world")))]

Find all the if forms

($ fragment [if])
=> '((if true (prn "world")))

3.2    Path

Find all the defn forms that contain an if form directly below it:

($ fragment [defn if])
=> '[(defn world [] (if true (prn "world")))]

Find all the defn forms that contains a prn form anywhere in its body

($ fragment [defn :* prn])
=> '[(defn world [] (if true (prn "world")))]

Depth searching at specific levels can also be done, the following code performs a search for prn at the second and third level forms below the defn:

($ fragment [defn :2 prn])
=> '[(defn world [] (if true (prn "world")))]
  
($ fragment [defn :3 prn])
=> '[]

3.3    Representation

Instead of returning an s-expression, we can also return other represetations through specifying the :return value on the code. The options are :zipper, :sexpr or :string.

By default, querying returns a :sexpr representation

($ (assoc fragment :return :sexpr) [defn :* prn])
=> '[(defn world [] (if true (prn "world")))]

String representations are useful for directly writing to file

($ fragment [defn :* prn] {:return :string})
=> ["(defn world [] (if true (prn \"world\")))"]

If more manipulation is needed, then returning a zipper allows composablity with rewrite-clj

(->> ($ fragment [defn :* prn] {:return :zipper})
     (map z/sexpr))
=> '[(defn world [] (if true (prn "world")))]

3.4    Cursors

It is not very useful just selecting top-level forms. We need a way to move between the sections. This is where cursors come into picture. We can use | to set access to selected forms. For example, we can grab the entire top level form like this:

($ fragment [defn println])
=> '[(defn hello [] (println "hello"))]

But usually, the more common scenario is that we wish to perform a particular action on the (println ...) form. This is accessible by adding "|" in front of the println symbol:

($ fragment [defn | println])
=> '[(println "hello")]

We can see how the cursor works by drilling down into our code fragment:

($ fragment [defn if prn])
=> '[(defn world [] (if true (prn "world")))]
  
($ fragment [| defn if prn])
=> '[(defn world [] (if true (prn "world")))]
  
($ fragment [defn | if prn])
=> '[(if true (prn "world"))]
  
($ fragment [defn if | prn])
=> '[(prn "world")]

3.5    Fine Grain Control

It is not enough that we can walk to a particular form, we have to be able to control the place within the form that we wish to traverse to.

($ fragment [defn (if | _ & _)])
=> '[true]
  
($ fragment [defn (if _ | _)])
=> '[(prn "world")]
  
($ fragment [defn if (prn | _)])
=> '["world"]

3.6    Pattern Matching

We can also use a pattern expressed using a list. Defining a pattern allows matched elements to be expressed more intuitively:

($ fragment [(defn & _)])
=> '[(defn hello [] (println "hello"))
     (defn world [] (if true (prn "world")))]
  
($ fragment [(defn hello & _)])
=> '[(defn hello [] (println "hello"))]

A pattern can have nestings:

($ fragment [(defn world [] (if & _))])
=> '[(defn world [] (if true (prn "world")))]

If functions are needed, the symbols can be tagged with the a meta ^:%

($ fragment [(defn world ^:% vector? ^:% list?)])
=> '[(defn world [] (if true (prn "world")))]

The queries are declarative and should be quite intuitive to use

($ fragment [(_ _ _ (if ^:% true? & _))])
=> '[(defn world [] (if true (prn "world")))]

3.7    Insertion

We can additionally insert elements by tagging with the ^:+ meta:

($ fragment [(defn world _ ^:+ (prn "hello") & _)])
=> '[(defn world [] (prn "hello") (if true (prn "world")))]

There are some values that do not allow metas tags (strings, keywords and number), in this case the workaround is to use the ^:%+ meta and write the object as an expression to be evaluated. Note the writing :%+ is the equivalent of writing ^{:% true :+ true}

($ fragment [(defn world _ (if true (prn ^:%+ (keyword "hello") _)))])
=> '[(defn world [] (if true (prn :hello "world")))]

Insertions also work seamlessly with cursors:

($ fragment [(defn world _ (if true | (prn ^:%+ (long 2) _)))])
=> '[(prn 2 "world")]

3.8    Deletion

We can delete values by using the ^:- meta tag. When used on the code fragment, we can see that the function has been mangled as the first two elements have been deleted:

($ fragment [(defn ^:- world  ^:- _ & _)])
=> '[(defn (if true (prn "world")))]

Entire forms can be marked for deletion:

($ fragment [(defn world _ ^:- (if & _))])
=> '[(defn world [])]

Deletions and insertions work quite well together. For example, below shows the replacement of the function name from world to world2:

($ fragment [(defn ^:- world _ ^:+ world2 & _)])
=> '[(defn [] world2 (if true (prn "world")))]

3.9    Optional Matches

There are certain use cases when source code has optional parameters such as a docstring or a meta map.

($ fragment [(defn ^:% symbol? ^:%?- string? ^:%?- map? ^:% vector? & _)])
=> '[(defn hello [] (println "hello"))
     (defn world [] (if true (prn "world")))]

We can use optional matches to clean up certain elements within the form, such as being able to remove docstrings and meta maps if they exist.

($ {:string "(defn add \"adding numbers\" {:added \"0.1\"} [x y] (+ x y))"}
   [(defn ^:% symbol? ^:%?- string? ^:%?- map? ^:% vector? & _)]
   {:return :string})
=> ["(defn add [x y] (+ x y))"]

4    Utilities

These utilities are specially designed to work with rewrite-clj;

(use rewrite-clj.zip :as z)

4.1    traverse

While the $ macro is provided for global searches within a file, traverse is provided to work with the zipper library for traversal/manipulation of a form.

(-> (z/of-string "(defn add \"adding numbers\" {:added \"0.1\"} [x y] (+ x y))")
    (traverse '(defn ^:% symbol? ^:%?- string? ^:%?- map? ^:% vector? | & _))
    (z/insert-left '(prn "add"))
    (z/up)
    (z/sexpr))
=> '(defn add [x y] (prn "add") (+ x y))

traverse can also be given a function as the third argument. This will perform some action on the location given by the cursor and then jump out again:

(-> (z/of-string "(defn add \"adding numbers\" {:added \"0.1\"} [x y] (+ x y))")
    (traverse '(defn ^:% symbol? ^:%?- string? ^:%?- map? ^:% vector? | & _)
              (fn [zloc] (z/insert-left zloc '(prn "add"))))
    (z/sexpr))
=> '(defn add [x y] (prn "add") (+ x y))

traverse works with metas as well, which is harder to work with using just rewrite-clj

(-> (z/of-string "(defn add [x y] ^{:text 0} (+ (+ x 1) y 1))")
    (traverse '(defn _ _ (+ (+ | x 1) y 1))
              (fn [zloc] (z/insert-left zloc '(prn "add"))))
    (z/sexpr))
=> '(defn add [x y] (+ (+ (prn "add") x 1) y 1))

4.2    match

a map-based syntax is provided for matching:

(-> (z/of-string "(if true (+ 1 2) (+ 1 1))")
    (match 'if))
=> true
  
(-> (z/of-string "(if true (+ 1 2) (+ 1 1))")
    (match {:form 'if}))
=> true
  
(-> (z/of-string "(if true (+ 1 2) (+ 1 1))")
    (match {:is list?}))
=> true
  
(-> (z/of-string "(if true (+ 1 2) (+ 1 1))")
    (match {:child {:is true}}))
=> true
  
(-> (z/of-string "(if true (+ 1 2) (+ 1 1))")
    (match {:child {:form '+}}))
=> true

5    Match Element

There are many options for matches:

  • :fn match on checking function
  • :is match on value or checking function
  • :or match two options, done using a set
  • :equal match on equivalence
  • :type match on rewrite-clj type
  • :meta match on meta tag
  • :form match on first element of a form
  • :pattern match on a pattern
  • :code match on code

5.1    :fn

The most general match, takes a predicate dispatching on a zipper location

(-> (z/of-string "(if true (+ 1 2) (+ 1 1))")
    (match {:fn (fn [zloc] (= :list (z/tag zloc)))}))
=> true

5.2    :is

The most general match, takes a value or a function

(-> (z/of-string "(if true (+ 1 2) (+ 1 1))")
    (match {:child {:is true}}))
=> true
  
(-> (z/of-string "(if true (+ 1 2) (+ 1 1))")
    (match {:child {:is (fn [x] (instance? Boolean x))}}))
=> true

5.3    :form

By default, a symbol is evaluated as a :form'

(-> (z/of-string "(if true (+ 1 2) (+ 1 1))")
    (match 'if))
=> true

It can also be expressed explicitly:

(-> (z/of-string "(if true (+ 1 2) (+ 1 1))")
    (match '{:form if}))
=> true

5.4    :or

or style matching done using set notation

(-> (z/of-string "(if true (+ 1 2) (+ 1 1))")
    (match '#{{:form if} {:form defn}}))
=> true
  
(-> (z/of-string "(if true (+ 1 2) (+ 1 1))")
    (match '#{if defn}))
=> true

if need arises to match a set, use the ^& meta tag

(-> (z/of-string "(if #{:a :b :c} (+ 1 2) (+ 1 1))")
    (match {:child {:is '^& #{:a :b :c}}}))
=> true

5.5    :and

similar usage to :or except that vector notation is used:

(-> (z/of-string "(if true (+ 1 2) (+ 1 1))")
    (match '[if defn]))
=> false
  
(-> (z/of-string "(if true (+ 1 2) (+ 1 1))")
    (match '[if {:contains 1}]))
=> true

5.6    :equal

matches sets, vectors and maps as is

(-> (z/of-string "(if #{:a :b :c} (+ 1 2) (+ 1 1))")
    (match {:child {:equal #{:a :b :c}}}))
=> true
  
(-> (z/of-string "(if {:a 1 :b 2} (+ 1 2) (+ 1 1))")
    (match {:child {:equal {:a 1 :b 2}}}))
=> true

5.7    :type

predicate on the rewrite-clj reader type

(-> (z/of-string "(if true (+ 1 2) (+ 1 1))")
    (match {:type :list}))
=> true
  
(-> (z/of-string "(if true (+ 1 2) (+ 1 1))")
    (match {:child {:type :token}}))
=> true

5.8    :meta

matches the meta on a location

(-> (z/down (z/of-string "^:a (+ 1 1)"))
    (match {:meta :a}))
=> true
  
(-> (z/down (z/of-string "^{:a true} (+ 1 1)"))
    (match {:meta {:a true}}))
=> true

5.9    :pattern

pattern matches are done automatically with a list

(-> (z/of-string "(if true (+ 1 2) (+ 1 1))")
    (match '(if true & _)))
=> true

but they can be made more explicit:

(-> (z/of-string "(if true (+ 1 2) (+ 1 1))")
    (match {:pattern '(if true & _)}))
=> true

6    Match Position

The positional options for matches are:

  • :parent match on direct parent of element
  • :child match on any child of element
  • :first match on first child of element
  • :last match on last child of element
  • :nth match on nth child of element
  • :nth-left match on nth-sibling to the left of element
  • :nth-right match on nth-sibling to the right of element
  • :nth-ancestor match on the ancestor that is n levels higher
  • :nth-contains match on any contained element that is n levels lower
  • :ancestor match on any ancestor
  • :contains match on any contained element
  • :sibling match on any sibling
  • :left match on node directly to left
  • :right match on node directly to right
  • :left-of match on node to left
  • :right-of match on node to right
  • :left-most match is element is the left-most element
  • :right-most match is element is the right-most element

6.1    :parent

matches on the parent form

(-> (z/of-string "(if true (+ 1 2) (+ 1 1))")
    (z/down)
    (z/right)
    (match {:parent 'if}))
=> true

6.2    :child

matches on any of the child forms

(-> (z/of-string "(if true (+ 1 2) (+ 1 1))")
    (match {:child '(+ _ 2)}))
=> true

6.3    :first

matches on the first child, can also be a vector

(-> (z/of-string "(if [1 2 3] (+ 1 2) (+ 1 1))")
    (match {:child {:first '+}}))
=> true
  
(-> (z/of-string "(if [1 2 3] (+ 1 2) (+ 1 1))")
    (match {:child {:first 1}}))
=> true

6.4    :last

matches on the last child element

(-> (z/of-string "(if [1 2 3] (+ 1 2) (+ 1 1))")
    (match {:child {:last 3}}))
=> true
  
(-> (z/of-string "(if [1 2 3] (+ 1 2) (+ 1 1))")
    (match {:child {:last 1}}))
=> true

6.5    :nth

matches the nth child

(-> (z/of-string "(if [1 2 3] (+ 1 2) (+ 1 1))")
    (match {:nth [1 {:equal [1 2 3]}]}))
=> true

6.6    :nth-left

matches the nth sibling to the left

(-> (z/of-string "(if [1 2 3] (+ 1 2) (+ 1 1))")
    (z/down)
    (z/rightmost)
    (match {:nth-left [2 {:equal [1 2 3]}]}))
=> true

6.7    :nth-right

matches the nth sibling to the right

(-> (z/of-string "(if [1 2 3] (+ 1 2) (+ 1 1))")
    (z/down)
    (match {:nth-right [1 {:equal [1 2 3]}]}))
=> true

6.8    :nth-ancestor

matches the nth ancestor in the hierarchy

(-> (z/of-string "(if [1 2 3] (+ 1 2) (+ 1 1))")
    (z/down)
    (z/right)
    (z/down)
    (match {:nth-ancestor [2 {:form 'if}]}))
=> true

6.9    :nth-contains

matches the nth level children

(-> (z/of-string "(if [1 2 3] (+ 1 2) (+ 1 1))")
    (match {:nth-contains [2 {:is 3}]}))
=> true

6.10    :ancestor

matches any ancestor

(-> (z/of-string "(if [1 2 3] (+ 1 2) (+ 1 1))")
    (z/down)
    (z/right)
    (z/down)
    (match {:ancestor 'if}))
=> true

6.11    :contains

matches the any subelement contained by the element

(-> (z/of-string "(if [1 2 3] (+ 1 2) (+ 1 1))")
    (match {:contains 3}))
=> true

6.12    :sibling

matches any sibling

(-> (z/of-string "(if [1 2 3] (+ 1 2) (+ 1 1))")
    (z/down)
    (match {:sibling {:form '+}}))
=> true

6.13    :left

matches element to the left

(-> (z/of-string "(if [1 2 3] (+ 1 2) (+ 1 1))")
    (z/down)
    (z/right)
    (match {:left {:is 'if}}))
=> true

6.14    :right

matches element to the right

(-> (z/of-string "(if [1 2 3] (+ 1 2) (+ 1 1))")
    (z/down)
    (match {:right {:is [1 2 3]}}))
=> true

6.15    :left-of

matches any element to the left

(-> (z/of-string "(if [1 2 3] (+ 1 2) (+ 1 1))")
    (z/down)
    (z/rightmost)
    (match {:left-of {:is [1 2 3]}}))
=> true

6.16    :right-of

matches any element to the right

(-> (z/of-string "(if [1 2 3] (+ 1 2) (+ 1 1))")
    (z/down)
    (match {:right-of '(+ 1 1)}))
=> true

6.17    :left-most

is the left-most element

(-> (z/of-string "(if [1 2 3] (+ 1 2) (+ 1 1))")
    (z/down)
    (match {:left-most true}))
=> true

6.18    :right-most

is the right-most element

(-> (z/of-string "(if [1 2 3] (+ 1 2) (+ 1 1))")
    (z/down)
    (z/rightmost)
    (match {:right-most true}))
=> true