lucid.publish generate documentation from code

Author: Chris Zheng  (z@caudate.me)
Date: 29 June 2017
Repository: https://www.github.com/zcaudate/lucidity
Version: 1.3.13

1    Introduction

lucid.publish facilitates the creation of 'documentation that we can run', the tool allows for a design-orientated workflow for the programming process, blurring the boundaries between design, development, testing and documentation. This library was originally developed as lein-midje-doc, and then hydrox.

1.1    Installation

Add to project.clj dependencies:

[im.chit/lucid.publish "1.3.13"]

All functionality is in the lucid.publish namespace:

(use 'lucid.publish)

1.2    Motivation

Documentation is the programmers' means of communicating how to use a library. There are various tools for documentation like latex, wiki and markdown, however lucid.publish should be thought of as an in-built html renderer for clojure code. It's features are:

  1. To generate .html documentation from a .clj test file.
  2. Render code and test cases as examples
  3. Latex-like numbering and linking facilities
  4. To promote code as communication

1.3    Walkthrough

Any clojure file can be made into a .html page. Let's give it a go. Create a file in test/example/sample_document.clj and copy and paste the following:

^{:site "hello"
  :title "world"
  :subtitle "this is a sample document"}
(ns example.sample-document
  (:use hara.test)
  (:require [lucid.publish :as publish]))
  
[[:chapter {:tag "hello" :title "Introduction"}]]
  
"This is an introduction to writing with **Lucidity**"

[[:section {:title "Defining a function"}]]

"We define function `add-5` below:"

[[{:numbered false}]]
(defn add-5 [x]
  (+ x 5))

[[:section {:title "Testing a function"}]]

"`add-5` outputs the following results:"

[[{:tag "add-5-1" :title "1 add 5 = 6"}]]
(fact (add-5 1) => 6)

[[{:tag "add-5-10" :title "10 add 5 = 15"}]]
(fact (add-5 10) => 15)

[[:chapter {:tag "two" :title "Another Chapter"}]]

[[{:title "map"}]]
(comment
  (map inc (range 10))
  => (1 2 3 4 5 6 7 8 9 10)) 

[[{:hidden true}]]
(comment
  (publish/publish))

Now either C-x-C-e on the (publish/publish) line or write in the repl:

(require '[lucid.publish :as publish])

(publish/publish 'example.sample-document)

The output will look something like this

2    API



copy-assets ^

copies all theme assets into the output directory

v 1.2
(defn copy-assets
  ([]
   (let [project (project/project)
         theme (-> project :publish :theme)
         settings (theme/load-settings theme)]
     (copy-assets settings project)))
  ([settings project]
   (let [template-dir (theme/template-path settings project)
         output-dir (output-path project)]
     (prn template-dir output-dir settings)
     (doseq [entry (:copy settings)]
       (prn entry)
       (let [dir   (fs/path template-dir entry)
             files (->> (fs/select dir)
                        (filter fs/file?))]
         (doseq [in files]
           (let [out (fs/path output-dir (str (fs/relativize dir in)))]
             (fs/create-directory (fs/parent out))
             (fs/copy-single in out {:options [:replace-existing :copy-attributes]}))))))))
link
;; copies theme using the `:copy` key into an output directory (copy-assets)

load-settings ^

copies all theme assets into the output directory

v 1.2
(defn load-settings
  ([] (load-settings {} (project/project)))
  ([opts project]
   (let [theme (or (:theme opts)
                   (-> project :publish :theme))
         settings (merge (theme/load-settings theme project)
                         opts)]
     (when (:refresh settings)
       (theme/deploy settings project)
       (copy-assets settings project))
     settings)))
link
;; {:email "z@caudate.me", :date "06 October 2016" ...} (load-settings)

publish ^

publishes a document as an html

v 1.2
(defn publish
  ([] (publish [*ns*] {} (project/project)))
  ([x] (cond (map? x)
             (publish [*ns*] x (project/project))

             :else
             (publish x {} (project/project))))
  ([inputs opts project]
   (let [project (add-lookup project)
         settings (load-settings opts project)
         inputs (if (vector? inputs) inputs [inputs])
         ns->symbol (fn [x] (if (instance? clojure.lang.Namespace x)
                              (.getName x)
                              x))
         inputs (map ns->symbol inputs)
         interim (prepare/prepare inputs project)
         names (-> interim :articles keys)
         out-dir (fs/path (-> project :root)
                          (or (-> project :publish :output) *output*))]
     (fs/create-directory out-dir)
     (doseq [name names]
       (spit (str (fs/path (str out-dir) (str name ".html")))
             (render/render interim name settings project))))))
link
;; publishes the `index` entry in `project.clj` (publish "index") ;; publishes `index` in a specific project with additional options (publish "index" {:refresh true :theme "bolton"} (project/project ))

publish-all ^

publishes all documents as html

v 1.2
(defn publish-all
  ([] (publish-all {} (project/project)))
  ([opts project]
   (let [project  (add-lookup project)
         settings (load-settings opts project)
         template (theme/template-path settings project)
         output   (output-path project)
         files (-> project :publish :files keys sort)]
     (doseq [file files]
       (println "PUBLISH:" file)
       (time (publish [file] settings project))))))
link
;; publishes all the entries in `:publish :files` (publish-all) ;; publishes all entries in a specific project (publish-all {:refresh true :theme "bolton"} (project/project ))

3    Syntax

Elements are constructed using a tag and a map contained within double square brackets. Elements tags have been inspired from latex:

Clojure strings are treated as paragraph elements whilst clojure forms are treated as code elements. fact and comment forms are also considered code elements. Elements will be described in detail in their respective sections.

3.1    Notation

e.3.1  -  Element Notation

[[<tag> {<key1> <value1>, <key2> <value2>}]]

for example:

[[:chapter {:title "Hello World" :tag "hello"}]]

3.2    Attributes

Attribute add additional metadata to elements. They are written as a single hashmap within double square brackets. Attributes mean nothing by themselves. They change the properties of elements directly after them.

[[{:tag "my-paragraph"}]]
[[:paragraph {:content "This is a paragraph"}]]

is equivalent to:

[[:paragraph {:content "This is a paragraph"
              :tag "my-paragraph"}]]

Multiple attributes can be stacked to modify an element:

[[{:numbered false}]]
[[{:lang "shell"}]]
(comment
  > lein repl)

displays the following:

> lein repl

4    Content

Content elements include :paragraph, :image, and file elements.

4.1    :paragraph

Paragraph elements should make up the bulk of the documentation. They can be written as an element or in the usual case, as a string. The string is markdown with templating - so that chapter, section, code and image numbers can be referred to by their tags.

e.4.1  -  Paragraph Element

[[:paragraph {:content "Here is some content"}]]

e.4.2  -  Paragraph String

"Here is some content"

e.4.3  -  Markdown String

[[:chapter {:title "Chapter Heading" :tag "ch-heading"}]]

"
# Heading One
Here is some text.
Here is a tag reference to Chapter Heading - {{ch-heading}}

- Here is a bullet point
- Here is another one"

4.2    sections

Sectioning elements are taken from latex and allow the document to be organised into logical sections. From highest to lowest order of priority, they are: :chapter, section, subsection and :subsubsection, giving four levels of organisation.

The numbering for elements are generated in sequencial order: (1, 2, 3 ... etc) and a tag can be generated from the title or specified for creating links within the document. :chapter, section and subsection elements are list in the table of contents using tags.

For example, I wish to write a chapter about animals and have organised content into categories shown below.

Animals
- Mammals
- Birds
- - Can Fly
- - - Eagle
- - - Hummingbird
- - Flightless
- - - Penguin

It is very straight forward to turn this into sectioning elements which will then generate the sectioning numbers for different categories

[[:chapter {:title "Animals"}]]
[[:section {:title "Mammals"}]]
[[:section {:title "Birds"}]]
[[:subsection {:title "Can Fly"}]]
[[:subsubsection {:title "Eagle"}]]
[[:subsubsection {:title "Hummingbird"}]]
[[:subsection {:title "Flightless"}]]
[[:subsubsection {:title "Penguin"}]]

The sections will be automatically numbered as show below:

Animals             ; 1 
- Mammals           ; 1.1
- Birds             ; 1.2
- - Can Fly         ; 1.2.1
- - - Eagle         ; 1.2.1.1
- - - Hummingbird   ; 1.2.1.2
- - Flightless      ; 1.2.2
- - - Penguin       ; 1.2.2.1

4.3    :image

The :image element embeds an image as a figure within the document. It is numbered and can be tagged for easy reference. The code example below:

[[:image {:tag "clojure-logo" :title "Clojure Logo (source clojure.org)"
          :src "http://clojure.org/images/clojure-logo-120b.png"}]]

produces the image below in Figure

fig.1  -  Clojure Logo (source clojure.org)

4.4    :file

The :file element allows inclusion of other files into the document. It is useful for breaking up a document into managable chunks. A file element require that the :src attribute be specified. A high-level view of a document can thus be achieved, making the source more readable. This is similar to the include element in latex.

e.4.4  -  :file tag example

[[:file {:src "test/docs/first_section.clj"}]]
[[:file {:src "test/docs/second_section.clj"}]]
[[:file {:src "test/docs/third_section.clj"}]]

5    Code

Code displayed in documentation are of a few types:

  1. Code that needs to be run (normal clojure code)
  2. Code that needs verification taking input and showing output. (midje fact)
  3. Code that should not be run (namespace declaration examples)
  4. Code that is part of the library's tests or source definition
  5. Code in other languages

The different types of code can be defined so that code examples render properly using a variety of methods

5.1    normal s-expressions

Normal s-expressions are rendered as is. Attributes can be added for grouping purposes. The source code shown below

e.5.1  -  seperating code blocks through attributes

[[{:title "add-n definition" :tag "c-add-1"}]]
(defn add-n [n]
  (fn [x] (+ x n)))

[[{:title "add-4 and add-5 definitions" :tag "c-add-2"}]]
(def add-4 (add-n 4))
(def add-5 (add-n 5))

renders the following outputs:

e.5.2  -  add-n definition

(defn add-n [n]
  (fn [x] (+ x n)))

e.5.3  -  add-4 and add-5 definitions

(def add-4 (add-n 4))
(def add-5 (add-n 5))

5.2    fact/facts

Documentation examples put in fact forms allows the code to be verified for correctness using hara.test as well as midje.

e.5.4  -  Fact Form Source

[[{:tag "fact-form-output" :title "Fact Form Output"}]]
(fact
  (def a (atom 1))
  (deref a) => 1

  (swap! a inc 1)
  (deref a) => 2)

renders the this output:

(def a (atom 1))
(deref a) => 1

(swap! a inc)
(deref a) => 2

5.3    comment

Comments are clojure's built-in method of displaying non-running code and so this mechanisim is used in clojure for displaying code that should not be run, but still requires display. Code can still output without interferring with code or tests.

e.5.5  -  Switching to a new namespace

[[{:title "Switching to a new namespace" :tag "c-com-1"}]]
(comment
  (in-ns 'hello.world)
  (use 'clojure.string)
  (split "Hello World" #"\s") ;=> ["Hello" "World"]
  )

e.5.6  -  Switching to a new namespace

(in-ns 'hello.world)
(use 'clojure.string)
(split "Hello World" #"\s") ;=> ["Hello" "World"]

5.4    :reference

Code in the repository can be directly referenced:

[[:reference {:refer "lucid.publish.theme/deploy"}]]

Gives this output:

(defn deploy
  ([]
   (apply-settings deploy))
  ([settings project]
   (let [target   (template-path settings project)
         inputs   (mapv (juxt (fn [path]
                                    (-> (str (:resource settings) "/" path)
                                        (io/resource)))
                                  identity)
                            (:manifest settings))]
     (doseq [[resource filename] inputs]
       (try
         (let [out (fs/path target filename)]
           (fs/create-directory (fs/parent out))
           (fs/write (.openStream resource)
                     out
                     {:options [:replace-existing]})
           (.closeStream resource))
         (catch Exception e
           (print "Cannot Deploy Filename:" filename)
           ;;(throw e)
           ))))))

Tests can be referred to by adding :mode :test to

[[:reference {:refer "lucid.publish.theme/deploy" :mode :test}]]
[[:reference {:refer "lucid.publish.theme/deploy" :mode :test}  :title "Test Output" :tag "source-1"]]

5.5    :api

An API table can be constructed using:

[[:api {:namespace "lucid.publish.theme"}]]

The above output will construct a table as shown below:

OUTPUT - lucid.publish.theme



apply-settings ^

applies function to the settings in the current `project.clj`

v 1.2
(defn apply-settings
  [f & args]
  (let [project (project/project)
        theme   (-> project :publish :theme)
        settings (load-settings theme project)]
    (apply f (concat args [settings project]))))
link

deploy ^

deploys theme into template directory

v 1.2
(defn deploy
  ([]
   (apply-settings deploy))
  ([settings project]
   (let [target   (template-path settings project)
         inputs   (mapv (juxt (fn [path]
                                    (-> (str (:resource settings) "/" path)
                                        (io/resource)))
                                  identity)
                            (:manifest settings))]
     (doseq [[resource filename] inputs]
       (try
         (let [out (fs/path target filename)]
           (fs/create-directory (fs/parent out))
           (fs/write (.openStream resource)
                     out
                     {:options [:replace-existing]})
           (.closeStream resource))
         (catch Exception e
           (print "Cannot Deploy Filename:" filename)
           ;;(throw e)
           ))))))
link
;; deploys the stark theme to the template/stark directory ;; overwriting any content in the directory (deploy (load-settings "stark" (project/project)) (project/project))

deployed? ^

checks if a theme has been deployed

v 1.2
(defn deployed?
  ([]
   (apply-settings deployed?))
  ([settings project]
   (let [target  (template-path settings project)]
     (fs/exists? target))))
link
(deployed? (load-settings "stark" (project/project)) (project/project)) => true

load-settings ^

load theme settings

v 1.2
(defn load-settings
  ([] (load-settings nil))
  ([theme] (load-settings theme (project/project)))
  ([theme project]
   (let [current (java.util.Date.)
         theme (or theme (-> project :publish :theme) *default*)
         opts  (-> project
                   :publish
                   :template
                   (assoc :theme theme
                          :date (-> (java.text.SimpleDateFormat. "dd MMMM yyyy")
                                    (.format current))
                          :time (-> (java.text.SimpleDateFormat. "HH mm")
                                    (.format current))))
         ns (cond (string? theme)
                  (symbol (str "lucid.publish.theme." theme))

                  (symbol? theme) theme

                  :else (throw (Exception. (format "Cannot load theme: %s" theme))))
         settings (do (require ns)
                      (load-var ns "settings"))]
     (->> (:render settings)
          (reduce-kv (fn [out k v]
                       (assoc-in out [:defaults k] [:fn (load-var ns v)]))
                     settings)
          (merge opts)))))
link
(keys (load-settings "stark" (project/project))) => (contains [:email :date :copy :tracking-enabled :site :time :manifest :icon :defaults :theme :author :render :tracking :resource :engine] :in-any-order)

load-var ^

loads a namespaced var

v 1.2
(defn load-var
  [ns var]
  (-> (symbol (str ns "/" var))
      resolve
      deref))
link
(load-var "clojure.core" "apply") => fn?

refresh? ^

checks the `refresh` setting for the project template

v 1.2
(defn refresh?
  ([] 
   (apply-settings refresh?))
  ([settings project]
   (boolean (-> project :publish :template :refresh))))
link
(refresh?) ;; by default, it is false => false

template-path ^

creates a template path, by default it is `./template`

v 1.2
(defn template-path
  ([] (apply-settings template-path))
  ([settings project]
   (let [template-dir (or (-> project :publish :template :path)
                          *path*)]
     (fs/path (:root project) template-dir (:theme settings)))))
link

The table can be customised with :title, :only and :exclude keys. The following generates a table with only an entry for deploy:

[[:api {:namespace "lucid.publish.theme"
        :title ""
        :only ["deploy"]}]]

This example generates a table with

[[:api {:namespace "lucid.publish.theme"
        :title ""
        :exclude ["template-path" "apply-settings"]}]]

To make links to a :chapter, use :link to create navigation links for the api

[[:chapter {:namespace "Theme API"
            :link "lucid.publish.theme"
            :exclude ["template-path" "apply-settings"]}]]
  
[[:api {:namespace "lucid.publish.theme"
        :title ""
        :exclude ["template-path" "apply-settings"]}]]

5.6    other languages

The most generic way of displaying code is with the :code tag. It is useful when code in other languages are required to be in the documentation.

5.6.1    python

The source and outputs are listed below:

e.5.7  -  Python for Loop Source

[[:code {:lang "python" :title "Python for Loop" :tag "c-py-1"}
"
myList = [1,2,3,4]
for index in range(len(myList)):
  myList[index] += 1
print myList"]]

e.5.8  -  Python for Loop

myList = [1,2,3,4]
for index in range(len(myList)):
  myList[index] += 1
print myList

5.6.2    ruby

The source and outputs are listed below:

e.5.9  -  Ruby for Loop Source

[[:code {:lang "ruby" :title "Ruby for Loop" :tag "c-rb-2"}
"
array.each_with_index do |element,index|
  element.do_stuff(index)
end"]]

e.5.10  -  Ruby for Loop

array.each_with_index do |element,index|
    element.do_stuff(index)
end

6    Project Level

6.1    Setup

lucid.publish can also be used to generate an entire site based on clojure code input. The default template can be changed according to need. lucid.publish can generates a .html output based on a .clj file. The :publish key in defproject specifies which files to use as entry points to use for html generation. A sample can be seen below:

(defproject ...
  ...
  :publish {:site   "sample"
            :theme  "bolton" ;; stark is the default
            :output "docs"
            :files {"sample-document"
                    {:input "test/documentation/sample_document.clj"
                     :title "a sample document"
                     :subtitle "generating a document from code"}}}
  ...)

This is the simplest example - more options can be added as needed. See examples for hara and lucidity

6.2    Templates and Themes

Templates can be set up to customise the site to look however fancy it needs to be.

e.6.1  -  Template

<html>
  ...
  <head>
    <title><@=title></title>
  </head>

  <body>
    <@=navigation>
    <@=contents>
  </body>

</html>

The <@=KEY> values can be found with lucid.publish.theme/load-settings:

(lucid.publish/load-settings)
;; {:email "z@caudate.me",
;;  :date "06 October 2016",
;;  :copy ["assets"],
;;  :tracking-enabled "true",
;;  :site "lucid",
;;  :time "08 12",
;;  :manifest [... files ...],
;;  :icon "favicon",
;;  :defaults {:site "stark",
;;             :icon "favicon",
;;             :tracking-enabled "false",
;;             :template "article.html",
;;             :theme-base "theme-base-0b",
;;             :logo-white "img/logo-white.png",
;;             :article [...],
;;             :outline [...],
;;             :top-level [...]},
;;  :theme "stark",
;;  :author "Chris Zheng",
;;  :render {:article "render-article",
;;           :outline "render-outline",
;;           :top-level "render-top-level"},
;;  :tracking "UA-31320512-2",
;;  :resource "theme/stark",
;;  :engine "winterfell"}

And more values can be added to the [:publish :template] or [:publish :template :defaults] entry in project.clj. Please reference the stark and bolton themes to see how to change a template to suit your needs.