hara.component constructing composable systems

Author: Chris Zheng  (z@caudate.me)
Date: 11 February 2018
Repository: https://github.com/zcaudate/hara
Version: 2.8.2

1    Introduction

hara.component is a dependency injection framework for clojure. The virtues of this type of design for composing large systems has been much lauded. This library places emphasis on building large systems using a toplogy of how subsystems fit together and data of how it is to be configured.

1.1    Installation

Add to project.clj dependencies:

[zcaudate/hara.component "2.8.2"]

All functions are in the hara.component namespace.

(require '[hara.component :as component])

1.2    Motivation

There are two existing component libraries, namely:

hara.component takes the more simplified idea proposed by the original library where there is whole system start up and teardown. However, it extends upon that concept by teasing apart configuration and application topology to give cleaner notion of design.

Configuration gives the ability to set the starting state of the entire program and should be easy as possible. Many a system become bloated due to not being able to properly manage configuration, therefore composing systems with configuration at the forefront will make for much simpler code and design. This library was build with this paradigm in mind.

Additionally, there are additional features that enable developers to write their systems in a clearer fashion.

  • support for dealing with arrays of component
  • control with nested systems

2    API

Because component is a framework for building systems, we have to start off with concepts of what we wish to build:

  • A Catalog searches through files in a smart way and consists of a FileSystem and a Database
  • A Filesystem stores and manages files
  • A Database stores indexed information about files

The most simple representation is:

(defrecord Database []
  (-start [db]
    (assoc db :status "started"))
  (-stop [db]
    (dissoc db :status)))

(defrecord Filesystem []
  (-start [fs]
    (assoc fs :status "started"))
  (-stop [fs]
    (dissoc fs :status)))

(defrecord Catalog []
  (-start [store]
    (assoc store :status "started"))
  (-stop [store]
    (dissoc store :status)))

These definitions are then used in context of specifying a system consisting of two Catalog instances, having the same Database instance, but different Filesystem instances.

component? ^

checks if an instance extends IComponent

v 2.2
(defn component?
  (extends? IComponent (type x)))
(component? (Database.)) => true

system ^

creates a system of components

v 2.1
(defn system
  ([topology config]
   (system topology config {:partial false}))
  ([topology config {:keys [partial? tag display] :as opts}]
   (let [full   (long-form topology)
         valid  (valid-subcomponents full (keys config))
         expose (get-exposed full)
         diff   (set/difference (set (keys full)) valid)
         _      (or (empty? diff)
                    (throw (Exception. (str "Missing Config Keys: " diff))))
         build  (apply dissoc full diff)
         dependencies (apply dissoc (get-dependencies full) diff)
         order (topological-sort dependencies)
         initial  (apply dissoc build (concat diff (get-exposed full)))]
     (-> (reduce-kv (fn [sys k {:keys [constructor compile] :as build}]
                      (let [cfg (get config k)]
                        (assoc sys k (cond (= compile :array)
                                           (array build cfg)
                                           (constructor cfg)))))
         (with-meta (merge {:partial (not (empty? diff))
                            :build   build
                            :order   order
                            :dependencies dependencies}
;; The topology specifies how the system is linked (def topo {:db [map->Database] :files [[map->Filesystem]] :catalogs [[map->Catalog] [:files {:type :element :as :fs}] :db]}) ;; The configuration customises the system (def cfg {:db {:type :basic :host "localhost" :port 8080} :files [{:path "/app/local/1"} {:path "/app/local/2"}] :catalogs [{:id 1} {:id 2}]}) ;; `system` will build it and calling `start` initiates it (def sys (-> (system topo cfg) start)) ;; Check that the `:db` entry has started (:db sys) => (just {:status "started", :type :basic, :port 8080, :host "localhost"}) ;; Check the first `:files` entry has started (-> sys :files first) => (just {:status "started", :path "/app/local/1"}) ;; Check that the second `:store` entry has started (->> sys :catalogs second) => (contains-in {:id 2 :status "started" :db {:status "started", :type :basic, :port 8080, :host "localhost"} :fs {:path "/app/local/2", :status "started"}})

system? ^

checks if object is a component system

v 2.1
(defn system?
  (instance? ComponentSystem x))
(system? (system {} {})) => true

start ^

starts a component/array/system

v 2.1
(defn start
   (start component {}))
  ([component {:keys [setup hooks] :as opts}]
   (let [{:keys [pre-start post-start]} hooks
         functions (or (get component :functions)
                       (get opts :functions))
         setup     (or setup identity)
         component   (-> component
                         (perform-hooks functions pre-start)
                         (perform-hooks functions post-start))]
     (if (iobj? component)
       (vary-meta component assoc :started true)
(start (Database.)) => {:status "started"}

stop ^

stops a component/array/system

v 2.1
(defn stop
   (stop component {}))
  ([component {:keys [teardown hooks] :as opts}]
   (let [{:keys [pre-stop post-stop]} hooks
         functions (or (get component :functions)
                       (get opts :functions))
         teardown  (or teardown identity)
         component (-> component
                       (perform-hooks functions pre-stop)
                       (perform-hooks functions post-stop))]
     (if (iobj? component)
       (vary-meta component dissoc :started)
(stop (start (Database.))) => {}

started? ^

checks if a component has been started

v 2.1
(defn started?
  (try (-started? component)
       (catch IllegalArgumentException e
         (if (iobj? component)
           (-> component meta :started true?)
           (primitive? component)))
       (catch AbstractMethodError e
         (if (iobj? component)
           (-> component meta :started true?)
           (primitive? component)))))
(started? 1) => true (started? {}) => false (started? (start {})) => true (started? (Database.)) => false (started? (start (Database.))) => true (started? (stop (start (Database.)))) => false

stopped? ^

checks if a component has been stopped

v 2.1
(defn stopped?
  (try (-stopped? component)
       (catch IllegalArgumentException e
         (-> component started? not))
       (catch AbstractMethodError e
         (-> component started? not))))
(stopped? 1) => false (stopped? {}) => true (stopped? (start {})) => false (stopped? (Database.)) => true (stopped? (start (Database.))) => false (stopped? (stop (start (Database.)))) => true

properties ^

returns properties of the system

v 2.1
(defn properties
  (try (-properties component)
       (catch IllegalArgumentException e
       (catch AbstractMethodError e
(properties (Database.)) => {} (properties (Filesystem.)) => {:hello "world"}

3    Config Driven Design

Following on from the general concepts we have have a look at how such a framework can be used in practice.

We will aim to create a system based upon a configuration file. As components are a very high level concept, using the pattern in code is more of a state of mind than following a set of APIs. Therefore in this documentation, it is hoped that a tutorial based approach will demonstrate the core functionality within the library. We seperate the guide into the following sections

The first two chapters are not really about components, but it is important in showing how to create functions based around a particular configuration. The rest of the sections take a comphrensive approach of how to configure and reason about entire systems based on the component approach.

3.1    The Bug Trapper

We are creating a simulation based on trapping bugs in different parts of the house, then tallying up the results and displaying it through a web interface. We can see how the sub-systems connect together in the diagram below:

fig.1  -  sub-system dependencies

At the very bottom is a statistical model for generating events ie, how often an insect would be likely to appear given a certain set of conditions such as brightness, dampness, if it is indoors or outdoors, etc. This is used by a number of traps, which are an array of devices that simulates events of an insect going into the trap. An insect may or may not be captured by the trap itself and this is also tallied. The app itself tracks events over time by putting results into a 'database' and the server takes live results and outputs a string via http.

3.2    Configuration

A datastructure can be created that customises various aspects of the simulation:

(def config
  {:server     {:port 8090}
   :app        {}
   :traps     ^{:fuzziness 0.1 :efficiency 0.6}
               [{:location "kitchen"  :brightness 0.3
                 :indoor true :rate 0.5}
                {:location "bedroom"  :brightness 0.1
                 :dampness 0.2 :indoor true :rate 0.3  :efficiency 0.2}
                {:location "patio"    :brightness 0.5
                 :outdoor true :rate 1.5  :efficiency 0.1}
                {:location "bathroom" :dampness 0.3
                 :indoor true :rate 0.2  :efficiency 0.3}]
   :db         {}
   :model      {:default {:fly 0.5 :ladybug 0.05
                          :mosquito 0.35 :bee 0.1}
                :linear  {:brightness {:bee 0.5}
                          :dampness   {:mosquito 0.4}}
                :toggle  {:indoor     {:fly 0.3
                                       :mosquito 0.2}
                          :outdoor    {:bee 0.1
                                       :ladybug 0.1}}}})

4    Probability Model

We have a model of what percentage of bugs and depending on location, brightness and dampness, we adjust our model accordingly. So for example, we should be able to write a function adjusted-distribution that takes in a model and some parameter settings and spits out a probability distribution in the form of a map:

(adjusted-distribution {} (-> config :model))
=> {:fly 0.5, :ladybug 0.05, :mosquito 0.35, :bee 0.1}

(adjusted-distribution {:indoor true}
                       (-> config :model))
=> {:fly 0.8, :ladybug 0.05, :mosquito 0.55, :bee 0.1}

4.1    linear-adjustment

We write a functions to adjustment for the linear increase:

(defn linear-adjustment [params linear]
  (reduce-kv (fn [m k stats]
               (if-let [mul (get params k)]
                 (reduce-kv (fn [m k v]
                              (update-in m [k] (fnil #(+ % (* mul v))

It can be applied to the model:

(linear-adjustment {:brightness 0.1}
                   (-> config :model :linear))
=> {:bee 0.05}

(linear-adjustment {:brightness 0.2}
                   (-> config :model :linear))
=> {:bee 0.1}

(linear-adjustment {:brightness 0.3}
                   (-> config :model :linear))
=> {:bee 0.15}

(linear-adjustment {:dampness 0.5}
                   (-> config :model :linear))
=> {:mosquito 0.2}

4.2    toggle-adjustment

The second function is for toggle adjustment, meaning that depending on a particular flag, we add a certain amount to the overall distribution:

(defn toggle-adjustment [params toggle]
  (reduce-kv (fn [m k stats]
               (if-let [mul (get params k)]
                 (reduce-kv (fn [m k v]
                              (update-in m [k] (fnil #(+ % v)

It can be applied to the model:

(toggle-adjustment {:indoor true}
                   (-> config :model :toggle))
=> {:fly 0.3, :mosquito 0.2}

(toggle-adjustment {:outdoor true}
                   (-> config :model :toggle))
=> {:bee 0.1, :ladybug 0.1}

4.3    add-distributions

A helper function is defined to add distributions together

(defn add-distributions
  ([] {})
  ([m] m)
  ([m1 m2]
   (reduce-kv (fn [m k v]
                (update-in m [k] (fnil #(+ % v) 0)))
  ([m1 m2 & more]
   (apply add-distributions (add-distributions m1 m2) more)))

The function is relatively generic and can be used to add arbitrary maps together:

(add-distributions {:a 0.1} {:a 0.1 :b 0.3} {:a 0.3 :c 0.3})
=> {:a 0.5, :b 0.3, :c 0.3}

4.4    adjusted-distribution

Combining the three functions, we can get an adjusted distribution based on the model taken from the config:

(defn adjusted-distribution [params {:keys [default linear toggle] :as model}]
  (let [ladjust (linear-adjustment params linear)
        tadjust (toggle-adjustment params toggle)]
    (add-distributions default ladjust tadjust)))

The adjusted distributions for each trap can then be calculated:

(mapv #(adjusted-distribution % (-> config :model))
      (-> config :traps))
=> [;; kitchen
    {:fly 0.8, :ladybug 0.05,
     :mosquito 0.55, :bee 0.25}
    ;; bedroom
    {:fly 0.8, :ladybug 0.05,
     :mosquito 0.63, :bee 0.15000000000000002}
    ;; patio
    {:fly 0.5, :ladybug 0.15000000000000002,
     :mosquito 0.35, :bee 0.44999999999999996}
    ;; bathroom
    {:fly 0.8, :ladybug 0.05,
     :mosquito 0.6699999999999999, :bee 0.1}]

5    Sampling Model

The sampling model is easier to construct. We wish to create a function that takes in a distribution and returns a key that is proportional to the values of the map:

(random-sample {:a 0.5 :b 0.5})
=> ;; either returns :a or :b
#(get #{:a :b} %)

5.1    cumultive

culmultive takes a distribution and turns it into a range, sorted by value:

(defn cumultive [distribution]
  (dissoc (reduce (fn [out [k v]]
                    (let [total (::total out)
                          ntotal (+ total v)]
                      (assoc out
                             k [total ntotal]
                             ::total ntotal)))
                  {::total 0}
                  (sort-by val distribution))

examples of its usage can be seen:

(cumultive {:a 0.3 :b 0.5 :c 0.2})
=> {:c [0 0.2], :a [0.2 0.5], :b [0.5 1.0]}

5.2    category

category takes a cumultive distribution and a point, return which section it belongs to:

(defn category [cumulative stat]
  (->> cumulative
       (keep (fn [[k [lower upper]]]
               (if (<= lower stat upper) k)))

examples of its usage can be seen:

(def dist (cumultive {:a 0.3 :b 0.5 :c 0.2}))
;; {:c [0 0.2], :a [0.2 0.5], :b [0.5 1.0]} 0.1

(category dist 0.1) => :c

(category dist 0.3) => :a

(category dist 0.8) => :b

5.3    random-sample

Now the random-sample function can be written:

(defn random-sample [distribution]
  (let [total (apply + (vals distribution))
        stat  (rand total)]
    (category (cumultive distribution) stat)))

We can now use this with a probability map:

(random-sample {:a 0.3 :b 0.5 :c 0.2})
=> ;; Returns either :a :b or :c
#(get #{:a :b :c} %)

As well as with adjusted-model defined in the previous chapter

 (adjusted-distribution {:brightness 0.3 :indoor true :rate 0.5}
                        (-> config :model)))
=> ;; Return either :fly :ladybug :mosquito :bee
#(get #{:fly :ladybug :mosquito :bee} %)

6    Implementing Components

6.1    Model

We create a record for Model. The data is just a nested map but a record is used purely for printing purposes. There is quite alot of stuff in the map and we should be able to only show the necessary amount of information - in this case, we only want to know the keys of the datastructure:

(defrecord Model []
  (toString [obj]
    (str "#model" (vec (keys (into {} obj))))))

(defmethod print-method Model
  [v w]
  (.write w (str v)))

We can now use the map->Model function to create a nicer new on the model:

(map->Model (:model config))
;;=> #model[:default :linear :toggle]

6.2    Trap

Trap is a component that needs to be started and stopped. It simulates a trap that knows what insect went inside the trap, what time it entered and if it had been captured. We create a basic function for one round of the trapping an insect:

(defn trap-bug [{:keys [rate efficiency fuzziness model output] :as trap}]
  (let [pause   (long (* (+ rate
                            (* (- (rand 1) 0.5) fuzziness))
    (Thread/sleep pause)
    (reset! output
           {:time (java.util.Date.)
            :bug (random-sample
                  (adjusted-distribution trap model))
            :captured (< (rand 1) efficiency)})

The usage for such a function can be seen below:

(-> (trap-bug {:rate 0.8
               :efficiency 0.5
               :fuzziness 0.1
               :model (:model config)
               :output (atom nil)})
=> (contains {:time #(instance? java.util.Date %)
              :bug #(#{:fly :bee :ladybug :mosquito} %)
              :captured #(instance? Boolean %)})

We create a record that implements the IComponent interface, making sure that we hide keys that are not useful

(defrecord Trap []
  (toString [obj]
    (let [selected [:location :output]]
      (str "#trap" (-> (into {} obj)
                       (select-keys selected)
                       (update-in [:output] deref)))))

  (-start [trap]
    (assoc trap
           :thread (future
                     (println (str "Starting trap in "
                                   (:location trap) "\n"))
                     (last (iterate trap-bug trap)))))

  (-stop [{:keys [thread output] :as trap}]
      (println (str "Stopping trap in " (:location trap)))
      (future-cancel thread)
      (reset! output nil)
      (dissoc trap :thread))))

(defmethod print-method Trap
  [v w]
  (.write w (str v)))

Finally, we create a trap constructor taking a config map and outputting a Trap record:

(defn trap [m]
  (assoc (map->Trap m)
         :output (atom nil)))

6.3    Partial System Testing

Having implemented the records for :traps and :model, we can test to see if our array of traps are working. The call to system takes two parameters - a topology map and a configuration map. The topology map specifies functions and dependencies whilst the configuration map specifies the initial input data. Note that to specify contruction of an array of components put the constructor in an additional vector:

(def topology {:traps [[trap] :model]
               :model [map->Model]})

(def sys (-> (component/system toplogy config)
;; Starting trap in patio
;; Starting trap in bathroom
;; Starting trap in kitchen
;; Starting trap in bedroom

(add-watch (-> sys :traps first :output)
           (fn [_ _ _ n]
             (if (:captured n)
               (println n))))
;; {:time #inst "2015-07-15T08:21:33.690-00:00", :bug :fly, :captured true}
;; {:time #inst "2015-07-15T08:21:34.216-00:00", :bug :mosquito, :captured true}
;; ....
;; ....
;; {:time #inst "2015-07-15T08:21:36.753-00:00", :bug :fly, :captured false}

(remove-watch (-> sys :traps first :output) :print-change)

(component/stop sys)
;;=> {:traps #arr[#trap{:location "kitchen", :output nil}
;;             #trap{:location "bedroom", :output nil}
;;             #trap{:location "patio", :output nil}
;;             #trap{:location "bathroom", :output nil],
;;    :model #model[:default :linear :toggle]}

6.4    App

The role of the app is to hook up the sensors to a datastore, in this case an ova, a mutable array of elements. We define initialise-app to setup watches to provide some summary and coordination:

(require '[hara.concurrent.ova :as ova])

(defn initialise-app [{:keys [db traps display total] :as app}]
  (let [data (mapv (fn [trap]
                     (select-keys trap [:location]))
    (dosync (ova/init! db data))
    (doseq [{:keys [location output] :as trap} traps]
       output :summary
       (fn [_ _ _ {:keys [success bug]}]
         (dosync (ova/!> db [:location location]
                         (update-in [:triggered]
                                    (fnil inc 0))
                         (update-in [:captured]
                                    (fnil #(update-in % [bug] (fnil inc 0))
         (swap! total update-in [bug] (fnil inc 0))))))

The opposite method deinitialise-app is also defined:

(defn deinitialise-app [{:keys [db traps total] :as app}]
  (dosync (ova/empty! db))
  (reset! total {})
  (doseq [{:keys [output]} traps]
    (remove-watch output :summary))

The two functions are then hooked up via -start and -stop protocol methods for the component architecture:

(defrecord App []
  (toString [app]
    (str "#app" (-> app keys vec)))

  (-start [app]
    (initialise-app app)

  (-stop [app]
    (deinitialise-app app)

(defmethod print-method App
  [v w]
  (.write w (str v)))

(defn app [m]
  (assoc (map->App m) :total (atom {})))

6.5    App Testing

We can now do a more testing by including a couple more constructors. Note that the keys :db, app and summary have been added. Also see the syntax for the :summary topology to expose the :total submap from :app.

The syntax for the :summary key should be further explained. What component/start sees a initialisation of {:expose [:total]}, it take the first dependency (in this case, :app), gets the :total submap and exposes it as :summary in the system map. The value of :expose can be either a vector (for nested map supprt) or a function for more generic operations. This promotes reuse and composition of multiple systems.

(def topology {:traps   [[trap] :model]
               :model   [map->Model]
               :db      [ova/ova]
               :app     [app :traps :db]
               :summary [{:expose [:total]} :app]})

(def sys (-> (component/system topology config)
;; Starting trap in patio
;; Starting trap in bathroom
;; Starting trap in kitchen
;; Starting trap in bedroom

@(:summary sys) ;; first call to :summary gives a set of bugs trapped
;;=> {:mosquito 101, :fly 120, :ladybug 6, :bee 28}

@(:summary sys) ;; second call to :summary gives an updated of bugs trapped
;;=> {:mosquito 148, :fly 184, :ladybug 12, :bee 37}

(component/stop sys)
;; Stopping trap in kitchen
;; Stopping trap in bedroom
;; Stopping trap in patio
;; Stopping trap in bathroom

;;=> {:app #app[:display :total],
;;    :db #ova [],
;;    :traps #arr[#trap{:location "kitchen",  :output nil}
;;                #trap{:location "bedroom",  :output nil}
;;                #trap{:location "patio",    :output nil}
;;                #trap{:location "bathroom", :output nil}]
;;    :model #model[:default :linear :toggle]}

6.6    Server

The server requires a couple of external dependencies:

(require '[compojure.core :as routes]
         '[ring.adapter.jetty :as jetty]
         '[clj-http.client :as client])

We define a very simple server with one route that just returns the summary as a string:

(defn make-routes
  [{:keys [summary] :as serv}]
  (routes/GET "*" [] (str @summary)))

(defrecord Server []
  (toString [serv]
    (str "#server" (-> serv keys vec)))

  (-start [{:keys [port summary] :as serv}]
    (println (str "STARTING SERVER on port " port))
    (assoc serv
           :instance (jetty/run-jetty (make-routes serv)
                                      {:join? false
                                       :port port})))

  (-stop [{:keys [summary instance] :as serv}]
    (println (str "STOPPING SERVER on port " (:port serv)))
    (.stop instance)
    (dissoc serv :instance)))

Again, print-method is defined for prettiness:

(defmethod print-method Server
  [v w]
  (.write w (str v)))

6.7    Server Testing

Again, we add an additional constructor to the system and start:

(def sys (-> {:traps   [[trap] :model]
              :model   [map->Model]
              :db      [ova/ova]
              :app     [app :traps :db]
              :summary [{:expose [:total]} :app]
              :server  [map->Server :summary]}
             (component/system config)
;; Starting trap in patio
;; Starting trap in bathroom
;; Starting trap in kitchen
;; Starting trap in bedroom

We can now use a client to access the summary via a http protocol:

;; First Time
(-> (client/get "http://localhost:8090/")
;;=> "{:fly 249, :bee 55, :mosquito 187, :ladybug 19}"

;; Second Time
(-> (client/get "http://localhost:8090/")
;; => "{:fly 305, :bee 70, :mosquito 227, :ladybug 26}"

Stopping is no different to before

(component/stop sys)
;; Stopping trap in kitchen
;; Stopping trap in bedroom
;; Stopping trap in patio
;; Stopping trap in bathroom

;;=> {:app #app[:display :total],
;;    :db #ova [],
;;    :traps #arr[#trap{:location "kitchen",  :output nil}
;;                #trap{:location "bedroom",  :output nil}
;;                #trap{:location "patio",    :output nil}
;;                #trap{:location "bathroom", :output nil}]
;;    :model #model[:default :linear :toggle]
;;    :server #server[:port]}

7    The Big Picture

7.1    Summary

We have created the bug trapping system based on our dependency diagram. We can visualize the essential components that make up our system in the diagram below:

fig.2  -  the system

The constructors and the dependencies form our system topology whilst the data that initialised our system form our config. There are significant advantages of doing this:

  • The final code is almost the same as the diagram of the system.
  • There is an isometric correspondence between process and data.
  • It clearly seperates data (which is normally loaded from a file) from process.
  • It keeps all the initialisations in a single place.
  • Systems can be built incrementally in the way that we have just done.

7.2    Further Extension

Say we needed to add more functionality to our system, in which we define the make-routes method to add an endpoint giving us information about the status of the datastore:

(defn make-routes
  [{:keys [db summary] :as serv}]
    (routes/GET "/total" [] (str @summary))
    (routes/GET "/db" []    (str (persistent! db)))))

It is very easy to redefine topology to include the extra dependency:

(def toplogy {:traps   [[trap] :model]
              :model   [map->Model]
              :db      [ova/ova]
              :app     [app :traps :db]
              :summary [{:expose [:total]} :app]
              :server  [map->Server :summary :db]}) ;; note the extra `:db` key

(def sys (-> (component/system topology config)

We can again visualize the extended system:

fig.3  -  the extended system

8    Links and Resources

Here are some more links and resources on the web: