hara.io.file tools for the file system

Author: Chris Zheng  (z@caudate.me)
Date: 14 March 2017
Repository: https://github.com/zcaudate/hara
Version: 2.5.2

1    Introduction

hara.io.file contain functions and utilities for file system manipulation using the java.nio.file package. Efforts have been made to allow filesystem operations to be as easy and straightforward to use as possible.

1.1    Installation

Add to project.clj dependencies:

[im.chit/hara.io.file "2.5.2"]

1.2    Motivation

There are a couple of other filesystem libraries for clojure:

  • fs - based on java.io and clojure.java.io.
  • nio - extends clojure.java.io functions to java.nio classes
  • nio.file - an early wrapper for java.nio.file.

hara.io.file aims to provide consistency of reporting for file operations. Operations are designed using the FileVisitor pattern. Therefore file and directory manipulation are considered as bulk operations by default.

2    Index

3    API

3.1    Path

Methods that construct and operate on the java.nio.file.Path object.



path ^

creates a `java.nio.file.Path object

v 2.4
(defn path
  ([x]
   (cond (instance? Path x)
         x
         
         (string? x)
         (.normalize (Paths/get (normalise x) *empty-string-array*))

         (vector? x)
         (apply path x)

         (instance? java.net.URI x)
         (Paths/get x)

         (instance? File x)
         (path (.toString ^File x))
         
         :else
         (throw (Exception. (format "Input %s is not of the correct format" x)))))
  ([s & more]
   (.normalize (Paths/get (normalise (str s)) (into-array String (map str more))))))
link
(path "project.clj") ;;=> #path:"/Users/chris/Development/chit/hara/project.clj" (path (path "project.clj")) ;; idempotent ;;=> #path:"/Users/chris/Development/chit/hara/project.clj" (path "~") ;; tilda ;;=> #path:"/Users/chris" (path "src" "hara/time.clj") ;; multiple arguments ;;=> #path:"/Users/chris/Development/chit/hara/src/hara/time.clj" (path ["src" "hara" "time.clj"]) ;; vector ;;=> #path:"/Users/chris/Development/chit/hara/src/hara/time.clj" (path (java.io.File. ;; java.io.File object "src/hara/time.clj")) ;;=> #path:"/Users/chris/Development/chit/hara/src/hara/time.clj" (path (java.net.URI. ;; java.net.URI object "file:///Users/chris/Development/chit/hara/project.clj")) ;;=> #path:"/Users/chris/Development/chit/hara/project.clj"

path? ^

checks to see if the object is of type Path

v 2.4
(defn path?
  [x]
  (instance? Path x))
link
(path? (path "/home")) => true

section ^

path object without normalisation

v 2.4
(defn section
  ([s & more]
   (Paths/get s (into-array String more))))
link
(str (section "project.clj")) => "project.clj" (str (section "src" "hara/time.clj")) => "src/hara/time.clj"

parent ^

returns the parent of the path

v 2.4
(defn parent
  [path]
  (.getParent (path/path path)))
link
(str (parent "/hello/world.html")) => "/hello"

relativize ^

returns the relationship between two paths

v 2.4
(defn relativize
  [path1 path2]
  (.relativize (path/path path1) (path/path path2)))
link
(str (relativize "hello" "hello/world.html")) => "world.html"

to-file ^

creates a java.io.File object

v 2.4
(defn to-file
  [^Path path]
  (.toFile path))
link
(to-file (section "project.clj")) => (all java.io.File #(-> % str (= "project.clj")))

3.2    Attributes



attributes ^

shows all attributes for a given path

v 2.4
(defn attributes
  [path]
  (-> (path/path path)
      (Files/readAttributes (str (name common/*system*) ":*")
                            common/*no-follow*)
      (attrs->map)))
link
(attributes "project.clj") => {:owner "chris", :group "staff", :permissions "rw-r--r--", :file-key "(dev=1000004,ino=2351455)", :ino 2351455, :is-regular-file true. :is-directory false, :uid 501, :is-other false, :mode 33188, :size 4342, :gid 20, :ctime 1476755481000, :nlink 1, :last-access-time 1476755481000, :is-symbolic-link false, :last-modified-time 1476755481000, :creation-time 1472282953000, :dev 16777220, :rdev 0}

set-attributes ^

sets all attributes for a given path

v 2.4
(defn set-attributes
  [path m]
  (reduce-kv (fn [_ k v]
               (-> (path/path path)
                   (Files/setAttribute (str (name common/*system*) ":"
                                            (case/camel-case (name k)))
                                       (attr-value k v)
                                       common/*no-follow*)))
             nil
             m))
link
(set-attributes "project.clj" {:owner "chris", :group "staff", :permissions "rw-rw-rw-"}) ;;=> #path:"/Users/chris/Development/chit/lucidity/project.clj"

permissions ^

returns the permissions for a given file

v 2.4
(defn permissions
  [path]
  (let [path (path/path path)]
    (->> common/*no-follow*
         (Files/getPosixFilePermissions path)
         (PosixFilePermissions/toString))))
link
(permissions "src") => "rwxr-xr-x"

shorthand ^

returns the shorthand string for a given entry

v 2.4
(defn shorthand
  [path]
  (let [path (path/path path)]
    (cond (Files/isDirectory path (LinkOption/values))
          "d"
          
          (Files/isSymbolicLink path)
          "l"
          
          :else "-")))
link
(shorthand "src") => "d" (shorthand "project.clj") => "-"

directory? ^

checks whether a file is a directory

v 2.4
(defn directory?
  [path]
  (Files/isDirectory (path/path path) common/*no-follow*))
link
(directory? "src") => true (directory? "project.clj") => false

executable? ^

checks whether a file is executable

v 2.4
(defn executable?
  [path]
  (Files/isExecutable (path/path path)))
link
(executable? "project.clj") => false (executable? "/usr/bin/whoami") => true

exists? ^

checks whether a file exists

v 2.4
(defn exists?
  [path]
  (Files/exists (path/path path) common/*no-follow*))
link
(exists? "project.clj") => true (exists? "NON.EXISTENT") => false

file? ^

checks whether a file is not a link or directory

v 2.4
(defn file?
  [path]
  (Files/isRegularFile (path/path path) common/*no-follow*))
link
(file? "project.clj") => true (file? "src") => false

hidden? ^

checks whether a file is hidden

v 2.4
(defn hidden?
  [path]
  (Files/isHidden (path/path path)))
link
(hidden? ".gitignore") => true (hidden? "project.clj") => false

link? ^

checks whether a file is a link

v 2.4
(defn link?
  [path]
  (Files/isSymbolicLink (path/path path)))
link
(link? "project.clj") => false (delete "project.bak.clj") (link? (create-symlink "project.bak.clj" "project.clj")) => true

readable? ^

checks whether a file is readable

v 2.4
(defn readable?
  [path]
  (Files/isReadable (path/path path)))
link
(readable? "project.clj") => true

writable? ^

checks whether a file is writable

v 2.4
(defn writable?
  [path]
  (Files/isWritable (path/path path)))
link
(writable? "project.clj") => true

3.3    IO



code ^

takes a file and returns a lazy seq of top-level forms

v 2.4
(defn code
  [path]
  (let [reader (reader :pushback path)]
    (take-while identity
                (repeatedly #(try (read reader)
                                  (catch Throwable e))))))
link
(->> (code "../hara/src/hara/io/file.clj") first (take 2)) => '(ns hara.io.file)

reader ^

creates a reader for a given input

v 2.4
(defn reader
  ([input]
   (reader :buffered input {}))
  ([type input]
   (reader type input {}))
  ([type input opts]
   (reader/reader type input opts)))
link
(-> (reader :pushback "project.clj") (read) second) => 'im.chit/hara

reader-types ^

returns the types of readers

v 2.4
(defn reader-types
  []
  (keys (.getMethodTable ^clojure.lang.MultiFn reader)))
link
(reader-types) => (contains [:input-stream :buffered :file :string :pushback :char-array :piped :line-number])

write ^

writes a stream to a path

v 2.4
(defn write
  ([stream path]
   (write stream path {}))
  ([stream path opts]
   (Files/copy stream
               ^Path (path/path path)
               (->> (:options opts)
                    (mapv option/option)
                    (into-array CopyOption)))))
link
(-> (java.io.FileInputStream. "project.clj") (write "project.clj" {:options #{:replace-existing}}))

3.4    Create



create-directory ^

creates a directory on the filesystem

v 2.4
(defn create-directory
  ([path]
   (create-directory path {}))
  ([path attrs]
   (Files/createDirectories (path/path path)
                            (attr/map->attr-array attrs))))
link
(do (create-directory ".hello/.world/.foo") (directory? ".hello/.world/.foo")) => true (delete ".hello/.world/.foo")

create-symlink ^

creates a symlink to another file

v 2.4
(defn create-symlink
  ([path link-to]
   (create-symlink path link-to {}))
  ([path link-to attrs]
   (Files/createSymbolicLink (path/path path)
                             (path/path link-to)
                             (attr/map->attr-array attrs))))
link
(do (create-symlink "project.lnk" "project.clj") (link? "project.lnk")) => true

create-tmpdir ^

creates a temp directory on the filesystem

v 2.4
(defn create-tmpdir
  ([]
   (create-tmpdir ""))
  ([prefix]
   (Files/createTempDirectory prefix (make-array FileAttribute 0))))
link
(create-tmpdir) ;;=> #path:"/var/folders/d6/yrjldmsd4jd1h0nm970wmzl40000gn/T/4870108199331749225"

3.5    Operation



copy ^

copies all specified files from one to another

v 2.4
(defn copy
  ([source target]
   (copy source target {}))
  ([source target opts]
   (let [copy-fn (fn [{:keys [root path attrs target accumulator simulate]}]
                   (let [rel   (.relativize ^Path root path)
                         dest  (.resolve ^Path target rel)
                         copts (->> [:copy-attributes :nofollow-links]
                                    (or (:options opts))
                                    (mapv option/option)
                                    (into-array CopyOption))]
                     (when-not simulate
                       (Files/createDirectories (.getParent dest) attr/*empty*)
                       (Files/copy ^Path path ^Path dest copts))
                     (swap! accumulator
                            assoc
                            (str path)
                            (str dest))))]
     (walk/walk source
                (merge {:target (path/path target)
                        :directory copy-fn
                        :file copy-fn
                        :with #{:root}
                        :accumulator (atom {})
                        :accumulate #{}}
                       opts)))))
link
(copy "src" ".src" {:include [".clj"]}) => map? (delete ".src")

copy-single ^

copies a single file to a destination

v 2.4
(defn copy-single
  ([source target]
   (copy-single source target {}))
  ([source target opts]
   (if-let [dir (parent target)]
     (if-not (exists? dir)
       (create-directory dir)))
   (Files/copy ^Path (path/path source)
               ^Path (path/path target)
               (->> (:options opts)
                    (mapv option/option)
                    (into-array CopyOption)))))
link
(copy-single "project.clj" "project.clj.bak" {:options #{:replace-existing}}) ;;=> #path:"/Users/chris/Development/chit/hara/project.clj.bak" (delete "project.clj.bak")

delete ^

copies all specified files from one to another

v 2.4
(defn delete
  ([root] (delete root {}))
  ([root opts]
   (let [delete-fn (fn [{:keys [path attrs accumulator simulate]}]  
                     (try (if-not simulate
                            (Files/delete path))
                          (swap! accumulator conj (str path))
                          (catch DirectoryNotEmptyException e)))]
     (walk/walk root
                (merge {:directory {:post delete-fn}
                        :file delete-fn
                        :with #{:root}
                        :accumulator (atom #{})
                        :accumulate #{}}
                       opts)))))
link
(do (copy "src" ".src" {:include [".clj"]}) (delete ".src" {:include ["test.clj"]})) => #{(str (path ".src/hara/test.clj"))} (delete ".src") => set?

list ^

lists the files and attributes for a given directory

v 2.4
(defn list
  ([root] (list root {}))
  ([root opts]
   (let [gather-fn (fn [{:keys [path attrs accumulator]}]
                  (swap! accumulator
                         assoc
                         (str path)
                         (str (permissions path) "/" (shorthand path))))]
     (walk/walk root
                (merge {:depth 1
                        :directory gather-fn
                        :file gather-fn
                        :accumulator (atom {})
                        :accumulate #{}
                        :with #{}}
                       opts)))))
link
(list "src") => {"/Users/chris/Development/chit/hara/src" "rwxr-xr-x/d", "/Users/chris/Development/chit/hara/src/hara" "rwxr-xr-x/d"} (list "../hara/src/hara/io" {:recursive true}) => {"/Users/chris/Development/chit/hara/src/hara/io" "rwxr-xr-x/d", "/Users/chris/Development/chit/hara/src/hara/io/file/reader.clj" "rw-r--r--/-", "/Users/chris/Development/chit/hara/src/hara/io/project.clj" "rw-r--r--/-", "/Users/chris/Development/chit/hara/src/hara/io/file/filter.clj" "rw-r--r--/-", ... ... "/Users/chris/Development/chit/hara/src/hara/io/file/path.clj" "rw-r--r--/-", "/Users/chris/Development/chit/hara/src/hara/io/file/walk.clj" "rw-r--r--/-", "/Users/chris/Development/chit/hara/src/hara/io/file.clj" "rw-r--r--/-"}

move ^

moves a file or directory

v 2.4
(defn move
  ([source target]
   (move source target {}))
  ([source target opts]
   (let [move-fn (fn [{:keys [root path attrs target accumulator simulate]}]
                   (let [rel   (.relativize ^Path root path)
                         dest  (.resolve ^Path target rel)
                         copts (->> [:atomic-move]
                                    (or (:options opts))
                                    (mapv option/option)
                                    (into-array CopyOption))]
                     (when-not simulate
                       (Files/createDirectories (.getParent dest) attr/*empty*)
                       (Files/move ^Path path ^Path dest copts))
                     (swap! accumulator
                            assoc
                            (str path)
                            (str dest))))
         results (walk/walk source
                            (merge {:target (path/path target)
                                    :recursive true
                                    :directory {:post (fn [{:keys [path]}]
                                                        (if (empty-directory? path)
                                                          (delete path opts)))}
                                    :file move-fn
                                    :with #{:root}
                                    :accumulator (atom {})
                                    :accumulate #{}}
                                   opts))]
     results)))
link
(do (move "shortlist" ".shortlist") (move ".shortlist" "shortlist")) (move ".non-existent" ".moved") => {}

option ^

shows all options for file operations

v 2.4
(defn option
  ([] (keys all-options))
  ([k]
   (all-options k)))
link
(option) => (contains [:atomic-move :create-new :skip-siblings :read :continue :create :terminate :copy-attributes :append :truncate-existing :sync :follow-links :delete-on-close :write :dsync :replace-existing :sparse :nofollow-links :skip-subtree]) (option :read) => java.nio.file.StandardOpenOption/READ

select ^

selects all the files in a directory

v 2.4
(defn select
  ([root]
   (select root nil))
  ([root opts]
   (walk/walk root opts)))
link
(->> (select "../hara/src/hara/io/file" ) (map #(relativize "../hara/src/hara" %)) (map str) (sort)) => ["io/file" "io/file/attribute.clj" "io/file/common.clj" "io/file/filter.clj" "io/file/option.clj" "io/file/path.clj" "io/file/reader.clj" "io/file/walk.clj"]

4    Advanced

As all bulk operations are based on hara.io.file.walk/walk, it provides a consistent interface for working with files:

4.1    depth

The :depth option determines how far down the directory listing to move

;; listing the src directory to a depth of 2
  
(list "src" {:depth 2})
=> {"/Users/chris/Development/chit/hara/src" "rwxr--r--/d",
    "/Users/chris/Development/chit/hara/src/hara/data.clj" "rw-r--r--/-",
    ... ...
    "/Users/chris/Development/chit/hara/src/hara/function" "rwxr-xr-x/d"}

;; copying the src directory to a depth of 2
(copy "src" ".src" {:depth 2})

;; delete the .src directory to a depth of 2
(delete ".src" {:depth 2})

The :recursive flag enables walking to all depths

;; lists contents of src directory, :recursive is false by default
(list "src" {:recursive true})
=> {"/Users/chris/Development/chit/hara/src" "rwxr--r--/d",
    "/Users/chris/Development/chit/hara/src/hara/data.clj" "rw-r--r--/-",
    ... ...
    "/Users/chris/Development/chit/hara/src/hara/function" "rwxr-xr-x/d"}

;; copying the src directory, :recursive is true by default
(copy "src" ".src" {:recursive false})
=> {"src" ".src",
    "src/hara" ".src/hara"}
  
;; delete the .src directory, :recursive is true by default
(delete ".src" {:recursive false})
=> #{"/Users/chris/Development/chit/hara/.src/hara"
     "/Users/chris/Development/chit/hara/.src"}

4.2    simulate

When the :simulate flag is set, the operation is not performed but will output as if the operation has been done.

(copy "src" ".src" {:simulate true})
=> {"src" ".src",
    "src/hara" ".src/hara"}

(move "src" ".src" {:simulate true})
=> {"/Users/chris/Development/chit/hara/src/hara/data.clj"
    "/Users/chris/Development/chit/hara/.src/hara/data.clj",
    ... ...
    "/Users/chris/Development/chit/hara/src/hara/time/data/coerce.clj"
    "/Users/chris/Development/chit/hara/.src/hara/time/data/coerce.clj"}
  
(delete ".src" {:simulate true})
=> #{"/Users/chris/Development/chit/hara/.src/hara"
     "/Users/chris/Development/chit/hara/.src"}

4.3    filter

Files can be included or excluded through an array of file filters. Values for :exclude and :include array elements can be either a pattern or a function:

(select "." {:exclude [".clj"
                       directory?]
             :recursive false})
=> [;; #path:"/Users/chris/Development/chit/hara/.gitignore"
    ;; #path:"/Users/chris/Development/chit/hara/.gitmodules"
    ...
    ;; #path:"/Users/chris/Development/chit/hara/spring.jpg"
    ;; #path:"/Users/chris/Development/chit/hara/travis.jpg"
    ]
  
(select "." {:include [".clj$"
                       file?]
             :recursive false})
=> [;; #path:"/Users/chris/Development/chit/hara/.midje.clj"
    ;; #path:"/Users/chris/Development/chit/hara/project.clj"
    ]

:accumulate is another options to set that specifies which files are going to be included in in accumulation:

(select "." {:accumulate #{}})
=> []
  
(select "src" {:depth 2
               :accumulate #{:directories}})
=> [;; #path:"/Users/chris/Development/chit/hara/src"
    ;; #path:"/Users/chris/Development/chit/hara/src/hara"
    ]

(select "src" {:depth 2
               :accumulate #{:files}})
  
=> [;; #path:"/Users/chris/Development/chit/hara/src/hara/class.clj"
    ...
    ;; #path:"/Users/chris/Development/chit/hara/src/hara/time.clj"
    ;; #path:"/Users/chris/Development/chit/hara/src/hara/zip.clj"
    ]

4.4    file system

The :file option takes a function which will run whenever a file is visited:

(select "src" {:include ["/class/"]
               :file (fn [{:keys [path]}] (println (str path)))})
;; /Users/chris/Development/chit/hara/src/hara/class/checks.clj
;; /Users/chris/Development/chit/hara/src/hara/class/enum.clj
;; /Users/chris/Development/chit/hara/src/hara/class/inheritance.clj
;; /Users/chris/Development/chit/hara/src/hara/class/multi.clj

The :directory option takes either a function or a map of function which will run whenever a directory is visited:

(select "src" {:include ["/class"]               
               :directory (fn [{:keys [path]}]
                            (println (str path)))})
  
;; /Users/chris/Development/chit/hara/src/hara/class

(select "src" {:include ["/class"]               
               :directory {:pre  (fn [{:keys [path]}]
                                   (println "PRE" (str path)))
                           :post (fn [{:keys [path]}]
                                   (println "POST "(str path)))}})
  
;; PRE /Users/chris/Development/chit/hara/src/hara/class
;; POST  /Users/chris/Development/chit/hara/src/hara/class

:options are passed in for move and copy

;; `:replace-existing` replaces an existing file if it exists.
;; `:copy-attributes`  copy attributes to the new file.
  
(copy "project.clj" "project.clj.bak"
      {:options [:replace-existing
                 :copy-attributes]})

;; `:atomic-move` moves the file as an atomic file system operation.
  
(move "project.clj.bak" "project.clj" 
      {:options [:replace-existing
                 :atomic-move]})

:with is either #{:root} to include the root path or #{} to not include the root path. It is set to #{:root} for copy, move and delete. It is set to #{} for list and select.

4.5    accumulator

:accumulator sets the atom that contains the accumulated values during the walk:

(let [acc (atom [])]
  
  (select "src/hara/class"  {:accumulator acc})
  
  (select "src/hara/common" {:accumulator acc})
  
  (map #(str (relativize "src/hara" %))
       @acc))
=> ("class"
    "class/checks.clj"
    "class/enum.clj"
    "class/inheritance.clj"
    "class/multi.clj"
    "common"
    "common/checks.clj"
    "common/error.clj"
    "common/hash.clj"
    "common/pretty.clj"
    "common/primitives.clj"
    "common/state.clj"
    "common/string.clj"
    "common/watch.clj")

For more examples of how it is used, please see the source code for copy, delete list and move.