Clojure

EDN data DSL

I keep some loosely structured habit and travel information in a set of EDN files. For a while I used an actual database, but I quickly discovered that I spent a lot of time tinkering with the UI for inputting data. I also tried spreadsheets, but the data is sparse and was awkward in a tabular format. Also, spreadsheets aren't very easy to query. The scheme I've settled on is to have a folder of EDN files that contain Clojure code, which when evaluated produce a list of maps.

To parse an individual file, I wrap its contents in square brackets and read it as a vector of EDN forms:

(defn parse-file [s]
  (clojure.edn/read-string {} (format "[%s]" s)))

Before evaluating a file, I bind the date in the filename to a dynamic var, then I evaluate the EDN forms as if they were Clojure code:

(def ^:dynamic *date*)

(defn eval-file [nmsp file]
  (binding [*date* (re-find #"\d{4}-\d{2}-\d{2}" (.getName file))]
    (->> (slurp file)
         parse-file
         (mapv (fn [form]
                 (binding [*ns* nmsp]
                   (eval form))))
         (remove var?)
         (mapv #(cond-> %
                  (not (:date %)) (assoc :date *date*))))))

In reality, EDN is only a subset of Clojure syntax. I can't use Clojure's reader macros, such as @ for dereferencing atoms, nor can I create anonymous functions with #(...). But non-reader macros such as def and defn do work, so after evaluating forms I discard vars. Finally, for any maps without a :date field, I add the date given by the filename.

A lot of power hides in that eval. Though I'm generating a list of maps, I typically don't write out the maps explicitly; instead, having Clojure available, I define functions that return maps, then call those functions. For example, instead of writing

{:lat 43 :lon -85}

I'll write

(defn location [lat lon]
  {:lat lat :lon lon})

(location 43 -85)

Defining and invoking functions like this keeps the maps for specific pieces of information structured consistently, and allows me to easily change details of a map, such as changing the names of keys (e.g., changing :lat to :latitude). It also avoids typos in the names of keys, which evaling doesn't detect, but eval will throw an error if I mistype the name of a function.

Using this style, I've built up a DSL of common entries. A typical file looks something like this:

(location 43 -85)

(outdoors 1030 1130)

(-> (walk)
    (route "park.geojson")
    (note "warm day, unusually busy"))

(no-tv)

Functions like route and note are modifier functions that add optional data to maps, and having most of the power of Clojure makes it easy to chain modifiers with ->.

Another subtlety in eval-file is that forms can refer to *date*. I don't reference it directly, but I do have tag-reader macros called #yesterday and #tomorrow that read *date* and rebind it to the date before or after, in case I want to refer to events that extend beyond the date named by the file. To use them, I supply :readers in parse-string:

(defn parse-string [s]
  (clojure.edn/read-string
    {:readers {'yesterday yesterday
               'tomorrow tomorrow}}
    (format "[%s]" s)))

Here's the main entry point, which reads the files in a directory:

(defn read-data [directory]
  (let [nmsp *ns*]
    (->> (file-seq (clojure.java.io/file directory))
         (filter #(re-find #"\d{4}-\d{2}-\d{2}" (.getName %)))
         (sort-by #(.getName %))
         (mapcat #(eval-file nmsp %)))))

Note that the files are evaluated in the current namespace, which means a function defined within one file is available in all subsequent files. Alternatively, you can group function definitions in a library file and evaluate it before processing the data files.

Querying this DB is as easy as getting a list of maps from read-data and filtering them with Clojure's built-in collection functions. The above code can all be run in Babashka, so I have a suite of bb tasks that print reports on various aspects of the data (such as how much time I spend outdoors). Those plain-text reports can then be piped to commands that generate data visualizations.

Overall, this setup allows me to have a low-maintenance log that produces well-structured, queryable data, yet is also sufficiently human-readable.

June 2023