Pivoting into new flows - life, location, Rabbit

The true life blood of a "data app" is the flow of data through all the components + how they relate to each other in the world...

Pivoting into new flows - life, location, Rabbit
A "Flow-based Clojure Notebook" with mixed front-end and back-end blocks? As If.

Over a year since the last post here? Damn.

Needless to say, a lot has happened - moved out of the Bay Area, new day job, some serious life stuff, etc. Given how tough the last couple years have been for all, I'm sure there are many other stories like it.

Confusing times. Hard times. Enlightening times. Anyways - as much has changed - I've never stopped iterating on my Data Rabbit project. This has become less of a blog - rather a personal diary of it's development history.

Hopefully this makes intuitive sense to someone other than me. ;)

I don't even know how many iterations I've gone through in terms of UI and backend micro-pivots over the years. It's funny how throwing away interfaces, ideas, code and refactoring so often feels like failure, at least at first - but is essentially just refining and rediscovering you own project.

What's the phrase in regards to literature - don't be afraid to "kill your darlings"?

Painful, but you learn much more about what you are building in this process.

In fact, without being 100% aware of it at the time, I basically flipped the entire perspective of the tool on it's side.


"a editable configuration-based system of pre-fabricated blocks, thoughtfully assembled"


"A river of REPLs" - blocks that can render UI, execute backend code, and pass that output to other blocks. Thus creating a set of "primitives" that more complex systems can be built out of.
Not a "REPL", but a... REVL? REDL? We might be beyond "Print". ;)
- Read
- Eval
- Visualize / Display (?)
- Loop
Maybe I don't even want a dashboard, maybe I just want an interactive look at all my SQL and relevant graphs in concert together. Here I'm inspecting some spicy Bigfoot sighting reports... This will eventually become a Bigfoot dashboard that features heavily in my documentation.

I just wanted to drop a quick update on some of those changes - and tease that a much larger set-by-step demo video will be released soon - followed by an Alpha1 version that everyone can use and run locally and hopefully give me some  (constructive) feedback on.

As I was saying - looking at my previous posts on DR, I focused a lot on presentation and composition of pre-defined 'bespoke' elements, while still keeping a "live-coding / live-config IDE" type feel.

Still somewhat close to the metal, but with very defined guardrails...

But at some point I realized, while all that is well and good...

The life blood of a [dashboard, data experiment, viz story, query set, whatever] is the flow of data through all the (individual and simple) components + how they relate to each other in the overall world.

Only then does some form of "visual composer" actually take center stage.

Server side and client side blocks created and then composed into a single UI element

Ah, so now we are back to "visual composition", so it shares some familiarity with my old approached but at the same time... not at all.

Note the block / layer navigator on the left - "composed-view" being a container for the previously defined CLJS and CLJ blocks - separating the logic of the block itself from the formatting as much or as little as you want.

A composed view block is just yet another block on the Rabbit canvas alongside the more "mechanical" blocks - as opposed to it being the canvas itself.

Of course we retain the ability to do compound recursive composition. With Horizontal, Vertical, and free-floating layouts - nested in each other...

Another view composer, this time also comprised of OTHER view composers - some vertical, some horizontal and one is free floating dragged arranged... Note: was testing out some simple VEGA LITE stuff here, so the charts themselves are just crappy placeholders.

And obviously, this wouldn't be very useful without blocks being able to "talk" to other block via UI elements. Enabling cascading filters and all sorts of other interesting interaction. The difference is you can SEE the interconnections playing before you put the output blocks together to form a presentation block...

Slightly ugly prototype Flowset example using BigQuery SQL calls, but I think you get the idea...

That being said, the core of the user-experience is now a series of "flow blocks" passing parameters and output (Reactions) from one to another.

  • Some blocks execute code on the server w CLJ
  • others render on the front-end w CLJS (with hiccup, re-com, vega, whatever)

Exposing all these REPLs makes things much more complicated than the "pre-fab with EDN config" blocks I was using before - but it opens up so much more opportunity.

Since now everything - UI elements, charts, server-side JDBC connections, SQL strings, Machine Learning process blocks - are made up of the same "stuff"  and can be saved piece by piece and re-assembled in new and interesting ways.

Server-side SQL queries (JDBC DuckDB in this case) surfaced to re-com UI boxes, grids, a front-end row count, random hiccup. Notice the edit panel of the "454" block showing it's inputs and outputs...

(was watching Scrooged that day, for the 100th time. "Niagara Falls, Frankie")

Simple brick by brick UI passing to UI example. Flow lines indicate data type. You would likely never do this in a "real" workflow (unless you were building a sub-flow for a "compound block" component) - but that fact that you CAN... awesome.

CLJS blocks that not only render the UI input dialog (in this case - sliders and color pickers) but also send the output as a parameter. No real hidden magic, just several forms of input and output - working together how you would expect.

A "real" flow-set I use for monitoring the backend JVM process and connected clients. Again, not a "dashboard", per se - but it doesn't need to be. Each block updates at different rates, I'm just "watching it work".

Affordable 8k Monitors can't come soon enough! I've got a river of REPLs to visualize. The real Buffy understands.

A JDBC connection map - browsing tables and "dragging out" SQL starter queries. Even though all our "primitive" blocks are base clj / cljs - there is no reason we can't have "special" ones also...

Here is a special case - since SQL is so important in my world - I wanted server code blocks to recognize JDBC connection maps and present a novel interface for pulling out simple exploratory queries from the metadata.

Not sure how you work, but I often find myself doing lots of "select *" and "select distinct" to taste the data. This expedites that even faster than my SQL IDE... plus the cached result blocks can stick around and be a visual reminder of table contents.

"Table Crawling"

Simple JDBC "Table Crawling"

A parting thought for January 2022...

It's interesting that I initially set out (during the FB prototype days) to build a "more code-aware version of Tableau for building dashboards" and ~4 years later that concept has evolved to be more like "an Unreal Engine Blueprints for data projects, visual apps and flows - in Clojure".

Quite a shift, yet in my mind it's still the same thing - just a much different set of abstractions.

Until next month, hope you all are doing amazing.