Hyperfiddle Technology Vision

Table of contents

Hyperfiddle isolates your web clients from I/O, so your UI code can stay pure. If React.js is managed DOM, Hyperfiddle is managed network and database. This enables a new kind of composable primitive for constructing web software, with paradigm-changing implications.

Motivation

The technical problem we set out to solve is Composable UI. The hard part of UI is actually not the frontend views, but rather the data-sync to the API (and then from API to database), because network I/O doesn't compose. This async, slow, failure-prone concern is the reason web developers code basically the same web boilerplate over and over again. GraphQL is a good idea in this space, but is insufficient because it requires
mapping API endpoint to database query, an abstraction resistant problem
that needs lots of coding.
We think Clojure/Script ecosystem presents an opportunity to fix this:
  • React.js virtual dom is a leap forward in composition of views, ClojureScript makes this better
  • Datomic is a leap forward in composition of databases and data services
  • How can we combine these things into some composable primitive for building applications?
  • BTW we also have Clojure, CLJC, eval, Transit, Pedestal, Reagent, funcool/cats, core.async, all of which advance the composition story in their respective spaces.
  • Most importantly: EDN, a data notation which shapes the way we think.

Hyperfiddle is like HTML

  • fiddles have links to other fiddles (a directed graph, called the fiddle graph)
  • Three kinds of link: anchor, button and iframe (not actually an HTML iframe, but individually URL addressable like iframes)
  • links understand data cardinality (so a link at the collection level will repeat for each row of the resultset)
  • Select options are iframes with style
  • buttons are like HTML form submits, they have a form body and a CLJC function to control the Datomic transaction
  • Fiddles compose in both view and in API. Views are React.js trees. APIs are graphs. API responses are a set of queries and results.
  • The fiddle graph specifies the structure of your application (both UI and API) as data (instead of source code)
  • Fiddle graph is stored in a database (instead of git) and is available for reflective query
  • which makes possible managed I/O

Managed I/O

  • fiddle graph captures API inter-dependencies as data
  • data sync reduces to a simple graph partitioning problem (find connected components of the graph)
  • Precise and total data fetching in one network trip
  • Userland not exposed to async / latency / failure, code with pure functions
  • Hyperfiddle is implemented in CLJC (your app is compiled into both frontend and backend)
  • In parallel, your app runs in:
    • Datomic peer to do database things
    • Web browser to do React.js things
    • Node for server rendering
    • JVM to do web service things
    • AWS Lambdas to do unsafe sandbox things like eval
  • All these places have the fiddle graph, so they are able to coordinate automatically
  • The distributed system as a whole is entirely data driven and abstract

Optimizing I/O Runtime

  • Fiddle graph is abstract; the concrete I/O implementation is unspecified
  • Separation of I/O from business logic
  • I/O runtime has fiddle-graph available for inspection, so it understands application structure
    • pages, links, data dependencies
  • Different features have different I/O needs
    • lazy loading, realtime push, server rendering
  • Different levels of scale have different I/O needs
  • Like an optimizing compiler, runtime can optimize in ways that hard-coded I/O cannot
  • Runtime can adjust transport strategies on the fly
  • Datomic (immutable log) captures information about what changed when, which the optimzer can query
    • means optimizer can partition responses into subgraphs with high cache cohesion
    • data that rarely changes is cached separately from things that change often (realtime chat)
  • (insert Machine Learning here)

Data-driven UI with core.match

  • Out of the box CRUD functionality (new/edit/remove, managed parent-child references, etc)
  • Forms and tables are driven by core.match on Datomic pull and Datomic schema
  • matching function is injected from userland – provide your own core.match and dispatch to your own widgets!
  • matching function is a "hypermedia function" which is like a web browser, but for fiddle graphs

Semantic ontologies (data driven ++)

  • Datomic is basically immutable RDF
  • Hyperfiddle datamodel is semantic, links have rels and classes (like HTML)
  • Semantics means machine readable, e.g. enables computer aided design and tool assistance
  • User interface understands what the links are meant for
  • Semantics are open to extension because everything is data and stored in Datomic
  • Not just for views. I/O Runtime uses semantics to optimize data sync (e.g. prefetch :next-page), this also is how Chrome works

Interactive Editor for CRUD apps

  • Imagine something like the Unity game engine
  • "source code" is EDN, stored in database instead of git. No more filesystem
    • There is some ClojureScript for the views, think of it as HTML templates... web designer stuff
    • This is stored as strings in the database and eventually baked
    • 1) Stateless 2) view is an expression 3) optional
  • Understands the core ontologies (Datomic, Fiddle graph, Forms, CRUD)
    • Can be extended with more ontologies
  • Bootstrapped – Hyperfiddle IDE is a hyperfiddle
  • A business model compatible with open source
    • busy people will pay for convenience and it gets better over time
    • git command line is free, time-saving git GUIs are a business
    • avoid "git clone whatever && lein new hyperfiddle --whocares"
  • Gets Hyperfiddle Server inside lots of companies
  • Already useful today as Datomic administration tool

Hyperfiddle Server

  • Hyperfiddle is HTML for Data
  • Run your own Hyperfiddle Server (Apache for Data)
  • Data security is first class – PII never leaves your controlled environment
  • All fiddles are APIs and always URL addressable
  • All responses are immutable, have a time-basis, are valid forever
  • Service discovery: Paste in a URI and see what APIs are now available
  • Open source

Like the world wide web, but for data (experimental)

  • Software is meant to interop with other software. Lets make that the default
  • All Hyperfiddle Servers work the same way, speak the data protocols
  • Distributed fiddle graph – links can be URIs
  • Everything is immutable and has a time-basis, caches are valid forever, so nodes work with whatever data they're allowed to see
  • "Schema stitching" (merging services into new services) falls out of distributed fiddle-graph for free

Data security and PII (experimental)

Hyperfiddle can express the entire continuum of data ownership:
  1. megacorp has your data and you can't see it
  2. megacorp has your data and offers filtered access via a UI
  3. megacorp has your data and offers snapshots / queries
  4. megacorp has your data and lets you mirror it effectively
  5. you have your data but let megacorp mirror it effectively
  6. you have your data but offer megacorp queries
Software built on a relational database constrains your choices to 1-3. HTTP/REST/GraphQL is, ipso facto, #2. Hyperfiddle's I/O Runtime abstraction decouples your application structure from your I/O choices and thus offers the entire spectrum of choices. You configure your data security choices, for whatever data you control, in a settings panel. Hyperfiddle Server will satisfy any requests in the most most efficient way it can, subject to your choices.
We believe we must not adopt a stance on which place is best – different kinds of data have different security requirements – we merely offer you the ability to make an explicit choice, rather than have it be accidentally established by implementation constraints.
  1. Getting Started
  2. Overview
  3. Tutorial
    1. Simple blog
    2. Seattle
  4. Concepts
    1. Design
    2. Technology Vision
    3. I/O Runtime
    4. Transactions
    5. Foreign integrations
    6. ctx
  5. Guides
    1. hyperfiddle.ui
    2. Select options
  6. Triage
    1. Architecture braindump
    2. Technical braindumps
    3. hyperfiddle.jar