Hyperfiddle, a developer-grade low-code database application platform, based on Clojure
Our mission: Empower all knowledge workers with database programming
September 12, 2019
September 12, 2019
Developers have the power to create database systems that positively impact our world. At Hyperfiddle, we believe any code-savvy person should be able to use software to help our world. To accomplish this, we’re doing what many think is impossible: bring the power of serious, developer-grade applications within reach of any code-savvy person. If accomplished, it would make it so that the value created by software can be captured by all of us, not just an elite minority.
Database-driven UI development, no coding needed
Well-formed Data at rest is as close to perfection in programming as it gets. All the crap that had to happen to put it there however..— fogus.db2 (@fogus) April 11, 2014
Database programming is really hard. Learning a query language to query the database is straightforward. But a finished application is so much more: it has a schema, a backend, and a frontend. It connects to APIs. It has a custom UI. All these things require coding to fetch and synchronize data between all these places, accounting for loading and error states, and update the app’s UI.
All of that just to present a person with some data.
"I am familiar with databases and APIs, but I'm not a web developer, which is a big sticking point for me." — beta tester
Imagine how simple it could be if all you had to do was type a data query, and the computer would take care of all the low-level details to display the data in a user interface that you can immediately use to solve your problem. All with no coding needed. This dream is what inspired us to create Hyperfiddle.
Hyperfiddle is a low-code database application platform — think Ruby on Rails meets Google Sheets — for both developers and code-savvy makers. Hyperfiddle lets both groups rapidly program transactional database applications that store valuable business information of record. Some example use cases are transactional data, business records, medical records, financial records, scientific records, inventory, configuration, departmental databases, and web applications.
"We should be able to write web apps simply by telling it what data we want to connect to, and handing things off to layout managers, io/control managers, etc" — beta tester
Hyperfiddle is fun, but it's not a toy. There is no abstraction ceiling as to what you can build. When the day comes that you need native code, an engineer can extend Hyperfiddle with plugins, or even bypass Hyperfiddle entirely and connect directly to the database in the usual way. Just use Hyperfiddle for the pages where it makes sense.
Links, forms and tables
80% of a database application is querying data, and then presenting the data as forms and tables. Thus Hyperfiddle is 80% of what you need to make a database application.
“I've only been doing web dev for a year, and during that whole year I've been constantly feeling that something like this should exist.” — beta tester
Business users and product owners already understand data. They have to, in order to ensure the right product gets built.
Hyperfiddle empowers a code-savvy business person to write the native database queries that define straightforward business rules, and then the system will take care of the user interface, database backend and datasync, letting you modify your system in real-time as you type.
8 years of research
We’re Karl and Dustin. We’ve been building the same cookie-cutter application — shuffling data between database and UI — over and over for our entire career. We thought there has to be a better way. So 8 years ago, we started asking deep questions about why things are the way they are.
We learned that the reason database programming is hard is that the data structures that we program with are not the same as the data structures in the database. This forces the maker to hold multiple programming paradigms in their head at the same time, and write difficult, performance-sensitive code that translates between paradigms.
This is why database applications are so complicated.
Solve this, and you solve database applications.
Clojure and Datomic
Clojure is an emerging programming language, known for its emphasis on simplicity and its elegant approach to state and identity.
If the goal is low-code, simplicity through elegance is the right place to start.
"How can we possibly make things that are reliable that we don't understand?" — RICH HICKEY, author of Clojure and Datomic
Rich has spent the last 10+ years of his life studying how to make programming simple:
In 2012, Rich unveils Datomic, a graph database (competitive with SQL databases) based on an immutable log. Datomic enables functional programming in the database, for a post-AWS world.
One interesting property (of many) is that Datomic and Clojure share the same immutable data primitives. In other words, Datomic + Clojure merge the database into the programming language — or is it the programming language into the database? — such that there is no impedance between them. With Datomic, the Clojure programmer can walk the database as if it were a local data structure. No massive rift with your process on one side and your database on the other side. This property of Datomic is called code-data locality.
Same data structures everywhere is the Holy Grail primitive we’ve been looking for, that can be used to vastly simplify application programming — for actual engineers, not just low-coders — by flattening all the same stuff into fewer layers.
Hyperfiddle extends this idea all the way through the UI:
All the same data structures. All work the same way.
It's just data.
Low-code, meet Clojure
Clojure already has a principled basis for low-code: data-driven programming. An example of something data-driven is a Datomic query. Datomic queries are declared as vanilla Clojure data structures. They are not strings. They do not have special syntax. They do not have if-statements, types, async, or failures. They are just data.
Because it is just data, Hyperfiddle simply inspects the query data and uses it to automatically generate working UI. Datomic query data (taken in context with Datomic schema data) contains precisely the metadata necessary to render a working UI, so this is a mechanical transformation.
For the write side, the Datomic schema describes perfectly how to issue a transaction. We can even stage transactions locally, testing the results of queries before pushing the transaction up to the cloud. This works because Datomic is structurally an immutable log, like Git, so you can stage database edits locally and test them before you push.
The end result is that Hyperfiddle is safe: use trial and error to learn and explore; it will never transact without your explicit command.
"Rather than reinvent the wheel for all our 'back of house' UIs it would be great to have something like Hyperfiddle solve that out of the box."
When the day comes you are ready for a proper engineering team, Datomic has your back. Datomic is not a toy, nor an experiment, nor a startup. Datomic is serious, enterprise-class data infrastructure for serious technology companies.
To build it, you must be willing to throw away everything
We think the software development industry has lost its way. Sometimes the fastest way to make progress is to realize that you're on the wrong path, turn around, and walk backwards.
Hyperfiddle has been hard to build. It was not clear from the beginning that it was possible to build. It still isn't clear. Hyperfiddle is currently not suitable for anything, unless you are an experienced Clojure programmer who is already using Datomic and is willing to work with us through our early days.
But we think, with computer science on our side, we may be able to make our low-code dream come true: letting anyone use software to innovate and help the world.
Here are some complexities you no longer care about because we manage all of it:
- REST, AJAX, HTTP, Async, Promises, Latency, Failure
- NPM, node_modules, Node, Express
- JVM, Servers, Infrastructure, Cloud, Deployment
- Rendering, History, PWA, SSR
"It's the biggest breakthrough I've seen since Meteor, in a web environment that feels lacking in imagination"
Lisp hackers have been doing this for sixty years.
It’s. Just. Data.