Lessons around creating UI builders

Web UI tools are something that I’ve been interested in for a couple of years now, mainly because of how painful web development currently is.

Very slow HMR

Developers have to wait a few seconds before changes appear in the browser, and that time adds up since UI development is so iterative. There had to be a faster approach that was more like painting, so I set out to create a better tool for building UIs.

As confident as I was about creating something that worked, trying to come up with something practical was far more challenging than I expected. Over the course of 4 years I’ve had many failed ideas, but I’ve had a fun time and learned a lot.

V1: Building a visual editor that could edit anything

My first approach to a UI tooling back in 2016 was ambitious. The idea was to allow users to open their existing app in the UI editor and allow them to start making changes immediately.

The editor was built to run any kind of web application, and could write back to source code by following source maps (React, Less, SCSS, Angular, you name it). Here’s another demo of the editor making changes to a live website:

However, things broke down quickly. For starters, the builder couldn’t write source code for all scenarios such as computed data. Take this for example:

Assuming a=b, and b=2, the user would see 3 within the UI builder. Since this is computed information, it would need to be uneditable since the UI builder wouldn’t know whether to edit a or b. This is a rudimentary example, but it turns out that a lot of app code is computed in a way that’s really hard to back trace and edit. My solution was to limit the UI builder to only parts of the application that it could easily edit, but the user experience was confusing since there wasn’t a clear boundary between editable and non-editable code.

There were also a number of other issues with the editor: sometimes it would write source code in the incorrect spot (source maps weren’t always accurate), some frameworks couldn’t be loaded without being monkey-patched, and certain native calls wouldn’t work such as getBoundingClientRect (this couldn’t be fixed since the editor used a browser VM). The issues kept piling on.

In the end the project was just too big, and took all of my time to manage. A better approach would have been to start small with clear boundaries, then iterate from there.

V2: Building a presentational component editor

In order to get on the right track, I started with a question: how do I create a UI builder using itself? I figured that if the editor could be used to build itself, then it it could handle just about any complex case.

Logic was eliminated from the scope from the get-go since I couldn’t imagine any scenario where I would build an editor and all of its logic visually. Handwritten code is the best medium in this case. The editor’s appearance on the other hand could be built visually, and the boundaries around that are pretty clear. Components, slots (ability to insert children in various parts of a component), variants, and dynamic attributes are really the only things necessary for a UI builder to cover, so I focused just on those features, plus an API that would allow the presentational components to be wired up with code.

Initially I wanted to build something that would allow users to freely draw canvas elements without constraints. The idea was to have a user experience similar to Figma, then allow users to annotate elements with layout information after they’re done, similar to how Pagedraw worked.

I figured that if users could freely drag elements around a canvas like this, the next evolution would be to allow Figma designs to be imported directly.

In practice, this approach broke down since I found that the structure of absolutely positioned elements doesn’t always reflect what‘s needed for a responsive UI.

Freely dragging elements around the canvas oftentimes produced HTML and CSS that I would have to refactor entirely in order to add layout information. It was a mess. To fix this, I thought of building UI controls on top of a custom layout engine that abstracted HTML and CSS, but that didn’t seem like a safe idea considering how complex CSS can get (in fairness, there’s a lot you can do with a limited set of CSS features, so the editor doesn’t need to do everything, but a custom layout engine was still too big of a leap). I needed to work on tooling that I felt certain could handle every case.

I felt that it would be better for the editor to guide the user into writing HTML and CSS correctly from the beginning, without abstractions. That required a user experience that was more of an augmentation on top of HTML and CSS. Here’s my first take on that:

This felt better. Next, I wired up some design files with the editor source code. At this point I just about stopped writing JSX code, and moved over to the UI editor.

UI controls were shaping up. Here’s an early screenshot of the style panel:

Here’s the editor finally coming together into something usable:

The user experience at this point was starting to feel polished. Here’s a demo of a radio button being created:

Fast forward a bit, features like split views, global CSS variables, and mixins are implemented:

Towards the end of this project the editor had the ability to start projects from scratch, run CLI commands, and run UIs live.

At this point I was gathering feedback about the app. After a few dozen 1:1s with developers and designers, I was getting mostly lukewarm responses. Some people were excited, most people weren’t. I realized that I built something that stood in the middle of what a designer and developer wanted, while satisfying neither group completely, including myself. Designers didn’t like being restricted by CSS, and developers didn’t want to switch environments in order to visually edit HTML and CSS — it’s just easier to write it by hand most of the time.

I had enough feedback to lose interest. It also didn’t help that there were a number big issues:

1. The UI files were in JSON, and you needed to use the editor to make changes. This created all sorts of issues for things like collaboration, debugging, and maintenance. It would have been better to use a file format that was readable and editable by hand.
2. The style panel got in the way in some cases. For example, If I wanted add a background color, I’d need to scroll through all options before I could do that. As a developer I already know the CSS property most of the time (I also have intellisense to help me out), so it’s usually easier to type it.
3. I sometimes needed weird CSS behavior that could only be represented in text at the time: calc, pseudo elements(:before, :after), and other fringy CSS features. I could have added more UI controls for these kinds of features, but that was a lot of work that I didn’t have the energy for.

The editor was just too big of a leap, and felt off. I think that many of the issues could have been fixed, but the shaky foundation didn’t leave me confident in the direction, so I quit. There were a few positive things that I learned though:

1. The biggest benefit to the editor was the live aspect of seeing changes appear immediately.
2. The style panel was useful for making fine adjustments to things like colors, typography, and shadows.
3. Creating presentational components in isolation felt very productive since I only had to think about HTML and CSS. The boundary between presentational components and logic was also clean and intuitive.
4. Aside from a few issues around the data model, overall it scaled pretty well. Heck, I was able to build the editor UI within the editor!

With that I decided to start over again.

V3: Building code-focused visual tooling

The first editor that I built was designed to handle all code, and that didn’t work well since it isn’t always possible for visual tools to edit application code. The second editor that I built didn’t feel right in part because JSON is a poor place to store complex HTML and CSS. That led me somewhere in the middle: A tiny language designed around visual development.

I started building a template language that was limited to presentational components. It was expressive enough to build most user interfaces, but limited enough to allow a layer of UI tools. Another reason why the template language focused just on simple behavior was that I didn’t want to have any confusion where logic should go. Logic goes in app code, UI code goes in the template language (just HTML, CSS, and primitive components).

Here’s the prototype I came up with:

Full-circle back to code and an HMR experience.

I called the project Paperclip to reflect it being minimal, lightweight, and compatible with multiple languages and frameworks.

One of the problems that I had with the previous visual editor was that performance started to degrade for large files, so I decided to build the fastest thing I could think of: a preview engine built in Rust. Design files were also constrained to HTML and CSS so that they could remain lightweight and wouldn’t put pressure on the Rust engine. Here’s what I came up with after many months of polish:

I added measuring tools, artboards, color pickers, and updated the syntax. This felt great, and seemed like a step in the right direction.

I introduced Paperclip at my work (Hum Capital) a little over a year ago, and I’ve been incredibly happy around how helpful it’s been for everyone on the team. There’s still a long way to go in order to make Paperclip more accessible to designers and anyone else that wants to build user interfaces, and I’ll continue to work with my team to help move Paperclip in that direction. Now that it’s matured a bit though, I’m really happy to share it. Next time, I’ll tell you more about it!