A few years ago, the discussion forum Hacker News, where engineers collectively decide what other engineers should read, developed a quirk. A new phrase had entered the coder lexicon, and it seemed to propel links to the top of the page with such force that to some the rankings might have looked rigged. The phrase—“local-first software”— had an artisanal, farm-to-table sort of ring, at once familiar and flicking at something new. Perhaps some engineers dismissed it as merely a marketing term. But others whittling away their workday afternoons seemed to see it as the solution to a problem they had long sensed: The software they were writing was broken.
One of the first Hacker News links referred to a white paper published in 2019, coauthored by a computer scientist then at the University of Cambridge named Martin Kleppmann and a group of open source developers at an independent “industrial research lab” called Ink & Switch. Kleppmann and the others were alumni of successful tech startups that had done what successful tech startups are generally meant to do: be acquired. They had taken a turn inside their larger buyers and emerged repentant, disappointed with certain aspects of their industry. There were more software developers than ever, but they weren’t coding better experiences for their colleagues or their users. They were coding for the cloud.
The lament was not exactly new. A slogan printed on bumper stickers, T-shirts, and water bottles in Silicon Valley has long mocked the hometown industry with the statement “There is no cloud. There is just someone else’s computer.” That “someone else” being a corporation. Come to Sand Hill Road with an idea for a consumer-facing app, and there are two routes to a check sizable enough to get you written up in TechCrunch: Either monetize your users’ data for resale or advertising, or charge them a fee for accessing that data. Whichever cloud-based business model you choose—“Senator, we run ads” or “Pay us or else”—it’s imperative that the data run through your own servers.
The local-first white paper (“manifesto” might be the more appropriate term) pointed to a third way. The beauty of the cloud, for the average user, is that it is accessible from many devices and allows collaboration between many people across rooms and continents. The authors proposed to keep all that, but with software that was essentially cloudless. The word “local” in the name refers to your personal computer. “First” means your computer is prioritized over “someone else’s.” If you and I wanted to work on a document together, we would no longer have to depend on some Google data center in the Oregon high desert to maintain the master copy. Instead, we would each have copies stored locally on our devices’ hard drives. I could edit my copy offline, and you could edit yours, and the two files would reconcile our changes anytime they connect, whether once a minute or once a week.
To build products like this would require fundamentally different ways of structuring data. Different math. The result of that effort? Less shitty software. Freed from worrying about backends, servers, and extortionate cloud computing fees, startups and indie developers could skip strings-attached VC funding and pursue more interesting apps. What’s more, they could take advantage of hardware improvements that cloud developers often missed out on. When an app is cloud-based, its performance is limited by the speed of its connection to the central server and how quickly that server can reply. With a local-first app, the user’s device runs all the code. The better your laptop or smartphone gets, the more the app can do.
To a developer, the opposing trends of accelerating machines and stagnant load times are rather silly. Offensive, really. You should be offended too, because it means you’ve been missing out on something. The cloud appears heavenly, until it isn’t. Haven’t you noticed, lately, as belts tighten across Silicon Valley, that your own personal internet feels less abundant than before? That certain things are getting a little more expensive, or a little less convenient? A monthly cost to keep all your photos in storage or back up your phone. A premium upgrade to allow multiple users to edit the same file. A video game that requires a subscription and lags just as you go for the win.
The journalist and science fiction author Cory Doctorow uses the term “enshittification” to describe how platform capitalism squanders useful technology. A new platform, flush with venture capital, is first good to its users. Then advertisers come for its audience, and the platform is good to them too. Then, still hungry for profits, it poisons the well. It begins to interfere with the features that you value until you’re fed up. This is “how platforms die,” Doctorow writes. Cold business logic blazes this sorry path, but technological choices pave it.
Maybe that’s fine. Maybe the process is regenerative, like a wildfire clearing the underbrush. It makes way for new platforms to be good to us again, at least for a while. But what if something different could take root? What if changing the innards of software, invisible to most of us, could help nudge technology out of the shit?
Kleppmann and I are suspended three stories above the parking lot of St. Louis’ City Museum, an old shoe factory turned architectural playground and salvage yard. It’s closing time, and the guards would like us to climb down and clear out. Kleppmann, however, is aiming for the highest point of the structure, a hollowed-out business jet from the 1960s, accessible through a steeply angled tube made of chain link. He wears a royal blue sweater that somehow instantly clocks him as European, his frizzled orange-brown hair pulled into a tight ponytail. As he slides into the fuselage, I imagine I’m pursuing a fox.
The night at the museum is Kleppman’s favorite part of Strange Loop, which might just be his favorite developer conference. It’s an event that merges joy and weirdness with practicality—his ideal combination. Kleppmann is perhaps best known for a textbook called Designing Data-Intensive Applications, which explains the fundamentals of moving lots of data around vast computer systems. A quirky survival guide for the modern dev, it has sold more than 200,000 copies—enough to merit celebrity status in this community. Fans stop Kleppmann by the yawning mouth of a life-size whale sculpture and as he emerges from a five-story slide, thanking him for helping them land their first software jobs.
The seeds of the local-first manifesto can be found in a little box on page 174 of Kleppmann’s book. It describes something called a conflict-free replicated data type, or CRDT, which he defines as a “family of data structures” that allow many people to collaborate on a file and “automatically resolve conflicts in sensible ways.” In the book, Kleppmann notes that the implementation of CRDT algorithms is “still young.”
By computing standards, though, CRDTs themselves were old. They were co-developed by a French computer theorist named Marc Shapiro about two decades ago, when the cloud revolution was still nascent. Shapiro, who subscribed to many of the ideals of the peer-to-peer movement, was beginning to fear where cloud computing might lead the web. While the internet protocol itself remained open and decentralized, the stuff being built on top of it was moving in a monopolistic direction. The tech companies were growing beautiful gardens to lure users in—then building walls to discourage them from leaving.
One area they hadn’t yet totally conquered, however, was online collaboration. Connectivity wasn’t good enough at the time. Shapiro and his colleague Nuno Preguiça wondered: Did people have to be online to collaborate online? Or could they work offline and collaborate peer-to-peer?
Conceptually, it wasn’t so difficult to imagine: Create many replicas of the same file, each of which automatically snaps to a state identical to its peers, like atoms in quantum entanglement. Whether you edit your replica first and then receive my changes, or I edit my replica and then receive your changes, the algorithm produces the same result for us both. In math-speak, that’s the “commutative” property. (In fact, it’s what the “C” in CRDT initially stood for.)
How should the algorithm go about this? In most cases, the answer is straightforward. If I add one paragraph and you remove another, the order doesn’t matter. But suppose we each tinker with the same word; you think it should be purple and I think it should be mauve. What stops the result from being purmaupleve? Different CRDTs solve this with different rules meant to preserve the intent of the various collaborators. They may rely on timestamps to order the new elements, or perhaps have some way of encoding the relationship of each element to the elements around it, preserving some notion of words or sentences. The possibilities are numerous.
These tricks to preserve order can also make the CRDT horribly inefficient. It’s too much data to keep track of. So the other task for designing a CRDT is one of editing: deciding the minimum amount of information that replicas need to send each other in order to produce a harmonious result, and how to package those changes efficiently.
Shapiro and Preguiça initially published their CRDT algorithm as a technical report. Shapiro thought about starting a company focused on collaborative editing. “A few months later, bang, Google Docs comes out,” he tells me. The new software used an older process for merging changes called operational transformation, or OT, and still relied on a central Google server. Shapiro believed he had invented something that was more theoretically sound—a stable foundation for truly peer-to-peer software. But by the time Kleppmann came upon his paper years later, few people were using the software.
Kleppmann had grown up in Germany toying with both computers and his viola. After an abandoned flirtation with a career in composition (ivory tower notions of “what was good and what was rubbish” did not agree with him), he had followed the classic techie career arc: He cofounded a startup (called Rapportive, it integrated data from social media profiles into email contacts); he moved to the Bay Area (closer to investors and social media giants); his startup was acquired by a tech juggernaut (LinkedIn). Kleppmann lasted a few years before leaving to take a research position at Cambridge.
The new job gave Kleppmann what he had long wanted: a return to creativity. He had flexibility to explore more unusual approaches to programming, including projects that might not immediately pay off. He explains his work with an analogy borrowed from his wife, a high school chemistry teacher. If you think of individual bytes of data as atoms, then data structures are like molecules. For any new coder, the next step after “hello, world” is to learn these arrangements—lists, trees, hashes, and graphs, to name a few broad categories. What Kleppmann wanted to uncover were stranger atomic arrangements that could enable different types of applications.
He describes Shapiro’s paper as “an awakening.” In CRDTs, Kleppmann saw the technical basis for a new class of software that no one was providing. But the algorithms were mostly useless for professional programmers. They were too inefficient and lacked the typical tools that developers actually use to make apps. Kleppmann realized he would have to make local-first developers’ lives easy, shepherding the idea from a set of mathematical proofs to production-ready code. He set about coding an open source implementation of CRDTs, which he called Automerge, that people could freely use to build apps.
I saw the fruit of this effort a few years later, shortly after the local-first manifesto broke Hacker News. I met Peter van Hardenberg, one of Kleppmann’s coauthors, in a café in San Francisco. He was, like Kleppmann, rebooting after a long journey through the cloud, first as part of the founding team at Heroku, which helped other startups get their cloud services going, and then inside its acquirer, Salesforce. He wanted to show me an app called Pushpin, envisioned as a digital corkboard.
Van Hardenberg pulled up a blank project on his iPad. I loaded a replica of the same file on my laptop. We began tinkering, adding images and text boxes to our own files, and then allowed them to merge. Sometimes this worked seamlessly; other times the changes stopped loading, or the pixels dragged with dial-up-era latency. Pushpin felt like a toy, the sort of app that a couple of bright-eyed Stanford undergrads might code in the common room with visions of a seed round and later shelve in embarrassment.
But van Hardenberg was far from embarrassed. The technical groundwork was being laid, he believed, for local-first versions of Slack, Discord, Google Docs, Photoshop. Better design apps, calendars, budgets. More complex programs, too, if they could make Automerge far more efficient. There was the possibility of private, end-to-end encryption for all these collaborative apps, since no server would get in the way. There were technical limits to CRDTs—and plenty of applications that the cloud would serve far better. But to him, the prototype felt like a revolution. There wasn’t a server between us. Yet it worked. Mostly. We were two peers communicating, as the first bricklayers of the internet intended.
Van Hardenberg’s vision was somewhat easier to see when we met again in St. Louis. The tech giants were slipping. Meta’s stock was at a seven-year low. Twitter was in the midst of a hostile Elon Musk takeover. Kleppmann was spending a few hours each week as a technical adviser to Bluesky, spawned by Twitter as a decentralized experiment and now suddenly thrust in the spotlight, poised to become its competitor. Its “federated” design promised to give people the option to leave servers and services that treated them poorly. Bluesky wasn’t using CRDTs, which would be much too slow for coordinating the feeds of millions of social media users, but the goal was similar: a better relationship with “someone else’s computer.” Computing alternatives were once again in vogue.
Among them, CRDTs. Strange Loop was teeming with local-first presentations—a surprise to Kleppmann and van Hardenberg, who had until recently kept track of every project through Google Alerts and word of mouth. CRDTs were turning up in the wider world too. Developers at The Washington Post had used them to build a tool for arranging articles on the homepage. People poking around in the code that runs Apple’s Notes app had noticed CRDTs. Jupyter Notebooks, a popular data science app, restored its collaboration tools using CRDTs after Google got rid of the cloud service it had previously depended on.
Among the presenters at Strange Loop was a Canadian developer named Brooklyn Zelenka, cofounder of a company called Fission. When she read the local-first manifesto, she recalls, “I was like, this is a great phrase. Before that, we had these awkward phrases, like ‘location independence’ or ‘user-owned data.’” Zelenka had been interested in the ideas of Web3—the moniker adopted by “decentralized” apps that use blockchain technology and cryptocurrency—but found its culture “aggressive,” which she attributed to the focus on money “so clearly, all the time.” It was nice to be getting into local-first early. “Everything is low-hanging fruit right now,” Zelenka told me.
Hers was a common trajectory. Crypto “brought out all the worst people,” van Hardenberg told me over lunch at the conference, but it also spoke to many of the same principles as local-first. In his opinion, it simply uses the wrong approach, promising users decentralization and independence but tethering them to speculative financial incentives. It’s also the opposite of offline-first: Cumbersome blockchains, controlled by whoever hoards the most resources, mediate every interaction. Still, crypto did offer a lesson in how hype can fuel the creation of new products. Van Hardenberg noted the large numbers of bored and disaffected programmers at the likes of Meta and Google who jumped ship at the height of the crypto bubble.
Local-first, he thought, could eventually elicit the same excitement, but with software that was actually good. What it needed, van Hardenberg said, was a big “exit” that would bring “signs of visible wealth” to some lucky group of local-first developers and help attract more talent and resources. Growth was also scary. Van Hardenberg and Kleppmann had so far eschewed venture capital funding for Automerge itself, fearing that it would force them into any variety of business models that “totally go against the values of local-first,” as Kleppmann told me. But at some point, they realized, growth would also be necessary. They hoped the software could stand for itself. “VCs love a replatforming,” van Hardenberg said.
A few months after the conference, “local first” was trending again on Hacker News. One commenter called CRDTs the “dragon slayer” sword that would allow local-first apps to compete with the cloud. Another lamented that every interesting technical post about CRDTs devolved into a “weird political discussion about decentralization.”
Despite many movements for tech decentralization, the dragon’s hoard of gold has kept growing. One problem is the perception that principles come at the expense of convenience. As fulfilling and virtuous as it might be to hew your own coffee table, it’s also hard. Eventually you get tired, and you buy your next piece of furniture off Amazon. So it goes for managing your data. “It’s so much easier to be lazy and let Apple or Google do it for you,” Shapiro told me. When I asked him what it was like to use the modern internet while adhering to his principles, he said he simply abstains from technology as much as he can. “It’s a terrible waste of your time,” he told me.
I was curious whether the term “local first” bothered Shapiro at all—whether he took it as an unwelcome rebrand of his technical creation. I was surprised when he told me loved it. There was magic to the phrase, he thought. Maybe the revolution has to be a little sneaky to land a blow: reel in developers with the technical possibilities, call it a “movement” to attract the politics-obsessed journalists (hello). Maybe it also needs to arrive at the right moment, when the Big Tech platforms seem ready to crumble, revealing the features lost and the abuses endured in exchange for convenience.
Kleppmann wasn’t demanding a return to analog or to smash all the cloud servers, which do many useful things. The sword was not a slayer but a tool to carve out something better—and even he would say that it still needs sharpening. When I asked him whether I could try out a new, CRDT-based text editor he had been working on, his usual expression of calm attentiveness transformed briefly into alarm. Of course, theoretically I could run the prototype that’s posted online, since it’s open source—“but please don’t do that,” he told me. He would let me know when local-first was ready.
Let us know what you think about this article. Submit a letter to the editor at mail@wired.com.
The Cloud Is a Prison. Can the Local-First Software Movement Set Us Free? - WIRED
Read More
No comments:
Post a Comment