tldr: Because AI generated slop artifacts are everywhere now, I encourage you to disclose how you utilized AI in your work – “AI Nutrition Facts”. This helps your colleagues understand how to review your work, all while helping you think more about where gaps in your judgement might lie¹.

AI artifacts are everywhere

Let’s embrace the fact that AI augmented work is here to stay. Let’s also acknowledge that much of our AI artifacts are, with the current state of the art, prone to hallucination and mistakes. As much as I want to shake my fist and lament the existence of AI slop in work artifacts, I think it’s all good-faith work done by colleagues doing their best with the tools and models they have. Heck, I look at my own output and sometimes am surprised at how much of it is subtly wrong, overtly wrong, or just plain cringeworthy².

Let’s also appreciate the fact that our colleagues on the other end – the ones reading and reviewing our code, memos and slide decks – are the ones who have the cognitive burden of reviewing this work. In this new AI generated content ecosystem, the verification and review of code is even more important and even more draining because as they read it, they need to know if the stuff they’re reading is unreviewed (or less-reviewed) slop or if it is fully represented by you.

Disclosures can help!

So here’s a simple suggestion: Add a footnote (or tag) to your docs or PRs disclosing how you used AI:

• “AI was used to proofread and suggest grammatical edits”
• “Claude/Gemini/ChatGPT was used to research and render this call sequence diagram”
• “I read coworker’s design doc and had Claude create this slide deck from it”
• AI_NUTRITION_FACTS=No AI was used in the creation of this PR
• “Claude/Gemini/Codex was used to brainstorm and add to the ideas proposed here”

With disclosure, this helps your coworkers contextualize your work and better understand which parts of the doc they can trust from you - and which ones they may need to scrutinize more. By providing them more information, you reduce their cognitive load when reading your work.

Secondly, this benefits you. By actually thinking (and writing out) how you came to your answer, code, or artifacts, you are able to better understand what parts of your work need to be reviewed before you throw it over to your colleagues.

You might be thinking “Geez Andrew this is going to make me look bad if I let on how much of my work is AI generated!” I think the opposite is true - disclosing AI usage is a gift of clarity you give your reviewers.

So drop a disclosure into your work. Your coworkers will thank you for it.

Someone wrote a doc and circulated it at work the other day - “Guidelines for vibe-coded docs”. My colleague got tired of reading super long design docs generated by LLMs. Too many docs at work are getting thrown around with reams of nonsensical text. “As an author, you must own your [AI-generated] work” my colleague wrote. It’s been getting out of control.

It got me thinking about how prevalant AI writing slop is these days. It’s especially egregious if you’ve ever surfed Reddit or X or.. God forbid, LinkedIn lately.

Lately I’ve been catching myself cringing when I read Claude-isms show up in my social feeds. “It’s not X, it’s Y.” Claude has a very specific voice and it’s all over the place. I absolutely freaking hate it. You see, to me, using AI signals to me that you’re not using your brain. You don’t stand behind your words. You’re just good at generating… words. But you don’t even know what it means, and I don’t want to engage with it.

I was asking myself why my experience of this was so strong. I think it’s the same phenomenon as the Uncanny Valley - the closer a mechanically-generated thing gets to appearing human, the more it elicits feelings of revulsion or disgust. The original scholarship suggests it may be an artifact of evolutionary biology, or of deep religious or cultural cues.

But is it that simple? After doing a bit of soul searching, I’ve come to realize reading AI slop makes it easy to discount the author and disconnect from their thesis. Reading AI content feels… generic. AI content is too wordy. Or too concise. Too flowery. Doesn’t make sense. I can’t hear the author’s voice behind it. I’ve already disconnected with your content the moment I caught whiff of the “It’s not X, it’s Y” LLM author-isms.

But if I went even further down the path of introspection… reading AI slop brings up emotions of fear in me. Fear that my genuine strength – writing! – is becoming commoditized. And where will that leave me? I was always a bookish kid, good with words, involved with the creating writing mag at school, deeply into the blogging scene when that was a thing. We craved authenticity, good writing, praised the ability of the internet to create real, authentic connection through digital media - most of all, through words.

It feels like all of this is threatened by the massive firehose of technologically-empowered slop that threatens to bury my small little skills with sheer volume. So deep down at the heart of it I’m deeply fearful - of irrelevance and of replacement.

I’ve been guilty of it as well - using AI to generate research docs or execute code snippets and only give it a passing glance of a review, declaring it “good enough”, then shipping it out for the world to see (or for my poor colleagues to review). I know it’s not as simple as being lazy, but there’s immense pressure right now to be “AI-first”, to fill up your hands with so many parallel things it’s impossible to review every word. So with that, your work and your craft becomes diluted because there is just so much less attention to go around.

But here’s a ray of hope.

Tomasz Tunguz made a quaint observation recently in his post “Observations on Writing with AI”. “What’s authentic?” he asks. “Imperfection.” AI editors and authors alike all sound the same. Your voice - with its imperfections - make your work uniquely stand out. We can lean into that.

Elsewhere on the internet, I came across Emily Segal’s Substack article on “tasteslop” where she argues that LLMs and the technology operators that own them, are attempting to capture the optimized taste of society. They do this through expert human raters, fine tuned reinforcement learning and real-world internet signals trained on billions of people. But what true taste is is socially constructed and always constructed in opposition to the dominant framework of the time. So by definition - the moment it gets captured by the machine, it ceases to be tasteful.

The evolving preference of human beings in the here and now, working together in sociotechnical systems - is still impossible to capture and distill (though I’m sure some PhD researcher somewhere is working on this as we speak). So here’s to keeping your own identity in the age of AI. Writing things (or typing things) out by hand. Viva la analog, and all things imperfect. It may feel like AI is capturing the world - and accelerating tons of good things - but it cannot come for the soul of the craft. That’s for you and me to own, and keep our voices authentically human in these weird, weird times.

There’s much fear and confusion these days in AI. We’ve got developers harnessing multi-agent swarms, with collective societies of agents running out and refining and defining their work. Agents are now running ML experiments. Articles speculate about how AI agents going to start improving themselves. It’s hard to imagine a world where there isn’t going to be a robot uprising. All this can present to us a special kind of of anxiety and fear about some new world order where we’re rendered obsolete.

When I read the “AI as Normal Technology” paper by Arvind Narayanan and Sayash Kapoor, researchers at Princeton, it all made sense. I’m going to do a quick recap of what their paper says, and reassure you that it’ll be okay. Really.

Tech’s adoption and impact is slowed by societal “speed bumps”. This is no different for AI.

The reason I’m so confident about saying this is that technology doesn’t develop in a vacuum. The dissemination, usage, application and improvement of technology happens in the context of socio-technical systems, whose processes are governed by human systems and have their own “speed limits”. Essentially, the authors argue that the “fast” view of AGI (superintelligent species or rapid human extinction) was unlikely to happen because of the socio-technical limits of AGI. It’s not like an AGI would just appear and humans would become fully AI-pilled and surrender completely to it. The authors note:

…the speed of diffusion is inherently limited by the speed at which not only individuals, but also organizations and institutions, can adapt to technology. This is a trend that we have also seen for past general-purpose technologies: Diffusion occurs over decades, not years.

The authors argue that a modern-day analogy would be more like electrification. Citing analysis from Paul A. David (“The Dynamo and the Computer, A Historical Perspective on the Modern Productivity Paradox”), who argued that despite electricity being invented and even with infrastructure built, it took decades for electrification to become utilized in such a way to dramatically affect productivity:

What eventually allowed gains to be realized was redesigning the entire layout of factories around the logic of production lines. In addition to changes to factory architecture, diffusion also required changes to workplace organization and process control, which could only be developed through experimentation across industries. Workers had more autonomy and flexibility as a result of the changes, which also necessitated different hiring and training practices.

Thus we see that there are practical limits to the adoption and transformative effects of new technology, and adoption – society-altering affects on a grand scale – will be slow.

Humans remain in control of AI systems and are unlikely to delegate control.

But isn’t there a chance that this time it’s different? That AI superintelligence could break out of a lab like a computer virus and start to wreak havoc on society? After all, aren’t there researchers broadly and deeply concerned about misalignment risks?

There was that old AI tale, about an AI that was instructed to make as many paperclips as possible. In the misaligned scenario, the AI takes over the world by creating paperclip factories and deeming humans a risk to itself, destroying humanity.

The authors’ response:

Misalignment concerns often presume that AI systems will operate autonomously, making high-stakes decisions without human oversight. But as we argued in Part II, human control will remain central to AI deployment. Existing institutional controls around consequential decisions—from financial controls to safety regulations—create multiple layers of protection against catastrophic misalignment.

Humans are the ones building or designing products; humans are the ones who decide how they’re deployed, what safeguards to deploy around them, how much or how little responsibility they should take on, how their outputs are parsed and used, and whether or not they should be automated and deployed into the wild. We are very unlikely to ever see the total delegation of decision making to an AI system.

AI is in service of a human; AI is in service to humanity. Humans use AI as tools, and if there’s anything to be reassured of in this day and age, it’s that human creativity and oversight will durably remain over the machines.

This is actually quite an insightful and perhaps insidious insight. The true risks and choke points lie in the providers and organizations who run the services, develop models, and license their technology for whatever technology. What are their values, what are their motivations? These are more important than worrying about what the machines are doing.

Disruption in the software industry

Circling back to the first point - if you’ve been watching the layoffs at Block and Oracle and (likely) Meta, you’re probably still just as discomfited as you were before reading this. After all, it doesn’t matter if AGI or superintelligence is here (or not). The AI hype / bubble is disrupting our industry, especially in software.

I know I started this article telling you that “it’s going to be okay”. The general anxiety around how AI will disrupt our industry is very real, and it’s certainly true for industries where the work is rote and easily automated.

The best thing to do is to understand the technology. Build product taste and exercise technical judgement. Be shrewd enough to learn from the new technology - how to harness it and understand how it works. But don’t delegate your thinking or your skills away.

Can I guarantee that mass layoffs aren’t around the corner? No, I can’t predict that. The sheer variability and noise in the modern world today is incredible. But the best thing we can do is lean in, focus on what we can control, and bet on ourselves. We humans are going to figure it out.

Let’s set aside the debate about AI productivity, its implications for the economy, the tech job market, and the existential questions about whether we’re in a bubble or not and the trillions of dollars in spending going into the ecosystem. Never mind the fact I just posted an AI-skeptical screed on the importance of artisan hand-crafting software with your old fashioned wetware. Ignore that a bit with me for now.

In my experience, there’s no doubt about it: AI-assisted coding is just plain fun.

It’s fun to blast through your backlog and move at the speed of thought. It’s plain fun to watch your little hordes of AI minions do their research and come back with all these great ideas. It’s fun to feel like things are moving where they’d been stuck before, maybe even for years.

There’s the sheer illusion of progress or speed, which reminds me a little bit of that typical scenario from the pre-AI era when starting up a software project. You’re maybe 20% of the way in, and you’re flying. Commits are flying off your fingertips. The team is sprinting, running toward that MVP. Your proof of concept materializes quickly. Maybe you’re in a greenfield project, and your test suite is blazing fast. Holy crap, you’re knocking it out of the park. Features are coming together and you just have this sense that the wind is at your back and the software gods are smiling down on you. ¹

AI coding feels like this all the freakin time. Your ideas? Launched out of an idea cannon immediately, fully formed. You’ve got three or four or five Claude Code windows open at a time, all working on something different. Your dopamine receptors are tingling with excitement as you guide each micro-team of agents toward their goal, each checking back with you every ten or fifteen seconds. Is this what you’re looking for? What else would you like me to do?. There’s really nothing like it, which is how I imagine Neo felt when plugging into the Matrix:

It’s pure dopamine. Sugar straight into the bloodstream. Man, it feels good.

I’d been setting up a Tailscale session to be able to SSH into my phone and build features on Wejoinin. Let me tell you, it’s probably just as addictive as social media, only arguably slightly more constructive. This time, whenever I want, I can tap straight into the thoughts of the computer and just direct these hordes of agents and send them off to build whatever harebrained feature I want to do next.

Regardless of whether AI coding is actually 10x more productive, or whether it will have macro-state impact on the economy or the entire knowledge economy, it can’t be denied that it’s just fun. And that alone makes it worth continuing to examine, to play with at the edges of your time, no matter if you’re technical or not. That kind of upskilling isn’t just boring and technical, like studying for an exam. It’s a positive reinforcement loop, all the worth diving into and getting your feet wet. You’ll soon find yourself going into the deep end.

Steve Jobs was remarking on the fact that computers would augment human abilities and allow humans to accomplish far more than they ever would have without. Now, with the dawn of the AI age, it would appear like we are about to make another leap in human capabilities. Or are we?

This last year I really jumped deep into the use of generative language and coding models in my work and personal life. Have I been seeing the payoff commesurate to the hype? Both yes and no - but this post will first start off with the pessimistic case.

AI generates hollow artifacts that can deceive

AI at this stage is impressively good at generating code. It’s impressively good at generating code that looks plausible and passable for real, thoughtful work. But looking like the real thing does not mean it is of the same caliber as a human. Why?

I’ve been trying to get to the bottom of this feeling; the feeling that after working with an AI system for any period of time that the work being generated is not intentional. Don’t get me wrong, the quality of the artifacts being generated - functional code, tests, and documentation - is fairly high quality. But it’s deceiving - peer at the code, and the system is doing things that look correct but fail in non-obvious ways. The code looks correct. It has all the right documentation. The test suite seems complete. But peer closer:

• New patterns and data structures are invented and injected when compliance with existing patterns would be preferred.
• The test suite blindly writes test cases that either don’t need to be written or are overly verbose.
• Blind adherence to application or codebase prompts (“ALWAYS follow TDD” or “You must always write a feature spec” leads to code bloat and reduced long term maintainability)

AI’s strengths (broad) and weaknesses (shallow) create distinctly unique artifacts that are unlike anything we’ve seen.

Now you might say, “hey, this is just the thing where an AI is like that overeager junior developer.” But it’s a wildly different beast. A junior developer usually has a clear limit of where she or he can operate. Maybe are able to implement some key feature within a system boundary, but a single misunderstanding of fundamental system assumption leads their solution to be nonoptimal or incorrect. That’s really easy to code review, or sit down and correct.

But an AI, they’re that intern but with wildly overconfident estimation of their abilities and a truly dizzying breadth of knowledge. They literally have the entire corpus of the Internet downloaded into the foundation model, with all of the patterns and knowledge. This makes correction and code review exhausting:

• AI models often overcorrect or overimplement in subtle ways.
• Sycophancy problems mean that AI models have difficulty pushing back meaningfully.
• With their vast knowledge, AI output often incorporates new or external concepts that do not quite fit the codebase. Depending on your intent, this could be a feature (hey, this new approach works better) or a bug (hey, this new approach is totally overbuilt)

AIs need consistent humans in the loop to correct or justify their approach. For humans, this is exhausting.

When Claude Code started to ask you “hey, is this what you mean?” and ask you questions to get you to clarify your intent, this was pretty amazing. It meant that the context that lived in my head, the implicit understanding that I would have never dumped out into the open was being pulled from me.

But on the other hand, it’s really exhausting and explains why long term using AI has not yet translated into wild productivity gains for myself. Having an AI continually prompt you for clarification on every micro-decision used to be something a programmer just did implicitly in their head; one of a million micro-decisions that formed the direction, vision and principles of a codebase. Having that pulled out into the open is not just wasting tokens, but wasting time.

Human writing (and human coding) is thinking, which is valuable in and of itself

My final and most important point; by delegating this work to AI, we lose out on the reward of reasoning and learning through thinking and work. I feel this is wildly overlooked by the AI conversation today, with most commentary along the lines of “work is now all about having product taste” or “programming is now becoming a higher level activity” as if humans just needed another layer of abstraction.

But the dark side of abstraction is ignorance, either knowingly or inadvertently. The foreman of a group of workers is no longer intimately connected to the craft. The manager of a team of coders is, purely by their (lack) of exposure to the direct work, less and less connected to the system. Our inability to interrogate an AI to understand its reasoning and the prolonged exposure to this new level of indirection is whittling away our ability to reason about and assure ourselves of the system’s integrity.

I’ve been working on a large project in a personal codebase of mine. Day by day I’ve been using AI to build things, carefully crafting CLAUDE.md instructions, reading best practices for context engineering, and dutifully doing my part to keep the intention of software clear, precise and defined in text. I’d be lying if I said it wasn’t satisfying; it’s so cool to see commit after commit land, sprinting from one checkpoint to another. I’m burning down the backlog at a rapid rate. I’ve started coding on my phone, managing my AI agent directly while running errands at the store. So it’s undoubtedly a satisfying experience for me.

On the other hand, it’s been taxing to manage this agent every few minutes. To directly manage its direction, provide additional context, and verify behavior. I’ve caught it duplicating code that existed already, or writing test cases that looked correct, but did not actually test anything of value. Repeat this hundreds of times over and it begins to dawn on you: I could have done this faster myself.

I was with a group of friends recently discussing our work with AI and one had a very astute observation: “I think working with AI is just less satisfying”. What he meant is that the distance from the work, and the work required to harness and manage an AI coding assistant was far more circuitous, far more taxing than just jumping in and doing the work himself.

I would agree. With increased use of these models, I feel the distance from myself to the produced artifact become dull. As a software craftsperson (professionally, I came of age in the XP and Software Craftsmanship movement), this pains me greatly. I am less connected, more cognitively taxed, and less satisfied with the work that is being done. As a writer, I relish the process of writing, refining an idea and polishing the gem. I love that the process of getting something down on paper (or a screen) is a dialog to refine the idea in your head, and back and forth it goes. That is the inherently creative part of the process I fear that I am losing.

A proposal for the future

I’d be lying if we didn’t acknowledge the times that these models have landed a particularly complicated refactor. Or the magical moment when an AI agent anticipated what needed to be done and did it directly in the background. It’s clear that the advances in technology are not going to stop. The annoyances I’ve listed here might not last another year at the rate of progress we’re seeing in the industry.

On the other hand, let’s not offload our thinking entirely, in order to stay knowledgeable and reputable. How might we…

• Think of less cumbersome ways to interact with LLMs than textually?
• Challenge ourselves to work unassisted for some time to better understand the system?
• Consider the use of these models in an assistive mode, rather than as a be-all owner?
• Keep the human in the loop, but make it an enjoyable loop?

I think tools are going to be the way forward. More on that in a bit.

An aside: I found myself recently on a flight back home with time to spare. I had intended to work with AI assistance (Claude Code) but my flight had wifi issues and I was offline for much of it. The two hours I spent digging in unassisted in the codebase gave me more insight and intuition than two months of work did.

Moving from low-coordination to high-coordination contexts

At Lyft and in roles prior, I worked on teams that were highly independent by design! By virtue of these companies having startup DNA and by virtue of existing during the zero-interest-rate 2010’s phenomenon, we had mandates to move fast and to be decoupled from other teams.

At Lyft I was in the Growth organization which was peripheral to the core Ride Booking experience. I was tasked with building user acquisition funnels and retention flows, which funneled into the core product experience. But because there were very clean lines in the user journey where we could draw a boundary on, the organizations had very clean lines of separation from each other. This gave my team a very long leash to execute our big ideas on. It was taken for gospel that we needed to ship fast and iterate quickly - my director told us his goal was to have a team ship a new experiment every week.

Now switch contexts to Google (YouTube), where I work now work on a core YouTube product surface - one that is embedded deeply in the product experience and with countless dependencies on other teams. For one my teams, I counted no less than 12 product- and infra-team dependencies. Want to ship a feature of any significant product impact? That’s how many teams need to convince, align and get approvals from.

As a result, you get heavy process- and planning-driven cultures, with technical program managers running the show (by the way, bless my TPgMs, they are a godsend and the only way anything gets done).

Moving from microservice to monolithic architectures

Following Conway’s law, it’s little surprise that our architectures reflected our org structures. Lyft was on the bleeding edge of the microservices wave in the 2010s (our frontend engineering guild of ~100 engineers counted nearly as many frontend microservices total - that’s nearly one service per engineer!). Want to get a change in? No problem - service boundaries cleanly match team boundaries, APIs are exposed between other teams and you can just basically write your code and ship it in 15 minutes.

Google, on the other hand, is famed for having a monolithic architecture (which is, as it is, experiencing a resurgence lately). This means that engineers are often queued up waiting for code reviews on other teams that are dependencies or owners of upstream services that interface with your feature. This is by design, though - the bar is set very high for any changes entering the codebase, and having a monolith gives engineers visiblity and accountability to maintaining its high bar.

Note that I’m not suggesting that a monolith necessarily means more perverse coupling or more technical debt - in fact, the Google way of doing things is incredibly elegant, and there is a lot of proper thought given to encapsulation of concerns.

But taken to the extreme, this can often mean playing review tag for days at a time with a team that’s on the opposite side of the globe, leading to very real implications for getting code shipped.

Product maturity and scale dramatically dictate how you approach product development

At Lyft, we were the #2 player in the market and always playing catch-up. We were always worried about losing users and market share to the competition, and this do-or-die mindset was always at the forefront of our decisionmaking. Thus, we were hyper-focused on shipping new features (even if the bets we were taking were a little risky). Reviews were fast and decisions were made quickly because the threat was existential. The question here is: are we shipping fast enough?

On the other hand, at the scale of YouTube as the 900-lb gorilla in the space, the business is designed to defend itself. There is so much more to lose than to gain from shipping new features. Want to do something radical? Product owners spend months debating, aligning, drafting annual roadmaps, then shipping them around to get reviewed and aligned with other teams to ensure that the feature you want to build is actually the right thing to build. Vision docs are written, alignment sessions are brought in, OKRs are hotly debated for quarters on end. The question here is: are we shipping the right thing?

The curse of data-driven decision making

At both of these companies you’ll find experimentation-driven cultures and advanced experimentation tooling, but Big Tech takes it to a new level. At Lyft, we definitely had a few north star metrics that we aimed to move (or used as guardrails) - but we really only tracked a handful, tops.

At YouTube, there are literally thousands of metrics that can move with a change, and you had better have a good idea why they move. Any change in metrics is inspected with a microscope, any deviations can send engineers and data scientists off on data expeditions to understand what happened. It could be real, or it could just be statistical noise. And God forbid this is a metric that you have never heard of - better go find the right person to ask and figure it out.

This leads to what we’ve internally called “metrics hell”. When a metric goes awry, how do you know:

• What actually is happening? What are our theories about why this is? Does it logically make sense to the change we are testing?
• Is it statistical noise - and can be ignored?
• What other kinds of analysis can we do to understand this outside our experimentation system? Manual testing? Deep-dive into log data?
• If it’s a real metrics drop, how much can we tolerate? When can we get it back?

Teams can spend quarters working on a big project only to get it up to an experiment then get stuck in Metrics Hell for months on end. It is very rare to see a project get the go-ahead if a core data concern is not resolved and investigated. This means we need to deeply instrument the system with the right data inspection tools, and train our engineers to query and analyze it.

Moving from zero-interest-rates to zero-headcount scenarios

Finally, this is maybe the most important point.

When I was at Lyft from 2019-2022, we were in a big wave of IPOs. Crypto frenzy was everywhere. Tech, coming out of the pandemic was white-hot, and 2022 felt like the year that workers held all the chips in their hands. I had never felt so much confidence in my prospects in the market.

It would soon all change as 2022 came to an end. Only a half year after my departure, I received news that my prior team at Lyft had seen heavy layoffs and more or less been disbanded. One by one, companies started to fold and layoffs started to materialize in places I had considered invulnerable to layoffs. The unspoken feeling was that we were all vulnerable, and that no job was safe.

Where in my last role there was a feeling of freedom, of experimentation and trust and play, the dominant experience of the next role was… stress. I had never felt so much pressure to onboard quicker, learn faster, work longer hours, and drive impact. It exhausted me, and work was no longer enjoyable.

So where does that leave me now? Well, I’m hoping to explore some of these learnings and themes in my next few posts over the next year. Some of the musings will be very practical about navigating a Big Tech job at Staff+. Other will be pretty personal about managing anxiety and stress. My goal is the write the Survival Guide I wish I had when I started out this new role - and hopefully it will be useful to you too.

In Part 1 we discussed the use cases for formal specifications and we looked at a simple transaction isolation bug in a financial institution.

Introduction

This is a distributed system transaction orchestration problem.

In this exercise, imagine we are a bank, and we are serving an API request from our banking mobile app to initiate a bank transfer from an external financial institution to the user’s account.

We are building a API endpoint that asks the other financial institution to move money over to us.

The wrinkle here is that internally, the transfer also needs to be synchronized with a third, internal transaction database (our internal source of truth) to formally recognize the balance in the user’s account.

How do we design this system to ensure the design is resilient to failures and outages?

Requirements:

• We must guarantee that balances are kept consistent between our system and the external institutions. No money should be lost/created on either side (obviously).
• The external service may fail to process the transaction for any reason (downtime, network partition, system error)
• The internal transaction service may also fail to process the transaction for any reason (antifraud rules, domain logic, ratelimiting, network partitions)
• The API request is synchronous, and must respond within 200ms @ p99

The Happy Path

We illustrate the system design using the happy path. Our mobile client calls an API gateway, which we use as a transaction coordinator.

The API gateway makes 2 calls. First, it calls the external financial institution to initiate the transfer. If the transfer is successful, the API then turns around and pings the internal balance store service to note the transaction was a success.

(Note that this is not representative of a real world bank! This is a contrived, simplified example).

Because we need our API to be synchronous, the API coordinator blocks until both responses are success, before returning a success response to the client.

sequenceDiagram
autonumber
OurMobileClient->>+OurAPIGateway: SubmitTransfer
OurAPIGateway->>+ExternalFinancialInstitution: StartTransfer
ExternalFinancialInstitution->>-OurAPIGateway: SUCCESS
OurAPIGateway->>+OurInternalBalanceStore: UpdateUserBalance
OurInternalBalanceStore->>-OurAPIGateway: SUCCESS
OurAPIGateway->>-OurMobileClient: SUCCESS

Internal API error with compensating transaction

Of course, we know that errors can crop up in the real world. If the call to either service borks, we will need a way to either retry or fail gracefully. Here, we consider the use case where our internal API service crashes.

We consult with the team and decide that if the API service crashes for any reason, we will want to undo the transaction in the external financial institution with a compensating transaction. We will throw this work onto an external queue as soon as an error occurs.

sequenceDiagram
autonumber
OurMobileClient->>+OurAPIGateway: SubmitTransfer
OurAPIGateway->>+ExternalFinancialInstitution: StartTransfer
ExternalFinancialInstitution->>-OurAPIGateway: SUCCESS
OurAPIGateway->>+OurInternalBalanceStore: UpdateUserBalance
OurInternalBalanceStore->>-OurAPIGateway: FAILED
OurAPIGateway--)BackgroundWorker: Enqueue Reversal
Note over OurAPIGateway,BackgroundWorker: Compensating transaction kicks off after a failure
OurAPIGateway->>-OurMobileClient: FAILED
BackgroundWorker->>+ExternalFinancialInstitution: UndoStartTransfer
ExternalFinancialInstitution->>-BackgroundWorker: SUCCESS

But wait! We see that there’s a bug. For there’s a race condition when users “button mash” after they hit an error dialogue in the mobile client and immediately retry their request again!

sequenceDiagram
autonumber
OurMobileClient->>+OurAPIGateway: SubmitTransfer
OurAPIGateway->>+ExternalFinancialInstitution: StartTransfer
ExternalFinancialInstitution->>-OurAPIGateway: SUCCESS
OurAPIGateway->>+OurInternalBalanceStore: UpdateUserBalance
OurInternalBalanceStore->>-OurAPIGateway: FAILED
OurAPIGateway--)BackgroundWorker: Enqueue Reversal
OurAPIGateway->>-OurMobileClient: FAILED
OurMobileClient->>+OurAPIGateway: SubmitTransfer
OurAPIGateway->>+ExternalFinancialInstitution: StartTransfer
ExternalFinancialInstitution->>-OurAPIGateway: FAILED
OurAPIGateway->>-OurMobileClient: FAILED
Note over OurMobileClient,OurAPIGateway: User's  re-submittal fails because there are no funds in account (race condition)
BackgroundWorker->>+ExternalFinancialInstitution: UndoStartTransfer
ExternalFinancialInstitution->>-BackgroundWorker: SUCCESS

This will fail.

Enter formal specifications!

OK, let’s try to model this behavior as a formal TLA+ spec. I’ll write out how the spec would look, and we’ll go through it line by line:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
variables
    queue = <<>>,
    reversal_in_progress = FALSE,
    transfer_amount = 5,
    button_mash_attempts = 0,
    external_balance = 10,
    internal_balance = 0;

define
    NeverOverdraft == external_balance >= 0
    EventuallyConsistentTransfer == <>[](external_balance + internal_balance = 10)
end define;

\* This models the API endpoint coordinator
fair process BankTransferAction = "BankTransferAction"
begin
    ExternalTransfer:
        external_balance := external_balance - transfer_amount;
    InternalTransfer:
        either
          internal_balance := internal_balance + transfer_amount;
        or
          \* Internal system error!
          \* Enqueue the compensating reversal transaction.
          queue := Append(queue, transfer_amount);
          reversal_in_progress := TRUE;

          \* The user is impatient! Their transfer must go through. They button mash (up to 3 times)..
          UserButtonMash: 
            if (button_mash_attempts < 3) then
                button_mash_attempts := button_mash_attempts + 1;

                \* Start from the top and do the external transfer
                goto ExternalTransfer;
            end if;
        end either;
end process;

\* This models an async task runner that will run a
\* a reversal compensating transaction. It uses
\* a queue to process work.
fair process ReversalWorker = "ReversalWorker"
variable balance_to_restore = 0;
begin
    DoReversal:
      while TRUE do
         await queue /= <<>>;
         balance_to_restore := Head(queue);
         queue := Tail(queue);
         external_balance := external_balance + balance_to_restore;
         reversal_in_progress := FALSE;
      end while;

end process;

Whew, ok! That’s a lot. Let’s go through it line by line:

First up, we declare variables and operators:

1
2
3
4
5
6
7
8
9
10
11
12
\* These are global variables
variables
    queue = <<>>,
    transfer_amount = 5,
    button_mash_attempts = 0,
    external_balance = 10,
    internal_balance = 0;

define
    NeverOverdraft == external_balance >= 0
    EventuallyConsistentTransfer == <>[](external_balance + internal_balance = 10)
end define;

There are two main blocks here, the variables block and the define block. The variables defined here track values that will be used globally throughout the model. The operators in the define block are properties that the model checker will use to make sure invariants and temporal properties hold true throughout the lifecycle of the model.

It’s imporant to note the properties defined here in the spec:

• NoOverdrafts is checked on every state combination, ensuring that there cannot be a scenario where the external financial institution is asked to transfer more money than is in its account.
• EventuallyConsistentTransfer is a Temporal Property that checks whether the system always eventually converges on the condition listed below - that external + internal balance equals $10, the starting amount. We are essentially guaranteeing that we cannot unintentially create or lose any money between our institutions.

Next up, there are two process blocks being defined here, representing the two internal systems whose interactions we are modeling here.

The first process is the API coordinator. Inside this coordinator, each action is marked by a label - so note the labels ExternalTransfer, InternalTransfer, and UserButtonMash. These correspond with various phases of our system sequence diagram. Let’s walk through the code:

1
2
3
4
5
fair process BankTransferAction = "BankTransferAction"
begin
    ExternalTransfer:
        external_balance := external_balance - transfer_amount;
        ...

This is fairly self-explanatory - the system is set up to first call the external institution and tell them to withdraw the money. For simplicity’s sake, we assume it always is successful. (It obviously won’t be, and we have the perfect tool to model failure scenarios around that!)

1
2
3
4
5
6
7
8
9
10
11
 InternalTransfer:
        either
          internal_balance := internal_balance + transfer_amount;
        or
          \* Internal system error!
          \* The system will enqueue the compensating reversal transaction.
          queue := Append(queue, transfer_amount);
          reversal_in_progress := TRUE;
          ...
        end either;

The next label is interesting. We use an either...or control structure to tell the model checker that there is possibly branching logic here (in this case, there is a success case and a failure case). Both these branches will be exhaustively explored.

In the successful case, we observe that the internal API is called successfully and the balance is correctly stored. However, the failure case will have us enqueue a compensating transaction (a “reversal”) that will be processed by an asynchronous worker.

1
2
3
4
5
6
7
8
9
10
11
          \* The user is impatient! Their transfer must go through.
          \* They button mash (up to 3 times)....
          UserButtonMash: 
\*            await reversal_in_progress = FALSE;         
            if (button_mash_attempts < 3) then
                \* But the UI blocks them from re-submitting until the transaction
                \* has finished being reversed/compensated.
                button_mash_attempts := button_mash_attempts + 1;
     
                goto ExternalTransfer;
            end if;

Ooh, the user, the user. You can always count on the user to do something unexpected. So now while the user is enqueuing the compensating transaction, our poor user is confused and is now retrying the original transaction (aka “button mashing”) the UI button in hopes that it will go through. Will it succeed?

Note that the way I’ve built the spec, I’m specifying a finite limit to the number of user retries, if only to make sure the program will eventually terminate.

Finally, observe the goto ExternalTransfer statement on Line 10. This basically tells the model checker to jump to the ExternalTransfer: label - i.e. the top of the program to re-execute the process all over again.

(Author’s note: I haven’t finished this yet, but thought I’d push this up as a work in progress. Do you see the error? Are your spidey senses tingling here? More to come!)

If you’ve been writing software for any amount of time, you may be familiar with the many tools we have available to us to ensure correctness, consistency and debuggability of our systems. They range the gamut of unit / acceptance / integration tests, QA plans, CI/CD automation and the like. System or language tools like type systems, interactive debuggers and profilers abound. Practices emerge like DevOps process, TDD/BDD, and even Agile process itself can be argued to be invented toward the goal of writing correct, robust, easy to maintain systems.

Surely these tools are advanced enough in the 70-plus years of computing to help! But no - with the rise of distributed computing, the classes of bugs that start to emerge start to get ornery and complex, are usually nondeterministic, and often beyond the reach of ordinary tools.

But what if I told you there was another option from the world of… math?

Enter formal verification

What if there was a way to guarantee that our systems and algorithms are performant, run correctly, are reliable against race conditions and the like?

Here’s how it works:

1. You describe your system (or program) in terms of formal logic statements. You assert specific conditions that must hold throughout the program runtime (invariants). You write this in the form of a proof (that lives outside your actual program).

2. The tool has a “model checker” which is a glorified BFS search algorithm that explores every possible state space of your program proof and lets you know if the invariant conditions hold.

3. If they do - congratulations! You’ve verified your system. If it doesn’t pass - congratulations! You’ve found a potential bug!

4. Using the results from the model checker, you can fix the proof to fix the model checking error. This will translate into a real world fix that you can then roll back into your program.

It’s not magical. It’s also a lot of work, and in all fairness, slightly out of the reach of the typical industry programmer. But it’s much more in reach than you think!

A simple example

I’ll be using a tool called TLA+, and writing a sample spec from a derivative syntax called PlusCal. I’ll walk us through a simple example that can be found on the Learn TLA web site.

In the book Designing Data-Intensive Applications, the Transactions chapter illustrates a scenario where read isolation is not correctly implemented in the database, leading to dirty reads - simultaneous queries may be able to read dirty data from complex multi-statement operations - leading to bad outcomes.

Let’s say we are a bank where a user attempts to transfer money between two accounts, and a separate query is being run by an auditor who wants to ensure that the bank software is working correctly and no funny money business is happening:

sequenceDiagram
autonumber
User->>Account1: Add $500
Auditor->>Account1: Query Balance
Auditor->>Account2: Query Balance
User->>Account2: Subtract $500

Alas, our system was implemented a bit naively, and we can see that the application makes two calls to the database, debiting from Account1 and crediting to Account2 in two separate statements.

Assuming both accounts have initial values of $1000, the User’s transfer completes successfully, transferring $500 from Account1 to Account2, maintaining the correct money flow (Account1 + Account2 = $2000).

However, the Auditor has had the unfortunate timing to look at the state of the world in between the two user operations and has a different view of the world, seeing that $500 has materialized out of thin air into Account 1 (Account1 + Account2 = $2500)!

Let’s model this behavior as a TLA PlusCal algorithm:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
variables
  transfer_amount = 500,
  account1 = 1000,
  account2 = 1000;

process User = "user"
begin
  StartUserTransfer:
    account1 := account1 + transfer_amount;
  FinalizeUserTransfer:
    account2 := account2 - transfer_amount;
end process;

process Auditor = "auditor"
begin
  DoAudit:
    assert account1 + account2 = 2000
end process;

High level explanation - the two process blocks model two independent activities happening here - the user initiating the transfer and the auditor running the query.

This will blow up! The TLA model checker will compute all possible computation states between the two processes as delineated by the statements inside the StartUserTransfer, FinalizeUserTransfer, and DoAudit labeled statement groups, including when the auditor runs before, during, and after the user’s inter-account transfer.

Just look - the model checker has run and blown up on a series of state transitions that got us to the Very Wrong situation we discussed before. The poor auditor has found that Account1 has $1500 and Account2 has $1000. That’s not good!

How do we fix this?

Clearly, this is incorrect. We will need to ensure that the database is not able to allow other queries to read values happening from within a transaction. So here, we say “OK, we’re going to wrap up these statements in a TRANSACTION block”. But hold up! We need to move that change in system design into our TLA model.

Recall that the TLA model checker can only test state combinations between each labeled state, meaning that statements grouped inside a label are considered atomic operations. Knowing this, we move both transfers to within the same label to tell the model checker “these two operations happen at the same time, as if they were running in a transaction”.

1
2
3
4
5
begin
  DoUserTransfer:
    account_1 := account_1 + transfer_amount
    \* Collapse this transaction with the one above to make them atomic
    account_2 := account_2 - transfer_amount

Run the model checker again - it passes.

More resources

This was a fairly high level overview on how to write TLA specs. This is much better explained on the Learn TLA site: please read more there!

For the sake of time, I will direct you to some great resources:

• Learn TLA - A beginner-friendly resource from author Hillel Wayne
• Practical TLA+ - Hillel Wayne’s more comprehensive resource for specification programming in TLA+
• The TLA+ Home Page - Leslie Lamport (Author of TLA+)’s resources for learning and running specs written in TLA+

Up next

I’d love to run through a real world example of using TLA+ to specify a distributed system, loosely based on a concurrency bug we saw recently at Lyft. Stay tuned!

When you think of a software engineering career path, you may default to the idea that you can climb the career ladder at various product companies and corporations, working directly with stakeholders and leadership to ship products to customers. The types of companies you might consider are early/mid/late-stage startups, established enterprises or Big Tech companies.

What you may not have considered, however, is how a stint in consulting can accelerate your learning curve and teach you lessons that can multiply your effectiveness across any engineering organization you join later in your career.

Now you may have a stereotype of a consultant - maybe of a management consultant that flies out to clients five days out of the week and works 80 hour weeks and lives out of suitcase and makes PowerPoint presentations all day. If life as a suit doesn’t seem appetizing, that’s okay. That’s not the consulting I’m talking about!

I spent four years working as a developer at a XP software consultancy shop and… really loved my time there.

What’s software consulting?

Software consulting is defined by working with a client on a short-to-medium-lived project that has a digital deliverable - a software platform, an updated platform capability, or an MVP to show to first customers.

Consulting may consist of process deliverables - I’ve been on many a project with the entrenched old guard in some industry realizing that the new upstarts are eating their lunch with software - and that they’d better get along with the innovation. That means teaching folks Agile process, or product management.

Now I can’t claim to know everything about consulting, as my experience is limited to one consultancy in my career. However, I can say that it was a huge springboard for my career because it increased my exposure to people and organizations. The following are some of my learnings and takeaways from my time:

1. The hardest problems are people problems

You can always make a tech problem work. It’s the people problems that are the hardest. From stubborn and resistant developers who need to be wooed to your side, or to surprise stakeholders surprising you right before ship date. The keys to project success are almost never at the execution layer.

As a consultant, you will learn to very quickly read the room and understand where the power structures are. There’s the account manager, who’s stuck their neck out to really get your team in the door. There’s the VP eng, who is somewhat skeptical of your team, but who needs to be shown results. This is no different from inside the walls of a product company, where teams need to know where and through whom the power flows - and properly seek to manage that relationship.

2. Relationships are the key (as is lunch)

As a consultant, you are an outsider and often met with skepticism if not outright hostility. Finding ways to build trust and rapport with your client partners (read: grumpy engineers or skeptical directors) are super important. To that end, it was important to show my face in the office(s) as much as possible to see. Making small talk was key, or grabbing lunch with the team.

As a consultant, you are always grabbing lunch with people. At a product company, you too will learn that it’s important to build bridges and relationships with the stakeholders and collaborators on your team and outside.

3. Even if you think you’re the smartest person in the room, be flexible and humble

Many of us are hired for our domain expertise or Thought Leadership(tm), which would seem to naturally imply that consultants have a lot of power or sway in what can or should be done. After all, they are expensive!

But wait! That also means consultants are often seen as a threat. After all, who has to maintain the codebase after these consultants build their thing into it? You can never roll in and assume that you have the permission of the entire team to build a new system/introduce a new process/launch a new product the way you think should be done.

Even though we held strongly to our product development principles, we would sometimes bend to the customers’ whims because we understood that not all the time, one-size-fits all. So if the client balks at writing tests a certain way, or if they really don’t want to name the class that name, or if they have really weird preferences around line breaks and indentation - we let it go.

4. Don’t chase the shiny (too hard)

In consulting, the pace of learning is exhilarating. One month you’re working on a kubernetes migration for a Fortune 500 company, the next month you’re dipping your toes in the latest React library, and the next you’re building an iOS app for a stealth startup (oh, and some coworker keeps talking about OCaml or Haskell or something). It’s easy to get caught up in the temptation to choose the latest shiny for everything you build.

My advice - Choose Boring Technology. More specifically, use the “innovation tokens” mentioned in Dan McKinley’s article and choose one (or maybe two) fun, new things to use. Don’t dump the innovation tax on your clients and customers. This is hard to choose into. At a product company, this will be important to learn as well as you learn to identify the tradeoffs of choosing The New and Shiny versus the stability of Boring Tech. At a product company, you are also responsible for long-term maintenance of your systems, so this lesson may emerge no matter what.

5. Use your energy thoughtfully - and rest

Billing hourly had the result of forcing me to think about where and what I allocated my hours to, every day. My consultancy had a rule to never bill more than 8 hours / day, and thankfully it was modeled from the top that we really would sign off at the end of the day¹. You’re forced to really work on the most important things - pushing your product counterpart to ruthlessly prioritize only the most important things. Then you sign off and you just don’t work at night. No emails.

Quite honestly, this is something I really find hard to do nowadays I’m at a product company. It’s not easy to put down the computer and not answer emails².

6. Pairs are a pleasure

Gosh, I loved pairing. I know that I’m a weird outlier. But I picked up so many technologies, architecture pointers, vim shortcuts, and other random things from all the pairs I had over the years. I was fortunate to really enjoy my coworkers, and by far, this was my biggest growth multiplier in my technical skills.

Now I’m back in Big Company life, I’m known as the engineer who keeps scheduling time to pair with teammates or folks on different teams. It’s the best way to learn a new domain or system, and to also build trust with the person you’re working with.

7. The importance of business development and sales

As a developer, I naturally shy away from the sales and business development process. As a principal engineer in the consultancy, I was often tasked to go on sales meetings with prospective clients to be the face of engineering and also to vet client systems. I gained newfound appreciation for our partners and business development staff and learned ways to properly put value on the process and art of software development. And even though lots of these BD trips ended up without an agreement, it gave me opportunities to meet staff at other companies and get a little window into how they worked.

Fin

So that’s my little spiel about how helpful my consulting background has been now that I’m at a larger company.

By the way - Gerald Weinberg said all this stuff better in Secrets of Consulting. It’s a great read - no matter if you’re in consulting or not!

I grew up on the stereotypical overachiever fast track. My dad was a Silicon Valley hardware engineer who got me into coding when I was in fifth grade, and my passions kept me scripting, coding and building web sites in middle school and high school. When I graduated from Berkeley with an EECS degree, it was pretty clear I was ready to dive headfirst into the industry.

I joined a fairly large company, filled with smart and friendly people. It was a pretty stable, comfortable place. My coworkers organized lots of social events and there was an obvious deep camaraderie between all.

I was the new grad hire on an older team¹. I had a great manager and teammates I could learn a ton from. Totally an ideal place to launch a career.

Except… a year and a half in, I quit. I decided I’d leave the industry for a bit. I was happy at work, but I wasn’t OK.

You see, I had just gone through my first big breakup, one that reverberated deeper than I realized. When that relationship ended, I went through a period of deep soul-searching and realized that I needed some time away.

Around that time, some friends let me know that they were entering a yearlong internship at our Oakland church community. I decided that I’d join them that year. And so it went - I moved out of my comfy Emeryville apartment and into the cramped quarters of our East Oakland community center. It was going to be the start of one of the most transformative experiences of my life.

Instead of daily standups behind big glass vistas of the San Francisco skyline, I woke up to daily meditation and time spent in the urban community garden. Where I took for granted the amenities and services of our big glass skyscraper, I was now the one vacuuming, scrubbing and cleaning the facilities². Instead of spending most of my day with high-earning tech workers, many days were spent chatting (and sometimes squabbling) with our unhoused friends who lived on the church steps.

I know it’s cliché, but having time to step out of the career hustle was so good for me. It was good for the young man that I was, who needed time to focus on himself and rebuild a grounded identity. It was good for me to spend among friends and trusted community. It was good for me to spend a season focusing my energies outward. It was good for my balance and sense of what was normal to see how folks way, way outside the tech bubble lived, especially in East Oakland as we served in the soup kitchen.

I think that if I had not taken that year off, I would have continued in the hustle - lost deep in the bubble that so many of us in tech ensconce ourselves with.

I fully understand that my time spent in East Oakland that year cannot fully be separated from conversations about gentrification and privilege. After all, I had the financial means to take a year off without worrying about debt. And a year later, I re-joined the industry, easily switching back into my privileged life in tech. To that end, the learning continues.

And yet, that year fundamentally transformed me - it gave me a perspective on life outside of the tech bubble. It gave me friendships that have lasted to this day and sweet memories (and uproarious stories) that will last a lifetime.

At 23 years old, I made a good decision to take a year off. I’d say it was worth it.