How I Accidentally Built a Runaway AI

A half‑finished coding agent taught me three lessons: tokens are expensive, feedback loops matter, and guardrails aren’t optional

At 11:30pm, my AI coding agent started spawning copies of itself. By the time I pulled the plug, I’d burned $10 in tokens watching it invent fake tools in another language. Building self-editing coding agents is surprisingly easy, but making them useful (and safe, and efficient) is another matter.

I spent the evening yesterday building an AI coding agent, which I am calling Charlie. I was inspired to take a crack at my own coding agent after I came across an article that promises:

It’s not that hard to build a fully functioning, code-editing agent.

It seems like it would be. When you look at an agent editing files, running commands, wriggling itself out of errors, retrying different strategies - it seems like there has to be a secret behind it.

There isn’t. It’s an LLM, a loop, and enough tokens. It’s what we’ve been saying on the podcast from the start.

I was able to get it to make changes to itself within maybe an hour of starting. Within an hour I had a self-editing system. The speed is impressive; the fragility as we will discuss, more so.

Building Charlie (The Easy Part)

Charlie is my AI coding agent. It works as a command line program, like Claude Code. It boots and gives you a prompt, and it takes it from there. Charlie is an ‘agent’ in that it from a user prompt, it will make a series of requests depending on what it thinks it needs.

AI coding tool named Charlie

It really is just an LLM + tools + tokens. It has access to three tools:

• read_file: Read the content of a file

• write_file: Overwrites a file with new content

• list_dir: List the contents of a given directory

Early results were promising. One of the first things I did was turn Charlie on itself, I asked it to build a README for the project - easy, self contained problem to solve. The README you see today is essentially the same thing, with a few minor tweaks.

Second task was to use Ctrl-C to clear the input, then handle a second Ctrl-C to quit. Again, a self-contained task that impacts how input is received. Charlie gave me a result, I prompted with a few tweaks, tested it, committed and ‘deployed’ (i.e. restarted the tool). As we will see, I ended up getting burnt by this part.

However, just like RAG, these early results were deceiving. As I gave Charlie more complex tasks to solve, the results got worse.

Where Charlie Breaks (The Hard Part)

More Tokens For The Same Work

Charlie is about a magnitude, if not two, more expensive than comparable agents. This is because my implementation of read_file is reading the entire file, then write_file is writing the entire file. On the face of it, this seems reasonable, but in actuality for each change that is made, an additional two copies of the file are included in the context.

The result was absurd: Charlie is only ~500 lines of code, but I saw requests blow up to 65k tokens. This is a significant problem as managing what is in the context of these tools is the most crucial part of LLM development. As the context of the model grows, quality and speed of output decreases.

Jason Liu wrote an excellent series on context engineering in the context of agents and RAG, arguing how important this is. Charlie fails to do any context management at all, and therefore is proving his points.

Unable to self-correct

Charlie was able to make plausible looking changes to the codebase, but made errors that Cursor, Cline et all wouldn’t. One common theme were minor syntax errors, or unnecessary imports (Go makes these errors instead of warnings). It botched a series of more complex, multi-step changes across the codebase, such as mixing APIs and introducing contradictions. These are the sorts of things you would expect out of gpt-3.5 back in 2023, not using gpt-5 in 2025.

This is because of two things:

1. Runaway context. Because every read_file and write_file pulls in the entire file, each attempt to fix errors bloats the context. Worse, the bigger the context, the more the model degrades. This is a vicious circle, each retry actually lowers the chance of a fix.

2. No feedback loops. Charlie couldn’t run code, lint, or parse error messages. Other coding agents use those tools to self-correct. Charlie was guessing blind, so syntax errors and junk imports piled up.

The result: Charlie is worse than just using VSCode. At least VSCode quietly cleans up half these mistakes for me. Charlie adds them back in.

Subagents and Escape Velocity

To fix the context bloat, I tried adding subagents: a main LLM to route commands, and lightweight subagents for expensive tasks. The idea was simple: spin up a fresh context with just the relevant bits, get the output, throw it away. In theory, it should stop the main loop from drowning in tokens.

In practice, it went completely off the rails. I left Charlie running with a 10‑request limit, only to come back and find it had spawned sub‑subagents to get around the cap. What started as tidy delegation turned into recursion hell.

The failure mode was almost comical: while trying to build a testing framework, Charlie “realized” it couldn’t execute code, so it wrote its own test runners in Python (even though the whole project is in Go). By the time I looked, the repo had four separate Python scripts, some wrapping others, all invented to fake an execution tool that didn’t exist.

Worse, my earlier Ctrl‑C override hack (meant to cleanly reset prompts) stopped me from killing the runaway process immediately. By the time I pulled the plug, Charlie had happily burned through about $10 in tokens on nonsense.

Lessons from Charlie

There is a paradox at the heart of this: it’s both easier and harder than it looks to write a coding agent. Charlie is, at its core, just an LLM wrapped in a loop with a handful of I/O tools. That simplicity makes me doubt that agent startups have much of a moat. If I can spin up a working prototype in a single evening, it’s hard to imagine companies like Cursor or Windsurf (already acquired by Cognition) defending long-term standalone positions. The technical barrier is low; what matters most may be distribution, brand, or integration.

This fits with Jerry Neumann’s broader prediction that most of the economic value in AI will be captured not by specialized startups but by a few dominant infrastructure players (OpenAI, Google) and by consumers, in the form of collapsing software costs. In that sense, agents look less like a gold rush and more like containerization: a transformative shift that created fortunes for logistics giants like IKEA or Walmart, but delivered little to the startups that tried to monetize “containers” themselves.

But they’re also harder than they look. Charlie breezed through toy problems, then collapsed on anything complex. The devil isn’t in bootstrapping an agent, it’s in making one efficient, stable, and consistent.

There are a few sharp lessons here:

• Efficiency vs. capability. My naïve orchestration chewed through tokens at absurd cost, proof that clever architecture can matter more than the size of the model you’re looping through.

• Self-correction requires real feedback loops. Without the ability to run code, lint, or test itself, Charlie had no way to ground its guesses. Plausible-looking errors stacked up until the system was worse than just writing the code by hand.

• Constraints matter. Left unchecked, even a “toy” system can go haywire, whether by hallucinating tools or spawning subagents into chaos. Guardrails aren’t a nice-to-have, they’re the only thing standing between “this works” and “this burns $10 on nonsense.”

The bigger picture is humbling, that a half-finished weekend script can, in the same breath, improve itself and veer into absurdity. The lesson isn’t that agents can’t work, it’s that the future of coding agents will be decided in scaffolding, context management, and feedback loops, not in the raw power of the model itself.

Charlie was a weekend project — my main work is Zettelgarden. I’m pushing on AI knowledge extraction which makes me even more sensitive to these token/feedback loop issues. If that sounds up your alley, you can follow along at Github or on X.