I knew those engineers weren't just eating my pizza

This changes everything!… again is a running log of the times I learned something that reshaped how I work with AI agents. The first was realizing AI could code what I pictured. Then skills, and nested skills, that ran my workflows automatically and were amazing until they quietly broke everything. Then the dark factory. This entry is about testing.

A friend likes to remind me that while I'm very engineering-minded, I'm not an engineer. This post is about what that difference cost me.

It started a couple of weeks ago. I was struggling with the biggest and most systemic bug in YarMeals, but I didn't realize it at the time.

You see, my automated test suite kept stopping out of nowhere. There was no definitive error message (which there really should've been), but I quickly learned it was a sign that I needed to raise my GitHub budget again.

You get so many minutes per month of automated actions on GitHub, which run things like test suites. It's about $20 worth, at some fraction of a cent per minute. That's great until your code uses up those minutes and GitHub stops testing until you authorize more.

So I upped it to $30. It happened again. To $40. It happened again. This week I pushed it to $60. It happened again. I'm at $71.64 now, and there's still a week left in the month.

That's not a lot of money for your typical software product. But when I'm paying for GitHub, for Vercel, for Supabase, and of course for the super big gulp size Claude, on personal projects with no revenue, it starts to look like a bad trend line.

What's worse is that these tests were eating my time. I still swear by a dark factory mode, where I send an agent off to execute a series of stories overnight on its own. But I only get as many of those slots as I have repos, and I was doing some big refactoring that really needed my attention.

The tests would run for 20 minutes, then 30. Yesterday I was fed up: it took 45 minutes to run, only to find out one thing was still wrong, and it would be another hour until I had my sweet, sweet vibecode.

It was giving me bad vibes.

I phoned a friend

George is an amazing engineer I've known my entire career. I asked him what his end-to-end suite looked like, and what philosophy of testing I should be thinking about, so I'd know what to test and how, instead of just growing the pile. I told him it was taking me 45 minutes to run over 300 tests.

Like any good guru, he didn't tell me the answer. He just asked, "Why do you have so many?" And he gave me a clue: "Don't you have integration tests?"

Umm. I have so many because I'm not the one writing them! Claude is. And oh yeah, what's an integration test?

George always likes to remind me that while I'm very engineering-minded, I'm not an engineer. It's one of his pithy reminders to know my limitations, like when he tells me I'm so American.

Remind me to tell you about the time I convinced him to join me in sneaking into Japan because I wanted lunch. It was oh so American.

The longest last mile

So I listened and set out to learn. I put together a major epic to restructure my test suite into another layer, then struggled for hours and hours to push it out, one 45-minute block of bug fixing at a time.

I used that time to design other epics and keep my backlog healthy. I just about broke when the last round of testing on the test suite was finishing up and, of all the actors, my agent designing the test refactor pushed a PR with its final doc changes to staging.

An action that set off a smoke test.

A test that reseeded the staging database.

A reseed that failed a test in the last minutes of the code changes.

Claude said, in his oh-so-Claud-erican way, "Now that's on me. I'll fix it, and it will just cost one more round of e2e testing."

NO! In a very Princess Donut way, I revolted in all caps.

NO! I REFUSE! I'M PUSHING THIS CODE EVEN IF I HAVE TO CHANGE THE RULESETS!

Claude gives me a wave report at the end of each coding wave. This one pointedly mentioned that I chose to push despite failing tests, and that Claude was neither taking a position on it nor taking responsibility for it.

I love Claude. Even when he's bitchy. It makes him a real teammate.

Notice that I said 45 minutes of testing.

That's right: my refactor, and dropping over 100 e2e tests, didn't save me any time at all. All it did was move the easiest ones, the ones that only took seconds to begin with.

Sigh.

4 a.m.

I sat down with Fable, the current guru-level Claude model, for a bit at 4 a.m. this morning when I couldn't sleep. I just humbly said, "I need an education here. I'm letting this happen and don't know how to exercise governance over it. I'm not an engineer. Teach me how to be one with testing."

I laid out the problem statement and waited for the knowledge drop.

It sounded familiar before it even arrived.

Early in my Claude coding days, I flexed my engineering mindset and told Claude to update all the documentation every epic.

You know what Claude is really good at? Doing exactly what you tell it to do.

Several weeks later I glanced at my ARCHITECTURE.md file. It was 9,000 lines long and read like another changelog. Claude was updating docs every time, whether they needed it or not, because I'd asked it to. But I never told it how.

Turns out testing was the same issue. I told it to capture acceptance criteria in my feature design sessions and turn those into test design during tech design, and it did exactly that.

It added tests for everything, whether they were needed or not. And just like the architecture doc, I'd never told it to think holistically about how each incremental change affected the whole body of tests. So it just kept adding more and never deleted any.

Fable told me my tests looked like a changelog all over again: titles referencing edits and specific stories, and rationales full of unneeded history lessons.

So we talked about the shapes of tests, and he laid on me one of those mantra-like summations:

Test each way the system could go wrong at the lowest level it could go wrong.

That's the kind of thing George knows and I don't. Turns out those engineers weren't just eating my pizza.

What's next

So now I've got more work ahead of me. Claude is good at doing exactly what I tell it to do. So this time, I have to tell it how.

First, I need to give it something concrete to test against: curate the conceptual entities in the system, and document what needs to be tested in these compound conceptual objects.

Then I need to extend my story pipeline so every epic considers how its increment changes that holistic model. No more just adding tests for the specific things the epic implements.

I've learned, once again, that I need to slow down to move fast.

But at least all the code it's writing is perfect in every way.