John Jeffers

How to Talk to Your Robot: Keeping Coding Agents on Track

As the maintainer of Luxury Yacht, I spend a lot of time talking to LLMs. A lot.

I wouldn't say this is fun. Honestly, a lot of it is drudgery. It's keeping the robots in line. It's waiting a long time between prompts, making sure they don't do stupid things, and that the code they spit out isn't total garbage.

No matter how much time and effort I put into documentation, agents files, skills, or whatever the hot thing is this month, LLMs only have so much context, they can't truly learn anything, and those **DO NOT IGNORE ME UNDER ANY CIRCUMSTANCES** critical instructions you carefully designed are eventually going to get dropped out of the context window to make room for who-knows-what. And that gets more likely the longer a session goes on.

Having said that, Luxury Yacht is largely written by LLMs, and I think it's a pretty good app. I don't like the term "vibe coded" because that implies that I don't care about code quality, and that is absolutely not true. I don't blindly accept whatever the robots churn out. I spend more time in optimization passes than I do in feature work, trying to make the code the best it can be, to the best of my ability as a person with far more experience in platforms and infrastructure than in software development.

With all that in mind, I'd like to share some of the things I've done that seem to help, at least a little bit. I'm not pretending to be an expert here. It's a lot of trial and error, and I have no doubt that there are things I could be doing better. We're all learning, and the AI landscape changes rapidly.

There Are No Silver Bullets

Let's just get this out of the way: for large, complex projects, you'll be wrangling the LLMs a lot. Keeping them on track can be difficult and requires vigilance. LLMs love to be "clever", so you'll be busy making sure that they stick to established patterns, reuse existing code, and don't reinvent yet another wheel. You'll be frequently reminding them to validate that a change they made in one subsystem doesn't have unforeseen consequences elsewhere.

Agents Files

Agents files are the first place an LLM looks for the lay of the land, so this is where you want to put your important rules about how you want the LLM to behave, and tell it where it can find more information.

It won't always follow these instructions, and sometimes it'll forget the very important things you put in here, but it's where to start. If you do nothing else, create an AGENTS.md file in the root of your project or repo.

Curious about what to put in AGENTS.md? Start here.

Claude has to be different

Unfortunately, Claude doesn't want an AGENTS.md file, it wants CLAUDE.md. If you only use Claude, put the instructions in CLAUDE.md instead of AGENTS.md. But, if you use a mix of LLMs including Claude, cover your bases by putting your instructions in the AGENTS.md file, then create a separate CLAUDE.md file with only this line:

@AGENTS.md

Using multiple agents files

For a large project, you might want multiple AGENTS.md files spread around the project. This can help if you're tasking an agent with working on a specific part of the app, and reduce the amount of instructions it has to read vs. dumping everything into a single agents file in the root.

For example, Luxury Yacht has an agents file in the root that talks about overarching rules for the entire project, but then has additional agents files in the backend and frontend directories with more specific rules and info about those areas of the app.

By the way, Luxury Yacht is open-source, so you can go look at the github repo to see the agents files I use. You're not going to want to use my exact files for your project, but they might give you some ideas of how to structure yours.

Documentation

I make sure my agents files contain info about where to find documentation for the app. Let the LLMs write their own documentation. Tell them that they are writing it for themselves, so it doesn't need to be human-friendly, but rather optimized for consumption by agents.

Every time the agent modifies the app, it shouldn't consider the task complete until it updates any relevant docs with information about the changes. Instructions for this are of course in the agents files, but I sometimes have to remind the agent to do this before closing out a task.

I put all of my documentation in a docs directory in the root. Unlike the agents files, I don't split it into frontend and backend docs because the split is not always that clean. Many systems have cross-cutting concerns. I usually let the LLMs decide how to organize the docs, since that's who they are for.

I drop a README.md into the docs directory that functions as an index to tell the agents what docs to reference. Rather than try to explain exactly what I mean, I encourage you to look at the file. As with the other docs, the agent is responsible for updating this top-level README any time it makes changes that would impact it.

Skills

Skills are bundles of instructions (a markdown file, and possibly other scripts and reference files) that tell an LLM how you want it to perform a specific task. It's a broad topic that I'm not going to go too deep on in this article. There is a wealth of existing resources for skills, far more comprehensive than I could hope to be here.

If you find yourself frequently repeating a task, you might want to write up a skill for it, or ask an agent to write the skill.

For example, in Luxury Yacht, I have a skill that gives the agent information about how to make changes to the refresh subsystem, the part of the app that makes sure the Kubernetes data being displayed in the app is fresh. This is a complex and fragile system, so it's important to give the agent as much help as I can when it has to interact with it.

There's some crossover with docs here. The way I think about it is that the docs are the what and the why and the skills are the how.

Tests and Linters

A very effective step you can take early on is to enforce tests and linters. Never allow the agent to consider a task to be complete until it has written tests and run a linter.

I find Red/Green TDD (or Red/Green/Refactor) to be particularly effective for LLMs, as it makes them think a little harder about how to properly implement something. If you're unfamiliar with Red/Green tests, it boils down to this:

  1. Red step - write a test that describes what you want. Verify that the test fails.
  2. Green step - write the simplest code to make the test pass.
  3. Refactor - improve the code while ensuring the test keeps passing.

After the tests, you should make sure the LLM runs a linter on the code it generated to catch the easy things that a linter is for. Good tests and a good linter will save you a lot of trouble.

Hooks

One of the newer things I've tried (and I'm not sure about its effectiveness yet) is using hooks to require the agent to perform a task at a certain point in its lifecycle.

The idea to use hooks came after a very frustrating Claude session where I just could not get it to behave. It kept making the same mistakes over and over, ignoring explicit instructions, and I asked it how to get it to stop doing that. It suggested creating a PreToolUse hook to force it to "map the blast radius" before making any changes.

The hook that I use consists of a few parts:

One reason I say that I'm not sure about the hook's effectiveness is that I currently only have the hook set up for Claude, and not Codex. But I don't think Claude does a better job with the hook than Codex does without it. It really depends on the specific task.

Adversarial Reviews

This is maybe one of the most important things I do. For any even moderately complex task, I'll pick a second agent to challenge the other. Typically I use a different model. If Claude is the author, then I'll use GPT as the reviewer, but even using the same model for both (in different sessions) can be effective.

I typically start by asking an agent (the author) to write a plan, then I'll point the other agent (the reviewer) at that plan and ask it for a review. LLMs seem to absolutely loathe coming up empty-handed, so it'll nearly always find something wrong.

I'll take the output of that review, feed it back to the author, and say something like:

Another reviewer has provided feedback. Do not assume these assertions
are correct. Validate all findings.

Then I'll paste in the reviewer's comments.

After the author agent has fixed (or not) the reviewer's findings, I'll go back to the reviewer and say:

changes made, review again

Typically it'll find something else, and I'll just repeat the process until both LLMs agree that everything is fixed, or we can at least get to a point where the disagreements aren't significant enough to continue. Depending on the complexity of the task, this can take a while.

If it's a particularly difficult or risky task, I might start a new session and repeat the process. As I said, LLMs hate to come up empty-handed, so it's rare that it doesn't find something to complain about for the other agent to fix even on a fresh second pass.

BTW, I know copy/paste isn't the most elegant way of doing this, but it's simple and harness-agnostic, and I'm trying not to get too in-the-weeds in this article.

The Wrap-Up

I said earlier that there are no silver bullets, and I hope this article helps you understand why. Building complex software with LLMs requires discipline. Every step you can take to reduce the amount of handholding and babysitting you have to do is worth it, but figuring out which of those steps are effective can feel like a mystery to solve.

If I had to distill it down, I'd recommend this:

None of this eliminates the supervision I talked about, but it should help to make LLM-assisted development in complex projects more manageable.

#ai #development