AI engineering

The cycle that feeds itself

links 5

The other pieces of the system are static: someone has to write the CLI, someone has to write the skill. This is the piece that produces the others.

The question it answers is: what guarantees that the tooling gets better without me having to stop to make it better? Because stopping for that is exactly what never happens when there is something to deliver.

The cycle

The entry bar is what makes it work

Only what cost more than one attempt and whose symptom pointed to the wrong cause goes into the dictionary. Typos do not go in, nor does what the next person would find out on the first try.

That bar is the difference between a useful journal and a list of everything that went wrong. Nobody rereads a list of everything that went wrong, and what nobody rereads becomes nothing.

The cross-referencing is where the value is

Recording the problem is not for remembering the bug. It is for the patterns to show up: reading the pile you can see that a whole class of problem has a common origin, and then the right fix is at the origin, not in the case.

That cross-referencing is automatic. The system detects the repetition, identifies that it became recurring knowledge instead of an accident, and proposes the skill that solves the whole class. I review and approve. It is the only point in the cycle where I come in, and I come in judging, not executing.

A real case, end to end

The best example of the cycle closing was not one problem. It was three, which did not look like the same one.

First symptom. A package installation failed saying it could not resolve the host. Right after, any curl started answering HTTP 200, with the HTML of a login page in the body. I went through “the package manager broke” and “DNS problem” before reaching the cause: the session on the network edge appliance had expired, and the captive portal intercepted everything answering 200. Some eight attempts. The wrong hypothesis was reasonable: the first error was a name resolution one.

Second symptom, days later. A mobile app build could not resolve a dependency, and curl on the same host worked. That makes no sense until you find out that the same appliance does selective TLS inspection: it intercepted one vendor’s artifact repository and did not intercept the language’s public repository, and that each tool picks its own truststore: curl uses the operating system’s, where the corporate CA is installed; the build uses the JDK’s, where it is not. Three attempts.

Third symptom. The app on the emulator froze at “no connection”, and it looked like an app bug. I knocked down two framework hypotheses before finding it: the operating system runs an HTTPS probe to validate the network, the probe failed against the corporate CA, and the system marked the network as partial connectivity. Without the validated network mark, the app’s library just repeated what the system said. Some ten attempts.

The cross-referencing. Three incidents, three symptoms with no apparent kinship: name resolution, build, emulator connectivity. None of them, alone, revealed the pattern. It was reading the pile that it showed up: it is the same appliance, with three faces. A whole class of problem, not three accidents.

What it became. A network skill the agent loads by itself when a symptom from that family shows up. It describes the three faces, explains why curl works and the build does not, and brings the procedure for each case, including where the exported CA is, to install in the right truststore.

The cost before and after. Adding up the three incidents, more than twenty attempts and a day lost in a diagnosis that went through package manager, DNS, framework and app bug: none of them the cause. Today, the symptom is recognized in the first message of the conversation, and the answer comes with the procedure attached.

And a case where the cycle did not produce a skill

It is worth more than the previous one, because it shows that the judgment step really exists.

There was a recurring problem: common commands failing with “command not found”, including basic system utilities. It looked like a missing package. The real cause was the shell’s PATH collapsing every now and then and losing even the system’s default directories. It happened twelve times, in five different projects, over a month before I recorded it.

Twelve recurrences is exactly the pattern that makes the system propose a skill. I reviewed it and decided it does not become a skill. A skill delivers knowledge to an agent; the problem here is not lack of knowledge, it is the environment breaking. A skill saying “when the PATH disappears, re-export the PATH” would be automating coexistence with a defect instead of fixing it, and worse, it would make the defect look solved.

The record stays in the dictionary, marked as reviewed, and that is what it should be: a known diagnosis waiting for a fix at the origin. Not every recurring friction is a tooling opportunity. Some are just debt waiting for someone to take it on, and the difference between the two is the only thing in the cycle that cannot be automated.

The second cycle: what slipped through in review

The friction cycle captures what cost me a lot. It does not see what went past me without hurting and was caught by someone else.

That is why there is a second journal, fed by code review, and its entry is not the defect on the line, it is the class of check that was left out, written as a check that can be run before opening the next MR.

Two decisions in this design that I find the most interesting of everything I built:

It records both directions. Each entry notes whether the finding was in my code or whether I was the one who found it reviewing someone else’s. It looks like a detail and it is the opposite: two occurrences of the same class in different directions prove that it is not one person’s habit, it is a process gap, and then the checklist really is the right place, and not an individual conversation.

It records what the reviewer checked and was right. Without that the checklist only grows, because nothing ever leaves it. It is the record of the hit that allows removing an item when that class started coming out right by itself.

The entry bar is the same as for friction: only what could have been checked and was not goes in. Disagreement over names and design preference stay out, because a reviewer with fresh context always finds something, and treating every finding as a failure makes the checklist swell until it becomes noise nobody reads.

Why this is the piece that matters

Everyone who works with an agent accumulates learning about how to make the agent work. In practice that learning stays in the head, or in a notes file that ages, and each person on the team relearns from scratch.

Turning learning into an executable artifact is the same principle that produces a shared library instead of duplicated code, and a delivery template instead of a copied pipeline. What repeats is extracted to a single place. The difference is that here what is extracted is knowledge, and the single place is a tool the agent loads by itself next time.

It is also the reason the system gets better with use instead of worse. Tooling only a human maintains decays, because maintaining is not urgent. Tooling that feeds on its own friction improves exactly at the points where it is hurting.

The limit

The cycle depends on me continuing to approve with judgment. A skill generated and approved on autopilot becomes accumulated noise, and accumulated noise in the context degrades the result of everything. The approval is the gate, and it only works while I take it seriously.

And it only sees the friction that shows up while working with an agent. A problem I solve from memory, without going through the tool, does not go in, and that is precisely the class of problem I already master too well to notice it cost something.

Pablo Mickael Quevedo Senior Software Engineer · Novo Hamburgo, Brazil

5 notes linked to this one

This star's neighborhood in the vault sky.

Explore the galaxy →