After the Prompt Series: After the Feedback, What If We Could Train Skills?

In the previous article, I discussed a simple idea: every time we correct an agent, we create knowledge, but that knowledge often disappears when the session ends. Warp captures those corrections, analyses them and turns useful feedback into reviewed skill updates. I find this valuable because human feedback becomes reusable. It also leaves another question open: once we start collecting that knowledge, how do we know whether our changes to a skill actually improve the agent?

SkillOpt offers an interesting way to approach this question. It treats the skill itself as something we can optimise. The model’s weights stay unchanged. What improves is the procedure that guides its behaviour: how it investigates, which evidence it looks for, when it uses a tool, how it checks a result and how it structures an answer. The model can remain exactly the same while the agent performs better because the operating knowledge around it has improved.

Improving the skill while keeping the model

When we want to improve an agent, we often consider familiar options: use a better model, provide more context, add tools or adjust the prompt. SkillOpt explores another option: keep the target model and execution environment fixed, and systematically optimise the skill used to do the work.

This changes the question we ask. We need to compare how different versions of a skill perform. SkillOpt runs tasks with the current version, records what happens and uses that evidence to propose edits to the document. The evidence can include the agent’s messages, tool calls, results and available evaluation signals.

At this point, it might sound like automated prompt rewriting. The useful part is how a proposed version is accepted. Clearer wording or a more detailed explanation is not enough: the candidate must improve performance on a held-out selection set, separate from the tasks used to generate the edits. Final test evaluation is separate again.

We can then judge a skill by the behaviour it produces, instead of relying on how convincing its instructions sound.

Small changes matter too

Another part of SkillOpt that makes sense to me is its limit on edits. The system can add, delete or replace text, but an editing budget controls how much it can change in an iteration.

Think about this from a software development perspective. If a skill works well in twenty situations and fails in one, allowing an agent to rewrite it completely could solve that problem while breaking several behaviours that already worked. An incremental approach makes sense: change one part, run the evaluation cases again and check whether the result has improved.

Rejected changes can also be useful. When an edit looks reasonable but makes the results worse, the optimiser can retain that information and avoid repeatedly trying the same direction. The comparison with training starts to make sense here, although the object being optimised is text and procedure, not model weights.

A current skill and a candidate update, with validation evidence and a human question: Does it actually help? — A proposed edit needs evidence before it becomes the next skill version.

Where human feedback fits

The previous article started with a person’s feedback. A developer corrected an agent’s recommendation, and that explanation could contain knowledge worth keeping. SkillOpt mainly starts from another signal: the agent’s executions and the evaluation of their results.

I think these ideas fit well together. Human feedback can explain why a decision was wrong, while execution evidence and validators can help us check whether a skill update improves behaviour.

Imagine a developer correcting an agent in a PR and explaining the reason. An improver agent considers that feedback alongside previous runs and proposes a small skill edit. Before accepting it, we run the candidate on known evaluation cases and compare the results with the current version.

Warp also recommends building a verification harness where the domain allows it. SkillOpt makes the propose-and-test process an explicit optimisation loop. The connection I am exploring here would combine human corrections with that systematic validation.

The proposed cycle could be:

Execution → human feedback → proposed edit → validation and review → new skill version

This gives the reviewer evidence about whether a reasonable-looking change actually helps.

What this could bring to AL and Business Central

In Business Central development, many stages already provide useful evaluation signals. For an AL implementation, we can compile the code and run tests. During a review, we can use analysers, validators and specialised knowledge such as BCQuality. For less deterministic work, such as architecture or specification, we can check specific properties of the result and see whether the next phase can proceed without another agent inventing missing information.

We cannot reduce everything to a number, and I do not think we should try. An architectural decision is not validated simply because the code compiles, just as a test alone does not establish the right customer convention. Still, the approach can be useful without making every aspect perfectly measurable.

We can start where the evidence is clear and build richer evaluations from there. In a framework such as ALDC, I would consider at least four types of signals:

compilation and test results;
findings from analysers, validators or BCQuality;
human decisions at approval points, reviews or PRs;
results from later phases, such as whether a specification supports implementation without leaving gaps to fill.

I would use these signals to understand how a skill works in practice, while keeping their different meanings visible. Combining everything into one score could hide useful information.

Today, we design skills that explain how an agent should work. A possible next step is to keep evidence of how those skills perform in real runs: which cases they handle well, where they fail, which edits we have tried and which ones we have had to revert. This is a direction to explore for ALDC, rather than a capability demonstrated in this article.

That would give us a stronger engineering basis for maintaining the instructions.

A skill should be able to evolve

We often design a skill, test it a few times and consider it finished once it behaves reasonably well. But agents do not work in a static environment.

Models change, new tools appear and we find cases we did not anticipate. We also discover that some instructions we considered essential add little value. A skill used for several months will probably face situations that never appeared in its first tests.

It makes sense for the skill to evolve. The important question is how. I would not let an agent freely change a production skill because it found a problem in a single run. I would prefer a process where the change is proposed, explained and validated, with human approval depending on the kind of skill and its impact.

We need to be able to trace what changed, why it changed and which evidence supports keeping the new version.

The harness question

SkillOpt also explores an issue that interests me because of our work around ALDC: whether an optimised skill remains useful with another model or execution harness, the environment that runs the agent and exposes its tools.

The project reports transfer experiments across models and between Codex and Claude Code. Those results support portability in the tested settings; they do not establish that every skill will work unchanged in every environment. GitHub Copilot is part of the ALDC context I am discussing here, not one of the harnesses reported in those experiments.

We may have GitHub Copilot, Codex and Claude Code working on the same project, each with different tools, primitives and ways of running agents. If our knowledge is closely tied to one harness, changing the environment can mean rebuilding a significant part of the system.

If some of that knowledge is expressed as a reusable procedure, part of its value may survive those changes. We still need to adapt tool names, capabilities and execution contracts to each environment, and evaluate the skill there.

For frameworks that need to keep working as models and tools change, I think this distinction matters. The model and runtime can evolve while we preserve useful operating knowledge.

From writing skills to maintaining their behaviour

SkillOpt suggests a change in how we think about skills. We often treat a skill as advanced documentation: we write it, version it and edit it when something goes wrong.

We can also treat it as a component whose behaviour we can observe and compare. We run a version, analyse the result, try an edit and check whether it improves or worsens the outcome. We keep useful changes and learn from rejected ones.

The final result does not have to be complex. For the production agent, it can still be a Markdown file. SkillOpt exports the selected skill without requiring the target agent to carry the optimiser’s memory. The more involved work is the process that produced that version and the evidence supporting its use.

Bringing both ideas together

The previous article ended with a question: how often have we corrected an agent and lost that correction when the session ended? SkillOpt adds another: of all the changes we make to our skills, how many do we know have actually improved the agent?

A longer skill is not necessarily a better skill. More rules can make the result worse, and even wording that seems clearer to us may cause problems when the agent follows it.

That is why I find these approaches useful together. Warp shows how to retain human feedback through reviewed skill updates. SkillOpt gives us a systematic way to test proposed edits before accepting them. A correction can be valuable without every change based on it being an improvement.

The process I can imagine is practical: agents work, people correct them, those corrections become evidence, and another agent proposes changes. Technical checks, execution results and human review then help us decide which changes are worth keeping.

I would be cautious about calling this agents learning by themselves. I see it as operating knowledge that can evolve through real work, within a process we can inspect, evaluate and review. For me, that is one of the most interesting directions to explore.

Read the Spanish version on Substack.