How do you make Claude skills actually improve over time?

I have a judgment agent that sorts my Gmail deciding: to keep message in the inbox, or push to a digest folder. It runs every hour since May.

The fear is that the script will make judgment errors at first. What fixed this was one rule: the agent is never allowed to rewrite its own prompt.

Here is the shape that works.

1. Split the judgment from the memory. The rules layer, the "how do I decide" part, is edited by a human only. It never touches itself. What updates automatically is a separate table of per-sender stats: digest count, keep count, false positives, false negatives. Corrections move numbers in that table. They never move a word in the prompt. That kills drift at the root. The rules cannot contradict themselves or bloat, because feedback flows into data, not into text.

2. The error log is just using the inbox. Nobody writes "this was misclassified" into a doc. When I move a message the agent got wrong, remove the digest label or add it, the next run reads the Gmail history and maps that change back to the original decision. Remove a label it added, that is a logged false positive. Add one it missed, false negative. The correction is the fix. In your CV case it is the same: the client re-files the profile into the right bucket, and the job reads that state change instead of parsing a prose error log.

3. Every decision is logged as structure, not free text. Which rule fired, the confidence, the sender, the tier. So when I review I see "rule X caused N false positives on domain Y," not a pile of anecdotes. That is what makes the feedback actionable instead of vibes.

4. The human sees a report, not a self-editing skill. A script reads the decision database and writes a weekly markdown review: every false positive with its subject line and the rule behind it, senders that have earned more autonomy, and a sample of the borderline calls to spot-check. Fifteen minutes on a Friday. I decide what becomes a permanent rule. The system proposes, I dispose. That is your "skill proposes a diff, human validates," except the diff is a readable report, not a prompt rewrite I have to audit line by line.

5. Autonomy is earned per sender, and it is reversible. New senders always go through the expensive LLM call. After 50 clean decisions with zero false positives, a sender goes on autopilot and skips the LLM. Three false positives and it drops back to manual, automatically. The system automates itself exactly where trust was earned, and self-corrects where it was not. No human in that specific loop, but hard guardrails around it, and always reversible.

Net: it gets better every week without me babysitting it, and it still cannot drift its own judgment. The only thing that updates unsupervised is a reputation ledger sitting behind red lines. The prompt changes when I change it, and only after a report has already done the aggregation for me.

Happy to go deeper on any of the five.