Version a prompt the way you version code: keep it in the repository, change it by commit with the reason in the message, and change the instruction and the check that enforces it in the same commit. That is what kept the agent that drafts posts for this site working through five changes to its instructions in its first week. The checks have automated tests. The prose of the prompt has none, so its regression test is the output: every draft goes through a quality gate and a human review. The part we have not versioned yet is the model itself, and this post says what we plan to do about it.
The prompt lives in the repository, and the scheduler only points at it
The agent runs once a day from a scheduled task in the Claude desktop app. That task has its own prompt, and it sits outside the repository, in a folder in the user's home directory, where git cannot see it. So we kept it thin. It says where the agent may work, tells it to update its clone of the repository, and then hands over: follow marketing/PLAYBOOK.md exactly, because that file "is the source of truth for every step, limit and rule".
That one decision means almost every change to the agent's behaviour goes through a commit and can be read, reviewed and reverted. The playbook is about 1,400 words of plain instructions. Around it sit three files that work as part of the prompt, though they are code or data: the pre-flight script that chooses a topic, the quality gate that checks a draft, and a config file holding limits and the list of sources the agent may cite.
If your prompt lives in a database field, a vendor's dashboard or a scheduler's settings screen, the first step of prompt versioning is moving it somewhere with history. Ours took a deliberate split: the part we cannot version stays short and stable, and everything that changes lives in git.
Five changes in the first week, and why each one happened
In its first week the playbook had six commits: the one that created it on 31 October 2026 and five changes. Each came from something that went wrong or was about to.
| Date | Commit | What changed | Why, from the commit |
|---|---|---|---|
| 31 Oct | a7bcb36 | Name draft branches after the topic id, not the post slug | The pre-flight could not tell which topics already had a draft waiting, so the next run would have drafted the same topic again |
| 31 Oct | 56a0e3f | Install dependencies before building; if the branch cannot be pushed, put the whole post in the final message and stop | A fresh checkout had no dependencies, and a sandbox that cannot push must not go looking for another route |
| 31 Oct | 97f7ce5 | Open and cite only hosts on an approved list | A run on a local machine can reach the whole internet |
| 1 Nov | fc3b733 | Excerpt 80 to 160 characters (was 200); page title 60 characters or fewer | An SEO audit: the excerpt doubles as the meta description, and search results cut titles after about 60 characters |
| 6 Nov | a5cdb87 | End each pull request with the one step code cannot do: request indexing in Search Console | New posts should become discoverable without anyone remembering how |
Two things stand out in that table. Every change is narrow, a sentence or a paragraph, which makes each one easy to review and easy to blame later. And most of them are about what happens around the writing, such as branches, builds, pushes and sources, and very few are about how to write. That matches where the failures were.
When the site moved, the versioned file redirected the unversioned one
The thin wrapper got its real test on 16 November 2026, when the website moved to a new host and its source moved to a new branch. The scheduler's prompt, the part outside git, still tells the agent to check out the old branch, main, every morning. Nobody had to find and edit it in time, because of the split described above: the old branch's copy of the playbook gained a short note at the top saying the site has moved, the branch is retired, and the run should switch to the new branch and follow the playbook there from the top.
The new branch's playbook changed in the same move. It gained a first step that checks out the right branch, and its build check was replaced by a check that the post can be rendered by the new site. Both changes went in by commit, like every change before them. The wrapper still names the old branch, which is now a known gap with a cheap fix, and a reminder that whatever you cannot version should hold as little as possible.
Change the rule and its check in the same commit
The source-list change and the SEO change each touched two files at once: the playbook, which tells the model what to do, and check-post.mjs, the quality gate that refuses a draft breaking the rule. On 1 November the playbook's instruction moved from "80-200 characters" to "80-160 characters", and in the same commit the gate's limit moved from 200 to 160 and the page-title check went from a warning to a failure.
Doing them separately goes wrong either way round. If the prompt says 160 and the gate still allows 200, nothing enforces the new rule, and the model will drift back to longer excerpts on the days it ignores that line. If the gate says 160 while the prompt still says 200, every draft fails and the agent burns its fix-up attempts on a rule it was never told. In one commit, the instruction and its enforcement cannot disagree.
There is a side effect worth knowing about. A rule tightened after a post shipped will flag that post the next time the gate runs over the whole site. When we did that for our post on the pipeline, the first post it ever published failed on exactly this change: its 176-character excerpt was legal on the day it shipped.
The checks have tests, and the prose does not
The scripts around the prompt have 22 automated tests between them. They read like a specification of the rules, for example:
- "a post full of AI tells, invented numbers and unresolved gaps is rejected on every count"
- "an approved topic is skipped, and explained, until its first-party material exists"
- "a topic with an open draft branch is skipped, and the next one is scheduled after it"
- "with an approved-sources list, a post may only cite approved hosts"
When the redrafting bug was fixed on 31 October, the fix came with tests for it, so it cannot quietly come back. That is ordinary software practice, and it is available to any rule that a script can check.
The prose of the playbook has no equivalent. There is no test that fails when a sentence about tone stops working. Its regression test is the output: every draft passes through the gate, and then through a person who reads it before merging. That catches a lot, but it catches it late and one post at a time. Anthropic's write-up on evaluating agents describes the state we are in with the prose: without evaluations, debugging is reactive, and teams cannot tell real regressions from noise. The next thing we would add is a small replay set: a handful of fixed topics with their material, run after any playbook change, with the drafts compared against the previous version.
Prompts written by someone else get the same review
The agent also follows six marketing skills, files of instructions written by an outside author and installed into the repository. They are prompts too, and an update to one changes the agent's behaviour as much as an edit to our own playbook.
So they are committed, and the playbook forbids installing or updating them during a run: "they change only by a reviewed commit". Updating them is a human step: run the update command, read the full diff, then commit. One of the automated tests fails if any skill gains a file that is not markdown, because a skill that ships a script has turned from instructions into code, and code deserves a different review.
What we have not versioned yet: the model
The scheduled task does not name a model. It runs on whatever the desktop app is set to use, which means the same playbook can meet a different model from one month to the next without a single commit in the repository.
OpenAI's own prompt engineering guide warns that different snapshots of models in the same family can produce different results, and recommends pinning production applications to specific snapshots and building evaluations to monitor prompt performance. We agree with the advice and have not followed it yet for this agent. The gate and the review limit the damage a model change could do, because a draft that changed style for the worse would still have to pass both. The cheap improvements are to record the model's name in every pull request the agent opens, and to run the replay set whenever it changes.
A minimum setup for prompt versioning
- Put the prompt in the repository. Anything that must live elsewhere should be short and should point back at the versioned file.
- Change prompts by commit, one change at a time, with the reason in the message.
- For every rule a script can check, write the check and change it in the same commit as the instruction.
- Give the checks tests, so a fixed failure stays fixed.
- Treat third-party prompts, skills and templates as dependencies: commit them, review their diffs, never update them mid-run.
- Record which model produced each output, and keep a small replay set to run when the prompt or the model changes.
In most agent projects the prompt is the least visible and most frequently edited part of the system. If yours has grown past what anyone can review, we can help untangle it.
