EVERY RULE HAS A SCAR_
I built a system with more than twenty agents. Then I spent three months building governance to stop them from breaking things. Every rule in that governance layer maps to a specific failure. None of it was invented prophylactically.
That's the honest version of how this works.
> THE INCIDENT THAT STARTED IT#
The first real wake-up call was a deepcopy bug on live MCP pipes.
I had a multi-phase implementation running. Three phases merged without any testing between them. The logic looked clean. The code reviewed fine. The problem was invisible until a user got a response missing context — and I started tracing backward.
MCP connections are stateful. When you deepcopy a pipe handle, you get a copy that points to nothing. The original connection is still alive. The copy silently fails. No exception. No error log. Just missing data.
Three phases. No smoke test between any of them. One silent failure that only surfaced because the output was wrong in a way a human noticed.
The rule that came out of it: deploy and smoke-test after every phase. Never merge more than one phase without QA. It's in the governance docs now. It's also enforced structurally — the phase lifecycle won't advance without a QA gate.
Here's the thing about that rule. It didn't exist before the incident. I didn't write it as a precaution. I wrote it because I watched what happens when you skip it.
That's the pattern. Every rule in this system has a scar behind it.
Agents don't respond to instructions the way humans do. A human engineer reads "test between phases" and understands the intent. They apply judgment. They know when it matters and when it's overkill. An agent reads the same instruction and follows it until it doesn't — usually under pressure, usually when the task is long, usually when skipping feels harmless. Instructions get ignored. Governance doesn't.
The deepcopy incident was the moment I stopped treating this like a documentation problem and started treating it like an infrastructure problem. You don't fix infrastructure with better docs. You fix it with gates.
> HOOK DISPATCHER: GOVERNANCE-AS-CODE#
Documentation that agents can ignore is aspiration. It's not governance.
Real governance intercepts. It blocks. It returns a decision before the action executes.
The hook dispatcher runs before every tool call. It reads the input, evaluates it against a rules registry, and returns a JSON decision. Block or allow. No ambiguity.
#!/usr/bin/env bash
# Blocking governance gate — PreToolUse hook for Bash commands.
set -euo pipefail
input=$(cat)
cmd=$(echo "$input" | jq -r '.tool_input.command // ""' 2>/dev/null) || exit 0
# Block raw runtime update calls outside deploy.py
if echo "$cmd" | grep -qE 'update_agent_runtime|update-agent-runtime' \
&& ! echo "$cmd" | grep -q 'deploy.py'; then
echo '{"decision": "block", "reason": "GOVERNANCE: Use deploy.py as the single deploy path."}'
exit 0
fi
# Block az CLI mutations (reads are OK)
if echo "$cmd" | grep -qE '^\s*az\s' \
&& echo "$cmd" | grep -qE '\s(create|update|delete|set|assign|add)\s'; then
echo '{"decision": "block", "reason": "GOVERNANCE: az CLI mutations blocked. Use Python SDK via @infra."}'
exit 0
fiThe rules registry is a table. Each row is a pattern, a tool type, and a decision.
| Write, Edit, Bash | locked_files:src_runtime | Locked file — delegate to @governance (Tier 2) |
| Write, Edit, Bash | locked_files:claude_md | Locked file — delegate to @governance (Tier 1) |
| Write, Edit | locked_files:workflow_json | Workflow state — delegate to @project-manager |
| Bash | destructive:git_force_push | Force push without --force-with-lease |
| Bash | destructive:git_no_verify | --no-verify bypasses git hooks |Each of those rows traces to an incident.
locked_files:src_runtime — an agent modified the runtime handler without understanding the session lifecycle. It broke session continuity for every subsequent call. The fix took longer than the original task.
destructive:git_no_verify — an agent skipped the pre-commit hooks because they were slow. Pushed failing code. The hooks exist for a reason. Skipping them is not a time-saving optimization. It's a deferred incident.
locked_files:workflow_json — two agents wrote to the same workflow state file concurrently. The file ended up in a partially-written state. Neither agent knew the other was writing. The state was corrupted. Both agents continued operating on bad data.
The hook dispatcher didn't exist at the start. It got built after each of those incidents. The rules registry grew one row at a time.
That's how governance-as-code actually works. You don't design it upfront. You encode failures as they happen. The registry is a changelog of things that went wrong.
> PHASE LIFECYCLE: USEFUL, EXPENSIVE, AND HONEST ABOUT BOTH#
The system runs nine phases: INIT, PLAN, EXECUTE, UNIFY, INTEGRATE, ITERATE, VERIFY, SHIP, LEARN, ARCHIVE.
Most of those names are self-explanatory. A few aren't.
UNIFY is where the agent reconciles what was planned against what was actually built. Plans drift. Code drifts further. UNIFY forces an explicit comparison before anything gets integrated. It catches the gap between "what I intended" and "what I shipped."
INTEGRATE is blast radius analysis. Before merging, the agent maps which files changed, which systems depend on them, and what could break. It's not a code review. It's a dependency audit.
ITERATE is the go/no-go gate. After INTEGRATE, the agent decides whether to proceed or loop back. It's the only phase where the answer can be "start over."
The git history tells the honest story. One project was archived four times before it stuck. Four complete cycles. The first three produced work that looked done but wasn't. LEARN and ARCHIVE forced the agent to document what actually happened — and each time, the documentation revealed gaps that required another pass. Annoying. Also correct.
Here's where I'll push back on myself.
Nine phases for a blog post is absurd. Nine phases for a two-line config change is absurd. The overhead is real and it compounds across every task.
Two things got removed after honest evaluation. The context-packer hook added latency on every phase transition and produced no measurable value. Gone. The tasks.md tracking system got reverted — the overhead exceeded the complexity of the work it was tracking.
The honest answer about why the system enforces uniformly anyway: AI can't reliably distinguish "fine to skip" from "will break production." Humans can. The system can't. So it enforces the full lifecycle on everything and accepts the speed cost.
That's not elegant. It's a tradeoff. The cost is speed. The benefit is that the agent doesn't get to decide which rules apply to it today. Given what happens when agents make that call themselves, I'll take the overhead.
> THREE-STRIKE RULE AND THE OBSERVE PIPELINE#
Agents brute-force past failures. That's the default behavior.
Same approach. Same root cause. Different surface variation. Fourth attempt. Fifth. The agent is not learning. It's iterating on a broken hypothesis and burning time doing it.
The three-strike rule is a hard stop:
When the same problem fails 3 times using the same root cause or approach:
1. STOP immediately — do not attempt a 4th variation
2. Append to ./status/session-feedback.md:
- What was tried (each attempt)
- What failed (exact error or outcome)
- What you intend to try next
3. Wait for user input before proceedingThe rule doesn't fix the problem. It stops the agent from making it worse. It forces a human back into the loop before the agent has burned another hour on a dead approach. That's the entire value. Not clever. Just necessary.
The observe pipeline is the other half of this. It captures failures automatically and routes them into a structured improvement cycle.
python3 .claude/skills/observe/observe.py add \
--type gap \
--summary "Scanner returned exit 0 on network timeout — deploy gate passed without scan" \
--context "Pre-deploy security scan. urllib3 timeout. Exit code not checked."The pipeline runs: observe, reflect, curate, improve. An observation becomes a pattern. A pattern becomes a rule. A rule becomes structural enforcement.
That scanner failure is a good example. The pre-deploy security scan was timing out silently. urllib3 threw a timeout exception. The exit code wasn't checked. The deploy gate saw exit 0 and passed. The scan never ran. A deploy went out without a security check because the tooling failed quietly and nothing caught it.
The observation captured it. The reflect step identified the pattern — exit code validation missing on external tool calls. The curate step grouped it with two similar failures. The improve step produced a hard gate: the scanner must return a non-zero exit code on timeout, and the deploy gate must validate it explicitly.
That's the full cycle. Break, observe, reflect, encode structurally. Never rely on memory to carry the lesson forward.
> ENCODING LESSONS IMMEDIATELY#
Decisions made in conversation are decisions that get lost.
The session ends. The context window compacts. The next session starts fresh. Whatever was agreed in the previous conversation is gone unless it was written somewhere durable. This is not a flaw in the tooling. It's a property of how these systems work. You either design around it or you repeat yourself indefinitely.
The system has a LEARNED.md file. Each entry maps to a failure. The entries are numbered. The numbers matter — they're referenced in rules, in code comments, in governance docs. When a rule says "see L5," there's a specific incident behind it.
L5: no exec(). An agent used Python's exec() to run dynamically constructed code. The code ran in the agent's process context with full permissions. The exec() call was removed as a hard security boundary. The rule is now enforced by the hook dispatcher.
L8: don't kill the proxy. An agent autonomously restarted the kiro-gateway proxy to "fix" a connection issue. The proxy routes Anthropic API calls through Amazon Q Pro. Killing it disrupted every subsequent API call until the user manually restarted it. The agent didn't know what the proxy did. It saw a process, saw a connection error, and killed the process. The rule now: proxy lifecycle is user-managed. Agents don't touch it.
L14: don't write to the home directory. In Bedrock AgentCore, the runtime user's home directory is ephemeral. Writes there don't persist. An agent wrote session state to ~/. The state was gone on the next invocation. The rule: all persistence goes through the memory manager to S3-backed long-term memory.
L26: don't rely on FETCH_HEAD. In a multi-fetch workflow, FETCH_HEAD gets overwritten by every subsequent git fetch call. An agent fetched from two remotes in sequence and then tried to use FETCH_HEAD. It was pointing to the second fetch, not the first. The merge was wrong. The rule: use explicit refs.
L31: grep all references before deleting anything. An agent deleted a secret from Secrets Manager. Three other places referenced that secret by name. All three broke silently. The rule: one deletion equals one grep for all references equals one atomic commit.
L33: don't pursue tangents mid-task. An agent noticed an unrelated bug while fixing something else. It stopped the original task, fixed the tangent, introduced a new bug in the tangent fix, and never finished the original work. The rule: log tangents via observe and continue the original task.
The pattern is consistent across all thirty-three entries. Break. Observe. Reflect. Encode structurally. Never rely on memory to carry the lesson forward. Memory is volatile. The file is not.
> WHAT THREE MONTHS OF GOVERNANCE ACTUALLY BUYS#
This system is not elegant. It didn't start with a clean architecture diagram and a governance framework. It grew from pain.
The hook dispatcher has forty-something rules. The phase lifecycle has nine stages. The LEARNED.md file has thirty-three entries. The rules registry has rows that trace to incidents I'd rather not have had. None of it was designed upfront. All of it was earned.
The cost is speed. Every governance gate adds latency. Every phase transition adds overhead. Every three-strike stop adds friction. That's real. That's not nothing.
What it buys: no unauthorized production incidents in three months. No agent has modified a locked file without approval. No deploy has gone out without a security scan. No concurrent writes have corrupted shared state. The failures that built the governance layer haven't repeated.
That's the only way governance that actually works gets built. Not from architecture diagrams. Not from best practices documents. From watching things break, writing down exactly what broke and why, and encoding the lesson somewhere the agent can't ignore it.
Every rule has a scar. That's not a flaw in the design. That's how the design works.
Joe Gajeckyj is a founder and infrastructure engineer with 19 years in IT, currently building AI-assisted development workflows at JRGWorkshop.