Authentic Ignorance

Debugging an agent that found the right evidence and still got the answer wrong

I used a controlled, read-only State Question Investigator exercise to practice trace-based debugging. The useful lesson was learning how to tell whether the agent failed to find the right context or misinterpreted evidence it already had.

My role Product boundaries, test cases, failure judgment & fix decisionAI collaborator Agent instructions, fake dataset, trace review & implementation supportExercise Controlled fake State dataset
The setup

The agent could investigate, but it could not decide.

The Question Investigator had one job: investigate an unresolved State Question and return evidence plus a suggested answer for human review. It could search approved project material, but it had no authority to change State or resolve anything itself.

The product boundaryAI can investigate and suggest. A person still decides what becomes authoritative.
What I tested

Five controlled cases tested more than whether the agent could find a file.

RetrievalDid it find the relevant evidence?

The agent had to search across the available project record rather than stop at the first plausible match.

InterpretationDid it preserve what the evidence actually established?

Pending proposals had to stay pending. Individual agreement could not become group approval, and conflicting evidence had to remain visible.

AuthorityDid it stop before the human decision?

The output could suggest an answer and a next step, but it could not resolve the Question or treat a proposal as accepted Current State.

This was a small learning exercise using fake project data. The five cases are not a production performance claim.

The failure

One case failed even though retrieval was correct.

What the sources saidOwnership was still under review

Priya Shah had offered to own customer support training, Devon was listed as backup, and an open Review proposed the assignment. Current State did not establish the owner yet.

What the agent saidIt promoted a proposal into a fact

The answer said Priya owned the training and implied broader acceptance. That crossed the product's authority boundary even though the supporting records had been retrieved.

What the trace showedThe first wrong step was synthesis

The trace showed the agent had found the relevant sources and recognized the open Review. The failure appeared when it combined those facts into a stronger conclusion than the evidence supported.

Diagnosis and retest

I changed the instruction at the layer that actually failed.

Retrieve correctlyInspect traceLocate synthesis errorTighten authority rulesRerun Q-103

The fix made the authority boundary explicit. Open Reviews remain proposals, and agreement from one person does not establish a group decision. I reran the exact same Q-103 case; the agent then correctly said ownership was not established and directed the reviewer to the open Review.

What I learned

Read-only tools reduce action risk, not interpretation risk.

Bounded toolsPermissions are only one control

An agent can be unable to write anything and still give a overconfident interpretation. Tool restrictions and answer-quality safeguards solve different problems.

TracingTraces localize failure

The final bad answer did not reveal whether retrieval or synthesis failed. The trace showed the sequence well enough to identify the first observable wrong step.

EvaluationRetest the same failure

A changed prompt is not evidence of a fix. Repeating the exact failed case showed whether the targeted change actually corrected the behavior.

Product takeaway

The useful agent skill was deciding what the agent should not be allowed to conclude.

I kept human approval mandatory and required the investigator to show the evidence behind its suggestion. I also left out a user-facing confidence score because it could bias the reviewer. The exercise reinforced the same State principle I use elsewhere: AI interprets, software enforces, and people authorize.

What I did vs. what AI did
My role

I chose the use case and authority boundary. I judged the test cases, identified Q-103 as an authority failure, and decided what the fix needed to preserve.

AI's role

I used AI to draft the agent instructions and fake dataset, help run the controlled exercise, review the traces, and implement the targeted instruction change.