Authentic Ignorance

State architecture and cost: what I expected vs. what I learned

Before AI integration, I worked through how State should divide work between ordinary software and a model, what information the model actually needed, and how I would evaluate cost. Building the product changed which constraints mattered most.

Before Architecture and cost-planning exerciseAfter Working AI-integrated productFocus Complexity, model choice, latency & cost
Before I built it

I deliberately avoided pricing an architecture I hadn't chosen yet.

The original exercise started with the product, not model pricing. I expected State to keep its data and rules in the application, and use AI only where interpretation or synthesis added value. For the first build, I used SQLite rather than adding infrastructure I didn't need yet.

Original cost decisionMeasure real usage before setting limits or estimating monthly cost.

The pre-build work focused on what to measure rather than trying to predict exact costs too early.

What actually got built

The working product kept more work deterministic than I initially expected.

Storage + authorityDeterministic application layer

Evidence, Current State, Reviews, Questions, History, versions, and authority checks live outside the model. AI proposes interpretations; software and people control consequential transitions.

InterpretationClaude Haiku 4.5

I settled on Anthropic Claude Haiku 4.5 after latency became a practical constraint. I kept the model behind an application interface so the product wasn't tied to one provider.

AskDeterministic context + generation

State prepares bounded project context in software, then uses the model for synthesis. The model does not get every record simply because it exists.

HostingSimple learning-build stack

The build used a FastAPI backend and SQLite initially, with the frontend deployed separately. The goal was enough real infrastructure to test the product thesis without pretending it was an enterprise architecture.

What changed

The biggest practical constraint was not the one I could model most neatly on a spreadsheet.

BeforeCost looked like the obvious quantity to model.

I planned to track calls, input/output tokens, model choice, retries, and non-model infrastructure, then stress-test low/base/high usage rather than optimize a single estimate.

AfterLatency became painfully visible.

An early Ask could take roughly 35–38 seconds, and instrumentation showed provider inference dominated the request. That changed the work from abstract architecture planning into model, context, output-budget, and interaction-design tradeoffs.

Model choiceCheaper/smaller did not automatically mean worse.

Haiku 4.5 remained the better fit for this workload. An OpenAI GPT-4.1 mini experiment was slower in the tested Ask path, so I did not treat provider or model labels as a substitute for measuring the actual workflow.

Context + outputLess can be a product improvement.

I reduced unnecessary context and tightened output limits. During hardening, I lowered the interpretation output cap and gave Ask only the context it needed, improving latency while reducing token use.

Estimate vs. reality

Some assumptions held. Others became more nuanced once the product existed.

Keep the architecture simpleHeld up. SQLite and a straightforward application layer were enough to learn the important product lessons before introducing heavier infrastructure.
Use AI selectivelyHeld up. Deterministic rules became more important after AI was added, not less.
Measure model usage before claiming costStill the right call. I do not have enough representative production usage to present an observed monthly State cost as if it were real operating data.
Cost would be the main optimization pressureChanged. In the working product, latency and reliability were more immediately damaging to the user experience, while many of the same fixes also reduce token use and unnecessary calls.
What I learned

Architecture, cost, latency, and reliability are one product decision, not four separate exercises.

The biggest shift was moving from “What will an LLM call cost?” to “Which parts of this workflow deserve an LLM call at all, what context does it need, how reliable must it be, and what will the user experience while it runs?” That framing led to a simpler system and made the cost model more useful.

Read the State case →Testing & debugging State →