A lot of AI knowledge work starts the same way.
Give a model a pile of documents. Write a careful prompt. Ask for an analysis.
The result can be impressive.
Then try to use the same approach for work that has to survive contact with another analyst, another dataset, another method, or another month of accumulated state.
The prompt gets longer. The agent gets more tools. The workflow gets more branches. The reasoning becomes harder to inspect. A useful prototype turns into a system that nobody is quite sure how to trust or extend.
I do not think the main problem is prompting.
I think difficult knowledge work is an architecture problem.
A model call is not the work
Consider a causal research task.
A model can read documents and produce a plausible explanation of why an event happened. That is useful. It is also only one part of the job.
A serious analysis may need to preserve the exact source passage behind each claim. It may need to distinguish evidence for one explanation from evidence for a rival. It may need to show where an expected piece of evidence was absent. It may need to let another person move from a conclusion back to the underlying record.
Those requirements are not solved by a better paragraph in the prompt.
They require state and data structures.
They require an evidence model.
They require method-specific operations.
They require a way to preserve what happened across the run.
That is why one of the research systems I built treats causal analysis as a source-grounded workflow rather than a text-generation task. The system keeps exact evidence anchors, rival explanations, mechanism traces, and explicit claim limits. The model still does important semantic work. The architecture carries the parts that need to remain inspectable.
Public Process Tracing workbench
The same pattern keeps appearing in other domains.
Let models interpret. Let software enforce.
Natural language is flexible. Software behavior is not.
That creates a recurring design problem for AI systems.
A model is good at understanding a requirement such as "a manager must approve requests above this threshold unless the customer is in an exempt category." It can connect language to concepts that were not written with a database schema in mind.
But once the rule is understood, I usually do not want the model improvising the rule again every time the system executes it.
I want the meaning to become explicit.
I want roles, entities, references, constraints, and lifecycle rules represented in a form that can be validated. I want reusable capabilities resolved against that representation. I want deterministic code to own transitions that need exact behavior.
That is the idea behind a semantic-compiler prototype I have been working on.
The model does interpretation. A target-neutral intermediate representation preserves the application semantics. Typed validation checks the structure. Reusable capabilities are resolved. Deterministic runtimes execute policy and state transitions. Platform-specific adapters come later.
The architecture is not automatically better.
In one same-session exploratory benchmark, the path-specific structural scores were 42/48 for semantic output and 43/48 for direct output. Because the paths produce different artifacts, those scores are not an apples-to-apples win/loss metric. The semantic path also missed a frozen paraphrase criterion.
On a behavioral comparison over gold-compiled cases, the semantic path was correct on 39/39 cases while direct generation was correct on 34/39.
That is mixed evidence.
I find it more useful than a clean marketing result.
It suggests that the question is not "does adding semantics beat prompting?"
The more interesting questions are whether explicit semantics improve reuse, behavioral stability, portability, inspection, persistent state, and composition as systems become larger.
Public semantic-compiler case study
Methods should be first-class parts of the system
Knowledge work is not one generic transformation from documents to answer.
Qualitative coding is not process tracing. Process tracing is not retrieval. Retrieval is not statistical estimation. Simulation is not literature synthesis.
Each method has different inputs, assumptions, failure modes, and output claims.
An agent can call all of them. That does not make them interchangeable.
This is where I think a lot of "agentic workflow" designs become too flat.
Everything becomes a tool call.
The tool boundary says what function can be invoked. It often says much less about what the result means.
For analytical systems, meaning matters at the boundary.
A capability should expose enough structure for the system to know what it accepts, what it produces, what assumptions apply, and how the output can be composed with the next step.
That does not require an ontology for every script.
It does require more than a function name and a natural-language description once the workflow becomes persistent or consequential.
State changes the problem
Many AI demos are stateless.
A user asks a question. The system answers. The run ends.
Knowledge work rarely behaves that way.
Evidence gets added. A coding scheme changes. A theory is revised. A source is superseded. An analyst changes the scope. A previous finding becomes an input to the next analysis.
The system needs to know what changed and what should be recomputed.
This is one reason I keep coming back to provenance, truth maintenance, temporal state, and explicit intermediate representations.
The value is not bureaucracy.
The value is that yesterday's work can become a reliable input to tomorrow's work.
A system that cannot distinguish current state from stale derived output does not compound very well.
Evaluation has to test the behavior you care about
Architecture changes what should be measured.
If the goal is one-shot answer quality, compare answers.
If the goal is stable execution, perturb the wording and inspect behavior.
If the goal is reuse, move the same capability into another domain.
If the goal is regulation, disturb the system after it reaches a good state and see what happens.
That last point comes from a separate line of experiments I have been running on collective competence.
In one simple system, local neighbor rules can produce global sorted order even though no component represents the global target. If the controller stops after success, a later perturbation is not repaired. If feedback remains active, the system steers back toward the criterion.
Reaching a good state and regulating toward it are different capabilities.
You do not see that distinction by staring at one successful trajectory.
You see it by designing the right intervention.
Public collective-competence research overview
The same principle applies to AI systems.
A benchmark should attack the property the architecture claims to provide.
The architecture I keep converging on
I do not have one universal pipeline for knowledge work.
I do keep seeing the same categories of concern:
Interpretation — models are useful where language, ambiguity, and semantic judgment dominate.
Evidence and state — source identity, provenance, persistent state, and transformation history need durable representation.
Methods and capabilities — analytical operations should have explicit contracts and claims that survive composition.
Deterministic execution — rules, validation, transitions, and exact calculations should move out of probabilistic inference when they can.
Evaluation and revision — the system should preserve enough structure to test what happened, revise it, and reuse the result.
Sometimes the right implementation is a direct model call.
Sometimes it is a small script.
Sometimes it is a retrieval system.
Sometimes it is a graph, a typed intermediate representation, a simulator, or a state machine.
The point is not to maximize architecture.
The point is to put structure where the task needs persistence, composition, inspection, or control.
The standard I care about
The long-term objective is not to make every analysis more elaborate.
It is to make analytical work compound.
Every analysis should make the next analysis easier.
That means evidence should remain usable. Methods should be reusable. Intermediate structures should survive beyond one prompt. Failed evaluations should narrow the claim instead of disappearing. New projects should inherit capabilities from old ones instead of rebuilding the same stack.
Large language models make more of the semantic work automatable than it was a few years ago.
That changes the feasible architecture.
It does not eliminate the need for one.
If anything, more capable models make the boundary question more important: which parts should remain flexible and learned, and which parts should become explicit, typed, persistent, or deterministic?
That is the engineering problem I am interested in.
I’m currently exploring Research Engineer / Applied AI / AI Systems roles around research automation, analytical and decision systems, retrieval and evaluation, semantic infrastructure, and agent systems.
Portfolio: https://brianmills.dev/portfolio/