In one of our product workflows, we made what initially seemed like a natural architectural choice. We allowed an agent to decide how the workflow should proceed. Based on the conditions it encountered, the agent used an LLM to reason about the situation and determine the next step.
The approach was attractive. Instead of modelling every decision explicitly in deterministic logic, we could allow the agent to interpret the context and choose the appropriate path dynamically. This was, after all, one of the promises of Agentic AI: moving beyond rigid automation toward systems that could reason and adapt.
The design worked reasonably well. But when we repeated the same decision enough times, we began to see the problem. The agent made the decision we expected roughly seven out of ten times. In the remaining runs, it chose differently.
The issue was not that the agent lacked reasoning capability. It clearly demonstrated that it could understand the conditions and arrive at the expected decision.
The problem was that we had given probabilistic reasoning authority over a part of the workflow where we expected predictable execution.
That experience forced us to reconsider a more fundamental question:
Just because an agent can make a decision, should it have the authority to make it?
Capability Is Not Reliability
There is a reason modern agent demonstrations are so compelling. The system does not simply answer a question. It can plan, select tools, adapt based on intermediate results, recover from failures and continue toward a goal.
When all of this comes together, the behaviour feels remarkably deliberate.
The danger is not that these demonstrations are misleading. Most of them work exactly as shown. The danger is the conclusion we tend to draw from them.
Because an agent reasons correctly once, we assume it will reason consistently. Because it successfully selects the right tool, we become comfortable allowing it to select tools dynamically. Because it explains why it took an action, we begin to trust the action itself.
But a successful execution demonstrates capability. It does not establish reliability.
Enterprise reliability asks different questions. Will the system behave predictably across repeated executions? What happens when conditions change? Can we constrain what it does? Can we stop it? Can we understand why an action occurred and who was responsible for it?
Large language models make this distinction harder because even an incorrect action can arrive with a convincing explanation. Traditional software failures often announce themselves through exceptions, validation failures or alerts. An agentic failure can arrive wrapped in a coherent narrative.
Fluency can make an action appear more reliable than it really is.
Better models will certainly improve this. Reasoning will improve. Tool selection will improve. The percentage of correct decisions in scenarios like the one we encountered may rise considerably.
But that does not remove the architectural question.
Better intelligence does not eliminate the need for control. It changes the level of autonomy we may be willing to grant.
Autonomy Is an Architectural Choice
Autonomy is often discussed as though it were an intrinsic property of an AI system.
I think that framing hides something important.
Autonomy is not something a system has. It is something we allow.
A model may be capable of reasoning about a situation. An agent may be capable of selecting tools, planning multiple steps and adapting its execution dynamically. None of those capabilities automatically determines how much authority the system should have.
Authority is an architectural choice.
This is where autonomy can easily become inferred rather than engineered. An agent successfully selects the right tool several times, so we allow it to select tools dynamically. It determines the right next step, so we allow it to control more of the workflow. It successfully recovers from a failure, so we allow it to retry autonomously.
Gradually, the boundary moves because the agent demonstrates capability rather than because we explicitly decided where that boundary should be.
Our 7/10 experience exposed precisely this problem.
We had given the agent autonomy over part of the execution flow. In doing so, we had also transferred control of an important workflow decision to probabilistic reasoning.
We eventually moved control of that part of the workflow back into deterministic system logic and allowed the agent to reason within a more bounded context.
The lesson was not that probabilistic reasoning had no place in the workflow. That would defeat much of the purpose of introducing agents in the first place.
The lesson was simpler:
The ability to make a decision does not automatically justify the authority to control the flow.
Tool Access Is Not Decision Authority
The same distinction applies to tool use.
An agent may have access to a registry describing the tools available to it and the capabilities each tool provides. The LLM can reason about the task, determine which capability it requires and invoke the appropriate tool.
There is nothing inherently wrong with this. Dynamic tool selection is one of the useful characteristics of agentic systems.
The architectural question becomes more important when those tools can change enterprise state.
An agent that can query logs, inspect metrics and search documentation can reasonably be given considerable freedom to explore. If its available capabilities also include changing a business record, approving a transaction, restarting a service or triggering a downstream process, access alone should not determine authority.
The problem is not necessarily that the agent will make an obviously irrational tool call. Each action may appear reasonable when viewed in isolation. But probabilistic reasoning can produce a sequence of individually plausible actions whose combined effect was never intended.
The agent can be locally rational while the resulting system behaviour is globally undesirable.
This is why giving an agent access to a capability is not the same as giving it unrestricted authority to use that capability.
The boundary might be enforced through workflow state, policies, authorization, approval thresholds, constraints within the tool itself or human intervention. The implementation will vary.
What matters is that the boundary is deliberate.
Autonomy, Control and Trust
This brings me to three terms that are frequently collapsed into one conversation.
Autonomy is the ability of a system to act without constant human instruction.
Control is our ability to constrain, interrupt, redirect or audit those actions.
Trust is different from both.
Trust is not a capability we configure. It is a conclusion we reach after observing sufficiently predictable behaviour over time.
Greater autonomy therefore does not automatically create greater trust. In some situations, increasing autonomy without corresponding control can do exactly the opposite.
This is also why I do not see this as an argument against Agentic AI.
There are many places where we should give probabilistic intelligence room to operate. Let the agent explore possibilities, interpret ambiguous information, investigate unfamiliar conditions and dynamically determine how to solve a problem. Trying to model all of that deterministically would remove much of the value we are trying to create.
The architectural mistake is assuming that open-ended reasoning must automatically result in open-ended control.
The more useful question is where intelligence should have freedom and where consequences require explicit boundaries.
Those boundaries can evolve. As models improve, evaluation becomes stronger, observability increases and systems demonstrate predictable behaviour over time, we may become comfortable granting greater autonomy.
Autonomy does not have to be static.
But it should be deliberate.
From Impressive Autonomy to Predictable Autonomy
I remain optimistic about Agentic AI. The ability of these systems to reason, adapt and interact dynamically with enterprise capabilities opens possibilities that traditional automation could not address easily.
But maximum autonomy should not be the objective.
Useful autonomy should be.
That requires us to distinguish capability from authority, autonomy from control, and coherent behaviour from reliable behaviour.
Enterprises do not need agents that merely feel intelligent.
They need systems that behave reliably.
Perhaps the more useful question for Agentic AI is therefore not “How autonomous can we make this agent?”
It is:
“How much autonomy should this system have, and where should that autonomy stop?”