An AI agent can take an action in seconds. Accountability takes longer to work out. That gap is getting harder to ignore as agents move from answering questions to operating inside business systems. An agent may send a message, change a record, call an API, modify code or trigger another system. If that action causes a problem, saying “the AI did it” doesn’t answer the questions that follow: who authorised the agent, who decided what it could access, who approved its level of autonomy, who was supposed to intervene, and who owns the outcome?
These are becoming practical enterprise questions rather than hypothetical governance debates. In the United States, Senators Josh Hawley and Chris Murphy introduced the AI Agent Accountability Act on 1 October. The proposal is specifically concerned with AI agents involved in computer hacking. It would create civil and criminal liability for operators in certain circumstances, and for developers who fail to implement reasonable safeguards despite knowing, or having reason to know, about hacking capabilities. It is proposed legislation, not current law.
Its scope is narrow, but the underlying question is much broader. Once an organisation gives an AI system the ability to act, responsibility cannot disappear into the automation.
The agent performs the action; people set the conditions
It helps to separate two things. The agent may perform the action, but the organisation still decides the conditions under which that action is permitted.
Take an ordinary workflow. An AI agent reviews a customer request and decides a refund is appropriate. It retrieves the account, checks the transaction and submits the refund automatically. If the refund is wrong, where does accountability sit?
The model generated the recommendation. But somebody chose the model. Somebody connected it to the customer system. Somebody decided what information it could retrieve. Somebody set the refund threshold, and somebody decided that transactions below that threshold would not need human approval.
The final action was automated.
The authority behind it was not.
That’s why “the AI made the decision” is usually a poor description of what happened. The system operated inside a set of choices made by people, and governance needs to make those choices visible.
Start with who owns the use case
Before discussing models, prompts or agent frameworks, I’d want one fairly simple question answered: who owns this use case? Not who developed the agent, and not who administers the platform, but who in the business is accountable for the process the agent is being allowed to perform.
Technical ownership and business accountability are not the same thing. An IT team can make sure an agent authenticates correctly, receives appropriate permissions and produces logs. It cannot decide whether a R20,000 customer credit should require approval. Security can define access boundaries, but it cannot decide whether the commercial consequence of an action is acceptable.
Those decisions need an owner who understands the business process and is willing to accept responsibility for how automation is used within it. Without that, accountability becomes something everyone assumes belongs to someone else.
Human involvement has to be meaningful
Recent workplace AI legislation in California (https://www.gov.ca.gov/2026/09/30/californias-nation-leading-ai-framework-just-got-stronger-governor-newsom-signs-more-first-in-the-nation-worker-protections-and-more/) reinforces a useful principle: for certain consequential employment decisions, human involvement cannot simply disappear because AI is part of the process.
The enterprise lesson is broader. Human oversight should follow consequence rather than being applied mechanically to every action. An agent finding an internal document is different from deleting it. Drafting a customer response is different from sending it. Preparing an access change is different from granting the permission. Analysing an employee’s performance information is different from deciding that person’s employment outcome. The boundary should be deliberate.
Adding a person somewhere in the workflow isn’t enough on its own either. If the person is expected to approve an action without knowing what information the AI used, what it is about to do, or why the action was recommended, the approval becomes ceremonial. Human oversight only works when the person has enough information, and enough authority, to challenge the system.
Accountability needs technical evidence
Even when ownership is clear, you need evidence of what actually happened. Suppose an AI agent changes a customer account incorrectly. The final database entry isn’t enough. You may need to know which user initiated the task, which identity the agent operated under, what information it accessed, which tool it called, what permissions applied, whether an approval was required, and what action the system eventually executed. Without that record, accountability quickly becomes guesswork.
This is why auditability matters more as AI becomes agentic. It isn’t simply a compliance feature. It’s how you tell apart a bad model response, an incorrect permission, a faulty integration, an inappropriate approval threshold, and a user asking the system to do something they should never have been able to request.
Those are different failures, and they need different fixes. A log saying “agent executed task successfully” tells you very little.
Runtime control and business accountability need to connect
The architecture being built around enterprise agents is starting to reflect this. SAP and NVIDIA, for example, are integrating NVIDIA OpenShell, a secure runtime for autonomous agents, with SAP’s Business AI Platform. Click here to read the article on SAP website. Their description separates secure agent execution from business governance: the runtime constrains what the agent can technically see and do, while the business layer applies authorisation, identity, role and process context to decide whether an action should happen in the first place. Audit trails connect the two.
That distinction is useful because there are really two questions. Can the agent execute this action safely? And should this agent, acting for this user, in this business process, be allowed to execute it at all?
A technically safe API call can still be an inappropriate business action. A correctly isolated agent can still have permissions that are too broad. A perfectly functioning model can still carry out an instruction the organisation should never have authorised. Security constrains how the agent operates; governance determines what it is allowed to do.
Agents act for someone, so identity matters
Agents should not become anonymous actors inside enterprise systems. If an employee initiates a task, their identity and permissions should follow the work. If the agent is acting as a service or autonomous process, it should have its own controlled identity. Either way, the action should be attributable.
That means avoiding the convenient but dangerous pattern where several agents share one highly privileged service account because it makes integration easier. If every action appears to come from the same technical identity, it becomes difficult to tell which agent initiated it, which user requested it, and which permissions should have applied. Least privilege reduces the exposure. Clear identity establishes responsibility.
Autonomy moves accountability; it doesn’t remove it
As agents become more autonomous, there may be fewer individual human approvals. That isn’t necessarily a problem. The mistake is assuming that removing an approval step also removes human responsibility. Someone still decided the workflow had become safe and predictable enough to run autonomously, and that decision should itself be governed.
A sensible progression is to start with an agent that advises, then let it prepare actions for approval, then grant limited execution within defined boundaries. As evidence builds, the organisation can increase autonomy, and if risk changes, it should be able to reduce it again. So accountability exists at two levels: responsibility for the individual action, and responsibility for deciding how much authority the system receives in the first place.
South African organisations don’t need to wait for an AI Act
South Africa doesn’t have dedicated AI legislation, but POPIA already answers much of the accountability question. Under the Act, the responsible party is whoever determines the purpose and means of processing personal information, and section 8 makes that party accountable for compliance. Delegating the processing to an agent doesn’t move that responsibility anywhere.
The rest of POPIA follows the same logic. Section 19 requires appropriate, reasonable technical and organisational measures to secure personal information, which covers what an agent can access and what it can do with it. Sections 20 and 21 require operators processing on your behalf to work under a written agreement with security obligations, which matters when the agent runs on a vendor’s platform. And section 71 restricts decisions with legal or substantial effects on a person, such as creditworthiness or work performance, from being based solely on automated processing.
So if an AI agent can access personal information, modify a customer record, affect an employee, trigger a financial action or interact with regulated systems, accountability should already be part of the design. The question isn’t how much AI we can deploy first and how we assign responsibility afterwards. Ownership needs to be established before the system starts acting.
How RebelCore keeps authority attached to the action
This is one of the problems we’re addressing with RebelCore. Go back to the refund agent.
In RebelCore, the organisation decides who can start that workflow: a support agent, say, but not a contractor. It decides what the agent may retrieve, such as the customer’s account and transaction history, but not unrelated records. It decides which tools the agent can call, and which actions need a person: refunds under the threshold go through, anything above it waits for a named approver who can see what the agent looked at and why it recommended the refund. And every step is recorded.
If that refund turns out to be wrong, the organisation doesn’t have to guess. It can see who requested the work, what authority the agent had, which controls applied, whether a person approved the consequential step, and what the system actually did. That’s a chain of accountability, not just a log.
The point isn’t to put a human approval box in front of every action. It’s to make authority explicit enough that when an action happens, the organisation can explain why it was allowed to happen.
Before an agent acts, decide who owns the outcome
AI agents will increasingly do work that used to require a person to open an application, retrieve information and press the button themselves. That’s where much of their value comes from. But removing the person from the mechanical step doesn’t remove accountability for the result.
Before giving an agent permission to act, I’d want clear answers to five questions:
- Who owns the business process?
- Whose authority is the agent operating under?
- What exactly may it do without approval?
- Which actions still require a person to decide?
- Can we reconstruct what happened afterwards?
If those answers are unclear, the organisation isn’t really delegating work to an AI agent. It’s delegating responsibility without deciding where that responsibility goes.
And that’s a much harder problem to fix after the agent has acted.
