This article was written by Prisha Verma. This article discusses: AI systems are no longer limited to carrying out simple instructions. Some systems can respond to changing situations, generate new outputs, and make decisions with limited human involvement. AI is already being used in areas such as healthcare, finance, employment, policing, and transportation.
Introduction
“Please rise. The court is now in session in the matter of The State v. ChatGPT 9000.”
It sounds like a scene from a science-fiction film: the prosecutor puts the AI’s code on a screen and argues that the system offended; the defence lawyer talks about algorithmic bias, and then the judge asks the AI whether it intended to cause the harm. That case does not exist, at least not yet. But the legal problem behind the imaginary courtroom is becoming harder to ignore. AI systems are no longer limited to carrying out simple instructions. Some systems can respond to changing situations, generate new outputs, and make decisions with limited human involvement. AI is already being used in areas such as healthcare, finance, employment, policing, and transportation. That raises a fairly simple question with a surprisingly difficult answer: if an AI system causes serious harm, who is responsible?
Take a self-driving car: if it causes a fatal accident, the answer may seem obvious at first. Perhaps the manufacturer is responsible. Or the software developer. Perhaps the owner or operator. But what if the system behaved in a way that none of them expected? What if everyone involved had followed the applicable rules and the harmful result could not reasonably have been predicted? This is where the criminal-law problem begins. It is one thing to say that an AI system caused something to happen. It is another to say that the AI itself committed a crime.
The machine is not the Defendant
At present, AI systems are not generally treated as independent criminal defendants. Criminal law continues to work mainly through people and legally recognised organisations.
If a self-driving vehicle kills someone, for example, investigators would not simply look at the decision made by the vehicle’s software and stop there.
They would examine how the vehicle was designed and tested, whether known risks were addressed, what warnings were given, how the vehicle was deployed, and whether the people responsible for it had complied with their legal duties.
The same would apply to an AI system used in another industry.
The system’s output could be extremely important evidence.
It might help investigators understand what happened or establish that a particular person or company failed to take reasonable precautions.
But the fact that a machine produced the harmful result does not, by itself, make the machine a criminal defendant.
Criminal law generally needs a legal basis for attributing conduct to someone who can be held responsible.
For many offences, it also requires proof of a particular mental state, such as intention, knowledge or recklessness. That is where AI starts to create problems for the traditional model.
AI is not just another knife
There is a familiar way of explaining the relationship between technology and criminal responsibility: a knife does not commit murder; the person using the knife does. The analogy works reasonably well for an ordinary tool. It becomes less convincing when applied to advanced AI. A knife does not learn from data; it does not change its behaviour after interacting with its surroundings; it does not generate several possible responses and select one; it certainly does not produce an unexpected answer because of patterns it developed during training. Some AI systems can do versions of these things, but that does not mean that they should immediately be treated as legal persons. It does, however, make responsibility harder to trace.
Suppose an AI system produces a serious harmful result. The developer did not specifically anticipate it. The operator did not intend it. The manufacturer could not reasonably have foreseen it. Who is responsible? If there is a clear human error somewhere in the chain, existing law may provide an answer. But if there is no such error, the situation becomes much less straightforward. This is often described as an accountability gap. The problem is not necessarily that no law exists. The problem is that the existing law may not have an obvious person to whom responsibility can be attached.
If Corporations can be Legal Persons, why not AI?
One argument sometimes made in favour of AI legal personhood starts with corporations. The law already recognises that a legal person does not have to be a biological human being. Corporations, for example, can own property, enter contracts, sue and be sued, and in some circumstances face criminal liability.
In the United States, the Supreme Court recognised corporate criminal liability more than a century ago in New York Central & Hudson River Railroad Co. v. United States.
The case is important because it showed that criminal law could, in certain circumstances, attribute the conduct and intentions of human agents to a corporation.
At first glance, that might seem to provide a possible model for AI.
There is, however, a major difference: a corporation is an organisation created through law. It has human members and agents, a defined structure, and legally recognised property and obligations.
Its legal personality exists because the law has deliberately created that framework. An AI system does not automatically have any of those things.
It does not ordinarily own property in its own name. It does not have an independent legal estate from which a judgment could be recovered. More importantly, it is not currently recognised by criminal law as an entity capable of being blamed for an offence.
So, even if AI were given some form of legal personality in the future, that would not automatically mean that it could be criminally liable. The two questions are separate. The law might give an AI system limited legal rights for one purpose without giving it criminal responsibility for another. The real issue would be what kind of legal status, if any, should be created and what consequences should follow from it.
Mens Rea without a mind?
Criminal law has another problem with AI, and this one is more fundamental. For many offences, it is not enough to prove that something harmful happened. The prosecution must also establish a particular mental state. Depending on the offence, that might be intention, knowledge, recklessness or negligence. This is the familiar idea of mens rea, but can an AI have mens rea?
Suppose an algorithm produces a discriminatory result. Was the algorithm itself prejudiced? Suppose an autonomous system chooses a dangerous course of action. Was it reckless? Or suppose a generative AI system produces false information. Can the system be said to know that the information is false? These questions sound strange because we normally use concepts such as intention, knowledge and recklessness to describe human mental states.
An AI system can certainly produce behaviour that looks intentional. It can even generate a sentence saying that it “intended” to do something. But that does not necessarily mean that it had an intention in the legal sense. There is an important difference between producing language that describes a mental state and actually possessing the mental state that criminal law requires. That makes mens rea one of the strongest difficulties facing the idea of AI criminal liability.
Looking at the people behind the system
If treating the AI itself as the defendant creates problems, one alternative is to look more closely at the people responsible for the system. Instead of asking what the AI intended, the law could ask who designed, trained, tested, deployed and monitored it. Was the system properly tested before it was released? Were foreseeable risks identified? Were users warned about its limitations? And if someone knew about a serious defect, did they do anything to fix it?
These questions are particularly important because modern AI is rarely the work of one person. There may be a developer, a model provider, a company that fine-tunes the system, a manufacturer, a deployer, an operator and an end user. Responsibility can therefore be spread across several stages. Looking at that chain may be more useful than trying to decide whether the machine itself had a criminal state of mind.
India: Accountability without AI personhood
India is also developing its approach to responsible AI. NITI Aayog’s Responsible AI for All framework places considerable importance on accountability. It recognises that responsibility can become difficult to assign when AI systems involve many different actors and stages of decision-making. The framework does not make AI a legal person and does not create a general system of criminal liability for AI. It is a policy framework rather than a criminal statute.
Its importance is that it focuses attention on the people and organisations involved in designing, developing and deploying AI. That approach is particularly relevant to the accountability problem. Instead of asking whether the AI should be punished, it asks how responsibility should be divided among the actors who created and used the system.
The real accountability gap
The most difficult situation may be one where nobody obviously did anything wrong. Imagine that the developer followed accepted practices. The manufacturer complied with the applicable rules. The company deploying the system followed the instructions. The user did not misuse it. Nobody intended the result. Yet the AI produces a serious and completely unexpected outcome. Who should be responsible? This is where the debate becomes much more difficult.
Some scholars have considered whether AI systems might eventually need to bear some form of direct liability in cases where responsibility cannot be traced to a human actor. Others argue that this would require major changes to criminal law and that existing rules should instead be adapted to deal with increasingly autonomous systems. There is no simple answer. The difficulty lies in deciding where responsibility should fall when an outcome has emerged from a system involving many different people, decisions, and technical processes.
What would punishing AI actually mean?
There is also a practical problem that is sometimes overlooked. Suppose the law did decide that a sufficiently autonomous AI system could be criminally responsible. What would its punishment look like? You cannot put software in a prison cell. Perhaps the system could be disconnected from the internet. Perhaps its operation could be restricted. A fine might be possible if it had legally controlled assets. “Probation” might mean limiting where or how the system could be deployed. A system could also be shut down, retrained, modified or permanently disabled. But these measures sound more like ways of controlling or preventing future harm than traditional criminal punishment.
Conclusion
The question of AI punishment ultimately comes down to a simpler issue: who should be responsible when an AI system causes harm? Under current law, AI is not generally treated as an independent criminal defendant. Responsibility remains with the people and organisations involved in developing, deploying or using the system. But as AI becomes more autonomous, there may be cases where no single person clearly intended or caused the harmful result.
The law may respond by strengthening human oversight, creating clearer duties for developers and deployers, or developing new rules for sharing responsibility across the AI lifecycle. Giving AI legal personhood is another possibility, but it would raise a further question: what would it actually mean to punish a machine?
For now, an AI defendant remains a fictional idea. The accountability problem, however, is already real. The challenge for criminal law will be finding a fair way to assign responsibility when machines can produce consequences that no single human being intended or predicted.
