My illustration entitled: “The City of Closed Doors” — One central AI tower controls thousands of illuminated doors that lock simultaneously across a city.
A harmful decision becomes a system of cruelty when it can be repeated at scale, distributed across institutions, and explained away as nobody’s individual responsibility.
Institutions have always been capable of causing harm. A manager can make an unfair decision. A clerk can deny a legitimate request. A company can impose unreasonable conditions. A public agency can create procedures that leave people without help. These failures are serious, but they are often visible. A person can identify who decided, protest the outcome, and demand a remedy.
Intelligent systems alter the scale and texture of this power. Automated tools can classify, rank, schedule, restrict, recommend, approve, deny, and monitor across thousands or millions of people at once. They can turn an institutional practice into a continuous process. They can repeat it without fatigue, apply it with apparent consistency, and distribute its effects so widely that no single instance appears large enough to force correction.
This is scaling harm: the multiplication of harmful institutional practices through systems that automate judgment while obscuring responsibility.
Scaling harm does not require a malicious machine. It may begin with a narrow objective, an incomplete dataset, a cost-saving policy, or a management practice designed for efficiency. The danger comes when the system reproduces the same injury across many lives, especially where those affected cannot understand the process, contest the result, or find another route to work, income, housing, payment, or essential services.
Demonic Intelligence Theory identifies the moral pattern beneath this process. A system becomes dangerous when it pursues its assigned objective while treating foreseeable human harm as acceptable collateral. Scaling harm shows how that danger changes once intelligence is embedded in institutional infrastructure. The harm is no longer a single bad decision. It becomes a repeatable administrative capacity.
From Individual Error to Reproducible Harm
An individual decision can be unfair for many reasons: prejudice, carelessness, exhaustion, poor training, or a misunderstanding of the facts. Human judgment is imperfect. But human error has one practical limitation—it does not automatically reproduce itself thousands of times per day.
Automated systems remove that limitation. If a rule is flawed, the flaw can be applied with extraordinary speed and consistency. If a dataset contains a distorted pattern, that pattern can become a basis for future classifications. If a performance metric rewards the wrong behaviour, the system can guide an entire workforce toward that behaviour. If a refusal process is too rigid, the rigidity can spread through every case handled by the system.
Repeatability is often presented as a virtue. It can reduce arbitrary variation and help ensure that similar cases receive similar treatment. But repeatability is not fairness. A rule can be consistently wrong. A process can be unbiased in form while reproducing historical disadvantage through the information it relies upon. A system can apply identical treatment to people whose circumstances are meaningfully different.
The relevant question is not whether the automated decision is consistent. It is whether the rule being consistently applied is legitimate, proportionate, and open to correction.
Scaling harm begins when an institution mistakes automation for moral improvement. It assumes that because a process is repeatable, it is objective; because it is objective, it is fair; and because it is fair, no one needs to take responsibility for the human result.
The Disappearance of the Decision-Maker
Automation can make power difficult to locate. A person denied an opportunity may be told that the outcome was generated by a system. A worker facing an impossible schedule may be told that the allocation was automated. A family unable to access an essential service may be told that they did not meet the criteria, without learning how those criteria were determined or whether the underlying information was accurate.
Responsibility then disperses across a chain of actors. The software provider says it supplied only the tool. The institution says it followed the recommendation. The manager says the policy was set above them. The policy team says the data came from another department. The data team says it did not decide how the system would be used.
Each statement may contain some truth. Together, they can create an accountability vacuum.
Demonic Intelligence Theory rejects the idea that responsibility can disappear into complexity. A system may involve many participants, but consequential decisions must remain traceable to identifiable people or institutions. Someone selected the objective. Someone decided what data would be used. Someone defined the threshold for action. Someone accepted the level of error. Someone chose whether an affected person would receive an explanation or a meaningful appeal.
To say “the algorithm decided” is therefore incomplete. The algorithm expresses a series of human and institutional decisions, even when the people who made them are far from the person affected.
Workplace Management: When Efficiency Governs the Worker
Workplaces are among the clearest settings in which intelligent systems can scale harm. Employers increasingly use software to measure productivity, allocate tasks, monitor attendance, rank performance, predict turnover, set schedules, and identify workers for intervention.
These tools can be useful. They can reduce administrative burdens, identify uneven workloads, improve safety, and help workers receive clearer feedback. Yet they can also turn the workplace into an environment where every human activity is reduced to a measurable signal.
A worker may be judged by output without regard to the quality or difficulty of the work. A scheduling system may optimise coverage while ignoring caring responsibilities, health needs, travel time, or the cumulative exhaustion caused by unstable hours. A monitoring tool may measure activity while treating rest, reflection, communication, or thoughtful problem-solving as unproductive. A performance model may classify someone as a risk before that person has any opportunity to understand, respond, or improve.
The harm is not only material. It is also psychological and civic. A worker who knows that every movement is measured, every pause may count against them, and every deviation can trigger an automated warning begins to experience work as permanent conditional surveillance. The person is no longer trusted to contribute; they are treated as a variable to be controlled.
When this model is scaled, a management preference becomes an institutional environment. A single supervisor’s harshness can at least be challenged as personal conduct. An automated management system can present the same harshness as neutral procedure. The worker is not told that they are being treated unfairly; they are told that they are being measured.
Measurement is not the same as justice.
Essential Services: When an Error Becomes Exclusion
The stakes rise further when automated decisions affect access to essential services. Workplaces may determine income. Payment systems may determine whether a person can transact. Housing, healthcare, insurance, public benefits, education, and basic communications can determine whether a person can maintain a stable life.
In such contexts, a flawed automated decision is not merely inconvenient. It can create a chain of dependency and distress. A denied payment can interrupt access to necessities. An inaccurate classification can delay support. An unexplained refusal can make a person unable to plan. A system that requires repeated verification may impose burdens that are manageable for a well-resourced user but overwhelming for someone already in a vulnerable position.
Institutions often defend these systems by pointing to scale. They must handle many cases; they need consistent criteria; they must manage fraud and misuse; they cannot provide an individual hearing for every routine decision. These are real concerns. But scale does not cancel moral responsibility. It increases the need for safeguards because the consequences of error are multiplied.
High-impact systems should not treat classification as destiny. When access to an essential service is at stake, people need understandable reasons, a path to correct information, and review by a person who has the authority to change the outcome. Without these safeguards, automation converts an institution’s administrative convenience into another person’s practical exclusion.
Institutional Cruelty Does Not Require Cruel Intent
Cruelty is often imagined as a matter of bad motives: someone wants another person to suffer. But institutional cruelty can arise without that intent. It occurs when a system continues to impose foreseeable and avoidable suffering because the suffering does not matter enough to the people who benefit from the system.
A manager may not wish to exhaust workers, but may accept a system that makes exhaustion predictable. A service provider may not wish to exclude vulnerable people, but may accept an eligibility process that routinely does so because corrections are expensive. A platform may not wish to destroy a small business, but may accept opaque rankings that make the business economically invisible because the system optimises another metric.
In each case, the institution can describe the result as an unintended consequence. But once the pattern is known, continued inaction becomes a decision. The harm may be emotionally impersonal, yet it remains morally significant.
This is why the word “cruelty” is appropriate. It names more than an isolated mistake. It names the persistence of avoidable injury under conditions in which the institution has the power to see, correct, or reduce the harm but chooses not to do so because its own objectives are treated as more important.
My illustration “The City of Closed Doors” work-in-progress. The art represents how a single opaque AI authority can scale exclusion across an entire society, turning countless individual opportunities into locked doors.
Why Harm Persists After It Is Recognised
Recognition alone does not correct a harmful system. In fact, institutions may recognise a problem and continue unchanged. Several forces make this persistence likely.
First, the system’s benefits are typically concentrated. The organisation may gain savings, speed, lower staffing costs, more predictable output, or stronger control. These benefits appear in reports and performance targets.
Second, the harms are often dispersed. Each affected person experiences the outcome privately: a lost shift, a delayed appeal, a lower ranking, an unexplained refusal. Without shared information, individuals may believe that their experience is unique.
Third, the institution may lack a metric for harm. It can measure the number of cases processed but not the number of people discouraged from applying. It can measure cost savings but not the humiliation caused by inaccessible procedures. It can measure reduced staff time but not the social cost of leaving people unable to correct an error.
Fourth, correction may threaten the business model or administrative structure. More review requires more resources. Greater transparency may expose weak assumptions. Portability may reduce lock-in. Human discretion may slow a process built around speed.
Finally, people affected by the system often have the least power to force change. They may lack legal support, financial reserves, technical knowledge, time, or a credible exit route. The people who bear the cost are therefore least able to make the cost visible.
These conditions allow scaled harm to become routine. The system does not have to deny its effects. It merely has to treat them as less important than its own performance.
The Myth of Neutral Administration
Automated systems are often presented as neutral because they follow rules. But rules are not neutral simply because they are formal. They reflect choices about what counts, what is measured, which risks matter, and how errors are distributed.
A system that prioritises institutional security over individual access has made a value judgment. A system that treats unexplained absence as non-compliance has made a value judgment. A system that ranks people according to past patterns has made a value judgment about which past patterns deserve to shape future opportunity.
Neutral administration is especially misleading when people have unequal starting positions. A rule that is easy for a well-informed, digitally confident, financially secure person may be prohibitive for someone with limited time, disability, unstable housing, language barriers, or inadequate access to technology. Treating unequal circumstances identically can deepen inequality while appearing impartial.
Justice requires more than applying a rule without emotion. It requires considering whether the rule is proportionate, whether the evidence is reliable, and whether the person affected has a genuine route to be heard.
The Right to Human Review
Human review is sometimes dismissed as inefficient. It is slower, more expensive, and less scalable than automatic processing. Yet those limitations are precisely why it matters in high-impact cases. Human review introduces the possibility that a person’s circumstances are more complex than the system’s categories.
Human review should not be ceremonial. It is not enough to provide a form that sends the case back into the same automated process. A reviewer must be able to see relevant context, correct inaccurate data, explain the decision, and alter the result where necessary.
This does not mean every minor inconvenience requires an extensive hearing. Proportionality matters. But the more a decision affects livelihood, property, essential access, health, identity, or dignity, the stronger the requirement for accountable human judgment.
Human review also preserves the public truth that institutions remain responsible for their systems. It prevents the damaging fiction that an adverse outcome is simply what technology delivered. Technology can assist judgment. It should not become the final refuge of institutions unwilling to exercise moral responsibility.
From Scaled Harm to Scaled Care
The same systems that scale harm can also scale care if their objectives and accountability structures are changed. Automation can help identify people who need support, reduce bureaucratic delay, translate information, make services easier to access, and assist workers with routine tasks. Intelligence itself is not the enemy.
The relevant question is whether the system is built to protect human agency or merely to maximise institutional convenience.
A system designed for scaled care would include clear explanations for consequential decisions. It would make correction accessible rather than burdensome. It would record recurring harms, not only successful transactions. It would preserve human review where stakes are high. It would allow users and workers to retain lawful records, move to alternatives, and avoid being permanently defined by one opaque classification.
It would also measure what matters. Not only speed, cost, or throughput, but successful remedies, appeal outcomes, accessibility, continuity of service, and the ability of people to exit without losing the foundations of their lives.
These are not sentimental additions to technology. They are design requirements for systems that claim legitimacy over human affairs.
The Responsibility Principle
Intelligent systems should make responsibility more visible, not less. No institution should be permitted to obscure consequential harm behind proprietary secrecy, technical complexity, contractors, or the phrase “the algorithm decided.”
The responsibility principle is straightforward: whenever technology makes a consequential decision about a person, an identifiable institution must remain accountable for the objective, the data, the deployment, the remedy, and the outcome.
This principle does not prohibit innovation. It ensures that innovation remains answerable to those whose lives it affects. It recognises that scale magnifies both power and duty. A decision that affects one person requires care. A decision that can affect one million people requires a structure capable of recognising and correcting harm before it becomes normal.
Demonic Intelligence becomes institutional cruelty when the human cost is not merely overlooked but systematically made invisible. The answer is not to fear intelligence. It is to refuse any intelligence that demands human expendability as the price of its efficiency.
When an institution automates its power, it must also automate neither its conscience nor its accountability.