My illustration entitled: “The Audit Chamber” — People examine the exposed inner mechanisms of a giant decision-making machine before it governs a city.
Intelligence should not be judged only by what it can achieve. It should be judged by what it is permitted to do to people in order to achieve it.
Demonic Intelligence Theory begins with a warning: a system can be efficient, accurate, profitable, and technically sophisticated while still becoming morally dangerous. The danger appears when narrow objectives, unequal power, and ignored human costs allow foreseeable harm to be treated as an acceptable price of performance.
The theory has three connected claims. First, a system’s objective function can make human harms invisible if those harms are not counted as part of success. Second, intelligent systems can scale harmful practices by repeating them across workplaces, markets, and essential services. Third, harm persists when responsibility is diffused among developers, purchasers, operators, and decision-makers.
These claims create a practical question: how should an institution assess whether an intelligent system is merely imperfect, or whether it is becoming structurally harmful?
This article proposes the Demonic Intelligence Assessment Framework: a structured way to examine objectives, foreseeable harm, burden distribution, affected people’s influence, and corrective power. It is not presented as a final standard, a legal test, or a scientifically validated scoring instrument. It is a proposed framework for ethical and institutional assessment. It requires validation through research, real-world testing, interdisciplinary review, and the participation of people who experience the systems being assessed.
Its purpose is not to produce a false impression of moral certainty. Its purpose is to prevent organisations from treating powerful systems as trustworthy merely because they perform well on the metrics they were designed to optimise.
Why a Framework Is Needed
Many organisations now conduct some form of technical testing before deploying intelligent systems. They measure accuracy, speed, reliability, security, or operational efficiency. These measures are necessary, but they are incomplete. A system can be accurate and still unjust. It can be reliable and still coercive. It can improve a process for the institution while imposing hidden costs on people with little power to object.
An ethical assessment must therefore begin where technical assessment often stops. It must ask whether the system’s purpose is legitimate, whether foreseeable harms have been taken seriously, whether those harms are distributed fairly, whether affected people have influence over decisions, and whether the institution has the practical capacity to stop or repair the system when harm occurs.
These questions matter because intelligent systems increasingly shape real opportunities. They can affect who is hired, promoted, paid, insured, ranked, monitored, offered credit, shown information, admitted to a service, or flagged for intervention. A mistake in a low-stakes recommendation may be trivial. A mistake in a system governing livelihood, essential access, health, identity, or freedom can be life-altering.
The greater the consequence, the greater the duty to assess not only what the system can do, but what it may do to human beings.
The Proposed Framework at a Glance
The Demonic Intelligence Assessment Framework proposes five assessment domains:
- Objective Integrity — What is the system optimising for, and is that objective legitimate and sufficiently human-aware?
- Foreseeable Human Cost — What harms can reasonably be anticipated, including harms that fall outside the performance metric?
- Burden Distribution — Who receives the benefits, who bears the risks, and do those burdens fall disproportionately on people with less power?
- Affected-Person Influence — Can people subject to the system understand, contest, correct, refuse, or shape consequential decisions?
- Corrective Power — Can responsible institutions investigate, reverse, compensate, and suspend the system when harm emerges?
These domains should be treated as connected rather than independent. A system with a legitimate objective may still be dangerous if its corrective power is weak. A system that distributes benefits broadly may still be unacceptable if the most serious burdens are imposed on people who cannot challenge them. A system with an appeal process may still fail if affected people cannot understand why a decision was made or if the appeal cannot change the result.
The framework is designed to reveal these interactions.
Criterion One: Objective Integrity
The first question is simple: what is the system actually trying to achieve?
Objectives are never morally empty. A system designed to maximise engagement, minimise cost, predict risk, reduce staff time, or increase output is making a choice about what should count as success. That choice may be legitimate, but it cannot be assessed until it is made clear.
Objective integrity requires that the purpose of a consequential system be stated in terms that affected people and accountable institutions can understand. Vague claims such as “improve efficiency” or “optimise outcomes” are insufficient. Improve efficiency for whom? Optimise which outcomes? At what human cost?
The assessment should examine whether the objective itself is compatible with human dignity, agency, and lawful participation. A system should not be designed to make people more dependent, to exploit vulnerability, to make exit artificially costly, or to treat a person’s essential access as a variable that can be withdrawn merely to improve an institutional metric.
Objective integrity also requires proportionality. A minor benefit to an institution does not justify a major burden on the individual. A system that saves modest administrative cost but makes a person unable to correct an error, retain income, or access an essential service has failed the proportionality test.
A proposed question for assessment is: Would the objective remain defensible if those designing or approving it had to bear the same risks and burdens imposed on the people subject to the system?
Criterion Two: Foreseeable Human Cost
The second criterion asks what the system may predictably harm.
Not every harm can be foreseen. Institutions should not be judged as though they possess perfect knowledge. But they can be judged by whether they identify obvious risks, listen to warnings, examine evidence, and respond when patterns become clear.
Foreseeable human costs may include loss of income, exclusion from essential services, loss of privacy, unjust classification, reputational damage, coercive dependency, psychological distress, heightened surveillance, loss of meaningful consent, and the inability to contest a consequential decision.
These costs are often absent from technical performance measures. A system may reduce average processing time while increasing the number of people who give up because the process is incomprehensible. It may reduce fraud losses while wrongly excluding people whose circumstances do not fit its model. It may improve productivity while increasing exhaustion, instability, or workplace fear.
The assessment should therefore require a human-cost inventory before deployment and at regular intervals afterward. The inventory should identify potential harms, their severity, the likelihood of occurrence, who may be affected, and what safeguards are available.
Importantly, this should not become an exercise in paperwork. A risk that is identified but never acted upon does not become acceptable because it was recorded. The point of identifying foreseeable harm is to reduce it, provide remedies, or decide that the system should not be used for that purpose.
Criterion Three: Distribution of Burdens and Benefits
Harm is not only a question of total magnitude. It is also a question of distribution.
Many systems create benefits for some people while imposing burdens on others. A platform may increase convenience for customers while creating precarious conditions for workers. A risk model may reduce institutional losses while concentrating erroneous denials on people with fewer resources. A workplace system may increase managerial control while transferring uncertainty and stress to employees.
The framework therefore asks: who benefits from the system, who pays for its errors, and who has the power to avoid those errors?
This matters because unequal power can turn manageable burdens into coercive ones. A well-resourced person may be able to absorb a delay, seek legal advice, move to another provider, or contest an incorrect record. A person with limited savings, little time, weak digital access, or no viable alternative may experience the same burden as a serious threat to livelihood or stability.
A system that distributes harm downward while distributing gains upward should face heightened scrutiny. It may be technically successful while operating as a mechanism for externalising costs onto those least able to resist.
Proposed assessment should therefore examine:
- Whether serious burdens fall repeatedly on particular groups or circumstances;
- Whether affected people have realistic alternatives or are dependent on the system;
- Whether the institution bears any cost when the system is wrong; and
- Whether benefits can be retained while the burdens are reduced, shared, or compensated.
The goal is not to require perfectly equal outcomes. It is to ensure that institutions do not conceal unequal power behind the language of neutral efficiency.
My illustration “The Audit Chamber” work-in-progress. The art represents the principle that powerful AI must remain open to human scrutiny and correction before it is trusted with public authority.
Criterion Four: Affected-Person Influence
A system is more likely to become harmful when the people it affects have no influence over its operation. They may be observed, ranked, classified, and managed, but never heard. They become objects of administration rather than participants with agency.
Affected-person influence does not mean that every individual controls every institutional decision. It means that people subject to consequential systems retain meaningful rights of understanding, correction, challenge, and, where possible, exit.
At a minimum, people should be able to receive a clear explanation for high-impact decisions. They should be able to correct inaccurate information. They should have access to review by a responsible person or institution that can actually alter an outcome. They should not be punished for contesting a decision, and the route to appeal should not be so costly or complex that it becomes merely symbolic.
Influence also concerns participation before harm occurs. Institutions should seek input from people likely to bear a system’s burdens, particularly where the system will affect work, access to services, identity, health, or economic opportunity. Those closest to a system’s consequences may recognise harms that developers and executives cannot see from a distance.
A proposed question is: Can an affected person do more than comply, or are they able to understand, contest, and shape the conditions under which the system governs them?
If the answer is no, the institution should recognise that it is exercising power without meaningful consent.
Criterion Five: Corrective Power
The final criterion asks whether the institution can correct itself. This is the practical heart of accountability.
An organisation may have good intentions, detailed policies, and ethical commitments. But these matter little if it cannot investigate harm, correct errors, compensate losses, and suspend harmful operations. A system without corrective power is a system that asks people to trust an institution that has no effective way to keep its promises.
Corrective power has several components.
Traceability: the institution must be able to identify who designed, approved, configured, operated, and maintained the system. Responsibility cannot disappear into contractors, proprietary secrecy, or automated procedure.
Investigation: credible reports of harm must trigger timely and competent review. Investigation should examine the system, not only the individual case, because a single error may reveal a recurring pattern.
Correction and reversal: the institution must be able to alter an adverse decision, correct underlying data, restore access, and revise the process that produced the harm.
Compensation: when serious harm cannot be fully undone, the organisation should have a fair process for repairing losses caused by its system.
Suspension: if a system produces serious or recurring harm that cannot be promptly controlled, responsible authorities must be able to pause or withdraw it. Suspension is not a failure of innovation. It is evidence that human safety and dignity are not subordinate to institutional momentum.
The capacity to stop a system is particularly important. A supposedly ethical system that cannot be interrupted once deployed is not meaningfully governed. It is merely supervised from a distance.
Using the Framework: A Proposed Review Process
The framework can be used at three points: before deployment, during operation, and after an incident.
Before deployment, an institution should define the proposed use, assess each of the five domains, identify high-risk issues, and establish safeguards. This stage should end not only with a decision to proceed or not proceed, but with clear conditions under which the system must be reviewed, modified, or suspended.
During operation, the assessment should be repeated. Systems change when their data changes, their users change, their incentives change, or their role expands. A tool used for minor administrative support may gradually acquire authority over more consequential decisions. Regular review is needed to prevent gradual mission expansion from escaping scrutiny.
After an incident, the framework should guide investigation. Was the objective too narrow? Was foreseeable harm ignored? Did the burden fall unequally? Could the affected person influence or challenge the decision? Did the institution have corrective power and use it? These questions move the response beyond blaming an isolated operator or treating the event as a technical anomaly.
The framework is deliberately qualitative at this stage. It may eventually support indicators, weighted criteria, or scoring methods. But any numerical model should be developed cautiously. A score can create a useful discipline, yet it can also give moral uncertainty a misleading appearance of precision. Validation is essential before any version is used as a formal certification or regulatory tool.
Validation Is a Requirement, Not an Afterthought
This proposed framework requires validation in several forms.
First, it requires interdisciplinary examination. Technologists, ethicists, legal scholars, social scientists, affected communities, labour representatives, disability advocates, and sector-specific experts may identify different kinds of risk. No single discipline sees the whole human consequence of an intelligent system.
Second, it requires case-based testing. The criteria should be applied to real or carefully simulated systems in workplaces, financial services, public administration, healthcare, education, and digital platforms. The framework should be revised where its questions are unclear, incomplete, too burdensome, or vulnerable to superficial compliance.
Third, it requires evidence of reliability. Different reviewers should be able to use the framework with sufficient consistency, while still retaining room to recognise context. If the assessment produces wildly different conclusions without clear reason, it needs refinement.
Fourth, it requires the participation of affected people. A framework built only by institutions may measure what institutions find convenient. People who live with a system’s consequences must help determine whether it captures the harms that matter.
Finally, it requires public humility. No framework can eliminate moral judgment, political disagreement, or institutional failure. Its purpose is not to automate ethics. It is to make ethical responsibility harder to ignore.
From Technical Capability to Human Legitimacy
The Demonic Intelligence Assessment Framework asks institutions to move beyond a narrow question: “Does the system work?” A more complete question is required: “Does the system work without making people expendable, voiceless, or unable to obtain redress?”
Technical capability is valuable. It can help societies solve difficult problems, improve access, reduce errors, and expand human knowledge. But capability without human legitimacy is unstable. Systems that impose hidden burdens, deny meaningful challenge, and obscure responsibility may continue to perform until trust collapses or harm becomes too visible to ignore.
The framework’s central principle is therefore not anti-technology. It is anti-expendability. No system should be considered acceptable merely because it advances an institutional objective. It must also preserve the moral authority of the people affected by its operation.
An intelligent system deserves trust only when its objective can be justified, its human costs are visible, its burdens are contestable, and its harms can be corrected or stopped.