AN ETHICAL APPROACH TO LETHAL AUTONOMOUS WEAPON SYSTEMS

Introduction

The increasing dependence on technological systems is a defining aspect of modern societies. Despite the significant benefits to humanity, the amalgamation of technology into every aspect of human life has also brought with it systemic risks, as an inevitable consequence of this development. The rapid pace of many technological developments and their impact on humanity are forcing global society and governments to reflect on the wider implications of these emerging technologies.

In the military domain, Lethal Autonomous Weapon Systems (LAWS) are often presented as the next big evolution in warfare. LAWS are defined as weapons systems that are capable of selecting and engaging targets without human intervention/ control.² These weapons can select and attack targets based on certain pre-programmed criteria and artificial intelligence algorithms^.3

Along with the leap in technology, the introduction of these weapons will mean, that for the first time in our history, direct human decision- making will not be involved in targeting and killing other humans. Instead, the decision may be delegated to an inanimate machine. Therefore, such weapons naturally raise significant ethical questions that need to be thought through before their development and deployment.

Ethical dilemmas in a war zone are not new and have been documented in historical epics such as the Mahabharata, wherein the dilemma faced by warrior Arjun on the battlefield resulted in Lord Krishna narrating the philosophical treatise, Gita. In modern times, the morally abhorrent behaviour of Axis powers was central to the Nuremberg trials and incidents such as the My Lai massacre have shaken the collective conscience of the world. However, what makes the LAWS dilemma particularly complex is that these weapons aim to replace a trained human with a machine that will anticipate decisions as would have been taken by a human, thereby also delegating the moral responsibility of lethal actions to machines. Hence, if we are to ensure that future wars are fought within acceptable humanitarian boundaries, the developers of these systems need to design them to operate within an ethical framework.

Such a framework was also sought by a German Ethics Commission report on autonomous vehicles. The question asked is vital, “What technological development guidelines are required to ensure that we do not blur the contours of a human society that places individuals, their freedom of development, their physical and intellectual integrity and their entitlement to social respect at the heart of its legal regime?”⁴ This is a fundamental question equally applicable to any agency developing LAWS.

Today the development and deployment of LAWS is beyond debate and is more a question of when rather than if. This paper thus aims to assess the use of these weapons within acceptable ethical boundaries, such that their use on future battlefields can be justified both on operational and moral grounds.

Ethics in the Warzone

Stanford Encyclopaedia of Philosophy describes Ethics as a branch of philosophy that deals with moral principles and values, such as good and evil, right and wrong, and fairness and justice. It states that human ethics is the examination of what is morally right and wrong, and how individuals should act in accordance with these moral principles.⁵ In our everyday lives, ethics are involved in nearly every decision one has ever made or will make, as well as asking us a fundamental question – how are we to live?

Conceptually there seems to be a dichotomy between two fundamental beliefs in philosophy. On one end is the Universality Condition, an idealistic concept, which indicates that ethics apply equally to everyone and morality is the same for everyone.⁶  On the other end is Consequentialism, which portends that an act is right or wrong based on its consequences and not based on the type of act.⁷ This ‘end justifies the means’ concept is often  considered to be cynical pragmatism. The challenge for any human (or a military commander) is to walk the tightrope between these two concepts.

In the context of warfare, ethics play a crucial role in determining the acceptable limits of military action. A core tenet of warfighting ethics comes from the Just War Theory, which defines two distinct concepts. Jus ad Bellum – a just cause for going to war, and Jus in Bello – just conduct during war.⁸ Therefore, ethics in warfare is all-encompassing, beginning with the justifications for using force, the treatment of prisoners of war, and the protection of non-combatants and civilian populations.⁹ To regulate the  actions of belligerents, laws of armed conflict, such as the Geneva Conventions, serve as a basis for ethical conduct in warfare and establish standards for the treatment of individuals in armed conflict.¹⁰

As can be inferred from the above, the importance of ethics in warfare cannot be overemphasised. Considering the lethal aspects of warfare, it is essential to codify certain moral and legal standards that govern the use of force. These regulations help ensure that military action is conducted in a manner that is consistent with the principles of just war and that it protects the rights and dignity of all individuals affected by armed conflict. They also help guide military operations such that they are conducted in a way that is consistent with fundamental human values and rights, with the core principle being that the protection of innocent individuals takes precedence over all other considerations.

The introduction of LAWS into modern warfare has kindled many debates, both due to their technical challenges and also due to the ethical concerns they raise. The ethical questions focus on accountability, responsibility, and the ability of these systems to comply with the principles of proportionality, which require that only military targets be engaged and civilian harm be minimised.¹¹ There are concerns that the deployment of  LAWS could undermine the laws of armed conflict and cause indiscriminate harm to civilians and non-combatants. International Committee of the Red Cross (ICRC) has succinctly summarised the problem statement, “the central argument is that it matters not just if a person is killed or injured but how they are killed or injured, and the process by which these decisions are made is as important as the results.”¹²

Preceding this discussion is the elemental question as to how much freedom of choice we are willing or allowed to transfer to programmers or even self-learning systems in the first place. German philosopher Immanuel Kant opined that “the freedom of the individual to enjoy the right of moral self-determination forms the basis of an existence determined by reason.”¹³ Therefore, to replace the human with a machine, we must first understand what the human was doing in the system and map the human reasoning system and moral code into a software code.

An Ethical Military Dilemma

Are we legally allowed to strike? Yes…; Does that mean we should? No… ¹⁴

The above conversation is from the 2015 motion picture Eye in the Sky. The movie is based on a fictional operation to eliminate a known terror operative via a drone strike. The key personnel responsible for the operational decision-making consist of a group of military personnel, bureaucrats, and politicians. The operation turns into an ethical quagmire for the group when a young girl sets up her bread stall adjacent to the target’s location. The movie aptly showcases the moral dilemma between completing an operational task of eliminating a confirmed target vis-à-vis knowingly sacrificing an innocent child as collateral damage.

The story focuses on the tensions arising between military action, political will, moral responsibility, and legality within one military operation.¹⁵ It brings out the impossibility of defining a set of guidelines or rules of engagement that can be specific enough to cover every eventuality in the scenario. During the entire operation, no individual has the complete Ironically those with the picture or understanding of the event, and authority to make the ironically those with the authority to make decisions are the ones with the decisions are the ones with the least the least information information. Yet, at some point, someone must make a crucial decision that may or may not result in the death of innocent parties. The operation moves at a fast pace in such a way that the right answer is never immediately obvious as there can be no good or bad decisions in such a scenario.

Responsibility and Accountability

The fundamental ethical concern with LAWS is whether the responsibility for the use of lethal force can be delegated completely to an inanimate robot/ machine. If the answer is yes, then can an autonomous robot/ machine be held accountable, for injuries inflicted or even deaths caused by it, which are morally unjustifiable? In traditional warfare, human soldiers are held accountable for their actions on the battlefield; but with LAWS, it becomes unclear as to who should be held responsible for any harm caused by these weapons.

Presently, legal responsibility relates to the duty of a person to account for decisions taken by him/ her and the attendant actions performed, to assume liability and, if necessary, to be willing to accept any legal consequences.¹⁶ In case of an accident involving LAWS, defining  responsibility is tricky, as it is split between the automated system (hardware) and/ or the software on which it is based, without any clear demarcation. Further, as the decisions are made by the system without the requirement of human oversight, this ambiguity in accountability between machines and humans could result in human decision-makers deploying these weapons without considering the consequences of their actions.

Let us consider a contemporary example. In autonomous vehicles, the accountability that was previously the sole preserve of the motorist will shift to the manufacturers and operators of the vehicle, associated software and to the bodies responsible for making infrastructure, policy and legal decisions. With this shift in responsibility, the option of reverting manual control to the driver is a complex question. Should the vehicle be completely autonomous or should manual override be available as an option to the human rider if it can avoid an accident from occurring? While a manual override option appears logical, such an option would introduce ambiguity in assigning accountability in case of an accident.

Lack of Human Judgment. The ambiguity in accountability also raises concerns about the loss of human judgment on the battlefield. Once human accountability is diluted, operators The ambiguity in accountability may delegate decisions to LAWS also raises concerns about the without considering the nuances and loss of human judgment on the complexities of a particular situation, battlefield thus increasing the possibility of unintended consequences. Such abdication of responsibility could be unintended, due to the lack of human understanding of these weapons, and their potential to cause collateral damage.

Situations and Dilemmas

Dilemmas are instances in which an individual/ machine must choose between two decisions, both with varying adverse implications, and between which there can be no trade-off.¹⁷ One famous dilemma situation is the ‘trolley problem’ thought experiment where the reader must choose between two actions, both of which result in the death of a certain number of humans.¹⁸ The situation in the above-mentioned movie could also be 18 considered a dilemma. To better illustrate the problem, consider a robot that is designed to provide medication to patients in a hospital. In the case of a patient unwilling to take the medication, what if the robot did not just deliver the medication, but forced the patient to take it, supposedly for their own good? Such imposed decisions, while for the benefit of the individual, could deprive him/ her of the right to take their own decisions.

Dilemmas have no right or wrong solutions and moral judgement is often ambiguous even when taken by experienced and skilled humans. Whether such situations can be ethically evaluated and morally acted upon by a software code is an open question. LAWS would be expected to prevent lethal action wherever practically possible, however, genuine dilemmatic decisions, such as a decision to choose between one human life versus another, cannot be standardised. Therefore, the challenge for designers is to develop an algorithmic model for an autonomous system, such that its decisions are ethically unquestionable, or at least explainable.

Human-Machine Interaction

Target classification is a tricky area. Since blips on radar screens do not communicate intention, the classification of a target as friendly/ hostile/ neutral has many layers. To take a generic and popular example, a defecting enemy aircraft will appear as a hostile, closing in for an attack and a valid target that needs to be engaged.¹⁹ But in case the defecting pilot’s intentions are revealed while he is still flying in, a human decisionmaker can adapt his/ her actions accordingly. However, an autonomous system, with no metric to measure the pilot’s intent, will surely take offensive action against the target. Even in case inputs regarding the defector’s intent are received by the human controller, one cannot disable the weapon system because threats still exist from other hostile aircraft. Neither can one alter the decision- making algorithm in such a short time, whilst the system is live. Therefore, a human decision-maker will need to be involved to apply discretion/ manual override. Further, this manual interface will have to be seamless so that human decisions can be implemented before the software initiates actions to engage. Thus, the human-machine interaction or decision matrix cannot be distinct but must be interwoven to function as one homogenous system.

 Compliance with International Humanitarian Law (IHL). These weapons will be programmed to evaluate and comply with certain basic legal considerations consistent with the laws of war, such as avoiding civilian targets and minimising collateral damage. However, the development of ethical decision-making algorithms is a complex and challenging task, and it is unclear how these weapons will make decisions in real-world situations. There is also the potential for these algorithms to malfunction or be hacked, leading to unintended harm. This raises the possibility that the deployment of LAWS may lead to an erosion of the norms and principles of IHL.

Bias. Despite being a software devoid of human emotions, AI applications have been accused of being biased in multiple situations.²⁰ There have been  allegations of AI systems providing results tilted against certain ethnicities and people of colour, providing results that favour certain political philosophies, etc. Biases may creep into the system due to lopsided training models or could also be attributable to the designers’ inherent preferences. Biases could also develop over time due to the restricted data set and situations the system is exposed to. In an operational environment, such biases could result in the selective targeting of certain ethnicities without any such military intent.

Solutions

The earlier sections raised several questions which would need to be answered before the deployment of LAWS in the real world. However, many of these questions have also been asked previously, as the shift from manual to automated systems is not a new phenomenon. The electrical elevator is a simple example whose development created fears of job losses, accidents and raised issues of manufacturer’s liability.²¹ Over time, these initial worries have been overcome and today elevators are accepted globally as a safe and efficient medium of mass transport. Similarly, LAWS too could overcome the present concerns, if we follow a patient and careful approach to finding solutions and assuaging these worries.

Before we start looking for possible solutions, we need to answer a fundamental question – what should be the role of autonomy? For this, it would be prudent to figure out the roles in which a machine can do better than a human in all situations, every single time; and focus on automation of those roles. As a corollary, in roles, where a machine does not provide a definite advantage over a human, resting the final control with a human is recommended. With this foundational objective, we can focus on answering more complex questions.

Responsibility and Accountability

In all automated systems, there will be issues relating to the demarcation of accountability and associated liability between manufacturer and operator. To resolve the ambiguity in accountability, it needs to be clearly distinguished if a system is being used in completely autonomous mode, or whether a human retains the option of overruling the system. In the case of hybrid systems, wherein a human can override the automated response, the human-machine interface must be designed such that at any time it is apparent on which side the accountability lies.²² Hence, such systems need to document their activity timeline, with time stamps and access records.

This applies especially to the human-machine handover procedures, wherein there needs to be complete transparency about who was in control of the equipment at what time, thereby having the decision-making Meaningful, effective or responsibility. This also includes the appropriate human control possibility of the autonomous system preserves human agency and handing over control to the human upholds moral responsibility operator if the situation develops in decisions to use force beyond the training models of the system. Any shift to manual control should be as per a defined protocol with due warning to the human operator because an abrupt handover would achieve no improvement in the decision-making despite the change in control.²³

ICRC has suggested that “Meaningful, effective or appropriate human control would be the type and degree of control that preserves human agency and upholds moral responsibility in decisions to use force.”²⁴ This does not necessarily exclude autonomy in weapon systems, but it necessitates limits on their autonomy, especially in decisions involving an ethical dimension. Such ‘controlled autonomy’ requires that a direct and continuous connection be maintained between the human and machine, so that the desired intent of the user guides the actions of the weapon system in a specific operation, and thereby also the eventual consequences of the mission.

Explainable AI

To reduce the controversies and ambiguities of accountability, we need to insist on the ‘explainability’ of the AI-generated solution. If humans cannot understand the solution being suggested, then having them in the loop (for manual override) is of no use. If we do not understand the core decision-making logic, instead just accept (and follow) a yes/ no solution offered by the algorithm, then we might as well let the software decide and keep humans out of the decision-making loop.

Explainability is certainly a complex proposition with AI-enabled systems where the algorithm is not a static piece of code, but one which keeps evolving with each experience (data set). The challenge is further complicated as there is a need to convey an abstract chain of logic in a compressed time frame under combat pressure. Therefore, designers will have to produce explainable models to enable end users/ operators to understand, appropriately trust, and effectively manage AI-enabled equipment. Such explainable models will also help in ruling out any biases that may creep into the system during initial training/ exploitation.

Human-Machine Interaction

LAWS may be able to act faster and more accurately than humans in several scenarios, however, the underlying AI technology (presently) cannot think. These weapons are capable of sensing, detecting, and taking corresponding action, but Human-machine interface in LAWS is a they remain limited by the crucial element of the system design, to analysis and actions coded in derive the best results from automation, their algorithms. Hence, the without letting the machine run amok human-machine interface in LAWS is a crucial element of the system design, to derive the best results from automation, without letting the machine run amok.

Making these weapons predictable and precise should be the first step towards the final goal of making them self-deciding. These weapons can be considered objectively more ethical than a human operator for a particular combat task, only if they are demonstrably more accurate and precise. To achieve this, both humans and machines should play to their strengths e.g. in war, many causalities result from poor target selection. Therefore, LAWS can help in target recognition, but the decision to act may still need to be human. As the technology matures, the role of human involvement would gradually evolve from directing machine actions to consenting for machine actions (as an interim step), and ultimately for goal determination only, with the machine performing the entire ‘detect-decide-act’ sequence^.25

As with any decision-making process/ system, there would certainly be errors. Therefore, there must be a defined metric for error acceptance which should be realistic. LAWS should be coded to operate within a defined set of limitations, and if the errors are anticipated to be beyond the tolerance limits, such cases should be referred to human decision-making. Also, to enable efficient and reliable human-machine communication, autonomous systems must adapt themselves to human communicative behaviour rather than the other way around.

Compliance with Human Rights

Since 2013 the governance of LAWS has been discussed within the framework of the 1980 United Nations Convention on Certain Conventional Weapons (CCW) and 65 states have actively participated in the debates. ²⁶ However, discussions on LAWS at the CCW are still at an 26 early stage as there have only been general discussions at the diplomatic level. Existing statements commonly acknowledge that this is a highly complex topic and that more discussion will be needed to generate a common understanding of the various issues.

An ethical-legal framework to define the development of LAWS is already available in the form of Human Rights Laws. These laws have coded ethical principles into globally accepted norms that have been developed and implemented over decades. Thus, a wide range of ethical views has already been distilled into legible legal principles. Human rights already comprise a universal language and framework of ethical norms, which have taken many years to negotiate and can therefore protect fundamental values and human dignity in an automated warzone as much as they do in a manual one.

Rather than grappling to invent new standards, governments and manufacturers should embrace existing human rights standards and processes as the starting point for AI governance. There are significant advantages to using human rights as the foundation of AI ethics. To begin with, these laws are already widely accepted internationally as the Universal Declaration of Human Rights (UDHR) with every country in the world being party to at least some of the United Nations human rights treaties.²⁷ Further, Human rights are also the existing legal framework of fundamental values and human dignity.

The challenge for LAWS, however, is not only to take human rights as the starting point for conversations about AI ethics and regulation but to embrace them in system design. Hence, manufacturers and governments should involve human rights experts to guide the development of these systems and integrate human rights laws within AI regulations and policies.²⁸ Doing so will be a reliable method to ensure that autonomous weapons indeed preserve fundamental values and human dignity. Such systems, designed with Jus in Bello principles at the core of their decision mechanism, limit the chances for violations of IHL.

Development of Global Standards

There are already a few initiatives taken by government agencies/ global regulators to instil an ethics engine in autonomous systems. The US DoD has committed to the ethical development of weapons by adopting five principles of Artificial Intelligence Ethics – Responsible, Equitable, Traceable, Reliable and Governable.²⁹ Further, the Institute of Electrical and Electronics Engineers (IEEE)’s Global Initiative on Ethics of Autonomous and Intelligent Systems, is working on ethically aligned design standards for AI and autonomous systems.³⁰ Such initiatives will need to be inclusive to have global relevance and strive to ensure the organic and sustained ethical development of LAWS.

AI risks can be mitigated through the development and publication of standards and guidelines around safety, privacy, ethics, and control in coordination with industry, academic, and international stakeholders. Failure tolerance is a critical aspect of any system; hence there should be some standards/ metrics to deal with failures when they do arise.³¹ Such transparent standards will also dispense fears that focussing on ethics and including it in the algorithmic design will result in trading away operational benefits or efficacy.

Josh Sullivan lucidly summed up the role of policymakers in AI regulation in The Atlantic, “While AI cannot be expected to solve philosophical conundrums that have existed across society for years, government leaders and policymakers are uniquely positioned to lead and fund long-term R&D on topics like ethics and bias where businesses may not be incentivised to invest.”³² Therefore, governments must frame policies that balance the need for innovation while honouring the obligation of safeguarding core human values.

Opportunities for India

The technology stack at the core of LAWS is extremely esoteric, resulting in increased reliance on manufacturers for maintenance and maybe even during operations. Thus, it is imperative that such weapons are indigenously manufactured as foreign dependence could pose crippling operational limitations. Without having complete transparency and knowledge of the inherent algorithms and the capability to modify them at will, these weapons can turn into a liability in adverse situations if the OEM is unwilling to cooperate. Therefore, indigenous development of LAWS, including the inherent algorithm and ethical controls is not an option but an imperative for India.

As India’s global standing improves, world leaders and multilateral organisations not only hear our opinions but also expect us to take the lead in certain areas, especially in technical fields, given our past expertise. We must leverage our unique position, Further, with our traditional historical advantage, and access to concepts such as Vasudhaiva global forums towards guiding the Kutumbakam, a proud history development of ethical standards of adherence to high ethical standards such as non-violence and an inclusive democratic society, India has a unique advantage in the development of techno-ethical standards. Opportunities like the present G- 20 leadership, International Solar Alliance, Quad, etc. are examples that we can indeed shape global discourse and more such opportunities will come in future. Therefore, we must leverage our unique position, historical advantage, and access to global forums towards guiding the development of ethical standards for autonomous systems, for both military and civilian applications.

India’s domestic military-industrial complex is emerging from a long period of relative lethargy and indifference and seems well-positioned to steer the development of LAWS. As this technology is at a nascent stage, we have the unique opportunity to do it the right way from the beginning. India can lead the development of standards for safety, failure transparency, accountability, integration of human values and human control in autonomous weapons. Ethically designed tools for defence will provide an opportunity to project our core values and can also aid in the export of military equipment in future.

One constraint for encoding of ethical regulations is that these laws are not widely understood, especially among scientists and technical personnel. Hence, the development of acceptable ethical standards for LAWS will need the participation of experts from a wide range of disciplines, including voices from fields such as philosophy, social sciences and technology. This suits a pluralist society like India, which is accustomed to encouraging healthy debates involving varied viewpoints during all public policy decisions. Kate Jones in a Carnegie article writes that “human rights need to be part of the mainstream dialogue, not merely a topic for legal experts.”³³ India could hasten this debate by introducing ethics as a curriculum in its technical universities and training programmes for technologists/ government functionaries.

The Indian Government for its part has been encouraging the discourse around various aspects of AI through official and informal channels. The Niti Ayog’s 2018 discussion paper on ‘The National Strategy for AI’ provided insight into the country’s AI ambitions.³⁴ The paper was comprehensive and covered issues beyond the technology realm; such as transparency, accountability, privacy, security, and ethics. Further, aspects related to explainable AI and its military implications are being debated in open forums by domestic think tanks such as ORF, which will help India drive the discourse in future.³⁵ A whole of government approach is also evident during events such as the ‘AI Def,’ which aims to broaden the interest in military applications of AI and their associated challenges.³⁶

Conclusion

Ethics and development are often mistakenly seen to be in conflict. Whether it be the felling of trees to make way for a bridge, the development of machines to replace human jobs or advanced software which is used to target humans during conflict; ethical quagmires will continue to challenge future designers and field commanders as they have since the time the humble bow and arrow was invented. However, solutions will emerge when we include ethical considerations as an integral part of all projects and imbibe them at the design stage as a self-check mechanism to prevent deliberate or inadvertent misuse of the technology. We need to understand that ethics do not hamper innovation, but help create a human foundation for technological innovation. Similarly, AI ethics not only prioritise transparency and accountability, but they also preserve fundamental humanitarian values and human dignity.

LAWS offer a wide range of novel possibilities for military leaders as it saves their troops from going in harm’s way and reduces physical and mental stress on their formations. Therefore, from a purely military point of view, the advantages of these weapons in terms of enhanced safety and protection of its forces outweigh the risks inherent in them. But, the development of these systems should be weighed against what is feasible vis-a-vis what is appropriate and ethically justifiable. Therefore, scepticism arising from these weapons should be expected and not be dismissed as naive criticism of technology. If they are not designed and exploited with care, LAWS will be in violation of IHL and the spirit of the Martens Clause. To avoid such serious concerns, our combined human conscience should guide the development and deployment of LAWS.

Philosophy and ethics are subjects that leave us with more questions than answers. These are ever-widening fields with increasing branches and theories. Hence, ethical development/ deployment of LAWS cannot be an end goal, instead, it should be a constant endeavour. Will LAWS let us be the good soldiers we were? Or will this new power bring out the worst in us by taking away accountability, making our actions indifferent to the consequences?

The famous deontologist Immanuel Kant once said, “An act is right only if it is motivated by a sense of duty and not desire.”³⁷ This line  encapsulates the spirit of ethical behaviour for most situations and all of humanity. There is not much wrong a soldier can do if he/ she lives and acts by this sense of duty, which should act as a guide to everyone who aims to fight their battles within the dimensions of acceptable human behaviour and wishes to morally hold their head high after the mission. It would bode well for the armed forces and the world if tomorrow’s autonomous weapons could also be guided by the above statement for their actions, both within and outside battle. The question though remains, how do we imbibe the sense of duty within a robot?