'Killer robots' pose a moral dilemma

The United Nations began talks on Monday on lethal autonomous weapons systems amid calls for an international ban on these "killer robots" that could change the nature of warfare.

The weeklong meeting of a disarmament grouping known as the Convention on Certain Conventional Weapons (CCW) in Geneva comes after more than 100 leaders in the artificial intelligence industry warned in August that these weapons systems could lead to a "third revolution in warfare."

"Once developed, they will permit armed conflict to be fought at a scale greater than ever, and at timescales faster than humans can comprehend," the signatories said in a letter. "The deadly consequence of this is that machines — not people — will determine who lives and dies." 

While the rapid development of artificial intelligence and robotics in the past decade have led to improvements for consumers, the transport sector and human health, the military application of greater autonomy in weapons systems has evoked images of Terminator-type sci-fi war machines entering the battlefield to hunt down adversaries without any human behind the controls.

What is a lethal autonomous weapons system?

There is no international consensus on what constitutes a lethal autonomous weapon system, also known as a fully autonomous weapons system. It is often defined as a system that can target and fire alone without meaningful human control. In essence, they are machines with built-in hardware or software that allow them to function independently of humans once they are turned on. They function on the basis of artificial intelligence — algorithms assess a situational context and determine the corresponding response. 

Multiple weapons systems from drones, precision-guided munitions and defense batteries already have levels of autonomy, albeit with various degrees of human control. Several countries' militaries also use human controlled robots to search for mines, traps and unexploded ordnance. 

"Many people underestimate the extent of automation and computerization of warfare today. The use of modern sensors and munitions has already generated greater distance between the human and the battlefield in some cases," said Michael Horowitz, a professor of political science at the University of Pennsylvania who researches autonomous warfare. "Any discussion of autonomous weapon systems should start by understanding how they are similar to, and different from, existing military technologies."

Are there lethal autonomous weapons systems in the world today?

Strictly speaking, lethal autonomous weapons systems do not exist today. Israel's Harpy anti-radar "fire and forget" drone is closest. After launch by ground troops, it autonomously flies over an area to find radar that fits pre-determined criteria and then unleashes a kamikaze strike.

South Korea has developed a sentry gun system to guard the heavily militarized border with North Korea. The weapons system, which includes surveillance sensors and tracking as well as automatic firing, can be made completely autonomous, but in its current use, it requires a human to approve before engaging. 

Slippery slope?

The Campaign to Stop Killer Robots, a group of NGOs seeking a ban on lethal autonomous weapons, says that sensors and advances in artificial intelligence make it "increasingly possible" weapons systems of the future would target and attack without human intervention.

"If this trend towards autonomy continues, the fear is that humans will start to fade out of the decision-making loop, first retaining only a limited oversight role, then no role at all," the group said in a statement.

In some cases, such as with cruise missiles, its sensors and terrestrial guidance systems can lead to more targeted strikes and fewer unintended casualties compared to traditional bombing. 

But the experts who signed the August letter expressed moral concern over the development of fully autonomous weapons systems out of today's semi-autonomous and human-supervised autonomous systems.

"Lethal autonomous weapons systems that remove meaningful human control from determining the legitimacy of targets and deploying lethal force sit on the wrong side of a clear moral line," they wrote.

There is no guarantee any technological system will work perfectly, "but the moment you give it lethal weapons, the danger increases manifold," said Ulrike Franke, a policy fellow at the European Council of Foreign Relations who researches drones at Oxford University. "A smart armed system can become a dumb armed system quickly." 

A fully autonomous system gone awry could take unwanted action in complex battlefield situations, target civilians or engage in friendly fire. The possibility of mistakes with little or no human role also raises questions around the laws of war and military policy, such as who bears responsibility. 

"To what extent can we hold a military commander that deploys such a system responsible, if there is no meaningful way for him or her to predict how it will behave?" asked Franke. 

In the hands of unsavory regimes with little worry about such questions, such systems could be used against its own people. And in the hands of terrorists or non-state actors, a "killer robot" could result in devastating destruction. 

Arms control

But not everyone thinks that advances in fully autonomous weapons systems will diminish the human role in warfare.

Retired US Colonel Brian Hall, an Autonomy Program Analyst at the Joint Chiefs of Staff, wrote in July that the advantage of autonomous weapons systems will "come from augmenting human decision making, not replacing it."

Still, he cautioned that given the pace of advances in science and technology the weapons capability of autonomous weapons in the future are difficult to predict, which may require legal and policy changes. 

Several countries, including the United States, Russia, China and Israel are researching or developing lethal autonomous weapons systems out of concern adversaries may not be bound by humanitarian, moral and legal constraints, resulting in a potential "killer robot" arms race in the years to come as the technology improves. 

As Russian President Vladimir Putin said in September, whoever is the leader in artificial intelligence "will become the ruler of the world."

While the powerful potential of autonomous weapons on the battlefield causes concerns, it also makes them more difficult to ban or regulate, experts said.

"This is the arms control dilemma. The more useful potential weapons are for militaries, the harder it is to regulate or ban them," Horowitz said. "Uncertainty about what an autonomous weapon is further complicates the discussion – states are unlikely to agree to regulations or bans if they do not know what will be covered," he addded.

For Franke, an outright ban or arms-control regime is unlikely. Lethal autonomous weapons systems are not like nuclear weapons since they cannot be counted, which is a key requirement for arms control agreements. They also are unlike chemical weapons, which have been banned. And with no strict definition of what a lethal autonomous weapons system is, "there is no way to identify it by just looking at it," she said.