As researchers push ahead with algorithms designed to give robots more autonomy — and possibly the capability to fire weapons without a human in the decision loop — ethicists and legal minds warn that not enough thought is being given to the implications of using unmanned systems to apply lethal force.
“We assure ourselves that we will always be ‘in the loop,’ and yet we’re working on systems that for very good reason, for very good battlefield justifications, are designed to take us ‘out of the loop,’” said Peter W. Singer, a Brookings Institute scholar and author of “Wired for War: The Robotics Revolution and Conflict in the 21st Century.”
Singer is part of a growing chorus of academics, legal experts and technologists who are suggesting that the research into robotics systems is outpacing the defense community’s ability to grasp the unintended consequences of introducing this potentially game-changing technology to the battlefield.
Col. Jeff Eggers, Air Force director of intelligence, surveillance and reconnaissance innovations, said, “These issues of ethics, especially in relation to developing concepts of autonomy, are very pertinent and we need to look at them now as we design our new systems.”
Maj. Gen. Charles Dunlap Jr., Air Force deputy judge advocate general, said there is a disconnect between the legal and research communities. Lawyers are generally “clueless” about technology and military strategy.
“It’s not hard to find lots people who know the law. It’s hard to find people … who understand systems and how systems are used in combat and the limitations and capabilities of those systems,” he said at the Association for Unmanned Vehicle Systems International conference.
Meanwhile, researchers are pushing ahead with the technologies that could make it possible for a ground or airborne robot to fire weapons autonomously. And one prominent Army leader, Lt. Gen. Rick Lynch, commanding general of Fort Hood, Texas, and the Army’s 3rd Armored Corps, called at the conference for the military to field ground robots that could autonomously apply lethal force. A weapon in a fixed position searching for insurgents who are planting roadside bombs was one scenario he cited. This technology is mature, and could be saving lives on the battlefield today, said Lynch, who also holds a master’s degree in robotics from the Massachusetts Institute of Technology.
The office of the secretary of defense, in its “Fiscal Year 2009-2034 Unmanned Systems Integrated Roadmap,” acknowledged that these legal issues have yet to be worked out. It suggested that Lynch will not be getting his wish anytime soon.
“For a significant period into the future, the decision to pull the trigger or launch a missile from an unmanned system will not be fully automated, but it will remain under the full control of a human operator,” the roadmap stated.
It outlined four main fields where the Defense Department should concentrate its robotics efforts: reconnaissance and surveillance; target identification and designation; counter-mine and explosive ordnance disposal; and chemical, biological, radiological and nuclear detection. Armed robots were not on the list.
That’s not to say that such programs don’t exist. The report identified three weaponized unmanned aerial vehicles: the Predator, the Reaper and the Extended Range/Multi-purpose unmanned aerial system.
The Army’s Armament Research, Development and Engineering Center at Picatinny Arsenal, N.J., three years ago attempted to field an armed ground robot, the special weapons observation remote reconnaissance direct action systems, or SWORDS. Three were sent to Iraq but never used as intended. Senior Army officers apparently did not have confidence in the system. Designed as a mobile platform, they were put behind sandbags in fixed positions. It’s unclear if their weapons were ever fired in combat.
Despite the lack of acceptance by battlefield commanders, and the goals spelled out in the technology roadmap, ARDEC officials at a separate robotics conference last year said they are pushing ahead with the next iteration of the system.
SWORDS and the three UAVs are all remotely operated. Humans ultimately pull the triggers, although from a distance. The roadmap stated: “Because the DoD complies with the Law of Armed Conflict, there are many issues requiring resolution associated with employment of weapons by an unmanned system.
“Many aspects of the firing sequence will be fully automated but the decision to fire will not likely be fully automated until legal, rules of engagement, and safety concerns have all been thoroughly examined and resolved.”
The Laws of Armed Conflict are international public laws that the Defense Department requires each service to incorporate into their training, rules of engagement and legal reviews for new weapon systems. They are part of the 1949 Geneva Conventions, of which the United States is a signatory.
Dunlap said battlefield commanders are likely to be cautious if autonomous armed robots are introduced. Every weapon system undergoes a legal review, but the challenge comes in the deployment. “Commanders are going to be accountable.”
Robots could potentially kill an innocent, or cause friendly fire incidents.
Nevertheless, he supports the use of armed unmanned systems in today’s wars. They put fewer American lives at risk, cause less collateral damage, and most importantly, they are efficient in killing enemies. And in his mind, that’s a good thing.
“If the purpose for which we are using force is appropriate, I’m a believer that technology can make the application of that force the most discrete,” he said.
As for armed UAVs, uniformed military personnel must be the ones operating these unmanned drones that carry out air strikes, he said. If contractors begin operating these machines, they may not have legal protection. There has to be a clear chain of command, and that requires operators in uniform at all levels of the decision-making process, he stressed.
Singer warned that there is danger in using technology “that allows us to go to war, but not under the old meaning of war.”
For decades, Congress has not exercised its prerogative to declare war. The nation now moves ahead with armed conflict without this formality. He noted that there has not been any national debate or vote in Congress authorizing strikes by unmanned aerial vehicles in Pakistan’s ungoverned tribal territories. These attacks, widely reported in the press as being carried out by the Central Intelligence Agency, are designed to eliminate senior Taliban and al-Qaida leaders.
“Most of those operations are being carried out by folks that are not in uniform,” he added.
Ron Arkin, director of the mobile robot laboratory at the Georgia Institute of Technology, has attempted to bridge the divide between the researchers and ethicists. He recently penned a book, “Governing Lethal Behavior in Autonomous Robots,” which tackles the subject.
His lab is working on an “ethical governor,” a program that can be added on to a ground robot’s specialized software architecture. It would monitor the outgoing behavior of the robot and transform it into behavior “consistent with any ethical system conceivably, but ideally the internationally agreed upon laws of war and rules of engagement.”
The program attempts to incorporate guilt into the system. He’s often asked why he doesn’t add compassion into a robot’s behavior.
“That would be great if I knew how to do it. For ‘guilt,’ there are models in cognizant psychology I can work on,” he said.
A robot should also be able to refuse an order. A “don’t shoot” override might recognize when a human operator is asking the machine to carry out an action that violates specified rules of the Laws of Armed Conflict. “I believe these systems should have the right to refuse an order.”
Singer said having robots comply with these rules may enter into the realm of “lawfare.” The term refers to a tactic that is being seen on battlefields today. The enemy is keenly aware of when a U.S. soldier is allowed to fire or not fire a weapon, and he takes advantage of these rules of engagement. A foe who is aware of a robot’s ethical governor may adapt in a similar way, he added.
Dunlap said if engineers inputted every international warfare law into a robot’s computer, the machine would freeze. “War is still an art. You can’t reduce everything down to an algorithm,” he countered.
Eggers also had his doubts. “I don’t think we can ever design a machine, or have enough money to build a machine, that will be able to consider every conceivable input into a decision for a military action.”
Arkin said a robot would be treated like any other soldier. It wouldn’t be necessary to cram every word of the rules of engagement or the Laws of Armed Conflict into its program. Soldiers train for specific missions and tasks. A robot would do the same.
Also, soldiers and marines are briefed on rules of engagement and the international laws of warfare all the time. Yet, horrific incidents of violence still happen.
“While many war fighters are quite noble, there are many others, too, who may have a propensity to commit atrocities,” Arkin said. Robots, he argued, can ultimately become “more humane than humans” on the battlefield.
These debates on where robotic technologies are leading the military are largely ignored, Singer said. Technologies come along that enable the nation to do things it was never able to do before.
Everything from the printing press to the atomic bomb has forced users to rethink the ethics involved. The robotics revolution is no different. “We have to weigh what’s proper when [we] didn’t have to think about what’s proper before,” Singer said.
Eggers added: “The future is happening every day and it is time to start thinking about that and talking about that now rather than designing all these systems and then start trying to patch [them] over.”