Damn Asimov's Laws, damn them to hell...
Here's the transcript, however I highly suggest listening to it...
Transcript
Hide
Alan Saunders: Music from James Cameron’s movie The Terminator, in which Arnold Schwarzenegger plays a killer robot from the future, But hey, The Terminator was more than a quarter of a century ago, and what was once science fantasy is now close to being fact.
These days, you can go to war without shouldering a pack and carrying a rifle. You can take out the enemy’s installations, and, indeed, take out the enemy just sitting in an office not far from home. We’re talking drones, robots which are doing the dirty work for us. But what are the ethics of a war fought for us by machines, where the only deaths we see are on TV monitors? How can we bring a moral imagination to bear on a world of robot conflict? Well that’s the question we’re asking this week on The Philosopher’s Zone.
Hi, Alan Saunders. This was one of the themes of a talk delivered at the Festival of Dangerous Ideas by Peter Warren Singer, Senior Fellow and Director of the 21st Century Defence Initiative at the Brookings Institution, the Washington think tank. He’s one of the world’s leading experts on changes in 21st century warfare. And he’s not just a policy wonk. He has a firm grasp on popular culture and he has advised on a video game series. So here’s PW Singer on robo-wars.
Peter Warren Singer: What I thought I would do to begin today is actually begin with a dangerous idea through the form of a story. And for this story you have to imagine yourself in Iraq. And ahead of you alongside the road is what looks like a piece of trash. But that insurgent has hidden that ‘IED’, that Improvised Explosive Device, with great care. Now, by 2006 there were more than 2500 of these roadside bombings every single month in Iraq.
Now the team that’s hunting for this IED, for this roadside bomb, is called an ‘EOD’ team: Explosive Ordinance Disposal. If you’ve seen the movie Hurt Locker, it’s about one of these EOD teams in action. Now, by this period in a typical tour, an EOD team would go out on just about two bomb calls per day. That is, they’d be asked over the course of their tour of duty to defuse about 600 bombs. The number though that’s maybe the better indicator of their value to the war effort is the fact that the insurgents put a reported $50,000 bounty on the head of one of these soldiers.
Unfortunately, this particular bomb call would not end well. And by the time the soldier got close enough to the device to see that it was a bomb, that it wasn’t a piece of trash, by the tell-tale wires coming out from it, it exploded. And so when the dust settled, and the smoke cleared, the rest of their team advanced and they found little left of them. So that night, the commander of the unit did their sad duty and he sat down to write a condolence letter back to the United States. And in the letter he apologised for not being able to bring that soldier home. He talked about how tough the loss had been on the rest of their unit, that they’d felt like they’d lost their bravest companion. And then he tried to talk up what he saw as the silver lining from the tragedy. This is what he wrote. Quote: ‘At least when a robot dies, you don’t have to write a letter to its mother.’
That scene, that story, sounds a little bit like science fiction but is actually battlefield and technologic reality. It’s the true story of the very first robot killed in action. That is, that soldier was a 42 pound robot called the PackBot. The destination of that condolence letter wasn’t a farmhouse back in Nebraska like in the old war movies, but was actually a factory building just outside Burlington, Massachusetts, that on the side of it says ‘iRobot’.
It is a real world company named after the fictional Isaac Asimov novels and not-so-great Will Smith movie in which robots start out doing daily chores. An iRobot that makes the PackBot also makes the Roomba, the little robot vacuum cleaner you may have heard of. But in both fiction and now reality, robotics started out doing our chores and moved on to fighting our wars.
Now, to pull back on all of this, there’s something big going on in technology, war and maybe even the overall history of humanity itself. The military that went into Iraq in 2003 had a handful of drones supporting it. Pilotless planes. Unmanned aerial systems. Whatever you want to call them. The US military right now has over 7000 of these systems. The invasion force that went in on the ground had zero unmanned ground vehicles, like that PackBot. We now have over 12,000 in the military inventory.
Last year the US air force trained more unmanned systems operators than it trained manned fighter plane and manned bomber plane pilots combined. Another example of the breakpoint is that right now, in the western states, out of all the defence manufacturers, none of them are researching and developing a single manned jet fighter. They’re all working on unmanned systems right now.
The interesting thing though is that when we talk about these PackBots and we talk about the Predator Drone, we need to remember we’re actually talking about the very first generation. We’re talking about the version of the Model T Ford or the Wright brothers’ flyer.
We’re really at the horseless carriage stage of all this. If you think about the way we describe them. We call them ‘unmanned systems’, like back then we called them ‘horseless carriages’. We couldn’t wrap our heads around what they actually are. That horseless carriages were automobiles. That unmanned systems are actually robotics. The point that I’m making here though is that that’s what’s happening right now.
Peering forward, one US Air Force three-star general I remember speaking with described how very soon it’s not going to be thousands of robots in our wars, as we have right now, but, quote, ‘tens of thousands of robots’. And it’s not going to be tens of thousands of these ones or the prototype ones you’ll soon see after that. It’s going to be the ones beyond.
The point that I’m making here is that the kind of things we used to only talk about at science fiction conventions like Comic-Con are becoming a point of incredible importance at places like the Pentagon. We’ve got to figure out what it’s like to live through a robots revolution.
Every so often in history, there’s revolutionary technologies that come along. Technologies, again, that change the rules of the game. And what’s important is not their capabilities, but the questions that they force us to ask. Questions about what’s possible that we never imagined was possible before. Questions about what’s proper. Issues of right and wrong that we didn’t have to deal with before.
Why was the horseless carriage important? Its ripple effects. It’s things like the mechanisation of war. The mechanisation of society. It’s changes that it has beyond. For example, because of this technology, we have the advent of what we in America call suburbia. That is, people living outside city centres, commuting back and forth. Didn’t have that before the horseless carriage. We have changes in geopolitics. A set of desert nomads at the time move in to a dominant geopolitical position because they’re lucky enough to live over a resource that’s valuable to the horseless carriage. We have changes to the world climate itself. Climate change coming out of this technology.
The point that I’m making here is that we often in discussions focus on the technology and how it works where what’s important in history is all of these ripple effects. The ripple effects on our wars. On our world. On our ethics. On our laws.
But there’s another ripple effect. It’s how it changes our politics, our governance. There are certain social trends that are happening in our democracies right now that robotics maybe takes to their final logical ending point. In most Western democracies right now, we no longer have a draft. We no longer have conscription. We no longer declare war anymore. The last time, for example, the United States declared war was back in 1941. We no longer pay higher taxes for our wars anymore or buy war bonds. And now we have this technology that allows us to carry out acts of war without having to worry about the political consequences of that condolence letter going to someone’s mother.
So what I’m saying here is that the barriers to war in our democracies, they were already lowering, and now we have a technology that literally takes those barriers to the ground. And I don’t think this is a piece of political theory. I think it’s playing out right now in the Pakistan war.
We’ve carried out more than five times the number of airstrikes using unmanned systems into Pakistan than we did with manned bombers in the Kosovo war just ten years ago. But unlike the Kosovo war ten years ago, we don’t call it ‘a war’. And I think the answer to this riddle goes back to this technology. Because of this technology, because of the unmanning, my nation’s congress has not had to have a debate about it either to support it in a resolution or to go against it.
The media reports it very differently than it would with manned bombers, than it did ten years ago in the Kosovo war. It’s being conducted not by a military force, not by the US Air Force, but it’s being conducted by a civilian intelligence agency.
The point that I’m making here is that what we used to call war, for some reason, isn’t anymore. But there is one thing that it is. And that’s a YouTube war. These technologies don’t merely remove the human operator from risk, they record everything that they see. And in so doing, they’re reshaping our relationship to war.
Now being able to see combat footage—there’s literally thousands of video clips of combat footage up on sites like YouTube.com right now—being able to see this is arguably a good thing. We’re seeing connections between the warfront and the home front that never existed before. But of course the ability to watch scenes of real-world battle on your iPhone is turning, for a lot of people, war into a form of entertainment. And the soldiers have a name for it. They call it war porn.
I got a typical example of this in an email that was sent to me and the title under the email said, ‘Watch this.’ In this case the video clip was a scene of a predator drone strike. Hellfire missile drops. Goes into target. Explosion. Bodies riding the crest of the explosion. It was set to music. We turned an act of war into a crappy music video.
And the danger of war porn is that, it’s not just that we’re watching, but it has a warping effect on us. This ability to watch more but actually experience less tricks us into thinking that we understand what’s happening. And I think there’s a good sports parallel for this: it’s the difference between watching for example a rugby match on TV where the players are these little tiny figures on the screen versus watching a rugby match in the stadium and seeing what a professional athlete really does look like face to face, which is a very different experience from playing in that match yourself and knowing what it’s like to be tackled or something like that. These are very different experiences. But notice how when we watch the game on TV, we yell at the screen when they do something dumb, like ‘How could you do that?’ because we think we understand. But we don’t.
But YouTube war is even worse because we’re not watching the whole game. We’re watching the sports highlight reel of the war. All the context, all the strategy. It just becomes highlight reels for us to blitz through. The irony in all of this is that, again, it’s not the technology that matters but it all comes back to us. It all comes back to our human psychology.
And there’s a policy example of this that we’re wrestling with right now that sounds very science-fiction-like: What is the message that we think we are sending when we use robots in war versus what is the message that is being received? And I wanted to know this so I went around interviewing people and I thought the best example of the message we thought we were sending was from a senior Pentagon official in the Bush administration who described how our, quote, ‘Unmanning of war plays to our strength. The thing that scares people is our technology.’ But what about when you go ask those people 7000 miles away? And this is what the leading newspaper editor of Lebanon had to say. And while we were speaking there was actually a drone flying above him at that moment. And he said, quote, ‘This is just another sign of the cold-hearted, cruel Israelis and Americans who are also cowards, because they send out machines to fight us. They don’t want to fight us like real men but they’re afraid to fight so all we have to do to defeat them is kill just a few of their soldiers.’ I’m not saying that’s accurate. It’s perceptions. And they’re perceptions that are going in completely opposite directions. And the danger of course is in a counterinsurgency in a war of ideas, perceptions do matter.
Alan Saunders: On ABC Radio National you’re with The Philosopher’s Zone and we’re listening to Peter Warren Singer from the Brookings Institution in Washington talking about the moral implications of robot wars and the ripple effects of changes in technology.
Peter Warren Singer: Another ripple effect though is the changing meaning of going to war. For the last 5000 years, whether you were talking about the ancient Greeks going to war against Troy or my grandfather going to war against the Japanese in the Pacific in World War Two, this meaning of ‘going to war’ really meant the same exact thing. At its fundamental level it meant going to a place of such danger you might never see your family again. When you went to war you were taking a risk that you might never come home again. That’s what it’s meant for the last 5000 years. Until today. And this is what a Predator squadron commander described of what it was like to fight insurgents in Afghanistan while never leaving the state of Nevada from his base just outside Las Vegas. Quote: ‘You are going to war for twelve hours. You’re shooting weapons at targets. You’re directing kills on enemy combatants. And then you get in the car and you drive home. And within 20 minutes you’re sitting at the dinner table talking to your kids about their schoolwork.’
This is a fundamentally different experience of going to war. And actually it’s turning out to be a very tough one. These units of remote warriors actually have, in certain cases, as high and in other cases even higher levels of combat stress and fatigue than the units physically in Iraq and Afghanistan.
Another ripple effect of this plays out in terms of the demographics of war. One of my other favourite stories is about a high school dropout. He dropped out of high school and he wanted to make his dad proud of him again. So he enlisted in the US army. The recruiting sergeant looked at his high school transcript and he asked him, ‘What would you like to be in the army?’ And he said, ‘I’d like to be a helicopter mechanic.’ Looked at the transcript and said, ‘I’m sorry son, you’re not qualified to be a helicopter mechanic because you failed your high school English class. Would you like to be an unmanned aerial systems operator instead?’
He turned out to be incredible at this. He turned out to be a natural. It wasn’t that he was a natural, it’s really that he’d spent his entire life training for this by playing video games. He turned out to be so good from it that they brought him back from his first deployment. They promoted him. They made him a specialist, which is effectively the second lowest rank in the entire US military. And then they made him an instructor in the pilot training academy.
Because of this technology, this young man found himself. He made his dad proud of him again. He’s serving his nation. I told this story at last year’s US Air Force national convention. They didn’t like this story of a 19 year old high school dropout who’s not just an instructor in the pilot training academy right now but has taken out more targets, saved arguably more American lives, than every single F22 pilot combined. The jet fighter pilots look at him the way the knights looked at the peasants just when the peasants were given guns.
Now robotics is not just little things that move beyond us. They are also changing our own bodies. A good illustration of this is a couple of months ago it was revealed that a British SAS trooper had had laser eye surgery. So what? What’s the big deal about that? Anyone in this room had laser eye surgery? Raise your hand. Okay, so let’s compare his laser eye surgery to what you got. Did yours allow you to see 400 metres at night?
Basically we gave him the version of a cat’s eye’s vision. Much of what you’re hearing hear is that there are always two sides to every revolution. We’re getting incredible science fiction-like capabilities, but we also have incredible science fiction-like dilemmas to figure out. These dilemmas sometimes evolve from very small things. From what a vice-president of a robotics company called ‘oops moments’. When things don’t work out with your robot, it’s just an ‘oops moment’, he said.
What are examples of ‘oops moments’ in war so far? Well, sometimes they’re kind of funny. Like, for example, when they tested out a machine gun-armed ground robot. It looked a little bit like a lawn mower with a 50 calibre machine gun on top of it. When they did a demonstration of it for a group of VIPs, it went, quote, ‘squirrely’. It started spinning in a circle and pointed its weapon system at the review stand of VIPs. They were very happy that there were no bullets in the machine gun at the time.
Other times ‘oops moments’ can be tragic. Just about two years ago in South Africa an automated anti-aircraft cannon had a, quote, ‘software glitch’. The cannon, which was supposed to fire upwards into the sky during a training exercise, instead levelled and it started firing in a circle. It killed nine soldiers before it ran out of ammunition. And the point is not just that this happened, but ‘How do we respond?’
Imagine if you were the young investigator who was asked to resolve this issue. What system of laws would you turn to for guidance? Because what he have playing out here is 21st century technology like a Reaper Drone that can take off and land on its own. That is smart enough that if it sees footprints in a field to backtrack those footprints. We have this 21st century technology being applied against 21st century actors in war. Like an insurgent who hides out in a house surrounded by women and children. Not because they’re ignorant of the laws of war, but because they’re deliberately violating them. So you have these two 21st century poles and the laws of war from the 20th century caught in the middle.
So, for example, the most important technology to come out the year that the Geneva Conventions were written was actually the 45 rpm record player. It’s a lot to ask of a law that’s so old to keep up with this technology. And actually when I was at Human Rights Watch doing interviews there, two of the leaders I asked them, ‘Well, what system of law do we turn to if a Predator Drone strike goes awry?’ And they got in an argument in front of me rather than answering me directly.
One of them was saying, ‘It’s the Geneva Conventions like we’ve always looked to.’ And the other one argued back to them, ‘No no no, the Geneva Conventions really don’t apply in this circumstance. We should turn to the Star Trek Prime Directive for guidance.’ And he was serious. Now, I’m a Trekkie. I love it. But the problem is we can’t actually call Captain Kirk as a real expert in a real court of law.
And so, in ending, it sounds like I’ve been talking about the future. But notice how every single example I gave you, every single picture you saw, is not from the future. The dangerous idea is that it’s actually from our present. And this sets an incredible challenge before us. Are we going to let the fact that this looks like science fiction, feels like science fiction, keep us from facing the reality of not only technology but the reality of war today?
So, in ending, I’d actually like to jump from reality into science fiction. What was fascinating to me in the research was finding that the perspectives on robotics are very distinct when you go around the globe. So in Western culture, the robot has almost always been the mechanical servant that wises up and then rises up. You know, it’s everything from the very first mention of the word in the play R.U.R. back in the 1920s to the Terminator movies. But if you go to East Asia—if you go to Japan, Korea or the like—the robot is actually usually the hero in things like anime.
So, what we’re seeing is actually very different attitudes towards things like the arming of autonomous weapons systems. And this may be the most dangerous idea of all.
A couple of years ago the American Film Institute gathered all of its members together and asked them, ‘What were the hundred top heroes and the hundred top villains out of every single movie ever made?’ That is out of every single character and every movie ever made, which characters represented humanity at its best and which characters represented humanity at its worst. And only one character made it onto the top 100 hero and the top 100 villain list. What do you think it was? The Terminator. A robot killing machine.
And I think that illustrates not only the duality of technology itself, that it can be used for both villainous reasons and heroic reasons, but it illustrates the duality of us, the people behind it. What has distinguished our species has been our creativity. We’re the species that created fire. We’re the species that created technology that took our species to the moon. We’ve created incredible works of art and architecture to describe our love for one another.
And now we’re doing something incredible. We’re literally creating potentially an entirely new species. If you believe both the scientists and the science fiction authors out there, that’s what they think we’re doing. But, if we’re being honest about it, the reason that we’re doing all this is just to get better at destroying one another. And so the ending question is this: is it our machines or is it us that’s wired for war? Thank you.
Alan Saunders: Peter Warren Singer, Senior Fellow and Director of the 21st Century Defence Initiative at the Brookings Institution, talking at the Festival of Dangerous Ideas in Sydney. For more on him and on his book Wired For War check out our website abc.net.au/rn/philosopherszone. And that’s also where you’ll find a link to our sister show All In The Mind’s discussion of robot wars. The show’s produced by Kyla Slaven, Charlie McCune is the sound engineer, I’m Alan Saunders and I, not the robot me, will be back next week.
