Unintended Consequences

Image: Maersk

In the quest for next-generation technology, one thing is often pushed aside in the race for the cutting edge: a hard look at the unintended consequences. Autonomy and artificial intelligence, cornerstones of the Defense Department’s Third Offset strategy, run this risk with the defense and security community. This is where Pure Risk, the winning entry from The Art of the Future Project’s “Third Offset” short-story contest comes into play. Jason Hansa wrote Pure Risk as a way to explore what might happen if wartime exigency removed the caution and boundaries constraining the use of autonomous weapons systems under the sea.

Max Brooks, Atlantic Council non-resident senior fellow and author of World War Z, wrote of Hansa’s story that “no one to my knowledge has fused imagination and hard data into such a scarily credible future. And as exciting as Pure Risk is to read, the story’s deeper value lies in its ability to jump-start a broader conversation on the unintended consequences of unmanned warfare.”

A prolific writer, Hansa is an active-duty Major in the United States Army who has deployed to both Iraq and Afghanistan.  He also served in both South Korea and Germany, and is currently assigned as a CASCOM doctrine developer at Fort Lee, Virginia. He has previously published several science-fiction stories on www.battlecorps.com, including Three Points of Pride and Irreplaceable.

The following are excerpts from an interview with Hansa in which he discusses Pure Risk with The Art of the Future Project:

What inspired this story?

I can’t really point to one thing in particular, except maybe Shattered Trident by Larry Bond. Set in a “now” period, it’s the entire Pacific versus China. It’s a fun read, and it stuck in my head. I wanted to do something in the same region, and I really wanted to highlight the fascinating designs and studies that are being done with submersible drones these days. As it turned out, I didn’t highlight any “current” tech, because this is set so far in the future.

Science fiction in the “near future” is a difficult beast: too advanced, readers call you on it; while if you’re not advanced enough, you can be out-of-date by publication.

The “unexploded mine” idea was just one of those shower ideas that you get after a couple of days of grinding on a story/scene – you let your mind wander, and process in the background, and suddenly, it hits you. I was really surprised it caught on like it did. I thought, “There’s no way they haven’t seen this before – they’ll call it cliché and send me a polite rejection email.” The fact I wandered into something so dangerous, obvious, and scarily probable is both exhilarating and kinda scary: what other sneaky tricks will our adversaries think up?

When you sat down to write Pure Risk, what kind of impact were you aiming for?

I wanted two separate reactions, one of which I hit, one of which I may have missed the mark on. For the first, I wanted readers to say, “Wow, this could easily happen.” The idea of rogue Chinese drones attacking randomly years later – it happens often enough on land with mines that a reader could instantly see this happening and wonder what the consequences were. I think I hit this mark. Again, it’s a little scary how easy this might become real. There’s a lot of choke points where a dedicated belligerent could make swaths of ocean impassible: the South China Sea, specifically the Malacca strait; the Persian Gulf; the Caspian Sea; and the Baltic Sea are some obvious ones. But even out-of-the-box ideas like outside the Suez or Panama Canals, or across the Cape of Good Hope would provide global effects at low-cost. The more you think about it, and look at areas on the globe where the water is narrow, the more possibilities you see.

The second mark I wanted to hit is how unprepared current technology is to deal with the Third Offset. I killed an entire fleet with an incredibly cheap swarm of drones. We have massively vulnerable platforms that are incredible expensive and nigh on irreplaceable. If we lose a carrier, we’re down a carrier. There’s no “backfill” for platforms like that.

What’s the plan? Do we go smaller, cheaper, lighter? Do we go for quantities over quality? How do we deal with waves of suicidal drones? What’s our defense against the swarm? These are valid questions I’ve seen asked, and I wanted to show, “Hey, if we don’t answer them, soon, people will die.” But I think because the scene was tucked into a sidebar, it got a little overshadowed by the drone-hunt.

Shed some light on the research that went into it.

I read a lot. I try to keep up on various blogs, military journals, etc. So I had a lot of the concept and ideas in my head, I just had to lock down specifics.

Now, specifically, I researched: the composition of the Vietnamese navy, the US fleet naming convention, minesweepers available for Pacific Rim nations, Google and the surrounding area, international time-zones to ensure the articles were all in chronological order, and finally, insurance terminology to make sure the emails had the right feel.

It was the final one that gave me the title. When I was crafting the final email, the piece was still called “Drone Story.”

How did you decide to use an email memo format along with news reports, and how did that change your portrayal of the story than if you’d written it from a single character point of view?

Originally, this story was going to be focused on just the ship and crew of a minesweeper. But then, I realized I needed to add the “so-what.” What’s driving them to be out there, what are the consequences? And, I wanted to make it a global issue, but, sketching it out, I know the global implications would eat a lot of words. I knew that something would lose out — either characters, or the world, or probably both. I had to pick, and I wanted the larger “world” more than I wanted a single minesweeper out alone and unafraid.

Converting the story from a traditional story to an email (which it is, all the news stories are attachments) allowed me to give massive info dumps to the reader and build the world. The email to open the story served two purposes: first, it gently introduces the reader to the world. “Bad things are happening, read on for specifics.” It set the mood. Second, it was designed to strike a chord with readers. Almost everyone has had to send a “Sir, FYI” type note to the boss. Writing it as an email similar to ones readers have probably experienced hopefully gave them a subconscious connection to the situation.

I put a lot of effort into the visual format so it wouldn’t look like a “story.” The formatting, I think, also helped provide a visual context for the readers: though they knew they were reading a story, their eyes believed they were actual newspaper clippings, pulling them further into the world.

What is the path from today’s tech sector’s engagement in defense matters to the one you portray?

This, I feel, is a natural one. I’m not the first author to make a connection between US businesses supporting the government in a time of war – notable examples include Debt of Honor and Ghost Fleet – but Google is slightly different. The US Defense Department (and intelligence agencies) has had a hard time pulling in the best and brightest in the computer science fields. Partially it’s because of the money, partially (as was accurately shown by Matt Damon in Good Will Hunting) because some people at the tops of their fields don’t necessarily play well with others, including the government. The leap from “hiring the best” to “subcontracting the best” is a short one; in the specific fields of computer programming, especially, it would be a way for the government to get the necessary talent while maintaining a buffer between the US military and the individual programmer.

Issues arise on the business side – how many customers would Apple lose, for example, if they sub-contracted for the NSA? Would their customer base truly believe in their independence? I didn’t explore the ramifications of Google’s cooperation; but Google’s motto is “Don’t be evil,” and I imagine working with the federal government on anything but the most benign of projects would have financial consequences.

How do you rate the odds of the unexploded ordnance phenomenon carrying over to the autonomous robotics realm?

This is where I have to throw a huge disclaimer up: these are 100% my own thoughts. I’m an active duty Army officer and a CASCOM doctrine writer, so it would be a very easy to believe I’m giving insight into the Army’s vision, or this is a peek into “doctrine of the future.”

I made up all of Pure Risk, and everything that follows is 100% Jason-the-science-fiction-geek speaking, not me as MAJ Hansa.

That out of the way, I believe the odds approaches certainty. There’s simply no way autonomous robotic hunter-killer robots won’t be invented, certainly within our lifetime. The “human in the loop” requirement will disappear before long, because it’s dumb. Point blank.

This goes back to culture. For our generation, humans-in-the-loop is a culturally-driven requirement because, when we think of machines intelligent to decide whether to kill a human or not, we think of a T-100 going back to kill Sarah Conner, or a computer named Joshua trying to launch ICBMs. Our generation has a cultural block that prevents us from accepting the idea that robots not only can kill, but we need to trust them to do it.

But our children have no such qualms: today’s kids, playing “Sprocket” (Skylanders) or Torbjörn (Overwatch) have no problem dropping a turret and leaving it there to cover their back while they complete the mission. The idea of not trusting the drone – the turret, the fighter, etc. – will come as an incredibly old-fashioned and out-of-touch concept for the so-called “Generation Z.”

More importantly – and it seems like an intuitive leap, but it’s not – other nations don’t have the same cultural touchstones. Every country has its own boogieman; the histories of past victories, defeats, and cultural shifts that drive their military procurement and doctrine. And very few of our adversaries are deeply worried about killer robots: they’ll let the drones take over the moment the programming can support it. The first truly autonomous drones will out-move, out-fly, and out-shoot any of their peers, and everyone else will be playing catch-up.

Long story short: autonomous hunter-killers are coming, and since no self-destruct is infallible, some will continue to kill long after the war has ended.

Do you foresee similar risks on land?

Absolutely. There’s two large constraints that dictate air and sea drone design: environment and power. Both the sky and the sea are unforgiving: gravity and physics ensure that any problems aloft are generally short-lived ones; underwater operations have both of those limitations plus the added complications of pressure and buoyancy. Systems built for land operations generally have none of those issues.

The second issue is power. Power storage, and/or generation, is the single biggest roadblock currently slowing the development of so-called exoskeletons, full suits of powered body-armor, and drone swarms. Land drones will have it a lot easier than their air or sea cousins: no energy spent to remain aloft or to shift depths, and with potentially limitless reserves if they are solar or wind powered.

Finally, people are sneaky, and they’ll program land drones the same way. There are so many places to hide an explosive on land, as we found out to our detriment in Iraq: drones smart enough to change hides if they’re at risk of discovery might continue to reappear and kill for years.

What about cyber UXO?

For the most part, I don’t believe it’s going to happen, despite what I wrote into Pure Risk and my earlier paragraph about Google assisting the DoD. Anything that can be hacked, will be hacked. I put that into the story for dramatic purposes, but no nation will be that stupid. Let me rephrase. One nation will be that stupid, and will suffer for it – their drones will be deactivated, or worse, turned on their own troops, and the rest of the world will immediately deactivate all remote access on their equipment.

This is, again, a mistake we currently make because, culturally, we’re still on the learning curve. We experience “beta versions” of software getting pushed out by companies; we expect our laptops to automatically pull and install updates. Our kids won’t make this mistake. After spending their entire lives in the public eye, they won’t be dumb enough to leave a hacker an opening.

Now, that’s not to say that the drones won’t communicate between themselves: they will, somehow – laser LOS, short ranged Wi-Fi that’s GPS tagged to not allow outside signals in, etc. But the smart money says drones will be capable of receiving orders and transmitting data, but their programming will be hardwired in. I’m sure they will have some system to receive system updates, but it will probably be via a physical cable, for example, to limit their vulnerability.

This does leave open the possibility of someone being able to sneak up to a drone on land and plug in to hack it, but, while making for dramatic short story, the most likely and efficient solution for a rogue land-drone will be a shaped charge.

Again, this is all Jason talking, not MAJ Hansa. I’m a history major, not an engineer, I wouldn’t know a design specification if it bit me. But I know my kids and my nephews, and even as tweens, they wouldn’t connect anything to the Internet that they’d rely on to save their life.

Describe your work flow and how writing fits into your daily life?

We have unit PT, work call, etc. Standard stuff. The thing that eats the most into my writing time is the kid’s homework – there’s been a lot written about the amount kids these days get, and it’s all true. I try to get an hour of writing in a day, but usually, I get a couple hours of writing in on an erratic schedule. I get more writing in during the summer when there’s no homework; weekends are almost a no-write zone because I try to focus on family time.

What are you reading, playing, watching or writing right now that you can’t put down?

Because the school requires 20 minutes of reading per day, it’s become “family reading” time, and I’ve finally been able to start knocking books off my pile. I just finished Isaac’s Storm, and am now reading Finest Hours by Michael Tougias and Casey Sherman. I try to balance “light” books with “hard” history, so I’m also reading Rising Sun, Falling Skies by Jeffrey Cox. Next up is The Dead and Those About to Die by John C. McManus, and my buddy Mike McCool’s Sleepernet.

We record stuff during the season and catch up during hiatus, so we’re a full season behind on Madam Secretary, Gotham, and Sleepy Hollow, but we’re caught up on Castle and Legends of Tomorrow. Light stuff, mostly; we read a lot and watch TV when we want to turn our brains off for a bit.

I still play “BattleTech” every other weekend; my kids and nephews love “Exploding Kittens” (the version with family-safe artwork) and “Sushi Go!,” I recommend those to other parents. They’ve also taught me how to play the Pokémon card game, so I tag along with them on league Sundays and run a “Snorlax attack deck.” Apparently, from what the local experts tell me, I might be the only person in the world running said deck at league events. I’m surprisingly OK with this.