Should we worry about Skynet?

Sep 11

ARTIFICIAL INTELLIGENCE AS AN EXISTENTIAL THREAT

AI DOOMSAYERS

It has become fashionable in recent times for doomsayers to call AI an existential threat, by which they mean something akin to the malevolent Skynet system in The Terminator movie franchise that goes rogue and launches nukes to annihilate humanity. I know regular people worry about this too, because one of the most frequently asked questions when I present on AI, going back at least six years, has been, Should we worry about Skynet? Yes, really.

So, is such a nightmare a probable part of our future?

In this post I’m going to run through some of the logic behind my answer. And a warning: it’s going to get a little dark, so strap yourselves in.

SKYNET PREREQUISITES

The nightmare requires at least three things to come true:

(a) AI will exceed human intelligence.

(b) A superintelligent AI will want to kill us all

SUPERINTELLIGENCE

As I’ve said previously, we can neatly side-step the philosophical hand-wringing about singularities and what intelligence is and whether AI will become conscious or not by simply accepting that AI is already, for practical purposes, indistinguishable from human intelligence in select narrow contexts, and will continue to exceed human capabilities on ever-broader fronts. I’m calling (a) inevitable.

THE MALEVOLENT AI

The malevolent AI (usually embodied in a robot) is as old as sci-fi itself. It connects deeply to our fears because it is mechanistic; once unleashed it cannot be reasoned with, there can be no appeals for mercy because it has no mercy, or indeed any emotions to appeal to. Even a hyena or shark has the capacity to change its mind, if only for self-preservation. The robot keeps coming. The lack of a soul chills our soul. We fear it intuitively, and writers and producers know we do, which is why killer AIs and robots make for such riveting movies.

But in the real world, no matter how smart AIs get, it does not follow that they will spontaneously metamorphosize into little Hitlers. It does not follow that they will develop survival motivations, or any notion that it might be a grand idea to start a genocide because humans ‘stand in the way,’ or are ‘inferior beings,’ or just plain suck. Further, NO ONE, so far as I am aware, has EVER published any science suggesting such a pathway. If you see any, let me know!

So (b) is hypothetical in the extreme, almost a non-starter.

But hey, we can’t completely rule it out can we? So, for the sake of argument let’s keep going and say malevolent traits do one day emerge. Which brings us to the crux of it (c).

To help break this down, let’s introduce the concepts of agency and what I call the reaction blind spot:

AGENCY

I’m talking here about the ability to take action or to choose what action to take.

We often delegate agency to machines.
Machines can sometimes seize agency, for example when viruses infect machines.
Sometimes power transfers more subtly and gradually via what I call unconsciously abdicated agency. I think of the way many of us allow various electronic media to decide what books we read, what news we consume, and so on. By habit and routines, we give away agency without really making a single conscious decision to do it.

There are probably more variations, but these are the ones I think about.

For an AI to be an existential threat, it would need some pretty serious agency.

LIMITS TO DELEGATED AGENCY

We love to delegate agency to machines. We delegate agency for convenience. We delegate to avoid tedious and small tasks. We delegate agency where it will make us more money. As Yann LeCun points out, we already delegate thousands of tiny decisions to AIs every day and think nothing of it. We will certainly be delegating many many more decisions to AIs in future.

Conversely, we do not rush to delegate agency for decisions where large numbers of people might be killed. We tend instead to keep humans in the loop. The greater the risk, the keener we are to keep humans in the loop. We get scared. Our fear protects us. This is the case in airliners. This is the case even in Japan’s ultra-reliable and ultra-automated railways. This is certainly the case in nuclear power plants. This is definitely, definitely the case with nukes, which I’ll get into later.

We already fear the malevolent AI. Almost everyone on the planet is already conscious of what they don’t want to delegate.

LIMITS TO SEIZED AGENCY

The AIs in our scariest movies seize agency. Okay, but in real-life, how much can they really seize?

The past, present, and future of infrastructure controllers is specialized, testable and stable (non-evolving) algorithms, not evolving AIs. The environments are heterogenous, which is to say, different controllers from different manufacturers are installed for different applications, for commercial reasons if nothing else. We do not have one type for all.

Furthermore, externally introduced viruses and malware are already dangerous. They cause damage. In certain types of infrastructure, such as electricity grids, they can shut down and kill. Viruses and malware have already taught us we have plenty to fear from seized agency.

So as a basic principle, we already compartmentalize damage-potential, both by chance and by design.

We get it wrong sometimes, of course, and doubtless, bad actors will try to combine AIs with viruses to make them more … virulent? But the notion of an all-powerful self-directed amoeba-like malevolent AI insinuating itself across enough controllers over enough infrastructure to perpetuate genocide? An uber-virus with a God-complex, if you will? Try as I might, I can’t see it.

I’m not saying errant AI won’t lead to loss of life once in awhile (as can happen when any critical software fails), and if that’s where the doomsayers stopped, I think we could have an interesting, rational discussion, but they keep saying existential.

THE REACTION BLIND SPOT

To my mind the most consistent deficiency in ALL doomsayer predictions on ALL subjects is the tendency to forget human response.

We are not cardboard cutouts. Our responses may be piecemeal and tardy (climate change!) and they may be surprisingly good and far-reaching (international bans on chlorofluorocarbons, removing lead from gasoline) but we do react, and the future unfolds as a multi-shaded tapestry of both our actions and reactions. No honest accounting of the future can include one without the other.

We already fear the malevolent AI. We are already reacting to it in the ways I described above. Everywhere you look in 2023 there are committees and enquiries and debates on the subject. This article too, is part of the reaction.

We will most certainly escalate our reaction if and when we feel more directly threatened.

REACTING TO THE ROGUE AI

Let’s imagine malevolent tendencies do emerge in an AI, sometime, somewhere.

If it happens in a lab, we’ll know about it. Why? Because the scientists involved will be screaming from the rooftops to be sure of collecting their Nobel! And we won’t be giving that flavour of AI any kind of agency over our dams, trains or power stations, will we?

If it happens in a commercial setting, we’ll spike it pretty darned quick. There is always an off-switch! And don’t give me any of that malarky about ethics. Some people think we’ll be assigning rights to AIs and we’ll be loath to pull the plug, conveniently forgetting that we’ve never hesitated to kill ANY life form, human or otherwise, that threatens us. Nor, for that matter, have we ever hesitated to kill any life form that offers more commercial value to us dead, or merely gets in the way of a new shopping mall project. If uber-intelligent aliens arrived from Sirius Major and threatened us, we’d kill all of them too. Believe me, if can do all that massacring of carbon-based life forms, we won’t so much as blink before pulling the pin on the software-plus-silicon variety!

BATTLEFIELD AI

One place where we do delegate agency, and we do it for the specific purpose of killing, is on the battlefield. Here, the killer robot is already a reality, and we will inevitably incorporate more and more AI because weapons that can make their decisions autonomously must logically be quicker on the draw. Winners on the battlefield “get there the firstest with the mostest,” as the generals like to say, and they shoot first too.

No amount of ethical debate or international negotiation will stop the nations racing one another for more capable autonomous weapons. As one side does, so the other must follow. On the tactical battlefield at least, delegated kill decisions without a human in the loop will be commonplace.

Commonplace, but not default.

Our self-preservation instincts still jangle. Soldiers and generals still fear the robot gone wrong – the glitchy sensor, the blue-on-blue attacks, the recriminations over dead civilians – and they try to keep humans in the loop wherever they can. Witness the machinations over drone strikes today where humans watching cameras are almost always kept in the loop, even for strikes on a single soldier. As we move ‘up the stack’ to bigger, deadlier weapons, the consequences grow and the operators are even more loathe to remove humans from the loop. Destroying the wrong tank is bad. Taking out the wrong convoy is worse. Raising the wrong village is incomparably worse again. Not to say it won’t happen, but the probabilities of delegated agency to an AI reduce and reduce.

Then there’s the power thing. When it comes to truly strategic weapons, even the worst and most hateful despots resist ceding control. Come to think of it, especially the worst and most hateful despots.

So again, the bigger and more existential the threat, the tighter the grasp.

Which brings us, finally, to nukes.

NUKES AND AI

Nuclear weapons ARE an existential threat to humanity, in the fullest sense: a single launch has the potential to trigger all-out war and an annihilation of homo sapiens. There have been many close calls. At this moment, approximately 12,500 warheads across 9 nations stand locked and cocked and ready to go. The doomsday clock stands at ninety seconds to midnight for a reason.

For an exercise in sober contemplation of the precariousness of human existence, I strongly recommend a visit to the Minuteman Missile National Historic Site in South Dakota or better, if you have time to get to the more remote location, the Oscar-Zero Minuteman Missile Alert Facility in North Dakota. At both sites you can descend into the underground launch control centres. At both you can view up close the control desks, the keys, the buttons [see the photo I took, above].

I promise you will come away a much more sober person than you arrived.

There is another reason for visiting. At both sites you will get an appreciation of the multi-stage and multi-authenticated and multi-human steps embedded in the launch sequence.

And this, remember, after many decades of embracing Mutually Assured Destruction (MAD) as a deterrent doctrine.

In our darkest nightmares we imagine authority delegated to AIs to auto-detect a launch from one side and auto-launch in response. But with ICBMs distributed geographically across silos and submarines, a devastating counterstrike is guaranteed. MAD still applies. There’s no need to auto-launch. So, we don’t. In fact we do the opposite. We insert more humans. We compartmentalize more. Hey, the US missile command only stopped using eight-inch floppy disks in 2019 — there’s a reason for that!

Bottom line, I see no one delegating agency over nuclear launches to AI.

Could they? Well, it would need multiple cooperating layers of utter stupidity, but hey, Daniel Ellsberg chronicled some pretty outrageous examples of stupidity in US weapons handling, and during the Cuban missile crisis Moscow made an outrageous delegation of (human) launch authority to local battery commanders – no phone call to the boss necessary – which placed us a hair’s breadth from Armageddon, and we can assume similarly chequered histories in other nuclear powers, so I guess they could. Nothing’s impossible. But for my money the probability of AI being given agency in any decision-chain is small enough as to be not worth losing sleep over.

I’m not saying any of this is good. By any objective measure, what we have now is insane! What I am saying is, no one is delegating direct launch control to an AI. No one, not even your worst dictator, wants to cede that kind of control. No one wants the mistake because no one wants the response. And if a mad dictator decides to trigger global war and turn everyone to ash, himself included, he will still want to do it on his terms and at a time of his choosing.

COULD AI TRIGGER A HUMAN-INITIATED LAUNCH?

On January 7 1983, Stanislav Petrov stared in frozen horror as his early-warning system told him that the United States had launched a ballistic missile strike at the Soviet Union. Moments later the system alerted him to additional launches. Five missiles in total, now flying skywards and northwards towards him. Could it be? Could it be? The USSR had just downed Korean Airlines flight 007. The US had moved nuclear-tipped Pershing II missiles into Western Europe. Aircraft and submarines on both sides had ratcheted up their cat and mouse games. Excruciatingly long seconds ticked by. Sound the alarm? Trigger Armageddon? Ignore and risk Soviet annihilation? Decide, Petrov! Decide!

To the eternal gratitude of every living breathing creature on this planet, Petrov decided correctly. And the thought-fragment that went through his mind, the deciding skillet of logic that saved our species, was this: “Why five?” In the awful calculus of nuclear war, when a first-strike should logically be an all-out effort to annihilate as much of the enemy’s counter-strike as possible, Why did they send only five?

We may take from this nearest of near misses a reminder that keeping humans in the loop is A Very Good Idea!

We may also take it as evidence that any computer error associated with launch-detection (in Petrov’s case, the satellite early-warning system was in error) DOES have the potential to indirectly trigger a human-initiated launch. It CAN happen.

So, if instead of asking, “Could a malevolent AI deliberately set out to nuke us?” we asked, “Could a bad AI trigger a bad human-in-the-loop decision?” then the answer must be, as it is for any software, yes.

Against that, as we’ve discussed, the fear- and associated care-factor is high, and unpredictability in installed technologies in this environment is, as a general principle, unwelcome. But the answer is still yes.

On balance, the risk to our future from nuclear weapons is already very great and the notion of a malevolent AI does not, in my opinion, make it any greater. I realize this is small comfort. Again: the doomsday clock is set at 90 seconds to midnight for a reason. Nukes are still nukes. Humans can still be idiots. But for me, it at least helps redirect my personal capacity for worrying to matters more worth worrying about.

As a postscript, instead of being lauded as a hero for saving the planet, Petrov was reprimanded and reassigned and eventually left the service. Only much later were his actions recognised, and then mostly by a very grateful west. His story is given prominence at the aforementioned Minuteman Missile National Historic Site, which he visited in 2007.

DEEPFAKES AS TRIGGER

What about broader misunderstandings? Could a malevolent AI generate deepfakes and escalate tensions between nations?

Most certainly. But note: from here onwards we are departing further and further from the Skynet scenario.

As I’ve been saying for years, deepfakes in all their forms represent the worst of the near-term threats. Deepfakes are already employed in cyberfraud and extortions. I’ve witnessed excellent demonstrations of a ‘live’ Putin deepfake where a bad actor speaks in English and the Putin avatar speaks the same sentences in Russian. Microsoft recently demonstrated an AI that can simulate anyone’s voice after sampling a mere three seconds of audio. You know those mysterious callers who hang up directly after your greeting? Can’t help thinking about a vast audio repository being built by criminals somewhere. And, despite all you hear about detection technologies and watermarking and so on, AI-generated deepfakes will be indistinguishable from the real. The inputs are hidden. The AI-engines are behind the scenes. The outputs are video and images and audio. Long-term, there will be no foolproof way of determining provenance.

Further, we know that technology can escalate tensions across an entire population. Witness the level of US political toxicity today. The escalation of divisiveness into fear and anger and outright violence. A certain level of divisiveness is not new. What’s new is the role social media echo-chambers and algorithms play by exploiting our biological predispositions: we tend to click more on what scares or angers us, more clicks generate more dollars, so algorithms show us more of what scares or angers us.

Further again, humans can get trapped into self-reinforcing decision-loops based on their own misinformation and the amplification of existing prejudices in a way that does lead to war, as we saw in the WMD hysteria precipitating the 2003 invasion of Iraq.

Of course, steering a population towards such a precipice as nuclear war is a massive undertaking, and definitely, definitely not the work of a single AI-generated video (the likes of Kim Jong Un, for example, ordering a launch — we will surely suffer this deepfake and worse). If a malevolent AI tried it, we’d probably notice, not least because the people being demonized would be doing jumping jacks and screaming ‘HEY, IT’S NOT ME!’ and at some point someone would probably pick up a phone.

If it was to happen, human-directed badness seems to me the far more likely pathway. I see plenty of humans wanting to use AIs to exploit other humans for short-term commercial gain (mostly) and for personal power and for politico/ideological agendas. Nation-states already employ deepfakes as weapons to sow political discord, and there will always be individual bad-actors hell-bent on causing chaos. In that sense – human-directed AI prioritising selfish gains over community harm – yields abundant dangers.

But again, this is not AI triggering war directly. This is not Skynet

FLASH CRASHES

On a related subject, one non-military application where I see delegated agency to AIs and AI-triggered mayhem is algorithmic stock trading. More than 90 % of all stock trading is already performed by algorithms, not humans. These trades occur at high frequency and are often tethered to fast algorithmic news analysis. They are prone to feedback loops which increase market volatility and can precipitate steep spikes and troughs. In 2007 in The Future of Business I predicted volatility was on the way because algorithmic trading systems are “designed to give competitive advantage, not to consider the best interest of the market as a whole. The complexity of algorithmic trading will increase, and unanticipated reinforcing effects will become increasingly chaotic. Neither the risk insights nor the intervention options available to regulators will keep up with the complexity of the market.” Three years later on May 5, 2010 — boom! We had the infamous Flash Crash on the Dow Jones, and it took regulators five years(!) to trace the cause back to algorithms which were manipulating the market by placing large quantities of orders with no intention to execute.

I noted earlier that one reason we delegate agency to machines is when it makes us money. This industry is driven entirely by money. And in this industry, speed trumps all; delegated agency is not simply desireable, it is mandatory. Except for a token role tripping the brakes after a derailment, humans cannot be inserted into the loop. Furthermore, this industry strives to maximize opacity, is already embracing AI aggressively, will always use the smartest AI it can lay its hands on, and will definitely push it to the max. Put it all together and we are certain to see more AI-triggered crashes in future.

A rich elucidation of the nasty possibilities, including AI manipulating the market as well as trading on it, can be found in the excellent novel The Fear Index (2011) by Robert Harris.