Autonomous Weapons

Autonomous Weapons: an Open Letter from AI & Robotics Researchers

Autonomous weapons select and engage targets without human intervention. They might include, for example, armed quadcopters that can search for and eliminate people meeting certain pre-defined criteria, but do not include cruise missiles or remotely piloted drones for which humans make all targeting decisions. Artificial Intelligence (AI) technology has reached a point where the deployment of such systems is — practically if not legally — feasible within years, not decades, and the stakes are high: autonomous weapons have been described as the third revolution in warfare, after gunpowder and nuclear arms.

Many arguments have been made for and against autonomous weapons, for example that replacing human soldiers by machines is good by reducing casualties for the owner but bad by thereby lowering the threshold for going to battle. The key question for humanity today is whether to start a global AI arms race or to prevent it from starting. If any major military power pushes ahead with AI weapon development, a global arms race is virtually inevitable, and the endpoint of this technological trajectory is obvious: autonomous weapons will become the Kalashnikovs of tomorrow. Unlike nuclear weapons, they require no costly or hard-to-obtain raw materials, so they will become ubiquitous and cheap for all significant military powers to mass-produce. It will only be a matter of time until they appear on the black market and in the hands of terrorists, dictators wishing to better control their populace, warlords wishing to perpetrate ethnic cleansing, etc. Autonomous weapons are ideal for tasks such as assassinations, destabilizing nations, subduing populations and selectively killing a particular ethnic group. We therefore believe that a military AI arms race would not be beneficial for humanity. There are many ways in which AI can make battlefields safer for humans, especially civilians, without creating new tools for killing people.

Just as most chemists and biologists have no interest in building chemical or biological weapons, most AI researchers have no interest in building AI weapons — and do not want others to tarnish their field by doing so, potentially creating a major public backlash against AI that curtails its future societal benefits. Indeed, chemists and biologists have broadly supported international agreements that have successfully prohibited chemical and biological weapons, just as most physicists supported the treaties banning space-based nuclear weapons and blinding laser weapons.

In summary, we believe that AI has great potential to benefit humanity in many ways, and that the goal of the field should be to do so. Starting a military AI arms race is a bad idea, and should be prevented by a ban on offensive autonomous weapons beyond meaningful human control.

signed by steven hawking, elon musk, woz, chomsky, a bunch of AI researchers/professors, etc etc

i get the feeling that not too many people care about this kind of stuff (yet) or maybe it's just hard to imagine, but i thought i'd go ahead and start the thread so that my last action before dying in Machine War IV can be updating the "first mentions" thread with my awesome first mention of autonomous weapons

If any major military power pushes ahead with AI weapon development

i would be shocked if the U.S., at the very least, has not been investing heavily AI weapon development already, but maybe i'm a weeee bit doomy

What makes autonomous weapons vastly different from piloted drones in terms of lowering the barrier to engaging conflict?

Like I feel like the only thing that would have the military replace human controlled drones with AI controlled drones is if they turned out to be much more accurate than humans in terms of identification & targeting.

http://dune.wikia.com/wiki/Arafel

I can't remember where I saw it (it may have been something drawing from Chamayou's Theory of the Drone) but as I understand it drone warfare is already a significant way towards this, through algorithmic recognition of likely targets that can't necessarily be determined by human viewers etc. In those cases it seems it's almost only the button press that launches the missile standing in the way of full AI automation.

it's amazing how these guys are always worried about the wrong problems

What would be the right problems for AI experts to be worried about if not the plausible use of non-human targeting systems to kill loads of people?

steven hawking, elon musk, woz, chomsky

are each a lot of things but "AI experts" would not be at the top of their cv, I don't think

yeah obv the real AI threat is the basilisk

I just think it's funny/sad how all these rich nerds are constantly worried about totally far-fetched/unlikely sci-fi movie threats - meteors! robot weapons! the "Singularity"! we have to build the internet *in space* right away! - and not like, shit that actually is a threat to the vast bulk of humanity like say climate change, vast income inequality, mass extinction events etc. (tbf some of these guys are worried about some of those things, but put them all together and it's "oh no what if we all get wiped out by nanobots!", it's ridiculous)

climate change, vast income inequality, mass extinction events etc.

a lot of those things won't matter after we've uploaded all human consciousness to the singularity. then the only issue will be finding indestructible hardware or how to keep necessary flesh custodians from unplugging the machine.

It's not remotely far-fetched given that we are 90% of the way there already, as Merdeyeux has pointed out. The list of signatories includes the heads of the AI / Comp Sci / Robotics departments at Berkeley, Stanford, Harvard, Oxford, Cambridge, etc.

https://www.youtube.com/watch?v=goPki_V34xA

i bought this on dvd for my best friend's birthday, lol

It's not remotely far-fetched given that we are 90% of the way there already, as Merdeyeux has pointed out

yeah I call bullshit on this. Drones don't function without human operators, and it's all about target ID'ing and judgment calls, which computers are nowhere near being able to accomplish

I've been hearing AI is *just around the corner* my entire life, and people were certainly worried about such a possibility long before I was born

Removing the judgement call is the issue. There is no obvious technological barrier to simply using pre-set criteria to trigger an automatic response that I can see.

there's no way we are close to the technology required of an AI to correctly and authoritatively identify a potential target, determine who they're with or what they're doing or whether or not current situation presents an ideal opportunity to attack, etc. Humans can't even be counted on to make these decisions - drone operators are constantly killing innocents as I'm sure you're aware. The idea that we're "on the verge" of having tech do that for us is some juvenile fantasy nonsense, it ignores all the things that go into, say, Obama determining what the odds are that someone on the US's "kill list" is actually in a given vehicle convoy and the risk of innocent life is worth dropping the bombs

sorry this type of thing gets me really irritated, the assertion that something as complex and inherently human as a "judgment call" is somehow easily quantifiable and artificially replicable - it isn't and it never will be

lol just a bunch of "rich nerds", nothing to see here

chomsky is actually pretty important w/r/t AI, with his work on formal language theory. it's a bummer that his kooky anarchist shit defines him for most people but it's his own doing, i guess.

it's not at all 'easily quantifiable' but it is becoming more and more artificially replicable, insofar as those human judgement calls have so many calculable elements themselves - determining targets is often based on not immediately visible things like patterns of movement, and the algorithms are increasingly learning what the human operators base their judgement calls on

i think it's not always (maybe not often) that the drone operator will see an image of a bunch of people with rocket launchers and decide those are the guys to shoot now, they can be making decisions based on the way a few pixels have been dancing around their screen for the last couple of hours

this isn't to say that it's of the utmost importance that we drop everything and make sure we protect ourselves from skynet right now, just that there are elements of this it's worth being concerned about

the thing tho is either a) human controlled drones are more accurate than AI controlled drones in which case we will continue to use AI controlled drones or b) AI controlled drones are more accurate than human controlled drones in which case we want to switch over bc it'll limit civilian casualties further. i don't see any ethical problems w/ AI controlled drones that weren't already introduced w/ human controlled drones.

er we will continue to use human controlled drones*

https://www.justsecurity.org/20825/autonomous-weapons-developed-secrecy/

While defense companies are pursuing advanced stealth systems, electric weapons and protected communications, companies focused on producing consumer goods and business-to-business services are driving many other key enabling technologies, such as advanced computing and “big data,” autonomy, artificial intelligence, miniaturization, additive manufacturing and small but high density power systems. All of these technologies – largely evolving in the thriving commercial computing and robotics sectors – could be exploited to build increasingly sophisticated and capable unmanned and autonomous military systems.

The fusing of these technologies is likely to lead to an incredible increase in capability from a military or intelligence agency’s perspective. (This is tacitly acknowledged in a November 2014 memo from the Pentagon’s top weapons buyer commissioning a study to find ways to “reduce or eliminate” barriers to DOD’s use of autonomous systems.) Therefore, anyone developing such weapons will likely want to keep their most potent new capabilities as secret as possible out of the natural desire not to reveal them to potential enemies, at least until there is little anyone can do about them.

Yet it’s imperative that such new weapons are developed with the highest possible levels of transparency and public scrutiny. If not, the first time we learn of them and their true potential could make Edward Snowden’s revelations seem quaint.

The weapons incorporating all of the technologies above will probably be linked, to some degree, to the same networks we conduct our civilian lives on. Indeed, these weapons will likely use and exploit civilian networks in combat since “cyberspace,” or the entire electromagnetic spectrum including the Internet, has been famously been defined by the Pentagon as its newest “warfighting domain,” on par with land, sea, air, and space. What happens when all of our homes, cars, clothes (the Internet of Things), and even our bodies are connected to a network that autonomous weapons ranging from robots to malware will also have access to?

AAGGHHGHGHGHHGHH! sorry, whoa

I've been hearing AI is *just around the corner* my entire life, and people were certainly worried about such a possibility long before I was born

yeah but that doesn't really prove or disprove anything, one way or the other. i do think it's the convergence of so many developing technologies at once that makes this a bit different, though. the thought of a drone+a 3D printer alone is pretty crazy. certain AI advances like real-time language translation/transcribing are not difficult to imagine at all.

there's no way we are close to the technology required of an AI to correctly and authoritatively identify a potential target, determine who they're with or what they're doing or whether or not current situation presents an ideal opportunity to attack, etc. Humans can't even be counted on to make these decisions - drone operators are constantly killing innocents as I'm sure you're aware. The idea that we're "on the verge" of having tech do that for us is some juvenile fantasy nonsense, it ignores all the things that go into, say, Obama determining what the odds are that someone on the US's "kill list" is actually in a given vehicle convoy and the risk of innocent life is worth dropping the bombs

You're talking about replicating human decisions, i'm talking about removing them. Is it far fetched to imagine, for example, drone systems that could be trained to identify and automatically destroy convoys of vehicles driving at night or human heat signatures on a mountain - or automated sniper systems that can detect and shoot anything that moves within an exclusion zone around a military base. An AI system can't tell you whether that's moral or proportionate but, alongside other potential efficiencies, i'd imagine that could be part of the attraction. Like other, more basic, technologies that can kill indiscriminately after they have been set in motion they would be illegal to deploy but unlike biological or chemical weapons they are not currently illegal to develop. The fact that the current US government would possibly choose not to use them doesn't mean that nobody ever would.

As Karl points out, the increasing use of connected technology poses other dangers. If it's theoretically possible to hack and crash one Jeep, or simultaneously turn on the gas taps and the toaster in a house designed with 'internet of everything' principles, it's not that hard to imagine the damage automated, self-targeting cyber attacks could do.

I'd ae thinking that the military might find it more economic to not have to pay an operator for each drone. &any ethical problems be seen as secondary to financial. After all its supposedly not 'us' that's being killed but always an apparently justifiable 'them'.
I thought there had already been a great deal of collateral damage from totally inanimate weaponry that had no autonomy. Trajectories being miscalculated, shells/bombs etc not exploding at the right time but remaining dangerous. Hate to think what adding autonomy into that would mean. There is always a factor of error looming over anything like that surely? & is the possibility of external override built in & not a 2 sided blade itself?

I think there's enough factors to be worried about that it is inherently problematic. But main thing for me is that possibility of error and possibility one couldn't correct it.
I'm intrigued by criteria of targeting, how accurate it would be in a computer anyway and moreover in a damaged one.

I've been hearing global warming is *just around the corner* my entire life, and people were certainly worried about such a possibility long before I was born.

neither accurate nor comparable but ok

If it's theoretically possible to hack and crash one Jeep, or simultaneously turn on the gas taps and the toaster in a house designed with 'internet of everything' principles, it's not that hard to imagine the damage automated, self-targeting cyber attacks could do.

I agree this is potential problem that does not seem far off. Although it seems different in degree and nature from the types of systems ID'd in the initial post (robot soldiers, autonomous missile targeting systems etc.)

We're totally going to be able to torture terrorists in their own homes by putting loud obnoxious music on their Sonos and flipping their Hue lights on and off.

it's all about target ID'ing and judgment calls, which computers are nowhere near being able to accomplish

Um....have you seen the latest XBox One Kinect? Consumer electronics that is always monitoring, can see through clothes, can monitor heartbeats, voice identification, etc. Glitchy as hell right now, but it is the consumer level.

I'd question the supposed need for judgment calls, would think that a programmed weapon might go ahead and do things without the ethical questioning I assume a manned one would be subject to. An operator is likely to prevent a missile from exploding in what appears to be a civilian crowd, I wonder if a computer would see the necessity of doing so?
So in short I would think that a computer would be more likely to bypass judgment calls and just act whereas an operator would recognise there was a judgment call there.

yes, kind of the central problem

have you seen the latest XBox One Kinect?

I would think it'd be kind of obvious that a system that you bought and willingly interacted with is a completely different scenario from a foreign/hostile system which you would (in all likelihood) be actively trying to evade

The self-driving cars from Google are working pretty well at this point.

I was gonna say isn't there usually a 20 year head start on technology in the military? They had the internet in the 60s. But perhaps that trend is ending, a lot of stuff is contracted out these days, and private contractors use their own off the shelf equipment. I think nowadays, just cos of the exponential growth of computer power in the 20th century, it's probably closer to a matter of months between patent and commercialization.

https://www.youtube.com/watch?v=YqpO58x7vuE

When you see a youtube robot do this running, it's clear that it can only function in a highly controller environment. The slope does not change and it is hanging from some kind of piping/wiring attached to the ceiling. I think this is for physics reasons, as well as piping in programming possibly from an external console. If there were killer cheetah robots they would be stationary and only a threat if approached. I suppose a pit could be dug and these robots thrown in, with collision detection, as a kind of cyberpunk Colosseum.

controller controlled

I think the military headstart is less existent these days, unless it's something that can be done with brute force. Even the military doesn't have the money to build a cutting-edge semiconductor fab. Also, private companies are super-interested in AI, so they don't even need military money to make it happen.

it does surprise me just how bad robots designed to replicate human and animal motion still are, c'mon guys even i can walk in a straight line without falling and i'm a fucking idiot

learn this one simple trick

http://www.washingtonpost.com/news/morning-mix/wp/2015/08/18/watch-googles-new-humanoid-robot-stomp-around-the-woods-like-a-drunk-iron-man

Like a drunk Iron Man who's cacked it's pants.

yeah that seems like a worthwhile thing to spend billions of dollars on

If Google paid me billions of dollars I'd have no problem stumbling through the woods drunk with cacked pants.

Hell I'd have probably even done it for free 15 years ago.

can't wait for google forestview

yeah that seems like a worthwhile thing to spend billions of dollars on

TBF that's rev 1.
Check out rev 8:
http://i.kinja-img.com/gawker-media/image/upload/s--cop0qLgT--/c_fit,fl_progressive,q_80,w_636/yv32dgqfhxmixwubbyr9.jpg

didn't realize google had bought them.

alright, let's check in on this, let's s--

The U.S. says humans will always be in control of AI weapons. But the age of autonomous war is already here.

oh.

The Pentagon says a ban on AI weapons isn’t necessary. But missiles, guns and drones that think for themselves are already killing people in combat, and have been for years.

Picture a desert battlefield, scarred by years of warfare. A retreating army scrambles to escape as its enemy advances. Dozens of small drones, indistinguishable from the quadcopters used by hobbyists and filmmakers, come buzzing down from the sky, using cameras to scan the terrain and onboard computers to decide on their own what looks like a target. Suddenly they begin divebombing trucks and individual soldiers, exploding on contact and causing even more panic and confusion.

This isn’t a science fiction imagining of what future wars might be like. It’s a real scene that played out last spring as soldiers loyal to the Libyan strongman Khalifa Hifter retreated from the Turkish-backed forces of the United Nations-recognized Libyan government. According to a U.N. group of weapons and legal experts appointed to document the conflict, drones that can operate without human control “hunted down” Hifter’s soldiers as they fled.

Drones have been a key part of warfare for years, but they’ve generally been remotely controlled by humans. Now, by cobbling together readily available image-recognition and autopilot software, autonomous drones can be mass-produced on the cheap.

Today, efforts to enact a total ban on lethal autonomous weapons, long demanded by human rights activists, are now being supported by 30 countries. But the world’s leading military powers insist that isn’t necessary. The U.S. military says concerns are overblown, and humans can effectively control autonomous weapons, while Russia’s government says true AI weapons can’t be banned because they don’t exist yet.

quit whining and prepare your own autonomous weapon systems to proactively protect-attack your home from any potential threats

The U.S. military says concerns are overblown, and humans can effectively control autonomous weapons, while Russia’s government says true AI weapons can’t be banned because they don’t exist yet

Forget about the question of AI or human control. The problem is autonomy. Autonomy is already a huge problem just from weapons as crude and passive as landmines. When you combine the lethality of a landmine with autonomous mobility and autonomous target-seeking, you've got a nightmare weapon all the same. The fact that their AI would be as stupid as a horsefly is not a point in their favor.

https://www.washingtonpost.com/opinions/2024/06/25/ai-weapon-us-tech-companies/

the first headline was

https://i.imgur.com/pCj7R2N.png

just so there's no confusion, i realize this is a poorly formed argument. also they work for palantir. just putting it here because i do think there is a non-zero chance of a terminator-style future with autonomous weapons, regardless of what a horrible idea that is and how nearly all of humanity will agree that no, we don't want that. just because something is a horrible idea and doesn't make any sense doesn't prevent it from happening.

a few minutes later they changed the headline to

https://i.imgur.com/wOxnZ8j.png

(the embedded html title was/is "Opinion | Alex Karp, Nicholas Zamiska: U.S. tech companies should help build AI weaponry")