IMHO they vastly misunderstand the scope and risks of the problem. I’m not at all convinced that “AGI” or whatever you want to call it is anywhere on the horizon, and have a hunch that there may be deeper theoretical barriers to it beyond processing power and algorithmic sophistication. It doesn’t help that AI/automation panic is cyclical (about ever 30 years), and the hand-wringing over AGI is part of the broader, more normie cyclical concerns over the robots taking the jobs (which is misunderstood in the same way).

Risks: uses of AI for corporate Orwellianism is a much bigger concern than a species extermination scenario from Stellaris because the former is actually happening right now. I don’t see the latter as a real concern at all given my prior comments about scope. AI has way more potential for quotidian evils and the enhancement of corporate and state power, from which the wealthy will be largely exempt.

Basically, I see Yud et al. as a self-important meme subculture. And it’s not like the “work” they’re doing is important or unique, plenty of more intelligent, better informed people are sounding the bells about things like facial recognition technology.

There is little reason to take it seriously. You are severely underestimating how many moving parts the cultists’ arguments have, each of which is highly speculative. To name a few:

• It has to be possible for AGI to be unsafe, but also possible to be safe.
• Humans have to be able to make AGI.
• Such human-made AGI has to be at least somewhat likely to be unsafe.
• Humans have to be able to do things that make AGI more likely to be safe.
• But it should not be too easy, because if it were easy then you wouldn’t need to spend any significant resources on it.
• Safety-promoting activities have to be possible now, not 500+ years into the future.

And this is all while assuming their scifi-inspired notions of AGI and safety even make sense.

Edit: If you read their writings, they only try to argue that AGI can be unsafe and that making it safe will be hard. Every other chain in the reasoning they just assume to be true. If you go along with that approach, AI Safety can seem like a much better sell than it really is.

If you’re interested in this, you should read Superintelligence - The Idea That Eats Smart People.

In my opinion, it does a good job of debunking the MIRI crowd’s premises, and providing a slew of simple refutations.

But AI is in fact a danger in the sense that it is already being used, by humans, to increase unchecked wealth inequality, and this trend will only get worse unless forcibly stopped, whether by government action, or revolution of some kind. The “means of production” becomes increasingly sophisticated and increasingly inaccessible to all but the most well-funded, which is a runaway effect. It’s not the AIs alone that are the risk, it’s how people use them.

In many ways the concern about the threat of AI is just projection: what if AI behaves as unethically and irresponsibly as many of us do? But the real risk is that AIs will help the selfish and unethical to increase the gap with the rest of humanity, and that systemic effects will conspire to prevent changing this, as the case with e.g. anthropogenic climate change.

that there’s a significant risk that an AI will be created that will murder humanity, or worse, and therefore we should dedicate substantial resources towards stopping that - is valid?

Imho one of the big problems with this reasoning is that it shifts focus away from the real problems with AI today. Where mediocre machine learning is used for all kinds of horrible purposes. ‘I’m sorry sir, the machine says you commited disability fraud, and your benefits are now removed. The computer says moops.’ kind of stuff.

Esp considering how many techies are absolutely convinced that unless there is another techie in the room, they are the smartest person there. The rationalist bias towards thinking they are not that biassed, and are rational also doesn’t help.

Now the main point, should we worry about AGI killing us all? If AGI is possible, and if AGI is possible of exponential self improvement, yes. But I’m not convinced of either, and I think big corps/govs with machine learning tools are a bigger problem. (And as somebody else already said, organizations are basically already AGI).

(I love thinking about it in a science fiction concept however).

Thing is, there’s a lot of variables here.

The basic premises aren’t that unreasonable:

1. Humans might one day be capable of creating a general AI that’s more intelligent than humans.
2. That AI would then also be capable of creating an AI that’s more intelligent than itself or be able to modify itself to become more intelligent (which creates a feedback loop of increasingly intelligent AIs).
3. You can have intelligent beings that don’t align with human values.

All those seem almost obviously true to me and we already have examples of simpler versions of this in action.

1. We already have “technology” that’s more intelligent than most humans. It’s called “organization” where we combine the insights and skills of multiple humans towards a common goal.
2. We already have simple software that can improve itself and we also have organizations that have found ways to self-improve.
3. Again, organizations provide us a good look at “intelligent” things that don’t necessarily align with human values. Alternatively, you can look at other sapient animals, some of which are quite weird from a human perspective.

In my opinion, the reasoning holds up. If you assume that AGI is possible, the rest follows.

The problem with Yudkowsky and other so-called rationalists is that they have an (in my opinion) disproportionate outlook on the urgency of this problem. “Friendly AI” is definitely a thing we should be looking into and having a MIRI-sized organization staffed by competent people is worth existing.

The idea that Friendly AI (or whatever it’s being called right now) is THE MOST IMPORTANT PROBLEM OF OUR TIME and we should all be throwing all of our resources towards them because having a benevolent AI god automatically solves all of our other problems is what I take issue with.

I think the problem with yudkowskys “ai go foom” scenario is the same fundamental problem at the core of rationalism in general: the belief that pure intelligence is superior to empiricism in forming knowledge. See for example his post about choosing BAYES over science, calling people idiots for not beliving in many-worlds, etc.

This is what leads him to believe that an intelligent AI will be superpowerful, as exemplified by the AI-box experiment, in which he claims an AI that is isolated from the internet can talk it’s way out through pure intelligence. But if you want to learn how to be a super persuader, you have to run experiments, ie: actually talk to human beings. Similarly, the machine cannot deduce all the laws of physics from first principles (he once claimed an AI could deduce einsteins graviation from a webcam looking at a piece of grass), it needs to run experiments with the appropriate equipment. Einstein could not have learned the secrets of the universe locked up in solitary confinement his whole life.

This means that in reality, any AGI will be limited by the speed of experiments and the speed of production. The idea that any AGI will become automatically omnipotent is completely overblown. Frankly, I think we could take it down as long as we aren’t idiots with deploying autonomous drones everywhere.

I think the rationalists are basically right that existential risk is very important, and that AGI represents a risk class that we should know more about. I’m skeptical that MIRI is especially effective, and that their framing - an alignment problem with a technical solution, especially one pursued outside of the normal academic/government channels - is right, but I do think they’ve actually done some useful work in bringing the question to a broader consciousness - certainly you see more discussion of this in more “serious” channels than you used to, and that’s a good thing.

Whether or not the concerns - either specifically or in broad outline - are reasonable, his methods are definitely not going to do anything about it.

Concern about AI ethics is far better directed at things like, “How will authoritarian governments use facial recognition technology?” or the antihuman nature of capitalism than mitigating the risks of acausal robot gods - especially as the problems you might roughly define as climate change are a slow motion version of what “AI murdering humanity” looks like when you realize “AI” is just “humans using technology”.

Very little.

What is telling is that an AI that is already on the way to destroying humanity already exists - it’s called the Youtube Recommendation Algorithm, as it’s literally making nazis. No need for Total Information Tactical Awareness Networks or exsurgent virii, just fucking stupid algorithms and the profit motive creating a fascist tide.

And those manbabies are shitting themselves over fucking Skynet. Jesus Christ.

Edit: Also, I think the only school, IMO, that could create an AGI, biomimetisism has fallen out of favor, replaced by neural net solutions that, while quite unlikely to be made intelligent, are far more easily made into tools.

It is 117% a defense mechanism against having to deal with real problems in one’s own life or out there in the world right now that you don’t need to write speculative fiction about.

The biggest problem with Yud’s brand of AI alarmism is that it is based on an “intelligence explosion” by recursive self-improvement, a hypothesis that under the hood requires a downright insane assumption–that it is neither the volume of data nor the processing speed that fundamentally limits intelligence, but the algorithm. Yud once claimed that a superintelligence would hypothesize general relativity from a few frames of video of a falling object. This represents a fundamental misunderstanding of how intelligence works.

If you haven’t seen Maciej Cegłowski’s talk about this, you are missing out.

And in general, if you’re not following everything he does, you are missing out.

Edit: ninja’d by u/antonvis

I think there are a bunch of issues that are weird even before we get into the practical issue.

Like “What is intelligence in the first place?”, “Can intelligence really be scaled up like they think it can, and won’t it just run into diminishing returns incredibly quickly?” etc.

And that’s before we get into “AGI is not only not technically feasible at this point, but we can’t even begin to think about how to do it.”

Nah. Francois Chollet, one of the prime movers behind Keras, a very famous ML library, had this interesting article on the subject: https://medium.com/@francois.chollet/the-impossibility-of-intelligence-explosion-5be4a9eda6ec

In particular, there is no such thing as “general” intelligence. On an abstract level, we know this for a fact via the “no free lunch” theorem — stating that no problem-solving algorithm can outperform random chance across all possible problems. If intelligence is a problem-solving algorithm, then it can only be understood with respect to a specific problem. In a more concrete way, we can observe this empirically in that all intelligent systems we know are highly specialized. The intelligence of the AIs we build today is hyper specialized in extremely narrow tasks — like playing Go, or classifying images into 10,000 known categories. The intelligence of an octopus is specialized in the problem of being an octopus. The intelligence of a human is specialized in the problem of being human.

Personally, I find the mathematical argument about there being no free lunch extremely compelling. I think it’s apparent to any honest observer of ML/AI that the algorithms at work are extremely narrow. You can feed an AI the entire corpus of Bach and produce more Bach music, but you can’t feed an AI the experience of a human life, and expect it to produce original music, or do anything but produce music like Bach’s. (That is, for Bach, the ‘training data set’ was being a human, plus Bach could do a bunch of other things like dress and feed himself.) The brittleness of current AI algorithms is enough to convince me that there’s no particularly interesting breakthrough coming by in the near future.

But even if those algorithms became more generalized, the no free lunch theorem guarantees they would not become these super-intelligent god-monster beings that the rationalists like to scare themselves with.

The whole article is well-worth your time, by the way–he goes on to explain that intelligence is highly situational, and so is the successful application of intelligence. Intelligence is not some kind of magical superpower. Believing that is only possible if you’re the kind of dork who was good at school and nothing else.

Blather about X-risks in practice tends to mean that folks ignore the real existential risks to civilization - Climate Apocalypse, The Facist Tide, The Bomb, Plagues, Bolides - the last one just got underscored in dramatic fashion, considering we just got buzzed by a citykiller asteroid - in favor of wank like Grey Goo or SHODAN.

I think it’s possible, but not necessarily likely, and much much farther into the future than they think it will be. Those text-generating algorithms Scott Alexander seems so impressed with are basically just a neat toy at this stage, it’s pretty weird that he acts like it could be remotely compared to real intelligence or sentience. Curing cancer, or something like that, would at this stage of history be a much more realistic and important way to improve the world.

I didn’t come up with the following analogy, but: people who see an algorithm do something clever and say “Oh my god, human-level AI is just around the corner!!!” are equivalent to people who see a magician do a trick and say “holy crap, that guy can do real magic!!!”.

In the case of AI; nobody has any idea how to even start making an AGI and anyone who tells you different is lying. It might be 50 years away, it might be 500 years away, it might be literally impossible for some reason we haven’t discovered yet. Nobody fucking knows.

[deleted]

NSFW tag?

Also what is your real concern? Is existential dread of climate catastrophe too much or you to handle? If that is the case fine. But why instead make up a future bogey man to haunt the closets of your mind?

Are you personally afraid that you will be killed by the big bad AI? If you have children, is it them that you are concerned for? Is your future grandchildren? Is it the future generations who will be the same as strangers to you?

Or do you believe that some future AI will reincarnate you simply to torture you, a la I Have No Mouth and I Must Scream. If that is the case, all I can say is get over yourself. You, like all of us, are insignificant. The omniscient AI will presumably be unlikely to waste its time with reincarnating you.

There are plenty of actual issues to worry about. Worry about those. If worrying about those upsets you, then don’t worry about them. But don’t go around making bogeymen for yourself. Particularly when the real issues we are facing are likely to degrade our societies to a point where we wont be able to make toasters, much less omnipotent AIs.

How would anyone ever form an opinion on this without doing a deep technical dive?

MIRI in particular is all about agent-based AI safety, which is irrelevant, but look at other areas.

People have been trying to build AGI since the 1950s. It’s hard. If you ask people working on AI today, they will generally say that AGI seems far away. There’s some interview with Demis Hassabis (head of DeepMind) where he says he thinks it’s far away, and he’s someone with an incentive to hype the prospects for AGI.

It’s okay, and good, for a handful of academics to study distant, speculative concerns. If you’re not one of those people, it’s not worth worrying about.

I actually think AGI is a potential problem worth worrying about. It has potentially huge implications, we’re not sure what the timeline for its creation will actually be, and in light of those facts, there’s comparatively few people working on it. I’m in favor of more research and strategy going into it.

The thing is, that research should be done by actual experts, not by an organization operating under the banner and philosophy of someone who compensates for a lack of academic or professional expertise by being utterly convinced of not just his own intelligence, but the awesome power of that intelligence (I’m talking about Yudkowsky and MIRI here).

Just to give an example, one can’t help but notice that MIRI doesn’t tackle things like deep reinforcement learning (which are at the forefront of advanced AI nowadays) but rather focuses on the very theoretical like game theory and decision theory. And yeah, game theory and decision theory are legit fields of study with actual experts and are quite possibly relevant to developing AGI, but they’ve also been endlessly abused and misused by self-proclaimed “very smart people” e.g. “rationalists”. All this adds up to making me doubt that MIRI is contributing anything useful. It’s a shame that many of the most prominent AI research organizations are contaminated by Yudkowsky’s influence.

I originally thought “well I’m neither interested nor super-smart enough to look at your problem, but I expect it’s good that someone’s on the case”

then I found out what LW actually did hoo boy

bit like cryonics really - I originally thought “plausible, long shot, who knows!” then I looked into it and went “holy crap what is this idiot nonsense”

We know that we don’t know what advances are necessary for strong AI; but we are pretty sure that at least some novel ideas are needed, and that relatively strong AI is possible in principle. It appears unlikely that we’ll crack that problem soon.

We don’t know the practical limits of AI (e.g. posed by complexity theory), but we know that some limits exist. Acausal robot god-style AI is probably fundamentally impossible (fantasy, not SF).

There are good reasons to expect that strong AI will be pretty dangerous once possible. We don’t know whether alignment of a specific future architecture will be easy or hard. It is at least plausible that some strong AI architectures will be easier to build than align.

I am doubtful that the rat movement has made, or will make, substantial contributions on this. It is plausible but not certain that meaningful contributions to AI alignment are simply impossible until we understand more about AI.

However, lots of tiny academic subfields exist and deserve to exist. I think it is good that some people are thinking about this, and it is a good idea for some tiny fraction of AI research to be directed at such safety concerns. On the margins, it is good that some people can think about this kind of safety concerns without getting laughed out of the room. I would have expected the rat movement to be damaging rather than conducive to that (“doomsday cult ahoi”), but it seems that this is not the case. So I give +1 to Yud et al for successful PR efforts for a cause that is good on current margins (moving from laughable to tiny subfield is good; moving from small subfield to mainstream would be bad).

an AI will be created that will murder humanity

Humanity is something that we can and will overcome.

The idea that AI will surpass human intelligence, historically soon, and that the values espoused by superhuman AI (whatever those values may be) will become the values that govern the earth, is completely logical. Given that, it is also logical to prioritize the task of identifying and codifying human-friendly values, with a view to ensuring that the first superhuman AIs follow values that are friendly to us, rather than unfriendly to us. And even now, MIRI, formerly SIAI, is one of the very few places directly tackling this task (e.g. under the rubric of “AI alignment”). So I certainly regard their concerns and their efforts as meritorious and highly important, even though they are now shrouded in a haze of controversy and notoriety.

But I’m an old-school Yudkowsky camp-follower, from the days before the detour into Less Wrong rationalism, and certainly from before today’s epoch of culture war. There was a time when skepticism about Less Wrong revolved around things like opposition to dogmatism about the many worlds interpretation, rather than concern that Less Wrong rationalism is a gateway to reactionary badthink. By /r/SneerClub standards I am undoubtedly a filthy rationalist sympathizer, so, you may wish to regard my opinion as both unrepresentative and unsurprising.

If you’re familiar with the AI/ML space, you’ll know that we can’t get symbolic reasoning from gradient descent. Without symbolic reasoning, the current generation of AI is just a fancy repackaging of computational statistics.

We are a long, long way off from AGI. The foundations for AGI aren’t even there yet, so there’s no reason to have an existential crisis over it.

What you should worry about is the way that AI/ML is being used to consolidate power. ML is really good at finding correlations and patterns that humans have had a hard time noticing in the past. In a capitalist context, this means that ML can be used to exploit others in novel ways and to accumulate capital rapidly.