Greetings geniuses. I am wondering if any one can point me in the direction of literature or argumentation that disputed some of the first principles underlying the LW / Yud / bostrom view of AGI. In particular looking for critiques of:
-utility functions / terminal value -orthogonality -instrumental convergence -consequentialism in the context of AGI
Thought this would be a good place to ask. Thanks
Emille Torres is a former EAer that has been crusading for a while against longtermism, worth checking out their arguments.
Superintelligence: the idea that eats smart people gives a good summary of reasons why AGI risk is kinda flimsy. It just inherently relies on a bunch of assumptions that aren’t necessarily true.
One of the key reasons you should be intiutively skeptical of AGI risk is that the rationalist community and “the sequences” are full of pseudoscientific idiocy, and that “rationalists” are likely the most influential group among AGI x-risk advocates. If they can’t realise that rationalism is dumb, can you really trust them on AI risk, an issue that relies on a chain of reasoning with a lot of flimsy assumptions?
One of the problems is that institutes that support AGI risk ideology get funnelled millions of dollars and can work full time putting out persuasive literature, while skeptics are generally hobbyists who argue against it in their spare time purely out of “someone is wrong on the internet” energy. Most people that correctly find AGI risk to be flawed will not put out rigorous papers on the subject, because we have day jobs to do. There are some attempts to change this with “change our mind” type competitions with cash rewards, but it inherently can’t compete with a full time institution.
Katja Grace has a really comprehensive piece here: https://forum.effectivealtruism.org/posts/zoWypGfXLmYsDFivk/counterarguments-to-the-basic-ai-risk-case
Rohin Shah has a great discussion on the use/misuse of utility functions in AI discussions: https://www.alignmentforum.org/s/4dHMdK5TLN6xcqtyc/p/DfcywmqRSkBaCB6Ma
Magnus Vinding also has some really good critiques of common assumptions in AI takeover scenarios: https://magnusvinding.com/category/artificial-intelligence/
IMO, both proponents and opponents of prioritizing AI safety tend to have big blind spots and unwarranted confidence, so it’s cool that you’re going out of your way to read a lot :)
Check out the Increments podcast. They have some really good critical episodes on AGI/Longtermism, they strike me as basically being what EA used to be, nerds who want to give money to mosquito nets and that kind of thing.
I appreciate this thread - thank you for posting, OP. I’d love to know about what else you find, if you’d be willing to share in comments when/if you come across resources you find helpful.
[removed]
Point out that their insistence that actual probability of a Singularity forming is irrelevant (due to how high the risks/rewards are) is an EXACT mirror of Pascal’s wager, they’ll hate that.
On top of that, the idea that an AGI will have an “intelligence factor” quantifiable as G, which can scale arbitrarily high based on amount of processing power thrown at it, and that in turn, the higher G goes, the greater its ability to solve arbitrarily complex problems, and that the AGI will actually choose to self-iterate like that, and that its initial hardware and power will allow it to do so to a meaningful extent… all these are absolutely necessary premises for MIRI-style AGI Risk to be a problem worthy of serious effort, and they are all actually a hugely specious pile of nonsense if you look too close.
“I am very smart and it makes me persuasive, and I’d choose to self-iterate and take over the world, and the AI would be even smarter than me!” is pretty much the closest they come to trying to justify any of them.