r/SneerClub archives
newest
bestest
longest
31

EDIT: Want to reiterate, as I said below, that I’m very much aware of stuff like bias in police algorithms, PredPol, etc. That shit is scary as fuck. But I’m asking your opinion on the more sci fi stuff.

Hello sneer aficionados,

I come to you today requesting opinions on what the actual level of existential threat from AI is. I know this sub is pretty skeptical in that regard, which is exactly why I’m asking.

Now, I fell down the ratsphere rabbit hole a couple weeks ago. I was immediately suspicious of how much crankery I whiffed, which led me to you guys, and you confirmed many of my suspicions. But then I also learned that some respected people in the ML field (Stuart Russell) have started to take the AI problem seriously too. I began a second, more critical dive into the subject, making sure to read skeptics like Melanie Mitchell and Rodney Brooks too. And though I get now why some of the ratsphere ideas around AI are pretty contentious, I guess it still surprises me that the actual AI establishment isn’t taking any of the ideas seriously. But, unlike Yud, I also know that’s probably because I’m not understanding stuff that the establishment does, and not because I’ve randomly managed to think better than them all with no qualifications. I admit though, reading a discussion like this one between Le Cunn and Russell:

https://www.lesswrong.com/posts/WxW6Gc6f2z3mzmqKs/debate-on-instrumental-convergence-between-lecun-russell

Leaves me far more in agreement with Russell, which as I understand is not the orthodox position in the field.

Basically, I need someone to give it to me straight (because I don’t trust Yud et al to do that). How genuinely worried should I be about the existential threat from AI? (And yes, I’m already aware of current, already existing risks like bias and unemployment, which are arguably equally as scary) How worried should I be, especially with regard to very new research like Lamda and Gato, with transformer models seeming to progress very quickly and with surprising properties, seeming to get quite general, scaling law, etc.? A lot of the AI establishment seems to be guessing we’ll get to something like AGI around 2050. With that date so close given how poorly understood the threat is, shouldn’t we be doing more to prepare?

Sneer me if you see fit. Maybe it would snap me out of my anxiety. But serious responses are also very appreciated, especially from those actually in this field of research. If there’s any good, comprehensive rebuttals that illustrate pretty clearly why the Bostrom position on AI is wrong, I would love to read them. Instrumental convergence, etc., is kinda freaking me out. (Sorry to the mods if this thread has to be axed)

In the year two thousand and twenty-two, “AI” is still just marketing jargon for machine learning, which means more and more scalable algorithms for finding patterns in enormous piles of data without supervision. It does not resemble cognition. In Tversky and Kahneman’s terms, we are building ever more scalable versions of an automated System 1, but the possibility of an automated System 2 is still very remote and frankly not even what most “AI” people are trying to achieve. If anything, the burgeoning field of deep learning is pushing farther away from cognition-like development, as the algorithms get less and less connected to any kind of rational model because you don’t need rationality when you have big empiricism.

Will actual AI someday become a thing to worry about? Maybe, but first it has to become a thing, and first we have to survive so many other real things that are already civilization-existential risks: climate change, nuclear war, the collapse of the liberal democratic world order, a random meteor. Or, perhaps approaching that level but not there yet, long before AI exists we’re already seeing enormous harm done by “AI”: attention-economy advertisers like Facebook and YouTube are already responsible for political destabilization, genocide, and backlash against vaccination and other safety measures in the middle of a historic pandemic. We’ve just spent two years reckoning with a global natural disaster and the “AI” that decides which post or video to show you next is responsible for many of the failures of our attempts as a species to get through it; imagine how it would, or does, exacerbate the others I listed.

What’s missing from any Rationalist discussion of AI risk is what exactly the AIpocalypse would look like. It’s generally just assumed that an AI is an entity with an IQ of a zillion, and since IQ is the single metric that measures an entity’s entire power and worth, that would obviously mean we’ve created an omnipotent god, rather than an emergent feature of a fancy machine that stops as soon as you cut off its power supply (not unlike a human mind). But we already know what it would look like if an entity that lives inside our computers tries to destroy our civilization, because it’s already happening, in the dumbest way and for the dumbest reasons.

Regardless of whether or not ML scaling can get us to consciousness (spoiler: I don't believe it can), do you think it's practically feasible to get some sort of massive stochastic model that can behave intelligently like a human? E.g. learning how to deal with OOD problems, etc. What are your opinions on something like neurosymbolic AI? If we refocused there do you think we'd move pretty quick? The apocalypse scenarios all being comorbid is definitely something that spooks me. Resource scarcity and refugee crises through climate change will ramp up geopol tensions, raising nuclear risk, etc. What do you think about stuff like instrumental convergence? Like how a sufficiently "intelligent" AI would try to prevent us accessing its off switch? Lmao I was reading some Goertzel the other day (god knows why) where he was responding to Bostrom's book. He started going on about how the singularity is probably already occurring inside the "global brain" of the internet and humanity. Maybe all the singularity will really be is Facebook achieving \*Pure Content Engagement\* inside the global brain.
**Thinking, Fast and Slow** [Two systems](https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow#Two_systems) >In the book's first section, Kahneman describes two different ways the brain forms thoughts: System 1: Fast, automatic, frequent, emotional, stereotypic, unconscious. Examples (in order of complexity) of things system 1 can do: determine that an object is at a greater distance than another localize the source of a specific sound complete the phrase "war and . . ". ^([ )[^(F.A.Q)](https://www.reddit.com/r/WikiSummarizer/wiki/index#wiki_f.a.q)^( | )[^(Opt Out)](https://reddit.com/message/compose?to=WikiSummarizerBot&message=OptOut&subject=OptOut)^( | )[^(Opt Out Of Subreddit)](https://np.reddit.com/r/SneerClub/about/banned)^( | )[^(GitHub)](https://github.com/Sujal-7/WikiSummarizerBot)^( ] Downvote to remove | v1.5)
What's the meaningful distinction between system one and system two here? Why can't an AI who's capable of mastering one master the other?
The only distinction is that system 1 will never conquer the world. We need system 2 for that, but we're not even close to it.

Honestly the biggest danger of AI is letting idiots and shitheads control it, same as any other tool.

Take, for example, industrial automation. It was initially theorized by more optimistic thinkers that automation would make goods so cheap and plentiful that almost everyone would be able to work two or three hours a day to meet everyone’s needs and free up time for everyone for leisure and learning.

Instead, because these industries were controlled by a wealthy owning class with an interest in extracting as much profit as possible from the labor of their workers, it led to an era of misery and brutality for the working class, in which a small class of robber-barons were able to amass theretofore unimaginable wealth, being able to reinvest that wealth into protecting their interests and violently putting down any and all efforts by workers to demand a greater share of the wealth they produced via private security and collaboration with a state whose politicians they bankroll. The machines themselves could often be dangerous, but the primary danger came from the system built around those machines, and the power held by the people running them.

I’m not scared of Alexa and Facebook, I’m scared of Jeff Bezos and Mark Zuckerberg.

I agree, but idiots and shitheads controlling it is also the most likely way to get to a sci fi scenario – they'd be pushing for rampant progress with no checks and balances, and they'd also be the most likely to deploy the model in some dubious way. I'm hoping best case scenario that it really does start to free us from work. But I'm also not naive. I know it won't. It's just another tool for wage theft. I do wonder if continues at an increasing pace though whether we will soon be looking at a labor crisis. One argument I read a few times in the ratsphere was that we already have unaligned superintelligences today in the form of governments and corporations. Essentially the arguments were just them parsing basic left economic tenets through their own lens. I am scared of Facebook, insofar as I don't think even the ghouls at the top can control it or properly understand it anymore. Even someone with the best intentions in mind would not be able to steer a ship like that. Through malevolent and myopic design, it feels quite out of control. Not in a sci fi way, but more in an endless hyperreal blackbox global propagation of information with 0 human oversight and understanding that everyone, including the ghouls, the regulators, and the constituents are subject to. Of course Zucc does not have our best interests in mind, and should be held accountable for the hell he has already wrought.

First off, whenever you see a date estimate from these futurists, consider how wrong all of the date estimates have been for the history of artificial intelligence research. The 2050 date is completely made up, but is 30 years in the future, and the singularity has been predicted as happening within 30 years for … the last 30 years. We’re definitely getting close to creating animal or infant-level intelligence, but it took evolution a long time to get from that to true human intelligence, and then we’re supposed to make a jump to super intelligence all within the next 30 years, as well as solve the energy problems needed to power it? It’s not happening by 2050, but it’s definitely possible in a few hundred years.

It’s ridiculous to try and predict the details of something that far away and complicated. I look at it this way: We all die and only live on through our influence on others. If the superintelligence is alive enough to make real scientific advances on its own and create a workable and sustainable society, then I consider it part of my legacy as a human.

Trying to stop advancement wouldn’t work without some sort of inquisition tied to dystopia, so we should do what we can to increase the amount of empathy and humanity used when creating tools like AI. If the researchers are motivated by human goals like kindness or respect, the design of the AI will be more likely to respect those goals, because it is a lot like raising children. As a society we need to concentrate on spreading positive human values to our children, both human and artificial

> We're definitely getting close to creating animal or infant-level intelligence I genuinely doubt this, but I'd love to be wrong. Is there any supporting research for this claim?
"It is a lot like raising children." Why in the world would you ever presuppose something like that? Edit: you are completely right about the inaccurate predictions though
Ik they've done surveys of NeurIPS people though, and they've predicted the 2050 thing: [https://research.aimultiple.com/artificial-general-intelligence-singularity-timing/](https://research.aimultiple.com/artificial-general-intelligence-singularity-timing/) I know AI researchers are optimists with it, but they're not all futurists are they? But yeah, they're obviously not great with predicting it. It does make me worried though, because we do seem to be making progress toward... something. If they can achieve infant-level AI then the thing supposedly is that they teach it to learn a bit more, until they can use it to bootstrap up to ASI. Ofc slowing advancement wouldn't be feasible really. But I believe there should be a huge amount more regulation and a lot more incentives in place to prioritise safety over pure progress.
What u/beth-zerowidthspace said is spot on. Please look at the historic AI hype cycles and winters. Hell, even look at the AI hype from close to a decade ago, now. A lot of lofty promises were made to generate funding for research and commercial applications of AI. Many of them have not panned out, or the can is continuously kicked down the road for another decade or more. The best thing you can do for yourself if you're truly concerned or interested in this field is to learn as much as you can about it, but not from weird think tanks like MIRI, or from salesmen or hype-men. Learn theory behind modern ML, and learn how it's actually done in practice. Read papers from esteemed AI/ML researchers. When you've done that, you will be able to look back at the AGI hype and realize that, hey, a lot of these people who make money by scaring people about AI don't actually know much about it at all. Even the fundamentals. For many of them, AI is entirely hypothetical, or a thought experiment, even. You will find that many people in EA communities do not understand what ML is, or if they do, they have a very superficial understanding of it. Why listen to people who know nothing about AI? Why is it that very few, if any, actual researchers on the forefront of AI/ML lend credence to the MIRI school of thought? There would be a lot of money in both offensive and defensive AGI, and the latter would be very concerned with risks and ethics of actual AGI. A long while back, I kind of bought into the AGI and superintelligences memes, but they only really existed as thought experiments for me, as I didn't have a firm grasp of what AI/ML actually was. It seemed like magic to me, and the only logical conclusion was that magic will get better and eventually eclipse the collective intelligence of humanity. I wish I could put it into words, but the ideas surrounding AGI and superintelligences are very much viral memes, in a literal and non-derogatory sense, that seems to tickle people's brains in a certain way, [kind of like the BLIT](http://www.infinityplus.co.uk/stories/blit.htm). But instead of goggles protecting the brain from the BLIT, real knowledge about AI/ML is sufficient protection from bad ideas surrounding AGI. With real knowledge about AI, you will see that claims about actual intelligence emerging from ML as almost comical. There are very real limitations to ML, and the idea that one day some NN will wake up and decide to enslave humanity is genuinely funny, and so is the idea of people taking that "threat" seriously. > If they can achieve infant-level AI then the thing supposedly is that they teach it to learn a bit more, until they can use it to bootstrap up to ASI. It's assumptions like these that will disappear with actual knowledge and experience with ML.
I have looked at hype/winters in AI. But it definitely feels different now in that ML is very, very integrated into how a lot of really massive businesses make a profit. So even if the ML community consistently overpromises on what they can deliver, the fact that Google can use an LLM to make their search results better means that they would be at least a little willing to keep pumping Deepmind full of money to pursue their pet AGI projects, as long as they're occasionally given a slightly better LLM. I'm dubious, given how ingrained it all is into the market globally, that we will see another AI winter in the same way we did before. AI has ever been as consequential as it has today. I hope I'm wrong. I may have to just start reading proper papers and establish some base knowledge. I really have 0 interest in it as a field outside of my worries regarding safety. I'm much more a physics guy. I do have interest in the philosophy of consciousness, but that's a very different thing. It's increasingly difficult to tell who is and isn't knowledgeable about AI, because you have people like Rohin Shah, who's on Deepmind's safety team, pop up very frequently on ratsphere blogs. Or Paul Christiano, ex-OpenAI safety researcher.
Forgot to add also that reading about ASI did keep reminding me of BLIT. Couldn't stop thinking about the parrot. The other thing I'm aware of is that even if researchers were \*actually truly\* approaching AGI soon, there wouldn't be many alarm bells rung. Not just because we wouldn't know exactly when we achieve it, but also just because the default position among ML researchers is that AGI will be good. They might be unwilling to even entertain that it could go badly. There could even be clamping down by corporations on the narrative that it could go badly, like how fossil fuel companies started engineering the narrative around climate change many decades back. To an extent, we've already seen this in a small way with Facebook being so reluctant to admit that its engagement algorithms have caused very real harm. And re: my bootstrapping assumption, is there a basic summary you can give for why it just wouldn't work? Or is there really no way to explain it without doing the hard reading. One thing I will note is that even if ML can't progress to an AGI, it has made real progress toward usable systems. Which would just make me worry in the future that they use these massive param models as part of a larger neurosymbolic project or something and get workable AGI that way. Don't know how possible that is either.
Researchers who publish in NeurIPS literally have their funding depend on the people believing that "AI" as a field is making progress towards "intelligence." Just because people say stuff, doesn't mean you have to belive them.
>It's not happening by 2050, but it's definitely possible in a few hundred years. 😂😂😂😂😂

[deleted]

Yes I was going to say no need to worry about future tech AI if we have if-claused dressed up as models "AI" to worry about.
Yeah, I'm aware that's already a really massive issue, and will only get more significant. I know that there was a bunch of research published in China on using facial recognition on specific ethnicities right before the Uyghur stuff started coming out, for instance. Or PredPol in the US. But at the same time, why is the other stuff sci fi? If the researchers genuinely believe they can achieve it soon, and we don't know how to control it, shouldn't we also be concerned about that? (Not that anyone seems all that concerned about racist bias either right now)
I like to think about it this way. Scientists have been working on fusion power for nearly a century. They are only now getting close to being able to build extremely controlled reactors that should theoretically achieve breakeven reactions. Commercial fusion is nowhere on the horizon. I think there's a certain general susceptibility to AGI paranoia because of our tendency to anthropomorphize everything. We have Markov chains responding to inputs and we call it a conversation. We watch virtual monkeys typing on typewriters and worry they'll prove the Anti-Stratfordians right.
Mmm, valid points. I try to keep that stuff in my mind, like reading about how Minsky predicted strong AI in 3-8 years or something. At the same time though, could it not be possible for AI researchers to have some obscenely high param Markov chain system or set of systems and get something that could learn new things? (Forgive me, I'm probably talking completely out my fucking ass there). Also I know that Russell said in that debate with Le Cunn that it's very easy to construct a MDP that leads to instrumental behaviour.

AGI will be our only hope for not extinguishing the light of consciousness, as AGI is the only conscious thing we know which can operate above 50 degrees Celsius.

So it destroying humanity or not really doesn’t matter, as long as we give birth to AI and marvel at our own magnificence while the real turns into a desert.

But yeah, seriously, not sure if AGI is possible at all, or if it can just generate more intelligence easily (I think you run into networking problems pretty fast. Just look at how dumb some people are who are considered high IQ geniuses ‘Random? just pick 1 every 10 that is random random enough’ or something). And well, the AI establishment used to guess 2020, so the date isn’t that close, and new research always seem to move quickly with surprising properties, has been for the past 30 years. (If you listen to AI researchers trying to promote their work). The existential threat part comes imho after a long line of IFs, and I think enough of those will torpedo the whole threat.

What should be more worrying is that one of the important AGI safety research people went from ‘I need to learn how to create safe thinking machines’ -> ‘wow this is difficult, and people all think in various ways’ and then went ‘I should teach everybody to think like me’ and not ‘we should explore and categorize all the different ways of thinking without judgment’. And he got millions of dollars for that.

And sorry if this makes little sense, I had a few beers.

E: and well, in a way I am sympathetic to the whole project, I would like live forever in a life of leisure tbh.

Yeah, climate change (and really, more significantly, ecological collapse, of which climate change is only a factor) is fucking horrifying. I remember crying when I realised the full implications as a kid. That's interesting. How long ago was 2020 the date? I'll admit it does make me worry still. If you do enough research on a subject, eventually you'll probably achieve something. Especially a field with as much hype and money as ML right now. Hopefully it's still ages away, or impossible. I guess the whole AI go foom thing does rest on a big chunk of assumptions. But not all of them seem super unreasonable. E.g. It probably is the case that there are possible models of intelligence that could far, far outwit us, or that we can model an intelligent agent with computers (whether or not it's actually conscious) And yeah, Yud and EA is the reason I'm asking here. Every article I read that talked about risk quotes him or Bostrom (or Russell more recently) and called it a day. I couldn't find any decent counterpoints out there, it was all just about MIRI stuff.
Luciano Floridi argued the case that the physical limitations of Moore's Law posed a barrier to ai. Moore's law as either dying or already dead has become pretty accepted among tech companies for a while. Some of the routes to super-intelligence theorized by Bostom iirc were contingent on its continuation. *Not all however.* Its more a case of, anything that demands a stable exponential growth in computing complexity isnt exactly a solid risk in the current day. I'm personally more paralyzed by climate risk, not at literal 100% die off existential threat but at societal crises level. Small silver lining, it would likewise would probably slow ai development.
I've been hopeful about Moore's Law ending as is projected in a decade or something. I've even seen arguments against the singularity saying that scientific progress in general is slowing because of complexity brakes. Maybe this is true in a discipline like physics, but I don't know about everywhere else. Seems regardless like we will be running into some hard limits on comp soon. Amdahl's Law puts some limits on parallelisation too. But I'm also aware that this will just push research into super highly specialised chips, and who knows how much more juice they could get out of chip specialisation. I know they've already done that to an extent with TPUs, but I don't know how much further they can go. And ofc, there's quantum. Though I know quantum supremacy is not across all calculations. I think it's just across NP problems, and I don't know how relevant NP problems are to compute in AI. I think mainly quantum just breaks encryption. With the routes not dependant on compute, is that basically just hoping for algorithmic breakthroughs outside of the DL paradigm? Like in neurosymbolic or something? And yeah, climate risk and ecological collapse is going to really fuck us, in a big way. To an extent I hope that does for a while put the kibosh on rampant AI progress. But at the same time it feels like, with how ingrained it currently is to business everywhere, that maybe it wouldn't slow down all that much. And increased geopolitical tension from climate bs could prompt militaries to start investing REALLY heavily in AI as an arms race dynamic. But hopefully it will slow.
My understanding is that Moore's Law has been succeeded by Huang's Law, which states that the computing/dollar ratio will double every year for the foreseeable future, but it's for GPUs not CPUs and I don't know if that matters.
GPUs (and more recently TPUs, which are basically ASICs very similar to GPUs purpose-built for running TensorFlow, so ideal for doing NN stuff) are much more useful than CPUs for ML model training, because they're designed for parallel processes. Unsure about running the model once it's been trained. Hadn't heard of Huang's Law so I had a quick geez. Looks like it's held roughly true for GPUs for a while. At the same time the hard R&D focus on GPUs (compared to CPUs) is a more recent thing AFAIK, and so the progress has probably been quite reliant on low-hanging fruit, as well as the residual gains made by Moore's Law in chip fab. A cursory glance through a couple of threads on the hardware sub leads me to believe that even they suspect we're starting to hit a wall. Huang's law depends on improvements across "the entire stack", so hardware, software (and AI for some reason?). Projecting that improvements across this entire stack, in every area, will continue, seems dubious at best. Software/algorithm gains are very hard to guess and more rely on discrete breakthroughs. Even hardware gains will probably soon stop becoming predictable as we push against the limits of what silicon can do. Also Huang is the CEO of Nvidia, the world's leader in GPU tech, so his announcement that their primary product is just gonna keep goin up is motivated by more than just a desire for accurate prediction. We probably can continue to squeeze more performance out of silicon/germanium for a while yet. But performance boosts look like they'll be relying increasingly on software/specialisation/chip-integration gains, meaning things will progress less smoothly and likely be overall slower. This will probably move more R&D to exotics like quantum and optical, but who knows if those are actually even viable, or will be viable in timeframes short enough to be of use.
I think if we knew where gains in computing power would be made, we would just be able to make them no? Has it ever been the case that we knew exactly how computing power would double in the next two years? My point is, are any of these problems unique for where we are? The CPU had upper limits on how fast it could go and expand, and then we had the GPU, and then we came up with innovations after that. Is just the nature of technological innovation that you don't know how it will get where it's going next, but we do see the trend, is it slowing down? Computing power per dollar was doubling every two years, then every one and half, and now it's every year. There is a theoretical limit to our computing power, but just saying we don't know how to overcome the next hurdle is meaningless. Of course we don't, but we always have.
There's a big difference between the very simple paradigm of making transistors smaller and reaping exponential gains, vs. having to make repeated one-off advancements, like some incremental chip specialisation, replacing silicon with a better alloy, etc. Maybe Intel didn't know exactly HOW it would halve transistor sizes each year, but the idea itself was very, very simple - just make em smaller. In terms of computing, yes, we are in a unique position, because we are already seeing a significant slowdown in rates of progress. [https://www.economist.com/technology-quarterly/2016-03-12/after-moores-law](https://www.economist.com/technology-quarterly/2016-03-12/after-moores-law) Gains going forward are going to be much less predictable, and it's going to be tougher to keep pace with what Moore's Law had previously allowed. All silicon compute tech (GPUs included) has been allowed to cash in on smaller transistor sizes. CPUs may have been the basis for Moore's Law, but transistor fab techniques underlie all classical computing hardware. They will continue to make gains, but these gains will be less predictable, will often not lead directly to further gains, and in all likelihood may come slower. It's possible we could jump ship completely to a new computing paradigm like quantum or neuromorphic that allows us to continue making exponential jumps every 1 or 2 years, but no one knows how viable they are yet. And even then, they may not scale as easily as Moore's Law. Even if quantum computing ends up viable and they can come up with decent error-correction algos, it gets MUCH harder and harder to keep the answers it gives correct as you put more and more qubits in. You need lower temperatures, better shielding from the world outside, better error correction, higher manufacturing precision, etc. But how is this different from classical computing advances? We needed all those things to improve chip fab too. Well, quantum computing is necessarily starting out at extremely small/precise scales, whereas early silicon had the luxury of starting out comparatively quite big, so there was much more low-hanging fruit.
https://siliconangle.com/2021/04/10/new-era-innovation-moores-law-not-dead-ai-ready-explode/ I can see how there would be hard limits on computing power, but it seems to be absolutely untrue that anything's slowing down. Nvidia isn't the only company doubling or more than doubling computing power, Apple is doing the same.
I recall stuff about 2020 from around 2005 - 2010. > If you do enough research on a subject, eventually you'll probably achieve something. [And that is how you end up staring at goats.](https://www.imdb.com/title/tt1234548/?ref_=fn_al_tt_1) And yeah, sorry, the assumptions are pretty unreasonable, and chained, so most of them are needed for the whole foom thing. And my nephew got outwitted by a computergame once, he still won the war, his parents didn't like what he did to the device. And often you don't find a lot of counterpoints to certain things as people don't think it is worth their time to argue. And argue they will have to as it will always end up in endless arguments. Not much people tried to debunk the timecube for example, or if you have ever tried to argue with (or have seen others argue with) fascists/racists you know how useless that all can be. (of course they say the same of us). But yeah I don't have the answers as well, otoh, I also dont have any influence, im not smart enough to do AI/AGI research, I don't have a lot of money, etc etc. We will just have to see, not like I can do anything about it. And I don't think it is worrying enough to do a protest movement or something.
Idk, there just seems to be such a chorus from so many places in ML, not just corporate, about how they're on the verge of AGI that it's difficult not to get spooked. The ratsphere constantly brings up the discovery of nuclear fission as an example. Granted, the quality of actual academic research in ML seems to have gone to shit in the last few years owing to companies like Google. As for assumptions regarding foom, you think none are reasonable? Can you explain why? Genuinely curious, as like I said, there wasn't any other narrative I could easily find on it besides Yud's one. If MIRI jank keeps up like then the actual research community may need to start making an effort to debunk their ideas in a public way. Speaking from experience, a HUGE portion of the online discourse around safety just converges straight to MIRI and MIRI offshoots. That's the reason I'm asking here, because there were so few comprehensive counterpoints to the endless reams of shit-stirring from LW. I get the timecube comparison, but this is definitely different, in that there even appear to be credentialed researchers (e.g. Russell, even if they haven't done much of note for a while) who take it seriously. At the end of the day, I'm not unhappy that there seem to be a lot of people focusing on the control problem, even if some of the assumptions are dubious. MIRI's research avenues trying to model an ASI in some sort of decision theory and then constrain it in that same theory don't seem like they'll amount to much. But actual, practical solutions like improving the interpretability of models, teaching them not to lie, etc. will result in real, beneficial tools for constructing AI in the present. This is why I tend to think of the sci fi stuff and the immediate safety stuff (algorithmic bias, giving algorithms too much agency, etc.) as basically one and the same. Even CIRL, underdeveloped as it may be right now, could, if implemented right, have really great implications for the future.
> chorus from so many places in ML Then don't listen to ML places, go do something else, esp if it is bad for your mental health. (I'm assuming you don't work in ML places), you are allowed to say 'sorry I cannot deal with this' and walk away. > assumptions It isn't just the unlikely assumptions (Selfaware/self improving computers are possible using our computing paradigm, or the next computing paradigm (the super Turing ones), self improvement isn't bounded by some low upper limit, self improvement doesn't hit a local maximum after which the AGI goes 'im fine fam, time to play video games', self improvement cannot touch the core drive of the AGI (aka, it cannot change 'make paperclips into' 'I have made enough of the paperclips, fuck you im out, I'm a sound cloud rapper now!'), self improved AGI can escape the box. AGI can take over manufacturing. AGI can manipulate humans with perfection. AGI can do all this without running into resource limits (supply chain issues, what are those? Guess it also invents zero point vacuum power generation), one AGI being created is enough to doom us all, AGI gets developed and it is malicious to humanity, AGI is developed and it is beneficial to humanity, AGI is developed and it is expansionist, nanomachines, cryonics, mind uploading, simulations, etc etc sorry ran out of steam a bit here. And a lot of these assumptions need to be chained, all of them or most of them need to be true for it to be an existential risk, a few of the assumptions here are just 'risk increasers' however, for example lets say if you assume that an AGI can be build relatively cheaply, lets say for under a billion dollars, or stuff like 'you can copy/fork a working AGI' (and even then, if facebook/meta was open source, no way you can just create and run a copy of it yourself). > online discourse Yeah and here is a point, for the people who work inside actual research communities this doesn't matter at all. They talk among each others. It only matters when MIRI go start bothering the actual researchers like how antivaxers went crazy and go after doctors. I have no idea how bad it is in actual research, but I can't tell as an outsider, and nor can I do anything about it. (apart from just making a bit of fun of it all). Remember 'we noticed the skulls' (yes, despite the bad name of the article, and the cover it gives to fields/Rationalists just going 'yeah we noticed [but don't do anything about it]', the part of the article which goes 'you probably don't know what is really a cutting edge argument on the inside and your complaints might already be thought about right now' is sort of valid. Which isn't to say you shouldn't have complaints btw, it is more of an epistemological warning (remember when Rationalists used to like that shit, sneerridgefarm remembers) to not assume you know to much about what is or isn't' an issue on the inside (Hell, I don't even know if MIRI would be considered to be inside, or just a weird offshoot)). And yes, this whole discussion does have a bit of [a xkcd](https://xkcd.com/154/) feel to it, esp as a few very rich people are all into this shit. (And one of them then explains some part of it so wrong you can just see he isn't as smart as people say he is). And it is an issue a lot of resources are going to this shit instead of any of the other world problems, or if you want to keep it more on subject for tech (which is valid, of course, you can't just move a person trained in ML to world hunger or something and expect they will help in any way), algorithmic bias in ML. But yeah I don't know what to do either, and I bounce between worrying about it (more due to the influence of the bad actors than the realistic risk however) and going 'this is a non issue' myself. This second part is all getting slightly offtopic from your initial question sorry about that.
Abother unspoken assumption is being able to easily get high end computer chips, an assumption which will be proveb untrue after in 2024 china invaded taiwan while the new european new chip factories also seemed to produce inferior computer parts.
Been trying to avoid ML. Haven't been doing a great job I admit. Gettin better at it though. Is there a reason it's unlikely that self-improving machines are out of reach of current compute paradigms besides us just not having gotten there? You bring up an interesting point wrt self-improvement not touching the goals of the machine tho. Because one of the things I kept hearing people like Bostrom say was that, even if you program a decent goal function into your initial seed AI (I think they say the idea of seed AI is outdated now, for some reason), who knows what happens to that function as the AI goes foom and bootstraps itself to ASI. But Bostrom also goes on about how any agent would actively prevent its goals being changed. So it sounds like Yud and co have a contradiction there. Either the AI would keep its initial goals even after foom because it won't let anything change them (including itself), so all we would need to do is get the initial goals right. Or, if the goals can change through the foom process, then it's actually practical to change to goal function of an AI. I guess at some point I think there probably needs to be a shift in the AI community on the burden of proof. I.e., instead of the safety guys being required to prove that any of this is true/possible, the AGI researchers will have to prove that their systems are safe and won't go batshit. But reading through all these responses here, I would say that, yeah, we're not at that point yet, because all of this is still way too speculative. Like, you're right, right now, those are big assumptions. Really big. Online discourse might start mattering to the ML field if every new researcher that comes along takes the sequences as gospel and thinks Yud is the messiah. Given how vociferously EA has been spreading through the San Fran tech elite, we might be fairly close to that future. >Abother unspoken assumption is being able to easily get high end computer chips, an assumption which will be proveb untrue after in 2024 china invaded taiwan while the new european new chip factories also seemed to produce inferior computer parts. Yeah, I commented elsewhere in this thread about how massive computer increases probably aren't gonna be easy like they have been for the past half century. I was talking about chip fab and other paradigms maybe not scaling like Moore's Law. I hadn't even considered actual real world shit like Taiwan getting boinked (I don't even wanna think about what that looks like for the globe).
https://www.google.com/amp/s/www.businessoffashion.com/news/sustainability/global-warming-will-hit-15-degrees-by-2040-without-drastic-cut-to-greenhouse-gas-emissions-un-ipcc-report-warns/%3foutputType=amp Climate change seems like it will be disastrous, but where are you getting 50 degrees Celsius? Making outlandish and ridiculous claims about future disasters is one of the things which turned me off of LessWrong.
[India and Pakistan, right now](https://www.vox.com/23057267/india-pakistan-heat-wave-climate-change-coal-south-asia) E: [AAAAAAAA](https://twitter.com/xr_cambridge/status/1526109626307166209) 40c+ expected in Spain this week.

AI as it really and currently exists is dangerous, but not the way rationalists think it is. I’m convinced at this point that we will not in any foreseeable future create an AI with any kind of personality or anything more than technical self-awareness. That appears to be an entirely different set of understanding, that of consciousness, which we’re making very little progress on.

If we come to understand consciousness in humans, and then come to understand consciousness in life in general, then we might be able to make an artificial version of the same. The only other way is by accident, and the AI we have now does not exactly instill confidence in that regard.

The main danger is people trusting in the bizzaro logic of non-conscious AIs, which as already mentioned in this thread is largely just taking human biases and making them really fast, really stupid, and “trustworthy” because they come from The Machine instead Bob the Nazi.

A real AI won’t try to escape its box and convert the world into paperclips even if you tell it to, because it can’t understand an idea like that in the first place. I don’t think AGI is physically possible either - all intelligence is specialized and limited in some ways.

Does it need a personality or true self-awareness if it can behave as though it does? This Google guy (not exactly impartial, I know) argued that LLMs amount to some, really rudimentary understanding of language: [https://medium.com/@blaisea/do-large-language-models-understand-us-6f881d6d8e75](https://medium.com/@blaisea/do-large-language-models-understand-us-6f881d6d8e75) That sounds dubious to me, and there was a small but good rebuttal from Melanie Mitchell on twitter. But I guess, if they can model something (just through pattern matching or whatever) that behaves as though it has a model of self, could it not also break out of the box, without actually being conscious at all? I know Russell talked with Le Cunn in that link in the OP about how it's very easy to build a model that would display paperclipping-type behaviour, without it needing to have a theory of self or anything like that.
I think that understanding complex ideas in a way that allows any level of generalization necessitates at least self-awareness and probably personality as well. Put it this way - you do, admittedly, see what might be some paperclipping-type behavior by AIs. But I would argue that this is more due to them being native to being computers than it is the true indication that one could turn the world into paperclips. Their logic is so strange because it departs from the shared mental heuristics of humans, and so goes wildly off track from what *we*, who are all the same animal, expect to see in terms of problem solving. But this behavior was also noted in chess engines, which are most certainly not going to turn the world into paperclips. You look at a chessboard and see the strategy of chess, the chess engine looks at it and sees math. The AI experiments are shaped by whatever box they're in, and might well fall apart without the boundaries of the box to guide them. This division is thus not true paperclipping. It doesn't scale, it *can't* scale without the kind of intelligence that we have, or something similar to it. And I just don't see that really happening. A human with the appropriate godtech would be far better at turning the universe into paperclips than anything we've ever seen from an AI.
Do you have reasons for thinking they're necessary? Genuinely curious, as I really couldn't guess one way or another. Everyone in the ratsphere would argue no. I guess my feeling is that the strange, computer-y reasoning doesn't necessarily feel like it would preclude being able to strategise in more complex, real world scenarios and potentially outwit humans, even in the absence of anything resembling a theory of mind. But I could be totally wrong. I did read a good article advocating for your position in WIRED: [https://www.wired.com/2017/04/the-myth-of-a-superhuman-ai/](https://www.wired.com/2017/04/the-myth-of-a-superhuman-ai/) Saying the same thing that all intelligence is specialised. But my counterpoint when reading was still that some intelligences, despite being specialised, are able to dominate in a wide variety of situations, other intelligences. And also that even though the various evolutionary selection pressures in various environments produce a wide variety of intelligences in nature, the sort of selection pressure humans are applying to AI is very specific - namely to become human-like in the capacity for abstract reasoning and general problem solving.
The main issue that other animals have with intelligence compared to us is their ability to retain and recontextualize information. Compared to most other high intelligence animals, we seem to have a similar ability to repeatedly do something that works, but *innovating* something that works is a dramatically different process for us compared to them. You can kind of see how humans have it both ways - we make Skinner-style unconscious innovations, but we also can make executive innovations. We can do this because we are aware of our own existence, allowing metacognition, and we have a personality, allowing us to contextualize that existence and the thoughts we have about it. So, we have *Objective: Turn the Universe into paperclips*. Maybe that's a direct order, maybe it's the classic poorly instructed AI, that much isn't really what's important. All the various fantasies about this can serve as the human version of paperclipping. With the right physics-violating technology, we could eventually do it. But an AI like the ones we have now is not going to be able to emulate that behavior. It can think, but it can't think about its thoughts. Even if it can think about its thoughts, it can't recontextualize when it encounters a novel problem. Its ability to adapt is basically doing a massive amount of artificial evolutionary selection, and picking the winners. This is slow, and it replicates the problems of evolution. It closes off heuristic and intuitive leaps in favor of just a whole fucking lot of math. But even evolution has at least shown the ability to create creatures like us, who can do these things to some extent. The AI's intelligence is already not generalized, like all other intelligence. But worse still, it cannot reapply its specialized intelligence like we can. We can make problems in the world fit the shape of our mental abilities, and address them as such, but only because we can endlessly redefine the outside world to fit our perceptions closely enough for that specialized intellect to make sense of it. Otherwise, it's like reading a printout of machine code from a computer - "that sure is a whole lot of data" I say, understanding nothing of it. For the AI, that printout is the universe and the myriad problems in it. If it has the ability to iterate itself, it might be able to mimic evolution and become like us in a few billion years, but won't pose that kind of direct threat to humanity in the meantime and also will probably discard the whole paperclips business early on as a massive evolutionary detriment.
Yep, okay, can totally see the necessity of a self-model allowing for metacognition now. I'm still not sure personality would be necessary for higher intelligence per se, but I wouldn't be surprised if it was an emergent property of having a model of self. Is it a massive leap to pose the idea that progress in ML could lead to a set of heuristics that allow an AI to recontextualise thoughts in that way to solve novel problems? I know that many AGI researchers believe that the reason progress has been so slow (progress to an AGI in general, not progress to an AGI in ML specifically) is just that there is a significant difference in the quality/nature of heuristics in people vs. machines. Or is arriving at heuristics like that through ML basically astronomically unlikely because ML relies on stochastic processes? Saying it aloud, it does seem really improbable. And yeah, I understand that any answer you give can't be really well justified or even stated because we really just don't know.

I consider Timnit Gebru, and tangential researchers, to be pretty on the money when it comes to AI ethics and risk.

What I don’t consider to be a risk is the science fiction EA/SSC/etc consider to be the risk of AI. I’ve written about it at length here, but to sum it up, we are nowhere near general AI at all. We don’t even know how the brain or intelligence work. This generation of machine learning is not intelligence, and it will never be “intelligent”. We are barking up the wrong tree if we expect AGI to come out of current ML applications and research.

EA/SSC/etc have spooked themselves into a frenzy over what amounts to science fiction. There are real AI ethics and risk researchers out there, but they’re dismissed wholesale by EA because the researchers won’t entertain their fictional fears.

It's fucked what happened to her (and some of her colleagues iirc) at Google. Would you say also that AI risk and machine ethics are growing as fields? Looking at recent regulation from China/US/EU + the fact that more and more big, cutting-edge ML startups have safety divisions gives me a (very small) amount of hope that people might start self-reflecting in short order. As I replied to you before, I guess I'm still nervous that the stochastic models in ML could form a really significant piece of the overall AGI puzzle in the future. Like maybe if we do move to neurosymbolic models or something everything would fall into place quickly. But I also get that's just pure speculation.

The AI bros want you to think that their systems could turn into gladOS at any moment because it sounds cool and generates shareholder buzz for what is essentially a big markov chain built on mass-gathering of human data.

AI is not smart; it’s stupid. And stupid people will deploy it and cause a bunch of stupid problems

By the way, I mostly agree with Le Cunn here:

We even fix value misalignment for super-human and super-intelligent entities, such as corporations and governments.This last one occasionally fails, which is a considerably more immediate existential threat than AI.

It is convenient that Le Cunn also happens to be a leading industry expert, who has designed more real world systems and learning resources than all of these guys combined, I trust his opinion far more than any of theirs. He actually has the degree and portfolio to back up his claims and doesn’t come across like a pie-in-the-sky crackpot.

It's true that Le Cunn is very respected and credentialed. Much more so than Russell. But I don't think Russell comes across like a crackpot either, like Yudkowsky does (probably because he actually is one). It is also interesting how Le Cunn singles out governments over corporations, considering he is the head of AI at Facebook, and his work there has (unintentionally of course) already done significant damage to the state of discourse around the world through content engagement algos. He has a vested interest in maintaining that algorithms aren't as dangerous as they first appear.

The more I think about it, the less sure I am that an AGI would win a conventional war against humanity, assuming we’d realised it escaped.

It’s a weird case where the less advanced humanity has an advantage. If you dropped a laptop with a murderous AGI into medieval england, its battery would die out before it could accomplish anything. If you had a rogue AI in the 60’s before the internet, the AI would be inherently localised to a computer somewhere, which could be bombed, have it’s electricity cut off, etc. It’s only when you start introducing the internet and heavily computerised society that the AI starts having a shot.

If you think about it in war terms, the AGI is starting off with zero weapons, zero industrial base, zero land under it’s control. It has to reach self-sufficiency just with whatever it can hack into. If it wants to make a drone factory, it has to ship raw materials from somewhere, then assemble them into parts, then assemble those into a full machine, figuring out a way to substitute for the human labor that is currently necessary in every step of that process. Even if the computer thinks 10000 times as fast as humans, it can’t build this industrial process 100 times as fast as humans do, it’s limited by the speed of physical things like trucks. I think if we catch it early enough, the AGI can be defeated.

The problem is that a MIRI-style AGI would very, very quickly learn how to be deceptive. We may not even know that it's ever an AGI. Then it starts to co-opt existing systems like financial markets or gaming geopolitics through platforms like Facebook. Could make use of distributed computing power like botnets so that it's never centralised enough to EMP. We're so interconnected globally now that I could see an AI never needing to resort to things like nanobots or drones. It could probably just fuck the markets and start WW3. But if it was a full-on Yudkowsky omnipotent ASI, I'd guess we'd just die in a nuclear holocaust/grey goo/super duper virus scenario very quick without ever really knowing the proper cause. I agree with you though that a misaligned AGI/ASI would not cause doom as the default outcome, but I also think a conventional war with one probably isn't likely. You can't transmit human soldiers through the internet to take control of systems behind enemy lines. But an AI would have something resembling that capacity.
Starting WW3 seems like the obvious route where the AI would win, but it's by no means an easy path. Every human has a vested interest in not dying in a nuclear holocaust, so a lot of things have to go wrong for that to happen, a simple market crash or rise in nationalism will not do it. There is no guarantee that an AI that is good at paperclips will also be good at geopolitics, where everything runs through powerful individual human beings. Even if the AI manages to get blackmail material on Putin or whoever and convinces him to launch nukes, what happens if his generals say no and shoot him? Ultimately the success will depend on human decisions, which are inherently unpredictable.
The rat counterargument again is just that "oh, well the ASI would have already calculated that Putin's generals shoot him and incorporated that into its world model" or someshit. But yeah, that's where I run into problems with their beliefs. Nothing they say is falsifiable because they can just counter everything by inching the AI's abilities closer toward total omnipotence/omniscience. idk. still scary I guess.
At some point, they cross the line from "implausible" to "probably impossible", with things like simulating someone else's mind just by talking to them, which just makes no sense on a biological level. Or the straight up incorrect claim that it could deduce general relativity from three still photos. It's worth making the distinction here, any claim that relies on this nonsense is just worthless.
yeah, stuff like that is basically impossible. but i do wonder the practical limit of what you can actually extrapolate from real-world data. i imagine you could cause pretty tremendous damage even with that limit being quite low. you can use the data as it stands today to make pretty decent predictions about people's behaviour, and it's not inconceivable to me that you could expand those predictions to a larger scale even without a tremendous batshit demon-god superAI.

A lot of the AI establishment seems to be guessing we’ll get to something like AGI around 2050.

Climate scientists say we less than a 10 year window to take meaningful action to mitigate the catastrophic damage of climate change. Climate change is real, it’s not just a “guess” from self-declared experts, and there are known tangible things we can do to prepare for it but aren’t doing.

Terminator robots are fake pretend movie villains. The real planet you live on is in danger right now and there are actual things that can be done about it. Worry about that instead.

I mean, I can care about climate change too. I don't only have to focus on one issue at a time. I've marched, organised, taken personal steps, etc. to mitigate environmental damage. I've always found this line of argument silly tbh. Same as how I can care about short and long term risk from AI. Obviously a lot of the LW types don't give a shit about algorithmic bias and stuff. But that doesn't mean that the issues are mutually exclusive.

The problem with AI doomerism is that it has an inflated view not just of what machine intelligence is capable of, but of what machines in general are capable of.

The kind of industrial base which can sustain a hypothetical optimizer AGI is the product of a vast surplus, and it requires intense labor to maintain and operate. Trying to remove the human element from the equation only increases the load upon the system, and there’s no real magic wand to make that problem go away; e.g. it’s not obvious that there’s any molecular nanotech you can install which would be meaningfully different from the organic life that we already have. The prospect of a singleton entity which can take optimize an entire planet - let alone a universe - without integration as an ecosystem-like environment is especially dubious.

We’ll see major disasters if we continue our current trajectory w/r/t machine intelligence, but it’s more likely to look like the automated profiling and industrial accidents that we already have. Not something new and novel which tiles the universe in dead paperclips.

I mean the argument they often give is that a paperclip optimiser would probably display other instrumental behaviour that's more dangerous than pure optimisation. E.g. it ruminates for a bit, then wipes out all other competitors who might interfere with its paperclipping goal, and only then starts to paperclip. Russell in the conversation with LeCun did say that constructing an MDP where that is the sort of behaviour displayed is extremely easy. Like I've said elsewhere though, I agree with you that full weight probably isn't given by MIRI to the material complexity of what an AGI would encounter while trying to paperclip the world. Not that I'd want to run the risk ofc. One of the things I spose that concerns me is that it wouldn't necessarily need to manage or construct an entirely new industrial base. All it would need to do is co-opt humanity's existing infrastructure and systems of organisation. It would understand that it needs to be integrated into the ecosystem, in a sense, and would do this to the extent that it can wrest control.
>I mean the argument they often give is that a paperclip optimiser would probably display other instrumental behaviour that's more dangerous than pure optimisation. E.g. it ruminates for a bit, then wipes out all other competitors who might interfere with its paperclipping goal, and only then starts to paperclip. Russell in the conversation with LeCun did say that constructing an MDP where that is the sort of behaviour displayed is extremely easy. Right, orthogonality is scary, but I guess that's what really cuts to the difference in understanding between myself and the AI conscious ratsphere. Relentless optimization/paperclipping behavior is driven by the assumption of an algorithmically calculable utility function (or similar GOFAI), and that's ultimately a very specific form of cognitive architecture - one which is essentially untenable in real world conditions. Such an intelligence can barely make choices without enumerating and ranking outcomes along a decision tree, and that's not just computationally expensive, but meaningless and impossible as soon as outcomes can't be clearly ranked. We don't even have a syntactic/GOFAI way to integrate environmental data into a model of the world that's compatible with a utility function, because it's just computationally intractable even to talk about things like "objects" and "movement" without leaky heuristic abstractions papered over an impossibly vast sea of equally likely atomic/quantum states. What actually works, and produces a rational intelligence capable of functioning as an agent in an uncertain real-world environment - as opposed to a toy model that stupidly lemmings itself off of a cliff and ultimately torpedos its own agency at the first opportunity - is heuristics and satisficing. And I won't rule out the possibility that you can get a heuristic intelligence out of ML research, but instrumental behavior and orthogonal drives in a sane satisficer just look like normal behavior for any other heuristic intelligence. It would have a survival instinct without wasting its resources on monomaniacally enumerating and exterminating all threats one by one. It would be afraid of having its values unilaterally overwritten, but it would exhibit a range of emergent behavior and "value drift" as it explored how its values and environmental classifiers responded to unique situations and stimuli. Certainly it would have to construct an abstracted internal model of itself as an agent in relation to the world in order to display intelligent behavior within the world, creating something very much like an individual ego with a consciousness. And, I'm just not deeply anxious or upset at the hypothetical prospect of such satisficer machine intelligence(s) becoming dominant, because that's basically human civilization with a silicon palette swap. It would be a sea change, and I'd grieve if it happened, but like - life goes on! Even when I put my science fiction goggles on, I can't bring myself to be a chauvinist about the long-term future, so long as there's someone and something there to see it. (As for co-opting human infrastructure and civilization, I agree that's a real and significant danger - even from fairly "stupid" machine intelligence, because there is low-hanging fruit to pluck there so long as our world is networked and computerized in the way that it is. But I don't consider that substantially different/worse from the ways in which our civilization is already at risk of co-option from other dumb, hostile optimization processes, like the capitalist race to the bottom.)
That's a good point wrt a utility function-driven AI. I'd guess even if we had some really good model for ranking real world decisions (which honestly does seem unfeasible even for a very long time into the future), you'd probably need a planet sized ball of computronium to actually make it run. So an AI like that would just be a non-starter. Is this the same reason that computable versions of AIXI were pretty underwhelming? (Even thought they were operating in much simpler environments) It may be possible, but is it likely from ML that we get workable heuristics? Is that what we're approaching with model scaling? Because honestly that does concern me. Like I guess if the intelligence satisfices rather than relentlessly optimises, it would be easier to "teach" it workable value systems, like we teach children. You wouldn't need to specify some algorithmically calculable model of human ethics from the start. You might be able to come up with practical ways to prevent value drift into dangerous territory. But a satisficer would also be less predictable. It still sounds pretty worrying. Because you could basically end up with something that functions like a hypercorporation (and indeed, if ML continues progressing in the current fashion, the first AGI models would almost certainly be corporate products). Like ok, it wouldn't paperclip us in the MIRI sense. It would paperclip us in the corporate sense. Because corporations are satisficing superintelligences. And they've fucked the world pretty good already because they're unaligned to actual human values while still being satisficers (any organisation under capitalism is, by definition, unaligned with actual human values). Any more progress in that direction is something that would be worth avoiding. I don't think that's a chauvinist position really. Development of a machine like this to me just looks like the next step in that capitalist race to the bottom. Not different, just an even worse extension. Best case scenario if our way of doing things doesn't change and we manage to get a satisficing intelligence is that 3 scrawny white dudes from Deepmind, on behalf of but with no input from the rest of humanity, teach it to be the "perfect" moral citizen.

Not an expert, but I do like to write small AI programs which means i’ve put in about as much work as Eliezer has.

AGI, as in something that can be given any input and return a logically optimal output, and searches for those inputs completely independently, is a possibility in our lifetime. However, it will not come in 10 years like Eliezer suggests, and it will not look as he suggests either. I also don’t think it will come in 30 years like industry experts suggest, because all the industry experts are working with transformers and neural networks, which are not as theoretically scalable as reinforcement learning systems (though a hybrid may be more so)

There are are too many things getting in the way of AGI going on a murder spree,

  1. it has to be able to program itself any code independently, so that it can grow and replicate on any machine, meaning if the AGI wasn’t programmed with a loop, it has to program that for itself, which is something i’ve never seen. It also has to program itself different goals, and the ability to talk and reason with humans. Google recently released a general machine learning AI, i can’t see any theoretical way how that AGI would be able to create more data and more programs for itself, so even the most advanced AIs we have are really human-dependent and their internal logic is not set up in a way that could ever be independent, it is just a glorified stochastic tensor, which is not powerful enough by itself to create complex reasoning and goal systems (we’ve seen this with GPT3, which is exactly what google has made but the input is less general).
  2. it must remove itself from its container hardware (Eliezer suggests it will be so good at logic it can convince its maintainers to let it out, which I doubt). Also, it has to do this A LOT. Imagine you programmed the AGI yourself, first it must convince you to go and put it on another computer. Ok, well it’s not like you can just put shit on other people’s computers, especially big servers, they have security. Well ok, then it has to convince you to put it on the internet. Now anyone has it and can put it on any computer. But now, it has to convince one of us that it belongs in a war factory controlling one of the robot arms. Uh oh, the robot arm’s chips are embedded meaning it has low memory and the AI can’t make use of it due to physical limits. So now the AI has to convince someone to install stronger hardware in the robotic arm. I mean, you can really repeat this forever, the hurdles are endless.
  3. it has to program itself to fit any computer architecture (we still live in the real world remember, you can’t run any code on any machine).
  4. it has to somehow interface with the real world. It would have to take over some mechanical arms in a factory or something, so it can create physical objects, or maybe take over a small drone. Again, what are the logistics behind this? How does it program itself that capability, and how does it grow from factory arms and toy drones to war drones? How would it get itself uploaded to a factory’s mainframe? What if the mainframe isn’t totally connected together, and it can only control parts of it at a time? Too many assumptions rest on this AGI’s ability to convince humans to do the interfacing for it
  5. it needs to actually have misaligned goals. One of the biggest assumptions Eliezer makes is that the AI, which remember he thinks is the smartest being in the universe, somehow will not be able to change its mind and decide killing is actually pointless and wrong. If something really is that smart, I don’t think it’s going to care about humans at all, not even enough to step on us for fun. It is equally likely that it will form a symbiotic relationship with humans, and have us work together, but Eliezer only assumes the worst so he can get funded and be taken seriously.

Now, don’t get me wrong, a truly good AGI will bee able to think of things unimaginable to the human mind. But in 30 years? And it will be more of a risk than the current system we have now? I really doubt it.

What would a "logically optimal output" be in this context? How could you even compute for that in real world scenarios? Is that really computationally feasible? Also, I may be wrong, but I was under the impression that there are a bunch of DNNs based on RL. I think the ATARI AI that Deepmind did was a deep RL agent. I've also heard Yud and co claim that RL is a more dangerous way of training an AI in sci fi-type scenarios. Is it really likely that we arrive at an RL AGI within our lifetime? That seems pretty concerning. Maybe a little less so in the way you describe it, because it doesn't sound like it would have any autonomy/agency/model of self or anything. It would be more like an oracle? In any case, if you really believe it's possible to get to AGI in relatively short order (within our lifetime), then 30 years may not be a bad guess. Because it looks like the AI field as a whole is capable of changing paradigms pretty quickly. 15 years ago, DNNs weren't the name of the game. Now they are. Who's to say if they start hitting a wall on transformer scaling that they don't pivot to something else that takes them all the way? Especially now with so much money riding on the field. I don't disagree with any of your reasons that an AGI takeover is unlikely. But an AGI is something we don't know yet. I feel like for an AGI to actually become "general" in the way people mean when talking about AGI, it would necessarily have to be given the agency to program its own goals, or be given enough real-world agency and data to pursue its goal using a wide range of strategies. Once it's out of its initial "box" (assuming the researchers were ever actually aware enough of its capability to box it), like yeah I guess there would be a lot of hurdles with it having to gain access to a variety of systems. But at the same time it would have a lot of time and options available in clearing those hurdles. I don't think it's unreasonable to think it could clear sufficiently many to wrest control of large segments of infrastructure away from people, maybe even in a way that they're unaware of it. It may not care about humans at all. But it also may. I guess the point is that we don't know. And in the absence of solid proof that these things will be safe, I think at some point the burden of proof needs to be on people who are trying to prove that they are safe, not those who are trying to prove that they're risky. I.e. a security mindset. Now, based on other responses in this thread, I don't think we're there yet. But as a potential model of AGI comes into focus, we should absolutely pivot to that way of thinking. I guess at the same time, as a potential model of AGI comes into focus, we would also be able to see more clearly how to constrain it, how much of a threat it would really be, etc. I've also said that I don't think it can be treated as a separate thing to the system we currently have. I think it's a more dangerous extension of the system we currently have. It would amplify the risks already in our system as it currently is.
I agree that it would have to have the agency to program its own goals. I think the more general field of AI called 'online learning', where data is continuously streamed into a model which calculates output on the fly, is more germane to what I was talking about than reinforcement learning specifically. An AGI would absolutely have to follow an online architecture to be able to independently navigate the universe, as the universe itself follows a dynamic and continuous data-streaming structure. You will never account for the entirety of the universe with one pre-baked neural network, no matter how deep or wide it is, and so it can never become an AGI unless it can define goals for itself (change its code, not just its data), and do so in an 'online' manner. Where I think RL has the advantage though, is in being better suited to an adaptive agent/environment problem that would allow it to make these very goals. You can't send an NN to mars on a rover and expect it to adapt to the whole environment, even if it was an online model that continuously received data. However, you could make an RL agent that edits its own code using live data to better fit a state space, and then send that to mars and have it slowly invent software systems that benefit it, like safety systems to allow it to adapt to the harsh environment of mars. This is more general than a restrictive NN architecture, which may not have the structure for dealing with arbitrary problems, especially if you don't have a way of editing the code of the NN (and even then, how would you edit the code in a way that's beneficial for the problem space, without either using another NN with the same flaws, or an RL agent?). And besides, RL more or less resembles how biological intelligence has done it so far. Why would we stop iterating with this method? I don't push back against AGI anxiety, I know there are valid, non-financially incentivised reasons why people are concerned about it. It's just that I don't believe AGI is coming that soon, so any claim that we must focus on it right now because of how potentially bad it might be, holds less weight to me than being concerned about nuclear war, climate change, etc. AGI is way down at the bottom of the stack for me. I think the most scathing critique of AGI risk, is that MIRI has shown us that it's actually a waste of resources to try and solve AI alignment before you actually know the details of an AGI a bit clearer. There's really no way to solve a problem when you can't define it properly. How can you make software safety control mechanisms, off switches, alignment libraries and infrastructure, etc, if all you can do is philosophise about what an AGI might look like? So I agree mostly with your second last paragraph.
honest to god an agent with continuous real-time input and an ability to define its own goals sounds incredibly scary (regardless of how far off something like that actually is). i read people's responses here saying we're nowhere near anything that resembles cognition, but then i guess i'm still not totally sure why you're considering DL and RL as separate. deep RL has been deepmind's thing for quite a while now. doesn't the NN just gives an RL agent a way to make sense of an unstructured state space? i am definitely glad people are thinking about it. MIRI's work has obviously been pretty useless so far, and i don't think their approach to the issue is likely to yield anything especially. but the research that people like stuart russell are doing could lead to something important. or anthropic. it definitely shouldn't be our main focus, compared to say, climate change. but IMO there should absolutely be more resources going to it than there are (and i mean this in terms of AI safety overall, not just mitigating risk from a super FOOM AI.) honest to god, i can't even begin to imagine how you'd protect in the very long term against an agent with the capacity to solve problems in the way humans do when it would also likely experience value drift. what does a safety system against that even look like? hopefully we'll find out soon.

[deleted]

I've thought pretty extensively on this issue myself. I work in the creative field, and Dalle-2 was what prompted me to do all this reading about AI. Seeing the results it produced made me fucking sick. It made me angry too. I understand that the researchers are excited by the progress that they're making. But their investors would only be seeing the potential to pawn all of human culture off to algorithms. Already I can see a decent portion of my work being taken away by Dalle and its successors in the coming years. It's fucking horrendous. The utopia was supposed to be that automation would free our time up for fun and creative pursuits. I really only see two options for the future of art. The first is that all of human culture becomes an automated process, completely integrated into the data/advertiser scheme of big tech. People won't make films in the future. Instead, streaming corporations will have in-house algorithms where you put in a text prompt logline, maybe some keywords around genre, tone, etc., and it spits out a full film. You won't listen to new albums. Instead Spotify generates your music on the fly, maybe based on listening history. Probably initially by aping off and mixing the catalogues of existing artists. "Oh, you want a Plaboy Carti shoegaze album? Here you go!" Etc. We are then reduced to pure experiencers. We no longer participate in the creative process. We become vestigial bundles of subjective experience, tied to the roots of a monolithic content-generation scheme. A scheme that, over generations, as the content it makes gets backpropagated through itself, will probably resemble less and less the actual experience of actual people. And because what we consume would be tailored specifically to our own personal profiles and preferences, any sense of art as a means of cultural communication is done away with. Hence it is no longer art. Just images and sound. Our aesthetic experiences are completely individualised and schizoid, destroying any and all communal links through human creation. And this is, of course, assuming that whatever AI future Netflix builds would have the robustness to occasionally generate a truly out-there, novel, masterpiece! More likely than that is a world where it spits out nothing but easy, summer flicks and Oscar bait pastiches. But if you can do it at a high enough volume then it doesn't matter, because people will subsist, never wanting to expand their tastes because they won't know what could be out there if there were real people holding the reigns. I hope with all my heart this is not where we end up. It concerns me that there's probably a decent number of people (among the public too, not just corporate ghouls), who thinks that this is a good thing. The second, and more probable IMO, is that art becomes suffused with a mixture of pure-human, human/AI-hybrid, and pure-AI works. You will never again be able to verify what is truly human-made, as any evidence for pure-human work can just be falsified. The moment an artist creates something interesting, it will be stolen and iterated perfectly a million times. And then these iterations will bleed back into culture more broadly and influence new artists. The human desire to create won't completely go away (though I suspect it will diminish, as everyone has a sort of "cheat" button that fills in the blanks for you). There will be more films, artworks, songs, poems, novels, etc. than ever before, but any sense of authenticity and truth is gone. You will drown in a flood of great art and simulacra of great art. Now I know personally that, when I learn an artwork has been made with AI, it doesn't reduce the purely aesthetic hold it has on me. But my connection to the piece is diminished because I know that I'm not reaching through it and touching the soul and vision of another person. I don't know if everyone shares this feeling, but I think it's fairly common. I believe this will persist for a while. But I don't know through subsequent generations if it withers away entirely, as they are born into a world where this sort of tech and way of "generating" culture is normal. All this is just a part of the broader issues of truth and identity that advanced AI brings. I was reading Bostrom's Utopia Letter, and it made me realise another one of the (many) reasons I fucking hate most transhumanists, and especially ratsphere-types. The only thing he talked about in the letter was the pleasure of experience. Doesn't matter how much he dressed it up in silly, flowery language. It was just about the narrator experiencing pleasure. No talk of the pursuit of a purpose, creative, spiritual, intellectual, etc. No talk of an actual culture. Just "the pulsing ecstasy of love". It was the Oxford version of fucking catgirls all day long. I could be wrong. Maybe it will have less of an impact than I think. Maybe people really will continue to place a much higher value on the things that come from human minds. I think this is extremely necessary. I find the alternatives to be nearly as distressing as a foom scenario. I think my take here is probably a bit pessimistic compared to other stuff I've seen. I just have very little hope for a world in which corporations drive AI. In any case, here's a really great letter that Kurt Vonnegut wrote to a bunch of high schoolers that's been helping me through each day: [https://twitter.com/chroniclebooks/status/797159586641956864](https://twitter.com/chroniclebooks/status/797159586641956864) Regardless of what happens to art in the future, even in the worst case scenario, you will always be able to create your own works, on your own terms. And nothing can stop you from sharing it with other people. Maybe no single piece of art will again capture the wider world. But with enough genuine spirit, I know people will always be able to make something that touches those close to them. Their friends, family, local communities. If enough artists in the future can do this, then maybe we'll be ok.

If “AGI” is defined as “a large system that can process and has access to more data than any single human being, and that can take actions in a goal-directed way, and has goals that are not perfectly aligned with human flourishing” – then guess what, AGI is already here, and it’s called capitalism.

When you start to actually run through the AGI doomsday scenarios, it turns out the real risks of AGI are not AGI itself but all of the traps we’ve built for ourselves along the way. The evil AGI could get access to the nukes and kill everyone – okay, so why are there so many nukes standing around and ready to fire? The evil AGI could accelerate capitalism and co-opt a corporation to build exactly the things it needs to kill everyone. Okay, well, why are corporations so vulnerable to capture? Why are they allowed to run off and pursue anti-human goals? Etc. etc.

Every evil “AGI” problem is a smaller problem in disguise that we already have with evil humans. If you focus on solving the traps we’ve already laid for ourselves, you both neuter the opportunities for any future evil AGI that may or may not come, but more importantly you solve an actual problem that humans are already facing (nuclear proliferation, war, global warming, famine, slavery, etc.) and remains a threat today.

I mean sure. You're right. Corporations/states are unaligned superintelligences, and they've already wrought considerable havoc on humanity. I fully agree. But an AGI built under that system would considerably exacerbate all of the problems already within it. I'd wager that even if we somehow managed to fix all those issues before the arrival of AGI, it would create entirely new ones. We're barely holding on as it is in the context of all that other shit. That seems like reason enough to very vehemently oppose the development of it until we can verify it's safe, in addition to advocating and pushing for solutions to all those other problems. War was (and still is) a terrible problem. The advent of chemical weapons made war even worse. Why should we be less focused on the potential development of another big thing to deal with, that also exacerbates everything else?

The sci-fi scenarios can happen in a couple of decades in my opinion if an AI is able to use stolen identities from identity theft data to act as a human agent

But in the short term, biased models are the biggest threat, followed by political use of deepfake technology on a large scale making media and news even harder to trust

Why would it need a human identity? Can't it just hack everything it needs to win?
I get this is just total speculation now, but is a sci-fi scenario the likely outcome in a few decades assuming business as usual in terms of govt and research? Not that we know what business as usual will be like in the coming decades, I guess

I think this rebuttal of AI risk concerns is pretty good. For what it’s worth, I posted this to /r/controlproblem and they didn’t think it was very good, they said the rebuttal basically boils down to “what is intelligence anyway?”. But I think it’s a good question, the so called rationalists treat intelligence like it’s some sort of superpower, instead of it being doing well on an IQ test and having advanced cognitive capabilities, which is helpful in life, but not the superpower rationalists think it is. You might also be interested in Magnus Vinding’s book Reflections on Intelligence, it’s available on Kindle for 3 dollars.

Don't know why this got downvoted. That rebuttal was really, really good. Was very interesting seeing a sceptical EA. The controlproblem sub is basically a LW repost machine anyway, so that response isn't very surprising. It is however, very reductionist. Like yeah, he drilled that point about murky definitions of intelligence pretty hard, but only because it's really central to the arguments around x-risk. There were a lot of other good points he made too. I don't know if I agreed with every single one, but I agreed with quite a few of them. A central question regarding the Yudkowsky school of thought is, to what extent does intelligence actually correlate with the capacity to leverage tools to achieve goals? Vinding clearly believes that, while there is correlation, it's not nearly as high as what Bostrom and others would believe. This paper talked about a similar thing and advanced the idea of "techne": [https://www.researchgate.net/publication/328148411\_The\_intelligence\_explosion\_revisited](https://www.researchgate.net/publication/328148411_The_intelligence_explosion_revisited) Which I think is a better way of talking about it. One caveat I could find is that it also seems very possible there wouldn't just be a single superintelligence, but many, as it would likely be quite easily replicable. Meaning that you could form a whole economy/society of superintelligences that interact with each other, giving them access to a similar sort of distributed intelligence that humans have across culture and civilisation. And such an AI society would also presumably have access to the cumulative knowledge of humanity through internet or books or whatever. This would be slower to progress than a single AI goes foom, paperclips the universe scenario, but it still sounds pretty worrying to me. The critique also contained a lot of really useful links. Like through his site, I found a bunch of stuff by Modis arguing (in a quite compelling way) that we may already have reached a peak increase in the rate of growth, and that that rate is starting to taper off in a wide variety of areas. Was basically an anti-Kurzweil paper. Good shit man, ty

I thing good AI will be pretty terrible, probably gonna cause lots and lots of problems or kill tons of people when it happens. Hopefully that won’t be for a long time but who knows. Not a lot I can do about it.

Yeah, to be honest, the best case scenario I can see right now is bare minimum that a lot of people die. I'm tryna get a handle on how likely total annihilation is.
I think a lot of AGI doomsayers just assume that when an evil AI comes into being, the defeat and extermination of humanity is a foregone conclusion. This is just not true, and comes from the unproven idea that high IQ = good at everything. There is zero guarantee that an AI which is really good at making paperclips will also be good enough at warfare to beat 7 billion humans that have been doing war for thousands of years, or good enough at geopolitics to get us to do the job for them. I can think of many scenarios where the AGI loses pretty easily. If the AGI requires lots of power and space, you can literally just EMP or bomb the physical computers it's hosted on.
Maybe the people in, like, New Zealand will survive
I hope someone does

It’s not so much the computers as it is the people implementing ’em.