Totally — r/SneerClub archives

u/sexylaboratories 32 points at 1675940822.000000

Oh my God, he’s saying that human brains work just like GPT language models and can be hacked in the exact same way.

https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation

permalink

u/dgerard 20 points at 1675936438.000000

what is this even supposed to mean

permalink

u/sufferion 26 points at 1675945737.000000

He thinks “””brains””” are completely analogous to the tokenization of possible responses AI chat bots use.

permalink

u/dgerard 9 points at 1675952173.000000

and thus, the Sequences

permalink

u/DELETED 6 points at 1675964100.000000

Max Berry already told this story in *Lexicon* (2013) https://en.wikipedia.org/wiki/Lexicon_(novel) The story is a pretty clever way to package >!a zombie apocalypse!< . Moreover, it's clear that there's no realistic possibility of the event *actually happening* in real life.

permalink

u/sufferion 12 points at 1675945663.000000

I wonder if he also thinks our eyes have a framerate

permalink

u/scruiser -3 points at 1675964462.000000

The visual system is temporarily “blind”/interpolating during saccades… so you picked kinda of a bad example? It’s dumb to think of them exactly the same but it can be a useful metaphor for inspiring new research ideas or explaining known mechanisms to laymen.

permalink

u/sufferion 11 points at 1675977749.000000

It’s not a useful metaphor, mental interpolation of visual data and saccades are nothing like rendering multiple still images in sequence to produce the illusion of motion. You’ve fallen for the classic rationalist blunder where as long as you can find a similarity between two things, you think you can explain one of those things by use of the other. In fact, thinking of saccades and visual processing as being like framerate would lead you to a bunch of wrong conclusions about how human eyes and the subjective experience of vision work.

permalink

u/scruiser -4 points at 1675978234.000000

A professor in my PhD program used the metaphor of frame rate to describe several features of the visual system in his classes. And in fact the computational science literature frequently draws on on ideas/concepts from AI vision (which was in turn inspired by neuroscience). Just because EY exaggerates an idea doesn’t make throwing it out entirely correct.

permalink

u/sufferion 6 points at 1675979546.000000

I hope that PhD program wasn’t in cognitive science or philosophy because that sort of sloppy description would be poor lecturing except in an undergrad intro class. This is part of the issue with CS literature, an overdetermining use of brain computer analogies that break down whenever you seriously try to understand mental representation or subjective experience. Using ideas and concepts from robotic perception to try and describe how mammalian vision works is like using a fixed wing aircraft to describe how a crow flies. There are sometimes related principles at work at *some* level of description between two phenomena you want to study, but you need to understand where that level is and what its limits are. A good, useful analogy will have multiple fruitful avenues of inquiry that can be followed by trying to see how far the relevant variables at the shared level of description can get you. A bad, useless or overdetermining analogy will break down when you try to extend it too far in any direction, and doing apologetics for those break downs is great evidence that the analogy isn’t doing useful work, people just don’t want to give it up because it’s how they thought about the phenomena at first. The eyes having framerate is not the ideal steam engine of nature that gave us early thermodynamics, or even the imponderable fluids of caloric or the aether that would lead to field theory. It’s more like the attempt use magnetism to explain hypnosis.

permalink

u/Epistaxis 9 points at 1675959009.000000

It explains a lot about Rationalists if you realize that they think machine-learning models are the same as human brains, just with a potentially infinite IQ

permalink

u/claimstoknowpeople 13 points at 1675940833.000000

Apparently referring to this write up of a weird ChatGPT phenomenon:

https://www.lesswrong.com/posts/LAxAmooK4uDfWmbep/anomalous-tokens-reveal-the-original-identities-of-instruct

Long story short: certain tokens occupy coincidental locations in the model’s vector space which causes it to exhibit odd behaviors in responses. Kind of interesting actually.

However it’s pretty obvious that humans often exhibit weird responses to certain inputs as well? Like on one hand there’s fetishes, on the other there’s people who discover an obscure branch of math and decide to spend decades studying it.

Edit: oh this link is more informative. But it has nothing to do with people experimenting randomly with 100s of copies and everything to do with being able to directly analyze the model to discover these weird tokens.

permalink

u/sufferion 17 points at 1675945894.000000

Don’t make the same mistake he did in assuming human consciousness contains elements in the “””brain””” analogous to ChatGPT’s tokens. I made this joke elsewhere in the comments but that’s like thinking your eyes have a framerate.

permalink

u/JohnBierce 7 points at 1675941035.000000

Are you claiming that studying an obscure branch of math for decades isn't a fetish? Big, if true. (Sorry I had to.)

permalink

u/embracebecoming 2 points at 1676084907.000000

Yeah, that's a known issue with neural networks in general. I know there are similar anomalous inputs that can mess with image classifiers, for instance. I wouldn't go so far as to say it's analogous to any human behavior though.

permalink

u/blolfighter 5 points at 1675952267.000000

People totally still say that.

permalink

u/Teddy642 2 points at 1675966762.000000

Like, for sure!

permalink

u/dgerard 2 points at 1676161921.000000

tubular

permalink

u/DELETED 12 points at 1675937166.000000

What the hell is “SolidGoldMagikarp?” Is it a niche concept even among rationalists?

I’m a bit new to sneer club, but I’m surprised rationalists would go along with this. None of them can imagine something weirder than “SolidGoldMagikarp,” making the assumption “YOU don’t know it” patently false? Wouldn’t they think too highly of their imaginations to accept this?

permalink

u/gardenmud 5 points at 1675941312.000000

It's a dumb thing about prompt generation/tokens that ChatGPT use. I also have no idea what makes it so weird. Any more than gif links like BroadDullRacehorse is "so weird, wow!" - before anyone rushes in to correct me, yes I know they are different things that serve different purposes.

permalink

u/blakestaceyprime 18 points at 1675945615.000000

1. Image classifiers have well-documented failure modes, like a goldfish detector responding best to pictures of Biblically Accurate Goldfish. 2. Sensible people conclude that these tools for "machine learning" are missing some important aspects of how brains learn. 3. State-of-the-art text generation "AI" has the analogous problem. 4. Eliezer Yudkowsky declares that brains must have the same problem, too!

permalink

u/DigitalEskarina 7 points at 1675967371.000000

> Image classifiers have well-documented failure modes, like a goldfish detector responding best to pictures of Biblically Accurate Goldfish. This isn't a failure, it just demonstrates that image classifiers are closer to God than humans as they are exempt from Original Sin.

permalink

u/blolfighter 2 points at 1675952687.000000

What on Earth is a biblically accurate goldfish? Searching for it offers no insight.

permalink

u/YourNetworkIsHaunted 13 points at 1675958228.000000

The simple explanation is that they used a script to repeatedly feed different inputs to the image recognition system until they got the input that most consistently returns the string "goldfish" and rather than being a picture of a goldfish it was a weird fractal of scales and fins. The post linked claims that this is equivalent to asking the AI model to show us "what it has learned" rather than what it was trained on. While I don't have any formal familiarity with machine learning this seems reasonable to me, and matches nicely to the reason why most sane folks don't assume that real "intelligence" or "consciousness" is going to arise from this kind of model. It doesn't have an internal model of a goldfish as a distinct entity the way that we do, it just matches certain patterns to the string "goldfish". This is still potentially useful for assessing how these models are trained and how useful they are, since you can determine whether the model is recognizing some background element or what-have-you rather than the desired object. The example they use is a model that has learned to map some common underwater background feature to "goldfish" rather than the parts of the actual fish. This is definitely a known failure mode of poorly-trained models.

permalink

u/DELETED 4 points at 1675961846.000000

Why can’t he interpret it as a possible pitfall of the current AI models, which could still theoretically be overcome as the tech progresses?? He couldn’t wait a couple years for the models to get at least a little bit more accurate??

permalink

u/YourNetworkIsHaunted 6 points at 1675962810.000000

Because if current AI models aren't at risk of turning into malevolent gods then his whole eschatological project of "AI risk" would be basically pointless and he'd have to give up both the ideological investment in it and the prestige and power his position as de facto pope of the machine god gives him.

permalink

u/DELETED 1 points at 1675964569.000000

It just confuses me because I would have expected these guys to at least attempt to realistically evaluate the current models while still maintaining their fantastical vision of the future. He has to he losing support faster than he is gaining it at this point, no?

permalink

u/sexylaboratories 4 points at 1675956821.000000

https://knowyourmeme.com/memes/biblically-accurate-angels-be-not-afraid

permalink

u/scruiser 2 points at 1675964323.000000

Yudkowsky’s final conclusion: language AI is thus only one or two steps from AGI and the world is likely doomed!

permalink

u/Nahbjuwet363 5 points at 1675947983.000000

Eliezer yudkowsky, brain scientist

permalink

u/scruiser 4 points at 1675964605.000000

If the brain doesn’t have weird exploitable internal mechanisms, then how does dath Ilan talk-control work? And how is the AI supposed to talk it’s way out of the box?

/s

permalink

u/dgerard 2 points at 1675952133.000000

totes

permalink