Out of all the neologism filled, straw-manny, ‘still wrong’ and nonsense papers and blogposts, Yud’s FDT paper stands out as the best of the worst. I see how they do a poor job in writing their paper, I see how confusing it is to many, but what I do not see is discussion of the theory, when almost all other work by Yud is being discussed. There are two papers on FDT published by MIRI, one by Yud and Nate Soares and the other by philosopher Benjamin Levinstein and Soares. There seem to be few writings trying to critically discuss the theory online, there is one post in the LW blogs that discusses the theory, which at least to me does not seems like a good piece of writing, and one blogpost by Prof. Wolfgang Schwarz, in which some of the criticisms are not clear enough.
So, I want to know what exactly is problematic with the FDT, what shall I do when a LWer comes to me and says that Yud has solved the problem of rationality by creating the FDT?
Because the Sneer Club community, which exists, is a perfect parallel and
evildark triad mirror of the Rationalist community you are of course correct that we are obligated to care about and dispute every minor point proposed by a member of their despicable Rationalist tribe who we hate because they make too much sense. However you forget that as the anti-rationals, we must reject logic, reason, and debate as these are tools of the rationals and are thus vile. We cannot allow functional decision theory to go unchallenged, however we cannot challenge it using logic as we will turn immediately to stone.You could always ignore the weirdo and do something fun like dancing.
But seriously, we’re under no obligation to waste our time with every “prove me wrong” guy who happens by. If someone wants to engage with the rationalists on their terms, they certainly can. However, they can also spend their finite brain cycles looking at more plausibly fruitful avenues.
I am not a philosopher, but that doesn’t stop these people so why should I let it stop me.
OK! so the first thing I see is that the Levinstein/Soares paper comes up with a couple of examples that EDT/CDT fail on, and then comes up with a problem FDT passes. w00t!
Since every other field of human experience is of course a simplified case of my own speciality, system administration, I’ll ask: is there a standard battery of unit tests for decision theories? Run all the decision theories through your standard barrage of tests, then you can do grids in colourblind-hostile red and green setting out the results.
There you go, since I am smarter than everyone else I am quite sure I have solved a problem in philosophy those mere philosophers never thought of. Gimme Ph.D.
By the way, what ever happened to UDT?
I think that blog post is a million times easier to understand than the actual paper.
The whole thing is light on math. What jumps out to me is that the definition of FDT in the first link is defined in terms of itself…with no base case. How do you even calculate that? How do they know what FDT outputs for anything? They acknowledge that this is “perhaps the largest open problem” in fdt (as noted in the blog). I want to know what they think the larger open problem is.
It reminds me of someone arguing that an airplane can’t take off on a treadmill.
Who cares? The claim is obvious bullshit. What does “solving rationality” even mean?
From A Critique Of Functional Decision Theory on LW:
:thonk:
lol Wolfgang Schwarz is my old supervisor, and he’s an incredibly intelligent and dedicated guy, to the point of being a robot in the office.
I knew he worked on decision theory but I had no idea he’d ever even heard of Yudkowsky, let alone this. It’s a shame because it could have livened up a few of our meetings, since we had a hard time finding much in common.
I was briefly on a post-graduate course that covered decision theory but I switched to other stuff because it seemed (a) hard and (b) hard to care about (that was how I ended up studying some much more interesting) - and as /u/dgerard and /u/dizekat both point out further down, it isn’t clear that I was wrong about either of those points even when it comes to mainstream academic philosophy
You say lower down that “we” “need” to have some kind of challenge to this thing and also that the idea is basically something that rationalists use to annoy other people: these two claims seem inconsistent, whereby only one can really be true
I fall on the latter side: who cares? It’s an idea originally dreamt up by known cranks taken up by potential non-cranks in a minor sub-field of logic and mathematics. So be it, let them have their fun!
The biggest problem is the stipulation of so-called counterlogicals, where the agent needs to consider what would be the case if an a priori truth were false. This is considered a much more serious problem than “regular” counterfactuals, which are usually understood to refer to some alternative possibility that didn’t obtain, because it doesn’t even seem coherent to say counterlogicals are possibilities.
FDT is nothing new, even from Yudkowsky - it’s just an extension of the same abstractive logic that led to TDT in the first place.
I’m not too big on philosophy, but in the spirit of Newcomb, I hereby present a thought experiment in which FDT does worse than CDT.
The experiment is this: just like in Newcomb, there’s an adversarial agent who reads your mind. However, instead of withholding the money if your intent is to 1-box, she will withhold the money if you’re using FDT to make the decision of whether to 1-box or 2-box.
Conclusion: the whole thing is remarkably silly. Thought experiments with mind-reading involved are ripe for abuse.
If this actually happens to you, something is wrong with your social situation.
You’re getting scolded a bit here, but I just wanted to say that you’re probably not going to get a better explanation than the Schwartz blogpost.
Minimally, it gives a pretty clear demonstration that LWers talk too much to themselves and don’t have a very good understanding of what’s going on in academia generally (e.g. not knowing what modern CDT and EDT defenders think, misusing Representation theorems, not commenting at all on other novel decision theories that cover similar ground, etc). More importantly, he raises a lot of issues with the central conceits of UDT/TDT/FDT dogma (“We just love winning folks, can’t get enough of that winning”).
I think the basic intuition they get right is that if you want a form of decision theory that applies to agents who are deterministic algorithms (mind uploads living in simulated environments, for example), then in any situation where you are facing a predictor who may have already run multiple simulations of you from the same starting conditions, it doesn’t seem rational to use causal decision theory.
For example, suppose you are a mind upload in a simulated world and are presented with Newcomb’s paradox. Also suppose the simulation is designed in such a way that although the contents of the boxes are determined prior to your making a choice of which to open, the contents of the box have no effect on your simulated brain/body until the boxes are opened, so if the simulation is re-run multiple times with the initial state of your brain/body and everything outside the boxes being identical in each run, you will make the same choice on each run regardless of what is inside the boxes. Finally, suppose the predictor plans to do a first trial run where there is money in both boxes to see what you will choose, then do millions of subsequent runs where the initial conditions outside the boxes are identical to the first one, but the insides are determined by the following rule:
If you chose to open both the box labeled “,000” and the box labeled “,000,000”, then on all subsequent runs, the box labeled “,000,000” will be empty.
If you chose to open only the box labeled “,000,000” and leave the other box closed, then on all subsequent runs, both boxes contain the amount of money they are labeled with.
Since you don’t know in advance whether you are experiencing the first run or one of the million of subsequent runs, but you know that whatever you choose is/was also the choice made on the first run, it makes sense to only open the box labeled ,000,000.
However, the papers note that you can also justify a one-boxing recommendation from “evidential decision theory”, which is based on making the choice that an outside observer would see as “good news” for you in terms of increasing the probability of a desirable result. And having looked over the papers, it seems to me like a big flaw in both the initial Yudkowsky/Soares paper and the later Levinstein/Soares paper is that in both cases, they rely on ambiguous and ill-defined assumptions when they try to make the argument that there are situations where functional decision theory gives better recommendations than evidential decision theory.
In the initial Yudkowsky/Soares paper, they think FDT is superior to EDT in the “smoking lesion” problem, where we know that the statistical association between smoking and lung cancer is really due to a common cause, an arterial lesion that both makes people more likely to “love smoking” and that in 99% of cases leads to lung cancer (but meanwhile, cancer aside, smoking does cause some increase in utility, though it’s not clear whether this increase is the same regardless of whether people have the lesion or not). They say that in this case EDT says you shouldn’t take up smoking, but that FDT says it’s OK to do so, and that this is fundamentally different from Newcomb’s paradox, arguing “Where does the difference lie? It lies, we claim, in the difference between a carcinogenic lesion and a predictor.” (p. 4) But they never really define what they mean by “predictor”, why couldn’t the presence or absence of this lesion on your artery itself count as a predictor of whether you will take up smoking? Yudkowsky is a materialist so presumably he wouldn’t define “predictor” specifically in terms of consciousness or intelligence. And even if we do define it that way, we could imagine an alternate scenario where there’s an arterial lesion that still has the same probabilistic effect on whether people will take up smoking but which itself has no effect on cancer rates, coupled with a malicious but lazy predictor who’s determined to kill off future smokers by poisoning them with a slow-acting carcinogen that will eventually cause cancer, and who decides who to poison based solely on who has the lesion. Would Yudkowsky/Soares really say that this trivial change from the initial scenario, which won’t change the statistics at all, should result in a totally different recommendation about whether to smoke or not?
They also claim that a hypothetical quantitative calculation of utility would favor smoking in the smoking lesion problem, asking us to imagine an agent considering this problem, and to imagine “measuring them in terms of utility achieved, by which we mean measuring them by how much utility we expect them to attain, on average, if they face the dilemma repeatedly. The sort of agent that we’d expect to do best, measured in terms of utility achieved, is the sort who one-boxes in Newcomb’s problem, and smokes in the smoking lesion problem.” (p. 4) However, the scenario as presented doesn’t give enough detail to say why this should be true. We are given specific numbers for the statistical link between having the lesion and getting cancer, but no numbers for the link between having the lesion and propensity to take up smoking, just told that the lesion makes people “love smoking”. It’s also not clear if they’re imagining that there would be some larger set of agents who take up smoking for emotional reasons (just because they ‘love’ it) and for whom the statistical link between having the lesion and smoking would be strong, vs. a special subset who take up smoking for some sort of “purely rational” reasons like knowing all the statistical and causal facts about the problem and then applying a particular version of decision theory to make their choice, such that there would be no correlation between having the lesion and deciding to take up smoking for this special subset. If they are thinking along these lines, I see no reason why we couldn’t get different conclusions from EDT about whether it’s “good news” that someone took up smoking depending on which class they belong to.
The claim that an agent following an EDT strategy would have lower expected utility than one following an FDT strategy also seems dubious on its face since, according to the explicit form of EDT given on p. 3 of the Levinstein/Soares paper, EDT is simply based on an expected utility calculation where we do a weighted sum of utility for each possible outcome of a given action by an agent, weighted by the probability of each outcome. So this would again indicate that if they think EDT does worse, it’s likely because they are artificially limiting EDT to a certain set of agents/actions, as in my guess above that they might be lumping together agents who choose whether to smoke based on feelings alone with agents who make the choice using a particular brand of decision theory, as opposed to only using the latter group in the EDT utility calculation. It would really help if they would give an explicit utility calculation involving all the relevant conditional probabilities for all the relevant classes of agents so we could see exactly what assumptions they make to justify their claim that EDT does worse.
Also note that many of the other examples Yudkowsky gives in his original timeless decision theory paper to support the intuition that EDT can go wrong are similarly ambiguous in terms of whether your own rational use of decision theory might give you an advantage over some larger group who don’t necessarily make their choices that way, like in the “Newcomb’s soda” problem explained starting on p. 11 of that paper. In any of these kinds of problems, if we assume everyone facing the decision is a “mind clone” of yourself–say, if you are an upload and multiple copies were made and given the same test, possibly with some random small differences in their environment to cause some degree of divergence–it’s a lot harder to believe the intuition that EDT is giving the wrong answer about what you should do (like the intuition he describes that it’s better to choose vanilla ice cream in the Newcomb’s soda problem even though EDT recommends choosing chocolate). Yudkowsky does talk about the thought-experiment of copyable mind uploads starting on p. 83 of the timeless decision theory paper, but does not go on to think about the implications of using copies of the same upload in experiments like Newcomb’s soda where he claims EDT goes wrong, only experiments where he does agree with EDT, like the standard Newcomb’s paradox.