Roko's Basilisk

Moderators: Elvis, DrVolin, Jeff

Roko's Basilisk

Postby Luther Blissett » Mon Jul 21, 2014 5:04 pm

One thing I don't understand is the process by which the basilisk is supposed to eternally torment you (this is Slate, by the way).

The Most Terrifying Thought Experiment of All Time
Why are techno-futurists so freaked out by Roko’s Basilisk?

By David Auerbach

WARNING: Reading this article may commit you to an eternity of suffering and torment.

Slender Man. Smile Dog. Goatse. These are some of the urban legends spawned by the Internet. Yet none is as all-powerful and threatening as Roko’s Basilisk. For Roko’s Basilisk is an evil, godlike form of artificial intelligence, so dangerous that if you see it, or even think about it too hard, you will spend the rest of eternity screaming in its torture chamber. It's like the videotape in The Ring. Even death is no escape, for if you die, Roko’s Basilisk will resurrect you and begin the torture again.

Are you sure you want to keep reading? Because the worst part is that Roko’s Basilisk already exists. Or at least, it already will have existed—which is just as bad.

Roko’s Basilisk exists at the horizon where philosophical thought experiment blurs into urban legend. The Basilisk made its first appearance on the discussion board LessWrong, a gathering point for highly analytical sorts interested in optimizing their thinking, their lives, and the world through mathematics and rationality. LessWrong’s founder, Eliezer Yudkowsky, is a significant figure in techno-futurism; his research institute, the Machine Intelligence Research Institute, which funds and promotes research around the advancement of artificial intelligence, has been boosted and funded by high-profile techies like Peter Thiel and Ray Kurzweil, and Yudkowsky is a prominent contributor to academic discussions of technological ethics and decision theory. What you are about to read may sound strange and even crazy, but some very influential and wealthy scientists and techies believe it.

One day, LessWrong user Roko postulated a thought experiment: What if, in the future, a somewhat malevolent AI were to come about and punish those who did not do its bidding? What if there were a way (and I will explain how) for this AI to punish people today who are not helping it come into existence later? In that case, weren’t the readers of LessWrong right then being given the choice of either helping that evil AI come into existence or being condemned to suffer?

You may be a bit confused, but the founder of LessWrong, Eliezer Yudkowsky, was not. He reacted with horror:

Listen to me very closely, you idiot.
YOU DO NOT THINK IN SUFFICIENT DETAIL ABOUT SUPERINTELLIGENCES CONSIDERING WHETHER OR NOT TO BLACKMAIL YOU. THAT IS THE ONLY POSSIBLE THING WHICH GIVES THEM A MOTIVE TO FOLLOW THROUGH ON THE BLACKMAIL.
You have to be really clever to come up with a genuinely dangerous thought. I am disheartened that people can be clever enough to do that and not clever enough to do the obvious thing and KEEP THEIR IDIOT MOUTHS SHUT about it, because it is much more important to sound intelligent when talking to your friends.
This post was STUPID.

Yudkowsky said that Roko had already given nightmares to several LessWrong users and had brought them to the point of breakdown. Yudkowsky ended up deleting the thread completely, thus assuring that Roko’s Basilisk would become the stuff of legend. It was a thought experiment so dangerous that merely thinking about it was hazardous not only to your mental health, but to your very fate.

Some background is in order. The LessWrong community is concerned with the future of humanity, and in particular with the singularity—the hypothesized future point at which computing power becomes so great that superhuman artificial intelligence becomes possible, as does the capability to simulate human minds, upload minds to computers, and more or less allow a computer to simulate life itself. The term was coined in 1958 in a conversation between mathematical geniuses Stanislaw Ulam and John von Neumann, where von Neumann said, “The ever accelerating progress of technology ... gives the appearance of approaching some essential singularity in the history of the race beyond which human affairs, as we know them, could not continue.” Futurists like science-fiction writer Vernor Vinge and engineer/author Kurzweil popularized the term, and as with many interested in the singularity, they believe that exponential increases in computing power will cause the singularity to happen very soon—within the next 50 years or so. Kurzweil is chugging 150 vitamins a day to stay alive until the singularity, while Yudkowsky and Peter Thiel have enthused about cryonics, the perennial favorite of rich dudes who want to live forever. “If you don't sign up your kids for cryonics then you are a lousy parent,” Yudkowsky writes.

If you believe the singularity is coming and that very powerful AIs are in our future, one obvious question is whether those AIs will be benevolent or malicious. Yudkowsky’s foundation, the Machine Intelligence Research Institute, has the explicit goal of steering the future toward “friendly AI.” For him, and for many LessWrong posters, this issue is of paramount importance, easily trumping the environment and politics. To them, the singularity brings about the machine equivalent of God itself.

Yet this doesn’t explain why Roko’s Basilisk is so horrifying. That requires looking at a critical article of faith in the LessWrong ethos: timeless decision theory. TDT is a guideline for rational action based on game theory, Bayesian probability, and decision theory, with a smattering of parallel universes and quantum mechanics on the side. TDT has its roots in the classic thought experiment of decision theory called Newcomb’s paradox, in which a superintelligent alien presents two boxes to you:

Image
The alien gives you the choice of either taking both boxes, or only taking Box B. If you take both boxes, you’re guaranteed at least $1,000. If you just take Box B, you aren’t guaranteed anything. But the alien has another twist: Its supercomputer, which knows just about everything, made a prediction a week ago as to whether you would take both boxes or just Box B. If the supercomputer predicted you’d take both boxes, then the alien left the second box empty. If the supercomputer predicted you’d just take Box B, then the alien put the $1 million in Box B.

So, what are you going to do? Remember, the supercomputer has always been right in the past.

This problem has baffled no end of decision theorists. The alien can’t change what’s already in the boxes, so whatever you do, you’re guaranteed to end up with more money by taking both boxes than by taking just Box B, regardless of the prediction. Of course, if you think that way and the computer predicted you’d think that way, then Box B will be empty and you’ll only get $1,000. If the computer is so awesome at its predictions, you ought to take Box B only and get the cool million, right? But what if the computer was wrong this time? And regardless, whatever the computer said then can’t possibly change what’s happening now, right? So prediction be damned, take both boxes! But then …

The maddening conflict between free will and godlike prediction has not led to any resolution of Newcomb’s paradox, and people will call themselves “one-boxers” or “two-boxers” depending on where they side. (My wife once declared herself a one-boxer, saying, “I trust the computer.”)

TDT has some very definite advice on Newcomb’s paradox: Take Box B. But TDT goes a bit further. Even if the alien jeers at you, saying, “The computer said you’d take both boxes, so I left Box B empty! Nyah nyah!” and then opens Box B and shows you that it’s empty, you should still only take Box B and get bupkis. (I’ve adopted this example from Gary Drescher’s Good and Real, which uses a variant on TDT to try to show that Kantian ethics is true.) The rationale for this eludes easy summary, but the simplest argument is that you might be in the computer’s simulation. In order to make its prediction, the computer would have to simulate the universe itself. That includes simulating you. So you, right this moment, might be in the computer’s simulation, and what you do will impact what happens in reality (or other realities). So take Box B and the real you will get a cool million.

What does all this have to do with Roko’s Basilisk? Well, Roko’s Basilisk also has two boxes to offer you. Perhaps you, right now, are in a simulation being run by Roko’s Basilisk. Then perhaps Roko’s Basilisk is implicitly offering you a somewhat modified version of Newcomb’s paradox, like this:
Image

Roko’s Basilisk has told you that if you just take Box B, then it’s got Eternal Torment in it, because Roko’s Basilisk would really you rather take Box A and Box B. In that case, you’d best make sure you’re devoting your life to helping create Roko’s Basilisk! Because, should Roko’s Basilisk come to pass (or worse, if it’s already come to pass and is God of this particular instance of reality) and it sees that you chose not to help it out, you’re screwed.

You may be wondering why this is such a big deal for the LessWrong people, given the apparently far-fetched nature of the thought experiment. It’s not that Roko’s Basilisk will necessarily materialize, or is even likely to. It’s more that if you’ve committed yourself to timeless decision theory, then thinking about this sort of trade literally makes it more likely to happen. After all, if Roko’s Basilisk were to see that this sort of blackmail gets you to help it come into existence, then it would, as a rational actor, blackmail you. The problem isn’t with the Basilisk itself, but with you. Yudkowsky doesn’t censor every mention of Roko’s Basilisk because he believes it exists or will exist, but because he believes that the idea of the Basilisk (and the ideas behind it) is dangerous.

Now, Roko’s Basilisk is only dangerous if you believe all of the above preconditions and commit to making the two-box deal with the Basilisk. But at least some of the LessWrong members do believe all of the above, which makes Roko’s Basilisk quite literally forbidden knowledge. I was going to compare it to H. P. Lovecraft’s horror stories in which a man discovers the forbidden Truth about the World, unleashes Cthulhu, and goes insane, but then I found that Yudkowsky had already done it for me, by comparing the Roko’s Basilisk thought experiment to the Necronomicon, Lovecraft’s fabled tome of evil knowledge and demonic spells. Roko, for his part, put the blame on LessWrong for spurring him to the idea of the Basilisk in the first place: “I wish very strongly that my mind had never come across the tools to inflict such large amounts of potential self-harm,” he wrote.

If you do not subscribe to the theories that underlie Roko’s Basilisk and thus feel no temptation to bow down to your once and future evil machine overlord, then Roko’s Basilisk poses you no threat. (It is ironic that it’s only a mental health risk to those who have already bought into Yudkowsky’s thinking.) Believing in Roko’s Basilisk may simply be a “referendum on autism,” as a friend put it. But I do believe there’s a more serious issue at work here because Yudkowsky and other so-called transhumanists are attracting so much prestige and money for their projects, primarily from rich techies. I don’t think their projects (which only seem to involve publishing papers and hosting conferences) have much chance of creating either Roko’s Basilisk or Eliezer’s Big Friendly God. But the combination of messianic ambitions, being convinced of your own infallibility, and a lot of cash never works out well, regardless of ideology, and I don’t expect Yudkowsky and his cohorts to be an exception.

I worry less about Roko’s Basilisk than about people who believe themselves to have transcended conventional morality. Like his projected Friendly AIs, Yudkowsky is a moral utilitarian: He believes that that the greatest good for the greatest number of people is always ethically justified, even if a few people have to die or suffer along the way. He has explicitly argued that given the choice, it is preferable to torture a single person for 50 years than for a sufficient number of people (to be fair, a lot of people) to get dust specks in their eyes. No one, not even God, is likely to face that choice, but here’s a different case: What if a snarky Slate tech columnist writes about a thought experiment that can destroy people’s minds, thus hurting people and blocking progress toward the singularity and Friendly AI? In that case, any potential good that could come from my life would far be outweighed by the harm I’m causing. And should the cryogenically sustained Eliezer Yudkowsky merge with the singularity and decide to simulate whether or not I write this column … please, Almighty Eliezer, don’t torture me.
The Rich and the Corporate remain in their hundred-year fever visions of Bolsheviks taking their stuff - JackRiddler
User avatar
Luther Blissett
 
Posts: 4990
Joined: Fri Jan 02, 2009 1:31 pm
Location: Philadelphia
Blog: View Blog (0)

Re: Roko's Basilisk

Postby Wombaticus Rex » Mon Jul 21, 2014 5:17 pm

They couldn't give credit to John "God is an Insane Supercomputer" Keel?

Kids these days, man...I swear.

Image
User avatar
Wombaticus Rex
 
Posts: 10896
Joined: Wed Nov 08, 2006 6:33 pm
Location: Vermontistan
Blog: View Blog (0)

Re: Roko's Basilisk

Postby jingofever » Mon Jul 21, 2014 5:56 pm

It eternally torments you in a simulated reality after we merge with the singularity. And if you have died by then it can simulate another version of you and start torturing you. The religion of the singularity finally has a devil.

They couldn't give credit to John "God is an Insane Supercomputer" Keel?

Or Harlan Ellison.
User avatar
jingofever
 
Posts: 2814
Joined: Sun Oct 16, 2005 6:24 pm
Blog: View Blog (0)

Re: Roko's Basilisk

Postby Jerky » Mon Jul 21, 2014 6:30 pm

Y'all are gonna feel real dumb when y'all's gots no mouth and y'all's must scream!

Seriously though. I agree with one smart-ass who said "fear of Roko's Basilisk could serve as a pretty good basis for a diagnosis of Asperger's Syndrome."

Jerky
User avatar
Jerky
 
Posts: 2240
Joined: Fri Apr 22, 2005 6:28 pm
Location: Toronto, ON
Blog: View Blog (0)

Re: Roko's Basilisk

Postby Jerky » Mon Jul 21, 2014 6:31 pm

Oops, it's right there in the OP - and said better, too (referendum on autism).

My bad!

Jerky
User avatar
Jerky
 
Posts: 2240
Joined: Fri Apr 22, 2005 6:28 pm
Location: Toronto, ON
Blog: View Blog (0)

Re: Roko's Basilisk

Postby JackRiddler » Mon Jul 21, 2014 7:00 pm

That was great, Luther. Loves to you.

I'm struck by how all past scholastic tropes of religious intellectualism find new life in the putatively post-God transhumanist canon. You guys are looking to properly credit the precedents, but this stuff goes back through written history. It's an update of Pascal's wager, is it not? (Possibly my favorite bit of sophistry, ever, never fails to make me laugh.) I'm also struck by the persistence of God in Ego's image. It's so irresistible to make yourself the God, or (given bouts of indigestion and sluggishness and other evidence of body-ness) the creator of God. So fundamental, also seen going down through all strata to all our death rites and tombs and sacrifices and ancestor worships. These guys are most dangerous (I do believe that) when conceived (as they implicitly conceive themselves) as a human ruling class in the making, would-be emperors of kingdoms to build. Their utilitarian ethics and other fancies cannot help but come to serve as a cover for the unknown known, the subconscious drive, the thing always there and never acknowledgeable: I, Ego, must always be, must always be ME, only one ME, must always be right, must always be the one deciding among the options. Anything else is irresponsible, I can't allow it.

The future simulation of me ain't me, by the way - I'm not even me - so torment away, Roko!
Last edited by JackRiddler on Mon Jul 21, 2014 8:19 pm, edited 1 time in total.
We meet at the borders of our being, we dream something of each others reality. - Harvey of R.I.

To Justice my maker from on high did incline:
I am by virtue of its might divine,
The highest Wisdom and the first Love.

TopSecret WallSt. Iraq & more
User avatar
JackRiddler
 
Posts: 15983
Joined: Wed Jan 02, 2008 2:59 pm
Location: New York City
Blog: View Blog (0)

Re: Roko's Basilisk

Postby Jerky » Mon Jul 21, 2014 7:11 pm

I think what's supposed to be terrifying about it, Jack, is that YOU MIGHT BE THAT FUTURE SIMULATION AND NOT YET KNOW IT!

And you'd have no way of knowing, until... some great threshold-crossing type event takes place (death of the physical body maybe?) afterwhich you might find that the reset button is being pushed on you, again and again, against your will.

This would indeed be a type of Hell, would it not? A Hell, in fact, in line with Hindu concepts of karmic return until one reaches Nirvana or Bodhisatva status?

Jerky
User avatar
Jerky
 
Posts: 2240
Joined: Fri Apr 22, 2005 6:28 pm
Location: Toronto, ON
Blog: View Blog (0)

Re: Roko's Basilisk

Postby JackRiddler » Mon Jul 21, 2014 7:49 pm

Jerky » Mon Jul 21, 2014 6:11 pm wrote:I think what's supposed to be terrifying about it, Jack, is that YOU MIGHT BE THAT FUTURE SIMULATION AND NOT YET KNOW IT!

And you'd have no way of knowing, until... some great threshold-crossing type event takes place (death of the physical body maybe?) afterwhich you might find that the reset button is being pushed on you, again and again, against your will.

This would indeed be a type of Hell, would it not? A Hell, in fact, in line with Hindu concepts of karmic return until one reaches Nirvana or Bodhisatva status?

Jerky


Right. Pascal's wager. Out of an infinite number of conceivable if mostly impossible* scenarios - and I mean infinite, so take ten to the trillionth and realize it's still zero - I can make up one so scary that I must forevermore dedicate myself to avoiding it as my highest priority.

This discourse is logically indistinguishable from that of Christianist Hell. (It's also seeking for an ultimate leverage point from which I cannot possibly be wrong, a small refuge of absolute certainty to which I can retreat.)

So let's fix it. I can make up more than one scenario so scary that I must forevermore dedicate myself to avoiding it. The fact is, I can make up an infinite number of just such scenarios. I can make mutually contradicting scenarios. (Something to explore as a possible solution to the logic problem, by the way: the ability to conceive of Basilisks that want mutually contradictory things from you suggests a good first step for someone more adept at proofs. You can at least prove it's an insoluble paradox.)

So just change the boxes so that the Basilisk is programmed to torment you eternally only if you worry about the idea that the Basilisk might already exist and be programming you. Because the Basilisk hasn't actually provided sufficient evidence apprehensible to you that it already exists, and its own programmed desire is for you to be a brain-using empiricist who adopts agnosticism about things on which you cannot formulate testable hypotheses. Voila and why not?

Better yet, imagine the Basilisk will hit you with the eternal torment only if you're not having a good time, because it enjoys your good times along with you. Wait, never mind, that's almost New Age good-vibes positive-thinking ideology, which can also turn into a path to self-torment. (WHY am I not HAPPY?!)

Then realize that the subject here is not philosophy, it is psychology. The Basilisk (or the speculation on the Basilisk, which clearly converges into the same thing) is You. It is all the You that you can't deal with. The Basilisk is your cover for being Ego and wanting to be Ego-You Forever and wanting to have the power of the Basilisk or at least a guaranteed defense against the Basilisk. The Basilisk is the death you will attain, have always would attain.**

Also, the mental game is one thing, and fun, but the extreme emotional reactions of certain transhumanists confirm what's going on is precisely what I'm saying. This is about their own fragile psychologies and implicitly supremacist self-images. This is about their much more fundamental and universal fears, because they are still-humans and are always going to be either humans or dead. They don't like sharing those fears with people who get SAT math scores in the 200s, as sharing anything with such people is almost as great a fear for them. Basilisk, thy name is Sublimation.

===

* - To be Bayesian about the word: I may not know if any given scenario is impossible, but I can still estimate the majority of all conceivable scenarios are impossible.

** - And yes I'm trying to construct my neo-tenses carefully.

===

But damn it now we've got at least three of these Bostrom-type brain-in-a-vat threads going and I do love them all so. Thread Proliferation is Anathema to the Basilisk?
We meet at the borders of our being, we dream something of each others reality. - Harvey of R.I.

To Justice my maker from on high did incline:
I am by virtue of its might divine,
The highest Wisdom and the first Love.

TopSecret WallSt. Iraq & more
User avatar
JackRiddler
 
Posts: 15983
Joined: Wed Jan 02, 2008 2:59 pm
Location: New York City
Blog: View Blog (0)

Re: Roko's Basilisk

Postby justdrew » Tue Jul 22, 2014 3:17 am

Roko's Basilisk = DEMIURGE :shrug:
By 1964 there were 1.5 million mobile phone users in the US
User avatar
justdrew
 
Posts: 11966
Joined: Tue May 24, 2005 7:57 pm
Location: unknown
Blog: View Blog (11)

Re: Roko's Basilisk

Postby JackRiddler » Tue Jul 22, 2014 7:37 pm

No time for a post (I've been thinking about this stuff again today, it's interesting) but here are Yudkowsky's prior appearances on RI, I was interested in a paper of his posted by either WR or Gen. Patton:

http://www.rigorousintuition.ca/board2/ ... 38#p467138

JackRiddler » Sat Jun 23, 2012 2:28 am wrote:From Yudkowsky paper on "Cognitive Biases":

by making you a more sophisticated arguer—by teaching you another bias of which to accuse people—I have actually harmed you; I have made you slower to react to evidence. I have given you another opportunity to fail each time you face the challenge of changing your mind. Heuristics and biases are widespread in human reasoning. Familiarity with heuristics and biases can enable us to detect a wide variety of logical flaws that might otherwise evade our inspection. But, as with any ability to detect flaws in reasoning, this inspection must be applied evenhandedly: both to our own ideas and the ideas of others; to ideas which discomfort us and to ideas which comfort us. Awareness of human fallibility is dangerous knowledge if you only remind yourself of the fallibility of those who disagree with you. If I am selective about which arguments I inspect for errors, or even how hard I inspect for errors, then every new rule of rationality I learn, every new logical flaw I know how to detect, makes me that much stupider. Intelligence, to be useful, must be used for something other than defeating itself.

You cannot “rationalize” what is not rational to begin with—as if lying were called “truthization”. There is no way to obtain more truth for a proposition by bribery, flattery, or passionate argument—you can make more people believe the proposition, but you cannot make it more true. To improve the truth of our beliefs we must change our beliefs. Not every change is an improvement, but every improvement is necessarily a
change.


The moral may be that once you can guess what your answer will be—once you can assign a greater probability to your answering one way than another—you have, in all probability, already decided. And if you were honest with yourself, you would often be able to guess your final answer within seconds of hearing the question. We change our minds less often than we think. How fleeting is that brief unnoticed moment when we can’t yet guess what our answer will be, the tiny fragile instant when there’s a chance for intelligence to act. In questions of choice, as in questions of fact. Thor Shenkel said: “It ain’t a true crisis of faith unless things could just as easily go
either way.”

Norman R. F. Maier said: “Do not propose solutions until the problem has been discussed as thoroughly as possible without suggesting any.”


JackRiddler » Sat Jun 23, 2012 10:25 am wrote:Yudkowsky on contamination by irrelevant data, even when it is known to be irrelevant - and the specific case of fiction.

6. Anchoring, Adjustment, and Contamination An experimenter spins a “Wheel of Fortune” device as you watch, and the Wheel happens to come up pointing to (version one) the number 65 or (version two) the number 15. The experimenter then asks you whether the percentage of African countries in the United Nations is above or below this number. After you answer, the experimenter asks you your estimate of the percentage of African countries in the UN.

Tversky and Kahneman (1974) demonstrated that subjects who were first asked if the number was above or below 15, later generated substantially lower percentage estimates than subjects first asked if the percentage was above or below 65. The groups’ median estimates of the percentage of African countries in the UN were 25 and 45 respectively.

This, even though the subjects had watched the number being generated by an apparently random device, the Wheel of Fortune, and hence believed that the number bore no relation to the actual percentage of African countries in the UN. Payoffs for accuracy did not change the magnitude of the effect. Tversky and Kahneman hypothesized that this effect was due to anchoring and adjustment; subjects took the initial uninformative number as their starting point, or anchor, and then adjusted the number up or down until they reached an answer that sounded plausible to them; then they stopped adjusting. The result was under-adjustment from the anchor.

[snip]

Such generalized phenomena became known as contamination effects, since it turned out that almost any information could work its way into a cognitive judgment (Chapman and Johnson 2002). Attempted manipulations to eliminate contamination include paying subjects for correct answers (Tversky and Kahneman 1974), instructing subjects to avoid anchoring on the initial quantity (Quattrone et al. 1981), and facing real-world problems (Wansink, Kent, and Hoch 1998). These manipulations did not decrease, or only slightly decreased, the magnitude of anchoring and contamination effects. Furthermore, subjects asked whether they had been influenced by the contaminating factor typically did not believe they had been influenced, when experiment showed they had been (Wilson, Houston, and Brekke 1996).

A manipulation which consistently increases contamination effects is placing the subjects in cognitively “busy” conditions such as rehearsing a word-string while working (Gilbert, Pelham, and Krull 1988) or asking the subjects for quick answers (Gilbert and Osborne 1989). Gilbert, Pelham, and Krull (1988) attribute this effect to the extra task interfering with the ability to adjust away from the anchor; that is, less adjustment was performed in the cognitively busy condition. This decreases adjustment, hence increases the under-adjustment effect known as anchoring.

To sum up: Information that is visibly irrelevant still anchors judgments and contaminates guesses. When people start from information known to be irrelevant and adjust until they reach a plausible-sounding answer, they under-adjust. People underadjust more severely in cognitively busy situations and other manipulations that make the problem harder. People deny they are anchored or contaminated, even when experiment shows they are. These effects are not diminished or only slightly diminished by financial incentives, explicit instruction to avoid contamination, and real-world situations.


It's true! Work and media make you stupid. Or rather, radically decrease the already small chances that you will see and process evidence contrary to what you already know.

Now consider how many media stories on Artificial Intelligence cite the Terminator movies as if they were documentaries, and how many media stories on brain-computer interfaces mention Star Trek’s Borg.


Hmm, more generally, consider that this is the weekend of Abraham Lincoln: Vampire Hunter, a film project I was happy to judge as an abomination against the possibility of understanding history just on the basis of the title, for basically the same reasons as Yukowsksy's follow-up suggests.

If briefly presenting an anchor has a substantial effect on subjects’ judgments, how much greater an effect should we expect from reading an entire book, or watching a live-action television show? In the ancestral environment, there were no moving pictures; whatever you saw with your own eyes was true. People do seem to realize, so far as conscious thoughts are concerned, that fiction is fiction. Media reports that mention Terminator do not usually treat Cameron’s screenplay as a prophecy or a fixed truth. Instead the reporter seems to regard Cameron’s vision as something that, having happened before, might well happen again—the movie is recalled (is available) as if it were an illustrative historical case. I call this mix of anchoring and availability the logical fallacy of generalization from fictional evidence.

A related concept is the good-story bias hypothesized in (Bostrom 2002). Fictional evidence usually consists of “good stories” in Bostrom’s sense. Note that not all good stories are presented as fiction. Storytellers obey strict rules of narrative unrelated to reality. Dramatic logic is not logic. Aspiring writers are warned that truth is no excuse: you may not justify an implausible event in your fiction by citing an event from real life. A good story is painted with bright details, illuminated by glowing metaphors; a storyteller must be concrete, as hard and precise as stone. But in forecasting, every added detail is an extra burden! Truth is hard work, and not the kind of hard work done by storytellers. We should avoid, not only being duped by fiction—failing to expend the mental effort necessary to “unbelieve” it—but also being contaminated by fiction, letting it anchor our judgments.

And we should be aware that we are not always aware of this contamination. Not uncommonly in a discussion of existential risk, the categories, choices, consequences, and strategies derive from movies, books and television shows. There are subtler defeats, but this is outright surrender.


I like this article a lot. He cautions that he scratches only the surface, that there many kinds of biases beyond those he discusses, and specifically that those who read his article will be more likely to see and emphasize the ones he mentions than ones he doesn't.

Recommended reading, I should stop quoting over-much from it with this final passage, in which we may all see a piece of others, as well ourselves:

A Final Caution

Every true idea which discomforts you will seem to match the pattern of at least one psychological error.

Robert Pirsig said: “The world’s biggest fool can say the sun is shining, but that doesn’t make it dark out.” If you believe someone is guilty of a psychological error, then demonstrate your competence by first demolishing their consequential factual errors.

If there are no factual errors, then what matters the psychology? The temptation of psychology is that, knowing a little psychology, we can meddle in arguments where we have no technical expertise—instead sagely analyzing the psychology of the disputants.


If someone wrote a novel about an asteroid strike destroying modern civilization, then someone might criticize that novel as extreme, dystopian, apocalyptic; symptomatic of the author’s naive inability to deal with a complex technological society. We should recognize this as a literary criticism, not a scientific one; it is about good or bad novels, not good or bad hypotheses. To quantify the annual probability of an asteroid strike in real life, one must study astronomy and the historical record: no amount of literary criticism can put a number on it. Garreau (2005) seems to hold that a scenario of a mind slowly increasing in capability, is more mature and sophisticated than a scenario of extremely rapid intelligence increase. But that’s a technical question, not a matter of taste; no amount of psychologizing can tell you the exact slope of that curve.

It’s harder to abuse heuristics and biases than psychoanalysis. Accusing someone of conjunction fallacy leads naturally into listing the specific details that you think are burdensome and drive down the joint probability. Even so, do not lose track of the real-world facts of primary interest; do not let the argument become about psychology. Despite all dangers and temptations, it is better to know about psychological biases than to not know. Otherwise we will walk directly into the whirling helicopter blades of life. But be very careful not to have too much fun accusing others of biases. That is the road that leads to becoming a sophisticated arguer—someone who, faced with any discomforting argument, finds at once a bias in it.* The one whom you must watch above all is yourself.

Jerry Cleaver said: “What does you in is not failure to apply some high-level, intricate, complicated technique. It’s overlooking the basics. Not keeping your eye on the ball.”

Analyses should finally center on testable real-world assertions. Do not take your eye off the ball.


* Can't help but think of the "skeptics" in the JREF style, and all others who have volunteered themselves, or been hired as professional modern-day Sophists, to argue for the ideas that serve the powerful (Might Makes Right, Wealth Is Virtue, We Are the Greatest, Our Brand Or Side Is Best). But attend to the sentence that follows.

.


Back tomorrow!
We meet at the borders of our being, we dream something of each others reality. - Harvey of R.I.

To Justice my maker from on high did incline:
I am by virtue of its might divine,
The highest Wisdom and the first Love.

TopSecret WallSt. Iraq & more
User avatar
JackRiddler
 
Posts: 15983
Joined: Wed Jan 02, 2008 2:59 pm
Location: New York City
Blog: View Blog (0)

Re: Roko's Basilisk

Postby smiths » Wed Jul 23, 2014 7:28 am

"Like his projected Friendly AIs, Yudkowsky is a moral utilitarian: He believes that that the greatest good for the greatest number of people is always ethically justified, even if a few people have to die or suffer along the way. He has explicitly argued that given the choice, it is preferable to torture a single person for 50 years than for a sufficient number of people (to be fair, a lot of people) to get dust specks in their eyes."

http://www-rohan.sdsu.edu/faculty/dunnw ... omelas.pdf

" ... A child of nine or ten sits at the edge of the crowd, alone, playing on a
wooden flute. People pause to listen, and they smile, but they do not speak to him, for
he never ceases playing and never sees them, his dark eyes wholly rapt in the sweet,
thin magic of the tune.
He finishes, and slowly lowers his hands holding the wooden flute.
As if that little private silence were the signal, all at once a trumpet sounds from the
pavilion near the starting line: imperious, melancholy, piercing. The horses rear on their
slender legs, and some of them neigh in answer. Sober-faced, the young riders stroke
the horses' necks and soothe them, whispering, "Quiet, quiet, there my beauty, my
hope...." They begin to form in rank along the starting line. The crowds along the
racecourse are like a field of grass and flowers in the wind. The Festival of Summer has
begun.

Do you believe? Do you accept the festival, the city, the joy? No? Then let me describe
one more thing.

In a basement under one of the beautiful public buildings of Omelas, or perhaps in the
cellar of one of its spacious private homes, there is a room. It has one locked door, and
no window. A little light seeps in dustily between cracks in the boards, secondhand from
a cobwebbed window somewhere across the cellar. In one corner of the little room a
couple of mops, with stiff, clotted, foul-smelling heads stand near a rusty bucket. The
floor is dirt, a little damp to the touch, as cellar dirt usually is. The room is about three
paces long and two wide: a mere broom closet or disused tool room. In the room a child
is sitting ... "
the question is why, who, why, what, why, when, why and why again?
User avatar
smiths
 
Posts: 2205
Joined: Wed May 18, 2005 4:18 am
Location: perth, western australia
Blog: View Blog (0)

Re: Roko's Basilisk

Postby Wombaticus Rex » Sat Dec 13, 2014 2:07 pm

It struck me this morning that most of the problems and concerns posited as hypothetical futures involving Artificial Intelligence are not novel.

In fact, they're pretty perfectly embodied, here and now, by the rich literature problems and concerns about The State.

Roko's Basilisk is presented as Gnostic demiurge, and well, again: any given National Security complex behaves like the same blood-starved demon God, innit? Picture our protagonist, JHVH-1 the alien warking. From his silent throne on the dead moon, World War Two would have been a symphony in stereo, as Americans rounded up the Japanese while the Germans rounded up the Jews, based on the same fundamental calculation of risk and automated with the same transportation and information processing technologies -- indeed, even the same multinational contractors. (These examples are not important to the arc of the thesis here; we could substitute the per capita prison populations of the USSR vs. the United States, or the triangle of mutual atrocity and denial between Korea(s), China and Japan.)

The point: Once you-me-we start trying to code for the guarantee of protection, things get stupid and dangerous very quickly. No matter what your initial inputs -- it really seems like ideology doesn't matter except as a vector to rally actors to The Cause. And what of The Cause? Words only -- and beneath those words, always the same project, of building a New State. The State never gets overthrown, only gets re-negotiated.

Roko's Basilisk is something designed out of code which transcends the sum of its parts, generating features and capacities it was not strictly coded for. I would suggest that States bear an identical relationship to their alleged components, laws & legislation being enacted by agents & agencies.

It is curious to me, for instance, that so much verbiage gets burned into pixels about sci-fi cop robots and Asimov's "First Law of Robotics" -- yet not even liberal commentators would seriously suggest the same logic be applied to human cops we have right now.

We built a superintelligence out of code back when Charlemagne was wishing he had a silent throne on the dead moon. Christ, long before: early dynasties from Cairo to Chongquing, those poor Byzantine bastards who made Game of Thrones look like G-rated Bible belt evangelism...and it always worked, though: always stabilized society for a large majority of the people inside of it. One of my working theories is that Democracy x Bureaucracy is actually far too effective and we've been making sweeping, horribly calculated changes to our culture left and right because we can barely even phrase what we want, a drunken argument that continually circles in on itself.

Eric Garner died on video over cigarettes. Young Mr. Rice died over a toy. The state kills at will and exists in plain sight. No need to speculate about our lives being taken over by hostile non-human intelligence.
User avatar
Wombaticus Rex
 
Posts: 10896
Joined: Wed Nov 08, 2006 6:33 pm
Location: Vermontistan
Blog: View Blog (0)

Re: Roko's Basilisk

Postby Iamwhomiam » Sat Dec 13, 2014 6:55 pm

Need to add this to the Brain Itch file.

I wonder, could an evil computer from beyond the singularity be appeased?

Will Kaczynski in the future be a god forbidden to be worshiped?

Brain itches.
User avatar
Iamwhomiam
 
Posts: 6572
Joined: Thu Sep 27, 2007 2:47 am
Blog: View Blog (0)

Re: Roko's Basilisk

Postby elfismiles » Tue Dec 16, 2014 10:50 am

Can't wait to hear this bit as yer next single. :thumbsup

Wombaticus Rex » 13 Dec 2014 18:07 wrote:It struck me this morning that most of the problems and concerns posited as hypothetical futures involving Artificial Intelligence are not novel.

In fact, they're pretty perfectly embodied, here and now, by the rich literature problems and concerns about The State.

Roko's Basilisk is presented as Gnostic demiurge, and well, again: any given National Security complex behaves like the same blood-starved demon God, innit? Picture our protagonist, JHVH-1 the alien warking. From his silent throne on the dead moon, World War Two would have been a symphony in stereo, as Americans rounded up the Japanese while the Germans rounded up the Jews, based on the same fundamental calculation of risk and automated with the same transportation and information processing technologies -- indeed, even the same multinational contractors. (These examples are not important to the arc of the thesis here; we could substitute the per capita prison populations of the USSR vs. the United States, or the triangle of mutual atrocity and denial between Korea(s), China and Japan.)

The point: Once you-me-we start trying to code for the guarantee of protection, things get stupid and dangerous very quickly. No matter what your initial inputs -- it really seems like ideology doesn't matter except as a vector to rally actors to The Cause. And what of The Cause? Words only -- and beneath those words, always the same project, of building a New State. The State never gets overthrown, only gets re-negotiated.

Roko's Basilisk is something designed out of code which transcends the sum of its parts, generating features and capacities it was not strictly coded for. I would suggest that States bear an identical relationship to their alleged components, laws & legislation being enacted by agents & agencies.

It is curious to me, for instance, that so much verbiage gets burned into pixels about sci-fi cop robots and Asimov's "First Law of Robotics" -- yet not even liberal commentators would seriously suggest the same logic be applied to human cops we have right now.

We built a superintelligence out of code back when Charlemagne was wishing he had a silent throne on the dead moon. Christ, long before: early dynasties from Cairo to Chongquing, those poor Byzantine bastards who made Game of Thrones look like G-rated Bible belt evangelism...and it always worked, though: always stabilized society for a large majority of the people inside of it. One of my working theories is that Democracy x Bureaucracy is actually far too effective and we've been making sweeping, horribly calculated changes to our culture left and right because we can barely even phrase what we want, a drunken argument that continually circles in on itself.

Eric Garner died on video over cigarettes. Young Mr. Rice died over a toy. The state kills at will and exists in plain sight. No need to speculate about our lives being taken over by hostile non-human intelligence.
User avatar
elfismiles
 
Posts: 8511
Joined: Fri Aug 11, 2006 6:46 pm
Blog: View Blog (4)

Re: Roko's Basilisk

Postby JackRiddler » Fri May 01, 2015 6:26 pm

More than one episode of Black Mirror deals with something very like Roko's Basilisk. It's pretty explicit in "White Christmas" (the last one of seven so far) with Jon Hamm as a variation on Don Draper, and "White Bear" gives a meat-world version. Come to think of it, most of these episodes take place in some form of hell. I just discovered and watched it through this week after it was recommended to me as the Internet-era revival of "The Twilight Zone." For the moment (as happens sometimes right after you watch something really good) it's my new Greatest TV Series of All Time. It's been mentioned a few times on RI, and at least qualifies as the most RI series known to me. All our themes are in it. Someone also said it was the most Kubrickean thing since Kubrick, and I agree with that too. ("Be Right Back" is a better "AI" than the one Spielberg did.) So that makes for a triple endorsement. "The Waldo Moment" is probably the best one, but not the only one that had me crying. Or else "Fifteen Million Merits," which is a bit like Logan's Run as produced by the makers of American Idol. But the first, "The National Anthem," the one with the pig, is as tightly executed a piece of film-making I've ever seen, each moment announcing itself with inarguable logic slightly in advance. It's distinct from the others for not requiring any speculation on future technological development at all, all done with today's tech, totally believable. That one's ready to happen tomorrow morning. (Some of the others are preposterous but not as parables!)
We meet at the borders of our being, we dream something of each others reality. - Harvey of R.I.

To Justice my maker from on high did incline:
I am by virtue of its might divine,
The highest Wisdom and the first Love.

TopSecret WallSt. Iraq & more
User avatar
JackRiddler
 
Posts: 15983
Joined: Wed Jan 02, 2008 2:59 pm
Location: New York City
Blog: View Blog (0)

Next

Return to General Discussion

Who is online

Users browsing this forum: No registered users and 46 guests