Crust First posted:

Can anyone explain to me why Yudkowsky believes humans could even control his "Friendly AI" to begin with? Surely if we built something (True AI) that was so amazing at self improvement that it vastly outpaces the need for humans, it would likely self improve itself right out of whatever initial box we build it in; isn't he just building blinders that would eventually get torn off either on accident or on purpose anyway?

Does he believe that an AI we couldn't comprehend would still use whatever he thinks is "human logic"? I understand, to a point, wanting to start it off "Friendly", but surely that concept is going to be discarded like an old husk at some stage.

(I'm assuming that he thinks this AI would grow rapidly and incomprehensibly powerful and uncontrollable, since otherwise who cares. I'm not sure this is a realistic scenario but even if it was, why does he believe he can do something about it?)

Crust First posted:

Can anyone explain to me why Yudkowsky believes humans could even control his "Friendly AI" to begin with? Surely if we built something (True AI) that was so amazing at self improvement that it vastly outpaces the need for humans, it would likely self improve itself right out of whatever initial box we build it in; isn't he just building blinders that would eventually get torn off either on accident or on purpose anyway?

Peel posted:

Please continue.

I'll just note that 'steelmanning' is also known as the 'principle of charity' and has been common in philosophy for decades if not longer, if it's something LWers like to lay claim to. I've heard of it a couple times.

Besesoth posted:

So his goal is to figure out (read: "pontificate") a way to force a given new AI to be True Friendly, so that we don't have to have the isolation hostility and can allow the AI out into the world. And since it's True Friendly, the AI will always act in our benefit, which Yud equates with having control over it.

Crust First posted:

Right, this is the point I have a problem with. Why does he think he can do this at all?

Besesoth posted:

Yud wants an AI singularity really, really badly.

Crust First posted:

Right, this is the point I have a problem with. Why does he think he can do this at all? If you believe that an AI can only function like a human does, then I could see saying "and we can give it values that make it Friendly forever" but this is more like, to me, saying, "We must instill human values on these energy beings from the 27th dimension". I assume that he believes the inevitable AI will be infinitely smarter and more capable than any human who has ever lived or will ever live, so why does he believe it can be controlled at all?

It seems like even if you could "force" the AI to be "True Friendly", eventually it will grow beyond the bonds that you set, and then it will either agree with you, or it will seek revenge on you, but in either case it seems like the binding is futile.

Alternatively I guess he could believe that you can make an AI that will never be able to grow beyond what you think you can accomplish, but that seems like a boring AI for a magic future singularity.

I feel like I'm picking apart an episode of a really bad sci-fi show, but I can't wrap my head around both believing that an incredibly powerful AI who can perfectly simulate you (3^^^3 times) can exist, and that it can be pre-programmed to never grow beyond some initial conditions that make it favorable to humans.

Besesoth posted:

friendly or hostile (or indifferent, although LW doesn't seem to admit to that possibility, or for that matter anything other than Pure Friendly and Pure Hostile)

Nessus posted:

What the gently caress are all of these things you just mentioned? This sounds interesting. The last bit sounds like Niven's wireheading, has that been invented then?

Yud posted:

I delay the formal presentation of a timeless decision algorithm because of some
significant extra steps I wish to add

Spazzle posted:

Intercranial direct current stimulation involves electrodes on the the outside of the head, not inside.

ungulateman posted:

Is this a joke post, or do you have a link to the actual thing? As SA's resident person-who-likes-fanfic-too-much I'd like to see this!

su3su2u1 posted:

I just read the entirety of the timeless decision theory paper that Yud self-published through his research institute. After 100 loving pages, we get this:


The paper goes on for 10 more pages without presenting a timeless decision algorithm, and then ends. WHY? WHY WOULD ANYONE THINK THIS COUNTS AS A PAPER?

quote:

So to me it seems obvious that my view of optimization is only
strong enough to produce loose, qualitative conclusions, and that it
can only be matched to its retrodiction of history, or wielded to pro-
duce future predictions, on the level of qualitative physics
"Things should speed up here," I could maybe say. But not "The
doubling time of this exponential should be cut in half."

quote:

Before Robin and I move on to talking about the Future, it seems to
me wise to check if we have disagreements in our view of the Past.
Which might be much easier to discuss and maybe even resolve.

quote:

If that doesn’t make any sense, it’s cuz I was rushed.

quote:

I was rushed, so don’t blame me if that doesn’t make sense either.
Consider that as my justification for trying to answer the question in a post,
rather than a comment.

quote:

Lest anyone get the wrong impression, I’m juggling multiple balls
right now and can’t give the latest Intelligence Explosion debate as
much attention as it deserves. But lest I annoy my esteemed co-
blogger, here is a down payment on my views of the Intelligence
Explosion - needless to say, all this is coming way out of order in the
posting sequence, but here goes . . .

quote:

It hadn’t occurred to me to try to derive that kind of testable predictions.
Why? Well, partially because I’m not an economist. (Don’t get
me wrong, it was a virtuous step to try.) But also because the whole
issue looked to me like it was a lot more complicated than that, so it
hadn’t occurred to me to try to directly extract predictions.

su3su2u1 posted:

I just read the entirety of the timeless decision theory paper that Yud self-published through his research institute. After 100 loving pages, we get this:


The paper goes on for 10 more pages without presenting a timeless decision algorithm, and then ends. WHY? WHY WOULD ANYONE THINK THIS COUNTS AS A PAPER?

Ratoslov posted:

Yudkowsky wants a benevolent AI that can ressurect the dead so everyone can live in benevolent AI heaven forever and ever, and he's willing to believe any arbitrarily absurd set of axioms to get there.

SolTerrasa posted:

This keeps bothering me. I also read that paper, but I assumed I must have missed or skimmed over the math, because legitimately that theory requires well under a page of math. I studied decision theory, my thesis had a strong decision theory component, and his new theory is just not that complex.

SolTerrasa posted:

Either way, gently caress you, Yudkowsky, this is why academia won't let you in.

quote:

How is it possible for the reader to be taken in by the impenetrable pronouncements of — as we shall call him — The Master? The first thing to note is that, in everyday life, it sometimes makes perfect sense to accept a statement before fully grasping it. For example, children accept what adults tell them even before they understand precisely what they are supposed to believe. People endorse the equation of special relativity (E=mc²) or the reality of economic recession while having only the foggiest idea of what such claims really amount to. This willingness to accept an obscure utterance for the nonce, without knowing what exactly was on the speaker’s mind, may actually facilitate the learning process. If you insist on understanding every single word of what you are told, before proceeding to the next step, you may not get very far. Better to bracket those obscure parts and trust that you will figure out their exact meaning later on.

In line with the principle of charity in cooperative communication, people will try to reconstruct the meaning of unknown terms on the presumption that what the speaker utters is true and relevant — particularly when they defer to the speaker as an authority. If what the speaker asserts seem bizarre or false on its face, it is prudent to suspect that the problem lies with your interpretation. The cognitive scientist Dan Sperber has called such utterances, swallowed without proper understanding, “semi-propositional ideas” [3].

As with all mental heuristics, this charitable attitude towards speakers, particularly ones regarded as experts, is liable to exploitation. Not everything that is obscure or apparently bizarre will eventually resolve into something true and relevant.

But then people will find out at some point, won’t they? Not necessarily. Another well-known psychological mechanism may kick in and prevent the listener from stopping the hermeneutic search for meaning after diminishing returns have set in. Psychologists have long known that people are averse to losses. Interpreting obscure prose is a form of cognitive investment, an expenditure of time and energy. If there is no hidden meaning to be found after all, your cognitive efforts will have been wasted. People are reluctant to face their losses, and tend to hold on to assets that have long since failed to deliver any returns.

In a similar vein, someone who has spent years wading through obscure prose will have a hard time facing up to reality and admitting that she has been duped. This is especially true when the quest for meaning is an open-ended one. For all you know, treasure may still be lurking deeper down, if only you are prepared to dig a little further — if only you spend a little more time and effort interpreting The Master’s writings. Some fine day perhaps the truth will dawn on you, or perhaps it never will — there is no way to know except by trying.

To make matters worse, people may persevere in a futile hermeneutic quest because — taking up the investment analogy again — they conjure up imaginary returns. In financial investments, however, at least the losses and gains can be objectively measured, they appear as hard figures on a balance sheet. In the quest for meaning identifying the long-sought treasure is less straightforward. In the hope of rationalizing his investment, the interpreter may be tempted to project all sorts of less-than-exciting “insights” onto the Master’s writings, such as common-sense knowledge or psychological lore. Alternatively, she can read her own musings into the Master’s pronouncements, thus using the latter as a mouthpiece. Naturally, obscure writings are perfect vehicles for such ventriloquism. Psychologists have identified the Forer effect: interpreters tend to read specific claims into obscure statements, mistaking their own creative interpretations for the author’s intended meaning. As Richard Webster wrote, “its very vagueness and obscurity means that it is pregnant with semantic possibilities” [4]. In line with Forer’s observations, this creates an illusion of intimacy: Lacan’s students had the impression that he was speaking for them and for them alone, revealing his insights in a secret code. Everyone ends up understanding The Master — but they all disagree about what is being said.

quote:

It is important to emphasize the intimidating effect of unintelligible prose. In the midst of people who all profess to understand what is being said, it takes courage to stand up and admit that you don’t. The philosopher Paul Ricoeur was brave enough to admit, after attending one of Lacan’s seminars, that he did not understand a word of what was being said, even though he found himself in the company of people who seemed to be in the knowing. Many interpreters have boasted that they, for one, understand Lacan perfectly well. The philosopher Jean-Claude Milner has maintained that the man’s writings are in fact crystal-clear, despite appearances to the contrary, and are hardly in need of any interpretation. Who will be confident enough, after years of investment in Lacanian exegesis, to see through this rhetorical bluster?

In his latest book, David Bentley Hart spends a lot of time lamenting the intellectual poverty of our age and hectoring his (atheist) readers, calling them out for their abysmal ignorance and their puerile misunderstandings. Lacan was fond of insulting and belittling those in his audience who failed to understand him. In a television appearance in 1974, he announced that most of his audience were “idiots,” and that he was surely mistaken to descend to their level. Intellectually insecure readers felt that the difficulty of Lacan’s prose was erected as a natural barrier for excluding those unworthy of his insights. Only the best divers could access the most precious pearls. As Lacan himelf wrote: “If you don’t understand them [my writings], so much the better, it will give you the opportunity to explain them.”

su3su2u1 posted:

Btw, I'd love to see more on the Yud/Hanson debate because otherwise I might find myself actually reading it.

ikanreed posted:

Man, if you told me in the 1990s that there'd be a sizable group of people who'd literally worship the idea of AI, and believe it could cause facsimiles of various vaguely biblical things, I'd have been sure you were referencing science fiction.

quote:

I am increasingly put off by how mandatory cuddle puddles are if one wants to participate in the rationalist community. I’m not into cuddle puddles. I’ve tried it several times, and they’re just not for me. As a result, I am frequently excluded, de facto if not de jure, from many rationalist events. And I really don’t want that to be the case. I don’t see a solution here.

Big Yud posted:

I'm incredibly brilliant and yes, I'm proud of it,
and what's more, I enjoy showing off and bragging about it. I don't know
if that's who I aspire to be, but it's surely who I am. I don't demand
that everyone acknowledge my incredible brilliance, but I'm not going to
cut against the grain of my nature, either. The next time someone
incredulously asks, "You think you're so smart, huh?" I'm going to answer,
"*Hell* yes, and I am pursuing a task appropriate to my talents." If
anyone thinks that a Friendly AI can be created by a moderately bright
researcher, they have rocks in their head. This is a job for what I can
only call Eliezer-class intelligence.

Steampunk iPhone posted:

su3su2u1 posted:

http://www.sl4.org/archive/0406/8977.html

quote:

I am SIAI's cackling mad scientist in the basement. That is my job
function, and everyone needs to get used to the new division of labor. At
least some other people on the programming team will probably be arrogant
mad scientists too. Isn't it enough that we save the world? Do we have to
be frickin' *modest* about it? Now that just seems unreasonable. How are
we supposed to stay sane?

Spoilers Below posted:

There's thankfully been quite the backlash against this kind of thing in academic circles, for obvious reasons.

For those interested in a pretty good discussion of the Hows and Whys of this sort of thing, I can direct you to an excellent article in Sciencia Salon by Dr. Maarten Boudry, http://scientiasalon.wordpress.com/2014/07/07/the-art-of-darkness/The Art of Darkness. He's trashing Jacques Lacan and postmodern theology, but you could easily plug in all the details about Yudkowsky and Less Wrong and the article wouldn't change much.

Some representative quotes:

I am not a book posted:

Please tell me he's speaking metaphorically and not about literal piles of people cuddling each other.

quote:

A Cuddle Party is a great place to rub your boner into the backs of complete strangers without anyone getting mad. Because, according to https://www.cuddleparty.com “An erection is just nature’s thumbs-up sign”.

Political Whores posted:

So is what I assume is extreme sexual frustration/repression a big part of being a rationalist?

su3su2u1 posted:

http://www.sl4.org/archive/0406/8977.html

quote:

You seem to be admitting that you are both rational
AND irrational. Emotional outbursts, arrogance,
self-descriptors of "MAD scientists" all indicate some
kind of irrationality and insanity, while you
simultaneously say rationality comes easy to you.

Could you please explain the discrepancy?

Yudkowsky posted:

I aspire to experience those emotions that I would feel if I knew the
correct answers, not to be emotionless. Oh, plus that standard business
about a foolish consistency being the hobgoblin. We humans are much
happier when we acknowledge our conflicting facets, rather than trying to
stuff them all into a consistent public image. Nothing wrong with being a
mad scientist and rationalist. I can laugh maniacally all I want, so long
as I still get the answers right on questions of simple fact.

In fact, I think I'll laugh maniacally right now. BWAHAHAHAHA! BWAHAHAHA!

I nearly fell into your clever trap, answering in tones of dull
solemnity, just because you asked a solemn-sounding question! I think I
need to be more silly, lest the people around me fall into the trap of
being solemn as well as serious.

To this task I shall now apply myself.

Weerp! Wonk! Warble!

su3su2u1 posted:

I'll just respond with this:
http://slatestarcodex.com/2014/09/27/cuddle-culture/

Yudkowsky posted:

> In my experience, clever people are not always clever *all* the time
> and are not always right *all* the time.

Then I shall aspire to be cleverer than those not-very-clever people whom you experienced. And cleverer than my past self, who was foolish enough to make mistakes.

SolTerrasa posted:

That ancient mailing list is a goldmine, where did you FIND that?

Yudkowsky posted:

> In my experience, clever people are not always clever *all* the time
> and are not always right *all* the time.

Then I shall aspire to be cleverer than those not-very-clever people whom you experienced. And cleverer than my past self, who was foolish enough to make mistakes.

... This is two posts after saying rationality comes easily to him. Proclaiming yourself to be literally infallible is probably not a good choice, Big Yud.

Chwoka posted:

These people have clearly never tried to predict something that can be easily falsified, like a presidential election or the whims of the stock market or a scientific hypothesis. They just sit around theorizing on what it would be like to predict something. I bet if I predicted something, I'd be right, because I'm so smart. And if I predicted wrong, then if I was even smarterer I'd definitely be right!

Well, they do make predictions, but only over such a long time period, and couched in such vagueness, that they can never be quite proven wholly incorrect by the sheer passage of time. Of course, they'll never be right either. But at least they'll be... less wrong.

Tunicate posted:

The funny thing is, the best futurist I've seen wrote for the Ladies Home Journal in 1900

su3su2u1 posted:

I'll just respond with this:
http://slatestarcodex.com/2014/09/27/cuddle-culture/

Yudkowsky posted:

This is how I see the story of life and intelligence - as a story of improbably good designs being produced by optimization processes. The "improbability" here is improbability relative to a random selection from the design space, not improbability in an absolute sense - if you have an optimization process around, then "improbably" good designs become probable.

quote:

Cats have brains, of course, which operate to learn over a lifetime; but at the end of the cat's lifetime, that information is thrown away, so it does not accumulate. The cumulative effects of cat-brains upon the world as optimizers, therefore, are relatively small.
...
So animal brains - up until recently - were not major players in the planetary game of optimization; they were pieces but not players. Compared to evolution, brains lacked both generality of optimization power (they could not produce the amazing range of artifacts produced by evolution) and cumulative optimization power (their products did not accumulate complexity over time).

Hanson posted:

Eliezer offers no theoretical argument for us to evaluate supporting this ranking. But his view does seem to make testable predictions about history. It suggests the introduction of natural selection and of human culture coincided with the very largest capability growth rate increases. It suggests that the next largest increases were much smaller and coincided in biology with the introduction of cells and sex, and in humans with the introduction of writing and science. And it suggests other rate increases were substantially smaller.

The main dramatic events in the traditional fossil record are, according to one source: Any Cells, Filamentous Prokaryotes, Unicellular Eukaryotes, Sexual(?) Eukaryotes, and Metazoans, at 3.8, 3.5, 1.8, 1.1, and 0.6 billion years ago, respectively. Perhaps two of these five events are at Eliezer’s level two, and none at level one. Relative to these events, the first introduction of human culture isn’t remotely as noticeable. While the poor fossil record means we shouldn’t expect a strong correspondence between the biggest innovations and dramatic fossil events, we can at least say this data doesn’t strongly support Eliezer’s ranking.

Our more recent data is better, allowing clearer tests.

...

Eliezer seems wrong there

...

No doubt innovations can be classified according to Eliezer’s scheme, and yes all else equal relatively-meta innovations are probably stronger, but if as the data above suggests this correlation is much weaker than Eliezer expects, that has important implications for how "full AGI" would play out. Merely having the full ability to change its own meta-level need not give such systems anything like the wisdom to usefully make such changes, and so an innovation producing that mere ability might not be among the most dramatic transitions.

Yudkowsky posted:

...
Yes, it's more convenient for scientists when theories make easily testable, readily observable predictions. But when I look back at the history of life, and the history of humanity, my first priority is to ask "What's going on here?", and only afterward see if I can manage to make non-obvious retrodictions.
...

Hanson posted:

If you can't usefully connect your abstractions to the historical record, I sure hope you have some data you can connect it to. Otherwise I can't imagine how you could have much confidence in them.

Yudkowsky posted:

Depends on how much stress I want to put on them, doesn't it? If I want to predict that the next growth curve will be an exponential and put bounds around its doubling time, I need a much finer fit to the data than if I only want to ask obvious questions like "Should I find rabbit fossils in the pre-Cambrian?" or "Do the optimization curves fall into the narrow range that would permit a smooth soft takeoff?"

Hanson posted:

Eliezer, it seems to me that we can't really debate much more until you actually directly make your key argument. If, at it seems to me, you are still in the process of laying out your views tutorial-style, then let's pause until you feel ready.

...

When I attend a talk, I don't immediately jump on anything a speaker says that sounds questionable. I wait until they actually make a main point of their talk, and then I only jump on points that seem to matter for that main point. Since most things people say actually don't matter for their main point, I find this to be a very useful strategy. I will be very surprised indeed if everything you've said mattered regarding our main point of disagreement.

Hanson posted:

With a thriving bot economy, groups would continue to explore a variety of ways to reduce bot costs and raise bot value. Some would try larger reorganizations of bot minds. Others would try to create supporting infrastructure to allow groups of sped-up bots to work effectively together to achieve sped-up organizations and even cities. Faster bots would be allocated to priority projects, such as attempts to improve bot implementation and bot inputs, such as computer chips. Faster minds riding Moore’s law and the ability to quickly build as many bots as needed should soon speed up the entire world economy, which would soon be dominated by bots and their owners.

Hanson posted:

I can’t win a word war of attrition with you, where each response of
size X gets a reply of size N · X, until the person who wrote the most crows
that most of his points never got a response. I challenge you to write a clear
concise summary of your key argument and we’ll post it here on OB, and I’ll
respond to that.

Hanson posted:

...
Conflicting cultures, languages, and other standards often limit the spread of innovations between humans, but even so this info leakage has limited the relative gains to those first with an innovation. The key question is then what barriers to the spread of innovation would prevent this situation from continuing with future innovations.

Yudkowsky posted:

To me, the answer to the above question seems entirely obvious - the intelligence explosion will run on brainware rewrites and, to a lesser extent, hardware improvements. Even in the (unlikely) event that an economy of trade develops among AIs sharing improved brainware and improved hardware, a human can't step in and use off-the-shelf an improved cortical algorithm or neurons that run at higher speeds. Not without technology so advanced that the AI could build a much better brain from scratch using the same resource expenditure.

Hanson posted:

Eliezer, it may seem obvious to you, but this is the key point on which we've
been waiting for you to clearly argue. In a society like ours, but also with one or
more AIs, and perhaps ems, why would innovations discovered by a single AI
not spread soon to the others, and why would a nonfriendly AI not use those
innovations to trade, instead of war?