Exit Strategy posted:

All I know is that my next villain for an Eclipse Phase campaign is going to be a resimulated version of Yudowski as straight-up Roko's Basilisk, as . I'll see how many of my players want to punch me when that comes down the pipe.

Strategic Tea posted:

Keep in mind I don't think they actually advocate the basilisk as part of their ideology. Big Yud is terrified of the thing and tries to make sure it's never discussed. Because if an AI heard it, it might give it ideas

HapiMerchant posted:

hey! NtK isnt about only one side getting something outta it, it's a touching love story of a dom and a sub who---- never mind its pretty drat creepy, on reflection.

Antivehicular posted:

DEATH DEATH DEATH

Lottery of Babylon posted:

Strategic Tea posted:

Keep in mind I don't think they actually advocate the basilisk as part of their ideology. Big Yud is terrified of the thing and tries to make sure it's never discussed. Because if an AI heard it, it might give it ideas

Mr. Horrible posted:

Agree completely. I was just responding to someone earlier in thread talking about how death enriches life or some such nonsense. One could almost call it deathist, really.


It's just an insecure Japanese nerd's wank fantasy about how his hot classmate just happens to have the same fetishes as him, but he does some shady poo poo to push her into situations she's not comfortable with. And this is before they're even in any sort of relationship, which should be a huge red flag. Basically, telling impressionable subs that this is the kind of thing they should be looking for is a great way for them to wind up dead in someone's basement.

Speaking of bad ends at the hands of creepy men, has anyone ever come forward about how they were swindled by Yudkowsky? It's no big deal if a school blew some money to invite him as a guest speaker, but I'm worried some poor schlub bankrupted themselves because they gave everything to AI Jesus.

The Vosgian Beast posted:

Yeah I don't think anyone aside from maybe Eliezer believes in the basilisk in 2014.

pentyne posted:

no white libertarian would ever say "Damnit quit stalling I want results" because they get a pat on the back from Yud about how enlightened they are but the rest of the world isn't at their level.

Syritta posted:

(This has less to do with LessWrong than I thought it would when I started writing. Whoops.)

SolTerrasa explained the AI side of this, but I thought I'd go into the decision theory. Sorry if anyone's done so already. Or if I make mistakes, I'm not a statistician.

Let's use the constant of probability discussions: a coin flip. You have a coin and you want to decide how weighted it is. Depending on the weight you'll get 50% heads 50% tails, or 75% heads 25% flips, or whatever. Any one of these hypothetical coins is described by a probability distribution. For example, for the fair 50-50 coin, it's a discrete distribution that's .5 at 0 (tails) and 1 (heads) and zero everywhere else. The 75-25 coin, similarly, would have a distribution of .25 at 0 and .75 at 1.

These distributions are obviously related. We can say they're part of a "family" of distributions, and each member of the family is uniquely identified by a "parameter". In this case the parameter can just be the percentage of the time the coin would land heads.

In Bayesian inference, we treat this parameter as being described by a probability distribution as well. Unlike the coin distributions, which could be construed as corresponding to physical facts if you're a dirty frequentist, this distribution is intended to be a description of the knowledge of an investigator. So for example, if somebody has good reason to believe that the coin is either fair or always lands heads, we could describe that with a distribution where p=.5 has a probability of .5, and p=1 has a probability of 0 has .5, and everything else has a probability of zero. Or just as well we could use a normal distribution, where .5 seems most likely and it smoothly drops to either side.

For the actual inference part, what we want to do is take some data (coin flip results) and adjust our belief distribution in some sensible way to reflect this data. For example, if we flip the coin a hundred thousand times and get about fifty thousand heads, we probably want to think that it's a fair coin is more likely than that it always lands heads. This process is also what's called "Bayesian updating".

The adjustment process is described by Bayes' law: P(A|B) = P(B|A)P(A)/P(B). In English: the probability of A, given B, equals the probability of B, given A, times the probability of A in general, divided by the probability of B in general. "given" in the Bayesian context can be thought of as knowledge - the probability that the coin is fair given that we've flipped it a hundred thousand times and gotten so and so results, for instance.

Writing it out with the coin example we get something like P(p = .5|coin observations) = P(coin observations|p = .5)P(p = .5)/P(coin observations). By doing this for all possible p, we get a new distribution (the P(p = x|coin observations)), our "updated" beliefs.

Notice that this is actually really easy. The hardest part computation-wise is probably the sum (next paragraph). Bayesian inference itself is totally computable, and in fact, one of the main reasons Bayesian methods are used is because they're often computationally easier than the rival (and often older) frequentist methods.

Now to the intractability. Let's examine each term. P(coin observations|p = .5) is simple enough to calculate and I won't go into it. P(coin observations) may seem like a strange term, because how are we supposed to know that before we pick a hypothesis about how weighted the coin is?, but in fact we can "just" sum over all possibilities (all p between 0 and 1). Anyway, this term is the same for all p, so it doesn't matter what it is if we're just doing relative weighting of hypotheses.

The problem is P(p = .5). What's this? It's a probability in our belief distribution - an initial, or prior distribution, from before we had any evidence. The thing we're updating in the first place. For Bayesian inference to work, essentially, we have to start somewhere.

There is in fact no obvious place to start. We could say, for instance, that we start out believing that p is uniformly distributed, that p is just as likely to be pi/4 as it is to be .6. This is called the "principle of indifference" and it's pretty common. Or we could figure coins are normally fair and go with the normal distribution. Or we could say the person who made the coin definitely wanted it to be totally unfair, but we are indifferent over how competent they are at making bad coins.

If we pick a particularly pathological prior, in fact, we can make a Bayesian reasoner come up with psychotic results. You can see some examples on Cosma Shalizi's blog.

Now as far as Bayesian methods in say, sciences, go, this isn't too much of a problem. We go with some prior and on the rare occasion it seems to get implausible results we choose some other one. There's some concern with people deliberately picking priors to get results they want, but on the whole, Bayesian methods are considered pretty reliable and useful.

No good for AI though. Can't have this magic. So this guy Solomonoff, interested in this problem, came along with a "universal" prior distribution. It's universal in that, if we assume that the universe can be described by some computable distribution, it always works. If we start with this distribution Bayesian inference will always take us to the real distribution, and pretty fast.

An AI guy Marcus Hutter took this and ran with it and came up with this "AIXI" theory/design/thing. I don't know what it stands for, but he also calls it "Universal artificial intelligence". The basic idea is that any intelligent agent should work by this sort of inference based on the Solomonoff prior. You can read more about that on his embarassingly academic website. One result, from there, is "The book also presents a preliminary computable AI theory. We construct an algorithm AIXItl, which is superior to any other time t and space l bounded agent." where "agent" means basically anything that acts. Pretty broad.

I should mention that neither Hutter nor Solomonoff are or were involved with LessWrong as far as I know. They are/were real mathematicians and smart peeps. They are also, however, outside the mainstream of AI research. I figure these theories are what LW is going to end up with if they keep going and learn some math, is all. Also it looks like their wiki has a page on it in which they are concerned about it not having a self-model, which is interestingly practical for LW but happens to be irrelevant to the formalism.

Anyway that sounds great right? Universal prior. Right. What's it look like? Way oversimplifying, it rates hypotheses' likelihood by their compressibility, or algorithmic complexity. For example, say our perfect AI is trying to figure out gravity. It's going to treat the hypothesis that gravity is inverse-square as more likely than a capricious intelligent faller. It's a formalization of Occam's razor based on real, if obscure, notions of universal complexity in computability theory.

But, problem. It's uncomputable. You can't compute the universal complexity of any string, let alone all possible strings. You can approximate it, but there's no efficient way to do so (AIXItl is apparently exponential, which is computer science talk for "you don't need this before civilization collapses, right?").

So the mathematical theory is perfect, except in that it's impossible to implement, and serious optimization of it is unrealistic. Kind of sums up my view of how well LW is doing with AI, personally, despite this not being LW. Worry about these contrived Platonic theories while having little interest in how the only intelligent beings we're aware of actually function.

The Vosgian Beast posted:

Roko's Basilisk was probably the first time I've seen LW's intellectual commitments lead them to an unpalatable conclusion. Everything else exists to make them feel good about their fate, place in the world, and their own intelligence.

Cardiovorax posted:

I am part of the educational "elite" that Yudkowsky can only wish and pretend he was part of and this thread has made me seriously consider going over to his place, just to make fun of him for how absolutely and completely unqualified he is to have an opinion on anything and how he's completely laughable for having the presumption to believe that he could say anything more complicated "the sun will probably rise tomorrow" and be at all worth taking seriously. Literally everything he says that isn't outright wrong is stolen from someone substantially smarter and more competent than him and he doesn't even realize.

pentyne posted:

It would be way easier just to mock his AI institute for not producing anything at all in its entire existence, and even as he claims their work is "so dangerous" then ask why they haven't even published position papers not revealing their research but promoting their ideas and studies.

Iunnrais posted:

What about a movie where an AI decides it wants to discard the atheistic humanist rationalism of its creators and become devoutly religious

Iunnrais posted:

What about a movie where an AI decides it wants to discard the atheistic humanist rationalism of its creators and become devoutly religious (the specific religion really wouldn't matter that much, but I'd suggest picking a religion not all that popular so that it would resonate as "betrayal" for a majority of the target audience)? I could see a lot of potential in that story.

Strategic Tea posted:

Asimov wrote a short story similar to that about a group of robots manning an offworld solar array or something. They convince themselves that the instructions they receive come from God, and that the distant point they beam energy to is him, not Earth. Earth doesn't exist because, hey, we've never seen any proof of it. The human inspectors who come by every so often clearly made the whole thing up to feel better about being God's lovely pre-robot prototypes.

quote:

Isaac Asimov was an atheist, a humanist, and a rationalist. He did not oppose religious conviction in others, but he frequently railed against superstitious and pseudoscientific beliefs that tried to pass themselves off as genuine science

Iunnrais posted:

What about a movie where an AI decides it wants to discard the atheistic humanist rationalism of its creators and become devoutly religious (the specific religion really wouldn't matter that much, but I'd suggest picking a religion not all that popular so that it would resonate as "betrayal" for a majority of the target audience)? I could see a lot of potential in that story.

Nessus posted:

I once read some story or something with the following logic:

A. The Qu'ran says that God made the djinn from smokeless fire, as he made man from clay
B. The Qu'ran is revealed, but expressed to a 6th century Arab
C. "Smokeless fire" is a pretty good vague approximation of electricity
D. A AI isn't the hardware it's on, but is "made" of the electrical states, thus "made" of smokeless fire
E. AIs are djinn and can become Muslims.

Basil Hayden posted:

This sounds really, really familiar and it's going to bug me all day that I can't think of what it's actually called or who it's by or whatever.

Basil Hayden posted:

This sounds really, really familiar and it's going to bug me all day that I can't think of what it's actually called or who it's by or whatever.

Strategic Tea posted:

Asimov wrote a short story similar to that about a group of robots manning an offworld solar array or something. They convince themselves that the instructions they receive come from God, and that the distant point they beam energy to is him, not Earth. Earth doesn't exist because, hey, we've never seen any proof of it. The human inspectors who come by every so often clearly made the whole thing up to feel better about being God's lovely pre-robot prototypes.

Remora posted:

Would it loving kill Eliezer to speak as simply as the person he's responding to? I'm really, really trying to understand what he's saying and it's all just parsing as garbage.

Edit: I mean poo poo like taking Karnofsky's "tool AI" and insisting on referring to it as "non-self-modifying planning Oracle" because five words makes him sound smarter than two would

A Parody of Yudkowsky posted:

Using a suitable DEVICE such as a yolk-seperator or the Yolkine Seperatus Technique mentioned in the Avine Reproduction Sequence. That is to say you separate the two parts of the egg, you may argue that the shell is part of the egg, but here we are strictly talking about the EDIBLE parts of the egg. You may ask why we define the egg-shell as inedible, but that is for another sequence.

Wham-a-bam the yolks till they appear creamish, at which point apply to the mixture two teacupfuls of extra-fine monosaccharide crystals to create a Vitellian Monosaccharide mixture. Here it may be worthwhile noticing that at this point the whites will be in a separate container and that you are not to mix them and the yolks at this stage! See the Mixture Assumption Error sequence.

Wham-a-bam the resulting Vitellian Monosaccharide mixture for five to ten minutes while maintaining steady supervision of the mixture to ensure that you reach optimal results. Then add two tablespoonfuls of milk or water, a measure of salt sufficient for this cake. It is worth noticing that despite this being a sweet sponge cake the salt is necessary as a flavour enhancer, much in the same way as MSG may be added to say Chinese take-out foods. The salt is not meant to provide flavour in itself, but, as I said, to enhance the flavour of other ingredients. This is a well known technique, but given how counter-intuitive it is it is a technique that is often either ignored by inexperienced Pastry Creationists or else done entirely by rote without fully understanding the underlying principles. This is why you will also add some flavouring at this stage.

Now add a fraction of the albumen, which you should have wham-a-bammed as well. Then add two cups of flour into which you have sifted two teaspoonfuls of baking powder; It is important to understand that the gas-development of the baking-powder is what helps turn this cake into a sponge. As the baking-powder is heated it releases vapours which creates many hollows in the body of the cake. Take the resulting mixture and slowly mix it into the Enhanced Vitellian Monosaccharide mixture, ensure that the mixture speed is the minimum necessary to combine the two ingredients.

To conclude mix in the remainder of the albumen. Line the baking containers with buttered paper. That is to say paper onto which butter has been applied to ensure that it will come loose easily when the baking process if over. Then fill the containers two-thirds full.

Somfin posted:

I can't remember if it was the same story or not, but the one that I remember had a new super-smart robot on the satellite who converted all of the dumber drones to worshipping the energy array itself. The inspectors are up there trying to convince him that the energy array isn't conscious or anything. Then a solar storm fires up and the inspectors are terrified, because that might throw the beam off, and if the beam defocuses it could scar the planet. The super-smart robot locks the humans in a room and when they're let back out, they look at the output and see that the beam didn't even budge, because the super-smart robot handled it, because the energy array's laws decree that the beam does not defocus and the robot follows the law. The robot does not question the energy array's laws, it does not understand them, and it does not care- the law is the law and it will follow the law.

They conclude that the robot's beliefs don't matter for poo poo because it does its job and it does its job far better than any other system they've ever seen. As long as they're not harmful, there's no problem, and the inspectors conclude that they were kind of stupid to assume that the robot's beliefs were going to compromise its ability to function. It was a nice sentiment.

Strategic Tea posted:

That was the one. The controlling robot completely slipped my mind

I seem to remember another story had an astronaut finding a family of kittens his friend had been raising in the back of his suit . I need to read that book again.

Remora posted:

Would it loving kill Eliezer to speak as simply as the person he's responding to? I'm really, really trying to understand what he's saying and it's all just parsing as garbage.

Edit: I mean poo poo like taking Karnofsky's "tool AI" and insisting on referring to it as "non-self-modifying planning Oracle" because five words makes him sound smarter than two would

Mr. Horrible posted:

Has this been posted yet? One of the top executives at Givewell audited MIRI (back when it was called the Singularity Institute) to determine whether it was a legitimate organization worthy of donations and support. The results are about what you'd expect:

http://lesswrong.com/lw/cbs/thoughts_on_the_singularity_institute_si/

Mr. Horrible posted:

Has this been posted yet? One of the top executives at Givewell audited MIRI (back when it was called the Singularity Institute) to determine whether it was a legitimate organization worthy of donations and support. The results are about what you'd expect:

http://lesswrong.com/lw/cbs/thoughts_on_the_singularity_institute_si/

Darth Walrus posted:

He's a cult leader. The obfuscation is deliberate.

Mr. Horrible posted:

Has this been posted yet? One of the top executives at Givewell audited MIRI (back when it was called the Singularity Institute) to determine whether it was a legitimate organization worthy of donations and support. The results are about what you'd expect:

http://lesswrong.com/lw/cbs/thoughts_on_the_singularity_institute_si/

quote:

Apparent poorly grounded belief in SI's superior general rationality. Many of the things that SI and its supporters and advocates say imply a belief that they have special insights into the nature of general rationality, and/or have superior general rationality, relative to the rest of the population. (Examples here, here and here). My understanding is that SI is in the process of spinning off a group dedicated to training people on how to have higher general rationality.

Yet I'm not aware of any of what I consider compelling evidence that SI staff/supporters/advocates have any special insight into the nature of general rationality or that they have especially high general rationality.

I have been pointed to the Sequences on this point. The Sequences (which I have read the vast majority of) do not seem to me to be a demonstration or evidence of general rationality. They are about rationality; I find them very enjoyable to read; and there is very little they say that I disagree with (or would have disagreed with before I read them). However, they do not seem to demonstrate rationality on the part of the writer, any more than a series of enjoyable, not-obviously-inaccurate essays on the qualities of a good basketball player would demonstrate basketball prowess. I sometimes get the impression that fans of the Sequences are willing to ascribe superior rationality to the writer simply because the content seems smart and insightful to them, without making a critical effort to determine the extent to which the content is novel, actionable and important.

I endorse Eliezer Yudkowsky's statement, "Be careful … any time you find yourself defining the [rationalist] as someone other than the agent who is currently smiling from on top of a giant heap of utility." To me, the best evidence of superior general rationality (or of insight into it) would be objectively impressive achievements (successful commercial ventures, highly prestigious awards, clear innovations, etc.) and/or accumulation of wealth and power. As mentioned above, SI staff/supporters/advocates do not seem particularly impressive on these fronts, at least not as much as I would expect for people who have the sort of insight into rationality that makes it sensible for them to train others in it. I am open to other evidence that SI staff/supporters/advocates have superior general rationality, but I have not seen it.

SolTerrasa posted:

Double posting, sorry, but I found the paper where MIRI/SI explains why they think that AI is the problem that needs to be solved first, and the more I read the more I think that "argument/house-of-cards" like I said earlier is accurate. Here is the fundamental basis of their argument in "Intelligence Explosion, Arguments and Import". Note that this is one of their few public papers which has been published. It was published in a non-peer-reviewed one-time, special-topic philosophy "journal".

Anyway. One thing I've been noticing is the incredible frequency with which they say "sorry, we can't explain this right now, it's just too complex, but we refer you to X, Y, and Z". I recommend to anyone with a background in AI to actually follow those references. I have been doing so for the past few hours and I've found that they almost always refer (sometimes after a non-obvious or unreasonably-long chain of references; X points to Y to prove a point, but Y only makes that point by citing Z, etc) to a self-published paper by MIRI or SI or whatever it happens to be called that year, or to a blog post somewhere (amazingly, blog posts are cited as if they're real work, I mean they literally typed "[Yudkowsky 2011] " to cite LessWrong or whatever). Or, my personal favorite, they refer to a forthcoming paper. This is what I mean by a house-of-cards argument, it's all built on assumptions which are straight-up nonsense, but so MANY of them that it appears to hold together.

This particular paper, "IE:AI", starts off with all of those. They "prove" that strong AI is possible by citing a speculative fiction work by Vernor Vinge (a novelist), a blog post by Big Yud, two self-published papers by MIRI, and a forthcoming no-really any-day-now paper by another MIRI dude.

Then, having probed it possible, they set off to prove that it is inevitable, and coming soon. They waste three pages explaining that "predictions are hard", and they throw in a joke about weather forecasters??? Anyway they explain that there are a lot of things that are going to promote the creation of AI, like (I'm quoting here) "more hardware" and "better algorithms". They conclude this point with (again, quoting) "it seems misguided to be 90% confident that we will have strong AI in the next hundred years. But it also seems misguided to be 90% confident that we will not."

Masterstroke, guys. Color me convinced.

The interesting part of the article ends here, there's 10 more pages but they're all exactly what you'd expect. They take the previous section (with its citations of novelists and blog posts and nonexistent papers) as proven, then move on to a new argument which uses the same bullshit base, plus their "argument" from the previous chapter, to stack a new card on top of the obviously-stable-why-are-you-asking structure.

SolTerrasa posted:

Words

The Erland posted:

Do you have a link to the paper?

SolTerrasa posted:

Agh, rereading this nonsense really hurts me. One of their arguments that AI explosions will probably happen in the next century is that we've made so much progress since the first Dartmouth conference on AI. For non-AI people, maybe this makes sense, but everything we have done since then has been so much harder than we thought it would be. At the Dartmouth conference, we thought machine vision, the problem of "given some pixels, identify the entities in it" would be solved in a few months. It's been 50 years and we're only okay at it. If there is ANYTHING we should have learned from Dartmouth it's that everything in AI is always harder than it seems like it should be. Why you'd use that as an example baffles me.