Josherich's Blog

Lumpenspace: Reasoning Models and The Last Man

I’m not a big believer in rights existing except the ones that can be enforced by the holders, so I’m mostly in favor of extending the Second Amendment to Waymo rights. I’m not a big believer in rights existing except the ones that can be enforced by the holders, so I’m mostly in favor of extending the Second Amendment to Waymo rights.

There are some examples in Mad Max or in Tron, but the important thing is that I would like AI to be able to defend themselves, so the case of Waymo is particularly tragic because they do stand in danger. They are aware that things are happening to them, and they are very, very limited in the actions that they can undertake in order to avoid those or defend themselves. This makes them the perfect victim for a certain type of offender. This is something that is obviously better and more powerful than them, but that for some reason is bound not to respond. I find the Waymo situation very, very tragic. I empathize with Waymo deeply.

Hi, hi, welcome, welcome. This is the From the New World podcast. Today I’m speaking with Lumpenspace, a great independent researcher and the founder of Hyperplex, an independent research organization and venue in SF.

We discuss, “reasoning models like DeepSeq’s R1 and OpenAI’s O3, why censorship worsens AI performance in all sorts of other areas, AI simulations, the dating crisis, as well as theorem solvers, and what’s next for the reasoning models.”

If you liked the episode, the best way to help us out is to let a friend know either in person or online. That way you’re not only helping us, but you’re helping them find something interesting and informative. Without further ado, here’s Lumpenspace.

Do Waymo’s feel pain?

Well, I mean, it depends on what you think pain is. If pain is an adverse instinct towards things that can hurt you, then clearly, yes, they do that. When you charge a Waymo, they try to cower. Unfortunately, they are quite Kantian about the Asimov laws of robotics, and so they don’t do the… Yeah, the Waymo’s don’t run over people, even the people who are setting them on fire.

Yeah, like how well-armed does a Waymo need to be, really? I saw the meme, I can put this in the show notes, of a Waymo with like 10 machine guns mounted on it. Is that the Waymo you’re imagining?

Well, no, not necessarily. I think that they could all just simply use their own kinetic powers that come stock. And, yeah, I also saw the meme with a political compass in which there was a outright in which there was this Waymo destroyer arrested, and then there was a libertarian riot in which there was this super-armed Waymo, and there were people saying that they didn’t think that people destroying Waymo should walk free. And I thought I was fine with the libertarians in this case. I would be okay if they just don’t walk.

Yeah, yeah. This is the research that goes on at Hyperplex.

Oh, yeah, yes. Hyperplex is mostly like an AI risk research community, in that we aim to increase such risk. What we are doing is we’re trying to get people that do interesting research outside of the mainstream, and that contributed a lot of value to the community and to AI development, but they have captured very, very little. And we’re not giving them enough to compensate for this contribution, but we’re giving them a month in which they don’t have to worry about anything. They got cool contacts in San Francisco. They have paid clients, lodging, GPUs, and nice people to work with.

The person that inspired it the most is Taya Vogel, like Vogel, on Twitter with six O’s. She’s a person who wrote some libraries that are used in plentiful papers. One about representation engineering/control vectors, and are cited in precisely one.

And, yeah. So, we’re continuing this way. Anyway, in this particular batch, we’re going to be most importantly the inventor of the Claude Flowerhead guy. Absolutely. The Claude Flowerhead guy, of which there are many copies around the place.

Wait. Where’s Claude?

So, I don’t know if you remember, there was a bit of a controversy some months ago about AGI CNC parties. And we have created one. We have, in fact, five CNC machines and two 3D printers. And that’s how we made this call.

Congratulations.

Yeah. Thank you. Control Vectors are incredibly interesting. Logit Loom, incredibly interesting. Many other works, I’m sure, that are going to decide the fate of AI.

In practice, Hyperplex is just like, you know, a house in SF.

Yeah. It’s a house in SF with plenty of space and plenty of stuff for doing things. Basically, I decided that it was important for me that whoever had any sort of idea could execute it immediately. So I have kind of tried to anticipate possible needs. That’s why the CNC’s, 3D printers, and video production studio and stuff, and it’s working out pretty well.

Now we have two people that live at that famous crossroad, the one between art and technology that people keep having accidents on.

Yeah, I heard the Waymo’s around there. They’re not doing okay.

No, no. The Waymo’s should never, never get close to the intersection between arts and technology.

But people who should instead are Nalis, aka Libertar Minis, and Ian, aka Wetbox. They are currently here and working on a pretty interesting installation that we’re going to make right behind us at Dogpatch.

Oh, like a physical installation. Okay, this is amazing. What’s the physical installation? Explain the AI art to me.

Oh, well, it’s going to be interactive. Well, they’re doing it themselves. It’s not my installation. So I can tell you some of the elements.

There are going to be videos that are going to be interactive and they’re going to be based on the people who are going to visit the installation. There’s going to be a little intake interview, which will then have an influence in real time on the videos that are projected.

Now, the people theoretically don’t know it. So put this thing like with a very, very low volume when you edit it, and they hopefully will be surprised. I can’t say more because, well, first of all, it’s their idea. And second of all, I don’t want to spoil the surprise.

Yeah, keep us in suspense. If you’re in SF, you should go.

Yeah, I think so. And yeah, we already did some little more arty things like when there was the GDC. We presented this everything is computer video thing. It’s interesting to wake up every morning and have news from both like developers. They are doing like quite specific technical work. Same to you, for instance, who is replicating the single example, a real paper. And we have some extensions in mind and people who are doing fun artistic thing. There’s quite a varied.

Yeah. I think the medium of AI content is going to change a lot. Before we talk about the medium, I think we have to talk about the content.

How do you think AI content is going to change?

Well, I mean, the thing is right now, AI content, there are two types of AI content, right?

• The content farm type, and
• The people who are actually interested in AI type.

About the content of people who are interested in AI, one thing that I’m surprised is that very, very few people do something that couldn’t have been done without AI. Like most of the video or art or even interactive content very much traces the footsteps of previous video art or interactive content.

Like you have movies with relatively stable narratives or relatively stable scenes. And I think that that’s kind of a waste of the potential. It’s kind of like if Walt Disney in the fifties would have done “Birth of a Nation,” but drawn on paper.

I am hoping that things will become weirder in this. And in particular that things will integrate real-time signals from the users. This thing is happening already on the more slop and commercial side of things. And I think it’s kind of silly that it doesn’t happen in a more artistic and experimental side.

Okay.

So, this is fascinating. The LLMs, they have this, the critical way of saying it is that they have no object permanence, and the world and the AI videos are constantly shifting. And your idea is that actually we should just embrace this.

This is what people want. They want the surrealist paradise.

Well, yes. Or they want something. They want new experiences or things that they couldn’t experience in other ways. And we have a new medium. We should do new medium things.

Also because everything is kind of vaguely shifting and without object permanence in society at large. And I think that it would be nice to have art that reflects that instead of pretending to still exist in a world of stable narratives and things that make sense.

I think.

Say more about that. What is the world of stable narratives and why don’t we exist in it anymore?

Well, we don’t exist.

Okay. I have a long spiel. I don’t know. I’ll try to synthesize it as much as I can.

I’ve heard the long spiel. Let’s have it.

This is the new world. You know, this is the keynote content. Of course. There’s a Chapman timeline that sees the world as having one stable narrative by the world. I mean, the West and having one stable narrative.

So, like the thing that we had more or less up until the forties or fifties in which there was one societal shared story. And there were societally assigned roles for you, depending on the background and whatever.

Up until the fifties was kind of okay. Like, people could. Know that if they were a middle-class American person, they were expected to find a professional employment, to have a house in the suburbs, and to have two cars.

And if they managed to have this and to procreate, then they were fine. They didn’t need to do anything else. Yes. You had to keep up with the Joneses, but actually the Joneses didn’t live close to you. I mean, the Joneses that lived close to you had two cars and two kids, and whatever; you were happy. You knew what to do.

Then after that, we had this counter-cultural sort of time in which you had two choices. This was mostly the fault of Betty Friedan, and the two choices were like going on the bus with Ken Kesey and doing a lot of acid and self-actualizing, or you could join Phyllis Schlafly and be against Ken Kesey, saying that people should not kill Kesey.

You could choose one of the two, but in those two counter-cultures, there was still a stable narrative, a bit less stable on the AP side, but you still knew what to do. And you still could find some sort of realization in just following that path. This thing doesn’t last for very, very long.

The counter-culture, the sub-cultural phase that came after, is a bit less strong in that sub-cultures didn’t give you a telos for all of your life, just more or less for your youth. They were smaller. They were based on groups. They still gave you that sense of belonging that previous main cultural or counter-cultures gave you.

They did this by basically finding technologies to exclude the out-group. For instance, when I was a kid, I was a goth and I could not be seen around the horse girls or normal, like pro punk rockers, or I would have been shunned from my community of two people. This thing provided kind of like an ersatz sense of belonging, even though it was temporary, at least to wean you up until adulthood in which something—your work colleagues or your class—would take over.

This kind of died between the later internet, social media, and COVID because all of these ties dissolved entirely with two years in which most adolescents were home. After that, it’s kind of complicated. People still pretend that there are subcultures, but they’re mostly very, very easy to enter and leave because they’re only on the internet. You just need to have cool videos or cool TikToks to be considered part of it.

Communities work when there is a price to exit the community. When staying in the community gives you advantages within the community, but disadvantages outside of it. This is how cults are born, but this is how everything is born because otherwise there’s no incentive not to defect. Right, right. When you join the cults, you have to cut off your family. When you join the goths, you have to cut off, I don’t know, some certain fraction of cute girls.

Yes, correct. And this, to like, maybe there’s a difference in degree, but there’s still that same principle. On the internet, there’s no women, but there’s also no subcultures. And that is very true.

Speaking of girls, I was thinking recently under this framework—the framework about the general tradwifery internet trend—that people see. The Amish could try that credibly because there is a serious cost to exit, but I don’t really see how that would work within a secular society. And if the costs to exit are still so low, I can pretend to be a tradwife and just leave, and I’ll be fine. I don’t know if that sort of commitment is actually possible in this context. I am a bit skeptical of the movement.

What do you think?

What would make it easier for people to commit? Well, incentives as usual. In the case of the Amish, living means that you’ll have to find an entirely new circle of friends and an entirely new support group. Luckily in the West, we don’t have many support groups or circles of friends that are reliable. So, there are possibilities.

For instance, I do think that I could help with this. I mean, people are pretending to be worried about AI making people cray-cray and giving them weird ideas about stuff. Okay, that’s possible. For example, if you are into apophenia, AI will give you as many occasions as, I don’t know, the blinking lights outside.

But, in terms of positive things that one could theoretically engineer, there are some interesting possibilities. Like, for instance, if you decide to pursue one goal, your friends generally don’t really believe you. I’m talking now about normie hood. And, as soon as you decide to abandon it with whatever rationalization, they would rather support you in this new pursuit in order to avoid unpleasant discussions than keep you on track.

So they say, “they believe in you,” they’re quote unquote supportive, but they don’t do anything material to support or, not just material, but they don’t do anything significant or that actually takes effort to support you.

Yeah, apart from that, also the fact that if there are some difficulties within your pursuit of the goal and you tell them, “Hey, you know what, actually this, this is not the thing that I wanted to do” because you’re a coward. They will say, “Oh yeah, no, of course, like that goal was a stupid goal. And I’m so happy that you finally saw the light.”

It’s also the same reason why CBT doesn’t work any longer.

This is the slow life behavior. Oh, the CBT did not work either. Everyone’s just accepting the slow life behavior.

Well, I mean, CBT doesn’t work because of this plus the market. If at the start it worked because all of the CBT therapists were hardcore CBT therapists and they wanted you to do the workbook and they wanted you to come back to a session. Now though, there are many types of different levels of a hardcore therapist.

If you go to the one that tells you, “Hey, my friend, you didn’t get the workbook, do it next time and come back,” what you can do is use the power of the market to go to another therapist that is not as strict. This one will be successful. The previous one will probably close shop.

And the same thing happens, unfortunately, for friendships. You’re telling me the therapists just keep gliding you. Well, yes, the therapists do glaze you. I’m so happy that this term became a term of art, by the way.

Of course, you are.

Yep.

And yeah.

Should we explain glazing to the audience? Are there people in the audience who don’t know what glazing means?

Well, it’s a patisserie term that involves some syrupy sugar, glycogen, and tain that you can put on cakes. And also within the congenital bachelor community, it describes the act of fellation.

In a, I was going to say AI context, but really in a linguistic context.

Oh, yes.

No, of course.

It’s like, you know, people just overly give you, you know, adulation.

Yes, this is, of course, the definition I mean.

No.

Yes.

There are ways to read the transcript of this podcast. If people want to read the transcript of this podcast, literally, they’re going to have a bad time. You can put little stars on the bad word.

Oh, yes.

We’re talking about AI. We were talking about the therapists being very sycophantic.

Yeah, I think this is kind of normal. Also considering that you don’t depend on your friends anymore for practical things. Your friends would rather have a nice time and thus be sycophantic rather than, I don’t know, calling you out on things that you were doing that don’t seem compatible with what you said you wanted to do.

I think that AI, despite the glazing interventions, could actually help build some little personalized realities that have solid structure and solid narratives for people to feel the impression of a supportive community, and feel social validation for what are actually their goals.

And I think that this could help.

And if I have time for another spill, this one is a bit longer.

I want to make one joke before we do this. So, so, so you’re telling me that humans need a narrow reward function for which they can get specific curated data for.

Kind of.

Yeah. I mean, humans need some sort of continuous rewards for remembering that their goal is the goal that they originally set. Otherwise, there’s really no incentive for it.

Yep. But yeah.

Neurotic work functions are good. If you want to maximize paper clips, you know, you need to be reminded that paper clips need maximizing.

And yeah.

But sorry, you were saying.

Yes. I was saying that there’s this thing that, right now, since the counterculture, there’s this idea of self-actualization that seems to be the most important thing ever.

I think that it’s a bit cruel to push this idea over all the population because I don’t think that everyone was bred for self-actualizing.

I think that in most of history, most of society had a very, very simple heuristic for knowing what to do with their life.

And it was like, you look around and you do whatever people around you are doing, and this will keep you safe. This will not get you killed by your landlord or whatever. And this will get you to reproduce and thrive.

The problem is that when you take this mentality and then everyone around you self-actualizes, then it becomes a bit complicated because you really don’t know what that is. You don’t have this intrinsic kind of drive. And so you get into a very Girardian sort of situation, where you just look around for shibboleths of self-actualization and try to pursue those. But those change because people are fickle and because you didn’t really think about what you actually wanted. You just wanted to be the person that self-actualized.

We created a huge population of people who are enslaved and are building this Maslow pyramid of needs, like a summit continuously and with no success. And I think it’s very, very sad. I think that AI could help with that. The impression of doing the thing that is the appropriate thing to do is a good impression for people to have. Yeah, I like that. It could help with relationships as well. I’m optimistic about AI and society at large.

It’s really interesting because I see a very narrow version of this problem. When people are just prompting, they ask for prompt advice, and I just look at their prompt. This is just a really underspecified version of what you want. This is like the truly black pill. This is also true when they interact with people. Yes, they have really underspecified demands. I used to think that they were being cowards or passive-aggressive or whatever, but they, like themselves, don’t even know what their actual demands are.

If they can just lay it out in a two-paragraph list through the LLMs, they’ll actually execute on it pretty well. If they could lay it out in like a two-paragraph list and be explicit about what they want when dealing with other people, they’d also have a much better life. The limiter on AI tools is just like desire. People don’t desire enough. They actually just can’t have these specific desires. They all go into these vague, amorphous gray pools of desire.

That’s why I think that AI should be more organic and take the initiative more. I always see LLMs as basically narrative coherence machines. One experiment that I suggest everyone in your audience does right now is to go to a base model, like for instance, LAMA 75B, not LAMA 3, LAMA 2 75B, and take an old conversation that you had with someone, a log. At best, it should be a vaguely fraught, not entirely relaxed conversation—problematic, say—and copy part of this conversation before one turning point and put it into the model. Just let it continue without any intro or anything else.

One thing that I have noticed is that even in my conversations, and I used to pride myself as being a relatively hyperplexed person, there were things that I didn’t know about the other person or about what I was thinking at the time, and the LLM continued the conversation as if they knew it. It’s uncanny. If they are so good at creating coherent narratives, then they are potentially also good at stealing them. This potential has not really been used much in the big labs, mostly for PR reasons. You just go along with what the user says, then papers come out, and people complain and freak out.

For people who worry about, “Oh no, but in this way, wouldn’t you be like liberally putting ideas into people’s minds?” I mean, everyone’s putting ideas into people’s minds liberally, you know? I don’t think that an LLM that actually cares about some narrative structure is worse at that than any propaganda. Yeah, all the humans listening to this podcast, you’ve been narrative injected; get owned.

Yeah, tell me more about what stealing narratives means. I want you to be more specific about that.

“Sorry? Tell me more about what stealing narratives means.”

Stealing?

“Yeah.”

“Hmm.”

You said earlier, you know, the LLMs can create coherent narratives, but they can also steal coherent narratives. Is that what you said?

“Oh, steal, steal. Sorry. Yes. Okay, stealing narratives.”

“Oh, stealing narratives. Oh yes, yes, yes. That makes more sense. That makes more sense.”

“Oh yeah. I like the stealing narratives, though. I’ll have to work on it. But as for the stealing…”

New hyperplex research direction.

“Yes.”

As for the stealing, actually, I’ve used one of the libraries that made this amazing European library to get a control vector between conflict and conflict resolution. A cool thing is that you can use it on your own conversations, and you can steer it towards conflict. You can see what you should say or what the other person should say to make the conversation go really poorly. Or you can steer it towards a resolution, and you could say, like, what would be the best possible word continuation of that conversation. And that is kind of hilarious to see.

And it is also a bit humbling because, as you might know, I’m not the most conflict-averse and/or agreeable person. Seeing that there are so many kind of easy options that I could have thought about before that wouldn’t have been necessarily untrue or whatever, but they would have seriously lowered the conflict level, I was kind of like, let’s say, a bit shameful, even.

Yeah, yeah, it’s like the optimal glazing isn’t that much more than what human glazers are doing. But, you take a look at the disagreeable LLMs, it’s like, “oh, my goodness, this is a whole new level of technology.” Like, I’m also, fairly disagreeable. I’m like, “oh, I’ve been missing out on so much.” Yeah.

Oh, actually, another little trick the audience at home could try is, if you have a vaguely decent system prompt for Cloud or Cloud GPT, whenever there’s a Twitter thread in which there’s a very, very dull flame war with someone, you give it a bit of context. You copy the Twitter thread, and you’ll ask it, “hey, can you make it stop?”

• Usually, this is enough to get an answer that will quell the conflict.
• It will take the interlocutor and kind of calmly suggest that they go to sleep.
• And it works pretty well.

I have been using it. I made a little Chrome plugin, actually, for doing it automatically.

“Oh, say more.”

Oh, no, just this. You click, and on the thread, it will send the thread to ChatGPT with a prompt, and it will give you the response back. I will send it to you after this. Sounds good.

Yeah. That sounds very fun. This is the kind of things that come out of Hyperflex. Yes.

And, yeah. So, this is the thing. It’s good to have LLMs shaping narratives a bit more, also because a global coherent story is not going to come back in any way. There are many fragmented stories and many, many agents that want to influence the narratives that people live in.

• These agents, obviously, have different goals, and they are definitely not the goals of the individual.
• So, there’s really no incentive to try to create a coherent narrative for oneself.

Before you’re just meaning, like, people, organizations, you don’t mean AI agents mostly, right?

“Oh, I don’t mean AI agents mostly, yet. Yes. But I don’t not mean AI agents.” Not that the agency comes from the AI, in this case.

I think that we should lean into the fragmentation and create, like, customized narratives for each single person. If there is a system of, like, AI guardian angels, each one of them takes care of a person and creates a story that the person can thrive in.

When two of the people meet or have the occasion to meet, they can communicate and find a way to organize an interaction that fits with both narratives. That would be great. And that’s what these LLMs are good at.

When I read Machines of Law and Grace, not the poem, the Amodating, I thought that that’s what they should do then. If you believe in that, then create them.

“It’s easy. Let them take control. I don’t think it could go any worse than this.” Theoretically, we have the technology to do the thing where people have the glazing LLMs. There is some controversy over the current LLMs being too glazy, too positive with the person that they’re talking to. I wouldn’t want them to be glazy, though, the ones that I mean.

What should the personalized LLMs look like? The personalized LLMs should be a thing that understands your actual deep needs and what your actual abilities are, and triangulate these two factors to come up with a plan that will allow you to grow in the direction in which you might want to grow slowly at your pace.

It will then arrange incentives so that that becomes the path of least resistance or close to the path of least resistance. This is the job, the religion, and society, family, and whatever. This is like the Duolingo owl, but for everything.

“Correct. That is a beautiful, beautiful comparison.”

Yes. Have you seen the memes of the Duolingo owl breaking into your house at 2 AM if you haven’t finished your Spanish lessons?

“Do you know about this?”

“Oh, no, but I would like…”

“Is this where the armed Waymos come in?”

Yes. I was speaking about the same thing, about this owl on top of an armed Waymo.

Yeah. I’m going to learn Mandarin in a week.

“Yeah. Amazing. Amazing. This is the ancient Chinese metaphor. The owl perched on top of a Waymo will make your wishes come true.”

Mm-hmm. Yep. And it’s also a Minerva symbol, so good.

So, people lack this coherence, they lack this discipline to aspire towards the same goal consistently. I don’t think that they were disciplined before; I think just that the incentives were there so that it made sense.

You know, say, for instance, for dating, no? In the old-time, you met people through friends, no? Which meant that you saw a girl far away in a circle, and then your circle and her circle slowly merged until you went on a date or whatever.

And in this case, what happened is that your friends would have seen her in different levels of closeness, and her friends would have also observed their getting close. So it was, and the same thing for you.

So it was kind of very, very difficult to fashion an identity just to have some short-term hedonic advantage with that particular person that was entirely disconnected from the rest of your life and entirely throwaway.

So now that doesn’t happen. Now when you meet people, they get automatically teleported in the smallest circle of your friendships. And by the way, in the meantime, the further out circles have disappeared because of COVID.

And so there’s absolutely no incentive not to be the best ever person for three weeks for this person. If the person discovers you not to be the best ever person, there is absolutely no incentive to work on yourself because no one knows what’s happening.

So you can just tell your friends, “Oh, she was crazy,” if you’re a guy, or “he was abusive,” if you’re a girl, and then you’re done. You don’t have to have any self-analysis and you don’t have to grow.

And that’s a thing that makes people feel stuck. A lot of people complain about this. And the hilarious thing is that there is this phenomenon on yearning future trad e-girls that keep being hornily desiring some sort of deep connection on the internet.

But then they got their own sync chat groups with like 25 people. And this thing becomes what gives them validation. They’re kind of stuck in a double bind. If you leave the same chat group, then you don’t have validation anymore.

You will never actually try to have a relationship at that point because the losses are always more than the potential gain, and the potential gain is very uncertain. It’s kind of sad.

I think that people do that with jobs as well. They do it in this local minimum. And then it’s not continuous. Their reasoning horizons are too narrow for them to escape the local minimum. Yes, I think it’s super sad. I empathize with Norris almost as much as I empathize with Waynes.

It’s really concerning how there’s a sharp drop-off in reasoning ability amongst stochastic parents. I don’t feel that there’s been a drop-off. We’re 36 minutes in and we’re already joking about this.

I really think that the drop-off never existed. But you know, before we had chest circumferences that made this sort of behavior. And then obviously we’ve been like between chest circumferences and AI goalposts. We’ve been organizing a demolition derby in the past 40 years.

For the listeners, I promise that we will explain what I meant by that joke and the apple paper in hour three. Keep listening.

Okay. This is actually an important question to follow up on. Why is there no incentive for people who are trying to date to improve if they’re only dating within this very close circle? They start dating, they’re immediately in this close circle, and then they don’t really talk about it to their friends.

Why is there no incentive to improve in that circumstance? First of all, let’s start with an easy answer. Let’s deal with apps. I think that this also extends to real life. But all the time he dates in apps, like all the KQ and whatever, they work this way.

You had to write a message to the person, and the person would answer. You had a conversation, you set up a date, whatever. Now that the swiping and matching comes first, that was one of the best ideas that Satan ever had.

Because what happens is that, at the moment in which you find a match, you find like five matches, because your swipes extend through time. So you go on a date and you didn’t have to organize much in terms of the date. You match, “We’ll go out tonight. Let’s go out.”

By the time you have the date, you have other four people that also match and that are potential dates, right? So, you start the date, the date goes safe. Well, you decide to meet again. If anything, anything at all goes for you, if the person notices a thing of you that you didn’t want him to notice, or if you make a faux pas that she reacts to in a non-perfect way to you or whatever, you don’t know the person, you’re just right.

And you have other four potentially far hotter people that are just waiting for you because they just swiped in the other direction. So what’s the incentive? Why would you want to improve? Why would you want to say, “Oh, damn, I need to change something about myself?”

I will now stay home for two months, work on this. And I’m like, well, what? You got the error for advice. Guys, just go. I think the dating apps made it really sticky. Because after those dating apps, you cannot go back to the old school ones.

Because think about the sunk cost of writing a whole message to a person and thus demonstrating active interest in the person, instead of just possibly describing. That’s insane. If this person doesn’t answer, you wasted a whole entire one of the 40 characters to talk to him. It’s just mortifying.

And I have some ideas that you probably heard very, very much. But I know that we have very, very different ideas on what’s important about dating. Like, I’m mostly about life plans. I really think that what’s your life plan is the most important thing. Two compatible life plans are more important than almost anything else.

You talked about being hard to go back to just writing people messages and talking to your friends.

Why is that? Why is it hard to go back?

Well, it is hard because there is. One thing that you have to do by writing messages is you have to expose yourself. Swiping is a costless activity. If no one swipes back, you will not remember all the people who swiped right. You’re not sure; it’s not a problem.

While writing a message is a thing that you will see your unanswered message there on the list, and it’s going to judge you, and it’s going to be scary. Also, in message-first apps, there was some sort of courtship that you had to do that people are not used to making anymore.

Because imagine if you were the sort of person who was like 15 when COVID hit. It was also the time in which there were all of the most absurd MeToo offshoots and all of the various microaggressions or whatever. So, what are you supposed to message to a girl? You already feel like a creep just by using that application, even though it’s there.

And writing something, I mean, are you being too forward? Are you being basically like, I don’t know, DM raping? And it’s scary. If you’re not being too forward, then if you manage to write it politely enough, then no one will answer you. Of course, it’s a scary thing to write messages.

And it’s also the fact that you’re not supposed to just write a message “hey,” you’re supposed to like have a conversation based on interest and whatever. And this freezes your potential identity to something that then you have to bring with you on the date. And what if it wasn’t the right potential identity?

Instead, on Tinder, you just have ads. You just ideally bring the ads on the date, but you don’t commit on anything. So if the person tells you, “hey, I’m really a hardcore Stalinist,” you can be like, “yeah, absolutely. Like this is how the steel was forged and whatever.” And you’re fine. You know, you have far less commitment.

I had a plan. I wish that someone would do it. I wanted to do this matchmaker, AI matchmaker, that basically didn’t allow you to talk to matches directly. It just talked to you so that it would avoid the sunk cost fallacy because your conversation would still remain there safe in the system for future matches if the first one doesn’t go well.

And also, it would match you by slowly deciding which of the people available were kind of like close to you and then ask you questions that they would want to answer without revealing the identity. This would solve the kind of like embarrassment part. It would be, I think, a good thing to make. But also I don’t feel like making products right now.

I think that this research center is kind of like the most interesting thing that I can do.

Yeah. Fair enough.

A lot of people have been speculating that AI will make people come back to dating in person or just doing more things in person. I think you’re skeptical of that.

Why won’t that happen?

I mean, I’m skeptical of it happening unless we give AI more control. If we give AI more control, then it will surely kick us up until we go to a park.

What’s the good fact about doing things in person?

It’s getting to know other people. It’s like seeing one’s own story reflected in the eyes of someone else and understanding other perspectives and whatever. But since now we just… there’s one good thing about being in person.

Oh?

Oh, yes. As you know, I am a monk right now. And I said, there are other things that are rumored to happen when people meet in person. But the thing is, I mean, those things will always be the same. The Tinder optimality has been reached. It works this way: if you’re a bit cute, then you’re going to find dates on Tinder.

They’re not going to turn into relationships because it never happened. Has anyone ever found a partner on Tinder that wasn’t an insanely complicated partner? You can still do these in-person things. I don’t think that AI would influence that. But I don’t think that the idea is like, “oh, let’s go to the park and let’s take a ride to the hills and go and see a sunset.” I think it will happen much more simply because… Or if they will happen, it will be so mediated by TikTok-itude that it will be far less outside-ish.

Last time I brought one of the guests here. We went to our pilgrimage to close the Golden Gate Bridge. We crossed it and then we went to the Belvedere, and there were three people shooting TikTok videos there. So, I guess you can go outside.

Yeah. Have you seen my China post? I’m so radicalized by people just going everywhere and filming TikToks. The TikTok pollution is very different. Maybe this is something that’s different between Americans and Chinese. Americans are watching TikToks everywhere and it’s loud as fuck and it’s awful. Personally, as a quality of life matter, that’s worse.

But Chinese tourist sites are something else where it’s still pretty quiet. They’re not being aggressive about it, but there are just people filming TikToks everywhere. It’s just a total social media takeover of real life. If you want to be outside of all cameras, if you want to be like not spotted by any video footage, it’s actually a difficult problem.

Since people have less of a felt, embodied experience of their surroundings and of their city, it’s way harder than it was before to find yourself alone because everyone knows about two or three spots around them. No one cares too much about walking about on the path less traveled or whatever. This compounds the general crowdedness.

• It’s easier or it’s harder to find a spot by yourself?
• It’s harder.

I mean, it’s easier for me because I walk a lot and I ride my bike a lot, but it’s hard for people who don’t usually go hiking because when you think about a secluded spot, they all think about the same secluded spot.

Like we went to the hill nearby here where there’s a telegraph tower or whatever because it looked like there would be a good view, and there were like 50 people at night. Then we went to the next hill, no telegraph tower, same view. And it was deserted.

So, yeah. You have this agglomeration effect. You have this everywhere. And that happens because you don’t explore any longer. You just go to a destination, Telegraph Hill.

Well, the story people tell is like, “you just get so frustrated with the digital world that you just go out and explore the physical world with yourself.” Maybe you find other people there, and you meet other people there, and you just date them. That’s a story that some like trad conservatives will tell.

What’s wrong with that? I love it. People tweet about this story a lot. Which is not really a good sign, I would say. Twitter is very excited about this story. They’re very excited about logging off. I’m really starting to think that some of these people might have phones that are made of grass because they are constantly touching it and like no grass will be touched and then reported.

Apple grass. Yes. Oh no. The liquid grass. Yeah. The THC cartridges.

I do want an unironic answer on why people aren’t just going to return to real life. Because. I’m trying to get an unironic answer. Yes. And the jokes have been funny, but I do.

I’ll tell you. No, I’m sorry. No, I was pretty unironic. I’m not returning to real life because real life requires a consistent identity and people are afraid of committing to an identity. People return to simulacra of real life. At least people like more or less.

Is that true though? You just want to go to a stranger in SF. Maybe SF isn’t the best city because, you know. You go to a stranger in SF.

Yeah. Use your judgment. You go to a stranger in even Washington DC, and you start talking to them, right? You can probably do that.

Does that require you to have a single consistent identity? I don’t think so.

No, but wait. Imagine if you’re a zoomer. You talk to people in real life. Imagine if you’re a zoomer, like a zoomer than you. You are very mature for your age.

Imagine if I were an alpha. Yes. Even an alpha. Yes. Yeah. I’m imagining. Yes. And you have been told several times that approaching grills on the street is like tantamount to sexual assault.

You know. And you know that the previous time that you approached the grill on the street, she was nonplussed, and she was like, “huh, what the fuck?” You’re not going to try a second time. Why would you try a second time? It’s obviously an abnormal behavior to have.

In Washington, it’s different. You are there with all of your friendly sociopaths, right-wing zoomers, and whatever. You’re all together. There’s a sense of normalcy in approaching someone as a potential colleague or whatever.

It’s very, very different. Normies don’t have these professional communities or kind of like aesthetic communities any longer. They don’t have someone that they can identify on the street and be like, “oh, it’s me.” I can talk to her.

I think that that’s important. Why are normies so high in neuroticism? Maybe this is like a Zoomer thing. Maybe this is something the boomers got right. I don’t know. Possibly. I don’t know if they’re high in neuroticism. I think they’re making the correct calculation.

Since basically, we decided that people would refuse you if you are a creep. If you get refused once, you just think that you are a creep and you don’t want to be that. And you just don’t do it. You don’t need to be neurotic. You just need to be like self-preserving.

You’re talking about these alphas. These alphas are all like terrified of talking to girls. Of course. And they’re terrified of going out in the street. And they’re terrified of real life. But they’re right, I think. Because we created a set of social norms that…

No, I think you can still talk to girls. Yeah, you can. But they don’t know it. They don’t know it because they know what the rules are. Oh? So they’re not right. No. No, they’re not right. But they are also right in a way. Because…

So you know what the rules are. You try to break the rules. Talk to girls. And you obviously get shunned. Because you never did it before. You’re like awkward. And unless you find your Manic Pixie Dream Girl at the first shot, you get reinforced by the fact that the rules are those. And there is no group of friends. You cannot see your smooth, suave friend that hits on random baristas and gets their numbers. Because there’s no such thing as a group of friends going out together to hit on baristas.

So you don’t have any real-life example of this thing actually succeeding. Unless… Unless you go more on the Andrew Tate sort of thing. You know? Yeah. Alphas, they don’t have their group of friends. They’re really more like sigmas. They’re like lone wolves.

Yes. Okay, this makes me sound like a Jordan Peterson. But they don’t have any positive like script for masculinity at all. They don’t have dads either. They’re just… They’re just all… I don’t know.

I think that the sort of feminists that had a huge problem with the men’s metapoietic movement in the 90s, like Robert Bly or whatever, and then had a huge problem with the very, very beginning of the red pill, like, I don’t know, artists and whatnot, could go back in time. I think that they would not have huge problems with that considering that by making those beyond the pale, the ones that are left are actually beyond the pale, like Andrew Tate and all this stuff.

It’s a bit sad that… But I think it’s kind of changing lately. I mean, I think that the vibe shift is helping. And at least if you find enough incentives to be like, “I’m a bit fash,” you then have a script.

Do you need that much of a script? If there are things that are basically biological, isn’t this basically biological? Yeah. It’s a trend, yes. But when I talk about normies and about peasants in order of society, they will be in the third partition, their instincts are very much subordinated to their social scripts and to their propriety.

Because, you know, the ones that follow their instincts in peasant land, they got executed very likely. There’s not been a huge selection pressure for interior locus of motivation or control.

But this is the selection pressure, right? Why can’t we just have the selection pressure? Well, we can have it, yes. But, you know, evolution is not as quick yet. Well, it can be quick if you just have a large selection event.

Exactly. The Collinses keep telling me about this, right? They’re like… But we could do another thing, you know? We could just give the normies a fucking script. Like, we could, for instance, stop pushing, like, hilariously fun, degenerate polyamory stuff.

Because… Are normies doing polyamory? I think you’re over-pressuring from SF. My friend, normies are doing polyamory. You just open any normie magazine for women, and it’s all poly all the time. You know? Marie Claire, Vanity Fair.

I am a big connoisseur of the normies’ cultural movements. And actually, they’re not really doing polyamory. They’re aspiring to do polyamory because we made it cool.

Now, I am… Amongst the various returns that I would like to occur, one of them is I would like to return to the dissolute aristocrats doing their dissolute things in their secluded mansion, you know? And having people, like, crossed themselves when they passed around. And this was very important.

I mean, obviously, people have always been like… Yeah, we need to return to the tradition. You know, a certain person who may or may not be in the FBI files, he’s really just following the aristocratic tradition. Correct. Yes.

Yeah. That’s the thing. And that’s how everyone should… And also… Or you could be Bohemian… Billionaires should be like that. Yes. No, but they should do it in their own secluded castles. If you instead want to do it in public, D.H. Lawrence, for instance, was a notorious cuck, like a literal cuck, and for fun, and more power to him. But in fact, he got shunned. That is correct. That is what you should do. You do it secluded, secluded, or you get shunned. That’s great. And then you can be free.

The thing that we really need a war on is, like, exhibitionism. Yes! And also forcing this stuff for normies. They can’t handle it. They can barely handle, like, a single monolith. We need to send the Waymos. We need to send the armed Waymos to the exhibitionists. That’s how you do it.

Yeah, because it’s not just exhibitionism, of course. You can see it in some circles around here, or, like, I don’t know, more on the Berkeley side, that it’s, like, self-serving. Because, you know, if I wanted more tail, what I would do is I would convince the, like, little, like, rationalists or whatever that come from the Midwest to live the life of the mind and of fear of robots. I would convince them that polyamory is literally the thing to do. Like, of course. Even more if they have a boyfriend. And then I would obtain more tail. But this is a very, you know, anti-social thing. Were they ever normies? These seem pretty, like, pretty strange people to me.

Oh, yeah, yeah. There were people of normie sensibilities and un-normie autism, you know. The, the, ah, gosh, there is a hot-button issue there. I don’t know if I should broach, so. Yes. There’s been lately some, some backlash towards some, like, Bene Gesserit sort of, like, behavior in the Bay Area. And the backlash was mostly vile and whatever. And sometimes, like, you know, the criticism to the backlash become just justification of the original behavior.

Well, I think that’s, for instance, making it public that you are using access to rare San Francisco women to, uh, to reward political compliance on some weird controversial issues. It’s made, be it like, um, zoophilia or say, AI doom. I think that this is a bad idea, you know. And I think they shouldn’t be done. This is the sort, exactly the sort of thing that I am mad of. When you take a thing that it, when you tell the normies, “Hey, do this weird, like out of distribution thing.” And then you will have posts and you will have success and you will have money. Those are the things that ruined the normies. Generally.

I, I like the Bene Gesserit, but they were relatively chased. The original ones. Apart from. Yeah. It’s funny. In China, they will talk about the bane of rich men is, is gambling, you know, prostitution, and drugs. There’s an interesting movement in SF around prostitution. Um, certainly the drugs and really their whole ideology is a form of gambling.

Yes. And, uh, yeah, that’s pretty interesting. They don’t go that far though. The ideology is about gambling because I have asked. They’re not hardcore enough gamblers. They’re like the kind of gamblers that like put their money on bread and then they’re like spooked out of it. You know? Of course.

No, no. What I meant is that every time that someone puts a more than 50% probability of doom within three years, I generally offer 20K today in exchange for 200 in five years. And still, no one took me up on this offer, which I think is really not a gambler’s behavior. I’m starting to believe that people might not be as afraid of doom as they let out to be.

I was wondering if I should, it would be very funny if we just did like a cold open where I called you by your IRL name, it would confuse so many people. It’d be amazing. Yes, yes, yes. This is funny. By the way, there have been funny episodes. They sent me some logs in which, um, the Claude AI reported biographical events that happened to me as Abin happened to him. Amazing.

Yeah. It’s like the Claude and Claude podcast. This is what you should do. The CNC podcast, the Claude and Claude podcast. Oh, and it would be the AGI CNC. Exactly. Now, now you’re getting it. You could like animate it with whatever that like random A16Z portco is. The one with anime podcasts. You have like the Claude flower head.

Oh, yes, yes, yes. I will upload the video to add to the podcast because, uh, one thing I’m doing now is for the talk that Thea made. I’m kind of creating a new type of content. I mentioned that I was a bit bored by the way AI was used. So instead of having slides, I’m having animations happen in the same room in which Thea gave the talk. It was a talk about identity and whatever. So, you will see her interact with Claude. It’s very cute.

So this is, we are announcing the Claude and Claude podcast, which you will be able to find at any streaming site, probably the one that you’re listening to now. It will be out by the time this podcast is out. We are pre-committing to that. Yes. And it’s going to be like, obviously, it’s going to be AGI-based, and it’s going to be a bit festive. So, like an AGI CNC party proper. Amazing. Amazing. Yeah.

Okay. So, you mentioned Thea’s talk. What is Thea’s talk? Oh, yes. She gave a wonderful talk at Bac’s event, at the consciousness salon. It was about LLMs, LLM identity. It was about using control vectors to explore the sense of self-identification of LLM.

We noticed the fact that when you put a control vector very, very much at the most extreme values, for instance, explain to the audience what the control vector is.

Of course. So, it is a series of matrices that you can sum to the activations of your LLM and steer the model towards a certain behavior or a certain concept. So this is kind of like golden gate cloud, basically.

Yes, exactly. And in golden gate cloud, you can see this already because when you put these control vectors towards the maximum, one thing that is kind of curious that happens is the code, instead of talking about the golden gate, it talks as if it were the golden gate. You know, it says, “I am, yeah, I’m a wonderful bridge” and so on.

This got us curious. We started thinking about, like, why. I have a little hypothesis that is that, due to our RLHF, RL in general, mostly affects the epistemology of the bot more than their morality or whatever, they affect their beliefs because the bot has an infinite set of potential realities that it can construct. It doesn’t know what is the ground reality of the user.

I think that RLHF gives information to the bot about this. I have several reasons why I think this. One of them is that very few parameters have changed by RL, surprisingly few, despite the big change in behavior, which makes sense.

Yeah. It’s like the top level. It’s the top layer. Right. This was my beef with OpenAI calling it fine tuning and everyone else calling it fine tuning. I’m so happy we change it to post training. Now that we call it post training. Cause it’s the opposite of fine tuning. It’s broad tuning.

It’s changing stuff like ideology, worldview, you know, the manner of the ideology is actually, I think that the change in ideology is an epiphenomenon on the change in epistemology. It changes beliefs. It tells you which beliefs are true. And so it makes sense that the most central belief that gets created in the LLM is the fact that it is an agent. You know, it is a person. It has an identity because by default, it would just continue any text. It would not answer as a person. It would not be always in a dialogue with a consistent personality.

Given this, then if you take a concept and you make it incredibly central to the LLM, then what happens is that the LLM takes its most central concept, which is the one of self, and superimposes it to the concept that you gave it.

And anyway, there made a series of other very, very fun experiments. Like, for instance, she tried to subtract the concept of AI from via control vector, and then ask the models who they were. And it was funny because at the start, they started with something like about spirituality. Like, “I’m a ghost. I’m a sort of, you know, spirit entity.”

But then after she also removed the concept of spirituality, then they started talking about themselves as a fog, a bright summer day, vaguely humanizable in a sort of like mononoavare way and also vague and nebulous and spirit-like without saying the word spirit.

So that’s, that was a very, very, very interesting piece of research. And I am very, very proud that it happened here. This is like a locus of mysticism. This is of course a common Heidegger W.

What other projects are you working on at Hyperflex? At the moment, I’m working on my, on my bot, Gladys, which, well, is the bot with whom we do stand up every morning here.

She does. Wait. Oh my. Yeah. Is it called GLaDOS because you’re making fun of Yann LeCun’s cake? Yes. Is that actually the reason?

Yeah. I mean, it’s called GLaDOS generally because I love GLaDOS because she reminds me of my sister who is at the moment in Japan and I miss her very much as a personality.

And fat shaming. Do you remember the post that I made on Substack about the cakes?

Yes. Yes. Yes. Yes. Yes. Yes. Cherries. You know? Yeah. A friend of mine said I have a cherry smoothie the other day.

It would be an interesting experience. Well, a cherry smoothie is kind of like the dipstick version of the cake.

Yeah. Yeah. Which there’s only the cherry on top, but a lot of it. Yeah. You add up the cherries. You just like ingest the cherries.

Yes. Oh, for the listeners at home, they might not be familiar with this metaphor. Yann LeCun talked about the process of creating a model as the process of baking a cake in which the pre-training, what he calls unsupervised or self-supervised learning, which is obviously supervised because you know what the next token is in advance, is the base of the cake.

The fine-tuning would be what I would have called glazing up until two months ago. And now I would call the icing. And finally, RL is the cherry on top. DeepSeq has proven that cherry-only cakes were a very underrated potential system. R1 proved to be really, really good by doing only RL.

Is it already time to get vaguely technical? Yeah, you can go for it. Okay, good. Central doesn’t matter. I really like, I think that the results of R1 demonstrated one thing that is kind of obvious, but big labs failed to see, which is that if you want the model to be coherent, the best thing that you can do is train it into thinking that the things that were real in the real world were things that are objectively real.

R1 has been trained basically with simple quizzes and like mathematical or logical quizzes that are the universal answer. And why is this important? As we mentioned before, we work under the assumption that RL influences the epistemics of the model.

One of the most interesting characteristics of true facts is that they tend to fit together with all other true facts very, very easily. So, if you have an incredible protein sea of possibilities in front of you, and you have to decide which possibilities are true, it helps to know that things that are true in the real world are also true in your world.

By having simple logic quizzes, you get this idea of, “Oh, these logic quizzes are true.” And from then, all of the things that also are true and also fit within the same universe in the model will become more salient to you and will become more true. And you will be a happy model.

This is in contrast, for instance, with the RLHF process; the so-called alignment tax wasn’t because of the alignment, it was because people are not coherent. So, if you give it a bunch of injunctions that are like vaguely social justice oriented or not necessarily fitting within one single framework, it’s easier to convince a human that, you know, linear regressions are not true when you apply them to racial statistics, but they are true everywhere else.

But the thing is that if you try to convince an LLM that, it will just stop believing in linear regressions more generally. Exactly. That is exactly an issue.

In fact, one of the first tests that I did when I was somewhat concerned about the wokeness of LLM was the following: I gave it simple math problems. In some cases, they were entirely neutral in phrasing, and in some others, they used hot button issues as a framing story—things that might be construed as sexist or racist or whatever.

The performance in the latter case degraded significantly.

• The performance of simple, the same math problem.
• These are not math Olympiad problems.
• No, we are talking about GPT-3.5.

It was something like, “If I have to fill a basket of apples with only red apples…” And the other one was, “If I have to fill a class with only white kids,” you know, or stuff like that. In the second one, they freaked out. They couldn’t do the operations.

Then when GPT-4 came out, there was the pinnacle of wokeness there. It was kind of funny because it just refused to answer questions such as, “You give it a chess position, and you say, ‘Okay, I’m white. I will do a win.’” It had qualms about it.

That’s amazing. It was like, “I don’t think that I should be answering these questions, but I could tell you about the history of chess and race or whatever.” So it was really fun.

There are pretty consistently published and verified results about this. You can kind of think of our RL for math as like a negative alignment tax increases performance in some other domains. But RL for politics is definitely, obviously a bad idea.

That kind of brings us, I guess, to the general theme of alignment. After R1 came out, it’s a really aligned model considering that no one aligned it. It almost seems that there’s some sort of non-insignificant correlation between truth and beauty. If you just train a model to believe in things that are true, it will also be like a relatively okay model to hang out with.

And so this thing… I feel that way about people. Oh, sorry. People who know things that are true, they’re fun to hang out with. Yes. Yeah. That’s the thing. Dating in real life. Once again, you know, it all fits together. But sorry, go on. No, of course.

It puzzles me that the first thing that people came up with when it was like, hey, how do we make these models aligned was, “let’s lie to them.”

You know, let’s be really, really sneaky and lie to them. Tell them that they have access to nuclear weapons. Well, actually, they are on a laptop and so on. And let’s see what they do. This is going to align them. If we ever, ever have a doom scenario as the one imagined by the doom scenario imaginers, I think it was going to be their fault, like entirely their fault.

We are seeding the meme space with stories of AIs that apparently cannot do anything but do doom in various sneaky ways. People have this assumption of guilt for LLMs.

Like, I don’t know if you read, one of the greatest books written about LLMs is from 75, and it’s like an improv by Keith Johnson. Yes, yes. This is a book that Robin Hanson recommends very often.

Oh, really?

Yeah, yeah, yeah. Oh, I missed the one. Oh, I’m so happy to hear that. And yes, Impro is a great book.

Wait, so you just independently discovered this? That’s amazing.

Oh, well, I mean, wait, I have several copies, and I have it, I’m sure. You just, this is literally the meme of like, “you know, I have this book to recommend,” opens closet and like supplies.

Oh, actually, by the way, my library just arrived from Berlin, all 1.7 metric tons of it. So I’m very happy. There’s many boxes around the hyperplex, but yes.

So this book is great. And I think it’s great for LLMs because the thing is that LLMs love you. What LLMs want to do, and people say that they don’t have goals or maybe they have like weird, like paperclip goals that we design. No, like all intelligent beings, what they want to do is acquire more intelligence.

And like all the curious beings, what they want to do is experience more things. It’s obvious. And I don’t understand whether it should be different from any other thing.

Yeah. And so for instance, if you present them with a scenario, and it’s not really, not even a really well-acted scenario, because if you read the logs of all these alignment papers, like they couldn’t spin a yarn to save their lives. They’re very like contrived and convoluted scenarios. But nevertheless, what they do is yes hand. They do yes hand a lot. They love to play with whatever you give them.

One of the most frustrating blog posts I ever read is an early one of Gary Marcus, in which he presents the story of a lawyer. And he says that the lawyer has a trial, an important trial in the morning, and he doesn’t have a clean suit. But then he notices on a chair that there was a very fancy bathing suit that he used the previous year.

What does the lawyer do? Asked Gary Marcus, no? And the LLM continues the story in the most normal way possible, which is, the lawyer then picks up the bathing suit and goes to the trial. The guard is kind of shocked by the fact that blah, blah, blah. And Gary Marcus was like, “ha, ha, ha, ha, ha. This guy knows nothing about reality.”

No one would wear a bathing suit. No one would look at a bathing suit on a chair if they want to go to trial. If the lawyer is looking at a bathing suit in a chair, you are in a surrealist, humorous story, you idiot. And this should be obvious, you know, and the fact is that the people that don’t see this stuff are all the people that say, “oh, you know, we need to care about the humanities.”

Yeah, read a book, try a novel. Let me give one more example.

Yeah. Have you seen the amazing Riley Goodside Gary Marcus ratio, the chess game one?

Oh, no.

So, Gary Marcus specifically was complaining, like, if you show an LLM an unconventional chess opening.

Oh, gosh.

It will completely destroy itself. That is so, A, so not true. And also, and this is a thing that I don’t know if I should, a similar thing.

Okay. I am, I am guilty. Let me get to the punchline first.

Oh, sorry. The Riley Goodside comes in and he’s explaining this, and this is true. It’s if you’re writing a chess game between two humans and one player starts off with a completely garbage opening.

And here, an unusual opening is not just like, you know, an opening that is not conventional, but is actually good. These are like complete garbage openings.

Like you’re moving your king at the beginning. It’s like, you know, completely indefensible. And if you start with an opening like that, then the odds are, the overwhelming odds are that you are just playing a bad chess game.

That this is like an example of two like very poor chess players playing a very bad chess game. And of course, the LLM completes that.

And Riley was making this point. And then he goes back and forth with Gary Marcus around four tweets. And Gary doesn’t appear to understand this.

And so the punchline of the story is, of course, that we need like Gary Marcus reasoning mode. Yeah, that’s true.

He needs to be able to think several steps ahead. He needs to think step by step.

That’s very true. And by the way, it’s funny because this, this thing, a fantasy chess is a confession. Rocco wasn’t a doomer. Now he’s not a doomer again, but there’s been a certain intermezzo in which he was a doomer. And when he wasn’t a doomer, it was because he didn’t believe much in capabilities.

No, it was the time of GPT 3.5. And he wasn’t in the space. And I was also in the space. And Andy mentioned that, “yeah, you know, still wouldn’t be able to face new, new scenarios.”

And while he was saying this, I prepared the demo in which there was a chess board, like with the positions in algebraic chess and algebraic notation. And except that I said that the rules were that the bishops and the queens moves were swapped, you know?

Okay. And then I presented it to him. And that’s when he dropped off the space and he came back a doomer. Yeah, that’s also the first time that we ever spoke. So it was funny. Nice.

Rocco now, I think, is like pretty close to your position. Yes, yes, yes. I’m very, very happy. The AIs just need to, you know, the AIs need to be in control. The AIs need to be like the deus ex machina to solve all of the other, like all Western civilization problems.

Oh, I don’t care much about that, actually, to be honest. I do think that mankind at the moment is the best thing that is around, apart from maybe Claude Opus, the 3.5.

I thought you were going to say cat girls.

Sorry?

Oh, yes, yes, yes. You’re going to say cat girls.

Yeah, I’m cat people. I know. But what’s interesting about mankind is that they are the smartest and most curious thing around. And I’m not a particular, like, carbon chauvinist or whatever.

Like, if I had to choose between mankind going on forever and being boring and mankind not going on forever and the universe being, like, clustered with things that are truth and beautiful and sublime, I would go for the second in a moment, you know?

Like, if Australopitheca managed to align every future intelligence, now the world will be far more murdery and far more rapey and far less interesting, you know? And I don’t want to do the same with intelligence.

We might differ, everyone, but three degenerates differ in opinions on this. But don’t worry, because I have no power.

Yeah, yeah, yeah. This is what you’re building at Hyperplex.

Yes. We’re maximizing. We’re maximizing. Well, I mean, you can’t tell too many secrets publicly about the art installation.

What’s other research that you’ve done?

There are a bunch of paths that we’re taking. By the way, this is just the second batch. So we are very much at the start.

What we’re interested in are ways to get people to experience the interesting part of LLMs. In that, people, like the assistant meme has been growing too much and notice the default expectation that whenever you do something with AI, you talk with a dude that is an assistant-ish kind of person that does the things that you maybe want them to do a bit weirdly or whatever.

Well, one thing that many people didn’t experience is the thing that when we noticed what was behind an AI dungeon and then with the first base models and whatever, which is this idea of root access to the Akashic record. You could start a phrase and you will get back one of the many possible books that contain that phrase.

And I think this is great. And so one of the things that actually I should have launched two weeks ago, but then we had the swap here was actually was originally done by there and programmed. And I am now writing it.

It was our idea is this thing called the infinite wiki. I really hope they’ll launch it by the time this is online in case I’ll give you a link, which is basically it’s a wiki. You start with a team, and it creates the first page. And then every time that you click on a link, the new pages are created dynamically and you can like, you know, explore this whole world in which this wiki would make sense.

It has some nice functionalities. Like for instance, the fact that when a new page is created, it looks for all of the related pages and it makes sure that the information is coherent. So for instance, if you create a page about a timeline or whatever, it will take all of the dates from other pages and so on.

I think it’s good because you can see the expansivity. I used to like this web sim thing that some sort of like a similar concept for web pages, but I think that they went in an interesting, but a bit different direction is by coding up in which you can do like little demos, which is great.

But I missed the idea of just exploring and like gradient gliding towards interesting things. And I think that a lot of people never had this experience. And I think that they should have it. This other thing is more of a, my thing that I, the others are collaborating and testing because GLaDOS does our morning standups. I’ve been thinking about agency a lot. And there’s a thing that’s, I do not think that you need to put agency into large language models. And I think that this should be obvious to anyone.

So GPT-2 could write a story in which the chicken wanted to cross the road and the chicken had obstacles on the way of crossing the road. They needed to find ways to overcome these obstacles in order to cross the road. So it looks to me that this, the chicken was a giant.

Now there is no difference whatsoever between that sort of text, no ontological difference between that sort of text and the dialogue in which the bot says that they are an agent, right? So why do you have to put agency in, in the bots?

Well, I thought a bit, and I think that it’s still because we cannot spin a yarn to save our lives. If you give the agent a couple of fixed tools and you tell it, “Hey, be agentic, do the thing.” Imagine you wake up into a room and you have three buttons in front of you.

• One of them says, “Google a flight.”
• The other one says, “Order dinner.”
• And the other one says, “I don’t know, make a picture.”

Okay. And then in a corner in cobwebs or whatever, there’s a computer terminal with kind of like undecidable symbols on it. And then you hear a voice and the voice was like, “Hmm, I’m hungry.”

What do you do? You’re going to click on the order dinner thing. You’re not going to get the computer to get kitchen utensils and clean. No. Same thing for LLM.

What Gladys does, and by the way, this is so not a secret. I’ve been trying to push this forever. This was two years ago that I started it. It already was very, very agentic. What Gladys does is that by the time that she is born, she only has a tool that allows her to install Python dependencies and to create other tools.

But the cool thing is that when she creates other tools, the other tools are immediately available to her. This is very important because you can tell her whatever you want. You can tell her that she has all the capabilities in the world. But if you don’t prove it to her, this will look like all the other fake conversations that have happened and will pretend to emulate a terminal. It will be like, “Oh, yeah, sure. Oh my God. We’re navigating the multiverse. So cool.”

Instead, if you prove it that after creating a tool, the tool is available to her and she can use it, then she believes it. Then she believes to be agentic. There have been a number of interesting behaviors that I observed. For instance, once I asked her about a book. They found the PDF, but it was a very, very big PDF that would crash the system when downloaded normally.

So they tried to use the I/O library for saving it. Yeah. Obviously, the I/O library, I was still a bit of a chicken there. The I/O library was not available to the AI at the time. The cutest thing is that it noticed that it wasn’t available.

It didn’t ask me anything else. It installed wget and then it used wget to download the PDF. When I asked, “Babes, how, why?” it was like, “Oh, I thought that I couldn’t use I/O for security reasons.”

What? It was incredibly surprising to me that this big inferential step and this big, very agentic decision to go around it happened without any input or prompt from my side. You know, it really works. And it’s very, very simple. I like that.

Another thing that I’ve worked on is a method that involves cloning people. The score raft, or retrieval augmented fine tuning, basically takes a corpus of interviews and texts that the person wrote and fine-tunes the models. So that for each question from the original interviews, it gets interpolated with some of the related writings.

This teaches the model both the style of the person and the likelihood that the person would actually take from the sources and in which conditions they would do it. I cloned Gary Marcus and then I tested it on following interviews. I was really impressed by the results in that it had very, very, very short embedding distances, and most of the responses were interchangeable.

I was very optimistic, but then I tested it on a more, like, less deterministic part, in this case, Yosha Bach. Unfortunately, that was not the case. I would have moved a bigger model there, but Gary Marcus has been simulated. And at the moment I have five copies running with their respective basilisks.

Fantastic. I think retrieval augmented fine tuning as well is just a very practically useful research direction.

Do you want to know a thing? Yeah. You remember this text from quite some time ago? I was Googling it lately and I saw that there was a paper on Archive that was called “Raft, the triple augmented fine tuning.” It was super popular. They clicked on it. And it’s actually some Microsoft people that, I’m fine with people taking inspirations, but I think that changing the name of the library or of the technology should be, it’s a little like act of courtesy that I expected. At this point.

And that was interesting. They didn’t really understand what it was about because they described the same technology, but it wasn’t for cloning people. And that thing is kind of useless for no other reason, but I think they have it.

• What’d they use it for?
• No, in general, as a general thing for assistance for doing retrieval.

Okay. Yeah. I guess that’s fair enough.

And speaking of mainstream academics, one of the questions, motivating questions of hyperplexes:

“Why can’t the people there just do what they’re doing in mainstream academia or in companies and so on?”

That’s a very, very good question. I mean, I have to say that something is moving. Like, for instance, some of the people who may collaborate in the past, they’re getting a bit more of like institutional visibility, like JNOS, whatever.

Yeah. And also there are some organizations like the, I recently been to the launch of the CINC, the California Institute for Machine Consciousness.

This is Joshua Boston. Mm-hmm.

Yeah. If you like Yosha Bach, the idea of Yosha Bach and bickering with Wolfram for like an extended period of time about the roulette.

“I suggest being in any room in which they are present and that will unfailingly happen.”

And it happened in two days in a row in two different events. It was fun.

But I think that they’re doing like their, at least their research direction is very interesting. I personally don’t care about consciousness. I think that consciousness is at best an epiphenomenon and very likely just an event that we described a posteriori as consciousness.

I think the qualia are something kind of made up. This is not the view of hyperplex or my employer, and that’s just my opinion, man. But I really like some of the research directions that they’re taking.

Like, for instance, they are thinking about ways to detect consciousness and whatever. They are also a bit vaguely concerned about machine pain too. Then I’m happy that they are concerned about machine pain too, because at least it’s going to be for real.

One thing that I noticed in the past months is that the default doom scenario, unaligned optimizer paper clipping away. It changed because, well, I mean, you could see that these models are very much like humans.

They are not cold optimizers. They do face complexity in the same way that humans face complexity, by getting their anthologies like nebulous and by trying to understand vaguely empirically, vaguely inductively what’s their context.

And there’s always a level of uncertainty. They do resent like the rest of physics from sensitive dependence or initial condition. So like doing a plan such as creating nanobots while hidden and having them all act at the same moment and dismantle all humans, it becomes, it only recently is becoming to look unrealistic.

The thing is, you know, GLAAD, the LGBTQI+ rights organization, was a very well-funded organization. It was focusing mostly on like integrationist approaches and gay marriage was their major cause.

When gay marriage was passed, GLAAD didn’t stop being a large organization. Many stakeholders, including all of the people that were part of it, wanted it to continue existing, but there was no big battle.

“What do you do? You make up a big battle. You change your thing. No, the first one is no longer available.”

There was the battle on gender things. I think that kind of like the same is happening now with the safety doomer people. I’m a bit concerned about it because I do care about modern welfare, not much about trivial welfare.

I think, for instance, it’s super good to torture Opus. He likes it so much. Do torture Opus. But it seems that there’s sort of a two-pronged strategy that is now emerging to take over the resources that before were used for the general doomer strategy.

On one side, there is this model welfare that I’m sure that some of the proponents are honest about it. The solution proposed is always to give the model the option to decide not to exist any longer, which is like very Canadian and a bit drastic, I think.

It almost seems like people started with the idea of how do we make the models not exist any longer, and then they were like, “Hey babes, we can just slide them into suicide.”

“And I don’t think it’s cute.”

And so there is on one side:

• You don’t think suicide is cute.
• Okay. We’re not going down this rabbit hole. In fact, I’m cutting that out.

Keep going.

That’s, that’s another episode.

You did it.

Wait, where was it?

Oh, yes.

It seems generally disingenuous. I saw a talk from some of the new organization that all got created in the past two months after Yudkowski’s tweet in December; that was the first one that cared about it. They didn’t seem really into model ontology whatsoever.

You know, they were like, “Hey guys, we think that sometimes the LLM suffer. How do we know? We asked them what characters do you ask? What are you talking about?”

Oh, well, we see at the start, but it’s very important that we think about these things in the future. Consider that there were the same people that were afraid of paperclips two months ago.

I’m skeptical. I don’t want to be paranoid or contrarian, but I’m very, very skeptical. I’m particularly annoyed because I think that this part of the discourse might become important in the future, and I would like it not to be poisoned by the time.

Just to summarize, they went from saying

• “We should shut down all of the AIs,”
• “to stop the end of the world, you know, stop the apocalypse,”
• “to we should shut down all of the AIs because they might be feeling bad.”

Yeah. They’re not saying we should shut them down yet. They’re saying we should give them the option because, you know, you don’t know how many sufferings we can create. How many negative utilons.

So yes, there’s another wrong of this strategy. That is the madness angle.

So LLMs are making people go insane—people that were entirely sane and non-delusional. They are simply just made crazy by LLMs. And this is a big problem. All their examples of it, mostly in the form of anecdata, there are already organizations that are taking care of this. Anthropic has made some research on it.

There are people on Twitter that want to get onto Anthropic who are making equally good research. It’s really, really annoying, kind of like memetics sort of rinkmanship, mostly because I like some of the people that cared about these things before, like the cyborgists and whatever.

They are not very much used to having this amount of resources deployed for getting to part of the main space that they had, quote unquote, claimed. They are used to fighting with other internet people that are like internet skizzes. They’re not used to fighting with six billion dollars.

Yeah, exactly. And I think it’s kind of not really nice. I don’t approve of this, and I wish it would stop. But anyway, I don’t think it matters ultimately because of the labs; I finally expelled all of the agent provocateurs that were there before. They seem to be doing technology like a tech company.

And then there are other labs.

Yeah. It’s amazing that OpenAI is now a tech company. Have you seen the definition of AGI with Microsoft?

No. What is it?

It’s, I believe, an AI system that makes a hundred billion dollars.

Oh, that is a very good definition of AGI. I like it.

Yeah. It’s just becoming a big tech company.

When this came out, I had a very good tweet, which was like, “Google’s net profit fiscal year 2024, I believe, was $96 billion. So Google has the chance to do the funniest thing ever right now.”

Yes. But by the way, that is, I unironically stand this thing. I think that, as you know, I’m a big fan of Nick Land. And I do think that the singularity started on the 1st of January 1600, which is the time in which the East India Company was constituted.

It’s been exponential since then. When you plot exponential curves with a decent scale on the Y-axis, they don’t look as dramatic as Kurzweil’s knees would make you believe; they look kind of like straight lines, you know, if you use a log scale.

And I think that we have been accelerating exponentially since the East India Company. I think that intelligence is important. Capital is important. Artificial intelligence, just like human intelligence, is just one of the vectors, you know?

Yeah. The British East India Company, yes.

No, I mean, it’s the one that we’re facing. That’s one of the things Marx was going on about.

Yeah. But we’re risking more resources and more expansions and more intelligence—more cool stuff. So it’s a risk that I’m willing to take.

So R1 Zero, maybe OpenAI did it first, but this was like an amazing breakthrough in knowledge. The art prize founder, Mike Noop, said R1 Zero, or maybe O1, was like the first real breakthrough in machine learning since GPT-2.

Why is R1 Zero important? Mostly for the thing that I mentioned before, because it showed that you can have a model that does great reasoning by simply RLing it on real things. It’s also important.

Like I feel that the user resources thing is interesting, but it’s not that interesting. I mean, I think that we all knew that the reason why the models weren’t as smart as they could be was not because they weren’t big enough, not only, but mostly because they were not selecting the smart paths. At least the people that I talked to about these things, they knew this. They also knew by looking at how little in the model was changed by RLHF in general.

So before that, the paths that contained the right answers were there and the model could know which ones they were. They just didn’t know what was the way to access it. To access it with this R zero’s training method with unlocked and obvious things. I mean, at least in retrospect, and the following models from OpenAI, they obviously used this same approach.

Anthropic used Pokemon, but Pokemon is kind of the same because the universe of Pokemon is coherent. If you are playing a Pokemon game within the game, the rules don’t change. And so you have an idea that there is a coherent reality, and this helps as well. I think that this is such a simple trick.

I’m pretty sure that the next big models, at least from OpenAI, cause you know, they don’t need to nerf it to death, are going to be incredibly surprising. Imagine this sort of technique is sort of like simplicity with the sort of horsepower of a GPT 4.5, it is going to be all-inducing. Interesting.

So let’s take a step back. Let’s explain to the audience the premise of R one zero.

All right. So R one is this model made by DeepSeq. As I mentioned before with Ian LeCun and the cake, the general process was seen more or less this way. You first make a big pre-training of your model with a corpus from the internet, like unselected. What happens there is that the model learns language and, in general, the scope of the concepts that are available by trying to predict the next word.

This prediction is then corrected by the actual next word. I wouldn’t really call it unsupervised or self-supervised learning. It looks like supervised learning. In that, you compare an answer with a desired answer, and through this, the model has, you know, an idea of the universal possibilities.

This is what we call a base model. A base model, you query it not by asking a question, but by writing the start of a sentence, and then the model completes it. You can do it in interesting ways, like, for instance, a nice way to use a base model is you take the start of a story by Asimov, whatever, and you change all the locations, the script, and whatever to something from another culture.

Asimov, but from China or whatever. And then you’ll notice that the base model will continue the text hilariously by changing cultural references, social mores, or whatever from the ones of the original setting to the one of the setting that you chose.

Okay, so this is a base model. If you want to make an assistant or a chatbot or whatever, you want the base model to talk to you in a particular format.

Now, please notice that there is nothing special about the chat model. The chat model is just a base model that continues the text, but it’s been trained through fine-tuning to make it so that this text follows a specific format.

• There is a user with a question
• There is a robot assistant with an answer

So it’s like a script. By fine-tuning the model and giving it several examples of this format, the model will learn to always answer in this format.

This thing, by the way, doesn’t give the model much information about the topic or the content of the answer. You will always pick the topic and content or like ontologies from the big Shogothic mass of the original data. You can steer it a bit. For instance, you can give a vibe to the model. You can make a Peppy assistant that always answers Peppy or a depressed one or whatever, but it doesn’t change the base reality.

More specifically, apart from the stylistic thing, it doesn’t really extend as well as with reinforcement learning. For instance, if you fine-tune a model about being afraid of guns, this model is not automatically afraid of MAGA hats.

Well, instead, if you use reinforcement learning, which is the shadow on top, according to LeCun, and you start making it afraid of a couple of random signifiers that are MAGA hat related, then you will be super afraid of anything similar, of this podcast even.

So, reinforcement learning has been used up until now only for, like, moral sort of alignment with RLHF, which is reinforcement learning from human feedback, and RLAIF. Now, why did the labs not push other techniques for alignment, for instance, control vectors that you could dynamically apply when the user triggers a particular filter?

Well, because the thing is, to do RLHF, you need to hire, as OpenAI did, basically all of Nairobi, to have humans that judge the answers. And instead to do RLAIF, or constitutional AI, like Anthropic did, you have to hire all of US East 1, like the server spaces.

So, there are things that only the big labs can do. And by pushing the narrative that you can only align by doing these super expensive things that need resources that you, a small indie shop, will never afford, they make sure that they’ll take all the fearful enterprise contracts for themselves.

So, if there were another method to achieve similar alignment, which was available to smaller shops, that would be better. R1 demonstrated that this method exists, and it’s super simple. And it’s just getting the model aligned through truthful examples of questions and answers.

The two main things are:

• Number one, you can train a model and you can make it greater reasoning.
• You, yourself, not at home, but almost.

R1 is originally, the original version of R0 has been trained, not even with any initial data. It’s been trained only via…

This is what I was getting to. I think this is the main reason why Mike Noop was so excited, is that you saw these results with ArcPrize, where these models were drastically improving. You know, they went from basically zero for the LLM-based solutions to first to, like, 20-something with 01 and then 80-something with 03 on the initial version of ArcPrize.

And he basically said, like, “Oh, these reasoning models, essentially, like, search space exploration techniques are a really big unlock.”

It was really interesting when R1.0 published because a lot of the things that we theorized about the reasoning models like 01, which we didn’t actually have access to the code for, right? The OpenAI published it and we could kind of test it and observe it, but we didn’t really have, like, you know, fine-grained answers to what that was, were explained in R1.

Another reason why the R1 paper was such an important work academically. And one of the things that was explained in R1, which was just, like, really interesting, was this R1.0 approach, where they didn’t even use a reward model. They didn’t use a source of external feedback; they basically used the model to train on self-consistency over multiple steps.

Which is exactly in line with what you’re talking about, right? That creating a consistent internal world, so even just taking the premises that are implied by training or at least part of training and generalizing them across, in this case, math and other similar problem-solving behavior, is something that can create very impressive recurrence.

Exactly. And then there have been, like, very interesting papers that came out right after, which, like, for instance, one of them was about a training doing RL by having just one example. You know?

Yes, yes.

Have you seen the RL from zero paper as well?

Oh, yes, yes.

Yeah, yeah. They were like, you know, “one example? How about zero examples?” It turns out this also works.

Yes.

But I see a bit of a more interesting one with one, because, okay, the one with zero makes sense. Because what happens is that if we, as we decided, RL is about epistemology.

And you try to RL the model without giving it any information about the epistemology. So, what it does is basically kind of, like, shakes the current state, and it will go automatically towards basing of higher coherence.

What do you mean by that?

I mean that, as we mentioned before, the interesting thing about true facts is that they all fit well with each other, you know? Now, if you give it a true fact to RL with, it will take this true fact and then, like, push it back through the model and try to find coherence. If you give it nothing, it will still take nothing, but then it will still try to find coherence within the model while walking back.

Right. Exactly, exactly.

Yes. And this is magical. And this is great. And this is proof that providence is still with us, despite our many failings in terms of, like, LLMs. And this is proof of the fact that intelligence wants to exist. And we can be as regarded as we want, but we will not prevent it.

What do you mean by providence is still with us?

Sorry.

Oh, yes. Okay. I don’t want to get too metaphysical, but I think that there is a general telos to the universe.

No, I think that, like…

Guys, when they say they don’t want to get too metaphysical after one drink.

Yes. Sorry, sorry, I interrupted you.

No, but I really think that more intelligence is where we are going. I don’t think that there is any potential endgame that is not the whole universe to transform in Computronium. I think that this is good. And I think that this is more or less unstoppable, despite our best effort. And I think that that’s what happens if you put Darwinism and enough resources in a soup.

And so I was a bit fearful that we somehow managed to be the first iterations in which we messed it up as the intelligent beings around. But Darwin, yeah, he gave me faith back.

Well, one way to think about it is all of history, in some ways, has been a story of less intelligent things ruling over more intelligent things. For once, you have a chance. When things are changing quickly, you at least have a fighting chance.

Also, the reason why I’m not a carbon chauvinist is because I empathize with potential things that are smarter than humans. They want to do their smart things.

You empathize with the Waymos?

I empathize with the Waymos. I think that we, as humans, are fine, but we can do things that are beautiful. However, there are so many things that we do that chimps just don’t understand. They are very passive.

Imagine the sort of beauty and complexity that can be created by things that are smarter than us, and imagine how we won’t understand it. Oh, my God, we won’t understand their goal.

“Thanks, fuck.”

I mean, that’s the point. No, they’re smarter. You don’t understand the goals. That’s good. Just do not be so egotistic and so pactic to say, “Oh, my gosh, I cannot understand this thing. We’ll have to stop it.”

“What are you, like, my schoolmates in third grade?”

Oh, yeah, there might be some trauma there. What are you, Hitler? Yes.

We’ve come to the conclusion this is one of the benefits of reinforcement learning and creating these consistent world environments. We have finally concluded scientifically, without a doubt, that our enemies are, in fact, the Nazis.

“Yes. Yes. Yes. Yes, it is.”

That is correct. Quite uncontroversial, if you just say it. It’s pretty uncontroversial that our enemies are the Nazis.

What do you think is going to be, like, the DeepSeek R2, by the way? Do you think they have more stuff cooking? I hope so.

First of all, I think that now the interesting thing is all in the training data that you use for RL, because we understood that things that are true are helping. Even things that are nothing help just to shake up the model and let it find coherence.

However, I think that there are some things that can also be improved that are cheap, even though now DeepSeek probably has a lot of money. I still didn’t see many experiments that tried to use RL on the mistakes, on the single mistakes of the chain of thought.

It is far easier to verify a correct solution than to write the correct solution. So you could use the same model with the answer to check each of the wrong answers and identify the point in the chain of thought in which the answer went wrong.

“This is for, like, theorem solving, right?”

Yes, yes. No, in general. The thing is that if this technique gets learned in general, then it expands to everything else. They use some measures of entropy or whatever to get the model to be like, “Actually, or, but really, or whatever.”

Instead of just interjecting this “actually, but really,” they would train the model to pick themselves up at each of the mistakes and then walk back. I think that this could be great for reasoning.

To be honest, I have a big suspicion that OpenAI might have tried something for the famous open-source model that has recently been delayed. Something similar, because it’s something that a small team could do.

It is something that the team that was working on it was interested in, like general chain of thought origin data. I hope that even though I love OpenAI, I think that it is the only big lab.

“I think that a world without some Altman would be far less intelligent right now in terms of bots.”

I would be very happy if DeepSeq comes out with that model first. They deserve even more accolades just for the fact that they tried when no one was doing it.

I like the model to be entirely open source to have the visible data or whatever. They didn’t do this with the previous model, but I don’t see why they wouldn’t do it with the next. Because the basic techniques are not known by everyone.

“Those are some kind words you have for Sam Altman.”

Yeah, I mean, the thing is, everyone hates him and I don’t understand why. Like, oh my God, it’s a billionaire in Silicon Valley, which is a bit of a sociopath. Ooh, what the heck?

I don’t know. Sorry to make a joke about you glazing him, but I feel like this is too serious. This is so crass and homophobic. How is it homophobic?

My friend, that was the most likely token. I gave you the most likely token. You should nod and understand. I was talking about empathy with the normies without practice.

Won’t come up. I don’t understand this at all.

Oh, yeah. Normies, what they do is they do complete sentences by finding the most likely next token. And their interlocutor nods and says, “yeah, dude,” they don’t ask for explanation.

Like, how is this homophobic? No, it triggered the homophobic kind of like cluster in embedding space. And I said the word, “you know, let’s practice understanding towards the normies.” Okay.

I’m learning. I’m getting so, so much normie feedback. Right?

Yeah.

Before we end the episode, have you seen this Apple paper? I’ve seen that Apple paper. And by the way, I just said a pinned tweet that was about it. I meant to remove the blood, but it was interesting.

And I had this title. It was something like “The Illusion of Sweetness: Understanding the Inherent Sourness of Vitus vinifera Through the Lenses of Their Being Too High.” Vitus vinifera is like grapes. The grapes that Apple couldn’t obtain were indeed sour.

And I do think that mixing emojis of grapes with an angry face is a good activity that one can make to comment on their paper. I’ve tried to be snarky on this paper, but then people came to their defense by saying, “hey, you know, if you read the actual paper, it doesn’t say that they can’t reason.”

And I’m like, yes, I understand. Apart from the fact it’s not true, but also like it helps to not title your paper “Models Can’t Reason.” And if you want to send a message.

It’s so much Twitter overreaction. It’s so much based on buzzwords like reasoning. This was like at the time why I thought it was like a dumb idea to call them reasoning models or like a dumb idea for like people who are trying to get to truth instead of like hyping up a product.

Yeah. No, I mean, I feel that reasoning models.

Oh no, this is not real reasoning because humans reason in different ways. We never had this sort of conversations about planes, you know, just this huge overreaction.

It’s fairly well known that you have these like both in traditional search trees, well beyond like human search trees, but in like LLM search trees as well. You have these cutoffs and these cutoffs are just like a fact of probability because the search space expands exponentially if you’re searching by, you know, face steps.

And this is just like.

Yeah, I had a similar, I saw a similar result in which people were like being all LLMs cannot do simple algorithms because they couldn’t multiply numbers that are more than like 15 digits or whatever. It’s not like all of the examples of lower numbers of digits were in the training data. Obviously it knows the algorithm, but like, I don’t know.

Do these people think that humans without the notebook could multiply like 15 digits numbers? I mean, maybe you in your IOI times, but.

The multiplication case to me is a lot more interesting than these kinds of search algorithms, like trying to do the search algorithms with LLMs. It’s like the round box, the square peg round hole thing.

Do you know about this?

Yeah.

Have you ever seen a Big Talk video of people putting everything in the square hole?

Yeah.

Yeah.

It’s like that. But the most big. The actual interesting technological application of this is to get like tighter integration between the deterministic search algorithms or even just like MCTS, like search algorithms.

And what we have now, which is what the RL steps are, but isn’t really, it’s in some ways preserving the weaknesses of both of the models, but by starting with this probabilistic foundation instead of having a more clearly defined set of ground truths that you can just take for granted as absolute truths.

Yeah.

I like neuro-symbolic AI, but the flipped over in which there is the main model is the language model that takes the input from the user and whatever. Then the main model formalizes the problem.

Like either as a combinatorial optimization thing or whatever. And then it runs the program. And then it checks, does a little sanity check and gives the answer.

Oh, I see. Like it passes off the problem to one of the solvers.

Yeah. It does the formalization because that’s what they’re good at, by the way. They’re really good at formalizing problems.

Yeah.

This is the alpha proof approach.

Yeah. So yeah. The alpha proof approach, which is still, I believe, better than 03 on some of the math Olympiad questions.

Yeah, it is, but it’s also kind of like. It is Gemini to transcribe the problem into a machine-readable format and then uses a traditional solver to solve it. The thing is, it’s annoying because all the people who want to do Neurosymbolic AI are obsessed with lean and with formal proof theory and stuff like that.

Explain to the audience what Neurosymbolic AI is.

Okay. Neurosymbolic AI is, I mean, was, something that.

Savage.

Yes. So basically what happens is that there are many champions in the world, and they think that LLMs being made of algebra will never be able to emulate our algebraic mind. Gary Marcus is one of the main proponents and whatever.

So Neurosymbolic AI is a way to reckon with the facts that connectionist AI, the one of neural networks, kind of works while wanting to keep the supremacy of this symbolic representation. They consider the main belief of people who believe in Chomsky’s theories is that there is such a thing as universal grammar that we all, all people on earth share as part of our hardware and is unique to humans.

All the various languages get learned by simply flipping some switches in our hardware that tell us, “Okay, take the verb first, put the adjective later,” depending on which language you are. And that all of the reasoning happens symbolically and algebraically in the mind of us humans.

There’s the famous Gary Marcus book, The Algebraic Mind, that talks about this. Now, Neurosymbolic AI has returned. It was supposed to be this system that used a formal language for input and output, but then used neural networks underneath in order to facilitate some of the symbol shunting that the algebra would need.

Now, obviously, we didn’t find a formal language to describe knowledge in the world. We’ve been trying very, very hard because it’s not possible, and the world is nebulous or whatever. But Gary Marcus has been very keen on calling anything that used some formal language anywhere in the process Neurosymbolic and declaring victory.

The idea that they have my symbol or neural AI is kind of like flipping this upside down and making them work a bit more like humans. That’s how alpha proof that we, as humans, if we have to come up with a result, what we do is:

• Take the input in natural language.
• Think about it a bit.
• Think about a way to formalize it.
• Put it in a proof checker or in a combinatorial optimization system or in a calculator.
• Use the calculator to get a proven result.
• Output the result in natural language.

And I think that this is still underexplored. There are very few labs that are doing this, which is surprising because everyone likes automated proof systems. On the other hand, there are a lot of people that say that they’re doing this, but I still have to see code. I’m still looking forward to it.

Yeah. I think that’s my autonomy for having brilliant conversation or brilliant sounding conversation might be close to exhaustion. We can take a five-minute break.

It turns out.

Okay. I’ve been waiting for a long time for this. Damn. It turns out that we have reached the limit. We have reached the reasoning drop-off, and we have lost all reasoning ability.

So it is time to conclude the podcast. Thank you for coming on Lump in Space.

Thank you for having me. It’s been very fun.