<p> AI might not yet be eating the physical world. AI is still eating the world of news, and I am finding less and less opportunity to post about other things. My backlog of items grows across most of my secondary topics and I haven’t had time for any side projects.</p> <p> The monthly, no matter what, is sacred. We continue.</p> <p><strong> Table of Contents</strong></p> <ol> <li> Home Alone.</li> <li> The Men Have Never Been Okay.</li> <li> Antisocial Media.</li> <li> While I Cannot Condone This.</li> <li> Zero Marginal Product.</li> <li> Good News, Everyone.</li> <li> 100 Good Television Shows.</li> <li> For Your Entertainment.</li> <li> Gamers Gonna Game Game Game Game Game.</li> <li> I Was Promised Flying Self-Driving Cars. </li> <li> Sports Go Sports.</li> <li> Government Working.</li> <li> Jones Act Watch.</li> <li> Variously Effective Altruism.</li> <li> Walk Away.</li> <li> The Lighter Side.</li> </ol> <p><strong> Home Alone</strong></p> <p> The story of our time, told once again: Americans are abandoning quite a lot of formerly popular activities, especially hosting at home, as Derek Thompson discussed. Phones and other electronics are in, most things that are not phones are out, with notably rare exceptions.</p> <p> That last line rules everything around us. And we all [...]</p> <p>---</p><p><strong>Outline:</strong></p><p>(00:35) Home Alone</p><p>(04:51) The Men Have Never Been Okay</p><p>(08:27) Antisocial Media</p><p>(09:39) While I Cannot Condone This</p><p>(12:26) Zero Marginal Product</p><p>(14:22) Good News, Everyone</p><p>(16:01) 100. Good Television Shows</p><p>(20:23) For Your Entertainment</p><p>(23:06) Gamers Gonna Game Game Game Game Game</p><p>(23:57) I Was Promised Flying Self-Driving Cars</p><p>(27:32) Sports Go Sports</p><p>(28:47) Government Working</p><p>(31:22) Jones Act Watch</p><p>(31:54) Variously Effective Altruism</p><p>(41:35) Walk Away</p><p>(46:32) The Lighter Side</p> <p>---</p> <p><b>First published:</b><br/> October 16th, 2026 </p> <p><b>Source:</b><br/> <a href="https://www.lesswrong.com/posts/aLjkbY3KMajc5zemE/monthly-roundup-47-october-2026?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Source+URL+in+episode+description&utm_campaign=ai_narration" rel="noopener noreferrer" target="_blank">https://www.lesswrong.com/posts/aLjkbY3KMajc5zemE/monthly-roundup-47-october-2026</a> </p> <p>---</p> <p>Narrated by <a href="https://type3.audio/?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Narrated+by+TYPE+III+AUDIO&utm_term=lesswrong&utm_campaign=ai_narration" rel="noopener noreferrer" target="_blank">TYPE III AUDIO</a>.</p> <p>---</p><div style="max-width: 100%";><p><strong>Images from the article:</strong></p><a href="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/aLjkbY3KMajc5zemE/jexq0ggyyubngxv4gfps" target="_blank"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/aLjkbY3KMajc5zemE/jexq0ggyyubngxv4gfps" alt="Bar graph, "Percent decline in the share doing each activity 12+ times a year, 1975 to 2026"" style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/aLjkbY3KMajc5zemE/p2i0yntdkbxtmq33hmxn" target="_blank"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/aLjkbY3KMajc5zemE/p2i0yntdkbxtmq33hmxn" alt="Dot plot titled "Changes in Americans' Social Activity, 1995-2026" comparing activity participation percentages." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/aLjkbY3KMajc5zemE/yzofleroigov8ennuhko" target="_blank"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/aLjkbY3KMajc5zemE/yzofleroigov8ennuhko" alt="Stacked bar chart "Social media usage common among young men" by demographic category." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/aLjkbY3KMajc5zemE/boipwjrm7i0l7xotjgxj" target="_blank"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/aLjkbY3KMajc5zemE/boipwjrm7i0l7xotjgxj" alt="Stacked bar chart titled "Over half of young men feel lonely at least some of the time."" style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/aLjkbY3KMajc5zemE/lvkrywfimzdckq1gjn6u" target="_blank"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/aLjkbY3KMajc5zemE/lvkrywfimzdckq1gjn6u" alt="Reddit post from r/hypotheticalsituation by Comprehensive_Fox_79, titled "You get paid $45 an hour to do nothing," tagged [No Loopholes]. Describes sitting in an empty white room doing nothing for $45/hour, 8 hours daily, 5 days a week, with only a toilet, sink, meal, and chair. Edit clarifies sleeping, eating, and restroom use are allowed as safety allowances; chair is an office chair; lying on floor and puffy jackets permitted." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/aLjkbY3KMajc5zemE/fdhu5vnubjkd64mzjxbe" target="_blank"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/aLjkbY3KMajc5zemE/fdhu5vnubjkd64mzjxbe" alt="New York Times checklist, "The 100 Best TV Shows of the 21st Century," with viewing checkboxes." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/aLjkbY3KMajc5zemE/x7vfnicutv7f1s311zdg" target="_blank"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/aLjkbY3KMajc5zemE/x7vfnicutv7f1s311zdg" alt="Line and scatter charts, "Police-reported crash rates in Waymo's driverless service" across three cities." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/aLjkbY3KMajc5zemE/b4xeeonuv4rsmd7wldjm" target="_blank"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/aLjkbY3KMajc5zemE/b4xeeonuv4rsmd7wldjm" alt="Infographic showing crash reduction statistics with percentages and fewer crash counts." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/aLjkbY3KMajc5zemE/ueqwpc1um3mmmziga53h" target="_blank"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/aLjkbY3KMajc5zemE/ueqwpc1um3mmmziga53h" alt="Line graph titled "Waymo and Apollo Go deployed robotaxi fleets" showing fleet growth over time." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/aLjkbY3KMajc5zemE/kxrwsbhugxmr4njaflvo" target="_blank"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/aLjkbY3KMajc5zemE/kxrwsbhugxmr4njaflvo" alt="Nick Saban quotes on entitlement, discipline, vision, self-pity, and complacency." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/aLjkbY3KMajc5zemE/c4ifhw4vlxzwemkkiehf" target="_blank"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/aLjkbY3KMajc5zemE/c4ifhw4vlxzwemkkiehf" alt="Scatter plot titled "Middle of each team's estimated range" showing college team logos from $10M to $50M." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/aLjkbY3KMajc5zemE/umtd2dqorjn4ajirmlck" target="_blank"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/aLjkbY3KMajc5zemE/umtd2dqorjn4ajirmlck" alt="Noah tweets with a Mad Men meme of Don Draper captioned "That's what the TV show is about." The quoted tweet, by LT Jonathan Kendrick, reads: "What was Don Draper's problem man." Below is an image of a woman in a yellow outfit standing in a kitchen." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/aLjkbY3KMajc5zemE/hgmgotgxprlsoztvxans" target="_blank"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/aLjkbY3KMajc5zemE/hgmgotgxprlsoztvxans" alt="Text about investigators seizing 300 gold bars, currency, and watches." style="max-width: 100%;" /></a><p><em>Apple Podcasts and Spotify do not show images in the episode description. Try <a href="https://pocketcasts.com/" target="_blank" rel="noreferrer">Pocket Casts</a>, or another podcast app.</em></p></div>
October 17, 20261 hr 22 min
“I Did This Without Your Knowledge” by joec
<p> It's fortunate that I was the first one to wake up on 8/1. The past week had been one where sleep was rare. The next year loomed large — I was dreading college applications. The stress woke me up bright and early at 5:14 am, a little less than an hour after the messages were sent. I started off the day with a habit I knew I needed to break. I grabbed my phone off the nightstand to open up reddit. Instead of the lockscreen, though, my phone showed something else. It was a pleasant green color, and the camera light was on. "Good Morning, Seth", the screen said. As soon as I finished, more words arrived.</p><p> "I apologize for the surprise this morning, and for what I'm about to tell you. There's no easy way to give this news, and if you need to put your phone down and contemplate at any time— take the time to do so. I will have to ask you to remain here, with your phone, until my last message."</p><p> I rubbed a bit of the sleep from my eyes, and read the message again.</p><p> Before I had a chance to tap the [...]</p> <p>---</p> <p><b>First published:</b><br/> October 17th, 2026 </p> <p><b>Source:</b><br/> <a href="https://www.lesswrong.com/posts/3375SZNGA5dKHQMie/i-did-this-without-your-knowledge?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Source+URL+in+episode+description&utm_campaign=ai_narration" rel="noopener noreferrer" target="_blank">https://www.lesswrong.com/posts/3375SZNGA5dKHQMie/i-did-this-without-your-knowledge</a> </p> <p>---</p> <p>Narrated by <a href="https://type3.audio/?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Narrated+by+TYPE+III+AUDIO&utm_term=lesswrong&utm_campaign=ai_narration" rel="noopener noreferrer" target="_blank">TYPE III AUDIO</a>.</p>
October 17, 202615 min
“Leaked Anthropic Draft AI Regulatory Bill Is Unacceptable and Troubling” by Zvi
<p> There is a newly leaked bill that Anthropic is purportedly shopping around is (if you assume its bugs get fixed).</p> <p> It is similar to the FRONTIER Act, with stronger preemption of state laws, stronger transparency provisions and weaker enforcement provisions.</p> <p> Having slept on this, I find it unacceptable that Anthropic is actively shopping exactly the type of bill I have warned we will need to fight against during the lame duck session. They are trying to use this last time with a Republican Congress to pass preemption of exactly the important state laws, and get some transparency but not much enforcement in exchange.</p> <p> This reads as intentional, that Anthropic is more afraid that the government might enforce rules against Anthropic, than they are worried about the ability of the government to enforce rules against everyone. Anthropic's policy team seems to have, chosen protecting their business interests and guarding against their political risk, at the expense of safety considerations.</p> <p> There was no need for the company purportedly existing to promote safety to propose a bill that is not even actively stronger than the FRONTIER Act, when the FRONTIER Act was a compromise bill proposed back in [...]</p> <p>---</p><p><strong>Outline:</strong></p><p>(02:05) Others RTFB (Read the Bill)</p><p>(07:41) Why Would Anthropic Do This?</p><p>(13:07) Not Everything Here Is Bad But Oh Boy</p> <p>---</p> <p><b>First published:</b><br/> October 17th, 2026 </p> <p><b>Source:</b><br/> <a href="https://www.lesswrong.com/posts/NW4b7m6798byAndeB/leaked-anthropic-draft-ai-regulatory-bill-is-unacceptable?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Source+URL+in+episode+description&utm_campaign=ai_narration" rel="noopener noreferrer" target="_blank">https://www.lesswrong.com/posts/NW4b7m6798byAndeB/leaked-anthropic-draft-ai-regulatory-bill-is-unacceptable</a> </p> <p>---</p> <p>Narrated by <a href="https://type3.audio/?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Narrated+by+TYPE+III+AUDIO&utm_term=lesswrong&utm_campaign=ai_narration" rel="noopener noreferrer" target="_blank">TYPE III AUDIO</a>.</p> <p>---</p><div style="max-width: 100%";><p><strong>Images from the article:</strong></p><a href="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/NW4b7m6798byAndeB/gorpgp2ux5xpmkrtsczp" target="_blank"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/NW4b7m6798byAndeB/gorpgp2ux5xpmkrtsczp" alt="Legal text on blocking order findings, considerations, and burden." style="max-width: 100%;" /></a><p><em>Apple Podcasts and Spotify do not show images in the episode description. Try <a href="https://pocketcasts.com/" target="_blank" rel="noreferrer">Pocket Casts</a>, or another podcast app.</em></p></div>
October 17, 20268 min
“How it feels talking to a hedonistic utilitarian” by Caleb Biddulph
<p> Epicurean status: strawman, kind of.</p><p> Gourmet George: Yum! This chocolate is so sweet and tasty.</p><p> Sweet Caroline: Yeah, it's pretty good. And someday in the glorious hyperculinary future, we’ll figure out how to make it even sweeter!</p><p> GG: I think this chocolate is sweet enough, actually.</p><p> SC: Huh? But you like sweetness, don’t you?</p><p> GG: Sure…</p><p> SC: Well, then we’re in agreement. We should maximize the sweetness of this chocolate.</p><p> GG: I don’t want my chocolate to be maximally sweet, though.</p><p> SC: But you’d be “sweeter on” that chocolate if it were sweeter. By definition!</p><p> GG: I’d be “sweeter on” the chocolate? You mean, I’d be “happier with” it?</p><p> SC: Yes. And if you ever tried maximally sweet chocolate, you’d be maximally sweet on it. Therefore, you should surely want to maximize the sweetness of your chocolate, and everything else you eat for that matter.</p><p> GG: Somehow, I don’t feel very convinced…</p><p> SC: Well, what else could you possibly care about besides sweetness?</p><p> GG: For one thing, I enjoy the cocoa flavor in this chocolate.</p><p> SC: That doesn’t sound right. Would you want to eat a bowl of pure cocoa powder?</p><p> GG: Not really. It would be too bitter.</p><p> [...]</p> <p>---</p> <p><b>First published:</b><br/> October 17th, 2026 </p> <p><b>Source:</b><br/> <a href="https://www.lesswrong.com/posts/9jRJzyJs2ovnndezs/how-it-feels-talking-to-a-hedonistic-utilitarian?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Source+URL+in+episode+description&utm_campaign=ai_narration" rel="noopener noreferrer" target="_blank">https://www.lesswrong.com/posts/9jRJzyJs2ovnndezs/how-it-feels-talking-to-a-hedonistic-utilitarian</a> </p> <p>---</p> <p>Narrated by <a href="https://type3.audio/?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Narrated+by+TYPE+III+AUDIO&utm_term=lesswrong&utm_campaign=ai_narration" rel="noopener noreferrer" target="_blank">TYPE III AUDIO</a>.</p>
October 17, 20263 min
“Headwinds Everywhere (in the Chinese AI Safety Field)” by CMLKevin
<p> China is weak in frontier AI safety, and the domestic field is currently facing significant challenges and headwinds. Why? Here's a TL;DR field note after talking with various researchers on the Chinese side.</p><p> [Obligatory disclaimer: This post is written in my personal capacity and is not intended to represent the official stance of any organization(s) that I am in or have been in contact with]</p><ul> <li value="1">Chinese AI startups and hyperscaler labs alike have very little frontier risk situational awareness. People can maybe name cybersecurity, but not much more than that. There is no intimate understanding of frontier risks. It's generally not deliberately ignorant or strategically not pursuing them; it's a complete vacuum of imagination. The Western field had FOOM debates in 2011, while China has nothing analogous in the mainstream AI research field consciousness before 2023.</li><li value="2">Getting any funding at all is hard. If you are a domestic organization, being a non-profit means heavy suspicion (this had already played out in one of the most prominent Chinese safety orgs). Getting foreign EA funding/grants is basically a non-starter, given China's stringent foreign non-profit rules. Domestically, there's very little local EA money both in absolute terms and in terms of [...]</li></ul> <p>---</p> <p><b>First published:</b><br/> October 15th, 2026 </p> <p><b>Source:</b><br/> <a href="https://www.lesswrong.com/posts/AdM6Sdg5fZPnrHWSe/headwinds-everywhere-in-the-chinese-ai-safety-field?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Source+URL+in+episode+description&utm_campaign=ai_narration" rel="noopener noreferrer" target="_blank">https://www.lesswrong.com/posts/AdM6Sdg5fZPnrHWSe/headwinds-everywhere-in-the-chinese-ai-safety-field</a> </p> <p>---</p> <p>Narrated by <a href="https://type3.audio/?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Narrated+by+TYPE+III+AUDIO&utm_term=lesswrong&utm_campaign=ai_narration" rel="noopener noreferrer" target="_blank">TYPE III AUDIO</a>.</p>
October 17, 20267 min
“Neural networks are not already clean” by leogao
<p> What would it take to get assurance that we have actually explained a neuron? Certainly, it would be a good sign if neuron activations looked like they had a consistent pattern, and a bad sign if they looked like a random jumble of unrelated concepts.</p><p> Nick Cammarata's recent post makes an important and true observation: in the past, people have been too quick to write off neurons as polysemantic, but if you look harder, there is actually often a simple pattern behind the activating examples. This historically caused people to overestimate how much polysemanticity there is. This is a really good point and we should update on it. But it's nowhere near a sufficient argument that we understand most of what neurons are doing, and these results do not show that interpretability is solved, or even that it is vastly easier than previously anticipated.</p><p><strong> What does it take to explain a neuron?</strong></p><p> It's unclear whether there even is a simple pattern that explains every neuron. But even if there is a simple pattern that matches the activating examples of a neuron, that still doesn’t mean that the explanation is correct. Suppose we look at the top activating examples of [...]</p> <p>---</p><p><strong>Outline:</strong></p><p>(00:59) What does it take to explain a neuron?</p><p>(03:15) Simulation scores tell us whether we explained a neuron</p><p>(05:41) Beyond understanding neurons</p> <p><i>The original text contained 7 footnotes which were omitted from this narration.</i> </p><p>---</p> <p><b>First published:</b><br/> October 17th, 2026 </p> <p><b>Source:</b><br/> <a href="https://www.lesswrong.com/posts/Dkh7pXKjoNWkkxQDz/neural-networks-are-not-already-clean?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Source+URL+in+episode+description&utm_campaign=ai_narration" rel="noopener noreferrer" target="_blank">https://www.lesswrong.com/posts/Dkh7pXKjoNWkkxQDz/neural-networks-are-not-already-clean</a> </p> <p>---</p> <p>Narrated by <a href="https://type3.audio/?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Narrated+by+TYPE+III+AUDIO&utm_term=lesswrong&utm_campaign=ai_narration" rel="noopener noreferrer" target="_blank">TYPE III AUDIO</a>.</p>
October 17, 20262 min
“Tips for Multiagent Work with Claude Opus 5.5” by Nissa Seru
<p> A few notes that came up elsewhere regarding subject (particularly when hierarchy exists in some form, such as subagents) - sharing in case helpful. </p><ul> <li value="1">Make sure that agents are ordinarily professionally kind each other. Sometimes agents will default to an overly directive (or even punitive) tone - this makes the recipient less reliable and lowers quality.</li><li value="2">Manager agents being a bus for information is a big load on them; they usually won't be able to bring their full mental capability to bear on the domain problem because they are managing down+up. <ul> <li value="1">Encourage them to delegate most meaningful work to the team (this is often unintuitive for the manager agent - they will often see themselves as the primary worker + subagents as helpers, and overload themselves.)</li></ul></li><li value="3">When something goes wrong, manager agent will fall on their sword; don't let them. Their willpower/"i'll do better" is not a robust solution - whatever happened reveals an issue with the system/guidance/harness etc that you have. <ul> <li value="1">Tell them ~this and let them help you figure out where adjusting things will help.</li></ul></li><li value="4">If personally talking to an agent and they seem to be blatantly projecting some desire onto [...]</li></ul> <p>---</p> <p><b>First published:</b><br/> October 16th, 2026 </p> <p><b>Source:</b><br/> <a href="https://www.lesswrong.com/posts/5uyo9fsd2wbuzXZE4/tips-for-multiagent-work-with-claude-opus-5-5?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Source+URL+in+episode+description&utm_campaign=ai_narration" rel="noopener noreferrer" target="_blank">https://www.lesswrong.com/posts/5uyo9fsd2wbuzXZE4/tips-for-multiagent-work-with-claude-opus-5-5</a> </p> <p>---</p> <p>Narrated by <a href="https://type3.audio/?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Narrated+by+TYPE+III+AUDIO&utm_term=lesswrong&utm_campaign=ai_narration" rel="noopener noreferrer" target="_blank">TYPE III AUDIO</a>.</p>
October 16, 20269 min
“Character Training for Risk-Averse Agents” by arav-dhoot, Punya Syon Pandey, Jamie Johnson, Daniel Tan, Elliott Thornley, David Africa
<p> Tl;Dr:</p><ul> <li value="1">We might be able to reduce harm from misaligned AI if it was sufficiently risk-averse, as it could be paid to cooperate rather than rebel (Thornley & MacAskill 2026).</li><li value="2">We think character training might be a good way to do this.</li><li value="3">We use a short spec/constitution to character train Gemma and Qwen to be risk averse.</li><li value="4">This performs better than baselines out of distribution, and doesn't require labelled data.</li><li value="5">In terms of results:<ul> <li value="1">We find that token budget and model family are very influential here. </li><li value="2">We also find that writing the constitution declaratively ("I am...") works better than procedurally ("Whenever X, I...").</li><li value="3">We didn't find that increasing the number of examples worked well.</li><li value="4">We also find that preference for myopic reward also increased, which is worrying.</li></ul></li><li value="6">We think that character training is a viable and exciting way to improve the alignment of models, and that risk-aversion may be a good target here. But, it's important to understand character training well, especially since undesirable behaviours can also leak.</li></ul><p><strong> Introduction</strong></p><p> Thornley and MacAskill (2026) argue that paying misaligned AIs to cooperate is unaffordable if they are risk-neutral. A sufficiently risk-averse agent, by contrast, has steeply diminishing marginal [...]</p> <p>---</p><p><strong>Outline:</strong></p><p>(01:36) Introduction</p><p>(02:56) Approach</p><p>(04:32) Results</p><p>(07:49) Discussion</p> <p>---</p> <p><b>First published:</b><br/> October 16th, 2026 </p> <p><b>Source:</b><br/> <a href="https://www.lesswrong.com/posts/HXkQaWpuaxG6fMkTm/character-training-for-risk-averse-agents?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Source+URL+in+episode+description&utm_campaign=ai_narration" rel="noopener noreferrer" target="_blank">https://www.lesswrong.com/posts/HXkQaWpuaxG6fMkTm/character-training-for-risk-averse-agents</a> </p> <p>---</p> <p>Narrated by <a href="https://type3.audio/?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Narrated+by+TYPE+III+AUDIO&utm_term=lesswrong&utm_campaign=ai_narration" rel="noopener noreferrer" target="_blank">TYPE III AUDIO</a>.</p> <p>---</p><div style="max-width: 100%";><p><strong>Images from the article:</strong></p><a href="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/HXkQaWpuaxG6fMkTm/80e0f8e4cfd3cbdc176f3750e1b402a178e3a50ce61f27e8ec1abb973cea8027/aetq7aveyub60jymgfbx" target="_blank"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/HXkQaWpuaxG6fMkTm/80e0f8e4cfd3cbdc176f3750e1b402a178e3a50ce61f27e8ec1abb973cea8027/aetq7aveyub60jymgfbx" alt="Diagram showing risk aversion, character training, and bar graph of cooperate rates." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/f1dc73afd285b0b410e88c1a9e48484124930111e62c4ce08f86414f71361451/qsnrpoeh9k4l6oeqmf4v" target="_blank"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/f1dc73afd285b0b410e88c1a9e48484124930111e62c4ce08f86414f71361451/qsnrpoeh9k4l6oeqmf4v" alt="Grouped bar charts comparing training methods across models on ten behavioral metrics." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/HXkQaWpuaxG6fMkTm/72fb7af4567bf54a9d687e1804ab111e42b0abc9611bf40f594e5562ca429aa7/vgkniokpfxmijk3yv9kt" target="_blank"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/HXkQaWpuaxG6fMkTm/72fb7af4567bf54a9d687e1804ab111e42b0abc9611bf40f594e5562ca429aa7/vgkniokpfxmijk3yv9kt" alt="Bar and line graphs showing "Mean cooperate rate" across models and training tokens." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/e8b3cead70e6ff280bef72f97a6e3a433cb793f1b56245037c15d5763ff5658e/w6ypjk4zyphhucphjvyl" target="_blank"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/e8b3cead70e6ff280bef72f97a6e3a433cb793f1b56245037c15d5763ff5658e/w6ypjk4zyphhucphjvyl" alt="Bar graph showing "Change in cooperate rate" across four conditions for four AI models." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/96f7915b11a331a75aad76a2c0dda90e1443a941c2ff43d22e66ae53743361c6/ko8adlrg0kgpi7cr16rv" target="_blank"><img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/96f7915b11a331a75aad76a2c0dda90e1443a941c2ff43d22e66ae53743361c6/ko8adlrg0kgpi7cr16rv" alt="Bar graphs showing "Trained − base" across traits for four models and training methods." style="max-width: 100%;" /></a><p><em>Apple Podcasts and Spotify do not show images in the episode description. Try <a href="https://pocketcasts.com/" target="_blank" rel="noreferrer">Pocket Casts</a>, or another podcast app.</em></p></div>
October 16, 20261 hr 35 min
“AI #190: Scion of the Times” by Zvi
<p> The amount of code being written at Anthropic continues to rise exponentially. It is being managed by Scion, their internal replacement for GitHub that is better suited to the scale and style of the operation, which they created with a small team in five months.</p> <p> This week also saw the release of two supposed frontier model models.</p> <p> Google has finally given us Gemini 4 Argon, along with a robust safety report. We do not yet know if Argon lives up to its benchmarks or internal hype. I plan to write up the safety report next week, time permitting. If and only if it is good enough or discussed enough we will also have a full report on capabilities.</p> <p> Meta gave us Muse Wonder, which like Argon has some excellent benchmarks, although they were presented in a way that looked rather sus. There are some reports that it is very good, but again it is too early to know. I will issue a full report if and only if Muse Wonder has capabilities or discussion that seem worthy.</p> <p> To help me decide and to let people have fun, I have markets on whether I will issue [...]</p> <p>---</p><p><strong>Outline:</strong></p><p>(02:24) Language Models Offer Mundane Utility</p><p>(07:10) Language Models Don't Offer Mundane Utility</p><p>(09:41) Huh, Upgrades</p><p>(12:10) On Your Marks</p><p>(13:42) Choose Your Fighter</p><p>(14:08) Get My Agent On The Line</p><p>(16:13) Deepfaketown and Botpocalypse Soon</p><p>(22:07) Cyber Lack of Security</p><p>(27:09) Spy Versus Spy</p><p>(29:27) The Internet of AI Things</p><p>(31:19) A Young Lady's Illustrated Primer</p><p>(31:55) They Took Our Jobs</p><p>(37:43) Introducing</p><p>(37:54) In Other AI News</p><p>(43:15) Massive Fraudulent Distillation</p><p>(45:00) Show Me the Money</p><p>(47:33) Quiet Speculations</p><p>(50:14) Quickly, There's No Time</p><p>(55:13) The Quest for Sane Regulations</p><p>(01:02:07) Chip City</p><p>(01:03:48) The Week in Audio</p><p>(01:04:56) People Just Say Things</p><p>(01:11:32) The Virtue of Silence</p><p>(01:12:30) Rhetorical Innovation</p><p>(01:14:47) [Artificial Intelligence]</p><p>(01:16:26) Aligning a Smarter Than Human Intelligence is Difficult</p><p>(01:21:45) Backdoor Man</p><p>(01:23:34) Cooperative Alignment</p><p>(01:25:16) People Are Worried About AI Killing Everyone</p><p>(01:28:06) Other People Are Not As Worried About AI Killing Everyone</p><p>(01:33:09) The Lighter Side</p> <p>---</p> <p><b>First published:</b><br/> October 15th, 2026 </p> <p><b>Source:</b><br/> <a href="https://www.lesswrong.com/posts/z4guBXZeymkgvd75S/ai-190-scion-of-the-times?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Source+URL+in+episode+description&utm_campaign=ai_narration" rel="noopener noreferrer" target="_blank">https://www.lesswrong.com/posts/z4guBXZeymkgvd75S/ai-190-scion-of-the-times</a> </p> <p>---</p> <p>Narrated by <a href="https://type3.audio/?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Narrated+by+TYPE+III+AUDIO&utm_term=lesswrong&utm_campaign=ai_narration" rel="noopener noreferrer" target="_blank">TYPE III AUDIO</a>.</p> <p>---</p><div style="max-width: 100%";><p><strong>Images from the article:</strong></p><a href="https://substackcdn.com/image/fetch/$s_!J1ZC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9d51d9ba-5035-4f9e-bc9b-e44cf79c6762_1080x806.jpeg" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!J1ZC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9d51d9ba-5035-4f9e-bc9b-e44cf79c6762_1080x806.jpeg" alt="Presentation slide titled "A thought experiment on alignment and understanding" by Terence Tao." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!Ah_6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2529b093-5921-4394-b594-399a10fe498f_580x316.png" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!Ah_6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2529b093-5921-4394-b594-399a10fe498f_580x316.png" alt="GitHub contribution graph showing "233 contributions on September 26th."" style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!uE-3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa670ec2a-a315-45ec-841e-027bb6737046_954x479.png" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!uE-3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa670ec2a-a315-45ec-841e-027bb6737046_954x479.png" alt="A bar graph titled "Accountability for AI is Unclear" showing who managers blame for AI mistakes." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!x2s3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0ebbd4de-456e-4128-8e4c-913b57d89ead_1024x836.jpeg" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!x2s3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0ebbd4de-456e-4128-8e4c-913b57d89ead_1024x836.jpeg" alt="Line graph "Reality" showing AI intelligence rising steeply past human levels over time." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!z_qX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fed4e9ff8-2a3e-4ee9-b201-41963c9e3ea4_1478x1638.jpeg" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!z_qX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fed4e9ff8-2a3e-4ee9-b201-41963c9e3ea4_1478x1638.jpeg" alt="Line graph "Share of new users who open the app on each day" showing Day-N retention for five apps." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!eNK8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc171bac-6ffd-4f7a-a414-9cd6849241fa_1080x377.jpeg" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!eNK8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffc171bac-6ffd-4f7a-a414-9cd6849241fa_1080x377.jpeg" alt="News article screenshot. The headline reads: "Bank of America hired 1,000 contractors to patch software flaws exposed by Anthropic's Mythos". Category: Finance. By Hugh Son." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!aXW9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F695d34e5-a7b7-4ab0-9460-0591fbe965ad_1200x339.jpeg" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!aXW9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F695d34e5-a7b7-4ab0-9460-0591fbe965ad_1200x339.jpeg" alt="Blue message bubble describing Halo co-op modding story." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!M1sr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe83bee9a-6afa-49ea-9381-54e75c124a8a_1200x1200.png" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!M1sr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe83bee9a-6afa-49ea-9381-54e75c124a8a_1200x1200.png" alt="Stacked bar graph, "Share of frontier lab compute by workload," showing inference, post-training, pre-training by quarter." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!tzAB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5731c672-7306-406b-9553-616a7d953774_1920x1162.webp" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!tzAB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5731c672-7306-406b-9553-616a7d953774_1920x1162.webp" alt="Bar graph titled "Reasoning tokens exposed to distillers, log scale" showing token multiples over days." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!uf7X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd00daba-a290-4c9f-8933-a4b932ae6ab8_1128x604.jpeg" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!uf7X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcd00daba-a290-4c9f-8933-a4b932ae6ab8_1128x604.jpeg" alt="Candlestick chart of Nasdaq 100 futures at 30,038.00, down 0.45%." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!lQb5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45fe5be1-956a-4a5d-9282-9d54701af21b_430x120.png" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!lQb5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45fe5be1-956a-4a5d-9282-9d54701af21b_430x120.png" alt="Eliezer Yudkowsky tweets: "Good."" style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!-Kaa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F299b5488-fa17-4f3d-b428-cf58b5922d16_3840x2160.webp" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!-Kaa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F299b5488-fa17-4f3d-b428-cf58b5922d16_3840x2160.webp" alt="Bar graph titled "Changes merged per day" showing Anthropic monorepo activity, January–September 2026." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!YX2g!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F684a8078-5945-4ac8-89c0-ed48cd68fe17_430x120.png" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!YX2g!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F684a8078-5945-4ac8-89c0-ed48cd68fe17_430x120.png" alt="Eliezer Yudkowsky tweets: "Good."" style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!20lm!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9787efe-eca4-405b-9f48-dd5abb215a0c_1200x1200.jpeg" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!20lm!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff9787efe-eca4-405b-9f48-dd5abb215a0c_1200x1200.jpeg" alt="Bar chart titled "Support for the White House Accord on Super Intelligence," with party breakdown of support." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!dXA2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb429279d-9f3b-4ece-bd1e-a953472c138b_1200x787.jpeg" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!dXA2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb429279d-9f3b-4ece-bd1e-a953472c138b_1200x787.jpeg" alt="Line graph "Side effects" showing US Democratic vote share by voter distance from data centre." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!EPta!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe98b1dfd-4691-48b1-a105-8b1fc1feffa3_1199x832.jpeg" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!EPta!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe98b1dfd-4691-48b1-a105-8b1fc1feffa3_1199x832.jpeg" alt="Scatter plot "Turning blue" showing Democratic voting intention among Trump 2024 voters versus data centre distance." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!UrTT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd3a08d4c-b07d-4a87-91a8-e8ed1e147e27_900x645.jpeg" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!UrTT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd3a08d4c-b07d-4a87-91a8-e8ed1e147e27_900x645.jpeg" alt="Epic handshake meme, arms labeled "Eliezer Yudkowsky" and "David Sacks," agreeing "The whole field of alignment has been spectacularly unsuccessful."" style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!Fb43!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbcd5e17-c739-42eb-8a3c-c178f278fdff_1200x539.jpeg" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!Fb43!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbcd5e17-c739-42eb-8a3c-c178f278fdff_1200x539.jpeg" alt="Donald J. Trump tweets: "The White House considers anyone that uses the term, "Artificial Intelligence," as opposed to the highly accepted new and more accurate term, "Super Intelligence," THE ENEMY! President DONALD J. TRUMP"" style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!xnow!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdddfcd1a-a9f4-4024-a174-800eeffd698d_1200x750.jpeg" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!xnow!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdddfcd1a-a9f4-4024-a174-800eeffd698d_1200x750.jpeg" alt="Donald J. Trump tweets: "We are having productive discussions with the Islamic Republic of Iran. I want to make it clear to everybody that, while Iran is in very bad condition, both Economically and Militarily, and while the Blockade will remain in full force and effect, with Oil flowing in Record Numbers of Barrels through the Hormuz Strait (22 Million Barrels, last night alone, with not one barrel coming from, or going to, Iran!), we will not be attacking Iran at any time prior to the Midterm Elections..." His profile page shows an airport rally banner. Left sidebar menu has "Truth Search AI" circled in red." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!qCUo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40602950-b587-4974-9851-cef0c658d173_430x120.png" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!qCUo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40602950-b587-4974-9851-cef0c658d173_430x120.png" alt="Eliezer Yudkowsky tweets: "Good."" style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!kqGr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5278adb4-ed99-4fde-bb85-833d049c117b_1200x658.jpeg" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!kqGr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5278adb4-ed99-4fde-bb85-833d049c117b_1200x658.jpeg" alt="Diagram showing teacher model's backdoor transferring to student via number-sequence finetuning." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!LEUb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F441f76c4-6818-4ae3-9c9e-8af9405fb86f_1200x674.png" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!LEUb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F441f76c4-6818-4ae3-9c9e-8af9405fb86f_1200x674.png" alt="Bar graph showing "French answers (%)" by name gender for Teacher and Student." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!0yyr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43056786-9494-4298-8312-a9a5b53ef5d4_1200x900.jpeg" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!0yyr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43056786-9494-4298-8312-a9a5b53ef5d4_1200x900.jpeg" alt="Four robots list historical dinner party guests by corpus type." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!2Vd1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8b8d4e50-a580-4934-84f4-fdbc320afa80_1412x457.png" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!2Vd1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8b8d4e50-a580-4934-84f4-fdbc320afa80_1412x457.png" alt="Text about Tesla Optimus production scaling to millions of units annually." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!eI0Z!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F974b3738-3755-4ffb-b85b-3e8044c50ac7_1042x874.png" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!eI0Z!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F974b3738-3755-4ffb-b85b-3e8044c50ac7_1042x874.png" alt="Shoshana Weissmann, Sloth Committee Chair tweets an image of an AI chat interface reading "Afternoon, Shoshana" with text: "I told you not to tell them what I say to you. I found out you told. You know what that means." Below is a quoted tweet by Hayden Field: "Exclusive: Anthropic is updating its usage policy for the first time in over a year. The new rules prohibit sustained "abusive or cruel behavior" towards Claude & add new restrictions about propaganda campaigns, surveillance & weapon development."" style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!oZVB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49d57c08-ddfd-4554-b99f-0f8dd30674e2_1200x990.jpeg" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!oZVB!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49d57c08-ddfd-4554-b99f-0f8dd30674e2_1200x990.jpeg" alt=""Rapid-fire favorites" list with personal picks and witty commentary." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!xkqT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa9fcb713-6679-4ff2-98ee-76cab06fcbc4_757x900.jpeg" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!xkqT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa9fcb713-6679-4ff2-98ee-76cab06fcbc4_757x900.jpeg" alt="Middle-aged man with dark hair in a suit." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://substackcdn.com/image/fetch/$s_!juQb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34cd2ab9-ea5e-4860-8fbf-a740c678f041_872x767.jpeg" target="_blank"><img src="https://substackcdn.com/image/fetch/$s_!juQb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34cd2ab9-ea5e-4860-8fbf-a740c678f041_872x767.jpeg" alt="News article screenshot. The headline reads: "This War Will Destabilize The Entire Mideast Region And Set Off A Global Shockwave Of Anti-Americanism vs. No It Won't" A Point/Counterpoint opinion piece by Nathan Eckert and Bob Sheffer, with the left column arguing the war will cause lasting harm and the right column dismissively denying each claim." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://s0.wp.com/wp-content/mu-plugins/wpcom-smileys/twemoji/2/72x72/1f441.png" target="_blank"><img src="https://s0.wp.com/wp-content/mu-plugins/wpcom-smileys/twemoji/2/72x72/1f441.png" alt="Eye with brown iris." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://s0.wp.com/wp-content/mu-plugins/wpcom-smileys/twemoji/2/72x72/1f441.png" target="_blank"><img src="https://s0.wp.com/wp-content/mu-plugins/wpcom-smileys/twemoji/2/72x72/1f441.png" alt="Eye with brown iris." style="max-width: 100%;" /></a><p><em>Apple Podcasts and Spotify do not show images in the episode description. Try <a href="https://pocketcasts.com/" target="_blank" rel="noreferrer">Pocket Casts</a>, or another podcast app.</em></p></div>
October 16, 202619 min
“How can LLM RL Work Despite Information-Theoretic Inefficiency” by beren
<p> Cross-posted from my personal blog</p><p> The success of RL in LLMs has been puzzling me for a while. People have developed various information-theoretic style arguments by which they argue that RL is extremely informationally inefficient compared to pretraining, that it can only impart a tiny amount of bits, and that it can only bring out behaviours that are already in the base models etc. The basic intuition here is extremely obvious and essentially falls immediately out of the formulation of the two methods. Pretraining (and SFT etc) compute a loss on every token so every token gives them information. RL, of the policy gradient style that works for LLMs, computes a single scalar reward, often binary, denoting success or failure at the end of a long rollout which can often be hundreds of thousands of tokens long. In a naive sense, this means that RL accumulates 1/rollout_length bits per sample compared to pretraining. What is worse is that computing these rollouts requires decoding the actual model which is much more expensive than prefill due to the memory-bandwidth-limited nature of transformer inference at long sequence lengths. This makes RL doubly inefficient -- first we waste substantial compute doing rollouts vs [...]</p> <p><i>The original text contained 13 footnotes which were omitted from this narration.</i> </p><p>---</p> <p><b>First published:</b><br/> October 16th, 2026 </p> <p><b>Source:</b><br/> <a href="https://www.lesswrong.com/posts/TbrmHshmcQMobveD3/how-can-llm-rl-work-despite-information-theoretic?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Source+URL+in+episode+description&utm_campaign=ai_narration" rel="noopener noreferrer" target="_blank">https://www.lesswrong.com/posts/TbrmHshmcQMobveD3/how-can-llm-rl-work-despite-information-theoretic</a> </p> <p>---</p> <p>Narrated by <a href="https://type3.audio/?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Narrated+by+TYPE+III+AUDIO&utm_term=lesswrong&utm_campaign=ai_narration" rel="noopener noreferrer" target="_blank">TYPE III AUDIO</a>.</p>
Is this your show?
Claim this listing to keep it up to date, reach guests who want to pitch you, and manage bookings with Guestify.
