“General capability - and capabilities generally - have no good y-axis” by Gregory Lewis🔸
<p> BLUF:</p><ul> <li> To determine whether AI is ‘improving exponentially’, ‘hitting the wall’, or any other claim which involves a quantity or magnitude (e.g. ‘This model was a big leap/small increment’). We need a good y-axis: an interval scale of AI capability which means +1 unit always represents the same degree of ‘how much better’, in the same way +1 degree Celsius is always the same amount of ‘how much hotter’.</li><li> Yet there is no good y-axis for AI capability. All our measures are of something related-to but clearly not identical-with it, thus ‘true’ AI capability can be a funhouse-mirror reflection of whatever was measured. Specifically:<ul> <li> Benchmark score: One small step in benchmark score can be a giant leap in capability, or the opposite, or whatever else. (My 6/10 vs. your 4/10 ≠ I’m 50% better at maths than you).</li><li> Elo et al: Can give a real y-axis in terms of winning chances, but doesn’t translate outside of beating others. (Going from 50% to 73% to 88% chance to get a higher score than you on a maths test ≠ gaining 0 → 1 → 2 units of maths ability over you)</li><li> Epoch Capabilities Index: Analogous to IQ, so [...]</li></ul></li></ul> <p>---</p><p><strong>Outline:</strong></p><p>(03:05) Introduction</p><p>(04:26) Both a poor reflection and a dark glass</p><p>(07:51) Human benchmarking also has a y-axis problem</p><p>(12:27) A metrological elegy</p><p>(12:49) The base case: benchmarks (cf. exams)</p><p>(13:30) Elo et al.</p><p>(16:25) (And maybe not quite 'game ability ≡ winning games', after all?)</p><p>(18:28) ECI (cf. IQ)</p><p>(20:56) Intervals Rarely True</p><p>(25:01) Measure endogeneity</p><p>(31:12) Forking IRT</p><p>(34:45) (Dimensions of being, and beating, a bat)</p><p>(39:40) Prediction (cf. chronometry)</p><p>(41:58) Time horizons</p><p>(44:34) Human "capability" is also exponential in time horizon</p><p>(47:38) Intuitive/interpretative prelude</p><p>(52:56) Time horizons and ECI share an axis kink</p><p>(57:13) Perhaps money, as a measure, stinks the least</p><p>(01:01:07) Finale: AI as normal epistemics</p><p>(01:06:48) Acknowledgements</p> <p><i>The original text contained 33 footnotes which were omitted from this narration.</i> </p><p>---</p> <p><b>First published:</b><br/> August 4th, 2026 </p> <p><b>Source:</b><br/> <a href="https://forum.effectivealtruism.org/posts/CQvdadxjCpd7i7kjA/general-capability-and-capabilities-generally-have-no-good-y?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Source+URL+in+episode+description&utm_campaign=ai_narration" rel="noopener noreferrer" target="_blank">https://forum.effectivealtruism.org/posts/CQvdadxjCpd7i7kjA/general-capability-and-capabilities-generally-have-no-good-y</a> </p> <p>---</p> <p>Narrated by <a href="https://type3.audio/?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Narrated+by+TYPE+III+AUDIO&utm_term=ea_forum&utm_campaign=ai_narration" rel="noopener noreferrer" target="_blank">TYPE III AUDIO</a>.</p> <p>---</p><div style="max-width: 100%";><p><strong>Images from the article:</strong></p><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/29d6c5d2e7c26d886214b917d4922bb31dde2e5b1567092303892a65d2dafcb3/cimlhjqxis26wlxddaxx" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/29d6c5d2e7c26d886214b917d4922bb31dde2e5b1567092303892a65d2dafcb3/cimlhjqxis26wlxddaxx" alt="Scatter plot titled "Epoch Capabilities Index (ECI)" showing AI model scores over time." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/c4760077d5542d3095044c6c7110566a67c3a8427b9be855072b32139facb3e0/vamt6ywqkdpqxijhkxuw" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/c4760077d5542d3095044c6c7110566a67c3a8427b9be855072b32139facb3e0/vamt6ywqkdpqxijhkxuw" alt="Scatter plot titled "Time horizon of software tasks different LLMs can complete 50% of the time"." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/6d264618f2ac3e227b1df46d83adaa572770eb2d2a5987c0a557f806668f04d9/b7aegckzbe4zcmqyheft" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/6d264618f2ac3e227b1df46d83adaa572770eb2d2a5987c0a557f806668f04d9/b7aegckzbe4zcmqyheft" alt="Line graph comparing "Scientific breakthroughs" and "Autopilot" curves against accuracy." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/CQvdadxjCpd7i7kjA/jw6yp72wsmbzntx2heaw" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/CQvdadxjCpd7i7kjA/jw6yp72wsmbzntx2heaw" alt="A line graph showing an S-shaped sigmoid curve on grid." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/29d6c5d2e7c26d886214b917d4922bb31dde2e5b1567092303892a65d2dafcb3/cimlhjqxis26wlxddaxx" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/29d6c5d2e7c26d886214b917d4922bb31dde2e5b1567092303892a65d2dafcb3/cimlhjqxis26wlxddaxx" alt="Scatter plot titled "Epoch Capabilities Index (ECI)" showing AI model scores over time." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/641ffb8f10bbd7ad75b8b9b9319f823cebf17db191460037e03fedcd921a8ab1/juccidt2cwnslgqttfie" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/641ffb8f10bbd7ad75b8b9b9319f823cebf17db191460037e03fedcd921a8ab1/juccidt2cwnslgqttfie" alt="Scatter plot titled "ECI (reverse ranking)" showing upward trend over time." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/571422b8e1aa0e2e75114020f094db5731f6c560f2cc0c2420cbe242e30071f9/xvtvhkzycyhyjobdigev" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/571422b8e1aa0e2e75114020f094db5731f6c560f2cc0c2420cbe242e30071f9/xvtvhkzycyhyjobdigev" alt="Scatter plot titled "-Rank vs Score" with upward trend line." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/8f1ba5a53fb61b685a2559489efedd5659651fb48f89c389a38760b93a76edd6/vtgycwwujfbiwdw05djq" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/8f1ba5a53fb61b685a2559489efedd5659651fb48f89c389a38760b93a76edd6/vtgycwwujfbiwdw05djq" alt="Three line graphs comparing capability versus time with benchmark ranges." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/9fc58e23311945b5719d41ea6d6f2ca1bc43ad35adfda81b6b2e7f3e740a8b1c/btchqzkd13wxcgknl7lj" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/9fc58e23311945b5719d41ea6d6f2ca1bc43ad35adfda81b6b2e7f3e740a8b1c/btchqzkd13wxcgknl7lj" alt="Scatter plot titled "ECI benchmark Discrimination vs. difficulty (unregularized)"." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/e41d787c6b855030a459968ce14535d48b84d3066c2babad0fd4adbb62713773/hajnuudpumbsnlpfhe7e" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/e41d787c6b855030a459968ce14535d48b84d3066c2babad0fd4adbb62713773/hajnuudpumbsnlpfhe7e" alt="Scatter plot titled "ECI benchmark discrimination vs. release date (unregularized)."" style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/e7bf1395ea4503f1c5ed1822a5f8b0628e0aca2a5285b68742218b918f6752f5/eomofdwknlehvpmw2gnh" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/e7bf1395ea4503f1c5ed1822a5f8b0628e0aca2a5285b68742218b918f6752f5/eomofdwknlehvpmw2gnh" alt="Scatter plot titled "Time horizon of software tasks different LLMs can complete 50% of the time"." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/bc84853978d03c8edebd524a32d45ea61af4b526cc16de50c889debdbee47eed/qcuwhltepo7r6asudioe" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/bc84853978d03c8edebd524a32d45ea61af4b526cc16de50c889debdbee47eed/qcuwhltepo7r6asudioe" alt="Scatter plot "Score vs Date" showing increasing scores over time." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/838616bc867575eaffeb3cbbd940b6408f7f873018e5382d9350ed11754795e8/oadhrc1epoyyx8gbu3oz" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/838616bc867575eaffeb3cbbd940b6408f7f873018e5382d9350ed11754795e8/oadhrc1epoyyx8gbu3oz" alt="Scatter plot comparing log(TH) values against average score percentage with trend lines." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/720acb637fd57317078529bb948575707fbd62ecea77d320d16a8adf34249b32/pznuqrbm6skg91e4naqt" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/720acb637fd57317078529bb948575707fbd62ecea77d320d16a8adf34249b32/pznuqrbm6skg91e4naqt" alt="Scatter plot comparing task duration versus average score for two thresholds." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/838616bc867575eaffeb3cbbd940b6408f7f873018e5382d9350ed11754795e8/oadhrc1epoyyx8gbu3oz" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/838616bc867575eaffeb3cbbd940b6408f7f873018e5382d9350ed11754795e8/oadhrc1epoyyx8gbu3oz" alt="Scatter plot comparing log(TH) values against average score percentage with trend lines." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/bc84853978d03c8edebd524a32d45ea61af4b526cc16de50c889debdbee47eed/qcuwhltepo7r6asudioe" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/bc84853978d03c8edebd524a32d45ea61af4b526cc16de50c889debdbee47eed/qcuwhltepo7r6asudioe" alt="Scatter plot "Score vs Date" showing increasing scores over time." style="max-width: 100%;" /></a><hr style="margin-top: 24px; margin-bottom: 24px;" /><a href="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/2c0397b987135961db0fe8f215d215c0f5ca303d4a80e6305e66cd70f6523319/h0hrwoulgvwmkjjtwnnq" target="_blank"><img src="https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/2c0397b987135961db0fe8f215d215c0f5ca303d4a80e6305e66cd70f6523319/h0hrwoulgvwmkjjtwnnq" alt="Line graph titled "Claude Opus 4.6 — Impact of success rates on <15 minute tasks on curve fit."" style="max-width: 100%;" /></a><p><em>Apple Podcasts and Spotify do not show images in the episode description. Try <a href="https://pocketcasts.com/" target="_blank" rel="noreferrer">Pocket Casts</a>, or another podcast app.</em></p></div>




