One Word Changed Every Answer: Ben Wills on Testing LLMs Directly
Ben Wills ran the same hotel-recommendation prompt over and over and changed exactly one word in it: the car he was flying to Los Angeles to pick up. Honda Civic, BMW, Ferrari. The hotels moved with the car, and they moved across every ChatGPT model and every level of thinking he tried. Ben started in SEO in 2001 at KeywordRanking alongside Andy Beal and Garrett French, co-founded the link building company ontolo with Garrett, then disappeared into 13 or 14 years of pure engineering. Large-scale crawlers and scrapers written in C by hand. Embedded firmware for an ESP32 inside high-end audio equipment. He is back in marketing now, building AI search visibility tooling at OppAlerts , and his argument is that LLMs finally hand SEOs the thing search engines never did: a fast, direct, testable feedback loop. In this episode The one-word experiment : hold the prompt constant, swap the car, and watch the hotel tier move with it Why you can test an LLM but never could test a search engine : no waiting for the index, no personalization to strip out The persona is a ranking factor : 100 industries, 10 personas each, 1,100 personas across the July analysis Backlinks, ranking strength and LLM visibility : a decent correlation, with a full page of caveats in front of it How Ben actually uses a correlation : as the anecdotal evidence that tells you which experiment to run Isolating the model : thinking off, fan-out off, then turn each one up a step at a time AI slop as an accountability question : producer standards versus consumer appetite, and whether his own reports qualify Inside OppAlerts : historical PageRank from Common Crawl, news-specific link gap analysis, 20,000+ taxonomy categories The Yandex source leak : a week in a Slack channel with Mike King and Russ Jones, and what the category vectors showed Chapters (00:00) Getting back into the industry after 13 years of engineering (01:32) AI is taking your tasks, not your job (02:58) The one-word experiment: Honda Civic, BMW, Ferrari (05:42) Why you can test an LLM but never could test a search engine (06:39) Twiddlers, patents and the black box (08:22) Ann Smarty, fan-out queries and the return of search principles (10:23) 100 industries and 1,100 personas: the July correlation study (12:45) Correlation is not causation, and Ben's own asterisks (14:29) Isolating the model: thinking levels off, fan-out off (16:25) Engineer brain vs. marketer brain vs. business brain (20:49) AI slop and where the accountability sits (25:03) KeywordRanking, ontolo, pure C crawlers and ESP32 firmware (27:46) news.oppalerts.com and building marketing tools (29:32) Historical PageRank, contact data and campaign personas (31:30) News link gap analysis and 20,000 taxonomy categories (33:36) Vectors, the Yandex leak and CatBoost (37:01) Homework: go build a search engine (38:46) Where to find Ben Wills Notable quotes "One of the most important things for people to understand about the prompts is how one word can completely change the results, the response that you get back." - Ben Wills "The search principles are gonna come right back. You gotta have good quality content. There's gotta be a lot of it distributed across the web. You gotta have backlinks." - Ben Wills "Ann Smarty said ... the fastest way to get into the LLM responses is via SEO. And that's the high...






