Why Your AI Focus Group Keeps Saying Three
You spend years building a product, polish the packaging, nail the pitch… then you hit the terrifying question: is anyone actually going to buy it? We dig into a 2025 research result from PyMC Labs and Colgate-Palmolive that aims straight at that fear with AI market research, synthetic consumers, and large language models that can simulate purchase intent at scale.TL;DR / At A Glancethe core problem with direct Likert ratings and why LLMs collapse to neutral threeshow semantic similarity rating converts free-text responses into numerical scores using embeddings and cosine similaritywhy follow-up AI grading helps but still trails the embedding-based approachwhat 57 real product surveys and 9,300 human responses reveal about accuracy and distribution matchinghow persona prompting reproduces real demographic patterns across age and income constraintswhy zero-shot LLM methods can beat supervised machine learning models trained on the same domainThe shocker is that the first attempt fails badly. When you make models like GPT-4 or Gemini answer a classic Likert scale with a single number, they hedge and pile up on neutral “3” ratings. The fix is not “better AI”, it is better questioning. Google Notebook LM Agents help us unpack semantic similarity rating: let the model respond in natural language, convert that text into embeddings, and map it to five anchor statements using cosine similarity. You get fast, automated scoring without stripping away the model’s reasoning.From there, we pressure-test the method against thousands of real survey responses across dozens of personal care product concepts, then look at whether AI personas actually reflect real constraints like age and income. We also compare the approach with traditional machine learning models such as LightGBM, and dig into an underrated advantage: synthetic consumers can produce richer, more candid qualitative feedback than many human panels.If you care about product testing, consumer insights, or the future of focus groups, listen through and tell us where you’d trust this and where you wouldn’t. Subscribe, share with a colleague, and leave a review with your take: would you let synthetic consumers influence a real launch?Paper: http://arxiv.org/abs/2510.08338Support the show𝗖𝗼𝗻𝘁𝗮𝗰𝘁 my team and I to get business results, not excuses.☎️ https://calendly.com/kierangilmurray/results-not-excuses✉️ kieran@gilmurray.co.uk 🌍 www.KieranGilmurray.com📘 Kieran Gilmurray | LinkedIn🦉 X / Twitter: https://twitter.com/KieranGilmurray📽 YouTube: https://www.youtube.com/@KieranGilmurray📕 Want to learn more about agentic AI then read my new book on Agentic AI and the Future of Work https://tinyurl.com/MyBooksOnAmazonUK
