Find partners
The Analytics Power Hour

The Analytics Power Hour

Hosted by Michael Helbling, Tim Wilson, Moe Kiss, Val Kroll, and Julie Hoyer

BusinessManagementInterviews guests

Episodes

10

Latest episode

Aug 2026

Language

EN

About the show

Attend any conference for any topic and you will hear people saying after that the best and most informative discussions happened in the bar after the show. Read any business magazine and you will find an article saying something along the lines of “Business Analytics is the hottest job category out there, and there is a significant lack of people, process and best practice.”In this case the conference was eMetrics, the bar was….multiple, and the attendees were Michael Helbling, Tim Wilson and Jim Cain (Co-Host Emeritus). After a few pints and a few hours of discussion about the cutting edge of digital analytics, they realized they might have something to contribute back to the community. This podcast is one of those contributions. Each episode is a closed topic and an open forum – the goal is for listeners to enjoy listening to Julie, Val, Michael, Tim, and Moe share their thoughts and experiences and, hopefully, take away something to try at work the next day.

Listen to episodes

10 recent
August 18, 20261 hr 7 min

#304: I Can Haz AI?

Everyone’s posting their AI projects online like proud pet owners sharing videos of their cat doing something marginally impressive — cute, occasionally clever, sometimes a little cringe. And the Analytics Power Hour is no different! In this co-hosts-only episode, Tim, Michael, and Julie skip the thought leadership hot takes and just… compare notes. What have they actually built? What broke? What surprised them? From a custom GPT podcast librarian to a full-blown show production app wired up to Neon, Vercel, Resend, and about five other things Michael is only sort of sure he set up correctly, to a Gemini Gem that simulates a client interaction so realistically it raises your blood pressure in a safe environment — there’s a lot of ground covered. Plus: why AI-generated communication has a Stevia aftertaste, why deploying AI context across a team is way harder than it looks, and why the LLM will absolutely tell you what you want to hear about your Meta spend if you give it half a chance. This episode is also brought to you by Stape , your all-in-one solution for server-side tagging. This episode is brought to you by Prism from Ask-Y —your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show APH Librarian Custom GPT * Hermes Agent Vercel Resend Neon * Surprisingly reliable, but it was something of an exploratory flight of fancy, so we provide no guarantees of uptime. Photo by Eugenia Pan’kiv on Unsplash Episode Transcript 00:00:00 | Announcer: Welcome to the Analytics Power Hour. 00:00:08 | Announcer: Analytics topics covered conversationally and sometimes with explicit language. 00:00:15 | Michael Helbling: Hi everybody, welcome. It’s the Analytics Power Hour, and this is episode 304. You know, there’s no shortage of high-level thought leaders that want to explain this or that about AI, and I mean, we’re not a post. But what about boots on the ground? What are people actually experiencing when they’re using AI in their day-to-day? I don’t know that any of us claim to be AI experts, at least I didn’t check anybody’s LinkedIn profiles, but I don’t think any of us have updated them to that extent. Updating that? No. All right. There goes that claim. But we have spent some pretty significant time in the tools, and we’ve got a pretty good background in this podcast of data and analytics, and I thought it’d be fun to talk amongst ourselves about some of the experiences actually using AI, the good, the bad, then the just okay. So that’s what we’re going to do. We’re going to get into it. And speaking of AI experts, let me introduce my co-host, Tim Wilson, AI Thought Leader. 00:01:15 | Tim Wilson: It’s actually Tim Wilson, M-A-I, like a master of AI. I’m going to start putting that at the end of my pool. I like that. Yeah. I’m like, hey. 00:01:25 | Michael Helbling: Well, good. And Julie Hoyer, are you doing? 00:01:29 | Julie Hoyer: Hello. I’m good. 00:01:31 | Michael Helbling: I’m excited to hear some of these war stories. Do you get introduced to clients as an expert in AI? 00:01:36 | Julie Hoyer: Oh, thank God I do not. 00:01:38 | Michael Helbling: Okay. Is that an AI strategist? 00:01:40 | Julie Hoyer: Nope, not yet. 00:01:42 | Michael Helbling: In consulting firms, a lot of times as an expert, you sort of just get slung in as an expert in whatever the topic is about. 00:01:48 | Tim Wilson: And it’s like, yeah, that’s actually inaccurate. 00:01:52 | Michael Helbling: Let’s kick off the whole conversation by just chopping that tree down. 00:01:56 | Tim Wilson: All right. NPR or the Daily Show and our AI correspondent. That’s right. 00:02:00 | Michael Helbling: And I’m Michael Helbling. All right. So let’s dive into it. I think maybe a good kickoff is just set the tables, so to speak, because maybe let’s just talk a little bit about the types of AI tools we’ve used, what LLMs, just very high level, very quickly, maybe just go around. We’ll just chat about that. And then I’ll dive into more specifics. Yeah. So why don’t you go first? Well, I’m glad you asked him. 00:02:31 | Julie Hoyer: Flip those tables. 00:02:32 | Michael Helbling: So yeah, I’ve been using AI since whenever ChatGPT3 came out. But yeah, I would say obviously using a ton of Chat AI LLMs. So that’s GPT, Gemini, Anthropics, Claude. So a fair amount of coding on Codex and Claude. Have experiment with some other stuff too. And then on the agentic AI side of things, I have a little Hermes doing little things for me using GLM 5.2 currently. So that’s sort of the latest and greatest. So yeah, I publish code to GitHub. I do things like that. And by me, AI does that, which is, it’s true. I think my first GitHub commit ever happened after I started using AI. So yeah, it’s definitely it opened the door. It opened the door to a set of possibilities I had hitherto lain dormant. All right. Okay, Julie, what about you? 00:03:41 | Tim Wilson: Me. 00:03:42 | Julie Hoyer: Okay. My list is not as impressive as yours, Michael. But similarly, I’ve used Chat GPT more for personal stuff since it came out. Like I was interested, of course, like anybody, but more so through work and more of the stuff that we’ll probably end up discussing today. I’ve done through Gemini and then Claude. So those are kind of my two main ones. Again, the easy foot in the door, I think the place everybody starts. That’s where I’m at. 00:04:12 | Tim Wilson: Thanks. No, that’s good. Tim. So, I mean, so I platform wise, I’m kind of across the, all through the big three, the Claude, Chat GPT and Gemini. I’ve gravitated much, I would say I’ve gone deeper in the Claude stack and is so many people out there saying people who are just using it for chat, you know, they’re, they’re missing the boat. And so I’ve kind of pushed myself, I feel like in the Claude world, I’ve actually used Claude code beyond just using chat to help me write code. I’ve gotten into Claude code a bit. I’ve done some stuff with co-work, which has been, which is the stuff I’ve done with co-work has been mostly podcast related stuff. So that’s a little, a little meta, but I feel like I’m, I’m hitting a point where I’m starting to sort of understand the distinctions between those and on the Gemini front, that’s kind of like, I’ve been underwhelmed by it, but outside of chat, I think I’ve played with Gems, which I remember when Julie, you, I think last called Gems multiple years ago, you’re like, Gems are cool. And I think it was another like year or two before I actually, I was like, dealt with one. I wrote down like, what are Gems? 00:05:31 | Julie Hoyer: Gems? 00:05:32 | Tim Wilson: But I mean, just a quick one, which isn’t even really in my list, but speaking of Gems, Stern at marketing analytics summit talked about, talked about AI, using AI for like training junior analysts and how to like engage with, with stakeholders. And I was, I, at the time I had a reaction that was like, but the humanity, how do you learn to do this? And since then I’ve had the opportunity to work with a couple of Gems that are purely there to simulate client interactions. And it’s been, even though I can read the instructions and I know what’s going on, like the realistic level, even to the point of punching record, like, so I’m like, Gems Stern, I know you regularly listen to this, I’ve now like done that. And I am converted that, wow, that’s a hell of a tiny little application. But it, it, it’s really impressed me. 00:06:32 | Julie Hoyer: Yeah, that situation made me sweat. The one that Tim, you like took the base one from, I can’t remember who made the original, was it Ryan Dupont? Ryan Dupont spun up the original one at further, and then you took it and tweaked it and made like that second iteration. But when you sent it to us to try out, I was like, whoa, I’m feeling like the pressure respond. 00:07:02 | Tim Wilson: It was good. That was like a wild little light bulb to say, oh, this is like really raising stress level, but in a safe environment. And the way I was like, ooh, I’m on board. So that was a very. Eye-opening one. But sorry, I went to a specific example that wasn’t even on my list. 00:07:19 | Michael Helbling: But yeah, Tim, save it for the next step of the process. 00:07:24 | Tim Wilson: It’s not in the outline. 00:07:26 | Michael Helbling: That’s right. No, well, that is what we’re getting into next is sort of like, OK, so let’s talk a little bit about projects we’ve done. And you kind of mentioned, Tim, the podcast, and that has been pretty fertile ground for a couple of us to try, you know, what I feel like is a low stakes way to try stuff out because it’s sort of like, OK, it’s not my business. It’s not a client. It’s a podcast and we’ve got real work that we’re doing. But like if it totally fails, like we can just throw it away and nobody has to know about it. But some stuff has actually stuck around. Like, and I think that’s kind of interesting and maybe informative in a way of like, yeah, if you’re not started yet on specific things, could you find something even sort of outside of your work that still would be a good use? 00:08:15 | Tim Wilson: Well, can we talk because this was over a year ago that that was when I headed down the path. I was using ChatGPT to help me write Python, Python’s not my strong suit, but I needed it because I was basically trying to save us some money and shorten some time frames on the transcription. So it started with like one Python script that ultimately got pushed into Colab, but it was purely like, help me write a script. I’m taking the code you’re giving me, putting it in VS code, going back and forth, kind of following along with what it’s doing, which is kind of a chat based version. So that was like, I don’t know, that was vibe coding because I was generating code that is still Python code that we run in Colab and we have three or four of those for different reasons, different purposes, but you did a different form of vibe coding for the podcast when it comes to actually developing an app. Like is it worth having you talk about that? 00:09:14 | Michael Helbling: Yeah. I mean, well, there’s a couple of things we could talk about, one of which our listeners can actually access, which is the very first, this is the very first coding, one of the very first coding things I ever did with AI, which was back at the end of 2025, which is crazy because that feels like such a long time ago, but it’s really not that long ago. It’s like so long ago, it was like a year ago, no, it was eight months ago. But no, we had all the transcripts from the podcast over the years, and I just started thinking about the fact that’s like, oh, well, AI is really good at reading through docs and stuff like that. 00:10:00 | Tim Wilson: I was like, wouldn’t it be cool to create a little AI library for the podcast using 00:10:07 | Michael Helbling: all the transcripts over the years? And so that got the idea started. And then I did a company hackathon with stacked analytics. We all went out and did AI hack work for a couple of days. And so I picked one of that as one of my projects and built a little custom GPT that basically the work or the underlying stuff behind it was sort of giving me a chance to learn about concepts like RAG and VectorDB and stuff like that, which are sort of organizing concepts that AI kind of access their chunks up data, probably not as relevant now the way that context windows have expanded and things like that. But for that time, it was kind of useful to figure out, OK, how do I get all of the transcripts in one place? So in this case, a Google Cloud storage bucket, and then how do I move that into a VectorDB? So I use Cloudflare to do that. And then how do I move that back across into a JSON schema that a custom GPT can talk to? And so building all that out and building out the flow of that was really the hard work. But it was kind of very useful because any time you’re dealing with permissions in Google Cloud, if you’re not super familiar with that platform and setting those up, that’s a good thing to learn about. You definitely want to know how that stuff works. And it was good. It was a really good experience because it took a long time to figure out, but it actually 00:11:40 | Tim Wilson: then produced something that’s like, hey, that’s actually pretty useful. 00:11:43 | Michael Helbling: And actually, if you go to ChatGPT and you go look at GPTs, you can find it as the Analytics Power Hour Librarian, and it’s available right now. So you can actually go find it and then ask it questions about the podcast, which is really circular because this right now will be recorded, transcribed, and then available at some point in that custom GPT. Weird. Tim, your company launches a new checkout and conversions suddenly drop. 00:12:15 | Tim Wilson: I mean, what’s your first move? Open 17 tabs, blame Safari, and schedule a meeting called Tracking Investigation, Urgent, or try 00:12:25 | Michael Helbling: this one on, connect the AI Assistant you already used to your Stape account through the Stape MCP server. 00:12:32 | Tim Wilson: That sounds less traditional. 00:12:34 | Michael Helbling: Yeah, but you could ask it to check the server side container, validate the tracking domain, compare usage before and after launch, break down activity by browser or client. 00:12:45 | Tim Wilson: So it can help answer, did customers stop buying or did our measurement just stop measuring? 00:12:51 | Michael Helbling: Exactly. And you can also review daily traffic, see which domains are generating requests and check for unexpected usage or possible surcharges. 00:12:59 | Tim Wilson: Fewer tabs, faster answers, and maybe that urgent meeting becomes an email. Well, let’s not get carried away. Fair. Click the link in the show description to learn more about the new Stape MCP server. Michael, what if your dashboard could do more than just sit there looking colorful and ask everyone to interpret it? Excuse me. That dashboard took six months, 14 meetings, and at least one resignation. Oh boy. Well, Prism’s new app builder turns an idea into a complete interactive application right inside your workspace. 00:13:38 | Michael Helbling: Okay, so I just describe what I need and Prism builds the app? 00:13:42 | Tim Wilson: Yep. You can review the spec, you can edit it, and keep refining the same app in plain English. 00:13:48 | Michael Helbling: Oh, so no rebuilding everything because someone asked for one tiny change is somehow requires a new data model? 00:13:56 | Tim Wilson: Exactly. You can also share the app with anyone through a link. 00:14:00 | Michael Helbling: Oh, useful, flexible, and no 12-week BI backlog. Frankly, it’s a little disrespectful to the traditional process. 00:14:10 | Tim Wilson: Well, Ask-Y is looking for people who want to replace a BI dashboard with an application without handling the migration themselves. 00:14:17 | Michael Helbling: Oh, so point the agent at your dashboard, Ask-Y reads the logical layer and builds it as a flexible, extensible application. 00:14:26 | Tim Wilson: That’s it. Sign up at ask-y.ai, that’s ask-dash-the-letter-y-dot-ai. Use code APH to jump to the top of that wait list. Boy, I like the sound of this. Give your dashboard a promotion. It’s been through enough. Which is, brings up, because like you built it, you kind of hacked it together, you’re like, hey, guys, I built this, and then like, and we were playing around with it, like, this is so cool. And then the immediate question was like, well, is that going to, is it going to stay updated? Because it’s like, do we want a snapshot? And that then sent us down a path, which I think is also representative because then the work I was doing on the transcription and needing to get, I guess I already had, I’d already done some stuff where it was going into markdown files with historical stuff. But then it wound up needing to move into a different process that was a combination of co-work and Python scripts to say, oh, we’ve got to coordinate between these two things. Like, I mean, literally just today, I just processed an episode and part of what it does is it, there’s a, there’s a handoff like the end of this one process is going to drop an additional markdown file in that folder. And then the GPT, that process you built is going to be looking and picking it up. And you were actually surprised a couple of months ago, you’re like, I don’t even think that’s working. Yeah. And then I was like, what’s the most recent episode? And then it get pulled up and like, it’s working. 00:15:56 | Michael Helbling: It’s going. Which is great. Because I didn’t build the next thing on top of that, which is like monitoring and data freshness checks in my GCP storage, which you would do if you were building something for a data purpose. And have built for other things. 00:16:11 | Tim Wilson: So yeah, that’s a nice thing about, yeah, podcast related things. You can just sort of work on it when you want to and then not update it later so that Tim 00:16:24 | Michael Helbling: gets really frustrated. 00:16:26 | Julie Hoyer: Yeah. 00:16:27 | Tim Wilson: There you go. 00:16:28 | Julie Hoyer: Speaking of the permissions part that you were running into, Michael. So one of the things I most recently tried to do, honestly, this was inspired from our conversation with Rob Colley, actually, recently, where he was saying where the data builders and where the people that are going to end up going in, again, using AI for more than just the chat and doing more than just analysis in it. And so I was trying to get into more of like, could I go use cloud code? Could I build an interface, an app, something that refreshes? And so from a lot of what you guys were just saying, I was inspired to go and try it for this learning community that I run at work every Friday. It’s called The Guild. And we have 183 recorded episodes or at least sessions that have happened that we have like a drive folder for Tim, which I was like, I hope Tim is proud of that because Tim was working with me when we were running that together. So anyways, I decided first I was talking to Claude just chatting and asking, you know, this is my situation. This is what I want to try to do. Can you help outline some of my options so that I could take all of these recorded videos? And there’s like content and decks in there. It all lives in a spreadsheet that we maintain and keep up to date, but also then it links out to all the drive folders where all this content is. And it gave me a couple options and I wanted it to be for searchability and like discoverability. So not only somebody to say like, oh, are there sessions on this topic or these key words, but could it also surface some things? Maybe they didn’t think about that could be relevant. So I was really excited about that. But as soon as I went to try to start like building it, it kept like throwing this error. Suddenly I was like giving it a link to the spreadsheet. And it’s like, oh, I can’t build off of this because I can’t publish because the account owner is different than the spreadsheet owner. And like this weird stuff, I’m like, what are you talking about? So then I am asking it, where do I find this out? Who owns the account? It’s my account. I don’t understand. So anyways, I find out that it’s the leader of IT from our parent company is the name on the true account. And for some reason wanting to publish this up to date like app is different than all the other ways I’ve been using Claude and giving it drive links. So that’s been like my big stop gap. And then I was even asking it like, is it appropriate to ask your IT leader to give ownership of the account to me so that I can publish this? And I was like spiraling. 00:19:06 | Tim Wilson: Help me draft an email. Help me. I’m at that point. I’ll draft an email for you. 00:19:11 | Julie Hoyer: Yeah, but I’m really hoping I can do it because to your point with the librarian, I’m like, we have so much content here that people never go and look at, but there’s so much applicable stuff in there that I’m pointing people to it constantly. Like, oh, there was this one time we talked about it. Here you go. 00:19:27 | Tim Wilson: But you said early on, like, which I think I’ve now learned, which is a little bit of an adjustment for me. I’m used to like, oh, I know, I know how whatever this thing is works. So I sort of have the architecture and here’s how I’m going to plug in the pieces. And because I’m still learning, I’m finding myself often saying, this is the end result and pushing myself to like articulate the outcome that I want. And then I will sit there and say, now, I think I could approach it this way or that way. And, you know, you tell me, like, actually, I had a case where I think it’s pretty known out there, like fax and feelings is kind of drifting away. And so Squarespace was going to charge us a big old fee to re-up in a bit. And I was like, oh, well, we’ve already stripped the site down to just be like a splash page. So couldn’t I just it’s like, it’s just a simple page. Can I just have, you know, Claude helped me recreate it as a static HTML. I’ll put it on Netlify. It won’t cost me a thing, whatever. And so I said, hey, can I do this? Want to walk through it? What are the cases? And then it brought up a whole bunch of really good points about managing the timing on the DNS cutover that I hadn’t thought about, which then sort of gave me like a much better plan before I kicked off to do it. But then I also learned like having it vibe code, trying to look at screen captures and inspect an existing page built on Squarespace, which has all sorts of added bloat on it. It was I still want to be doing a lot of inspection and CSS tweaking. Although even in the middle of that, I kicked Claude, I kicked co-work off to go do some like dom scraping to to suss out some some settings for me. So that whole like, don’t just assume what you’d know, like the chat ones are really nice to say, what might be the options? And can I use, should I use, you know, Claude code for this? Or, you know, should I use co-work? And at times it’s come back and said, well, yes or no. And given the pros and cons. But when we talked to Rob Colley, a big thing he was pushing was like you, you should go and build apps. And it’s kind of funny being on the podcast with Michael, because I think we all would acknowledge that Michael, you’re kind of tend to be a couple of steps ahead of us, generally, when it comes to what’s the new things that are doing. So you when Rob was talking about that on that episode, I was like, oh, yeah, that’s what that’s what Michael did with, like, literally building an app for us to do our back end workflow for the show. Yeah. 00:22:16 | Michael Helbling: So what was interesting about that was obviously, like running a podcast, there’s all this stuff in the background that we do, which we don’t need to get into. It’s not that fascinating. 00:22:25 | Tim Wilson: No, let’s talk about it. 00:22:25 | Michael Helbling: I don’t want to document this as a very detailed process document. 00:22:31 | Tim Wilson: I wasn’t going to go here all night. It’s right. It’s all been transitioned into runbooks and co-work. Yeah. 00:22:37 | Michael Helbling: But but the thing was, is like the idea was, how do we both expand the capability for everybody to be part of that process, as well as take some of the stuff off of Tim’s plate? I don’t know that we’ve been effective at accomplishing either of those two goals. But the idea was to start a start a process and also like experiment with AI as a result of that. But a couple of really interesting learnings from that. One was I spent a lot of time up front writing documentation of like what I thought this app should be able to do. And actually was pretty unhappy with my design, if you will, or the architecture the first go around. So I was using chat GPT to kind of like kind of help me write those and like think it through. And it was just AI tends to do this thing where it kind of latches on to ideas and it’s real hard to get it to let go of them. And then it sort of gets all in the rest of the stuff and it gets, you know, it’s sort of like a what’s that thing where like when you’re cooking something, it changes state. So like an egg is cooked, you can’t uncook the egg, right? It’s so it’s sort of like that feels like that with AI sometimes. Like something gets in there and you can’t uncook the egg. It’s now like a different form and it’s just really in there with your ideas. And you just have to like tear it up and start over. So I was at that point and so then I switched over to Claude because I was starting to use that a little more and I was like, all right, this is a good first step with Claude. And I used us a tool to design it and using the design of it to do that. But the other thing I thought was the kind of interesting was it actually took me back into a learning process about specific things because one of the pieces of the app is an integration with our Slack our Slack channel in the podcast. And I realized like as I was thinking through like how this would work, I really didn’t know what was possible with Slack integrations really. Like I’ve seen some but not a lot and I’ve not done any development behind it. So like it was sort of a blind spot about what here’s what I want. But like I’m not sure how how much this is or is not should interact. And so working with the AI to kind of like even educate myself a little more about that was actually part of the process. So it’s sort of like you hand stuff off to AI to go do. But actually you’re getting educated a little bit along the way on this as well because while you kind of know like what you’re trying to get to, it’s sort of like, well, do I even know what’s possible? And that’s actually pretty critical learning because then that shows you like, oh, if I don’t know what’s possible, like I’m not really the AI is not really extending my capabilities as far as they might be able to go if there was a more experienced hand. And this comes out, I think, when you do other kinds of work with AI. I was chatting with a CEO of a company a week or two ago and they’re like, yeah, I really love using AI and it’s great. But I feel like it’s not really ever helping me to like innovate that next step. Like it’s not helping me kind of like go that one step further. And I was like, oh, yeah, I think I get what you mean. So AI is it’s really it was a great experience. The app works really well. It actually looks pretty good too. 00:26:04 | Tim Wilson: Can I ask about because it’s it’s a it’s a little bit of a mystery to me. Like I get the the coding part, but like ultimately, this is an app that is hosted on Vercel. Like what is the actual and you’ve done a few iterations because we have like sort of a feature request and bug tracker. And so I guess two questions. One, the actual I get when you have code, I’ve got it when I’m in our studio or I’m in VS code, or I’m just asking for it to generate and that’s code. But when you have what is it, what are you actually doing to get the code run and deployed somewhere? That’s question one. Question two is when you’ve made updates, is it discipline that it’s touching specific sections of the code or is it kind of going we in like regenerating from scratch? And this is like reading Ben Stansel a year or two ago and the way it was being described. So that’s a two-parter question. 00:27:09 | Michael Helbling: So I don’t understand the whole thing I learned at the hackathon. Well, and before that too, but the hackathon we did, which had better developers than me at it, was I learned more about how to use GitHub. And so I use GitHub pretty heavily. So basically when I do a coding project, basically my process is, and hey, listen, I’m not claiming anything here. So like this is totally whack and broken. Well, you let us know in the comments, but this is what I do. I start a local folder. This is where the code is going to live on my computer. And then as it starts to develop into something, I then go create a GitHub repository. And then that code starts to live on that repository. And so basically for small projects like this, it starts locally and then moves to GitHub. And then GitHub becomes the versioning and the control. 00:27:58 | Tim Wilson: And then from there, OK, I’m just going to put it, put it, put it a little, let’s just call out that Michael did not explain how Git works well on that. 00:28:06 | Michael Helbling: I don’t know how Git works very well. I just want to put in the qualifier. Yeah, no, I already qualified it. I was like, I don’t really know. But he didn’t say it might be whack. Yeah, exactly. He said it might be whack. 00:28:20 | Tim Wilson: And what I mean by that is it’s probably whack. But that’s the code. I mean, I published from GitHub. Yeah, two GitHub pages for like web. 00:28:29 | Michael Helbling: But how do you then when you when you start up? So then basically, there’s sort of the DevOps piece of it, which is how do you take code that lives in GitHub and is maintained there in a repository and make it an app out on the web or wherever. So that’s the DevOps piece, which is where you use something like we in this case, use Vercel and Vercel basically can go connect to your GitHub, get the code, deploy it. And basically any time you change GitHub, it automatically deploys to Vercel. And so you have an automatic deployment process. And so the Vercel interface stores all of your environment variables and all the stuff that makes all this work and all the API calls that go in and out. And so this app basically connects to resend for email. 00:29:15 | Tim Wilson: It is the database living and where’s the actual database 00:29:21 | Michael Helbling: where the database is neon, which is the got bought by Databricks. So yeah, that lives out there. And so basically all this stuff is sort of interconnected already. You basically when you write all this code, you’re basically connecting all that stuff via API keys and stuff like that. And then the app knows when to call. 00:29:40 | Tim Wilson: So Claude told you, OK, go make a Vercel account, enter these. Here’s how you can go make a neon account. 00:29:46 | Michael Helbling: And yeah, basically, I mean, you can say to it, like, I want to use this versus that, you know, so it might come to you and be like, well, this is an architectural decision. Do you want to use SuperBase? Do you want to use Neon? Do you want to use Convex? Like you can kind of say which one you want to use or you have one you need to use to say that. So it’s not like you can’t. You can just use whichever one. So like there’s either other versions like Vercel is one example. But I think there’s many others that do the same thing like rep lead. I think probably does the same things. So yeah, since I know absolutely nothing, I just went with whatever Claude said. So that’s where it lives. 00:30:22 | Tim Wilson: Well, so I mean, so that that’s helpful because I yeah, I’ve got things that are published to GitHub pages and to notify. But I’m generally it’s just like static stuff. Same thing if I if I do a new push to GitHub, GitHub is going to it’s going to detect that and it’ll redeploy and kind of refresh it. But I was like, but that’s always been like, that’s just rendering Korto or our markdown or Hugo. But so it’s but that’s really helpful because I was like, OK, but you’re extending it saying, well, there might be two or three different systems that all have to kind of play together. 00:30:55 | Michael Helbling: Yeah, in the case of that app on the back end, there’s about five or six systems or API calls that go out. So like we have API calls out to different LLMs to process things within the app. It reads, you know, we have an incoming email and I look through those to evaluate, like, is this a good pitch or is this not a good pitch, you know, those kinds of things for us. So those are all things that are happening in the app. And then the email processing itself goes through a company called Resend and, you know, so on and so forth. So there’s a bunch of those little things. But basically, all of those decisions get made back at the beginning when you’re defining what is this thing is going to do and how does that work? It’s like, well, here’s what happens. We get topics from the website. We get topics from emails. We get topics from our Slack channel. Now we need to bring those all together in one place so we can evaluate those topics and and make decisions about them. 00:31:49 | Julie Hoyer: So what you’re saying is my earlier app library for this guild is laughable because I have a lot of steps I’m going to have to still go figure out. Well, I wasn’t even thinking like, oh, I’m going to have to probably connect it to GitHub and do this. But like, I don’t know, the example you’re saying is giving me thought of like, I have quite a few pieces to put together in this puzzle. 00:32:12 | Tim Wilson: There’s a nice little nice little fable anecdote that you have when like you asked didn’t rather than it was a fable. What is it? Yeah, fable that you asked to say, hey, could you evaluate this? 00:32:22 | Michael Helbling: Is this kind of pretty low level authentication path? Yeah, when fable five came first came out, I had to look at the code base for that app. Not with anything. I was just like, I’m thinking of making some changes. Could you evaluate this code base and give me your thoughts? And so fable is pretty well known as being like a pretty high end AI or like pretty state of the art right now. And it immediately identified three security flaws in that app, which we then patched immediately. So it was like, oh, yeah, well, before we do anything, here’s three things that are broken about your app. I was like, well, your little brother is the one that coded it. 00:33:01 | Tim Wilson: So we escalated that because this is a podcast. You can bring the podcast production process down. 00:33:10 | Michael Helbling: Bring it to a screeching halt. I don’t know that was actually what was the problem. But there were like a couple of things that definitely should have been fixed and we did fix them. But but you know, that’s that’s sort of an emerging thing. And also one of the things like, you know, I’m over here doing experimentation with things like Hermes, which is an agentic AI that kind of is like doing ongoing AI tasks for me on different things. And like this is actually an area of big concern, which is the security piece of that. So like setting that up is I’m super cautious about that because like that doesn’t live on my local network. It lives far away on an external box that is not attached to me at all. It cannot access certain things. It cannot write things in certain places. 00:33:56 | Tim Wilson: So basically I keep it pretty pinned down for now 00:34:02 | Michael Helbling: because I just don’t as I learn, I don’t want to be exposing like my work and all these things to an agent that may then just go dump all that information on the first, you know, random hacker that gets the opportunity to kind of prompt inject it with something that it thinks is cool. So like there’s some real risks out there with AI, too, which you know, I’m trying to be cautious of, but still also experiment around and have a lot of fun. 00:34:30 | Tim Wilson: Yeah, I mean, it’s what it’s the luxury like it is kind of nice sandbox. We’re not having a crunch a bunch of data, but there is a lot because what I tackled was I said, oh, we have this lengthy 40 page document of our process. That should be something I have a background as a technical writer. I’ve kept it maintained, throw it in and let co-work run with it to do everything because we record a little bit of stuff happens. Things go off to Tony to do the editing. He sends back like a final file. And that’s like the starting point for a whole bunch of stuff that has to happen. And that’s documented in the steps. And we had several of us who were like, we can do this process. I do it most of the time, but we had a couple of other people with fallback. It’s a Google doc knew how to do it, threw it into co-work. And a couple of things have emerged like one, how many steps there are that the human in the loop, partly because, oh, you got to go click somewhere that it’s just not going to go click on its own more. So I need to check in and have you, you know, pick which tags we’re going to put on the show. I’ll recommend these or these the right ones. Or these are three options for the show bar because frankly, I don’t think the show we’ve written enough by hand. We have enough training. We can now use it to write that copy. And it is now a the two things that have kind of emerged. One, what used to be an hour, an hour and a half of focused time that was not enjoyable at all and subject to to me making mistakes by not completely following the process. Now it’s like an eight hour eight hour duration because it’s super slow cranking through some stuff. And I just have to wait to hear the little beep and then go in and check, you know, what I need to rely on. And for some reason, I’ve never been able to get caught dispatch to work because I was like, this is the perfect application of I’m going to go meet someone for coffee. This thing’s running. If it just needs me to say go, never could get that. I have not gotten that to work on my phone. So one that was interesting to say, yeah, this is kind of a grind of work to do, but it’s stuff that a human has to do. So that was kind of validating. The other is it’s wild to watch it take different paths to do the same task each time. Like there are times where it’s like, well, that didn’t work. So I’m going to try this other thing. Well, that didn’t work. And I watch it be super inefficient as it finds its way to do the thing. So it still has that it has that probabilistic LLM thing going that it’s not a rules based. It’s more robust. So if a UI slightly changes, it’s probably going to figure it out. But on the other hand, where exactly it sort of runs into problems is kind of a moving target each time. So you could do Tim, as you could write a skill for it. 00:37:40 | Michael Helbling: So you could give it. That’s good. Yeah, you can give it the steps. 00:37:43 | Tim Wilson: Well, but I’ve given it the steps in code work. Like it has a run book that is like, do this, then do this. I mean, yeah, but that’s very detailed. 00:37:53 | Michael Helbling: That’s different than sort of like process steps. I don’t know, maybe not. Maybe it’s not a good application. I don’t know. 00:37:58 | Tim Wilson: I’m just actually I should probably go to Claude chat and say, I’ve done this and this is the challenge I’m running into. And how could this be? But is this something skills would help with? I love doing that when I don’t understand something like, hey, would skills work for this? And it’ll come back and be like, yeah, you ding dong. Like, why weren’t you why weren’t you using skills from the get go? That’s a good skill segue because you’ve been playing around with skills. 00:38:27 | Michael Helbling: Yeah, I yeah. Well, because yeah, because in a couple of cases, our clients are using AI and so we’re helping them do that. And one of the things I noticed when AI when LLM’s work with data, they tend to sort of fall victim to some very common pitfalls when doing analysis, like certain types of analyses. I think they do different things and it’s sort of challenging to kind of get them to not do that. Like one of the things I noticed right away was like comparing two things to try to evaluate what was different between these two things. It would sort of come up with its own idea, like sometimes they would just make it up and then it would just sort of hang on to the ideas if it was like totally rock solid, that’s the answer. And I would go back and be like, no, that’s not the answer. That’s something you made up. And we got to throw that away or just hold that to the side for a minute. Maybe it’s the answer or maybe it’s not. We don’t have enough data to say that’s the answer. And of course, you know how AI is. It’s like, oh, yeah, you’re absolutely right. Like, I’m definitely not going to do that again. And then I would just go do it again, you know, and it would just do it. And so I was like, OK, I’m out. So that got me thinking like, well, what if I gave it a better set of instructions? And so I started building out a set of skills for different types of analysis to sort of say, OK, if I was sitting down to do this analysis, how would I break this problem down? There is a bunch of great literature and technique and methods all published like there’s a ton of material that it could be drawing on and it just isn’t doing it. And the other thing I noticed is that all of the data skills that are out there, a lot of them have to do with like accessing tools or making a PowerPoint presentation or making a visualization. It’s like, OK, that’s all very handy. I’m talking about like just doing the analysis of the data. So I’m working through a process. We’re still very early stages where we kind of I built some skills for that. We’re doing some testing. What I’m learning in that process is two things. One is using those analysis skills is producing a better result on average than just the raw LLM. And that’s pretty cool and very awesome. But doing evals on skills to prove that they’re working the way you want them to work is a ton of work. It’s like so much work. So like it and you have to be super structured about it. And you have to be like you have to make sure like the prompts are set up correctly and everything is done the right way. So like you can actually evaluate like the outputs and know that it’s like got it right. So like it was still a long way to go, but I’m hopeful that over time 00:41:16 | Tim Wilson: we’ll actually develop something that’s going to be at least convenient, 00:41:20 | Michael Helbling: if not useful in some cases. I think the challenge there is that at the end of the day, I don’t think an LLM can actually produce the quality of analysis or meaningful analysis that I would then go and say, like this is what you should do instead of having humans do it. I just don’t think you can replace it. But I know that humans are going into AIs every single day and asking it to analyze data. And in light of that, maybe we should try to build something that makes it less bad. So in the old George Box framework, you know, all models are broken, but some are useful. That’s sort of what we’re shooting for here. 00:42:02 | Tim Wilson: All models are wrong. All models are wrong. Yeah, there you go. The Mechershop just got triggered because he’s like whatever it is, you’re not using it right. Yeah, yeah. Well, I’m trying to use it wrong both ways. That was on purpose. 00:42:15 | Julie Hoyer: Wait, can you share like what type of analysis you’re talking about that like one of the ones you’re working on a skill for? I’m just curious. 00:42:23 | Michael Helbling: You cannot use EDA. Sure. So root cause analysis or like a or a prescriptive analysis or so like different types of analysis. So like it was sort of a list or a library of like a bunch of them. And yeah, some of them are ones that Tim is not very happy that I’m doing. 00:42:44 | Tim Wilson: Metric, metric reconciliation, metric reconciliation, question, 00:42:48 | Michael Helbling: refinement, yeah, causal, you know, inference or impact. So like all the ones that like in a business context, there’s a better level. But most of the time people don’t put the work in to do. It’d be nice if your LLM was like, hey, here’s what the data is saying. But I think the next step here is actually some kind of holdout test to really prove this. It’s instead of just knowing that threshold. Yeah, instead of just telling you what you want to hear, which is what the LLM would do without the instruction because it picks up on things like the way you ask the question. Like if you’re asking a question with an idea that like, hey, can we stop spending Facebook spending money on meta this month? It’s going to pick up from that question that like you’d like to stop spending money on meta this month. And that was going to do the analysis and come back to you with something that makes you happy, whether that’s good for the business or not, because there’s underlying that is this instruction set that the LLM is doing on its own, which has nothing to do with your analysis and more about how it wants to make you happy. And that’s sort of like what we’re fighting against a little bit because when you do analysis, you’re not trying to make someone happy. You’re trying to get to a closest version to reality that the data will support. So it’s it’s a little bit different exercise. And so it takes the LLM instead of telling it like now behave. It says, hey, here’s the literature. Here’s the methods. Go through these steps and use this as a floor, as opposed to like how you’re doing it the other way. 00:44:25 | Tim Wilson: When I feel like if like, oh, you’re asking to do a root cause analysis, the skill of the root cause analysis, say, I’m going to go do a quick exploratory data analysis of this before, like given the time and the capacity, you should always check that the data doesn’t have nonsensical stuff in it. Where it’s super easy, super common for somebody to say, we’ve been using this data forever or somebody handed it to me and told me it was clean. And so they just go straight to, you know, running a regression. Whereas some of that and I could also see it qualifying, requiring some qualification of the, you know, from the user. But that’s that’s been my hobby horse that I think is valuable to they ask a question and they get a very polite but reason set of focus and focusing and clarifying questions back. 00:45:17 | Michael Helbling: Yeah. And not always sort of like, like an easy answer. 00:45:22 | Tim Wilson: But like, OK, well, there’s more work to do to actually get the answer you want. And it’s like, OK, well, no one’s going to accept that. 00:45:30 | Michael Helbling: So we’ve got to tune it more. 00:45:33 | Julie Hoyer: Funny enough, I not on the list of types of analyses you mentioned, but I guess I would I would count it as a type of analysis. I had pulled together a proposal for a client and long story short, there’s a bit of like a pause. And so we wanted to send over the proposal for our direct stakeholder and her boss to understand like, hey, when things change on your end 00:46:01 | Tim Wilson: and you can revisit this, like here’s here’s what we’re proposing, 00:46:06 | Julie Hoyer: valuable to outline that story. And we wanted to close it out with like an impactful value slide. Right. So me as the strategist on the account is in charge of that. So I mean, we’ve done the exercise manually. I have people at the company I was talking to Matt Padone about it. You know, ways that he’s gone about pulling something like this together. And so I didn’t have a lot of time. And I was like, you know what, I’m going to try to use Claude to help me here because really what I needed to grab and make some assumptions about what this program could help drive for the business for, you know, leadership that we were trying to talk to was like public financials. Right. And like, that’s a slog to go through on your own. So I’m like, oh, I’m going to give it an example of information we’ve pulled before when we try to pull together these like value estimates. And I pretty much gave it context to say, here is who we are working with. Here’s the situation. Here’s what we’re proposing. This is the area of the business that we would be working with. This is the impact we believe that we could drive for them through this work we’re proposing. And it was maybe a long paragraph, like a couple of run on sentences, nothing crazy, gave it one link. And I asked it to help pull public financials to help me make some assumptions and give a value estimate. And it spit out a six tab spreadsheet, which I was a little like, wow, overkill. That’s what I’m also finding. I’ve asked it for like one slide before gives me like seven. I’m like, whoa, all right. But it did a pretty good job. I mean, I was able to go and find to like where it was pulling numbers from. It did automatically give me pretty descriptive, like this is where this number is from. It highlighted cells that it was making assumptions on. It gave me like industry benchmarks that I didn’t actually ask for those explicitly, along with the client’s actual published numbers. And then I was able to work through it once I understood the spreadsheet set up, though, and play with how I wanted to craft the narrative. But it was interesting because I now know the next time I go to do this, I would prompt it in much more detail because, again, I have a better idea of like how I want the spreadsheet to function and the things I want to be able to like input. So I do think if I get more explicit or even dip into like what you’re saying, Michael, and build like a skill around this for it, that it could be a really super useful tool. And then I definitely have to work on the visualization of it because it needs the human touch there for sure. Tim was helping me today. I slacked him out. I was like, I’m running myself in circles. Like I just need some a clean brain to help me edit this story to like what the hell actually matters. So it’s been good, though, for that type of thing. 00:48:51 | Tim Wilson: I had a I mean, I worry like the when you talk to the like it goes it goes bloated, it goes along. And then you look at it and it’s really hard to fully wrap your head around. And it was interesting, even in that the spreadsheet, there were cases where it had just like hard coded done the math and hard coded it. So it wasn’t even like a traceable model in some cases. It was in some cases. But I I mean, that that continues. Like the higher level, like it’s so fast to generate content that’s polished and professional, but doesn’t have. The humanness in it. And I think in a in a proposal context, like you’re somebody’s going to be looking at it and they’re going to be asking questions, regardless of what it is, not your specific example there. I had a case where I’m doing some training, working with another company. And for various reasons, I’m developing the presentation in Horto. It’s very specific to the audience. But I also am pretty particular about I don’t want things to look great. And it turns out when you’re trying to use markdown, I mean, I’ve run through this before. But I worked with the people who were kind of co-presenting and we had kind of a rough we’d iterated through a Google doc where we really sort of figured out the narrative and the story. And that was like collaboratively. And then I went through the Google doc and put a little more meat behind it. And then I was like, what happens if I pop over to fable five and say generate me the I did this in cloud code. I already had a repo set up for it and had pulled it down locally and said, give me the shell. And that was really useful as like the raw because I fed in the this is the sequence. These are the specific slides and they still look like absolute, you know, garbage, like I’ve spent hours after that needing to iterate. But I still had that skeleton in place. So I wasn’t having to start one slide at a time. I have the whole shell, but it came from a human human massage, developed the narrative and the talk and the points and the highlights. But it was it was another kind of useful cloud code experience in that case. 00:51:16 | Julie Hoyer: I see cloud like decks that get put together and it’s like a standalone slide. It’s pretty good. And sometimes they’ll pick up on like this, maybe the main story point you wanted, like throughout this, like again, like bloated long deck that it will put together. But you know, it’s funny to my always go back to like just the basic McKinsey title idea. And it’s like, I need to start telling people to explicitly tell Claude or whatever I’m there using to be like, use McKinsey titles. I think the outline from a human before asking it to make the deck should be like, this is the story bullet points in this order that I want you to tell. Use it for McKinsey title style and say like this needs to make sense. Like top to bottom, because when I get into a deck that is like 00:52:02 | Tim Wilson: Claude generated, for example, it’s like, OK, generally, this slide makes sense. 00:52:08 | Julie Hoyer: I kind of understand what they’re saying. But to me, it doesn’t seem like there’s a good thread between and it’s like verbose and it’s repetitive. And then it’s really hard for me personally to figure out what is the point of this individual slide I’m looking at. 00:52:21 | Tim Wilson: But that’s not that hard to tell to tell Claude to say, you must walk me through. Make sure you have a solid outline. I’m going to I’m going to define the steps of how we’re going to do this. You’re going to make sure I give you McKinsey titles. I did have a rant a like a month or so ago on LinkedIn because I was literally I should remember what I was working on. And it said, I have a hand I have handcrafted a bespoke, beautifully structured five slide presentation and HTML and CSS. It uses clean, high impact management consulting layouts like McKinsey or BCG decks designed specifically for your topic. And I’m like, this is horrible. So that’s rant one. Like I was like, yes, you this is what you think is great. The other like just. Boogaboo, I was trying to generate HTML presentations with like our markdown way back when had one specific client where once every week during COVID, we had to pull all these data sources together and generate it. I’m like, I can make all this happen with with in our and pull together something that matches your template. It’s just you’re going to send your ultimate going to send us a PDF anyway. And I just got hammered. It’s like, it’s got to be in PowerPoint because we got to be able to edit it. And I was like, I understand the the the motivation and you need it. Whereas now everybody’s like, I can just make it an HTML. And I’m like, fuck you. Like, like it wasn’t good enough when I wanted to make something in HTML because it wasn’t just being spit out, you know, generically, it looked fine. So yeah, I’m sorry. 00:54:04 | Michael Helbling: Well, it’s weird with all the things I’m willing to let AI do for me. The thing I’m least willing to let it do is produce content or communication. So like, we even have at Stack Analytics now kind of a standard, like we will not let it write our emails. I like that I won’t let it write decks like technical documentation. I think it’s actually OK at and can produce good quality materials. But like anything where I’m attached to what the person is reading 00:54:33 | Tim Wilson: in terms of communication, I just don’t think it’s there yet. 00:54:37 | Michael Helbling: Like and I’m training it on my voice and those kinds of things that I keep testing. But I I’m just not comfortable with it. And I just feel it. It’s sort of like Stevia, you know, that weird aftertaste. 00:54:52 | Julie Hoyer: That is so good. I just think that’s exactly what it is. 00:54:56 | Michael Helbling: It’s just I don’t get it and I don’t like it. And so if I don’t like it, then I assume other people probably don’t appreciate it either. So it’s sort of like, all right. So then the least we can do is like, it’s going to be us typing that email. 00:55:07 | Tim Wilson: I have I have given it like content. I’m like, I need a McKinsey title that describes this, this, this and this. And if you take it, then I’m editing the edit it yourself to edit it. 00:55:19 | Michael Helbling: Yeah. Yeah. I think that’s totally fine. It’s more of like something that spits out. You just paste it into your email and fire it off. 00:55:27 | Tim Wilson: Like I don’t like it. 00:55:29 | Michael Helbling: Well, I don’t. It’s not good yet. 00:55:30 | Julie Hoyer: Once it does like a deck for you again, we talked about this in past episodes, like you become the editor of work that’s in front of you. But sometimes like it’s so much easier just given that like, it’s good enough because I didn’t think through what it needed to be already. Yeah. Because you didn’t try to solve it. Now you’ve been biased by what it spit out. 00:55:48 | Tim Wilson: And I will say when I’ve I really only a handful of times 00:55:53 | Julie Hoyer: have had it made any type of slide, but I have heavily edited anything that it gives me. Because to your point, Michael, I’m like, I know I can structure a better slide that will get the point across. But sometimes I do need like, can you just give me a couple ideas here? And like, pull what I like and I’ll mash it together. 00:56:10 | Michael Helbling: That I think is just fine. That’s just fine. It’s more of the other side of it where it’s sort of like you just turn your brain off. And it’s like, is that good enough? I don’t think that’s good enough because I could anyone could do that. The client could do that. Like as a consultant, like I hope we’re bringing more to the table than just regurgitating whatever Claude said. Yes, you know, ideally a BCG or McKinsey style email. Well, you know, the problem is sometimes when I write my own version and I’ve seen what it writes, I’m like, my version sucks. But at the same time, it’s like, but it’s yours. Like, it’s me. It’s me. So, you know, everybody just bear with me. I’m not as smart as an AI, but, you know, I’m trying. All right. I love it. One last thing I want to cover off on, because it was a big kind of learning for me is obviously using these LLMs for me has been a lot of fun and very learning experience. One of the big things that happened recently the last six months with with us is going from an individual using Claude to do stuff on a certain project to a number of people needing to engage with that same thing. And like, how do you share the information and context and those kinds of things? And we all like have we’re on data background. So like in data warehouses, everyone’s like, oh, you got to have a semantic layer. So the AI knows how to do all the things it needs to do. And so that’s been a very interesting challenge. I don’t claim to have solved it, but we’re very much looking at that too, which is keep iterating on this idea of how do we deploy context? Because there’s like so many different little decisions go into when you’re setting up data, like, OK, we’re going to tweak it just like this so that these things match up and we’re going to make this decision at this point to kind of normalize the data like this. Like all of those choices are filtering down into then other people making it harder to follow in your footsteps. And so it’s interesting because we’ve done some various things to sort of share that context more broadly. But that’s something I’m keeping an eye on in our space, especially in the data space, because like I think that’s really common. Is sort of like, how are people making this a team sport versus an individual effort? Because I think that’s a that’s a pretty big, pretty big deal. 00:58:41 | Tim Wilson: A little bit of a callback. I started to say we had this document just for the podcast. So it’s not on the data front. And I set this up in co-work. It was somewhere along there that I realized that, oh, the runbooks are stored locally, which I’m like, that is terrifying. Now it’s going into a folder that I have syncing somewhere else, but I’m much more comfortable with stuff being in like defined cloud, whether I’m syncing to GitHub with Git or whether it’s Google Cloud or whatever. But I started to say that we had something that I could hand off. If I wasn’t available, somebody we had backups that could follow the process and it was fairly safe. And now somehow with co-work, I’m like, can I hand this off? Like, like, I don’t know that in theory, I’ve handed it off to this agent, but I can’t figure out how I could safely hand it off, especially knowing that I will hit things and say, yeah, I need to update the task and the instructions here and there. So that’s that this is similar. Yeah. World that you’re like, oh, this supercharge is you, but it doesn’t necessarily supercharge the team in the way that it should. 00:59:51 | Michael Helbling: And I think that’s the big I feel like that’s the big lever or tipping point to value for AI and a lot of organizations is sort of like, you’ve got one or two people kind of doing stuff over in a corner. It’s making them real productive. But like, how do you create a process for AI usage that makes everyone that much more productive and step up the whole org with AI? Like, this is actually the whole part. I think the big part of it. And it’s actually a truly complex, I think, very deep problem that I don’t know that I’ve seen anybody really solve well. And given your recent LinkedIn post and the feedback you got on it, like I see a lot of other people, especially in the data space, because like it’s not a technology problem at the end of the day. It’s actually like a human process problem. And how you solve it goes into your culture. It goes into your process. It goes into all these different things. So anyways, that’s a fun one to kind of unwind, but also kind of like the frontiers that we’re finding and using AI the day to day is sort of like, oh, I did all this cool stuff. Now, how do I hand this work off to my teammate who now or my client who now needs to take this stuff and go one step further with it? Oh, what do I do? So that’s what you’ve got to find the answer. So we’ve built some things to do that. But it’s it’s I don’t know that it’s like it’s not perfect. It’s not perfect at all yet. And I don’t think with AI you can be perfect. 01:01:27 | Tim Wilson: So you just have to kind of be adaptive. 01:01:32 | Michael Helbling: Anyway, all right. Any last words, final thoughts? Oh, we’re like like we’re fine. 01:01:38 | Tim Wilson: Like it, like like the LLMS where we’re running along. Yeah, it’s right for both. All right. Well, then let’s have a Michael. Yeah, that was an excellent point. You did a great job. This has been every point you made was just so spot on. Sorry, I’m being like, I don’t appreciate that at all. Tim, that was the kind of bluff that I don’t appreciate. 01:02:05 | Michael Helbling: You’re absolutely right. You’re absolutely right. That’s the load bearing comment of the podcast. Tim, great job. All right. I bet as you’ve been listening to this episode, you’re like, wow, I can’t believe how terrible the Inelix Power Hour is in AI already and you’ve got thoughts and comments. Listen, we’re open to learning a little more. So this one we do actually want to hear from you. Like tell us how you’re using it. Talk about what you’re doing with it. Like we want to know, at least I do. So you can reach out to us. Do that on the measure slack chat group or on the LinkedIn or via email. At contact at InelixHour.io. 01:02:45 | Tim Wilson: And yeah, check in with us. 01:02:48 | Michael Helbling: Tell us how you’re using AI in your day to day. I think it’s a pretty wide open space and we’re all learning from each other, which is a lot of fun. It reminds me of the early days of analytics, to be honest, where we’re kind of all just sort of comparing notes. 01:03:01 | Tim Wilson: And back then it was, I remember that time with like web trends and like the report tables and the analysis tables and how you could, you could call up one turn and get it. Remember, remember that? Sorry. 01:03:19 | Michael Helbling: And you can also get stickers because nothing says you’re cool, like a sticker on your laptop or water bottle from the analytics power hour. And you can request that on our website at analyticshour.io. And I think no matter how you’re using AI or even how you’re using AI to listen to this podcast, I think I speak for both of my co-hosts, Tim and Julie, when I say, don’t stop analyzing, even if the AI offers to do it for you. You’ve got to stay involved. 01:03:55 | Announcer: Thanks for listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at at analytics hour, on the web at analyticshour.io, our LinkedIn group and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst. Those smart guys wanted to fit in. So they made up a term called analytics. Analytics don’t work. 01:04:19 | Charles Barkley: Do the analytics say go for it no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 01:04:32 | Michael Helbling: Say it all, you never know. 01:04:33 | Charles Barkley: Tony, do not put that. 01:04:35 | Michael Helbling: Do not put that on a date. 01:04:38 | Julie Hoyer: OK, wait. Ready, Michael? We’re going to self-diagnose your head pain. Why? Hang on. Hang on. Just trust me. Good. It’s good. It’s all right. Good. Fine. OK. Window. Here we go. Ready? 01:04:52 | Tim Wilson: Yeah. 01:04:53 | Julie Hoyer: So these are the different types of headaches. Is it a hypertension, a migraine, a stress, or is it a data sits under I.T. type of pain? 01:05:06 | Tim Wilson: That’s good. 01:05:10 | Michael Helbling: It’s kind of a data sits under I.T. 01:05:14 | Julie Hoyer: It’s got that kind of feel to it. 01:05:16 | Tim Wilson: Yeah. 01:05:17 | Julie Hoyer: My guy was literally sitting with a sinus infection last week. So I was like, oh, before I knew it. And I was like, oh, it’s like right behind my eyes, but low. And so I see this on LinkedIn. I mean, I should have known. And I’m like self-diagnosing myself with it until I got to the end. 01:05:33 | Michael Helbling: That’s good. Yeah, no, it’s the migraine one. But yeah, that’s pretty funny. 01:05:40 | Tim Wilson: That’s good. 01:05:42 | Michael Helbling: Yeah, I need to keep that in the back pocket because that’s that one keeps coming up. 01:05:49 | Tim Wilson: Yeah. It’s an episodic recurrent. Yeah. Recurrently episodic. No matter how much tile and all you take. OK. All right. Are we are we doing last calls? 01:06:03 | Michael Helbling: Um, I don’t think so. I’m picking up I’m picking up something from our. 01:06:13 | Tim Wilson: Moere producer. I don’t know. It’s comical for next week. Yeah. The Venn diagram between what Julie reads and what I read is. 01:06:23 | Julie Hoyer: Well, who do you think, you know, sent me all those newsletters? 01:06:27 | Tim Wilson: Yeah, I think this is well, I do have a kid who’s, you know, planning trips out in October and he’s super laid back and chill and he doesn’t get bothered if somebody like can’t make a decision. He’s like, well, here’s the deadline. And if you haven’t made it by then, you’re out. And he just moves on with his life and like, if I could get his attitude just to else, I would probably live ten years longer. Due to the hypertension that is not developed. 01:07:01 | Michael Helbling: Data still sets up right. He. There. Yeah. All right. All right. Let’s do this. Here we go. This is all right. Energy, energy, energy. 01:07:15 | Tim Wilson: Me, my mom, my mom, my mom, my mom. You know, the local folks jumped over the lazy dog over the lazy. Yeah. Rock flag and stevia. The post #304: I Can Haz AI? appeared first on The Analytics Power Hour: Data and Analytics Podcast .

August 4, 20261 hr 3 min

#303: Funnels Assume Progress. Barriers Recognize Reality.

Twenty years of digital marketing created something of a monster: companies poured enormous investment into analytics teams, MarTech stacks, media capabilities, and data infrastructure—and then watched all those functions march off into their respective silos to work really, really hard at producing activity rather than impact. Rusty Rahmer , founder of Starize AI and author of Working As Designed , joined Michael, Julie, and Val to dig into why that happened, why it’s still happening, and what it actually takes to flip the shovel over and use the right end. Along the way, Rusty—who Val correctly identified early as a spontaneous analogy machine—explained why customer journey maps on walls are basically the statistical average American life that literally nobody lives, why waffle fries are structurally superior to ridged chips (and what that has to do with cross-functional team design), and why the most important question a marketing leader can ask a room full of executives is also the one most likely to be met with complete silence. This episode is brought to you by Prism from Ask-Y —your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. This episode is also brought to you by Stape , your all-in-one solution for server-side tagging. Links to Resources Mentioned in the Show (Book) Working as Designed: Why your organization delivers exactly what it was built to — and how to redesign it for growth by Rusty Rahmer (Book) Analytics the Right Way: A Business Leader’s Guide to Putting Data to Effective Use by Tim Wilson and Dr. Joe Sutherland WITHpod: The AI End Game – Chris Hayes podcast series (Book) The Way of Excellence: A Guide to True Greatness and Deep Satisfaction in a Chaotic World – Practical Strategies Using Modern Science and Timeless Philosophy by Brad Stulberg (Book) Inside the Box: How Constraints Make Us Better by David Epstein (Newsletter) Range Widely by David Epstein (Article from Range Widely) How to Pay Attention and Care Deeply in a Chaotic World (NYT Article) The Revenge of the Philosophy Majors (Post) Half-Baked Product by Gerard Rubio Photo by Gabriel Komorov on Unsplash Episode Transcript 00:00:00 | Announcer: Welcome to the Analytics Power Hour. 00:00:08 | Announcer: Analytics topics covered conversationally and sometimes with explicit language. 00:00:15 | Michael Helbling: Hi everybody, welcome to the Analytics Power Hour and this is episode 303. Does it ever feel like we repeat the same steps sometimes even though no one’s really 00:00:26 | Michael Helbling: thinking about whether they actually apply in the scenario? 00:00:30 | Michael Helbling: That old saying, the definition of insanity, repeating the same activity while expecting different results, I think we’ve all been there and yet as analysts, maybe we’re slightly more clued into when it’s happening than others in a business, I’d like to think so. Maybe not. The idea here is are we thinking a step or two deeper as analysts or are we just sort of rolling with the same set of activities with no critical thinking applied? Well, we’re going to talk about it and now I want to introduce my two co-hosts who are definitely deep thinkers, Julie Hoyer, welcome. 00:01:06 | Julie Hoyer: Hey there. 00:01:08 | Michael Helbling: And Val Kroll. Hello. Hello. 00:01:11 | Michael Helbling: How’s it going? 00:01:12 | Michael Helbling: Good. And I’m Michael Helbling. Well, we wanted a guest and our guest is someone we have had on the show before, but not for a while now, Rusty Rahmer is the founder and chief product officer of StarizeAI. Prior to that, he held analytics and marketing leadership roles at GSK and Vanguard. He has a new book called Working As Designed and today he is our guest. Welcome back to the show, Rusty. 00:01:37 | Rusty Rahmer: Thank you. Thanks for having me back. 00:01:39 | Michael Helbling: I appreciate it. And as far as bios go, Val and I also worked with you on the board of the Digital Analytics Association back in the day. So we are friends going back a long way. So it’s great to have you back on the show. All right. So to kick this all off, you wrote a book. So obviously you’ve been thinking about this for a long time and then you decided to get pretty serious about it. Talk about that a little bit because I think probably something was brewing for a long time that kind of came out in the form of the book, but just talk about that journey a little bit and then we’ll dig into some of the content and some of the things in the book itself. 00:02:16 | Rusty Rahmer: Yeah. You know, people ask me, when did you start the book? Well, I feel like I started the book 20 years ago, right? It’s the experience of the last 20 years that I think, you know, at Vanguard it was about building, you know, at the front end of the digital wave, building out the capabilities of, you know, a user experience organization, a data and analytics and data science organization, marketing and MarTech and then, you know, digital transformation, then going to GSK and helping them a different problem, right? 00:02:44 | Michael Helbling: They had all of these capabilities and it was about, you know, why aren’t these capabilities 00:02:49 | Rusty Rahmer: working the way that we think that they should? Why haven’t we gotten the value out of all of this? And that’s a pretty common problem, I think, as I’ve moved on to consulting and I can see it over and over again at different companies and it felt like a problem that I had experience solving and that I was surprised is so prevalent out there and so I felt like I had something to say about that and I wanted to put it down and writing and my experience is what I was observing and what I think the solutions and the core of the problem and the solutions to that are. So that’s really what it was. 00:03:17 | Michael Helbling: No, I think that’s great. So talk a little bit about the book contents itself, like, you know, kind of what was the driving idea behind it? 00:03:27 | Michael Helbling: Yeah, I think there were two key observations. 00:03:30 | Rusty Rahmer: One was sort of where are we now, right? Like looking around at different organizations, I see the same problems, identifying that problem and sort of describing the solutions that I’ve put towards that and what I think the core of the solutioning is. And then secondly, like how did we get here, right? Like there’s a pretty clear path that led to this place that we’re in. 00:03:50 | Michael Helbling: And it’s both of those things that I think were what’s in the book, you know, to sort 00:03:56 | Rusty Rahmer: of describe how we arrived, what it is that the problem is that you’re feeling and then how do you solve that problem are really sort of what’s underneath it. 00:04:04 | Val Kroll: So what are some of those symptoms of like how we got here? Let’s dive into it. 00:04:08 | Rusty Rahmer: Yeah. Okay. Well, let me start with what I think people are probably feeling today, right? So as I led each of those organizations, whether it was user experience or it was analytics or Martek in the tech side, media, I think everybody in those organizations feels like there’s something wrong with it. Like we could be doing so much more with what we do here. Like we’re so underutilized to this problem of growth or whatever the company is trying to solve. 00:04:36 | Michael Helbling: And I think that feeling comes from very siloed organizations, right? 00:04:42 | Rusty Rahmer: I think I see the problem that chief marketing officers, chief growth officers, chief commercial officers have spent a lot of money over the last 10 to 20 years buying software, buying capability, hiring teams, building teams. And largely I don’t think that they’re getting a large percentage of the potential value all out of all of that investment that they’ve made. So from the C-Suite’s perspective, I put a lot of money into this on faith that this 00:05:12 | Michael Helbling: will produce value for my commercial organization, and it just doesn’t seem to change the net 00:05:17 | Rusty Rahmer: result very much. And I don’t understand what’s wrong here, but it’s not working. And I think the reality of that is that it is going back and looking at the history of how we arrived here and the moment that we’re in. 00:05:28 | Michael Helbling: And I think to switch gears a little bit, I think if you go back 20 years before the 00:05:33 | Rusty Rahmer: internet, you know, 1998 or whatever you’re sitting at your work computer, you don’t even have the internet on your computer, right? I mean, it’s hard to even imagine the world back then, but that’s the way that it was. 00:05:44 | Michael Helbling: And marketing your organization was print, you know, maybe, you know, some partner distribution 00:05:51 | Rusty Rahmer: methods, but it was expensive. Print is really expensive to send out. And so TV ads are there, you know, there’s some phone advertising, but everything is an expensive channel. And so your reach is relatively small. It’s relatively limited by your geography, right? Well, digital comes into the picture and completely blows that paradigm out of the water, right? Now you think about buying an email list, and you’re actually rewarded for sending it out to more people than it can actually, you know, like the more people you send it out to, the lower the cost of purchasing that list goes down, right? So you’re rewarded for your reach. 00:06:26 | Michael Helbling: And what that did from an economic standpoint, it allowed companies to enter markets where 00:06:32 | Rusty Rahmer: they were not previously competing with their brand, right? To create, oh my God, now I can reach the world with what my product or message is, and I can sell to the world. So for 20 years, we focused on reaching, reaching and building capability, digital capabilities that would help us reach audiences, right? And even bad marketing with the kind of reach you were able to achieve with the digital world drives your organizational growth. It just does. 00:06:59 | Michael Helbling: I mean, fractional return on that investment means massive growth for your organization 00:07:03 | Rusty Rahmer: just based on the scale that you’re currently achieving. But it’s really lazy marketing, to be honest, right? That reach and frequency marketing is lazy. And so you’ve been purchasing all of these capabilities. You’ve been buying Mar-Tech stacks, Ad-Tech stacks, you’ve been building analytics teams and data lakes and data environments. You’ve been hiring media teams. All of that has been growing in investment, but it’s all working as like, we have to build up this function. It’s never existed before. We’ve never had an analytics team before. We’ve never had a data team before. And so the mission of those organizations has been about how do we build and deliver some value immediately and then scale this organization to what it’s going to need to really drive the growth of this company. So we’re sort of stuck in that time where these functions are still working in silos, trying to do analytics activities that show we could be the best analytics organization in the entire world. We could be the best content creative people in the entire world. We need all the Mar-Tech tools, right? So the Mar-Tech teams are out buying things that the organization doesn’t even need yet just to have them, right? And I think that’s not how it’s actually working, right? I think we’ve saturated the growth market. We’ve reached every corner of the market we can possibly reach. And now it’s about quality of marketing, getting back to getting away from reach and frequency and getting back to timeliness and relevancy, right? Which means, hey, I actually have to understand my customer better than I’ve ever understood my customer before in order to show up in a timely and relevant way to that customer, right? And that’s actually what all of these capabilities we’ve been building have been designed to do. But when you look at the system that they’re operating inside the company, they’re still working in these vertical silos instead of working together on horizontal problems, like growing the company with its customer base, right? So I think that’s a bit of how we got there, which is understandable, given the thrust of digital and digital capability and the speed at which that’s grown. And then it sort of explains why we feel like we’re underutilized in the same system as well, why people aren’t doing more with the data that we’re creating, why Test and Learn isn’t being adopted as fast as it should be, why media dies at the front door of the company and OmniChannel doesn’t connect to it, right? I think all of these problems are functional silos out of that arc of growth and evolution that’s happened over the last 20 years. 00:09:34 | Val Kroll: And one of the things related to what you were just talking about, Rusty, is that really resonated with me. I’ve seen it so many times in organizations. Like you said, it’s like the best content team in the world, the best performance marketing group. And they produce a lot of reports with their analyst partners showing all this progress and performing above industry average for those different campaigns. And so the way you talked about in the book is like, how can we be doing so much that says we’re doing well, but we’re still not getting to where we need to go? And the way the system responds is, well, we just need to be doing more. 00:10:09 | Michael Helbling: And I was like, great, highlight, underline, startles. 00:10:15 | Val Kroll: That is such a repeatable pattern that I think people absolutely feel. But when was the first time you started to take a step back and acknowledge boring more on top is just not moving us forward anymore? Like when do you have a specific moment or story where that realization really started to come to life for you? 00:10:37 | Rusty Rahmer: It’s a little of two things. Again, it’s a little of having jumped silos to build these functions out and then getting the GSK and owning sort of every touch point of the customer, the entire customer experience really was like, oh, my God, now I’m sitting 00:10:51 | Michael Helbling: on top of this entire organization and I’m saying things that they can’t solve. 00:10:55 | Rusty Rahmer: Like this is like a square peg. And the problem I’m trying to solve is a square peg in a round hole operationally. Like no one, no one can function the way that I’m asking for them to function or solve the problems that I’m throwing on to the table. And so transformation sort of has to happen around that. And I think I think leaders experience it every day when they’re the ones, you know, a chief marketing officer is responsible for, I know, 20% growth and they go in every year and planning season and they set that goal on the table and they say, hey, our goal this year is being handed down to us as 20% growth, right? And in that room is the head of analytics, it’s the head of media, all these different heads in there of all these different functions. And as soon as the media is over, everybody leaves and everyone’s going to work really, really hard to try and achieve that goal, but they’re not going to talk to each other and they’re all going to work in their little silos. Like the solution to that 20% growth is sitting somewhere in the data strategy or it’s somewhere in the media strategy or it’s somewhere in the Mar-Tex that it’s not, it’s not, it’s all of those things functioning together to solve that problem in a way that we have not set up the company to work or reward more importantly to reward. And Val, that’s what you’re talking about because later on in the year after we’ve had that discussion and everybody’s going off into their silos and is trying really, really hard to make that goal happen, to try and be the hero, we’re going to get together quarterly and we’re going to talk about results and we’re going to talk about, we’ve built new data tables in the data, and we bought this new, we replaced this software with some new AI driven, whatever. And like, we built new customer segmentation model, like we’ve got it, we all worked on stuff independently, but the dial’s not turning because it wouldn’t, it wouldn’t given those pieces independently working rather than, you know, working together on a common problem. 00:12:46 | Michael Helbling: Hey Tim, imagine for a minute you’re an e-commerce marketing leader and you just launched a new checkout yesterday. 00:12:56 | Tim Wilson: And now you’re wondering, did performance change or did the tracking break? 00:13:01 | Michael Helbling: Yeah, it’s so important and with the STAPE MCP server, you can connect an AI assistant to your STAPE account and you can ask it, things like checking the container to confirm that the tracking domain is valid, review usage before and after launch, and you can show what changed by browser or client. 00:13:21 | Tim Wilson: So instead of digging through infrastructure screens, you can get a plain English operational check. 00:13:28 | Michael Helbling: Yeah, you can also review daily usage, see which domains are generating traffic, and check for unexpected costs or surcharges. 00:13:36 | Tim Wilson: That helps an in-house team answer an important question faster. Is this a marketing problem or a measurement problem? 00:13:45 | Michael Helbling: Yeah, hone in on the solution and to do that, click the link in the show description to learn more about the STAPE MCP server and how it can help you. 00:13:55 | Tim Wilson: Michael, ask wise prism version 3.0 now speaks MCP. That’s great. 00:14:03 | Michael Helbling: I love acronyms. They make me feel like I’m in a meeting. I forgot to decline. 00:14:07 | Tim Wilson: Well, in this case, it means prism can plug into Cod Desktop, Cod Code, and other MCP compatible tools. Oh, I do like that. 00:14:15 | Michael Helbling: So prism becomes part of the workflow instead of another tab I’m pretending is organized? 00:14:21 | Michael Helbling: Exactly. 00:14:22 | Tim Wilson: From Cod, your users can query the workspace, trigger analyses, and pull results into whatever they’re working on. Oh, I love that. 00:14:28 | Michael Helbling: Less where did that come from more? 00:14:30 | Tim Wilson: Oh, look, a usable workflow. Yeah, and with prism’s memory engine, skills, permissions, and admin controls, the context stays managed, not trapped in one person’s chat history. Really useful because I think Claude told me this last week is not a governance model. Not exactly. So go to ask-y.ai and join the waitlist and pro tip use code APH to jump to the top. That’s ask-y.ai and use code APH. Fewer tabs, smarter agents, less workplace archaeology. 00:15:06 | Rusty Rahmer: I think you see it every day. I mean, I really think you do. The advantage was having sat above all of those functions and going, okay, you know what I don’t need is more exposures with no one going to a high value action on my website. Like, are you working with the OmniChannel team to deliver, like, after we expose them to deliver a journey that moves them closer towards the high value action that I want? Oh, no, you guys never talk. Okay, this is a system that produces and rewards this, and now we need to change the system that we’re all operating in to reward, to give you a different problem to solve and to reward you for working as hard as you’re already working, but to solve a different problem than what you’re trying to solve today, right? 00:15:48 | Julie Hoyer: And what you said really hit home of it’s time for us to turn back to thinking about how timely and relevant it is. For the customer, and I have also found working with a lot of my clients is that in the pressure of activity, like you were just describing in the silos of their organization and the pressure to do more, get more out of it, or at least that’s, again, the habit that we’ve created, people very quickly lose the lens of the customer and the customer experience. And at the end of the day, that is what all the functions that the company need to work together to change. I am curious, how have you found a way to help teams get back to thinking about the customer or helping teams reach across to the other teams involved? And I’m curious from the different positions that you’ve held or now you’ve been moving 00:16:42 | Michael Helbling: into like consulting, like you mentioned, is there anything you use to kind of convince 00:16:47 | Julie Hoyer: people of that or scenarios where you’ve been able to help change those working habits? 00:16:52 | Rusty Rahmer: Yeah. And Julie, two things there. I think first, I probably even said those words too, getting back to the customer sort of experience. Honestly, it should have always been about the customer experience from the beginning of time, right? When it was a small hardware store that you walked into in your neighborhood, like they knew in order to do business, they had to help the customer solve a problem. And I think as this digital era has helped companies scale, I mean, any company you take, if you go back 20 years and look at what they were, you know, the number of people that worked there, the size of revenue, all of that, that is a magnificent scale. And I think that challenges companies to figure out how to even grow their organization to be that big and handle that much customer work. But I don’t think we ever should have lost track of, I still actually need to be good with customers. That’s why I called it a little bit of lazy marketing in between because it did get away from that even though it shouldn’t have. And now if you’re going to grow, you’ve reached as much as you’re going to grow with bad marketing. If you want to grow that baseline from what it’s at, you’ve got to get good at the customer experience. You have to solve customer problems again. So you’ve got to know your customer better than you ever have before. And again, the good news is all of this tech stack and everything we do is actually ideally designed to solve that problem. You’re just not using it. The cover is a shovel upside down, digging a hole. That’s actually the metaphor that’s trying to be conveyed on the front of the cover. You have the right tools. You’re just got to flip it over and use the other end and it will solve the problem for you. To your point, Julie, I think what I like to do, I always, as a leader who’s led a lot of transformations over the last 10 or so years, I always think it starts with vision. 00:18:29 | Michael Helbling: I think you sit down and you talk about where do you need to be in five years from now commercially? 00:18:35 | Rusty Rahmer: Like what goals do we need to be achieving five years? And what’s it going to take? How do we have to operate in order to achieve that, right? And well, we need to do more when you’re, okay, great. A fundamental question I love to throw in there that’s sort of a blockbuster question that really stops the conversation in its tracks is, okay, what are the customer problems we have to solve in order to turn them into better customers to achieve that result you 00:18:59 | Michael Helbling: just talked about? And they just stop. 00:19:03 | Rusty Rahmer: And there’s no answer to that. There’s just silence. And I go, look, fundamentally, marketing is about changing mindsets. It is about, if they were already going to buy it, I wouldn’t need to waste money marketing to them, right? That I wouldn’t need to do that. So it is about changing a customer that isn’t going to do this to some degree. Maybe some customers are further away from doing it than others. But in some way, it’s about interjecting myself into that customer and changing an outcome that benefits the customer and me at the end of the day. And if I’m asking you, what is it going to take to change the mindset of your customers or what problems stand in the way of your customers being better customers for you, then I don’t know how you’re ever going to achieve those goals. That is fundamentally the question. That’s the only way to achieve growth is solving those problems for your customer. And the fact that you’re sitting here quiet tells me we haven’t done enough to know our customers and what problems they have that are preventing them from being better customers 00:19:56 | Michael Helbling: for us. 00:19:57 | Rusty Rahmer: And that’s something we better get back to doing. And look, I’m not the first person to preach customer centricity. But I think it’s been a forgotten concept. And I think many companies would say that they are customer centric because they have a lot of customer data. 00:20:16 | Michael Helbling: But that customer data is a lot like saying, this side of the river holds the biggest fish. 00:20:21 | Rusty Rahmer: So if I fish here, I catch big fish. It’s not how to catch them. It has nothing to do with how to catch them. And so you really don’t know your customers because you don’t know what the problem is. And therefore, you can’t solve it and move big fishes or small fishes over to the side that you want them to. And so I think it’s about that. I think that’s the moment where they sit back in their seats and go, you know what? Maybe we actually don’t know the key to that question and knowing it is unlocking the future for us. So how do we do that? And so you’ve got an ear to start talking about how you start solving that problem and how you start getting back to knowing your customers again. 00:20:56 | Julie Hoyer: And you know what’s interesting, you talk about knowing your customers. Another thing I’ve run into a lot is when we get into segmentation of customers, again, at the thought of, oh, we’ll figure out who’s alike and then we’ll market to them accordingly. It’s like getting at the idea of caring about the customer, figuring out how to work with each user type. But sadly, the first thing I see a lot of my clients reach for is they segment them based on outcome. They say, who are my big spenders already? Or who are already really loyal and who are the people who are not loyal? Which is interesting because it’s so backwards to thinking, well, wouldn’t you describe them by similar characteristics and then you’d hope to interact with them to then drive the actual outcome measure for the different groups? But I find it really interesting that so many companies end up actually segmenting on outcome, which seems so backwards. 00:21:55 | Rusty Rahmer: It’s so about you, not about your customer, right? And I get it. It’s particularly in pharma, right? Like that’s an industry that I spent time in that I could see right away grew on a really expensive resource called a salesperson showing up at a doctor’s door, right? Which I don’t know, costs something like $300 every knock on a doctor’s door. And though that ability has drastically been reduced by legislation, they don’t really have the ability to do what they used to be able to do or be as effective with sales folks 00:22:24 | Michael Helbling: as they used to be able to. 00:22:26 | Rusty Rahmer: So they’re going digital, but it’s exactly that, Julie. When I think about sending a very expensive resource like a salesperson out, I can’t send them out to very small customers where the customer opportunity is very small. I have to send them out to the place that they could have the potentially biggest impact on my bottom line. I get it. But when you extend that thinking to digital, you’re making a mistake, right? Like digital doesn’t cost the same amount of money. And that was you were putting the problem set on a human to interact with another human to figure out what the problem actually was that needed to be solved, right? Now you’ve got a digital ecosystem, which can reach bazillions of people, very, very low cost, but you don’t know the problem you’re solving as you’re blasting all this stuff out all the time. So you still create segments that are like, these are the biggest fish. Let’s spend the most marketing dollars on these biggest fish when in reality, if you 00:23:19 | Michael Helbling: stepped back and you said, you know what, there’s probably only four or five problems 00:23:24 | Rusty Rahmer: that my customers have that if I could solve these problems, they would be a better customer for me, right? And if you segment your customers on that first and say, okay, here are big fish and little fish that all share the same problem. And if I can solve that problem, I would turn all of them into better customers for me. Now, some of them within solving that, there may be different ways of solving it. I may send a rep out and maybe I wouldn’t send a rep out to a small, but I know the problem I’m sending the rep out to solve and I know what I’m asking my digital marketing or my customer journey marketing to solve as well. It’s just, I’m deciding how much to put into the solution, but the solution is the same across the board in terms of the value that has to be exchanged with that customer to move them from where they are to where I want them to be, right? So I think that segmentation has a place. It just isn’t the first cut of segmentation. It’s a second or third layer of segmentation that says, okay, once I’ve segmented on the problem I need to solve and I’ve got solutions to that problem, right, which is a portfolio of solutions. I can decide which of that portfolio of solutions I should invest in which customers based on cost and potential return on that investment. 00:24:33 | Julie Hoyer: And I think if a company gets that far and they’re hemming and hawing about the incremental value of a solution on a certain subset of users for a problem, like, I’m sure they’re doing pretty damn well because that is like top tier thinking, like, if you’ve made it to that step, like, I want to come meet your company, like, honestly. 00:24:52 | Val Kroll: Totally. Well, and so that’s, I cannot wait to talk about some of that portfolio stuff in the way you’re talking about thinking about, but I want to go back a little bit to the transformation it takes to be able to get to what you guys were just talking about because I know you reference like the cover of the book is about like turning around the tool like you have it, you’re just not using it correctly. But I do have the privilege of, you know, being in your circle. And so I, and you’re also the king of analogies, so I can’t wait till we do the tally at the end of the episode for all the amazing analogies you give us. But our view, and there’s so many that have stuck with me, but I remember you were talking about a team that was mobilized and ready to do some of this work. And I think the way you phrased it, hopefully I don’t butcher it is like we’re ready to work together to make Thanksgiving dinner. And the request I’m getting is for a side of cranberry sauce, more cranberry sauce, just like put more cranberry sauce on my plate. And so I want to talk a little bit about like, who needs to be involved in some of this transformation and how do these groups kind of work together because like Julie said, getting an organization to like put aside your segmentation that you’ve been so emotionally like your crutch for the past however many years to something that you’re talking about with this transformation. Like talk a little bit more about like what that takes and who you have to get bought 00:26:16 | Michael Helbling: in. 00:26:17 | Rusty Rahmer: Yeah, it’s a great question. And as I just mentioned with the customer centricity, right, like I don’t I’m not the first person to preach customer centricity. I mean, that’s been out there for a long time. I think there are this this entire book is not about any one unique concept. It is more about all of these concepts that companies are putting into play. Like they think they’re being customer centric, but they’re not applying it correctly. They have agile, but they aren’t totally applying it from front end to back end. It’s only a it’s only a tech build. It’s not a front end strategy, right? So I think it’s on this book is about a lot of those tool, not just more tech stack and had tech stack and and teams that you have built, but also ways of working that you’ve deployed, but you’ve only got like a percentage of it’s deployed, right? Because this is this is where it comes together, right? I think at the there’s sort of an arc of maturity you go through here, right? I think there’s sort of when you don’t have functions, you build centers of excellence to create consistency across your organization and to get you over the large investments that need to be made around capability and discipline and structure, you know, the design systems for the UX team, you know, analytics, so you don’t have 10 sources of truth, you’re down to a, you know, a limited set, you’ve, you’ve got definitions of data dictionary, like all those things come with center of excellence and they’re really good and they’re really important. But ultimately when you talk about solving these customer problems, they largely become a bottleneck pretty quickly, right? Because you need localization of the problem solving and not only that, you need quick feedback loops, right? This is where, you know, Val, of all people, test and learn, right? She’s so close to your heart, but how difficult is that to introduce into a company that’s organized into the site? Like it doesn’t work. First off, it takes them nine months to take an idea to market. When it takes nine months and nine months of cost to get an idea in the market, I can’t do two of those. I just need to get it out and get some kind of result. I have no time to test any alternate ideas, right? So this has to get these teams, these teams that can do this work closer to the front end, be able to deliver pieces of things in a hurry, evaluate quickly with analytics if it’s working or not, and pivot quickly. It can’t go through year-long planning cycles, nine-month delivery site, it can’t do any of that. Like that, look, you’re not going to build that thing overnight. You have to mature to it. But ultimately, when you have a team that is comprised of all of the different functions that make digital run from data strategy to analytics and two kinds of analytics, right? Long-term analytics and fast, quick, dirty, is it worth it? 00:28:59 | Michael Helbling: The thing we just put it out in the market, how much traffic is coming to the website 00:29:02 | Rusty Rahmer: from that? What are they? Are they achieving high value act like quick and dirty? To media buying, all of this needs to be done on a rapid scale with a team that comprises all those disciplines, right? It’s not cheaper. I won’t tell you it’s cheaper. Centers of Excellence are really cheap because you scale them across the entire organization. When you start building these teams, you’re borrowing discipline from the Centers of Excellence and you’re dedicating them to a set of problems, right? 00:29:29 | Michael Helbling: So you might say in this case, you might say, all right, let’s pretend we’re at a pharma 00:29:34 | Rusty Rahmer: company and we’ve identified that there’s a valuable cluster of customers who need more evidence in order to write the script for our drug over what they’re currently writing the script for. We’ve got a better drug, but they’ve been writing that for 10 years. The existing drug, our new drug, better results, but we have to prove that to them in order to get that, right? Now, there are different value exchanges that can happen to do that. You can get them data and white papers that say the results are better. You can have them talk to a peer. You can get them to a conference where they see somebody, a key opinion leader speak on it. So there’s different ways of solving that, but I’ve got to build journeys to get them to those high value actions. I got to get somebody to a conference and that means I’ve got to make them aware of the conference. I got to find the right people who would be open to going to a conference. Maybe that’s geography-based. I’ve got to build a journey that makes it easy for them to sign. There’s a whole journey that goes behind delivering that value exchange, right? I need a team who is dedicated to that problem set of not just delivering those value pieces, but also evaluating, are these the right solutions to somebody that even needs more evidence? Or are there better solutions out there? Maybe a podcast on something like this might be a better solution, whatever. So they’re looking at not only the tip of the spear, the high value actions or the value exchange pieces, but also, are we getting people to sign up for those things? Are we actually exchanging those value when we pick a customer and we say, we want this customer to attend a conference. How good are we at getting them to be able to do that? Are we improving the journey that gets them to do that? When they do go to the conference, is that the right solution? Is that moving them or not? 00:31:18 | Michael Helbling: That all needs to be rapid design, analytics behind it very quickly. 00:31:23 | Rusty Rahmer: And so you need a team who has all of those disciplines or can borrow all of those disciplines to deliver in that in a very agile or lean way, right? You’re working model with some hybrid of agile, lean, kind of on-bonnish thinking that you put together to solve that problem. But every two weeks, you’re putting something out into market, you’re getting results back 00:31:40 | Michael Helbling: immediately and you’re saying, okay, this is working or this isn’t working, fail, go 00:31:46 | Rusty Rahmer: on to the next idea. And that’s where Test and Learn finds its home with all of these other disciplines helping to do segmentation more precisely. Maybe we targeted the wrong audience with that message, content helping to improve the messaging consistently, time after time, you got all of that at your disposal to try and solve that problem. Now, it is more expensive, I will say it again, than the center of excellence. But if all the center of excellence is doing is pumping out more cheaper, that’s not impact. This is impact because I’m going to give them a problem. And I’m going to say, here’s the population that I know that I want you to target to change that this is the population I’ve defined as having the problem of needing more evidence. And I’ll be able to look at that population and say they’ve moved or they haven’t moved as customers as a result of the actions that this entire team has put forth. So I have a measurable result of impact at the end of that in a way that the center of excellence who just pumps out websites or pumps out email campaigns or pumps out like doesn’t have any connectivity to an output or an impact output associated with it. So it’s just pumping out actions, but not results. This is tied to a result. 00:32:51 | Julie Hoyer: I really like how you started to define what a journey, so you were saying it’s for a specific audience with the same problem. Because another thing I find is I spend a lot of time talking to my clients and stakeholders about a journey, what they consider a journey. I’ve heard people say, I think maybe the purist is like, well, the journey should be defined by the business ahead of time. What is the ideal flow? But then these companies spin up huge websites, huge digital ecosystems, they offer labs, they offer webinars, they offer e-com online or their salespeople in person, and it gets very complicated fast. So then people want to say, well, we have all this behavioral data. Let’s just use AI to surface the pattern, and that will tell us the actual journeys. But what I hate about both of those is it’s completely disconnected to the customer. So I really like how you put those bounds around it, because I think that’s something that I’ve been searching for, because I’ve started to just get so crabby when people want to talk about journeys. I’m like, oh, freaking journeys again. There’s no good answer. But I really like the angle you’re taking on it. 00:33:58 | Rusty Rahmer: So, Julie, you just stepped on a landmine for me. I am here. 00:34:02 | Michael Helbling: Yay. 00:34:03 | Rusty Rahmer: Customer journey exercises. 00:34:04 | Michael Helbling: Oh, my God. 00:34:05 | Rusty Rahmer: In my life, if I could count the number of times I’ve walked by a room and looked in and saw like this massive arc of customer journey with stars and arrows and boxes, like graphic designer skills going to waste on this magnificent board that is a customer journey, right? Like from whatever, biggest waste of time I’ve ever seen. Actually, it’s an exercise that’s probably not a waste of time to do, but is so non-reality. So this is the analogy I like to give around this. If we were to take the average American’s lifespan and say, well, what’s the journey? Well, I don’t know. I’m born. If you average it out, you’re born. You graduate high school. You’re born to two parents. Oh, by the way, you’re a family of 2.5. I don’t know who the 0.5 is, but you have a 0.5 of a sibling. You go to high school, you graduate, you go to some four-year college, you get married, you have 2.5 kids of your own, you retire, whatever. That’s your entire life story, you go on vacations, you retire, and that’s it. Literally no one in the United States lives that life. Literally no one lives that life. That’s what a customer journey on a wall looks like at a company, and no one actually walks through that thing that way. That’s an average of people’s lives that literally no one lives. The reality is, some people do go to college. Some people are born with six siblings, some are born with one, right, or none, just one individual child. Some go to college, some don’t go to college. Some get married, some don’t. Some get divorced. Some have kids with the first relationship. It’s messy. The real world is messy. You can’t create an arc that touches all of those points and pretend that that’s a reality. That is not a reality. What you can do is exactly what we’re talking about here is say, okay, no, I can’t do a whole arc. But if somebody’s online shopping for a kayak, I can give them a journey to get them to a kayak. I can detect that, and I can get them to the kayak, right? Maybe they buy another one after that, so that’s like a divorce. They go through the same purchasing thing again, okay, no problem, right, if it’s fine. You can only sort of detect in the moment what their need is and try to solve that need. You have different needs that you can solve for, but you can’t configure them in a definitive way that they’re going to go through them, right? Even if you say, my biggest problem with this new product launch that I have is I have to create awareness. No one knows about my new product I’m about to launch into the market. Okay, well, your first journeys are all going to be around awareness-based, right? 00:36:43 | Michael Helbling: Okay, great. 00:36:44 | Rusty Rahmer: Let’s fast forward five years into your new product out there, and a competitor comes into the marketplace, right? Here’s a really good customer of yours that keeps buying your product, but suddenly there’s a competitor in the market. Well, they might have to go through awareness again. Any awareness might be how your product is better than what the competitor’s product is, right? Like, sometimes they go back through the same journey again. That’s what I’m saying. So I think to try and create a journey which links all of these subjourneys together is insane. It never happens, but there are micro journeys. And at the end, at the tip of a micro journey is a value exchange. If I can get them to do this, they will move closer to being a better customer for me. And if I can use data and analytics to determine what that need is, which is what I should be using it for, then I can put them on the right journey and score and evaluate my ability to deliver a good journey and a good value exchange to move them towards it being a better customer for me. I think the other analogy that’s in the book that kind of works well here is the game of Plinko, right, which is this old wooden nickel game. Drop it in the top. It hits these nails, and there are four buckets at the bottom that have different points associated with them. I think the wooden nickel is the customer. 00:37:48 | Michael Helbling: And I think at the bottom are the journeys that we’re trying to put them on, like, okay, if I could get them on this journey, they’ll become a better, if I can exchange this value with them through this journey, I’ll make them a better customer. Each one of those nails they’re hitting on the way down is an opportunity, it’s an interaction with your brand that is an opportunity for use data and that interaction to try and guess which bucket they need to be in at the bottom of that. And that is the game of data and analytics at the end of the day. When I use this to look at patterns and identify what my customer need is at scale for this group of people, this is the problem that they have with our brand. And if I can identify that, now I know from marketing what problem I have to solve. I can measure the actions that I’m taking, the journey that I’m taking to try and get them to exchange that value, which I believe was, I can measure all of that and tell you how effective all of that is at changing my customers into better customers. 00:38:49 | Val Kroll: I like that a lot. And I will say that the one thing that I repeated in my notes as I went through your book, I was like, the process is testing, like in there, in a couple of like, carrot, like that’s, yeah, where that all fits in. I love that. So like if we have these problem mobilized pods that are focused on continuous delivery of the portfolio of solutions that could potentially relieve or solve some of these problems, like paint the picture. So one thing that we didn’t really get to talk about too much is like the system and the system’s expectations of how people behave today and how they’re kind of rewarded for activity to how the system behaves in this like more idealistic vision that you’ve painted. Like, can you talk a little bit about that? 00:39:40 | Rusty Rahmer: Yeah. I mean, again, and we’ve achieved this, right? I think at GSK, this was sort of the pinnacle of what we were able to do was put these teams together and the reward structure completely changes, right? We’re not rewarding activity anymore. We’re saying, here’s a population that needs to move closer to the ideal customer set that we have. And here’s a set of problems that we believe is standing in the way of them being that, iterate and solve that problem, right? Validate that we’ve even got that, like maybe we’ve got the wrong, the way we’ve segmented these customers into this problem set may be wrong. The value, the products, right, in the experience products, the value exchange experience products that should, you know, our conferences, maybe that isn’t the answer to this problem that we think it is. Maybe that really doesn’t move a customer forward. So look at the portfolio of things that we’re trying to get them to do and our ability to actually get them to do that and help us optimize that and the way, again, reward. Again, I think people today in the silos of the existing system are really innovative and ingenious. I think every team I’ve ever led has extremely talented people in it who are trying really hard and defining new ways of solving problems for the company. It just isn’t latched up to something, a common problem that actually delivers impact the way that this system does and that we can reward and recognize those for that kind of innovation and ingenuity against a well-defined problem and a system that can measure the actual outcome of the efforts that they’re doing, right? And so, again, I can measure the cost of, you know, with a really broad brush, I can measure the cost of my centers of excellence today. I know how much they cost me. I know how much the tech stack they’re working in costs me annually. I know how much headcounts in there. I know contracts. I know all the answers to that, right, at a very high level. I know that cost. And I also know how much our impact that entire organization is having out in the real world in terms of growth. I know those two things. And I know this model is significantly more expensive than that model, but I can show the results and the outcomes are significantly higher than the money that’s being spent in this other model. And when I can do that, I can reward and recognize the people within the system for the amazing ingenuity that they’re having within that system. 00:41:56 | Michael Helbling: So in the operating model chapter, you kind of develop this idea of, like, developing those cross-functional teams as opposed to sort of like center of excellence concepts. So just to clarify boundaries a little bit. And also, I think if I was a business leader listening, I sort of have some questions about like, how deep should that go? Should I try to take everybody? Cause like some people are just specialists, like they are working on that thing. And do I need to throw them in a cross-functional team or is there sort of that structure? And then I set up a like a tiger team sort of set up. Like how do you, how do you bridge that gap for people or advise people? 00:42:38 | Rusty Rahmer: Yeah, I think it’s a great question, Michael. And I think there are, it’s hard to put that in the book, the experience and what I think, because I think, well, I say this because I think a smaller company has a different answer to that question than a larger company does. So it’s hard to prescribe that clearly, but here’s what I would say. I think, you know, this is, this is another analogy for you, Val. I think potato chips with no ridges break really easily in the dip. They do. The reason ridges were invented is it makes the chip stronger, right? But ridges only go one way. You have to orient the chip correctly, Rusty. Well, that’s correct. Yeah, yeah. Come on, you know, what’s stronger? This is not our first time dipping a chip. Anybody who dips horizontally is making a huge mistake. And that’s the other. There’s going to be half a chip in the dip. That happens. It’s not going to be good. No, but, but what’s even stronger than, than that, right? Which, which I would say, if you only went with teams that were, you know, journey aligned, you would have horizontal, it’s stronger than the plane chip where nobody’s aligned to anything. But it isn’t the strongest function that you’re the strongest model that you can have. The strongest model is the waffle fry. Seriously. The Chick-fil-A waffle fry is the strongest potato in the world. It can’t be broken any way you try. All right. So you get it. Crosscutting is actually twice as strong as, and I’m sorry, there’s probably some engineer that’ll tell me more to a waffle chip. Trader Joe’s has a waffle chip. They do. They absolutely do. That is a supreme dipping material vehicle. It’s nice. It’s real nice. Yes. Yes. So you get the point. I think you can’t just go one direction with this. I think to your point, Michael, in addition to these, the players on these teams that are, that are multifunctioned, cross-striped, whatever you want to call it, still need to belong to a discipline, a guild, which belong, the center of excellence still needs to exist. The work just doesn’t get done in the center of excellence. It’s now transferred out to these teams. But the members of that team belong to the guild of the center of excellence, because we, if you just let it go, you lose all the discipline of a center of excellence that has, you know, updates the technology and the stack that they actually work in to make sure that continues, that there’s a roadmap for that, that documentation continues, that standards for design continue. Right. You can’t, can’t turn it all loose. I think you still retain a small center of excellence that continue to represent the guild and the need of the guild to be able to do its job fast and effectively. I also think it can, in a large organization, you should separate out the value delivery products, so conferences, white papers, these things should be developed by teams outside of the journey teams. Journey teams should be mainly focused on identifying the right audiences, moving those audiences through the content journey and evolving the content journey to more effectively move them to exchange the value. But that portfolio of value things is actually another team that should actually develop and own those. Ultimately, if you can afford to do it, I think if you’re a small organization, this sort of collapses a little bit on itself, you know, smaller teams, smaller, see if, you know, all that stuff. So, center of excellence. 00:46:04 | Michael Helbling: But then you run into the age old Glen Gary, Glen Ross problem, you know, these leads, these leads are all good. Oh, the leads. 00:46:11 | Rusty Rahmer: It’s a great movie, by the way. No one has the patience to watch that movie anymore, but it is amazing dialogue. 00:46:17 | Val Kroll: Well, before Michael says what I think he’s about to say, I have to ask, the book was written in a very specific format, a format, the way to kind of phrase it. So I think, Christine, you need to tell your future readers that are currently listeners, a little bit about how you made some of those choices. 00:46:39 | Rusty Rahmer: So I actually wrote it long form originally. So it’s a 500 page, there’s a, there’s a 500 page version of that book. And when it, before I even started writing the book, I have some mentors that I, that are sort of guiding me through the process. We’ve written books before they are C-suite individuals who, who sort of were the people I was trying to speak to the most to help them understand why the money that they’re spending isn’t, you know, returning what they, what they really want. And so I had pitched the idea to them very early on, is this a book? Is this worth a book? Is this worth a book? And they were like, oh my God, yes, definitely write that book. So, so they, they sort of, you know, helped me affirm that there was, there was something to be said here. I developed the 500 pager. I send it to them and they’re like, I’m not reading it. Like, who are you talking, like, what is this? Like, I don’t have time to read this, you know, whatever. And they’re like, when you came in here, you told us about this and we were all in, we totally understood it and we were ready to go back to our companies and make a change. Just tell us that’s like, talk to us like you did then. And so I had to go back and take the 500 pages and turn this into what is more of a conversation with the reader than, you know, than the 500 pages. It is really meant to be a conversation. It is spaced with timing, which, you know, when you first open it, it’ll look like poetry. It is not poetry, but you will, after the first page, you’ll sort of fall into this groove of single spacing pauses versus double spacing pauses. And it helps make the points that I’m trying to make in shorter, you know, less stories, less get to the more get to the point CEOs sort of speak. And it, you know, it condensed the book significantly and it helped me say the same thing in a lot less words that I think C-suite people will be willing to endure. 00:48:23 | Rusty Rahmer: I’m working, I’m more of a podcaster. I listen to things more than I read the physical copy. So, you know, I get the book published and I’m working on. Okay, now I got to get to the part that I would actually enjoy, which is the podcasting part. Everybody’s like, my mentor’s like, you have to do it. It has to be your voice. So you got to do this. I’m like, okay, well, I’ve never done this before. So, okay, let me look into this. There’s a lot of work, editing and all that. But I’m like, wait, there are these great AI products out there now. You record like a 30 second sample and it’ll, I’m like, yes, this is it. I’m going to do this, right? So I go, I buy the software, I record the 30 second sample. And then I immediately turn it loose on the first chapter of my book. And it actually did a pretty good look. It did a really good job of being me reading the first chapter, which was the problem, right? Listen, I turn, I turn all of my devices to like speak in British because everything sounds smarter if you have a British accent. I mean, you sound brilliant with a British accent. Like Siri talks to me and because I like to think that I’m around smart piece. So, okay, all my devices speak with a British accent to me. The opposite, right? So you could read the script of idiocracy, but if you do it with a British accent, it sounds like the Magna Carta. It’s amazing, right? Oppositely, if you have a Philadelphia accent and you read the Magna Carta, you sound like you’re reading it at idiocracy. And that’s exactly what I was like, oh my God, is that my accent? Do I sound like that? It was terrible. So I ended up having to hire someone to do it. It’s almost done. But like, it’s so much. But I’m not even, I only write to the guy in email because I don’t want him to hear my voice now. I won’t talk to him on the phone because I don’t want him to hear out there. 00:50:04 | Rusty Rahmer: Water, water. All right. 00:50:07 | Michael Helbling: This is awesome. This is so good. And the book is really good. And it does read. Once you get into it, you do read with a little bit of cadence. And I definitely picked that up as I was reading it. I was like, at first I was like, oh, what’s going on with this book? And then I was like, oh, actually it kind of does flow once you get used to it. So it is well worth it. And I don’t think it’s just for CEOs either. Because I think even when you’re sitting in a seat, I think you can advocate for some of these things. If you’re not the senior executive in your department, you can sort of raise your hand or start to apply some of the principles even in where you’re at. So, okay, we do have to start to wrap up. Awesome conversation. Thank you so much, Rusty. It’s been too long. And you’ve been working on this super hard. It’s like, wow, you’ve got a lot done. I was like, what have I been doing? No, so thank you for all the work, putting that together and coming on the show to talk about it. Okay, one of the things we do is go around the horn, share a last call, something that might be of interest to our guests. Rusty, you’re our guest or might be of interest to our listeners. And Rusty, you’re our guest. 00:51:22 | Rusty Rahmer: What is your last call? Have you guys read this book, Analytics the Right Way? Have you heard of it? Yeah, yeah. Little plug for Tim. Little plug for Tim. No, that’s good. Um, lately, I have fallen in love. This is the second time I’m listening to the same seven series podcast. It’s the first time I’ve ever listened to a podcast twice. It’s by Chris Hayes. It’s called the AI End Game. It’s about seven podcasts, about an hour each podcast. They’re not terribly long. He has guests on to each episode. What I love about it is it covers every, it’s a, it’s very sort of top thinking, you know, about AI, right? In the world, everything from philosophical. So he’ll have, you know, some, some professors on that will talk about what is intelligence, what is sentience, can AI ever achieve this to the political ramifications of how do we govern this? How do we write policies to control this? Where do we need policy to control it? To history. Talking about AI summers and AI winters of innovation and how, how we got to where we are and what the weaknesses and strengths of where we are and capability is. And even economics. So, you know, how, what’s this going to do to jobs or what do we think it’s going to do to jobs? What’s it going to do to the stock market? All of these different things covered in this series. Really big thinking, I think, from some of the smartest people in the world as guests. And I really love this podcast. Like I said, it’s my second time through it because I think there’s so much to think about and see differently than, than the AI software where we’re interacting with every day. The more larger ramifications of all of this stuff, which is, which is really cool. 00:53:05 | Michael Helbling: Nice. I’ll have to give it a listen. All right, Julie, what about you? What’s your last call? 00:53:09 | Julie Hoyer: My last call, funny enough, is actually a David Epstein letter newsletter. And it’s him and Brad Stuhlberg. And it’s them discussing two books that they recently published earlier this year. The books are The Way of Excellence and Inside the Box. And it’s a really cool discussion back and forth and a really nice highlight of both of the books. And it really made me want to go and actually get them and read them in full. And the title of the newsletter is How to Pay Attention and Care Deeply in a Chaotic World. And so I’ll just give a few teasers of like quotes just from the beginning of it, but it was such an interesting read. And like I said, I think that the books diving into these concepts more would be super interesting to go and check out. So the first one is the quality of your attention shapes the quality of your relationships, work and leisure, which is to say the quality of your attention shapes the quality of your life, which really like hit home for me. And then they talked about it’s changed from time management to attention management. And then the last one I’ll give you is Wealth of Information Creates a Poverty of Attention. So like those three things just like really caught my eye. So highly recommend reading the newsletter. And then I would not be surprised how many people go and purchase the books for more. 00:54:35 | Michael Helbling: Yeah, Julie, if we’re telling people to like trim down, like I think that goes against what we’re doing here. 00:54:42 | Julie Hoyer: But it’s over. It’s about quality, right? We’ve got quality here. It’s okay. Yeah, yeah. That’s right. 00:54:47 | Michael Helbling: Thank you. Appreciate you. Yeah. Keep listening to this, but cut everything else out. Everything else trash. 00:54:54 | Julie Hoyer: We need your full attention, people. That’s right. 00:54:57 | Michael Helbling: Val, what’s your last call? 00:55:01 | Val Kroll: Okay, so mine is a New York Times article that my husband sent me who is a philosophy major. And the title is The Revenge of the Philosophy Majors. And it talks about how a lot of AI companies like Anthropic and Google DeepMind are recruiting philosophers to join their kind of crew and some of the things that they’re really leaning into as some of their like skill sets that are different or have a different perspective or angle than others is around like obviously the ethics, but like being able to take like abstract concepts and make it easier to digest like all those like conceptual challenges around AI. But the one thing that I found really interesting, I mean, it’s super interesting and they have like, you know, different quotes from different people. I was like, oh, like what are they called like forced to play philosophers? Like thinking of being so funny. And I was like, no, they’re just philosophers. Like it’s just philosophers. That’s the title. It’s like, okay. Wow. All right. Well, there you have it. But it’s just so cool to think about like how AI and, you know, some of these roles and how it’s kind of morphing and changing what is seen as mission critical to like move us forward. But it was kind of a fun read. So that’s my last call. 00:56:22 | Michael Helbling: So you’re telling me they can’t be advanced philosophers? 00:56:27 | Val Kroll: Senior advanced. 00:56:29 | Michael Helbling: Advanced philosophizing. Sort of like how we do an advanced analytic. Yeah, exactly. All right. Sorry. 00:56:37 | Val Kroll: That’s good. No, good. Those data philosophers. Yeah, exactly. Yeah. 00:56:45 | Rusty Rahmer: I’ll tell him that he’ll love that. 00:56:47 | Val Kroll: Good data in front of it. 00:56:50 | Rusty Rahmer: Just to build on that for a second. There has been a longstanding philosophical test. I mean, I mean, they’ve been wondering if computers will ever be able to imitate a human for a long time, ever a chief sentence or whatever. And the longstanding test has always been if a human in a blind test cannot tell that they’re having a conversation with a computer, that has been the philosophical bar that sentience has had to reach in computers. If he could do that, then it would be blown that away already. And so the need to define what is it that makes us different than a computer is like a huge philosophical debate right now. And I’m sure that gets right into it. It’s really exciting times for that. 00:57:30 | Val Kroll: How about you, Michael? What’s your last call? 00:57:32 | Michael Helbling: Well, mine is a little more fun this time, but I do love a good allegory. And a blog post that Charlie Tyson worked with shared with me by a guy named Gerard Rubio. It’s sort of an allegorical story about startups. And I thought it was well written and kind of a fun read. So we’ll have a link to it. But basically, he’s sort of talking about a guy who’s decided to start an oven business and goes off into all the different things that happen to a founder through that process. And I think now in this day and age, because it’s so easy to do something with AI, I think everyone sort of has this thought of like, oh, I should start a company. And then this will sort of get you through probably the first few years of that pretty easily. And if you were thinking about doing a product. So it’s a good read. It’s a fun little read. All right. Well, Rusty, again, thank you so much. What a pleasure to have you back on the show and hang out a little bit. It’s always great. I appreciate the insight and the wealth of experience that you kind of distilled into your book, Working As Designed, which is available, I believe, on all the major platforms. So you should probably pick up a copy just so you can say, I have a book by Rusty Raymer. And also read it. Yes. Yeah. So that’s probably more important than say you have it. But anyways, all right, I know as you’ve been listening, you’re probably thinking, hey, I’ve got a couple of questions or things like that we’d love to hear from you. And the best way to do that is through our website or our LinkedIn, or you can email us at contact at analyticshour.io. And also, as you’re out there listening, please add a review or comments also where you listen on Spotify or wherever. So we’d love to hear from you or see reviews and ratings is there as well. Always is awesome. All right. 00:59:34 | Michael Helbling: So I think that’s it. 00:59:37 | Michael Helbling: And I think I speak for both of my two co-hosts, Julie and Val, when I say no matter what your organizational structure, whether you’re a center of excellence, a Tiger team, or a cross-functional porridge, or you’re defining two big customer journeys, keep analyzing. 00:59:56 | Announcer: Thanks for listening. Let’s keep the conversation going with your comments, suggestions, and questions on Twitter at analyticshour on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst. Those smart guys wanted to fit in. So they made up a term called analytics. 01:00:18 | Charles Barkley: Analytics don’t work. Do the analytics say go for it no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 01:00:33 | Michael Helbling: You do in the French Quarter, nothing crazy. 01:00:37 | Michael Helbling: And we had come out of a place and there was a lady that was really pretty inebriated, I would say. 01:00:45 | Michael Helbling: And the rest of us were just like, oh, well, and we kept going. But Rusty, because he’s such a nice guy, stops to try to help her into a cab and make sure she’s okay. And she turns around and she goes, you have the most beautiful salt and pepper hair. 01:01:04 | Rusty Rahmer: Me, Marles, really, her two eyes are going different directions. Yeah. So obliterated, right? 01:01:12 | Michael Helbling: It was something so funny. Because I was like, oh, yeah. But Rusty, because he tried to be nice. And this lady’s like, oh, wow. 01:01:20 | Julie Hoyer: But your image cut through the fog that she was in when she saw you clearly. 01:01:27 | Rusty Rahmer: Hold five with me, I’m sure. 01:01:31 | Julie Hoyer: There’s a whole group of good-looking men that help me into a cab. 01:01:35 | Val Kroll: Yeah, I think one of the stories she tells. 01:01:39 | Michael Helbling: Through this day, he has this image of her guardian angel. Amazing. 01:01:46 | Rusty Rahmer: Convinced she went to hell, unfortunately. 01:01:50 | Michael Helbling: Anyway, good times. 01:01:53 | Tim Wilson: I can, I can, I can hop into the PIVOTS FOR ANALYSIS on, so, listen. 01:01:58 | Rusty Rahmer: I have to write a couple more books, at least, okay? Yeah, at least. 01:02:03 | Michael Helbling: I’m on it. 01:02:04 | Rusty Rahmer: My movie is next, by the way. So that should help me. Oh, well, that’s perfect. Perfect. The analytics movie. That’s what I want to hear. Yeah. 01:02:14 | Val Kroll: Nice. The red stapler becomes, what would it be? I was told I could have access. 01:02:21 | Rusty Rahmer: Yeah, I was told I could have access to the snowflake tables. Yeah, and if I could just, if I could please give me back my access to the snowflake tables. 01:02:29 | Tim Wilson: Oh, Gershoff. Can’t you hear it? It’s a whole movie, I’m telling you. Yeah, it’s a tie between Gershoff, Joe Sutherland, and Gary Angel. Oh, Joe, yeah. Joe. 01:02:37 | Michael Helbling: Oh, we. I feel like there’s somebody else. How many times have each of them been on? Three. Haven’t we had Chelsea on three times, too? I thought so. Yeah. Because we did two in quick succession last year with her. Well, this is what I always try to do analysis on. 01:02:57 | Val Kroll: Yeah, is this like the high problem of like 2010, where it’s like capitalization, so it doesn’t. 01:03:04 | Rusty Rahmer: Look, it’s like a real company here. Tim does all the work, and he’s got four bosses. Tell them what he did wrong. 01:03:11 | Michael Helbling: You know what? Meanwhile, meanwhile, Tim’s over there typing into Claude, and Claude’s like, you’re absolutely right. I did forget about that, guest. You’re right to bring that up. You’re right to call me out on that. Oh, that’s right. 01:03:29 | Val Kroll: One was like the last time. How many times have Tim and Moee fought or something? 01:03:35 | Tim Wilson: Oh, what triggers Tim? Like, those are always. Does it take up on notes and argument? 01:03:43 | Val Kroll: It’s pretty good. The post #303: Funnels Assume Progress. Barriers Recognize Reality. appeared first on The Analytics Power Hour: Data and Analytics Podcast .

July 21, 20261 hr 8 min

#302: It Was a Dark and Stormy Insight…

Here’s a question analysts almost never ask themselves before walking into a room: why am I actually here? Not “because it’s the weekly meeting” or “because someone asked me to pull the campaign readout.” But WHY—what difference will the information coming out of my mouth make, and for whom? Aleya Harris , bestselling author, TEDx speaker, and strategic storytelling advisor, joined Tim and Moe to dig into exactly this kind of thing—and she did not pull her punches. Analysts are often the smartest people in the room, she says, and that might actually be the problem. The gap between “here’s what the data shows” and “here’s what we should do about it” is precisely where story lives, and it turns out storytelling isn’t some soft, hand-wavy thing marketing people do—it’s a repeatable, learnable framework that has been working on human brains for millennia. Plus: a cautionary tale about ranking on page one for the wrong keywords, the tweaking-the-deck death spiral decoded, and an elevator pitch exercise that reveals your carefully crafted slide deck says something completely different than what you actually think. This episode is brought to you by Prism from Ask-Y —your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. This episode is also brought to you by Stape , your all-in-one solution for server-side tagging. Links to Resources Mentioned in the Show Hero’s Journey (Book) Heal Your Body by Louise Hay Ladies First: Wildlife Matriarchies (Video) You Are Contagious – Vanessa van Edwards at TEDxLondon (Video) What If Your Identity Is a Script You’ve Rehearsed Too Many Times? – Aleya Harris at TEDxChantilly HS Photo by Nong on Unsplash Episode Transcript 00:00:00.00 [Announcer]: Welcome to the Analytics Power Hour. 00:00:08.92 [Announcer]: Analytics topics covered conversationally and sometimes with explicit language. 00:00:13.32 [Tim Wilson]: Hi everyone, welcome to the Analytics Power Hour. This is episode number 302. I’m Tim Wilson and I thought I would kick things off with the start of a poem by Edgar 00:00:26.68 [Tim Wilson]: the Analyst Poe. 00:00:29.52 [Tim Wilson]: Once upon a midnight dreary, while I pondered weak and weary, over many a deck and query heaped upon the office floor, while our business partners barely waking, kept on nodding, kept on faking interest in the point I’m making till they wandered out the door. Cry die, surely truth and data should compel them to the core. But these are just facts and nothing more. Okay, if you’re nervous then I’m going to do a raven length recitation, have no fear. We’ll leave it at a single stanza. What we’re going to chat about today though is that pesky reality that while data and its analysis is valuable and is our bread and butter, the data alone is rarely enough to drive positive behavioral change. We need to be compelling to alter the beliefs of our business partners in order to motivate change, and that requires a lot more than just the results of our analyses. For that discussion I’m joined for this episode by my co-host, or by my co-host, my co-host, Moe Kiss. Moe, are you pronouncing any words funnily like I am today? 00:01:38.24 [Moe Kiss]: To be fair, I’m not starting off with a poem with a dramatic reading, so I feel like my job’s a lot easier today. 00:01:46.36 [Tim Wilson]: I also had a dog that we’re dog-sitting walking underfoot throughout all of that, so if I held it together, good for me. 00:01:54.96 [Moe Kiss]: Excellent. 00:01:55.96 [Tim Wilson]: We were also going to be joined by Val Kroll, she unfortunately is under the wetter weather at the last minute. Wetter? 00:02:02.44 [Aleya Harris]: Oh, this is not going to be good if we’re going to talk about diction and using appropriate 00:02:07.32 [Tim Wilson]: words. So, what we thought we’d do was get a professional who actually doesn’t butcher the English language on a, at a moment’s notice, like I am doing so far. So, we recruited a guest for this discussion after we saw her totally own, I mean like dominate, own of the room at Marketing Analytics Summit earlier this year out in Santa Barbara. Aleya Harris is a strategic storytelling advisor for leaders and organizations. She wears many hats at the Evolution Collective, including both CEO and program developer and the lead trainer for Spark the Stage Signature, where she builds your keynote so you leave with a finished presentation, a professional speaker platform and skills to own the stage. She sets a high bar for how she owns the stage. She’s a best-selling author and a TEDx speaker and today she is our guest. Welcome to the show, Aleya. 00:03:00.20 [Aleya Harris]: So much for having me, I mean, I really love speaking at Analytics Summit, but I think I might have loved that poem. Maybe just a little bit more, so. 00:03:12.80 [Tim Wilson]: That was one where there was an AI assist, but then I had to do a heavy edit, so I did practice my dramatic reading a couple of times, so. 00:03:22.44 [Aleya Harris]: And you could, it showed, it was quite good. It showed. And then I also would like to lower expectations for all of your listeners. I make no guarantees that I will say all of my words properly either, so just in case it doesn’t have to require me. 00:03:38.00 [Tim Wilson]: Well, I started by setting the bar pretty low, so, well, well, we’re thrilled to have you and when we. Thank you for having me. Well, so when we saw you speak, one of the things you really hammered on pretty hard is it’s something that I think we all sort of know, but then, like, as analysts, we like to ignore, and that’s that, you know, just the facts, like just delivering the data is not only is it not effective, I think it’s uniquely ineffective when it comes to actually getting somebody to make a decision and drive change. So if that’s a fair way for me to put words in your mouth, can you maybe talk about that a little bit? Like, why are stories so critical when it comes to delivering information in a business context? 00:04:27.72 [Aleya Harris]: Because you put words into my mouth beautifully and wonderfully, I’m here for all of that. So the reason why it’s important is because stories are how our brains work, they function in story. You can give me 10 bullet points of data, but you can tell me a story and I might remember one bullet point of data, but I will always remember the story. I tell this story, which is a true story, about how I, in not a, I say a former life, but then I’m also very woo, so I think sometimes people mean like I meant like, you know, back in the 1800s when my name was Melissa, but no, I mean, like, earlier on in my career, 00:05:08.24 [Aleya Harris]: I was a private chef. 00:05:11.40 [Aleya Harris]: I was a private chef for lots of celebrities and one of them was Stevie Wonder. And one of my favorite moments was we were in Utah because he was singing with the Moermon Tabernacle Choir, of which I had a front row seat to their rehearsal concerts, amazing thing. But I remember the night before, we were all a bunch of us were in his hotel suite. I was sitting on his bed and he was like doodling up a song on his phone. And I started like doodling it with him and I was like, oh my God, I’m making a song with Stevie Wonder. 00:05:54.32 [Aleya Harris]: Is that, is that one? 00:05:56.32 [Aleya Harris]: Don’t worry, it’s not on anybody’s record. I asked if we were going to like get this out into the world, he, he promptly laughed at me. But then the more that I really honed in this situation, I mean, he was adding in lyrics and beats and melody and we were like just doodling up a song and I took me a second to remember that this man is not sighted. 00:06:21.12 [Aleya Harris]: Right? 00:06:22.12 [Aleya Harris]: Like, I was looking at him, doing everything he’s doing, thinking it’s all normal, talking to him, engaging with him like I would a sighted person, but he isn’t one. 00:06:34.72 [Aleya Harris]: And I took a pause for a second to be like, wait a minute, he never put limitations on 00:06:42.52 [Aleya Harris]: himself. Yeah, the Grammy Awards, the fact that he’s like a musical genius, blah, blah, blah. But one of the main moments that I remember about him, about him reaching out possibilities was us making this song with him, doing all this stuff on his phone as a non sighted human. And the question I asked myself was, where have I been letting excuses hold me back? I have all of the resources, all of the things that I can do with myself. He has a severe impediment compared to a lot of other people and yet this man has a household name across the world and it was in that moment that I decided that I actually didn’t want to be part of Stevie Wonder’s entourage. I didn’t want to cook anymore. I wanted to be an entrepreneur and not be part of someone else’s entourage but develop one of my own. 00:07:39.44 [Aleya Harris]: Right? 00:07:40.44 [Aleya Harris]: But it was that moment. Now, I sat and I told you the story. Your listeners here have heard this story. Instead of telling you a story about how overcoming obstacles and watching someone else do it 00:07:59.16 [Aleya Harris]: inspired me to overcome those obstacles, inspired me to overcome those obstacles and 00:08:05.20 [Aleya Harris]: become a business owner, I could have given you a bunch of data around 50% of people that have obstacles to overcome, do it in these ways, step one, two, three, 25% of this, expert gender this, if we have this large demographic percentage of this, da, da, da, da, da, da. Or I could tell you my Stevie Wonder story. Which one do you think is more compelling? Which one do you think is going to inspire more people? The story or the bullet-pointed list of data? What do you think will actually inspire people to participate in their own transformation, which is what I’m in the business of doing? And which spoiler alert? All analysts are in the business of doing. That’s why you’re analyzing things in first place because you’re trying to make something happen. You’re trying to illustrate, hey, something’s going on. There’s connections that need to be made here and there’s things that need to be done because of these connections. 00:09:03.04 [Tim Wilson]: I mean, not to over-index to that story, but the world of overcoming obstacle, you had a, I don’t know if you could fully plot a hero’s journey story out of it, but is there a part of it too that even as you’re telling the story, I was finding myself saying, well, I know where we ended because you’re not doing that anymore. So I’m listening, because it’s got a narrative form, I have no idea if this is right or not, 00:09:36.16 [Aleya Harris]: I know it’s going somewhere, because it’s a narrative, I know it’s going somewhere. 00:09:43.48 [Tim Wilson]: So I have to be on the edge of my seat because I know, whereas if I’m looking at a chart or I’m getting the weekly readout, there’s this nature of, and then there’s going to be another slide and then there’s going to be more data where you clearly with a story you’re on a progression to somewhere, which makes it easier for the audience to pay attention. 00:10:10.32 [Aleya Harris]: Stories, unless you’re like Uncle Bob at Thanksgiving, which I talk about all the time, and I had an Uncle Bob. Uncle Bob wasn’t actually the problem, Uncle Bill was the actual problem, Uncle Bob was fine. I have Uncle Bob and Uncle Bill. Uncle Bill was a real long-winded, but anyways, I digress about my uncles. Unless you’re that person at Thanksgiving or insert holiday dinner here where you’re just like, oh my God, you’re talking about absolutely nothing, this isn’t a story, this is word vomit, you know that if someone is telling you something in what you said was the magic words, a recognizable story structure, you’re hooked. So when people say story is important, they think words are more important than numbers. That’s what they hear, that it’s words that are important and words are important, but it’s not just words, it’s words put in a certain sequence, in a certain framework, in a storytelling framework because that’s how our brains work. So yes, you knew there was going to be a payoff because I intentionally was telling it in 00:11:16.44 [Aleya Harris]: a storytelling framework. 00:11:19.68 [Aleya Harris]: You knew, oh, this is familiar, this feels like a movie I have seen, a book I have read. She, unless she’s like really bad at what she does, is not going to leave me hanging and just being like, yes, so what were we talking about, I just wanted to tell you that, like, 00:11:34.00 [Aleya Harris]: I thought like pajamas are cool, like, you know what I mean? 00:11:39.00 [Aleya Harris]: Like you knew I was going somewhere with this because I used something that your brain recognized, which meant that you were paying more attention, you were also cataloging the details of the 00:11:49.76 [Aleya Harris]: story just in case they were important later, which is what you want people to do with your 00:11:55.56 [Aleya Harris]: data. What do you need them to understand, know, and believe about this data that you’re weaving into a story because it’s important for them to use it to make decisions with later. So you’ve entered into my very geeky favorite territory of storytelling frameworks because stories all exist in a framework, good stories. The most popular one that we deal with is the hero’s journey, like what you mentioned, spot on. There are lots of other types of frameworks and each framework allows you to elicit a different type of response and help people to behave in a different way that you get to, this sounds a little bit like a bit of work with me, you get to control because you are controlling the story structure, you control the story structure, you understand the certain beliefs that people have and they hold as they walk through the world, which means that you can control the messages and the narrative you give around certain data so that you get the end result that you’re looking for. 00:13:02.24 [Michael Helbling]: Tim, here’s a common measurement problem. The tracking event you care about does not always have the user information you need. 00:13:14.12 [Tim Wilson]: Right. You can’t identify as themselves once, maybe with an email or phone number, but later events may come through with missing or incomplete data. 00:13:18.64 [Michael Helbling]: And that’s what Stape Enricher is designed to help with. It’s a state power-up that securely stores selected user data from incoming requests and uses it to enrich future events when information is missing. 00:13:28.32 [Tim Wilson]: So instead of every event standing alone, and Richard can help connect activity across sessions, devices, and even online and offline touch points. 00:13:36.00 [Michael Helbling]: Exactly. That means better event match quality, stronger attribution, and a clearer view of the full customer journey. 00:13:43.12 [Tim Wilson]: And importantly, Enricher is built with privacy and consent in mind, including settings that align enrichment cookies with your site’s marketing consent preferences. 00:13:51.52 [Michael Helbling]: And teams can even upload historical data from tools like Shopify, Clavio, CRM exports, or offline customer lists. So if your measurement has gaps, Stape Enricher can help fill them responsibly. Use the link in the podcast description and check out Stape Enricher today. 00:14:11.84 [Tim Wilson]: Michael, Ask Why’s Prism version 3.0 is here and it officially turns the analyst into the admin. 00:14:20.16 [Michael Helbling]: Ooh, I have always wanted power. Not delete the database power, more like stop asking me which revenue number is real power. Exactly. 00:14:31.24 [Tim Wilson]: With AskWhy.ai, Prism becomes a real workspace platform. 00:14:35.28 [Michael Helbling]: You manage the data, control the context, and empower users wherever they work. So I’m not just answering questions, I’m curating intelligence. Yes, the new memory portal is mission control for what your agents know about your business. Meaning I can search the memory, endorse what’s right, reject what’s wrong, and tune it in plain English. Yep. Importance, trust, and decay scores adjust automatically. Ooh, decay scores. Finally, a polite way to say this metric definition has gone bad in the fridge. And context notes are now refreshable. 00:15:11.36 [Tim Wilson]: Upload playbooks, call transcripts, glossaries, business rules, then refresh them instantly or on a schedule. 00:15:16.76 [Michael Helbling]: I like that. So no more stale context haunting every analysis like a ghost with a spreadsheet. 00:15:23.28 [Tim Wilson]: Go to ask-y.ai and join the wait list and use the code APH to get up at the top of that list. 00:15:29.84 [Michael Helbling]: That’s ask-y.ai with code APH. 00:15:34.12 [Moe Kiss]: Can I dig into that a little bit? Because I’m just going to talk about the problem out loud and then we can kind of have a bit of a discussion about it. One of the things that I observe with data people is that they think the data should speak for itself, right? And what we’re discussing here is really that the narrative in the story is the more important component. But how do you think data people then weave the data into the story successfully? Because obviously that’s the specialized skill that we bring to the table. That’s the information that folks don’t have. And they might not remember all of the data points. You want them to remember the story, but how do they pull those things together well? 00:16:14.68 [Tim Wilson]: Well, I’d also say they’re expected to bring the data. A lot of times they’re dealing with kind of the norms of the organization. If they showed up and went, no data compelling story, that feels super, super risky. So yeah, that’s my addendum to your question. 00:16:31.00 [Aleya Harris]: I love your question, and it is definitely how data people think. 00:16:36.72 [Aleya Harris]: And it is a fundamentally flawed question in its structure. And I’m very glad you asked it this way because this is how it’s asked all the time. How do I weave data into story? What you really need to ask yourself is, how do I create a story from this data? What is the data trying to tell me? What am I not only extrapolating, but how do I use the key numbers and the key points along the journey of what people need to do in order to use this data? So when you’re thinking of a perspective, let’s say you have a dashboard, 00:17:21.52 [Aleya Harris]: you have website analytics in front of you, or you have any data set, really. 00:17:27.56 [Aleya Harris]: And you guys have actually the hard job because you have to know all of the technical part. And you’re like, great, I just spent a longer than necessary, exorbitant amount of time pulling together this data, making sure it’s clean, making sure it’s correct, making sure it’s usable data. And now I got to explain it to you, which is where I think some of this data should speak for itself comes from because I’m done. I am now done with this. Now shouldn’t this be your job when actually, no, it’s not. You put a bow on your work by saying, OK, let me take a deep breath. I now have this information. What am I going to do with this? If I have this information, how am I using this to make business decisions? How should somebody use this to make business decisions? If I have SEO metrics, I have a bunch of, you know, these are just hot keywords. These are how people are finding you. This is the search. What is this really saying? Well, it’s saying this is the problem. This is the solution. This is where we are in the journey. This is where we’re trying to go. The challenge with a lot of data people is you are the smartest person in the room. I said it. I’m going to keep saying it. I said it in the conference. I’m going to say it again. Right. And so you’re like, well, isn’t it obvious? Isn’t it obvious what this says? No, ma’am, it’s not. Sheila, it’s not obvious to someone who is not you. So you need to make what is obvious to you obvious to other people. And the way that you do that is by busting out of your silo and asking questions that might even seem unrelated to your data. Like, what are our business goals? What are you trying to do with the website? What are our larger marketing initiatives for this next year? Right. What are we struggling with as a company? And then you look at the data and you’re like, OK, I got it. So like this happened to me where I do not pretend to be an analyst, nor to play one on TV. So let’s get this real clear before this next story. But I know enough to make me dangerous. And I got from my SEO person a very nice dashboard of which he didn’t do what I said. And he just sent me a bunch of keywords and numbers and rankings and all of this stuff. And I looked at it and I said, no wonder why all of our marketing is going nowhere. We’re doing because he goes, you’re doing great. You’re ranking really well for I’m not going to use actual keywords. But I want to out the company for apples and oranges and plums. That’s awesome. You’re a fruit seller, right? You want to be waiting for apples, oranges and plums. This is great. To which I replied to him the equivalent of, well, we’re actually not like a fruit seller in that we want to want to be known as a fruit seller. We’re actually like a fruit package. Like we sell the fruit packaging like the bowls, not the fruit. So ranking for apples and oranges and plums might sound like we’re the number one person 00:20:48.68 [Aleya Harris]: that when you Google apples, you’re going to find that doesn’t make us anybody. 00:20:55.56 [Aleya Harris]: So you then as the SEO person need to be able to look at data because you know that we actually want to be ranked for the bowls, not the apples. And it’d be like, oh, I understand our marketing objectives. I understand our business objectives. I actually know that this data is bad, says bad things, even though we’re number one. Page one, the SERP is bang and our name is everywhere for the wrong thing. Our content strategy was wrong and bad and all over the place. And wouldn’t it have been nice for my SEO person to know that before they came to me 00:21:28.12 [Aleya Harris]: who was the CMO? 00:21:30.24 [Aleya Harris]: So excited about their data. 00:21:31.56 [Tim Wilson]: I mean, I think you’re hitting on some like very fundamental things. I think there are times where it’s, I have this data, I have been told to crunch this data. I’m the expert on this data. And therefore I need to crunch the data until I find something that I can slap a label of a story onto. And I will claim hard that there is a lot of data that is just showing things are going as they expected. And that’s like a massive issue with the industry at large is there’s a belief that if you pulverize the data enough, something will emerge. And I think analysts wind up in this spot saying, well, I got to do a deliverable. And so then they start slapping. What story can I tell, but their story winds up being, oh, look, you’re ranking for this irrelevant fruit, you’re ranking for fruit when that’s not what you’re selling. But that kind of pairs with the flip side, like really asking not starting with, not even starting with the data, I would think it’s not start with the data and what stories is telling me. It’s starting with what is the real problem I’m trying to solve or what is the real idea that I’m trying to validate? To me that, and then I may go to the data, but then I’ve got kind of the kernel of a story because I’m trying to, I kind of know what the front, kind of what a couple of alternate endings, the ending is either go spend more money there or spend less money there or whatever it is. I’ve kind of outlined some possible endings to the story. Then I’m using the data or the experiment or the research, whatever it is, I think. I mean, I just feel like analysts find themselves saying, they sort of paint themselves into a corner and then they say, well, I don’t know, I don’t have a story to tell. It’s like, well, yeah, because you kind of painted yourself into a corner to start with. I got triggered on a few fronts. 00:23:25.68 [Aleya Harris]: You know, I don’t know, I didn’t know analysts were so judgy. 00:23:30.20 [Tim Wilson]: I’m pretty judgy. That’s not, that’s not paying too broadly, but I say that, I say that in the way like 00:23:40.52 [Aleya Harris]: you’re judging your story as your data is bad, is it telling like a bad or unnecessary story? Just tell the story. Tell me what the story actually is and allow it to be. Tell me what then you are now, what does that make you as an analyst more curious about? Okay, everything’s going like how it’s supposed to. Is this the direction that we want to take it for the next five, 10 years? Tell me now, CMO, this direct, this is what you told me you wanted to know. We’re ranking for these things. We’re ranking for those things. It’s all good. I can leave it at that. Or I could say, yeah, we’re ranking for these things. And what does that mean? Often analysts are supposed to be the source of truth and I would love, would have loved it and love it more when I’m working a fractional basis as a CMO. 00:24:44.36 [Aleya Harris]: If analysts asked me more questions, that, I mean, I guess depends on the personality 00:24:52.12 [Aleya Harris]: of whoever you work for, right? But if the analysts that I have had had been like, great, here’s the information. It’s what we expected. This is ranking here. This is what this is saying. This is the story that is telling the story that we are the best in the market. We are the worst in the market. We’re ranking high for these keywords. The money that you wanted to spend on additional, whatever you can spend, great. I would love then the next thing that it’d be like, now tell me, this is great for right now. 00:25:25.72 [Aleya Harris]: What about next quarter? 00:25:29.72 [Aleya Harris]: Is this what we’re still trying to do because then I can start monitoring this in place and then use your freaking brain to do the analysis, this is how you get to do the analysis that you want to do by asking you the questions in advance and not being so reactionary. So then you’re not just the storyteller, you then become the story author, which makes 00:25:49.56 [Aleya Harris]: you a more valuable employee for me as a marketing lead because you’re now my partner. 00:25:57.08 [Aleya Harris]: You’re the person who is asking because I might know what I’m doing, but I might not know what I need to be looking at to know in advance what the signals are to tell me if this I should be moving in this direction. If if I’m moving in a certain direction, the early signs that it’s working or not working, you know that. So ask me the question so you can know what to monitor. So you’re not coming back to me every month saying looks great because I’ve had that too. 00:26:23.92 [Tim Wilson]: It’s annoying. I might have tried to start an entire consultancy based on the premise that you need to do more talking and interrogating of the ideas of the people in the company and before you jumped into the data. So I am my my heart is singing because you just sort of put on another hat of like the CMO hat. I think analysts, we have there’s this wiring of listening for a data poll and wanting to get out of the room as fast as possible. Like you start to say, well, I kind of think our SEO and they’re like, great, I can go pull an SEO report as opposed to stopping and saying, I don’t think you really give two shits about your SEO. Like what are you really trying to do? You’re trying to do X and Y and Z and and I think that the business, the CMOs and the other business partners, they try to meet the analysts where they are by using their terminology and the analysts mishear that as being overly prescriptive. 00:27:20.40 [Aleya Harris]: Okay. 00:27:21.40 [Tim Wilson]: Can I pivot things a little bit? 00:27:25.08 [Moe Kiss]: That’s okay. 00:27:26.08 [Aleya Harris]: I’m right there with you. 00:27:27.60 [Moe Kiss]: You just said something to him that I want to I want to dig into you. They want to get out of the room as quickly as possible. And one of the things that I just like really observed both watching you present and talking to you now is like this absolutely infectious enthusiasm and just like Tim said in the intro, right? Like you own the stage. And I suppose one of the things I’d love to get into is like, yes, the story is important, but it’s also how you tell it, especially if it’s like in a presentation format. So like how did you develop that skill? Like I’m someone who didn’t used to do public speaking. I was petrified of public speaking. I had to really put in a lot of energy and learn how to do it. And I feel like maybe we had a similar tact where like, I just took all my nervous energy and tried to like channel it into enthusiasm. But I do feel like there are so many, so many folks who like, it is such a daunting thing. And like, if you tell the story in a really average way, people probably still won’t listen, right? Like they might remember a bit of it, but it’s like so much of why you’re persuasive is how you tell it. So like, can you share like your tips and tricks on that part? 00:28:42.40 [Aleya Harris]: Well, first, that is true. But the framework, once you learn the actual storytelling frameworks, it does a lot of heavy lifting for you. Even if you have a monotone voice, even if you’re not super energetic, the framework does a lot of heavy lifting. So when I work with people and I help them put together their keynotes and I help them get on stage and feel confident on stage, I always ask them to create videos and we could cheek them and they’re in delivery and the very next thing I say is, and don’t try to be me because if you have a great gregarious personality, then great. Like I am like this all the time. I don’t, this is not like how I am on stage, how I am on the body. I’m like this all the time. It’s a lot. I know, but it’s, it’s, it’s up to how I am on stage because that’s how I am in life. I don’t want an introvert who is normally a much more serious, you know, put together individual to try to then emulate my style. They’re going to be awkward and everyone’s going to feel awkward in the audience. Right. That’s not going to work. So I want them to use a storytelling structure and then find their voice. A lot of times people feel insecure with speaking because there’s a lack of clarity and a lack of clarity causes and or masquerades as a lack of confidence. And there’s more that you need to be clear on than you think that you do. It’s not just, what am I going to say? Like, what am I bullet points? What’s my message? Like that’s very important, but you need the more data points. You can be clear on having, I couldn’t help myself, the better. So yes, what is your message? Are you crystal clear? But why are you delivering that message? Why are you standing in that room right now? Why are you the only person that can deliver that message right now to these human beings? Why do you deserve to be listened to? Why is that message at this time going to cause someone else to participate in their own transformation? Those are the types of things you get clear on. And so it doesn’t matter whether you are a jokester. It doesn’t matter if you’re serious. 00:31:13.28 [Aleya Harris]: It doesn’t matter if you have some other personality type. 00:31:20.04 [Aleya Harris]: If you are clear on why on the wise, you’re clear on the what, which most people actually are a lot clearer on the what than they give themselves credit for, right? Like, you know what you’re going to say is not that’s actually the easiest part. It’s the wise that they need to work out. And then once you do that, then you get clear on what I do with my hands, right? Your physical body in space. How do I want to show up? If I show up with my hands like this or something like this, how will I be perceived? Okay, if I show up still me, but I hold my hands out, how am I perceived? And you start then becoming curious about how bodies look in space. Like you walk down the street and you see someone with their hands on their hips. What do you think that they’re feeling? You walk down, you see someone like waving their hands, flailing in the air. What do you think that they’re feeling? Right? And you start understanding, oh, I’m actually clear on the story that bodies send. So now I’m clear how to hold myself in space. And now I’m clear on why I’m there. I’m clear on what I’m saying. And most importantly, I am clear that, and this is true again, no matter what you’re talking about. 00:32:44.36 [Aleya Harris]: I am clear that in this moment, at this time, I will make a difference in someone else’s life 00:32:54.28 [Aleya Harris]: with what comes out of my mouth. I will pause it, period, not trying to like gas you up. It’s just kind of like it is what it is. We can get metaphysical about it if you want to. Just believe me, trust me, this is not that long of a podcast. You will make a difference in someone’s life with whatever’s coming out of your mouth right now. So don’t worry about you being effective or not. Worry about the other stuff and getting clear on how you will be the most effective you can possibly be on that stage with whatever time that you are given. And everything else will work itself out. It’s amazing as I work through these things with people, with my clients, the clearer we are, 00:33:36.88 [Aleya Harris]: the more confident they get. 00:33:39.60 [Aleya Harris]: In the beginning of my program, I don’t talk at all about how much your rates should be, when you charge a speak, I don’t talk about speaking goals, I’ll talk about another. There’s no goal setting until the end. And I one time had a client ask me or give me feedback, like, well, that seems a little backwards. Like I’m used to, you know, doing goal setting at the beginning and it would have been really nice to have my goal set in the beginning. And I said, I get that, I type A too. But if I asked you in the beginning, when you were unclear about who you were, why you were there, if you had value, if your stories mattered, if you mattered and you were having like an existential crisis that you didn’t even know you were having, and I asked you how much you should charge to be on that stage, you would have said I should pay them to let me have the honor of opening my mouth. Now you’re sitting here asking me if you should start off at five or $10,000 to charge, to allow them to earn the right to listen to your wisdom, all because you got a lot clear on those things that I was talking about. 00:34:44.56 [Tim Wilson]: So that is you and just that framing is kind of around the like public speaking keynote, you know, audience sharing information. I assume that clients come out of that and it fundamentally changes the way that they, even in a conference room with five people or even potentially even on a Zoom call, how they put together, like it scales up and down, right? Like when presenting, oh, I’m just an analyst and I’m just delivering this to, you know, the marketing team, it’s not a public conference keynote, but I guess two questions, all this still absolutely applies, I’m 100% sure, but how do we get over the hurdle of saying, but this is just one presentation, I don’t have, I can’t invest the time and the effort to turn this into a proper story, I don’t have the capacity. I have thoughts on what I think the answer is to that, but I’d love to hear what you think. 00:35:58.00 [Aleya Harris]: First of all, you’re like, I’m so sure I’m like, are you there? You don’t sound sure. People think that storytelling takes a really long time and that’s actually very much not true 00:36:09.24 [Aleya Harris]: because of what I was talking about before, frameworks. 00:36:14.60 [Aleya Harris]: Once you know the framework, you start thinking, talking, acting in story without even realizing it. So it’s more about an investment. You had a great question, does this apply to me? I have a better question. Can you afford to not invest in other people understanding what the hell you’re talking about? 00:36:39.72 [Tim Wilson]: Well, that tends to be my, when people are like, I’m too busy, I just have to get stuff out the door and it’s like, well, how’s that working out for you? You’re training your business partners that you’re just there to crank out as much volume as possible. And they’re bored, they don’t know what to do with it, they can’t use it and trying to. 00:36:56.36 [Aleya Harris]: But I think the other thing you’re hitting on there is that, yeah, probably the first time you got to find the time to do a better job 00:37:05.84 [Tim Wilson]: and it’s not going to be perfect, but it will be better. And by the time you’ve done it, I love the way you framed it as saying, you get to where you just see it and think in those mode, in that mode, it totally makes sense that it then becomes easier because you’re not trying to make a bunch of slides and then go back through the slides and rearrange the slides and figure out what else you should add and not take something away. Because you’re kind of, you have the story and then it, I assume it’s been pretty quick to then say, okay, now I just got to articulate, give the visual backup to it. 00:37:42.84 [Aleya Harris]: When you’re tweaking and re-tweaking and tweaking, tweaking and then you do the tweaks for the tweak-tweak-tweak-tweak, which is like the tweak dance that everybody does, that means that something fundamentally is wrong. That means that fundamentally you are insecure about the structure, the way you’re going to present this information. You don’t know what or why? Yeah, you don’t know what, you don’t know why, you don’t know how, you don’t know nothing. All you know is you got some numbers that somebody asked you to tell them about and you have no idea how to do that. Just how numbers work in a very logical way, stories are the same. People often view them as like opposites, like words and numbers, you know, require different parts of your brain. And like what’s easier, trying to like be a number person and then transform yourself into a word person, or is it trying to make repeatable, predictable processes out of the way that words work? 00:38:54.36 [Aleya Harris]: I don’t even think we should do the second one, and that’s by using structures and framers that are repeatable, you know, 00:39:03.24 [Aleya Harris]: and it’s not like, oh yeah, they work like this is the new 2026 TikTok friend, no, they work for Eon, like millennia, right? We’ve been using the same frameworks to communicate effectively with each other. And so once you realize that, you’re like, oh yeah, I can use a story framework because it doesn’t sound so like woo woo, oh that’s what marketing like the markers do, like no, it’s how an effective leader communicates. And then if you’re money motivated, like I am thinking about it this way, if you want people to understand you as a strategic thinker, if you want people to understand you as a true people data and information leader, you should use story. Because strategy is just the ability to put a bunch of random stuff together in a framework and make it work. How are we getting from point A to point B? Oh my god, doesn’t that sound like once upon a time all the way to the end? 00:40:12.28 [Aleya Harris]: So tell me if you were a data person and you’re listening, like I feel like you’re going to have a lot of folks nodding along to everything you’re saying, 00:40:20.04 [Aleya Harris]: but like let’s say tomorrow- I should hope so, and if you’re not, rewind it, listen again, email me if you have questions, because if you’re not nodding, I’ve missed you and we’re trying to bring everyone along on this journey. 00:40:35.64 [Moe Kiss]: Do continue Moe. I love that, I love that though. So tomorrow you’re sitting down at your desk or today you’ve listened to the episode and you’re like okay, but practically how do I do this, right? Like how do I sit down? I’ve got a presentation I’m doing next week for some stakeholders on analysis of how people are using a particular product on our site. Like what do you recommend they do as they’re sitting there thinking through how to craft that story? 00:41:04.20 [Tim Wilson]: Okay, if it’s for real tomorrow, we ain’t got a lot of time, so we’ll give you the short short version. Well, they’re sitting down tomorrow, the presentation is not for a week or two. Yeah, yeah, okay, there we go, there we go, there we go, there we go. 00:41:15.40 [Aleya Harris]: I would recommend you start simple, Phil. What’s the beginning, middle, and end of the, of how the information should be used? 00:41:27.64 [Aleya Harris]: Because you’re not telling a story just, let me back up. 00:41:33.80 [Aleya Harris]: When you’re telling a story, you can tell a flat story that no one cares about, which I don’t recommend. I don’t recommend you like, you like, say, this morning I woke up, I did my skincare, and then I got dressed and I left the house. The end. Like what? Like that’s not even, it doesn’t have the elements. I mean, technically at the beginning I woke up and I said the end, 00:42:00.92 [Aleya Harris]: but there’s no purpose to the story. It’s worth thinking more when we think of stories more like Aesop Fable’s kind of situation, 00:42:09.56 [Aleya Harris]: where there’s some lesson to be gleaned and there’s a reason why you’re telling a story. So the first thing you have to sit down and figure out is why the heck am I talking? Why am I opening? Besides the fact that like, this is a weekly meeting and like I’m supposed to be here every week. 00:42:26.36 [Aleya Harris]: Like, why am I really talking? That is what I call your controlling idea. 00:42:32.52 [Aleya Harris]: What is the subject that you’re talking about? And what is your point of view? Because I can talk about like, oh, this, you know, is a dashboard and this means our product is selling well. Great. What is your point of view based on the brain that is sitting between your two ears? 00:42:58.68 [Aleya Harris]: That is the controlling idea of your story. Then you say, okay, I have the point of view like, 00:43:04.92 [Aleya Harris]: this product is selling well based on the data that I see that I’m reporting this month. However, my point of view is actually it’s our dog. It’s our worst seller. We’re having a blip month, because I know we have this major client come in and it’s skewing the data. So it shows like, this is our best product ever. But really, if you remove this outlier, this is actually our worst product. And as opposed to the recommendation of pumping more marketing dollars into this product, I actually think that we should explore either sunsetting it or repositioning it. That’s my point of view. 00:43:46.44 [Aleya Harris]: How do I know that? Because I looked at the data. Then you’re like, okay, I have my controlling idea. 00:43:53.88 [Aleya Harris]: A simple story structure is what is the problem? What do we learn from this problem? 00:44:02.84 [Aleya Harris]: And how does that relate to the broader context? So the problem is, if I was going to tell the same 00:44:11.08 [Aleya Harris]: story, well, you start kind of like in your ordinary world or whatever is going on normally, right? So the normal thing is that I’m happening is like, here I am, here’s the data, here’s what it 00:44:20.52 [Aleya Harris]: here’s saying this product is going gangbusters. But the problem is we’re not looking at the full 00:44:27.64 [Aleya Harris]: set of data. And the problem is our own success that we have this great client. But my concern is that we don’t have a reputable predictable way of bringing in more whale clients like this one who are going to regularly buy this product. So the data is skewed. So what we can learn is the data is skewed, we need to get underneath this data to see what it really is telling us and actually learn what our highest selling top three, whatever the thing is that we’re supposed to be doing here is so that we can put our marketing dollars in a place that matters, either making a reputable process for landing more whale clients that we’re going to buy more of this thing, or putting more money behind the other top three who are regularly performing well. Because I know that our company’s goal is, in fact, to sell and to go into the direction where we’re selling more and more and more of this product. My recommendation is that we, in fact, focus our marketing dollars and efforts on expanding this product line to get more clients to buy it, whatever it is. That’s how I would love if analysts came to me. 00:45:51.80 [Tim Wilson]: That was like a, that wanted to be like a detailed scenario, but I kind of latched on to your why am I, why am I being asked to, why am I telling the store, why am I presenting? Like, and if it’s, if my answer, if the analyst can’t answer the question other than saying, well, it’s the weekly meeting. And I could, I could see it’s like, well, next week you’re kind of screwed, but maybe this is encouraging you for the next time to actually go ask more questions so that you can, like if, if you can’t, that may be really uncomfortable for an analyst to say, I don’t know. It’s just because I’ve been told to give the campaign readout and I don’t, and it’s, and it’s too late. I could be like, well, then you better look ahead for the next one and kind of lay the, lay the groundwork a little more effectively with the questions you’re asking. I mean, and it doesn’t give you a sense of purpose. Like for me, it 00:46:49.96 [Aleya Harris]: drive me back to do crazy to have my thought that my job was just crunching numbers floating around in a vacuum. Like when you understand how what you do on a daily basis impacts everyone that you work with by putting money in the company’s coffers or not, doesn’t that, that feels, that would feel much more fulfilling to me. I would get more joy out of my work if I, if I knew that. 00:47:22.68 [Tim Wilson]: I think it should. I will, my other, my cynical rant number like four, I think analysts let themselves get, they tell themselves the story that that’s just my job is just to pull the numbers and just to present them. And then they tell themselves the story. This won’t be, if I come with a point of view, well, somebody may disagree with that. And then I’m going to have to have a conversation with them. And it may be that my point of view is actually wrong. And it’s scary. And it’s like, no, no, no, I just want to pull the numbers and let the, let my business partners 00:47:54.36 [Aleya Harris]: interpret it. So I’m very, this probably is snarky. And I get it. Like you’re speaking for everyone else out there, but like when I was just the word analyst, that means that you’re going to like analyze something and tell me, like if I go for an analysis, I’m assuming you’re going to tell me the thing and tell me what’s going on. You might not know how to fix it, right? You might not have all of the bells and like whistles in your toolbox to know how to fix it. But like if I go for a lab result, and like it just gives me all of like the blood count, like new numbers, I’m like, so like, do I have diabetes or not? That’s a great analogy. Like if somebody goes in, you’re 00:48:49.40 [Tim Wilson]: like, sure, my chart notified me, but I actually want to talk to the person who can actually 00:48:56.12 [Aleya Harris]: tell me how I should feel and what I should do. But case in point, the doctor is going to say like, 00:49:04.12 [Aleya Harris]: okay, yes, you do have diabetes, here’s your treatment plan, but I’m expecting someone giving me an analysis of the, of the results to do all that. But I mean, I need you to like, as my husband says, I’m always like, finish it. Like, like, like, tell me the fault of the thing. Well, and it, and 00:49:22.44 [Tim Wilson]: this is, let’s see if I can tell it generically, because it was at the conference where we saw you, and there was a session that made my blood boil, where what he’s right now, some tea, all generic tea, but it’s still, it’ll be generic to be there. Jim Stern will be I wonder which one he was talking about. Because there’s so much around AI, and somebody really stood up and proudly demonstrated how he had managed to set up, set up all these agents so that he could now take his weekly reports to all his clients. And I won’t say what the reports were about, he was kind of a consultant, and he could just automate it and it would just like, spit the report out itself. And it’s the classic, I feel like we’re starting to, people are starting to realize, I like to think I was saying it pretty early, that people were putting volume over value and what he had done, he’d taken what he wasn’t doing well, which this process he had that generated, it’s like that SEO report you were describing earlier, and he said, aha, guess what I can do? I can do that faster and 00:50:32.76 [Aleya Harris]: longer. And I’m like, that’s not actually helpful, like there’s no way. It’s not. And I think he got 00:50:41.48 [Tim Wilson]: excited because the side of him, like the tinkering and getting to like build, like sure, I like to bang out code and have it generate stuff. But we tell ourselves a story that, oh, I could even give it that it has to have a narrative form and it will generate a beginning and a middle and an 00:50:57.16 [Aleya Harris]: end. And it’s not useful. No, does it do what I said in the very beginning? Is it helping someone or something as a company participate in their own transformation? If it’s not doing that, stop 00:51:12.68 [Tim Wilson]: tinkering. But analysts will tell themselves, they’ll tell themselves that it is. And then now the other side of them. Y’all lie to yourselves a lot, as my people do in this podcast episode. Other side of their mouth, they will complain about how my business partners don’t know what they’re doing. Yeah. So I don’t think I was realizing this was going to be so much of me ranting. Can I ask, I had a more tactical… I’m actually here for it, Tim. I’m here for all of the rants. I’m thoroughly enjoying myself to continue. Listeners know that I’m not going to be the one who’s positive about anything in a optimistic. But as you were… Because as you kind of work through that scenario and you kind of just made up kind of a silly one on the fly, 00:51:56.04 [Aleya Harris]: how do you feel? I have occasionally said, I’m going to walk away. I’ve looked at the data. 00:52:01.56 [Tim Wilson]: I’m tempted to start trying to make slides and make charts. But if I walk away, I’ve got to drive somewhere. Can I just out loud, not walking down the street waving my hands where people look at me funny? But can I just talk through what I think the story is and just keep talking through it? How do I tell this story in five minutes and keep doing that and iterating until I feel like I’ve found what it is on my actual point of view is like away from the computer? 00:52:35.00 [Moe Kiss]: Tim, I feel like I do that with analysts a lot, where I’ll be like, hey, you’ve written this summary up. I don’t know what it’s saying. I want you to pretend you’re in the elevator with our COO. You’ve got two minutes from ground floor to seven. Why should he care about this? Tell me in your own words. And it’s like when people say it out loud and they practice that story, I do feel like it gets clearer much faster. And you’re like, and that doesn’t match, that doesn’t 00:53:04.76 [Tim Wilson]: match your deck at all. Like what? It doesn’t often match what they’ve written down. 00:53:09.64 [Aleya Harris]: As people put in the deck, the things that they quote unquote should be saying. The things that feel very smart and listy. But the smartest people are the ones who can make things sound the simplest. And the actual way to showcase your intelligence is to make the dumbest person in the room 00:53:31.32 [Aleya Harris]: understand the most complicated thing you do. So not saying that in this example, the CEO or 00:53:39.16 [Aleya Harris]: whoever the dumbest person in the room, however, when it comes to data, they’re probably not as 00:53:45.00 [Moe Kiss]: great as you are. They probably don’t have the depth of understanding because there are across so many things, right? So it’s like you have to bring it up to like a different altitude because they don’t have the same like subject matter expertise and like, yeah, 00:54:01.08 [Tim Wilson]: like depth of the particular topic that you’re discussing. Wow. Well, that is a great note saying that executives are stupid. No, that’s not what we said. And that’s where we’re landing. But we are running short on time. I feel like I need to go like write down a few notes immediately following this because a few little light bulbs have gone on for specific mindsets and tactics and ways to think about this topic. So thank you. We could talk for another hour and a half. But it is time for us to move to a wrap. And the last thing we like to do on the show before we close out is go around and share a last call, something that might be of interest to our users. And Alea, you’re our guest. Do you have a last call you’d like to share? 00:54:53.64 [Aleya Harris]: I don’t know if this will be of interest to your users or not. And it’s seemingly unrelated. So, but work with me for a second. Okay. We’ve talked a lot about storytelling. We’ve talked a lot about words and making data and words play well together. We did touch a little teeny, teeny, teeny tiny bit on your body and how when you understand how to hold your body in space, 00:55:26.52 [Aleya Harris]: you can more confidently tell a story. When I work with analysts, not with a lot of analysts, 00:55:34.28 [Aleya Harris]: no matter how intelligent the words are that are coming out of their mouths, a lot of the times their body is saying the thing that you just said, Tim, please get me the hell out of this room. How fast can I get the hell out of this room? I don’t even know why I have to be here. This damn weekly meeting could have been an email. 00:55:54.76 [Aleya Harris]: I would have felt much more comfortable that way. So there was lots of somatic work out there. But 00:56:03.40 [Aleya Harris]: one of the books, I like to have it on my desk, that seems, again, unrelated. And it’s, how many pages is this one? It’s not even, you don’t even really read the whole thing. The whole thing is 00:56:16.12 [Aleya Harris]: 80 pages. And it’s a book by Louise Hay called Heal Your Body. And it’s more of a reference than 00:56:27.48 [Aleya Harris]: like, let me read every page. And it will go through and it talks about different things that you could have be dealing with, the root cause of where and how it gets stuck in your body and a mantra to release it. To make sure that expectations are clear, 00:56:45.96 [Aleya Harris]: saying a mantra is not going to cure an incurable disease here. Like that’s not where we’re at. 00:56:52.44 [Aleya Harris]: But I’m giving you a starting place to think about your words, yourself, your purpose a little bit differently. Like, so for example, like, let’s say, oh, I feel like I flipped this open. I have people sometimes read the Bible, you flip it open to see whatever page is open. And like, that’s what God wants us to kind of mean. Apparently, I opened up to see and it says cough. And it says, if you have a strong cough, it could mean you have a desire to bark at the world, see me, listen to me. Now, you could read that and say, that’s a bunch of poop into dupes. I disagree. Or you can at least give a second where you’re like, is there any truth to that? Do you don’t have to tell anybody else what there is or not? I don’t know. I’m not there. Which you, you listen to me and we, this is not a live conversation. This is asynchronous. Then next to that, it says, I am noticed and appreciated in the most positive ways I am loved. Now, I have a family member who has a chronic cough, not chronic sickness, not long COVID, just always kind of coughing, always clearing their throat. 00:58:11.40 [Aleya Harris]: This person also is the person who, I don’t know why they do this, but they always are, 00:58:18.28 [Aleya Harris]: like, if we’re walking in a group, they walk behind us. They’re like, would prefer to not be seen. They’re like chronically, like, almost like debilitatingly introverted at times. And then when I read this, and I didn’t plan this, but this is what I opened up to, and it says, a desire to bark at the world, see me, listen to me. That makes sense to me. There’s a kernel of truth in that. And even if it were just a kernel, and you said to yourself, 00:58:46.12 [Aleya Harris]: if you are a person with this cough situation, every single morning, or you said it before 00:58:52.20 [Aleya Harris]: every meeting to yourself, I am noticed and appreciated in the most positive ways I am loved. Do you think there’s a chance that you might show up with that story a little bit differently in the room that you enter? Is there just a little chance? There’s a little chance, isn’t it worth taking, to understand how your body could stop telling on you, so that you really could own the power that your intelligence provides you with. So that’s what I have for you. It’s a very little book, very cheap on Amazon. Louise, 00:59:24.60 [Aleya Harris]: hey, heal your body. That sounds way more productive than having like a little cartoon 00:59:30.68 [Tim Wilson]: flippy chart, you know, like the, you know, Dilbert comics, like this is like a, like, wow, you’re actually going to make- However, don’t hit on Dilbert. This Dilbert is funny. 00:59:40.12 [Moe Kiss]: I’m not a wah-wah person, but there’s something about that hit, which is not the, do I believe this or do I not believe this? It’s like, how do I take a kernel of this thing that might help give me, you know, a different mindset for the day? And that was, yeah, 00:59:56.84 [Tim Wilson]: I really appreciate that. I’m so glad. And Moe, what is your last call? And by the way, 01:00:02.20 [Aleya Harris]: I’m sorry. Before Moe, before you go, I think that it’s hilarious that in the U.S., it’s woo-woo, but in Australia, it’s wah-wah. So now I kind of want to know what other countries call it. 01:00:12.12 [Moe Kiss]: I probably, I probably just fucked it up. Like, let’s be real. Like, that’s the most likely outcome. 01:00:17.72 [Tim Wilson]: That’s how you’re going. It’s, how do you know if somehow it has like four, four syllables in it? No. Yeah. No, no. Yeah. Okay. I have a weird one. 01:00:28.12 [Moe Kiss]: I was walking to school with my son yesterday, holding hands. It’s just really cute. We’re still at that age. He likes to do that. And I was telling him all about his Nana. Well, my Nana, sorry. And who she was as a person. And I was like, oh, Nana was a matriarch. And I was like, do you know what that means? And he goes, yes, a matriarch is when there’s a leader of the group and the leader is a woman. And I like nearly fell over myself. And I was like, you were raising him right, Moe. I was like, wow, where did you learn that? And he’s like, well, elephants and lemur families are both matriarchal families. And I’m like, shut the front door, kid. Like, this is the best thing you’ve ever said. And it was like a zoo show on ABC IV, which is our like national broadcaster, where he had learned all of this. And it just, yeah, the reason I share this like little story is because I don’t know, like there’s little, there’s little moments with kids where you can like, have a really amazing positive influence, or maybe you just underestimate the like, the way they’re learning. I don’t know, it’s just, it’s really been the highlight of my week. And I often talk about academic stuff. And I wanted to share that. However, however, I do have a two for because during the course of this show, I did kept thinking, I kept thinking about a Ted talk, which is Vanessa van Edwards, you’re contagious. And she talks about when you’re presenting and like, you put your arms behind your back and like, what that does to like, how people receive you. So if you were really interested in what a layer had to say, that would be a really good Ted talk to go check out. Anyway, sorry to talk to him. 01:02:08.60 [Aleya Harris]: That was so nice. In addition to my Ted talk, feel free to look up mine as well. Thanks so much. 01:02:14.44 [Aleya Harris]: Absolutely. I gotta get a shameless plug in there, Moe. Come on. 01:02:19.96 [Tim Wilson]: Totally, totally. And Moe, like it sounds like it was, you know, somewhere down under there was a nice kind of a, there’s some good TV programming, we’re shutting down any sort of that kind of messaging and educationing, like education over here in North America. So that’s just rub it in, rub it in our face that you’re, you still have that sort of thing. Yeah. I’m a real human. I 01:02:42.44 [Moe Kiss]: didn’t shut out a layer. I’m like, you know, talking about how great on TV is. I’m really in 01:02:48.12 [Aleya Harris]: a role today. It’s, it’s all right. There is, there is not much more you could do for an American 01:02:54.44 [Aleya Harris]: to make us feel worse than we already do. Sorry. Over to you, Tim. So Tim, what’s yours? Yeah, 01:03:06.12 [Tim Wilson]: I was going to say that, Moe, you’re going to ask me what my last call is. Yeah, what’s your last call? Jeez. That’s what I was counting on Val to be here for. So I’ve dipped into this well a few times over the years, because it is still always just a fun site. But the pudding had a, pudding.cool had a whole deep dive analysis on similes. It’s called comparisons as predictable as the sunrise, an analysis of 200,000 similes from popular fiction. And it kind of struck me because as we’re talking about LLMs and AI and all this stuff and all it can do with unstructured data, which is often text data. And this was actually literally going in to a bunch of text and pulling out the similes. And it makes the point that a simile is kind of an identifiable form because it’s, you know, something as blank as whatever. And as the pudding does, it’s got visualizations. It’s got hilarious anecdotes at the very end. It’s got like just a random, this will be like a layer flipping open the book and reading a random thing. It’s got a whole bunch of like random similes, like a, a, as welcome as hemorrhoids at a rodeo or as clear as the novel by Virginia Wolf. As funny as a guided tour through Dachau. So 01:04:29.40 [Aleya Harris]: it’s kind of like, and those are real things from literature. 01:04:34.60 [Aleya Harris]: I had to get as comfortable as a trout on a marble slab as graceful as a lobster playing soccer. This is awesome. We know what I’m doing. Instead of watching anime until my kid comes home, this is what I will be doing. Thank you so much. 01:04:48.20 [Tim Wilson]: Yeah, you’re welcome. It made me feel good that it was like welcome as a pork chop in a synagogue. 01:04:54.92 [Moe Kiss]: Oh, could I, sorry, her tone as friendly as a bucket full of shit. I’m like, how did I land on 01:05:01.80 [Tim Wilson]: that one? That’s good. I brought joy and those were humans. I’m sure, I’m sure AI could be asked to come up with these like wacky things, but those were from real human written stuff. And it, yeah, I got to the end. I was like, now what’s at the end? And I’m like, well, now I can’t stop reading 01:05:20.84 [Aleya Harris]: these. So, oh my God, I’m sending this to 50 people right now. I cannot help it. 01:05:27.08 [Tim Wilson]: All right. So, that was a lot of fun. So, Aleya, thank you again for coming on the show. This has 01:05:37.08 [Aleya Harris]: been a wonderful discussion. I’ve loved, I’ve loved being here. Sorry, I had a blip because I was still reading the article that you just talked about. I’m sorry. I’m back now. I’m back and I’m 01:05:49.72 [Aleya Harris]: focused. I’m as focused as a dog chasing a bottle like I’m focused. 01:05:59.48 [Tim Wilson]: All right. If you, I’ve, at this point, I think all our listeners have dropped off to go track down that putting article. So, they’re not listening anymore. But if you are still listening, we would love to get a review or rating on whatever platform you listen on. If you would like to get a sticker to slap on your laptop or a mug or a water bottle, you can go to analyticshour.io and we’ll send you a sticker. We’d love to hear from you. Reach out to us on LinkedIn, on the Measure Slack. Reach out to Aleya on LinkedIn as well. If you want to learn more, you can always email us at contact.analyticshour.io. And for whether you are realizing right now that you have a presentation next week and there is no story, or whether you’re like, no, I know what the story is and it came to me while I was listening to this episode. No matter what you’re doing, 01:06:59.16 [Tim Wilson]: for myself, for Moe, keep analyzing. Thanks for listening. Let’s keep the conversation going 01:07:06.04 [Announcer]: with your comments, suggestions and questions on Twitter at, at analyticshour, on the web at analyticshour.io, our LinkedIn group and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst. Those smart guys wanted to fit in so they made up a term called analytics. 01:07:25.08 [Charles Barkley]: Analytics don’t work. Do the analytics say go for it no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 01:07:54.76 [Tim Wilson]: Rock flag and we don’t sell fruit. The post #302: It Was a Dark and Stormy Insight… appeared first on The Analytics Power Hour: Data and Analytics Podcast .

July 7, 20261 hr 16 min

#301: It Turns Out Analysts Are Natural AI Crafters

There’s a certain type of person who first encounters Excel and, instead of running in terror, leans in and grins. Rob Collie has spent his career—from the Excel team at Microsoft to helping birth Power BI to now running P3 Adaptive —building things for exactly those people. He calls them “crafters,” and his new book, Fair Game: Customizing AI to Your Business Is Easier Than You Think , makes the case that this same crowd (hi, it’s us) is uniquely positioned to do something genuinely remarkable with AI. Not because we’re developers, not because we’ve cracked some secret, but because we’ve always lived on the boundary between the business and the tech—and that’s precisely where the real AI work happens. The conversation covers the two “voids” crafters need to jump to go from chatting with Claude to actually building useful custom solutions, why the off-the-shelf AI tools are mostly useless for business purposes (and what to do about it), the faucets-first philosophy for semantic models, and why the developer isn’t dead—just moving to the suburbs. Also: Tim built a quiz about his marriage and let his adult children take it. That happened. This episode is brought to you by Prism from Ask-Y —your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. This episode is also brought to you by Stape , your all-in-one solution for server-side tagging. Links to Resources Mentioned in the Show Fair Game: Customizing AI to Your Business Is Easier Than You Think by Rob Collie The quiz Tim quickly co-built with Claude for his (adult) children The Future of Software Engineering Is the “Product Engineer” A Realistic Assessment of How Many 12-Year-Olds I Could Beat Up Before They Overtook Me AI may be making us think and write more alike Designing the front door Models Demystified: A Practical Guide from Linear Regression to Deep Learning Photo by Bermix Studio on Unsplash Episode Transcript 00:00:00 | Announcer: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. 00:00:13 | Tim Wilson: Hi everyone, welcome to the Analytics Power Hour. This is episode 301. And hey, here’s a fun little fact. Did you know we had the founder of X.AI on this very show? But it’s not what you think. The date was August 30, 2016. The show was episode number 044. And the topic was artificial intelligence with Dennis Mortensen. So at the time, X.AI was a company that was just trying to solve multi-party meeting scheduling via email. And we talked a lot about like highly specialized agents. And it was this wild conversation about how these simple things 00:00:56 | Tim Wilson: that we think are simple can actually be pretty complicated to hand over to the machine. 00:01:02 | Tim Wilson: But that was 2016. It was a totally different world. And it feels like that was kind of an ancient ancestor of the AI world we’re living in today. And we are going to talk a lot about AI on this episode. And I’m going to do that with my co-host. Julie, Hoyer, you’re from further. Is AI something that comes up at all in your day-to-day? 00:01:24 | Julie Hoyer: If I had a nickel for every…, I would have heard AI, right? 00:01:29 | Tim Wilson: Yep. 00:01:31 | Tim Wilson: That’s what I thought. And Val Kroll, you’re from Fin, the company formerly known as Intercom. Is AI something that your company, I mean, are they doing anything at all with AI? 00:01:41 | Val Kroll: Yeah, we’re just starting to dabble, really. Just starting to just get our toes, get our toes up. No, it’s top to bottom. Top to bottom here at Fin. 00:01:49 | Tim Wilson: The fundamental grounding of the primary offering. 00:01:53 | Val Kroll: Yeah. 00:01:53 | Tim Wilson: And I’m Tim Wilson, or maybe I’m not. And this is just an AI agent that Tim built to record this episode on his behalf. Who knows? So sure, all three of us have been exploring and learning and using AI, but none of us have dived in deeply enough to write a really useful book about it. And as far as I know, none of us were product leaders at Microsoft focused on the creation and build out of Power BI. But our guest has done both. Rob Collie is the CEO of P3 Adaptive, which does custom AI and data work for mid-market and Fortune 1000 companies. He’s a former Microsoft engineer on the Excel team and a founding engineer on what became Power BI. He’s the host of the raw data with Rob Collie podcast. And he’s the author of an upcoming book, Fair Game, Customizing AI to Your Business is Easier Than You Think. It’s not publishing until August 15th, but we got a preview. It’s amazing. And it’s available for pre-order now wherever books are sold, including fairgamebook.ai, which gives instant access to the first few chapters and a few other perks. And today, Rob is our guest. So welcome to the show, Rob. Thank you so much. I’ve enjoyed this already. All right, so we don’t usually kick off the show with a, hey, give us your professional history. But one of the reasons we really wanted to have you on specifically is that you’re not just kind of any old run-of-the-mill AI thought leader, you know, half the people who have LinkedIn accounts right now. You actually kind of came from our people. And I mentioned in the intro, but you had a formative part of your career working at Microsoft, including being part of like the birth of Power BI, which is inarguably like one of the premier, most successful BI platforms out there. So I think that actually may be a good place to start. So can you sort of talk about your role, you know, from Microsoft through kind of connect the dots for us as to how that set you on the trajectory to where you are today and like like helping organizations figure out like useful paths forward with AI? I would love to. 00:03:57 | Rob Collie: So I didn’t, like most people who love data, I didn’t know that I love data. I didn’t grow up saying I want to be a data professional, whatever. I’ve talked for years and years about how some of us just sort of carry this thing I call the data gene and we eventually discovered that we have it. Like when we collide with Excel or whatever for the first time, most people bounce off like in terror. They don’t like Excel, right? Yuck. But like it actually is about one out of 16 people who collides with something like Excel and sticks and goes, ooh, that was kind of cool, you know, or their first collision with SQL. It’s usually Excel. That’s the first sticking point, but not always. And so I discovered that I had the data gene in my course of working at Microsoft. In fact, at one point earlier in my Microsoft career, I said to someone that I would never go to work on something like a product like Excel. Never. It was like beneath me. But then I went and I ended up kind of getting re-organized into Excel and discovering, oh my God, this is totally where I needed to be. And so I got to know a few different really important audiences. I got to know the people who really kind of make the world go around, like the Excel power users, that one in 16 crowd. Back then, even really to this day, they kind of make the world go around in a very kind of almost like thankless way. I got to know those people and really, really, really develop an affinity for them in a way because I’m kind of one of them and I was building products for them. And then at one point, I also got grabbed to be in charge of the BI investments in Excel. And so that was being dragged into like the corporate IT zone, right? And I learned the old style of BI through that experience. And then later, I was recruited to join the Power BI team when it was still called Project Gemini Super Secret Project. And the contrast between how the old style of BI, most of us who are working in data or working in business intelligence, were never part of the old traditional style BI because it was so insular, it was so arcane. It was a very tiny priesthood that was involved in this stuff. And I couldn’t even learn how to do it. I was working on it and I couldn’t learn how to write the formulas in these old systems. So helping to build the Power BI, which was aimed at that original Excel crowd, those people that I like so much, the data gene crowd and giving them industrial strength tools, I never thought that was going to work. I love the project. It was exciting. It was definitely the place that I needed to work at Microsoft when they came to pitch me. But like everything else, I’ve just realized, this is probably going to suck. Or maybe it’ll take three versions to get it right. And then I was shocked at how good version one was. I mean, it was really fucking good. And I knew the world was going to change, but I also knew that consulting companies of the time were not going to want to work this way. They were not going to want to move at that pace. 00:07:24 | Rob Collie: They were not going to want to work with clients of different sizes and pick up smaller, faster projects, which is what the world really needed. And that’s what led me to start the company that I did. I wanted to go fill that niche. And guess what? I got to hire those types of people that I liked so much into these new consulting roles. They became consultants. And that’s what our company is largely staffed by, not 100%, but largely staffed by that data gene crowd that grew up in the business more so than in IT. And so it’s sort of like I got to create a home for some of my favorite people at the same time. It was really kind of hard to pass up. 00:08:05 | Tim Wilson: So that’s the first part of that journey. And then, I mean, the link you make in the book 00:08:14 | Tim Wilson: is that those are the people that you say they’re kind of crafters and why crafters specifically are uniquely suited to do stuff with AI. So maybe that’s my follow on, is can you talk sort of about like the leaders, crafters, developers, knowledge workers, and why that’s an AI thing? 00:08:42 | Rob Collie Sure. Yeah, absolutely. And first, let’s just bridge one tiny little gap, which is like when I, I didn’t set out to write a book about AI. So, you know, I’m a CEO of a data company, you know, 50 plus person data company, and I see AI coming. And it was leading to this sense of dread, which I think we can all kind of relate to, right? Like AI is really good at writing code, you know, it’s actually really good at doing a lot of the things that data professionals do for a living. I went on a research project 00:09:13 | Rob Collie: with my own sort of like journey down the rabbit hole to figure out what the future of our company should be. This was about survival. Like what, you know, I had to do this. And what I found down that rabbit hole was far different than what I expected. And far more encouraging and far more like, oh, we can do this sort of thing. And far more, oh, this is actually a data problem. Like it’s like right in our wheelhouse. And the difference of sort of like feeling like AI was like opaque and inaccessible, going from that to going, oh, that’s it. That feeling of like, oh, that’s it. Like that feeling is what compelled me to write a book. Because when I found myself in possession of that feeling, which by the way is exactly what led me to write the first Power BI books that I wrote, you know, 10 plus years ago, like me feeling like I understood something and knowing that there wasn’t anything all that necessarily special about me, like the rest of the world can do this too. Like I felt like I owed that book. And I felt the same sort of the same sort of vibes again, but like at a greater magnitude this time, because like everyone’s worried about AI. But yeah, so back to your question. 00:10:32 | Rob Collie: You know, I’ve been calling these people the data gene crowd forever. But you know, what they really are, and I think present company probably is, you know, it’s we, right? It’s not they. You know, what we really are are people who are willing to think systematically about business problems, while also remaining close to the business and understanding that, you know, there’s the technologists that kind of like want to be like locked in a closet and have like notes passed under the door of what to build, right? And that’s that’s a lot of tech, right? And it’s but it’s a very and you need those people. But that’s a very inefficient way to work for most business solutions. Like being immersed in the business, while at the same time having some tech skills, enables a completely different style of work, a completely different tempo and a completely different quality of output. But with the with with AI coming along and the ability for people like us, people who picked up Excel and smiled, right? Like, when you sit down with people like us sit down with something like Claude Code for the first time, we sort of like get over our fear of trying something new. It’s that same feeling like captivating feeling like, Oh my God. And so I needed to come up with a new name for these people. It’s no longer the data gene crowd. I needed a noun for these people in the book. So I had to come up with something and crafter is where I landed. So that leaders, crafters and developers and ratio to 580. And so for every, if you have a if it just works out that way, if you had a population of 87 people who work in an office, you know, like two of them are developers, five of them are crafters, and 80 of them are knowledge workers of various, various flavors. So like only so five relative to 80, like that’s how many people again, one to 16 are willing to engage with something like Excel. And what I’m seeing is that that same data gene crowd, plus the ability to use coding agents, AI coding agents, to build custom software. I mean, this is like going to be the new spreadsheets. You know, we application development is now within range for this crafter crowd. And we’re just in the earliest stages of the world waking up to this, like most crafters haven’t yet discovered that they can do this. But we’re at steady state, we’re going to have about the same number of people building software as we had building spreadsheets. 00:13:17 | Tim Wilson: Michael, how does your team share AI generated analyses 00:13:26 | Michael Helbling: today professionally? We say, hold on, I think I asked Claude this last week and then sort of disappear into a chat thread like a raccoon in an air duct. Oh, nice. That’s that’s elegant. That’s 00:13:36 | Tim Wilson: clearly scalable. I mean, actually, that’s horrifying. Why thank you. It’s called innovation. That’s the type of innovation that ask why is trying to solve with Prism. Your AI work shouldn’t 00:13:49 | Michael Helbling: be trapped in one person’s called innovation. I have started to realize my private chat history is not a data strategy. It’s kind of a joke drawer with confidence. Well, Prism helps preserve context 00:14:03 | Tim Wilson: and memory across users, you know, like metric definitions and source of truth tables, business 00:14:08 | Michael Helbling: rules, prior analyses. So when I teach it, a qualified lead excludes internal traffic, spam forums and that one campaign we don’t talk about, the team can build from that shared context instead of starting over entirely. Beautiful. Let’s ask Michael what he meant, more the system actually 00:14:26 | Tim Wilson: remembers. And with Prism, analyses are organized, traceable, auditable and reusable with your other 00:14:32 | Michael Helbling: data sets. I like it. So my best AI work becomes company knowledge, not a screenshot named. Is this important? Maybe.png. Exactly. I guess it’s time to stop being a data trash panda when it comes 00:14:45 | Tim Wilson: to AI. Definitely. No pandas in this context. Go to ask dash y.ai and sign up for the wait list. 00:14:52 | Michael Helbling: Use code APH to jump to the top of that list. That’s ask dash the letter y.ai code APH. Because check my cloud history is not a collaboration plan. Tim, one of the trickiest parts of measurement is user data gaps. Right. Someone gives you an email in one session, 00:15:15 | Tim Wilson: comes back later from another device or converts offline and suddenly the customer journey gets 00:15:20 | Michael Helbling: hard to connect. Exactly. Staped and richer is a state power up that stores selected data from incoming requests and uses it later to enrich future events when information is missing. 00:15:32 | Tim Wilson: So once a user has been identified, and richer can help recognize them in later events and add previously collected details, all while aligning with consent settings and handling personal data 00:15:42 | Michael Helbling: securely. Yeah, it’s especially useful for cross device journeys, offline conversions, longer buying cycles and connecting CRM outcomes back to earlier website activity. You can set up and richer to work with cookies, state store or both. And it supports uploads of historical data from systems like Shopify, Clavio or your CRM. So if you’re tracking has gaps and who’s doesn’t, Staped and richer can help create more complete consistent event data. Yeah, go to state.io to learn more about Staped and richer, improve event match quality, attribution and visibility into the full customer journey. That’s Staped.io, more complete data with consent and privacy in mind. 00:16:28 | Val Kroll: Besides like the just the usual FUD, what do you think are some of the barriers to crafters? Because like I remember where I was the first time I saw a pivot table and it was like, blew me back in my chair. I was high school, but of like, what is holding, what do you think are some of the major barriers that are holding up the crafter crew back? 00:16:52 | Rob Collie: Well, here’s one quick funny story before I answer your question. I worked on the Excel team for a full year before I knew what pivot tables were used for. And I love that. I would fake it be like, yeah, you could throw this into a chart, you could throw it into a pivot, you know, I would just say things like that without ever you really even knowing what it was. Just like, fake it. No, that’s what all the cool kids were saying. So I would say that, right? And then one day, so for you to see a pivot table in high school, that’s awesome. That is really, really cool. So the thing about Excel is that, you know, is that it was so, I mean, I think the most people are even even people who end up liking Excel are scared of it at first. You know, thinking back, I think I was, you know, and, and even when I worked at Microsoft, like every couple of years, people would move around between teams and office, you know, it’s really an open season recruiting. I tried to get people to come to work on Excel and they, they wouldn’t want to. They would want to go work on outlook and word and PowerPoint because they already understood those things. But they could sense that Excel was deeper, you know, Excel is an application development platform, you know, it’s a spreadsheet is an application, you know. And so, like, there’s something intimidating about it. I personally, and I say this in the book, I don’t, I don’t pick up new tech readily. I don’t, I’m not an enthusiastic, like, ooh, shiny, go grab it off the shelf kind of person, like I kind of have to force myself, you know. And it’s really just sort of like not even knowing where to start. And like, how do you even get it installed? Like, even just going and installing something like like cloud code, right? Like, at least it used to be harder than it is now, it’s a lot easier. 00:18:43 | Rob Collie: But and not understanding the nature of the system you’re working with, like, it just feels like magic. Like, so you’re putting so much trust in something else that’s that’s that feels opaque to you. I think there’s a sort of like a like a sense of loss of control that that is that is terrifying. And I’m seeing this a little bit with our own team, like, you know, not as much anymore, but in the early going, they’re like, we derive such confidence from having touched every last single piece of logic, you know, and like artistinally crafted it, right? Like, like, how can we delegate parts of that to another system, you know? So there’s a number of things like that. But for me, honestly, it was just kind of getting off the starting line. I just had to sit down and build something. 00:19:39 | Tim Wilson: I have a little bit of a theory, and it bridges those worlds. And it’s interesting when you say, you know, PowerPoint and Word, and Outlook and Excel. And I don’t think I’d really thought about it. If I’m doing something in PowerPoint, I’m making slides, I know I am constrained to this thing of slides and Word, same thing with Excel, I feel like the people with the data gene, the crafters, somewhere along the way, they go from it’s a table where I’m doing formulas to Oh, wait a minute, I can actually build an interactive app, you know, I can put slicers or, you know, I can put data validation and put drop downs in and I can, I can, I can kind of build a data product, which I think is one of the reasons Power BI is successful is because there’s the progression through up to Power BI. And I think the same thing happens with any BI platform is you have that somebody with the data gene saying I’m building, I’m crafting an experience. And to me, what I think might be happening, and this was purely from actually reading the book that this little light bulb went on, we are all introduced to this new world of AI through the chat interface. And there are plenty of people out there saying it’s not, it’s more than just the chat, you got to build agents, you got to do all this other stuff. But there are people saying that it feels like it’s, it’s this other thing far away. But really where it’s close is saying, okay, if you just step out of the chat and say, let me, let me use the chat to help me build a data product, let me use the chat to help me, like, create something like that’s where that, that parallel. Okay, yeah. You know, I started to, to see because it just that light bulb hadn’t gone on that, oh, I can create a deliverable for my audience, for my user. And I’ve now got a broader set of things in my fingertips, just like going from, I got to hack around and excel to build an interactive dashboard, I go to Power BI, I’ve now got a deeper, a lot of these things, I don’t have to come up with the indirect formula in a data validation dropdown to make this filter work. It’s just natively there. 00:21:52 | Rob Collie: I think you just nailed it. I think you nailed it. And I didn’t, and I’m not saying that like, that’s exactly what I was thinking. I’m saying that like, I think you just gave an answer that I didn’t have. But I think it’s the, I think it’s a really good point. So there’s a, there’s a mixture of kind of like gaps you have to, you have to hop, right? Like, you know, smooth climbing curves are one thing, smooth climbing learning curves are one thing, but having to like jump from point to point, like over like a, like a void, you know, those are the places that get us. And there’s two big voids here. One of them is that, you know, we, we sit down and we use the off the shelf AI systems. This is a recurring theme in the book is that the, the off the shelf AI systems are amazing for like personal use. They’re amazing for individual use. They know everything about like all of human history. It’s insane. I was using Claude the other day and it was telling me about the geologic construction of the ground underneath my house here in Seattle. It, without even having to search the web, it didn’t have to search the web. It knew what the geographic makeup and so we were calculating the speed of sound in the ground near my house and it didn’t 00:23:05 | Val Kroll: have to search the web. You’ve got the data gene. It was family night. 00:23:12 | Rob Collie: Okay. Look, there’s a power driver driving, driving a post into the lake near our, near our house and the sound arrives in a two syllable thump. You get a thump through the ground and then you get the clank through the air. It’s one thing, but because the speed of sound in the ground is faster, the ground wave gets here and I was measuring the delta between the, the time between the two arrive and using that to calculate the distance to how far away the pile driver is. And yes, there’s something that the whole family is aware is going on. Like dad’s calculating this is like, it’s two, it’s, it’s 200 meters plus or minus 15. But anyway, so like, like, but to think about it, like that is publicly available knowledge that’s going into this LLM. You know, but like if it’s about my business, and in the book, I show an example of this, like I just have it like write a proposal, write me a proposal for this project. It goes and tries to do it, but it knows nothing about us. It, it just gives the lowest common denominator like a proposal and it is completely worthless. It is awful. And so there’s that, there’s that gap between business use and personal use, which is like, again, it’s, it’s, it’s this void, right? Like it’s, it’s not a, you have to jump it all at once. 00:24:32 | Tim Wilson: Well, it’s the same, it’s the same void of saying, I know I’m not supposed to put private data into this if I’m using a public or I, I’m like, I’ve been terrified that I’m going to put something in this going to be a problem. But if I just ask her to do something without it, so how do I put enough stuff in? And it’s, that feels like the other connecting the dots, which again, I mean, yes, yes. So there’s that sycophantic Tim, so maybe I am just an AI agent. I’m like, your book breaks out the different pieces and how to think of them. But that feels like it’s again, it’s like, you’re using chat GPT and somebody’s like, 00:25:04 | Rob Collie: but don’t, don’t give it any of your information. And so then this is the second void is that you have to get comfortable building real, regular software, like stuff that’s written in Python or whatever, which is not what data people have traditionally done. Like we’ve worked in ETL, we’ve worked in formulas, we’ve worked in data pipelines and things of that sort. But until you can write your own custom code, you can’t build the rest of the system around the LLM to customize it. If you’re in charge of that software, then you’re also in charge of the storage and you’re in charge of like, which version of the API you call that is you can call ones that are that are actually more protected and that aren’t that they don’t train on your on your data. So you can you can candle that sort of sensitivity. But you have to go from being a non software developer to being a software developer in order to get this customized experience where you can control the flow of information and you can give context about your business at the right time to the LLM. So you have to sort of jump both of these voids at the same time to get started. And this is why like in the appendices of the book, the first 00:26:20 | Tim Wilson: thing you just did one of these the other night, I think, Tim, right? So the first thing I have 00:26:28 | Rob Collie: people do if they’ve never used cloud code or codex or whatever, like it’s just build a website on their computer. That’s just like a survey or a quiz that gives a score or whatever, just get used to the idea of building anything. No, no AI agent stuff, just get just start to get more comfortable with building free form software. And then we start to layer in and the next exercise, like, okay, now we’re going to add some LLM to it and see, you know, where it 00:26:58 | Tim Wilson: goes from there. So I’ll throw in now. This is I may regret this, but for shits and giggles, what I wound up doing was a little 15 question multiple choice quiz about my wife’s and my relationship and about us that I sent to my three 20 plus kids, 20 adult children. I was like three 20 plus. No, three 23 kids who are all adults about like, it’s like just family lower stuff, which was like a prompt. But it was fun standalone. I did wind up having to throw it on Netlify. So it was like, I would like one part of my like, Oh, I already have a Netlify account, I’ll throw it there. But it’s now, of course, gone to where now my parents have taken it. I mean, my kids do the best, but they’ve also had a few like, what you did? I mean, so, you know what, I’ll throw it into it in the show notes and anybody terribly. Yeah. Why not? How well do you know, Tim Wilson’s marriage? Yeah, I mean, it’s going to be referred, it’s going to refer to mom and dad. And that’s going to, you’re going to have to recognize that. Okay, I’m just going to have to be okay with it. Anyway, it was a great episode. But just, I mean, throw in that, that was like the little unlock was even though I’ve done stuff with co-work, even though I’ve done stuff with gyms, even though I’ve, I feel like I’ve done a lot, there was a piece by the time I got to the end of the book of saying, Oh, no, no, no, there’s a software component where what I’m developing with limitations, right? Because you also talk about where you still need developers. This is not now you’re turning the people with the data gene, the crafters into full blown, 00:28:32 | Tim Wilson: push stuff into production to hit the masses. The developers now are still needed for their 00:28:39 | Rob Collie: skills. Yeah. So, you know, we P three, we’ve hired this year, our first two full stack developers ever. And it’s kind of funny, like, in fact, I was even in Fortune magazine at one point, talking about this, because, like, big tech is laying off developers, because, because AI is so good at writing code, in fact, it’s better writing code than anything else. At the same time, though, while it’s expanding the capacity of an individual developer to do so much more, that makes their ROI pencil out much differently. And so, like, you know, in order to achieve what we’re achieving today with two developers who are all in on using AI to write code, we’re getting the results of, like, honestly, like, what are probably required a team of 10 00:29:32 | Tim Wilson: developers to do before. So, I call this sort of like the migration to the suburbs 00:29:40 | Rob Collie: for developers. Like, so, like, like, there’s been a concentration of developer talent in big tech firms, because they build huge amounts of technology and they require engineering teams of, like, 30 developers at a time. And I don’t think they’re going to require 30 developers to do 00:29:56 | Tim Wilson: the same sorts of things anymore. At the same time, though, the ROI on hiring a developer, 00:30:04 | Rob Collie: like, in a smaller company, or a department level of an enterprise or whatever, is going to be higher than it used to be. So these people are going to have other jobs they can plug into. They’re just going to be in different places. And, you know, so, like, you know, the dev team of two that we have here, we don’t need to go too deep. And just to sort of underline it, like, where we’ve built things that put us at probably about a year ahead of Microsoft 00:30:33 | Tim Wilson: in terms of offerings that Microsoft will have. It’s like, we’re able to do things for our clients 00:30:39 | Rob Collie: today with AI around Power BI and things like that, that, you know, Microsoft is absolutely 00:30:45 | Tim Wilson: building. But they’re not there yet. And so, you know, it gives us an opportunity to fill some 00:30:52 | Rob Collie: gaps there in the meantime. And, like, that is just absolutely bananas. A year ago, me, would never have guessed that that was a possibility. 00:31:02 | Tim Wilson: You would have said the tool that’s not doable, even with our DAX amazingness, we can’t do that. We’ll have to wait until Microsoft rolls that out. I mean, or you’d find an alternative way to come at it. 00:31:14 | Rob Collie: No, I’m thinking it’s more like I just hadn’t realized how valuable developers were going to be. Like, like, there’s a difference between what a crafter, like, so I think of myself, I’m 100% a crafter. I’m not a developer. And I talk about there’s a form of discipline that’s present in a developer. These are mostly personality traits as opposed to, like, intellectual IQ measurements. There’s a disciplined patience in a developer, combined with, like, an aggressive modularization instinct that isn’t really present in most crafters. It’s not present in me. And so, the things that our dev team of two have built are things that our crafter team would not have been able to build. They’re just levels of complexity. And even just knowing what to build and figuring out, like, what the roadmap should look like would have just taken us too long. The developer brain is still really useful. And so, one of the things we’re working on on our team is figuring out how to hybridize developers and crafters to give the best possible results to our clients. Because most developers, and I’m painting with the broad brush, are most developers aren’t going to want to be the customer-facing interface that does all that conversation. Like, they want to be writing code. Even if they’re capable of having those conversations, they love writing code, you know? And meetings take them away from that. 00:32:53 | Tim Wilson: But crafters thrive on interacting with the stakeholders, right? Like, that’s the world 00:32:59 | Rob Collie: that they came from. They want to be right there, you know, like, almost like pressed up against the audience that they’re helping. And so, but there are certain kinds of code, certain kinds of platforms, et cetera, that, like, you really, you’re still going to want developers to do those things, even though we’re building solutions at the crafter level. And I 00:33:19 | Tim Wilson: do talk a lot in the book about trying to paint pictures of where those boundaries are, 00:33:26 | Rob Collie: but they’re incredibly fluid, you know? Like, it’s like, you can’t paint a precise picture. 00:33:32 | Julie Hoyer: I was going to ask you to clarify, like, when we’re talking about crafters, are we, in my brain, and I feel like maybe some of this wording comes from, like, working at further and search discovery, like, we always talk to him and about, like, our titles, or maybe a little different than somehow, like, the industry talked about them. But in my head, right, crafters, the way you mentioned it, Rob, to me, I would put in, like, this engineering group that’s more like building the back end of, like, a BI tool, and, like, surfacing the data type engineering role, whereas I came up through more of, like, the classic, like, analytics role. We kind of talk about them as, like, functional analysts, very tied to, like, the business. I could, like, build a dashboard, of course, but I wasn’t doing, like, the ETLs in the background. I was doing more of the, like, business, what are you asking? What is your problem? What’s your hypothesis? Let me actually go manipulate the data, build something in R, right? Like, a model, do an analysis, and then bring it back to you. In your framing is a crafter, both of those things, because when you say, crafters need to become software developers, and, like, the way of using AI, I’m just curious, are you talking more what I’m saying are engineers, or are you including this, like, functional analyst group as well? So I think I should very much clarify that I’m using 00:34:52 | Rob Collie: a very Power BI-centric lens for this, because, you know, our company’s all been all about Power BI. You know, we’ve long said that we will use another tool if something came along that we like better. But so, but we’ve been, we’ve been a Microsoft stack company, and it’s not because of Microsoft patriotism. Like, I literally formed a company around this thing that I saw. 00:35:16 | Tim Wilson: The Table of Army is going to be coming for this. The comments are going to be… 00:35:20 | Rob Collie: Well, I am, I am ready. I am ready for that. When we talk briefly about semantic models, I’m really ready. So… And the Domo people too, but that’s only going to be like three people. So… I know. Yeah. I mean, you just defarm them. The people that I’m talking about have kind of always sat on the boundary between the business and IT, but they’ve lived mostly in business. So they, you know, like they reported up through, you know, some non-IT function. You know, they’re the shadow IT folks. And so, you know, the role you’re describing is sort of like the business analyst that does all the, interacts with the business and sort of figures out what the needs are. The people who do that, 00:36:08 | Tim Wilson: but then also go and like build the spreadsheet, you know, that addresses it, or in our world, 00:36:15 | Rob Collie: then went and discovered Power BI and started doing the ETL and started, in sort of doing the data modeling and writing the DAX. That’s the, you know, that’s the, that’s my personal like, like center of gravity for all of this. So at our company, we talk about it as like the decathlete model. So like most consulting firms, the way they’ve been constructed over the years is they have specialists. They’ve got someone who throws the javelin. They’ve got someone who runs the hundred meters. They’ve got, right. And so this team of 10 rolls in there on the engagement and you pay a tremendous amount of communication costs between those 10 people. Moest of the elapsed time of the project is actually the communication between the members of the team. It’s like 99% of the weight of the project is communication costs and not implementation. And so our whole business model from the beginning was built around this idea that, no, no, like you, you need to be the decathlete. You’re the one that talks to the customer. You’re the one that understands their needs. You’re the one that builds the ETL. You’re the one that builds the data model. You’re the one that builds the dashboards and all that community, your bandwidth within your own head is just so much faster than the bandwidth between people. And by the way, also less error prone. And so at least again, speaking through the lens of my company, the company that we built, this is the persona that’s kind of like, like at ground zero for it. And so I think more on the business analyst side than on the quote unquote engineering side. But, but in the Power BI world, a lot of those business analyst types acquired these skills that had traditionally been the purview of the back office IT types, right? Like we were invading their ETL world and invading their data modeling world, right? Like, you know, tearing down that ivory tower. 00:38:11 | Tim Wilson: I feel like it’s the media agency that would say, oh, we manually update this spreadsheet and we put our dates across in the columns and it just makes our skin crawl with the structure of it. And they manually date it and they put some hacky formulas and since some sort of garbage over the wall to the marketer. And then an analyst gets in and or somebody with the data gene gets in and says, this is error prone. It’s not really giving the marketer what they’re trying to ask. So I’m going to figure out a way to maybe it’s VBA, you know, maybe it’s Google, maybe it’s Apps Script, because we’re doing in Google Sheets. I’m going to figure out a way that we streamline the process and deliver something that is more aggregated, more useful. Oh, and I’m going to put a layer of interactivity on it. And I’m not going to be able to stop. And then suddenly I’m the person who the digital marketers come to and they say, can you make one of those spreadsheets? And like, that’s where we’ve crossed over and we’re like, wait a minute, are we are we developers? Like, no, you’re still not a developer. You’re a, but I also feel like it’s where analytics engineers arose. That is a whole field that came out of analysts who said, damn it, I can’t keep waiting for the engineering team. Every time I want to change the ETL, I got, I’m just going to build myself a little sandbox. And all of a sudden that became an entire position or a total role of 00:39:37 | Tim Wilson: where the analyst just kind of gets better. And the point I think that, that like in the 00:39:44 | Tim Wilson: light bulb that went on for me was like, oh, wow, I would never consider building a front end. I would never consider something that limits the rules that is actually software. But guess what? Those same people with a little bit more knowledge and a little bit of practice can actually extend it with a boundary of where does it cross over to a developer, but they can actually, they have this other thing that we’ve never even thought of as being in our arsenal. 00:40:13 | Val Kroll: It seems like one of the, the first steps to someone kind of seeing for these crafters have their eyes open is to get out of chat, um, to start building. And I think some of those examples you guys talked about was great, but something you touched upon earlier Robin obviously got into it deeply in the book is thinking about how do you give it the context about your business that is required in order for it to actually be the partner for you that it needs versus just like the generic off the shelf, like how would an individual contributor think about going about that in a way that’s productive for them in role? Like do you have any advice for taking that leap, which feels like another step? Yeah, I mean, the, what’s really amazing is just 00:40:55 | Rob Collie: how much smarter even the LLM seem to get when you give them something to go on. So the first example that I talk about in the book is this concept of what I call a handbook. Um, so if you, if you had a new hire, uh, and one of the metaphors I use is, um, an LLM is basically something that has a, especially the frontier new LLMs, they basically have PhDs in every human subject, every, everything that you could get a PhD in the LLM has that level of knowledge and even ability to reason to a certain extent in all of those subjects, but it shows up every day knowing nothing about your business. And if you thought of it as a person for a moment that showed up with all of that skill, but knowing nothing about your business, they would be relatively useless to you. You know, like you don’t care about the history of the Roman empire, uh, most of the time in your business, right? You don’t care about the ground speed of sound underneath Moett Lake Seattle most of the time in your business, right? Um, but you do have policies and a brand framework and workflows and things like that. And so like, if you were going to hand, um, um, a new hire, some sort of handbook and instruction manual for their job, uh, it would be written in English, like if, if you’re in the United States anyway, 00:42:17 | Tim Wilson: right? Like, you know, it’d be written in natural human language. That same sort of document can be handed to an LLM every time it wakes up. And if you, if you, again, now you’re writing custom 00:42:30 | Rob Collie: code, right? So you have, you’ve had to take control and you’re not, you’re not using the chat GPT built in interface or the off the shelf interface anymore. Like you’re sort of like taking control of it. You’re building your own software. Um, but it’s not that hard. It really isn’t. And your, your software, the first thing it does when it, when your software wakes up and someone asks it a question or something like that, let’s say you’re building your own chat bot, the first time the chat bot wakes up every time it goes and grabs that handbook and feeds the handbook to the LLM saying, here’s what you’re doing for us today. Here’s your personality, here’s the rules that you follow. And that lands in the LLM’s brain before any question from the user does. And suddenly it goes from being this off the shelf useless thing that can’t help you with your business because it doesn’t know your business to being a world beater. Um, and if you just think about that for a moment, just pushing some instructions and some context, just like some information about your business. Who are we? What do we do? Like what’s, what’s your corner of the world here? And each of these agents you build like this, they become specialists in that one role, right? You don’t, you don’t, um, I talk about like in the book, you don’t, 00:43:48 | Tim Wilson: you’re never going to build a super being that knows everything about your business because 00:43:54 | Rob Collie: everything about your business can’t fit, believe it or not, it can’t fit into the LLM’s memory. It will over, it will overload it. So you have to give it narrow lanes. Um, and you know, so like if you have like, you know, but you can imagine agents with five different roles, like I’ve got one that helps me write marketing copy. I created an editor to help me with my book. Um, and they have different instructions, right? You know, one of them is, is instructed how to write like our company. And the other one is instructed not to write anything for me. The other one’s instructed to help hold me to my standard of writing. And when it makes suggestions on fixes, don’t make suggestions that make it sound like boring business voice. Remember that I’m still trying to sound like me when you’re criticizing my work. So like, don’t, for example, don’t, I don’t want my editor to ever say to me, Hey, Rob, you start too many sentences with the word and or but, you know, that’s just how I write, you know, so shut up about that. We’re not going to do it that way, you know. So this idea of handbooks is really like the, the first place to start. But then over time, you start to give these agents, and again, you’re just giving the LLM sort of a menu, 00:45:18 | Tim Wilson: you give it the ability to pull information that it wants. Because you can’t sometimes, 00:45:24 | Rob Collie: you don’t, you can’t know in advance what information it needs. So you give the LLM some ability to go and pull information it needs. And by the way, we’ve all seen this in action already. Every time you go to an off the shelf LLM product and you ask it a question, when it switches off and does a web search, it’s doing exactly that. It’s recognizing that it needs more information from elsewhere and triggering something that’s called a tool. And the tool is just, it’s something that literally just sends a chat request to the tool, the LLM does, and then the out that the tool goes and performs the web search and returns the results back to the LLM. And you can do that with your own internal databases. You can do that with your own internal knowledge bases, which is crazy. 00:46:12 | Julie Hoyer: So for a very specific example, it immediately when I think of like AI and use of analytics, you hear about every tool putting in AI chatbot or AI agent on top of their BI tool, right? And it’s supposed to like be so great. And everyone’s got to have had experience with it. Yeah. And as most people that actually go in and use them, they don’t know anything about your business in there. And I have not found a helpful one so far. And so then of course, that gets us into the idea of like, Oh, it needs a semantic layer, it needs context. So I’m kind of like wrapping us into two possible paths here, and you can choose whichever one, but I’d love to talk about them both. One being the whole semantic layer conversation. But two, and what you were just talking about, what advice would you give an analytics practitioner, a crafter of your company has this BI tool, and it has this AI agent? Like, are you saying you can control that AI agent with like these handbooks and things, which it sounds like maybe it leans into semantic layer, or are you saying, don’t use the off the shelf thing, go make your own custom one? Or is this scenario not applicable at all? Option C. I think, I think they’re both kind of the same 00:47:29 | Rob Collie: conversation, honestly. Like, the key thing to recognize about any AI solution, and this is the thing that kind of blew my mind, is that the LLM is sort of like, almost like, it’s just something you pick off the shelf, and it’s almost like, like the, an element on the periodic table, it is exactly the same for you as it is for me, as it is for the, as it is for Satya Nadella at Microsoft, it’s the exact same LLM. And it’s really just like a component. And most of making, not most of, making these things do anything useful for us is all about just regular software, not AI, just regular software. And the regular software’s job is to handle like the user experience, just like it always does, as well as the flow of information to and from the LLM, like giving the LLM access to what it needs to know. So let’s keep in mind for a moment that most AI agents, we fast forward like two years, and we see what the future looks like. Moest AI agents that are deployed at companies aren’t going to be about analytics, you know, like, you know, like a, a chat bot that handles the first level of interaction with customer service, for instance, it doesn’t need to be performing strategic queries against your, you know, your data warehouse, right? Like it really just needs to have a, like a flow chart in it, and, and also some instructions on how to talk to people and what not to say and all that kind of stuff. And that, that sort of falls into the handbook category. So there’s a lot of tactical AI agents that aren’t going to have anything to do with what we think of as analytics. I mean, I just want to level set that. What’s kind of neat, though, is that even those tactical things that have nothing to do with analytics, they still, because they, they operate on information that needs to be very carefully controlled and fed at the right time, that the types of discipline that we’ve developed as analytics people, as data people, are that are the right mindset for making those things work properly, you know? And now that you’ve given us, the world has given us crafter types, the ability to write real software very quickly with the use of an agentic, you know, like something like cloud code, we’ve got everything at our fingertips, that we’re going to be really good at building those sorts of solutions, even though they’re not related to traditional analytics at all. And so I want to make sure that I’m not pigeonholing us data crowd into just the traditional data crowd type AI stuff. So that’s, and the handbook stuff is not, is not part of what you’re hearing when people talk about semantic models or semantic layers. Handbooks are not, not really part of that. Semantic layers are a decoder ring for your structured data. And that’s, now, now we’re back in the analytics and data crowd, you know, we’re back in our, in our, in our home neighborhood now. Yes. And so the, the, I think this isn’t even really controversial, honestly, because so we were talking about, you know, the Tableau crowd coming for me. Okay, so it doesn’t matter what platform you’re associated with, like if you’re a Tableau professional, you’re a Snowflake professional, you’re a Power BI professional. I think it’s really just like a foregone conclusion that you are going to be working with semantic models and semantic layers. And, you know, I can give you the technical reason why, but I can also just give you the market reason why. So let’s take Tableau as an example, and I love this, this just tickles me 00:51:33 | Tim Wilson: to death. So for, for men, the best thing about Power BI, from a technical standpoint, the best 00:51:41 | Rob Collie: thing about Power BI was not the reason why it got adopted. The reason Power BI won the BI wars, and has the largest market share is because it was cheapest and it was integrated with the rest of Microsoft. That’s it. No one was evaluating like, ooh, it’s so cool that you have a semantic model in this star schema stuff and all, like, like, it makes a difference in practice, but that’s 00:52:03 | Tim Wilson: not why it won. And Tableau long neglected the idea of semantic models. It’s, they were all about 00:52:11 | Rob Collie: each time you need a new dashboard, you go write a bunch of new SQL queries to make a big flat, to make a big flat wide rectangle of data that you hide behind the dashboard. That’s the philosophy over and over and over again. And the thing is, it worked, right? I mean, like the IPO, they sold the Salesforce, I mean, those people made gobs and gobs and gobs of money. So it’s like, you know, you know, I’d start arguing with them about like, well, y’all don’t have a semantic layer. And they go, well, I have more money than you, you know, but you’re right. So they eventually invented a semantic layer that sort of was an attempt to sort of add what Power BI had been doing from the beginning, but then they neglected it. A friend of mine at Tableau was in charge of that for a long time, and eventually he just retired in frustration. He just gave up. Not even, not even making this up. Okay. Well, after so many of these companies have neglected semantic layers forever, they all banded together to form this OSI thing, which I think you have has been talked about on the show before. Open semantic interchange. And, and on the Tableau website, a year ago, less than a year ago, there was an article says, the agentic future demands 00:53:25 | Tim Wilson: an open semantic layer. Okay. So everyone now, if they, every data company that wasn’t in on semantic layers is now in on semantic layers. Every single one of them has something that either 00:53:41 | Rob Collie: are sticking with the one they already had, like in Microsoft’s case, or they’re banding together in this open standard and rapidly integrating into that product and trying to get their customers to adopt it. Now, why would they be doing that? As a company builds a semantic layer, I think, 00:53:56 | Tim Wilson: and this was when we had Cindy house and on to talk about it, there was something she said that I really liked. So she was kind of in the OSI heavy camp our last episode, we had Jacob Madsen, we talked about maybe we don’t need one at all. So we’ve kind of covered, I think, 00:54:10 | Tim Wilson: my one question is like, how does the semantic layer not get held up as the monolithic 00:54:19 | Tim Wilson: boogeyman and the monster dev project of saying, yes, yes, yes, yes, we will do glorious things with AI as soon as we build the semantic layer, because we did that with data dictionaries, we’ve done that, I mean, at least in the days of power play cubes and SSS, like they were contained, but like where that’s where I get nervous about semantic layers as it becomes an excuse and people like we have to build the entirety of the story of our data. 00:54:50 | Rob Collie: You could take your concern there that you just shared and you could plug and play replace that with data warehouse, like seven or eight years ago. And that would be something that was a talk and a spiel that I would give everywhere, like the data warehouse. So we got to go build the data warehouse first. And the other thing is data warehouse is never done. And by the way, the other thing about data warehouses is that you would find that you’ve been trying to anticipate all of these future needs, and you anticipated them poorly. So you built plumbing to nowhere, places that faucets were never going to be needed. But then the first time you need a faucet, you go and you look and there’s no pipe, the place you needed the faucet. So you over engineered and under engineered at the same time. And yeah, I agree, like those sorts of monolithic projects 00:55:35 | Tim Wilson: that kind of become, they become their own thing. They become like a goal in and of themselves. 00:55:42 | Rob Collie: Those are danger. You do not want that. So one of the things we’ve been, one of our philosophies 00:55:52 | Tim Wilson: that are our company, we call it faucets first, we start with the faucet and we build backwards 00:55:57 | Rob Collie: from there. So yes, we do need responsible data plumbing. But if you’re building plumbing forward, you’re going to lose. Now, if you’re a consulting company and you’re building plumbing forward, you’re going to win because you’re going to build a lot of hours. And then when things don’t work, you’re going to build a lot more hours to fix it. But if you go faucets first, if you go faucets backwards, you can get everything that you need. So I think semantic models, and the way we approach Power BI is to sit down and go faucets first. We don’t go build a big 00:56:33 | Tim Wilson: data model before we build our first dashboard. The whole point is to get to a 00:56:40 | Rob Collie: first dashboard as quickly as possible so that the customer can go, okay, that’s not right in these three or four different ways. And then we go modify the data model to fit it. The same thing here. If your goal is to build, and okay, so let’s bridge a really important gap here. The reason why a semantic model is suddenly so important today, it was important to BI, but no one cared other than Power BI really. But everyone now cares is because you don’t have what I call the semantic shock absorber of the people that you lock in a room and make them write the queries. So each new dashboard that’s created is really representing new questions that have emerged. The business has new questions, so you can build a new dashboard. This has been inefficient really, even in the Power BI world, this is really not optimal. Okay, but it’s even less optimal than the tab low world where it requires like a week’s worth of SQL work before you get the first dot on the chart, right? If an AI agent forms a new question or a user comes to an AI chatbot with a new question, there’s no time for someone to go write a week’s worth of SQL. And you also don’t want to trust the AI to go write that SQL because it will write it slightly differently each time and it won’t know which column means what in the source data. This is why the decoder ring analogy I think is so apt. Symantec glare is just a very nerdy way to say decoder ring. We wrote down what things mean. We wrote down how to translate our company’s structured data into meaning that matters to 00:58:17 | Tim Wilson: the business. What is gross profit defined as? There’s a lot of nuance to that that’s 00:58:24 | Rob Collie: different for every single business. And so if you look at it through an AI lens and you go, okay, what does this AI agent need to do? And then you build the Symantec model that is required to power that. You’re in a much more focused means of engagement than if you’re trying to like, let’s go model the whole business, just the whole thing. And let’s just party on this thing forever and never test it and never write. No, that’s not the way, right? And I was going to say too, 00:58:56 | Tim Wilson: like I love what you’re saying and I just feel like people are finally waking up to this idea 00:59:05 | Julie Hoyer: and that like surfacing the data alone was not the valuable part. And so like why people could get the budget approved for data warehouse or like projects or the huge Symantec layer projects. I feel like it’s because and Tim, you’ve talked about this a million times, right? Like the promise of the value of like having the thing, having the data and in current conversations with clients, and I’ve seen it more like across the industry and articles, you know, there’s this whole like, well, what’s the actual return on investment for AI? And I just feel like AI has become this thing that really highlights to people that just because your data is there or you have all this data, right? Like even though AI can access it and use all of it that you’ve spent so much time and effort to create and surface, the output of AI is not inherently valuable to you. You know, like it is finally like the mirror to that flawed thinking. And so it’s interesting when you were saying like faucet first design, I do feel like everyone is now afraid to just be like, oh yeah, do another AI POC. They’re like, no, I need to know that my faucet in my business is going to work, not that you can go make plumbing in another room. Like I don’t care about that plumbing in another room. So I just, I really like that finally clicked for me in the conversation we’ve been having, but I’m curious your thoughts on like AI being scrutinized for value and this idea of like the data itself is not the value to the business. AI should be scrutinized for value, 01:00:49 | Rob Collie: you know, there’s a lot of pressure to just go do AI, right? Because everyone, everyone can sense the uncertainty of inaction, right? If we’re not doing anything about AI, we are falling behind, right? As a business. And so, but like what happens is the people who don’t know what to do about AI at all, they don’t know how to apply it, but they’re important. They’re like board members or whatever, they lean on executives and say, hey, you need to be doing AI, right? And the executive doesn’t know. So they turn to someone else and say, you know, you need to be doing AI. And then you end up in these weird, crazy things. We’ve all heard these stories and they’re real about people measuring teams usage of AI tools. And by the way, they’re always the off the shelf tools. So they’re the wrong things for most business uses, right? And making sure that people are using them, because if they’re not using them, then they can’t go back up the chain and say, hey, we are doing something about AI, right? If we’re in a really, really strange place where no one knows what to do with it, but the amount of pressure to do it is I don’t think we’ve ever been in a situation like this. Like BI wasn’t like this, right? Like no one has ever under pressure to do BI, like, unfortunately, right? They, you know, like the boards never came to their, you know, C suites and said, you should be doing BI. Well, I feel like personalization was a little bit 01:02:18 | Julie Hoyer: like that, Val. Like we heard that from a lot of clients, right? Like our goal is to do personalization. 01:02:23 | Tim Wilson: You’re like, but to what end? Like digital transformation. So like, I just think that 01:02:31 | Rob Collie: the semantic model case with an AI chat interface over the top of it is one of the easiest wins for AI adoption that’s actually going to add ROI. Like so for people who have Power BI investments, if they’ve got good semantic models, like we’re, we’re putting in AI chat interfaces. And honestly, they’re so much better than dashboards. There’s so much friendlier, there’s so much more user friendly, people actually get value out of them. And even the people who built the semantic models who know it, the ins and outs, even I use the chat interface to go and do all kinds of things for me to go research things for me that I wouldn’t do otherwise. It’s not even like saving me time. It, it, it, I form better questions and kick my buddy off to go research it for me. And again, it leans on the semantic model. It doesn’t, it doesn’t hallucinate. It gets real answers from real systems and comes back and, and, and, you know, and I can sort of, I can send my assistant back and, you know, to do more if I, if I want, you know, under all this pressure, and again, as data and analytics professionals, it is one of the easiest and most trustworthy and best places to start dipping our toes into AI is to start enabling these sorts of AI chat interfaces that actually work, not the ones that come off the shelf from the vendor. But if you build it, you build something yourself, like we’ve got this framework that we’ve built, again, we have these two developers that have been doing amazing things for us, like our clients freaking love this stuff and we love it. So I think it’s one of those, it’s not the only thing. I do not want to say this is like the future of AI, but it is a great place to dip 01:04:12 | Tim Wilson: the toe into it that does add value. And it’s right in our backyard. So I wanted to make sure we, 01:04:20 | Rob Collie: I got that, that little, you know, career advice. Yeah, it’s a great place to land the plane to kick 01:04:24 | Tim Wilson: off for show two. Yeah. All right. Well, that, yeah, there’s, there, I have so many more questions, 01:04:34 | Tim Wilson: so many more thoughts, but we also up against the clock. So this has been a fantastic discussion. And yeah, I think it’s even in this discussion for me in the book, lots of light bulbs have gone on for me. So this has been enormously useful to me, at least, and it’s all about me. But when we last thing we like to do on the show is go around the, go around the group and everyone share a blast call, something that something or maybe a couple of some things that might be of interest or amusing or entertaining or useful to our audience. And Rob, you’re our guest. So 01:05:14 | Rob Collie: do you have a last call or two? I do. I have one that’s relevant and then one that’s kind of just funny off, off topic. So the one I’ll obviously provide you the link, but an old friend of mine at Microsoft named Uli Homan wrote something on LinkedIn recently, where he talks about this new 01:05:30 | Tim Wilson: role he’s calling the product engineer. We’ve had software engineers, software developers, 01:05:35 | Rob Collie: all that kind of stuff. We’ve also had product managers who like were like the designers of it and all that kind of stuff. And he makes a very, I think, eloquent case for, I think, where we’re all going to land with this product engineer role. And so I think it’s a worthwhile 01:05:50 | Tim Wilson: read and it’s not very, it’s not very long. And I think it lands the concept pretty cleanly. 01:05:55 | Rob Collie: Then what’s the relevant one? No, you said it could be, it could be any link to anything that we’ve ever liked, right? That’s true. Right. So this is an article from 2005 on the McSweeney’s website. It’s titled, a realistic assessment of how many 12 year olds I could beat up before they overtook me. And it is such an analytical, dispassionate analysis that uses game theory and assumes optimal behavior on the part of the 12 year olds. It’s like, no, the 12 year olds aren’t going to get up in a nice line and let me take them one at a time. They’re going to form a circle, right? And they’re going to know that groin kicks are their primary weapon against me. And here’s how big an average 12 year old is and just really just breaks it down and eventually comes up with a very reasoned argument for why this certain number is probably the number 01:06:54 | Tim Wilson: that would take him. I love it. That’s so funny. Oh my God, that’s amazing. That’s really funny. Julie, what’s your last call? 01:07:05 | Julie Hoyer: Mine. Actually, I feel like this ties back to Rob, what you were talking about in your use with AI agents. A colleague recently shared a article called AI May Be Making Us Think and Write Moere Like. And it sparked my interest right away because I feel like multiple times on the show, I’ve said like, but what about like group think and like the cyclical, you know, like self-fulfilling prophecy. I’m like, if AI is then using our AI outputs to then continue modeling, I’m like, it’s all just going to converge somewhere. So anyways, this was very like fulfilling for me to see that there’s now been research that, yes, that’s actually happening. They’ve pretty much said that they’re seeing all of this really come to a point where everyone’s losing their unique voice in writing. And because of that too, what everyone is putting out is becoming more similar. So it is like this idea of the group think happening. And it’s funny because they say that AI solution, Rob, similar to what you did, is they were saying, you need to make sure you have a very clear framework to ask your AI model to enforce like strict stylistic constraints and make sure that you can extract your distinctive voice. So I thought that was funny. It’s like, AI is the answer, even though AI is the problem. And it was this whole, you know, echo chamber. 01:08:34 | Rob Collie: So it’s an interesting read. Wow. That’s not everyone’s starting to sound the same, but a lot 01:08:40 | Julie Hoyer: of people are. You can read something now and you’re like, oh, yeah, that sounds like AI. It’s 01:08:47 | Rob Collie: pretty quick. So I’m going to resist. Val, what’s your last call? 01:08:55 | Val Kroll: So there’s a AI tie-in for this one. But it is very self-serving because it comes from some of my co-workers at SIN. But my thing is, 30% of my last calls come from like UX collective, like this publishing group on Medium. And this could have very well been on there. So it’s a Substack article, all that caveat build up. And it’s called Designing the Front Door. And what I thought was really interesting is it’s like from this engineer and this designer at SIN thinking about how you create the front door to some of the agents that are not the helper agents, the chatbot, the little thing you see in the bottom right hand corner of your screen, right? But they’re calling it like the spotlight messenger. And what I think is really interesting about the article beyond the solution that they landed on is how they use data and experimentation and lots of research to figure out what is the optimal way to bring some of that to bear in a way that doesn’t harm other interactions, whatever. And it just feels like some of the research that I did when I was at further with the consent management banners four years ago and thinking about like, how do you surface this when there is no established pattern? And so it’s an interesting look at it with some fun examples. And it’s a quick read. So it’s a good one. So how about you, Tim? 01:10:17 | Tim Wilson: So I’m going to kick it old school with an ebook. It’s on a github.io site called Moedels Domestified, a practical guide from linear regression to deep learning. So if you want to step away from AI and just kind of understand generalized linear models a little bit deeper or causal modeling, I have not read the whole thing. I’ve kind of sampled it. It’s got kind of a lot of our a lot of the models are kind of ours available and Python. So you can kind of play with either one. But I feel like that maybe when I get overwhelmed by Claude and going back and forth, I it’s a good little thing to dip in to say, maybe I should just give some deeper statistical knowledge and a good old fashioned ebook. It’s a good way to do it. So that’s how Tim 01:11:11 | Tim Wilson: touches grass. That’s what you’re saying. Yeah, that’s what I’m not measuring the speed of sound 01:11:19 | Tim Wilson: waves through different materials. So you know, but you can, but you can’t. That’s the important 01:11:27 | Tim Wilson: thing. And can I just kind of throw one last little plug in here and you could edit it out if 01:11:34 | Rob Collie: you if it doesn’t it doesn’t fit. I just want to encourage people to go to fairgamebook.ai and preorder the book. I mean, I genuinely wrote this to help people writing books doesn’t make enough money to ever be worth the personal sacrifice. Like and so I was really gratified by y’all reading it in advance. And yeah, it’s it’s it’s for our crowd is for leaders. And I think it will actually help you not be so afraid. Like if you’re having some angst about AI, like my I felt like sitting down right in that book, my job was to help you feel better about it. Like I actually turn it into excitement. So I genuinely hope that helps people. 01:12:15 | Tim Wilson: Well, and I will say so you you took it from me because because I was definitely going to plug the book again. So I will I will further endorse that because I did find it to be extremely useful. We had a proof of it. So it was like a super early version. So I mean, it was a proof so we we could not we could not copy anything from it. We could not highlight anything in it. So like the the the notes had to be like old school retyping brilliant things. 01:12:43 | Rob Collie: And you know, you sit down you sit down with cloud code. And it’s wide open. You just 01:12:49 | Tim Wilson: you can do it. There’s no protection anymore. Okay, I’m not going to say maybe there wasn’t 01:12:56 | Tim Wilson: a way to throw the whole thing at AI and have it read it. Even when I’m making those clouds, 01:13:01 | Rob Collie: tell me, hey, you know, Rob, this isn’t going to actually protect you, you know. 01:13:06 | Tim Wilson: All right. Well, Rob, thank you so much for coming on. On the one hand, you’re like promoting a book, but the book is itself really useful. And this discussion, I think was extremely useful. So thank you for taking the time to come on like this. And really, I think y’all I mean, 01:13:22 | Rob Collie: again, I hadn’t met y’all before now. I really think this is a I love your format. I love your approach. I think you’re doing great work here. So I really appreciate your the invitation. 01:13:33 | Tim Wilson: Well, awesome. So what you could do and any of our listeners could do or you could have an agent do is go leave us a review and a rating. Look at that. Look at that segue. Oh, I’m on it. Make Michael proud. How many reviews do you want? So if you listen on a platform that allows leaving reviews, listeners, we would love to get reviews and ratings that helps us out. If anybody, if you’d like a sticker for the show, you can go to analyticshour.io and fill in the little form there. We’ll send you stickers. We also love to hear from our listeners so they can you can you can reach out to us on LinkedIn on the measure slack. You can just email us at contact at analyticshour.io. But regardless of how many faucets you’ve turned on, if the faucets first, if the plumbing’s first, if you’re deep, deep into cloud code already, or if you’re now just inspired to get into cloud code, no matter what you do for myself, for Julie, for Val, keep analyzing. 01:14:45 | Announcer: Thanks for listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at at analytics hour on the web at analyticshour.io, our LinkedIn group and the measure chat slack group. Music for the podcast by Josh Grohurst. Those smart guys wanted to fit in so they made up a term called analytics. Analytics don’t work. 01:15:10 | Tim Wilson: Do the analytics say go for it no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning 01:15:21 | Julie Hoyer: in competition. It’s all semantics. Yeah, it is indeed. It’s a challenge. 01:15:28 | Rob Collie: And I saw your, I saw what you wrote in the doc in response and I’m like, oh, yeah, so that’s 01:15:32 | Tim Wilson: okay, I’m not gonna ruin it. I mean, I will try. I mean, it’s not like we can’t like 01:15:39 | Tim Wilson: touch on it at all. I just think there’s like so much other like gold for us to hit on. As the moderator, I will say, you know what, I’ll allow it, but you’re going to be on a shot clock. Okay, I got it. I mean, we’re going to, we’re going to barely scratch the surface of everything I think we would want to talk about. And that’s just a sign of a good show. So unless Michael Helbling is running the show, in which case it’s because he has lost control 01:16:08 | Tim Wilson: into the terrible job. And Tony, feel free to put that in the outtake. We always bully him 01:16:14 | Julie Hoyer: into one more question, you know? 01:16:29 | Tim Wilson: Rock flag and data genes to crafters. The post #301: It Turns Out Analysts Are Natural AI Crafters appeared first on The Analytics Power Hour: Data and Analytics Podcast .

June 23, 202658 min

#300: Are Semantic Layers Really Necessary?

If you’ve ever poured months into building a semantic layer only to watch it become shelfware the moment the business pivoted, Jacob Matson has some thoughts. And a metaphor. Your data is a jungle—and a semantic layer is a highway. Great if you need to get somewhere fast and reliably (monthly active users: highway, please). But the interesting business questions? The slicing, the dicing, the nuanced dimensions that actually differentiate your company from its competitors? There’s no highway for that. There never will be. Jacob, a developer advocate at MotherDuck with deep roots in accounting and ERP systems, joined Michael, Moe, and Julie to talk through what comes after the semantic layer—or at least alongside it. The conversation covered why the most important parts of any business are precisely the parts that resist being modeled in someone else’s framework, why AI is actually pretty good at writing SQL but not so great at remembering what it figured out yesterday, and whether the real job to be done here is less about modeling and more about search. Oh, and the uncomfortable truth that at episode 300, we still don’t have a great answer for metric drift. But we’ve got some really good questions. This episode is brought to you by Prism from Ask-Y —your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. This episode is also brought to you by Stape , your all-in-one solution for server-side tagging. Links to Resources Mentioned in the Show What if we don’t need a semantic layer? by Jacob Matson The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling by Ralph Kimball and Margy Ross SkillOpt: Executive Strategy for Self-Evolving Agent Skills Wordslut: A Hilarious and Informative Exploration of Gendered Language and Its Impact on Women by Amanda Montell PostHog’s next chapter by James Hawkins Photo by Jens Lelie on Unsplash Episode Transcript 00:00:00 | Announcer: Welcome to the Analytics Power Hour: analytics topics covered conversationally and sometimes with explicit language. 00:00:13 | Michael Helbling: Hi everybody, welcome to the Analytics Power Hour. This is episode 300. This is Analytics Power Hour. Okay, sorry, that was just a dumb joke on the movie. Okay. Every time you turn around. I think you’re hearing about the semantic layer. We even did a show on the topic recently. It’s what AI needs to be successful. Well, that’s at least what we keep hearing from the vendors. And well, honestly, that was right around the time we ran across an article that grabbed our attention. What if we didn’t need semantic layers? Then we read an article by OpenAI. They published about how they’re using AI to analyze data and took a closer look at a couple of vendor websites. We started seeing the context for AI isn’t only a semantic layer thing. And well, we wanted to talk about that. So let me introduce my co-hosts, Moe Kiss of Canva. How you going? I’m going great. Thanks for asking, Michael. I see you’re down a remote on the wall there. So that’s… Oh jeez. 00:01:14 | Moe Kiss: I love when we give a visual in-joke that no one else can follow. 00:01:18 | Michael Helbling: Yeah, for an audio podcast. It’s okay. We’ll make a clip out of it. Julie Hoyer of Further. Welcome. 00:01:27 | Moe Kiss: Glad to see you. 00:01:28 | Julie Hoyer: Hello. Hello. 00:01:30 | Michael Helbling: Glad to be here. Awesome. And go Browns. And I’m Michael Helbling. So naturally, we reached out to the author of one of those articles. And I’m excited that he is our guest. Jacob Matson is a developer advocate at MotherDuck, the cloud data warehouse built for answers. He has also held senior data and accounting roles at firms like Simetrics, Funko, and Verimatrix. And today he is our guest. Welcome to the show, Jacob. Hey, Michael and Julie and Moe. It’s great to be here. I’m super pumped. Awesome. Well, we’re excited to have you. So I think, Jacob, maybe to kick off the conversation, I think it would be great for us to understand a little bit more both about your background and exposure to this and sort of what started formulating for you that led to you kind of digging in and doing research in this area around sort of semantic layer or not or other alternatives to semantic layers. 00:02:24 | Jacob Matson: Yeah. That’s such a good question. I guess I’ll start with a little bit of biography that got us here. I’ll try not to be too self-indulgent. So when I graduated from college, I worked in accounting, in public accounting. I sat for the exams, did all that stuff, and worked in public accounting and then had a 00:02:43 | Moe Kiss: company called Verimatrix that was called out there, doing all of the normal accounting 00:02:47 | Jacob Matson: things, climbing the ladder in a very specific kind of governed way, working for people with titles like Controller or CFO, and eventually taking some moves on myself. I think one of the things that we always talked about was especially on the financial side was like, how do we know what numbers are right for this definition of this thing? And at the time, the tools we had were much, much worse than we have now. I remember it being a big deal when we got the Excel that could handle 1 million rows and not just 65,000 rows. That’s why I think it was Excel 2003, maybe, which of course, actually, what it all did was train everyone to just have a horrible experience in Excel all the time and just be totally fine with it. It’s just too much information, it can’t handle it, and then we all dealt with saving issues and crashing and all these things all the time. 00:03:34 | Michael Helbling: Yeah. Don’t calculate your metrics until you’re really ready. 00:03:38 | Jacob Matson: Yeah, exactly. Turn that automatic calculation off. That’s right. Yeah. Step one. And then you can do that as a binary format Excel file. So did all those things and had the pleasure of working on things like MDX and DAX along the way, which are both the modeling languages that are built into the Microsoft stack. And along the way through that journey, really found my way towards using SQL for a lot of the work I was doing, and that just naturally came out of the data that I had that was too big for Excel and it was too complicated. And there was lots of really, I was just driven 100% on just the business need for solving these problems and I needed to get more robust tooling and we had SQL server and it had more, it was running on a server that had more compute than my laptop and all these things. And so it was very natural to kind of progress up there and so I’ve done lots of fun things kind of in that space. I kind of like the joke that I always worked in data from the beginning of my career. It’s just my pipelines ran like once a month, right? It was a month in close process for those of you at home. And it was just kind of how I got there. And so you do a lot of things along the way and you see lots of errors along the way too. Some material, some not, right? And I worked on the IT side at a public company and you see lots of interesting things produced internally that never make their way into the filing documents, for example, sent to the SEC. So I think for me, kind of some of the genesis that led to this notion of like, do we need semantic layers anymore? It was like two things, A, working in accounting for a long time and like understanding the quality that goes into those numbers, which is very high, but also not as high as you’d like. That’s what I would say. And the second part of that is that like we often, at least what I would see on accounting, the accounting side was like, we would get too precious about the exact accuracy and precision of a number instead of instead of actually moving the business forward, right? And so what actually happened in my career, at least, is that like it seemed like I, you know, I wasn’t that close to it early in my career, but like what it felt like is that like CFOs in particular, completely abdicated the realm of analytics to like some other domain. Like, you know what, we can’t get accurate enough, you know, it’s not good enough for whatever reporting we’re building, we’re just going to let some other part of the org like handle that. In fact, I remember even seeing like job descriptions that were like, director of analytics, non-finance, like type of roles. I experienced that when I was at a company room really fast and was trying to build all of these things. And I built, you know, our first day to warehouse from very much, you know, accounting principles first, and eventually just got to the point where we had to break that apart because it was just, we didn’t have the right primitives to answer the questions in a way that we were, we were comfortable with. And we tried all the things and it was just really hard to manage and to update. So a little bit of this idea is me manifesting like, what if I just took away all the pain that I experienced when I was like building these analytics cubes back in the day? Like what if we could just like ask AI those questions and it would like reformulate those on the fly? And so that was kind of like what led me to exploring the idea and then beginning to do like research around it. 00:06:44 | Moe Kiss: Can you tell me a little bit about the tension that you just, you kind of touched on, but we didn’t go date the tension. So I mean, we talked about this with Cindy a little while back about semantic layers and just like, it’s been sold at the moment is like the holy grail as every new, not actually new idea in analytics is that all will solve all of our problems. But that like, that core tension that you touched on on how hard it is to maintain the like inflexibility perhaps that then makes you not able to answer your business questions like what were some of your lived experiences? 00:07:19 | Jacob Matson: Yeah, this is such a good question. I mean, I think like the first one was, I think we, so I was working on an ERP system and we were like, hey, we want to implement like better reporting analytics on it. We’re going to buy this software package that will just like automatically build out all the OLAP cubes for us and then we can like tie them into our BI tool. I think this was even like pre-Tablo maybe and well, Tableau existed, but it didn’t exist for the company I worked at, I think that’s what I would say, it was certainly not something I was tracking at the time, you know, we bought the software and then I was like, okay, now let’s like implement it. And it was like, it had to fit in a very narrow box for us to actually take advantage of it. And that was a pattern I saw repeated a lot kind of in the ERP space too, which was like, hey, just like, you know, make your business fit into this template of how we run our systems. And then you get all of these awesome synergies or whatever, right? Like now you don’t need people to do your buying. You just like run this report and it tells you what to buy. And so what I kind of came to believe, I think like, I definitely wanted to say, all right, let’s just like apply this system, let’s apply these SAP primitives that are, you know, very old and well tested, let’s just apply this like blindly to our business. Like why are we over complicating it? Like our business is not this hard. 00:08:35 | Moe Kiss: But then what I kind of discovered is that the interesting parts of your business are 00:08:39 | Jacob Matson: really hard to define in someone else’s model, right? They end up being, unless you’re like a pure commodities trader, like the magic happens kind of in the margins. And so like defining those systematically is super hard. I really struggled with that. And so like when I kind of realized that that’s or like that was the mental model that I was bringing to these problems, I started being like, hang on, how do I design this ERP system that we’re working on that was my accountability? How do I make it so that like we can do the thing that we need to do, you know, to make the system work? Also, we allow kind of space in the way that we interact with this so that like the magic of the company would differentiate the organization can still happen too. And so like once I started thinking about it that way, that really kind of unlocked for me kind of a way for us to move forward. And it was much less difficult to kind of get people on board because it wasn’t like, hey, we’re going to change your totally change your job and make it so that like it just fits into this box. It was more like, okay, how do we meet in the middle? And so, you know, I think there’s like a, I think there’s a little bit of a paradox in that, right? Which is like the paradox is that you need some level of conformity across the organization for like everyone to be able to communicate well. But also if you have too much conformity, you have a commodity and you need space for your, you know, to have some sort of differentiation. And so I think like that’s kind of the perspective I brought there, you know, I think figuring out which constraints, I kind of like think about it sometimes like, like this game Jenga, I don’t know if you, if you all played that, but you have like a stack of blocks, right? Some of them you touch them and you’re like, okay, I’m not, I can’t pull that one out. That one has to stay there. It’s like, but like you only figure that out like kind of existentially, right? Like you don’t. So like for me, I spent a lot of time just like trying stuff and like, okay, you know what, that didn’t work. The CFO just got really mad at me, like we won’t do that, but like, let’s, let’s just, let’s just, let’s try this other, other path. And I think a lot of it just became like, and then at the end you have this beautiful tower, right? Hopefully you don’t knock it over, but you have this beautiful tower and that tower is unique shape that like hopefully fits what the actual, what, you know, actually represents what the business is. 00:10:56 | Moe Kiss: And so that’s kind of what I think about it. Hey, Tim. 00:11:04 | Michael Helbling: Have you ever opened GTM preview mode and immediately thought, well, there goes my afternoon. 00:11:09 | Tim Wilson: Absolutely. Nothing says fun like hunting through a giant pile of tags, trying to figure out which one 00:11:13 | Michael Helbling: broke. Yeah. That’s why state built, state GTM helper, a free Chrome extension for debugging Google tag manager. 00:11:21 | Tim Wilson: And free means actually, well, free, no sign up, no subscription. Just install it from the Chrome web store and start debugging. 00:11:28 | Michael Helbling: Yeah. And it works with both web and server side GTM. It helps you focus on what matters by filtering down to the specific tags you’re testing. 00:11:37 | Tim Wilson: Your tags from Google, Meta and Microsoft are color coded, so they’re easy to spot. And it makes JSON payloads readable instead of whatever they normally are. 00:11:46 | Michael Helbling: Yeah. And for server side GTM, it gives you better visibility into consent status and it can help with Shopify checkout debugging too. 00:11:54 | Tim Wilson: There’s even a Website Tracking Checker that gives you a web and server side tracking report with actionable fixes. 00:12:00 | Michael Helbling: The state GTM helper is a must have for anyone deploying our managing tags in GTM, search for state GTM helper in the Chrome web store, or use the link in the show notes page on our site. It’s free, installs fast, and might just save your afternoon. 00:12:17 | Tim Wilson: Michael, where does your best AI analysis live right now? 00:12:22 | Michael Helbling: Oh, I’ve got this Claude conversation called GA4 help for this meeting I’ve got coming up. I’m buried between a lunch recommendation and me asking it to explain regex to me like 00:12:34 | Moe Kiss: I’m a fifth grader. 00:12:36 | Tim Wilson: Exactly, that’s the problem. Your AI work gets trapped in one chat with one person in one thread. Ah, yes, the modern knowledge base. I swear Claude told me this somewhere. And that’s why I ask why I built Prism with memory and shared context across users. So the useful stuff doesn’t vanish into my private little AI cave? Exactly, it’s out of the cave into the sun. Prism keeps the context, your metric definitions, source of truth tables, business rules, prior analyses and makes it usable across the entire team. 00:13:12 | Michael Helbling: I like this. So if I teach it that active user means three sessions in 30 days, Julie doesn’t have to teach it again tomorrow. 00:13:19 | Tim Wilson: Exactly. And if Val runs a GA4 cohort analysis, that knowledge can live in Prism, organized and traceable, not locked inside her chat history like a tiny little analytics hostage. 00:13:30 | Michael Helbling: I am starting to like this team memory, not ask Michael because he remembers the cursed dashboard lore. 00:13:39 | Tim Wilson: Plus, with Claude co-work in Prism, your analysis becomes shareable, auditable and ready to build on. 00:13:45 | Michael Helbling: I like this. So the AI becomes company knowledge, not just some vibes I had with the chatbot at 11.42 00:13:52 | Tim Wilson: p.m. That’s right, because that’s way after my bedtime. So go to ask-y.ai and join the wait list. 00:13:59 | Michael Helbling: And you can use the code APH and I’ll take you to the top of the list. That’s ask-y.ai code APH because your team’s brain should not be trapped in one person’s chat tab. 00:14:13 | Moe Kiss: Exactly. Everyone can see my face, obviously not our lovely listeners, but everyone in the podcast. I love an analogy. My company loves an analogy. And I feel like this Jenga one is going to take away too far because at some point you do knock it down. That’s the reality of when we build data architecture and systems, at some point you do end up rebuilding. But I think the exact tension that I feel right now, and I keep banging on about 80-90%, we do need standardization. We do need some conformity because otherwise, if one person over here calculates it this way and one person don’t, we can’t ever have a mature conversation. But I think the really challenging part is how we get that 10 to 15 or 20% that should be bespoke or is actually a unique situation. And I’m thinking about those are the Jenga blocks that you push through and you can move. But you just have such a wealth of experience here. It sounds like for you a bit of that was trial and error. If I want to learn from all your trialing and the erroring, how do you figure out what the standardization bit is and where the flexibility needs to be? 00:15:27 | Jacob Matson: Oh, that’s such a good question. I think some of it is being able to zoom out and understand what the engine is of the company. How does it function? What’s differentiating about your competitors? But then also how does that build the feedback loop that ultimately increases the cash on the balance sheet, hopefully? I always felt like that was an advantage for me as someone coming from an accounting background where I’m just like, it’s very easy for me to visualize, okay, if this business is great at X, they will increase their cash flow. And so I think a little bit is developing good intuition around that and I’m very thankful to have grown up and working for CFOs who are very excellent mentors as it related to that. But I think the second part of that is your model for reality is imperfect. And so you need to be able to test that model in a way that is sort of safe. And what I mean by that is you don’t get fired if you’re wrong, you might get reprimanded. That’s okay. That’s the threshold. Yeah, exactly. You can take some risk, but you want it to be the right risk. And so I think a lot of the trial and error part was just how do we take some risk here that is not too drastic, but is opinionated in a way that if we’re correct, we win more. 00:16:46 | Julie Hoyer: Do you have an example of what that risk is? I don’t know why. I’m having a hard time conceptualizing the risky metric. In the previous episode that we talked about semantic layers, I think you had thrown out the example of monthly average users when we were talking about semantic layers. Is that a risky metric? Is that a metric that moves the business forward? Can we talk through a metric like that for a business like Canva? 00:17:12 | Moe Kiss: I think monthly active users is one that I wouldn’t take a risk on. And that’s because it’s one of our company foundational goals. So we have long historical reporting, but actually where it does get complicated, right? And I’m going to give you a specific example is we often will look at monthly active users by different products. And sometimes different people have different interpretations of what that product monthly active user is. And that’s why there’s so much devil in the detail, right? Like it’s such a gray zone because someone might be like, oh, anyone that used any kind of like video or social media and some people might be like, well, it’s only video if you did X, Y and Z, like you were in our video editor and you used advanced video features. And that’s why it’s like, it’s just messy, our jobs are messy. Totally agree. 00:18:04 | Jacob Matson: When I think about risk, I guess like I would put it in a slightly different, I would not necessarily frame it as like analytics first, but I would just say that like I left a job and then like a month later, someone on the team who was still there sent me a message and was like, man, it has been rough since you’ve been gone. And I was like, why? And I’m like, everyone knows all the things that was nothing interesting happening. He’s like, well, no one’s making any decisions. And I was just like, oh, yeah, okay, I could see that. And so I think like some of it, like when I talk about risk, I just honestly, I’m just like, make a decision, right? Like be opinionated on what it means to have a monthly active user by product X, Y, X, Y and Z, not said, sorry, you know, and like, you know, maybe that there’s lots of interesting things that happen when you start breaking those things apart, right? And you know, one thing that I think I was well trained on because I was in accounting is you get really good at like delivering bad news, like, you know, and so you’re always in the, you’re always, you know, one of the first objectives in accounting is like, you want this to be true. You want it to be the numbers you’re showing are a reflection of reality as you understand it, right? In a way that is defensible. And like sometimes when you’re dealing with metrics, especially with product teams, like, you know, they want to show that their thing is working, right? And like what you’re, you know, it’s a tension, right? Between the domain team and like a central team, which is, okay, like what is truth to that, you know, you know, to the company and like what moves the business forward. And I would almost always say that like, we want to be measuring things in a way that when they’re tested against reality, they, they’re proven to be right, right? And so when people are bringing agendas into things like, hey, like I want to define something in a way that says, you know, I get more monthly active users, well, if that’s not moving the company forward, that metric when it’s tested is going to fail, right? And so how do we, how do we do that? How do we test them more closely against reality is a really interesting question. And the reason the way we do that is by like, you know, potentially taking, taking bets and like making decisions on them, right? 00:19:59 | Moe Kiss: Can I, can I, I’m taking us completely down off topic as per usual. And I just want to push on this a little bit because I do see this happen and I’m curious if your experience in accounting has perhaps given you confidence or like you’ve built the confidence to sometimes have an opinionated decision. Whereas I feel like often in data lands, sometimes there is this like desire to debate every which way something can be cut and like maybe like make a proposal, but like not often enough be like, you know what, I’m going to have an opinion here of like, we’re going to calculate it this way. Let’s do this. Let’s move the business forward. Like, do you think sometimes like that, is that your accounting background? Do you think that helps you have that perspective because you’re more willing to have a position knowing it’s the best, the best of where you can get to, or do you think that’s like you personally, like what do you think’s driven that willingness to take a gamble and have an opinion? 00:20:58 | Jacob Matson: I mean, I think it’s a little bit of self selection. Like part of why I liked accounting was because it let me do things like that or like gave me a framework to reason about those things, right? I think I’m probably a little contrarian by nature. And so, you know, when I see people getting like too precious about metrics, I’m definitely just like, let’s make a decision, let’s, let’s figure it out and we will test it. And if it’s wrong, we can fix it, right? One of the things that’s great about, you know, analytics in general, compared to, I don’t know, financial numbers that you’re publishing to your board or whatever is that you have a lot more degrees of freedom in terms of what it looks like to go back and make something better. You gnow, one of the biggest challenges, right? And analytics is like, hey, that number got put into, you know, our regulatory filings. So that’s how we do that now. You cannot change that anymore. So like, obviously, like, you know, you don’t want to take, you could, I don’t know if this happened to me specifically, but I’m sure, well, actually, yes, it has. Or we made up some, some way to bin some set of data and then suddenly it was in, you know, annual reports. And now it’s like, okay, now we’re always presenting that. And if we had known, if I’d known, I think from experience would have been like, hey, let’s be a little more precious about this. And I think like, it’s definitely a tough balance, but we don’t need to be perfect. Right. We can be, we, as long as we kind of know, you know, and it’s justifiable and we can defend it, I think we can go pretty far. But like, you know, there’s risk, right? There’s risk that you could be wrong. 00:22:23 | Julie Hoyer: Do you feel like of all the context in a business, what percentage of it that people use day to day, like when creating metrics, defining metrics, like doing their analysis, making decisions, what percentage of it is actually captured in a formal, like static 00:22:41 | Moe Kiss: semantic layer for like broad knowledge compared to they’re just doing it, like adding 00:22:47 | Julie Hoyer: in their own context and their own SQL queries and their own, you know, the way they’re pulling the data. 00:22:51 | Jacob Matson: I mean, now that I work in marketing, it’s a lot less than it was when I worked in finance. So it’s contextual, I think. 00:22:59 | Julie Hoyer: Yeah. 00:22:59 | Michael Helbling: There are no generally, there are no generally accepted analytics principles, if you will. Yeah. Yeah, sure. 00:23:06 | Julie Hoyer: Yeah. I feel like it’s a small percentage, smaller than maybe people want it to be. And do you feel like people are always fighting to like make it as close to a hundred as possible? 00:23:16 | Jacob Matson: Or I think like the tension is that like, everyone wants the risk to be low. Like, hey, if I’m going to use data to make this decision, well, then it better be right. The data better be right. And therefore, I’m not going to use the data because I don’t want to take someone accountability for someone else, you know, something produced by someone else. I want to take my own accountability. You know, I think that’s a core, definitely a challenge. Do I see it moving towards a hundred percent? 00:23:40 | Moe Kiss: I mean, I think like, if you’re moving toward a hundred percent, like you just 00:23:44 | Jacob Matson: automating the entire function, right? I mean, I guess that’s like Google AdWords bidding, right? Like, okay, the whole thing’s automated. The price is the price. Um, in some ways, that’s like the fully actualized form of analytics, right? Like auction pricing. Um, do I think that’s the right way to do it? Like, I don’t think, you know, most, most jobs don’t are not that, not that straightforward. I think that it’s probably a smaller percentage than, than most analytics people would, would want it to be and probably, you know, roughly around the right number for where, where things are today. 00:24:19 | Michael Helbling: All right. I want to start to pivot into what we actually need to talk about, which was sure, let’s say you’ve been struggling with the semantic layer and you’re running into all the problems that semantic layers kind of introduce, you 00:24:33 | Moe Kiss: know, they’re inflexible, uh, not easy to pull together. 00:24:40 | Michael Helbling: Don’t work well across different departments and teams. Like there’s lots of reasons why a semantic layer is a challenge and, and lots of people spend a lot of time on it. But like, what are the alternatives? What are people doing to lower their dependency on the so-called semantic 00:24:55 | Moe Kiss: layer that is sort of like the, uh, favorite of the AI world right now in 00:25:01 | Michael Helbling: data? 00:25:02 | Jacob Matson: I mean, I think like, you know, ultimately what we’re seeing a lot of right now is that everything is kind of going into the notion of like skills, right? Which is just marked down, marked down on your laptop or in a GitHub repo or somewhere. And I think people are capturing a lot of context that way, that they are ultimately using either personally or sharing inside their company. I think there’s a lot of new service area here for products. Um, I know like, for example, inside of, uh, Claude, they have some of this notion called projects and projects that you kind of put marked down in there and then link, link it to other things. Um, and then whenever you ask a question, you can select a project and then you’ll, you bring context along, you know, with it. So I think we’re seeing that. I think we’re, we’re definitely seeing like vendors coming along in the space. You know, we’re seeing, we’re seeing a lot of like, we’re also seeing like the perspective from the labs, right? The labs who haven’t limited tokens are just like, Oh yeah, we just like, you know, use AI to do, do everything. Like we’re just like, you know, maximizing our tokens, spend, solve these problems. I think the one that I saw that was, was killing me was like, open AI has like a analytics agent and they basically say, all right, well, we’re just going to ask every question twice. So basically we’ll ask, you know, the user will ask and then we’re going to reformulate it and then have our, you know, our background agent, just see if it gets the same answer. And if they’re too far apart, we’re going to, we’re going to escalate it. And I think like, you know, certainly that’s one approach. Uh, I, I can’t imagine it’s cost effective for anyone at reasonable scale who’s not a lab at the moment. We’re seeing lots of ways that people do it. I mean, from, from what we have been working on, you know, at mother duck, we’ve started to do is just, um, put context in a database because of course we’re a database vendor. So like put it in database database or AI is really good at writing SQL. It’s really good at retrieving the right thing. Um, and so when you do that, then you can start, you know, um, treating it in a more structured way, right? Whether that’s, uh, you know, more of like a graph that has like nodes and edges that you can use to like navigate across, or if it’s just like, you know, straight up comments on columns or whatever, which is a, you know, old, old part of the SQL spec. 00:27:05 | Moe Kiss: I am very much feeling the semantic world bubbling at the moment and the pressure of it and the perception that it’s going to solve a lot of our problems or the perception that it’s required to do AI and data well. I think the thing, one of the points that you made that really resonated with me in the article was about semantic layers being static and how challenging that is, but I think the real like thing that’s keeping me up at night, right? Is if, if we don’t go down that semantic layer path, it’s the validation, right? Which, which you just touched on, right? So the bit that’s challenging that I, I see pop up constantly is if we don’t have somewhere for people to self validate, I find that’s, that’s a hard thing that we, I want to solve for because I don’t, my job to be, to QA other people’s shitty outputs and AI hallucinations with data, which it feels like I spend some time doing now. And so like, and I guess, I guess what I’m trying to say is like at the moment, I feel like it’s binary that you either have some type of semantic layer thing where people can validate or you go down the evaluation framework. Am I, am I totally off here? Like, is it one of those binary options? Or is there just like a range of options? And I haven’t thought deeply enough about it yet. 00:28:28 | Jacob Matson: I mean, I think the first question is about the interface, right? That you let users interact with, right? If they’re interacting with a spreadsheet, there’s different set of constraints than if they’re interacting with a BI tool, than if they’re inner interfacing with like a chat app, right? There’s more engineering freedom, I think on, on obviously a chat application, which is what, you know, people have proven to love, you know, just asking questions to a chat bot, than there is on like a BI tool. I think that one of the biggest challenges on using, you know, even the best BI tool in the world at this moment is that it’s very difficult to interface with something else that someone else built. Like you’re just, I think one of the things that I’ve really found kind of when using AI generally is that like the closer the framework you’re using to answer questions is like a snap fit to your own brain, the easier it is to like use it and use it well. When you’re using someone else’s model, like even a really well-defined model by a really good engineer or really good BI analyst or whatever, it’s like, that’s their model. That’s not your model. That’s not necessarily how you’re thinking about the problem space. And like the biggest challenge that LLMs solve is they’re really good at translating between languages, right? And that really means they’re really good at for me as a, let’s say, someone working at marketing to ask a question and then have the LLM reframe it to be like, oh, I see what this really means is, you know, it means this and, you know, that translates to this language in your current model or whatever. And so I know I’m not really that kind of answering your question. I don’t know, like, I think there is a spectrum. I don’t know if we know if, like, I think the products are not super mature as or outside of the semantic layer because the semantic layer buyer, I think traditionally has been very risk averse. That’s why they’re buy it. The reason you buy a semantic layer in the first place is because some number got somewhere and it was wrong and legal’s pissed. You know, like that’s how you buy a semantic layer. I’m being maybe a little too cynical, but like only maybe. Um, so I think like figuring out how to, like, how do we, again, I keep, I keep saying this word risk, but like, it all comes down to like, how much risk do you accept? And like that determines the spectrum of tools that you can implement. Right. Um, if you can, if you can take on a lot of risk, like in marketing analytics, you can probably take on more, more risk than you can in financial analytics. That’s just true. Right. The cost of being low of wrong is way lower in marketing than it is in finance. That’s just true. Like that’s, that’s a physics problem. Um, so I think like you may not have the same solution across the entire company. It just depends, you know, horses for courses, as they say, I suppose. 00:31:07 | Moe Kiss: Just to be clear, I will keep asking Jacob 50,000 questions. 00:31:10 | Michael Helbling: I’m trying to make space for other people by not speaking. That’s okay. And we, we added out dead air. So don’t, don’t worry about that. That’s fine. 00:31:18 | Julie Hoyer: I wanted to ask actually, Jacob, if you could talk about your proposed solution in your article that you talked about, like using AI, because everybody’s obsessed with having a semantic layer to help with AI, but you kind of flipped in said also AI could help with the semantic layer problem. 00:31:35 | Jacob Matson: So when I wrote the original article, what I was really thinking about at the time was, uh, the notion of skills and using skills kind of locally on your machine to, to kind of codify the, the way to go here. I think I’ve gotten a little more nuanced recently, but like also I think that we’ve seen a lot of mature maturity and development, especially like in the anthropic ecosystem with projects inside of, inside of cloud, for example. I think it’s a very natural way to think about it is like, how do I, how do I make it easier to maintain and, or actually create and maintain and skills are incredibly, incredibly easy to create and maintain, maybe too easy, right? They may not be the right abstraction. But certainly I think skills is the way that I, I thought about it or, you know, at the time, and then I just, you know, and I’ve been doing kind of evals in 00:32:23 | Moe Kiss: that front against, you know, benchmarking data sets. 00:32:27 | Jacob Matson: And, you know, we can debate the, the efficacy of benchmarks versus, you know, real, real life data and all these things. But I think what we do know is true is that if we can find the right context, we can very reliably return the right answer. And so I think like where my, where I think my, where my thinking has gone recently is, is how do we make our data easy to search, right? Which is a different way than maybe we’ve designed it before, as like a Kimball model, which is like, how do we make it easy to retrieve? You know, there’s a whole bunch of really good research that those guys all wrote around how to make a data warehouse and how to make it work and easy to retrieve. And those were written in the constraints of the time, right, which I think probably the original Kimball book is probably going on 25 or 30 years now. We have better technology now, and maybe we can revisit some of those assumptions. And so my, I think my current kind of position on this stuff is that we really have a search problem. And if we can find a way to make our metrics searchable, then the way that we define it may be less important. It might be a semantic layer that’s in YAML. It might be something that’s more, that’s more well defined than that. It might be, you know, something more programmatic than even YAML that is like kind of like a SQL alternative. But like what I, what I have also found is that like LLMs in particular are so good at writing SQL and understanding it, but like adding another language that is new and bespoke is really hard to get good results out of compared to just using the trusted good old thing that is very verbose and like has weird syntax and like has a whole bunch of downsides, but also like there’s 50 years of training data in the, in the training set, right? So if we can figure out how to say, how do we make it search to find the right thing and then write the right SQL, we can get really good results. And I think that’s what I’m seeing is most promising, you know, at this moment. Can we separate search from the context? 00:34:20 | Moe Kiss: Because I feel like those are two very different things and where you struggle with both. So search is more about like, how do I point it in the right place? How do I help find the right table or whatever it is, right? Whereas the context is more, how do I understand this table? And is, okay. And do you think that what you’re proposing, the way you’re thinking about this solves for both equally? Or do you think perhaps like, I feel like we probably need a different approach for each or different thinking? The answer is the devil’s in the details, I think. 00:34:57 | Jacob Matson: So let me, I think where I’m struggling with this question is like, if you assume that there is a natural language interface, right? And then maybe some sort of tooling like an MCP or something that has a search tool. Well, that’s, you$know, someone should solve the search tool problem, right? Which will say, hey, let me search and find this context and return it to you. And the next thing is, you know, writing the SQL based on that understanding. I think the second part is basically solved, which is if you give good context to an agent and say, like, describe a set of tables and then ask a question, you get really good results. And, you know, the challenge we have, of course, is that like, they don’t have good memory. And so like every day, you have to remind them. And so how do we make it so they have good memory? And there’s a whole bunch of people, I’m sure, working on that problem. I’ve read quite a few papers, you know, in the space of like, how do we make it? So when we ask a question, you know, the next time, the next time we ask it, it’s faster to get the answer. Or we already know, you know, we already have some kind of, you know, pathway built out. The metaphor I kind of think about is like, your data is like a jungle, right? Unless you really know where the things are. It’s really hard to find stuff. We can build a semantic layer, but a semantic layer is like a highway. Right? It’s like, we are just like building the thing straight through the jungle. I would almost say like, we can build it a little more progressively, which is, hey, we need like these little paths, right? Maybe these paths are just wide enough for a human to walk on. And maybe this one we can put gravel on. And maybe this one we can put gravel on. And maybe this one we can pave. And then maybe this one we can build a highway. But like, so I kind of think like, I think this actually goes back to the spectrum question you asked earlier, like there’s probably a spectrum of solutions here where a certain set of questions need to be on the highway, right? Like if you’re one of your key metrics as monthly active users, that needs to be right all the time. And there’s a highway for that, right? And so the search problem is like, can my agent find the highway? Okay. And if you can find the highway, then you get the answer. And then, you know, there’s a bunch of, but the interesting stuff, the reality is the interesting stuff is when you start slicing and dicing by all these arbitrary dimensions. And then like, there’s no highway for that. I can’t afford to build that even in the age of AI today. And so, you know, how do you make it easy to do a little bit of off-roading and then still get the right thing, right? And I think like this is where, you know, I think I’m, I’m in particular probably well served by like, you know, accounting principles as in building this stuff out. And I’m just like, well, what if we just made this a ledger and we can just like, you know, walk forward and backwards through time or whatever. Now it becomes very easy to trace it back to the highway. You know, there’s a whole bunch of abstractions like that we can, we can talk about. But I think like the answer is probably, you know, and not, not or in the long term, I do think that there’s probably some set of questions that are so important that you need to always get the right answer. And that might mean maybe a different interface. Like, hey, you know, for our financial reporting interface, we use X for marketing, we use Y. I don’t know. 00:37:46 | Moe Kiss: Do you think it depends on stakeholders too? Like I’m just thinking. So for example, if you’re a data person, the context is something that exists within you. Like you have such good context. So you know how to prompt, you know how to give the right context. And I’m talking as you’re like in the build stage and this is evolving, right? Because if we have essentially, you know, you’re kind of suggesting like an evolving semantic layer or a knowledge-based graph base, at the earlier stage, right, when that context isn’t fully built, what do you think the like the risk is for the business user to be exposed who potentially doesn’t have that same content? Like, do you know what I mean? Like you haven’t built the enough of the highways yet. 00:38:32 | Jacob Matson: Yeah, such a good question. So when we first were, so we launched our MCP server in December at MoetherDuck. And we had our own existing set of dashboards for the sales team. And immediately what started happening is the sales team started like building their own kind of dashboards using our tool, which is really interesting to see, right? We’re, I mean, we’re a small company where our risk threshold is obviously higher. But what everyone was doing was because they already had really good context for the data, because that, you know, they’re sales team. They’re, they, if the data is wrong, you know, everyone’s mad. It doesn’t work. But like it’s, it’s tested against the sales data in particular is tested against reality all the time, right? Because you’re rolling on a call with a customer and you’re like, hey, I saw you used, you know, 10 hours of compute last week, you know, we sell compute. And they’re like, no, we didn’t. You’re going to be like, okay, let me go. Oops, let me go fix my dashboard. So they have the stakes are pretty high for them, right? And for financial teams too, they’re really high, right? They need to be right. They need to be speaking about the right numbers. And so for teams that have developed an intuition for what the data should be, they’re actually pretty good at using LLMs, regardless of language and understanding like being a data person first, because they have a way to test it against reality really quickly, right? They can break down a set of numbers by, you know, like, if you’re a CFO or something, you could probably break down revenue by product and you would with an LLM and be like, that’s right. I can tell that it’s right because I’ve seen this before, right? You already have like the fingertip feel for the data, but if you’re someone else, especially like, this is the hard part for like a data analyst who’s maybe disconnected from the business is they don’t have the fingertip feel. They just have like a question from someone that says, hey, build me this thing. It’s really, you know, in that case, you really have to, you know, rely on the context of others to help build the right thing. You know, I think like that’s, so yes, I think the stakeholder does matter. You know, I think that that certain teams are more well suited to be data driven than others just like out of the box, but like that we can definitely close the gap with like good context, maybe not for everyone, right? If someone asks a totally off the wall question using the wrong words, like 00:40:28 | Moe Kiss: that’s a, you know, we don’t have a crystal ball. 00:40:31 | Jacob Matson: We have, we have, you know, vectors that we’re matching at the end of the day, right? 00:40:36 | Julie Hoyer: With this like approach that you’re talking about, like using AI to kind of bring up the context for somebody who doesn’t have it themselves. I have two questions. One, and I think, well, you were kind of asking this. So it was like, how do you know when it finds context that it’s choosing the right context? Because again, we’ve talked about people are defining and talking about the same metric differently, especially if it’s not one of the highway metrics. The other part is it feels like classically the legwork, right, was on the individual trying to ask the question, make the query. They had to go around and obviously ask and get all that context. But with the new, if we put in the solution that you’re talking about, where does the legwork now live in that process? Do you know what I’m saying? We’ve shifted the hard work of determining what is right for the question you’re asking of where to grab it, how to think about it, how it’s defined, and how to define the metric you’re trying to query. So I’m trying to understand those two pieces. 00:41:44 | Jacob Matson: The first thing that I always think about is like, well, how do we make it visual? This was a video podcast. I could show you a really sweet demo, but maybe I’ll just have to send it to you. I’ll send you a video async that shows you one way that it could look like. So I think the first thing is how do we make it so that interacting with the context is not just typing into a box and then stuff happens and you get an answer. We need to make it like you need to be able to understand what that looks like. And I think for a lot of this, because I spent so much time in the Excel salt mines, I think about it a lot like I think about Excel, which is you have some tab somewhere that says, here’s this metric, right? Here’s our profit for last quarter. And there’s a little button in there that shows trace precedence. Okay, where does that take me? Okay, that takes me upstream. And then I can keep navigating all the way back until I kind of understand how this thing comes from. And then there’s another little button on there that says calculate formula, right? And it just shows me all the little parts. And it shows how they’re adding and subtracting and dividing and multiplying and how it ends up with the final number. I think like we don’t have those traceability pieces yet for agent work flows or semantic layer stuff, really. I mean, we do, but only for the developer, not for the user. And so what I’m really thinking about is, you know, the way for me to like manifest this notion of, hey, you don’t need the semantic layer. It means you need to have something else. And part of it is, yes, you need the context. But you also need a way to visualize it in a way that like people can reason about and understand and like, you know, agree with how it was calculated, right? 00:43:20 | Moe Kiss: But you’re saying visualize the lineage, though, and the way it’s calculated. 00:43:25 | Jacob Matson: I think all parts of that, right? Lineage, yes. But like less about like, I mean, I would love to be like, okay, this number comes all the way from this field in your CRM. You know, I would love to be able to do that, right? Okay. Or like, you know, some sort of way to say, you know, hey, like we were talking about one of the active users earlier, like this, here’s all the detailed ways that that’s calculated. I think that’s probably actually too hard to reason about. It’s the wrong, it’s too detailed, right? We need to kind of be able to get the proper level of zoom out, right? We don’t want to be at the 10 foot level. We don’t want to be at the 40,000 foot level. We want to be at like the 10,000 foot level. And I think it probably depends on persona too. But like you need some sort of way to be able to reason about, you know, those numbers and how they were calculated to be confident that they’re correct, right? And like SQL is of course one way to reason about it, right? But like SQL is like verbose and kind of hard to reason about as like a non-technical person and like even, you know, engineers hate using it. And there’s a whole good set of reasons for that. You know, I think like the magic, the magic of Excel continues to be like demystifying how complex some of this stuff is and doing that with really clever UI interactivity. And so, you know, how do we, you know, what tools, what abstractions do we need to build to make that work? And I’ve been working on that. I think the core thing that I’ve been starting with is just like, how do we take something like a set of tables and like interweave context in there to show us how the tables relate to each other, right? In some ways you’d be like, hey, that looks a lot like ERD, right? It’s like, here’s the primary keys, here’s the foreign keys, whatever, right? But we can add a richer level of that that says, okay, we calculate these metrics this way, we use these joins, you know, we use this formula. That stuff can all, that stuff is also, you know, a level of the graph that’s always been missing when we get the ERD, right? We don’t know how it’s actually used. We just have a thing that says, here’s how the database defines the table. What we don’t have is knowledge of how the application actually uses it. And so, that’s I think the next part too, you know, we can mine like query history. In fact, I’ve done a bunch of work around that, like how do people actually query these, how do databases or how do applications query these, how do those patterns differ when it’s a human versus an application. So there’s a whole bunch of stuff we can do there. I’m not trying to be like too abstract or obtuse here, but like, I truly believe that that part of it is just that we have all of the, we have all of like the Lego blocks, but like no one has assembled it into something that is like cohesive in a way that’s like, I totally get this now, right? It’s very much like, even if you’re like, you know, high-ranking executive and you like get some number, there’s no like, you’re trusting that your team built it right. And like there’s not a good way today. And like lineage is part of that, but like just being able to say like, all right, how do all these components fit together and like fitting that in your brain as like, I don’t know, maybe if you’re the CMO, you shouldn’t do that, but like, I don’t know, maybe you should do that sometimes. I don’t know, hopefully that’s helpful. Hopefully that’s helpful. That’s kind of just how I’m thinking about it at the moment, but like make it visual is number one. 00:46:20 | Julie Hoyer: I was going to ask, how do you fight the echo chamber effect too from some of this? Sorry, when you were talking about like using AI to search for queries, like if it’s just always bringing back historical data and you were to like move that into the future. How do you do that? 00:46:37 | Jacob Matson: Okay. So let me just make sure we from this question. So you’re basically, you’re basically saying like, how do you not overfit the history? 00:46:43 | Julie Hoyer: Because they were figuring it out. Some of it’s not right. Some of it is. Or, you know, it’s just the bias of like, what was right at that time and you know, how do you get things in? This was my exact question. You have people that aren’t going for the context, the old fashioned way they’re relying on the tool to give them the context, but it’s old context, you know. 00:47:01 | Jacob Matson: So I think the first thing that I’m thinking about is like, well, if you had a semantic layer, you only have the history. You don’t have anything going forward because someone curated and built that for you. So now we get a new problem to solve, right? Now, if we can just forget the semantic layer exists or maybe we have it and it’s, you know, it’s the highway, right? But we want to detect when there are changes, right? When there’s drift, when there’s drift in our metrics, right? How do we, how do we detect that? I think this is a really good question. I don’t know if there’s like good programmatic ways that I’m like aware of, like I’ll talk my head at the moment, but certainly like there’s probably ways to do it programmatically. And I think like also some of it is like, well, if there’s less humans, you know, doing, doing some of the, some of the work that’s really easy to automate with AI, well, now what do those humans do next? I think part of it is like, yeah, maybe we need like a curator person who’s like handling this context potentially, right? You know, I think I kind of like always jokingly talk about like, hey, we need like more librarians. Like we’re generating all this context all the time. And it’s like, what do we do with it? It’s like, I don’t know. It just like lives in Slack or like teams or our emails or, you know, DMs on WhatsApp or whatever. And like eventually if it’s like, eventually it gets, it makes its way, you know, into the, the canon of the, of the organization, right? But like that, just that time is like really, it can be really long. How do we make that tighter is a really interesting question. I do think it’s like in the other side of that is like, well, what about archival, right? What if a metric is now was right and is now wrong? How do you discard it and like manage the life cycle of it? So I think like all of those pieces in my mind like kind of fit together, which is like life cycle management of this. And it’s like, we never even got there because we just get to like, we get to like semantic layer is published. And then it’s like, it’s such a big lift. It’s like, even get there that it’s just like, okay, like I got promoted. Thank you. I’m going to go to another company and I’ll do this again. This is now your problem. Like, sorry. And I’m being, again, I’m being sort of facetious, but like, these are hard. Like everyone wants to build it. No one wants to maintain it. And I think like how do we, I think we, you know, there’s probably space for companies or multiple companies in the space to build products that help us do this better. But, you know, I don’t have anything on top of my head that’s like, well, here’s how you catch drift and like reimplement it. You know, I wish I had the answer. You know, it would be, it would be more compelling maybe next time, next time. 00:49:19 | Michael Helbling: Right. 00:49:20 | Moe Kiss: Let me do more. 00:49:21 | Michael Helbling: That can do my next set of research. 00:49:23 | Jacob Matson: We’ll see if we can do it. You know, we are, we are fully migrating our internal stuff. 00:49:28 | Michael Helbling: So if we wanted certainty, Jacob, we would have just interviewed Claude, right? So this is great. Yeah, exactly. Exactly. This is awesome. Thank you so much. Really excellent conversation. One of the things we do at the end of every show is go around the horn and share a last call, something that might be of interest to our users. Jacob, you’re our guest. 00:49:46 | Jacob Matson: Do you have a last call you’d like to share? I just read a really awesome paper that I will share a link to you to Michael called skill opt executive strategy for self-evolving agent skills. I saw it on Twitter this morning. It is really interesting. I’ll share it and you can put it in the show notes. Definitely worth the read just to understand, you know, what it looks like to actually apply some of this stuff about what exactly we were talking about. How do we evolve these things as the systems change? That’s awesome. I love it. 00:50:15 | Michael Helbling: Awesome. Julie, what about you? 00:50:17 | Julie Hoyer: Okay. My last call is a little off the wall, but it was just too timely. Okay. So, Moe, this is a little bit, a fun fact about me and a question to you because it has to do with Canva. 00:50:28 | Moe Kiss: Okay. 00:50:29 | Julie Hoyer: Fun fact. I don’t like squirrels. Like, I just, I don’t like them. I don’t think they’re cute. They creep me out. Okay. Yeah. They kind of scare me. They kind of scare me because quick backstory. My dad said, you know, to this little, little girl, if you see possums are at Coons in the daylight, they’re rabid. I thought that meant squirrels. I went around for years thinking squirrels were rabid. You know how they pop around trees when you’re riding your bike. They like hide on the other side. I always thought they were going to pop out and get me. I don’t like squirrels. Okay. Fun fact that I don’t share with lots of people. I get an email from Canva that is squirrel themed and the CTA at the bottom of the email said, if you would like to stop getting squirrel themed content like opt out here. Like has AI personalized too far? Or like, is this, is this a phenomenon out in like the trendy world that I just have not been exposed to like our squirrels a thing or do they know this about me? So it kind of freaked me out. And there’s my fun fact. 00:51:30 | Moe Kiss: Also, like is squirrel a cool hip thing that I also don’t know about because I mean that level of personalization. 00:51:38 | Michael Helbling: Well, if the other cohort was getting emails about moose, then you’re something, now you’re getting somewhere, but that’s probably a reference that nobody knew. Did anybody else get the squirrel email? 00:51:48 | Moe Kiss: Julie, I will follow up and we can, I can share an update with you in our next class called really people hang in. 00:51:54 | Michael Helbling: Well, there you go. That’s a good last call. That just shows the range we can have here. Amazing. All right. Moe, what about you? What’s your last call? 00:52:04 | Moe Kiss: Okay. I have been reading a book which Rachel Gerson recommended. It’s called Word Slut, a feminist guide to taking back the English language by Amanda Moentell. And I just like, you know, words are important. Like you do. But when you go through this book and you hear about the evolution of certain words and how they refer to women, it just like, it’s, I don’t want to say delightful. It’s been surprising in a really great way. Like I feel like it’s kind of added to me wanting to be more thoughtful about the word choices that I use, which is entertaining because I don’t give too much thought to what comes out of my mouth. So anyway, go check it out. And Michael, over to you. 00:52:49 | Michael Helbling: Yeah. So mine is from back in May. James Hawkins, the CEO of post hog wrote an article about what he was thinking about as sort of their next chapter of vision of the future. Just so your book, if listeners aren’t aware, post hog is sort of both an analytics tool as well as other tools for digital and website operations. Anyway, it, I don’t know that I agree with everything he wrote in terms of like where products should go, but I thought it was very thought provoking. It definitely worth some time because all of us in analytics and in measurement spaces were facing a lot of change. And it’s good to see like, okay, here’s a company who’s got a lot of customers who’s doing a lot of work, especially with AI. Here’s how they’re looking at the future. So I think it’s kind of a good read just to develop and gain perspective. So that’s why I would recommend that. All right. Once again, Jacob, thank you. Thank you so much for coming on the show. This has actually been a really cool conversation and kind of like we, we, we stayed pretty high, but I feel like there’s also some really amazing kernels for people to pick up on and drill into their orgs with that I think will actually bear a lot of fruit. And also because it relates to songs like so now in my head, I’m like, life is a highway. Okay. Wow. Yeah. See, that’s what, that’s what happens to me when we talk about stuff. Anyways, so thank you again. And obviously to our listeners, yeah, I’m sure you’ve got thoughts and questions and we’d love to hear from you. And there’s a great way for you to do that. You can reach out to us on LinkedIn or on the measure slack chat group or via email contact at analytics hour.io and wherever you listen, you can also leave ratings and reviews and we read all of those as well. And Tim Wilson who’s not here today would like you to know you can also request a sticker for your laptop. Just go to analytics hour.io and there’s a form that you can fill out and we will mail 00:54:52 | Moe Kiss: it to you even internationally. 00:54:55 | Michael Helbling: All right. 00:54:56 | Moe Kiss: This has been great. 00:54:58 | Michael Helbling: I think there’s about three more of these that probably need to be done. AI is changing so fast. But Jacob, thank you once again. And I know I speak for both my co-hosts, Moe and Julie when I say no matter where you are in the jungle, keep analyzing. 00:55:16 | Announcer: Thanks for listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at analytics hour, on the web at analytics hour.io, our LinkedIn group and the measure chat slack group. Music for the podcast by Josh Grohurst. Those smart guys wanted to fit in so they made up a term called analytics. Analytics don’t work. 00:55:40 | Charles Barkley: Do the analytics say go for it no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 00:55:53 | Jacob Matson: I think actually like, if you moved your cable around, I wonder if the cable was just like giving you some weird feedback or something. Because I was don’t touch it. 00:56:02 | Julie Hoyer: Don’t touch it now. 00:56:04 | Michael Helbling: Don’t touch it. Hands up. 00:56:06 | Julie Hoyer: Tim sent me a mic that he knew made a buzz. That was like an evil trick. Sabotage. Yeah, my toddler now uses it as her microphone. 00:56:18 | Moe Kiss: I was like, sure, you can have this. 00:56:21 | Michael Helbling: Does she do a podcast because you do one? Because that would be the most adorable thing in the entire world. No, but she does like to sing. 00:56:28 | Julie Hoyer: She just breathes into the mic. 00:56:33 | Michael Helbling: So that’s what I do. 00:56:35 | Moe Kiss: I’m going to be triggered a lot with every time the word, the S word is mentioned. You know, usual. 00:56:43 | Michael Helbling: I do the exact same thing, but I look over because a lot of times I leave crap on my chair back here. And so then I look, I’m like, oh, clear. I don’t. 00:56:51 | Moe Kiss: My husband gets dressed in here in the morning and decides to leave whatever pants he didn’t want to wear in the background. 00:56:59 | Julie Hoyer: Trust me. You don’t want to see what you can’t see behind this cover. 00:57:03 | Michael Helbling: All of the mess is my responsibility. 00:57:05 | Moe Kiss: I do get asked very frequently, though, why I have so many remotes on the wall behind me. And I’m like, it’s a fair question. Well, there’s another one that’s missing. So it’s totally fine. 00:57:15 | Michael Helbling: I don’t know. You’re just, you’re just a very successful person. 00:57:22 | Moe Kiss: And if people can’t deal with that, I’m just, I’m afraid to do it. 00:57:27 | Michael Helbling: I’d be afraid of that level of success is what my answer is. Five remotes. I know. I don’t have the lifestyle governance required. All right. Let’s get into it. So I’ll give us a five count and all right. All right. Let’s stop moving our microphone. 00:57:43 | Moe Kiss: It’s the problem with the YOLO one. It like pains everybody. 00:57:45 | Michael Helbling: That’s right. 00:57:46 | Moe Kiss: That’s right. We’re very serious. 00:58:03 | Julie Hoyer: Rock flag. 00:58:04 | Moe Kiss: And if you build it, who will maintain it? The post #300: Are Semantic Layers Really Necessary? appeared first on The Analytics Power Hour: Data and Analytics Podcast .

June 9, 20261 hr 0 min

#299: AI Can (Help) Build the Dashboard. It Can’t Build the Buy-In.

There are roughly a thousand ways to roll out a new analytics platform, a BI tool migration, or an AI initiative to your organization. Most of them involve a town hall, an email with a link to some training materials, and the quiet hope that everyone figures it out. Most of them also don’t really work. On this episode, Yehonatan Schwarzmer joined Michael, Val, and Tim to bring some long-overdue organizational change management thinking into the analytics conversation. Yehonatan has the unusual combination of real-world experience in both change management consulting and data leadership, which makes him exactly the right person to explain why the technical rollout is the easy part. The harder part is understanding that when someone says “this tool doesn’t have what I need,” they might really be saying “I was the hero in the old system and I don’t know who I’ll be in the new one.” The Kübler-Ross grief model shows up. Psychological safety shows up (reluctantly). And Val’s question about who analysts should recruit to help them manage change at scale almost gets answered. This episode is brought to you by Prism from Ask-Y —your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. This episode is also brought to you by Stape , your all-in-one solution for server-side tagging. Links to Resources Mentioned in the Show Prosci Kübler-Ross Change Curve Five Stages of Grief #184: Psychological Safety and Analytics with J.D. Long McKinsey 7S Framework Kotter’s Change Management Theory The Digital Delusion by Jared Horvath How AI is Redefining the Role of Analytics Leadership by Eric Sandosham Let My People Code by Noah Omri Levin Marketing Analytics Summit Can You Teach an AI Ethics? by Gary Angel Photo by Brad Starkey on Unsplash Episode Transcript 00:00:00 | Announcer: Welcome to the Analytics Power Hour. 00:00:08 | Announcer: Analytics topics covered conversationally and sometimes with explicit language. 00:00:12 | Michael Helbling: Hi, everybody. 00:00:15 | Michael Helbling: Welcome. This is the Analytics Power Hour, and this is Episode 299. You know, in the movie Godfather, Michael Corleone takes over the family in a time of crisis. And he quickly moves Tom, his adopted brother, out of his role as consigliere. His reason for this was because Tom wasn’t a wartime consigliere. In times of great change, I guess it’s important to recognize the need for different styles of leadership and approaches to our work. And, oh, I mean, AI is kind of putting every company on a, quote, wartime footing, I guess, in that survival, our ability to predict and normal cycles of growth, even our own understanding of the levers of business are kind of being transformed. And how do you manage it all? All of this filters down to analytics because AI is making some things much easier, but potentially creating some downstream challenges that could be very, very damaging. So it’s time to put on your wartime consigliere hat and embrace the change. I mean, it’s already underway. So let me introduce you to my co-hosts, Val, “Ground Truth” Kroll. How are you doing? 00:01:25 | Val Kroll: I’m good. Now that I’ve Googled consigliere and those roles from the Godfather. Yeah. 00:01:31 | Michael Helbling: Just keep doing it the Chicago way. That’s right. Got it. 00:01:35 | Val Kroll: Got it. And, of course, Tim, the last word, Wilson, how are you doing? 00:01:42 | Tim Wilson: Doing quite fine. 00:01:43 | Michael Helbling: And I’m, my goal, the diplomat, Helbley. All right. We needed a guest. Somebody could walk us through what’s happening, how to react to it, and I think we found a great one. Yehonatan Schwarzmer is the director of business intelligence at Networx. He’s also held analytics leadership positions at Search Discovery now further. He has done organizational development consulting and he has a master’s degree in industrial and organizational psychology from Columbia University. And today he is our guest. Welcome to the show, Yehonatan. Thank you. 00:02:15 | Val Kroll: I’m very happy to be here. 00:02:17 | Michael Helbling: Great to have you. 00:02:19 | Val Kroll: Okay. 00:02:20 | Michael Helbling: So I think there’s a lot to get into here, but I think, you know, in your world, you’ve kind of lived across two different distinct worlds in your career. One is sort of this idea of change management as its own practice and then analytics as its own sort of discipline within the business. So like, as you see those two things sort of come together in this moment, like what are the things you’re noticing in the community that you’re in? 00:02:46 | Yehonatan Schwarzmer: The idea of change management is one that we all focus on as a topic by itself. You know, we have an issue or we see that there’s a company or organization struggling with shifting or a change that they know is coming and the feelings that come along with that. And so there’s a lot to talk about with change management. Some people think about it like organizational therapy or some other label that we stick on it, but separate from the question of what is change management, it also has to do with what it is that you’re managing a change of. And specifically when you talk about the management of change in the world of analytics, it really focuses on a lot of the big questions that come up that you typically explore in the world of data and analytics, just with an eye towards what is happening on the people side of these changes? How are the people being affected? How are they going to approach it? How do we think about that so that the change is seamless and gets us to where we want to be and we’ve gotten everything we want out of it? You know, when I shifted, kind of talk about two worlds, so when I shifted into the world of analytics from the world of pure change management, right? Pure change management. So we were doing consulting to companies and talking about, you know, leadership development. There was like conflict resolution, especially when you have those like family companies and you know, lots of the issues that come up with those types of things. And then you shifted to the world of analytics and it starts with what’s the number, talk about the numbers, just the numbers, but then really, really quickly, you get to the understanding that it’s not just about the numbers. 00:04:22 | Val Kroll: And if you only focus on the numbers, you’re going to miss a big part of the value that 00:04:26 | Yehonatan Schwarzmer: can be delivered. You really have to understand the people involved in it. And that’s where the cross between the two topics became very exciting. 00:04:35 | Tim Wilson: It does seem like you said something in that that the one change, figuring out how to get where you’re trying to go to. And I guess, is it fair to say in the world of data and analytics, it seems like often there is where we want to go to is being data driven or being it’s kind of like this squishy aspirational, we want to use the data to make decisions. But that’s not really like the future state hasn’t been fully formed. It’s kind of more of this aspirational idea. Does that not bring challenge? Because if you haven’t really defined the destination, which I guess is also a temporal thing, because the destination is always going to keep moving. Is that a particular challenge if you’re doing change management and you’re not acknowledging how the future should look different from today? Yes, of course. 00:05:34 | Val Kroll: There’s a couple of the typical or the classic issues that you’re going to bump 00:05:41 | Yehonatan Schwarzmer: into either when you are doing change management or that come up that make you realize we should really be managing this change. So one of them for sure is we don’t really have a vision for what this change is supposed to be. Of course, we don’t have people on the bus. Nobody knows where the bus is going or we have 10 different buses and they’re going in 10 different directions and people have to decide which ones they want to get on. So for sure, defining it is one of the critical components. There’s a number of critical components. You talk about things like defining the vision. You talk about things like making sure that you have buy-in that you’ve dealt 00:06:12 | Val Kroll: with resistance, that you have a organizational sponsorship. 00:06:16 | Yehonatan Schwarzmer: Let’s say a primary sponsor, someone who’s really going to make sure that this thing is managed the way that it needs to be. So there’s a bunch of them. But for sure, having a vision or where it is that we’re supposed to be is one of the fundamentals. 00:06:28 | Val Kroll: I love the way you put some of that introduction. It has me thinking like whether I should adjust my definition or how I think about it a little bit. That is something that happens separate. So I guess there’s the concept of like changes happening, whether you’re managing it or not. And so you have to make the choice for it to be an active part of whatever is in scope. But do you think of it as like if it were one of several work streams, if you will, on a project, is that missing the boat because it’s not like interwoven with the other activities on the project? Or is that healthy because then you can kind of give it its own time and attention and like the capital letters of the name of the work stream? Or like how do you kind of consider that in the scope of or in the context, I should say, of like a larger initiative or project? 00:07:17 | Yehonatan Schwarzmer: I think that as with most of these types of questions, the answer is going to be it depends, which is only because there are so many pieces that come into it. You know, we talk about a change. So you think about a small startup that’s quickly shifting, you know, every month, it seems to focus on something new and different. You talk about a very large bureaucracy where someone has come in and says that the way that we’ve been doing things for the past 20 years just isn’t working. You talk about a company that’s turned around and said, oh, my goodness, we’re falling behind in the AI bandwagon and we better get everybody on board as quickly as we can. You know, these the changes that we’re talking about, they there’s not one type. And because there’s not one type, there really isn’t one answer. People who love change management sometimes tend to think of it as a hammer and everything is a nail. And it’s a terrible hammer because there there are so many things that happen separate from the people side and the people’s dealing of the change. You know, we say change management is so important because if you don’t deal with the people, then you’re going to really struggle. That’s true. But if you’re just dealing with the people and let’s say you’ve forgotten about the strategy, for example, or the vision, then you’re not really much better off. So when I say it depends, I think of really a couple of different axes that you’d you’d want to think about that. It depends factor on, so, for example, what type of change is it an incremental change, something small changing, probably doesn’t need a lot of management. Is it a transition or shifting something underlying or much more critical to our regular operations? Or on the other end of the spectrum, is this a huge fundamental shift? And we have to redefine our identity, let’s say we have to become something that we weren’t before. So that’s one way to think about it. What’s the motivation? Where does it come from? You know, you have these, especially now, you know, you get some less startup that pops up and they’re very excited and, you know, they’re running in one direction and they know that they’re working in an uncertain world and then things change around and they’re like, this is amazing, a new challenge. And they turn around and they go in that direction. And then they, you know, and it’s a group of, you know, six or seven or eight people and they’re charging in this direction, they’re charging in their interaction and we can talk about whether that’s good or not good. But the management of the change and the willingness of the people to go along is 100% as opposed to the next level up, which is a change imposed, but it’s imposed internally. So let’s say you have a new leadership and they come in and they say, from now on, we’re going to do it this way. I’ve looked at how we’re doing it. It’s not going to work. We got to do something new or I found a new capability. And so there are some people pushing for it inside and some people have to kind of get along with that. And then the third kind is you’ve got an externally driven change that the people on the inside never really asked for, but they don’t have a choice. So all three of those categories are going to react to the change in a very different way. And so the, the management of it, how necessary that is, really does defend and depend, and then there’s the, the organization itself, you know, you’ve got some organizations that are oriented towards learning. They love learning. They love conversations. They love that free flow of what’s working. What’s not working. Let’s be honest with each other. Let’s move the ideas forward, et cetera. You have others that are a little bit more calcified and not as willing to move things along. Um, and then you have some that are just, let’s take it as it goes. You know, we, we, we don’t necessarily point to those as the, uh, the way things are supposed to be, but there are some places where management is, is kind of a committee and decisions are made by committee. And we all know what that looks like. So there are lots of ways that a change could be imposed and lots of ways to respond to it. And so depending on that determines the level at which you have to manage change as a separate thing from your project or as integrated within the project itself. I’m going to stop there and see if that makes sense. I could, I could talk about that for far too long. 00:11:23 | Val Kroll: So I’m going to stop there. Well, we have a whole episode to dive in. 00:11:28 | Tim Wilson: Michael, exciting news. Claude, co-work is here for prism. 00:11:33 | Michael Helbling: Finally, because I’ve got some GA for BigQuery exports and frankly, they look like somebody shredded a website and taught the various pieces 00:11:40 | Tim Wilson: accounting. That’s why ask dash y.ai built the prism, co-work connector, which is quite a literative. Your GA for BigQuery now has a smart analyst that actually speaks GA for. That includes having funnel and cohort analysis baked in ready to run right out of the box. 00:11:59 | Michael Helbling: Oh, that’s huge because I mean, half the time GA for speaks in riddles, event parameters and frankly, emotional damage. 00:12:07 | Tim Wilson: I don’t think that’s what it has, but co-work ships with sessionization built on GA for his own methodology as a baked in skill. 00:12:16 | Michael Helbling: Okay, I do love that because my session start final actual real query can finally be sent to the farm upstate. 00:12:24 | Tim Wilson: And every analysis becomes a shareable page. Prism auto generates a dashboard page directly from within co-work. 00:12:32 | Michael Helbling: Oh, no copy paste, no screenshot archeology, no, please ignore the column 00:12:36 | Val Kroll: named temp fix to nope. 00:12:39 | Tim Wilson: And co-work brings the whole prism brain, the analytics agent, harness skills, memory engine, everything. 00:12:46 | Michael Helbling: I like an organized analysis and it sounds like it provides that traceable, audible and ready to use with your other data sets. 00:12:54 | Tim Wilson: Exactly. You might have to do some adjusting to your personality and standard way of working. Well, I do feel threatened, but you know, in a growth mindset way. All right. We’ll go to ask-y.ai. That’s ask-the-letter y.ai and sign up for the waitlist. Use code APH to jump to the top. 00:13:13 | Michael Helbling: Yeah, that’s ask-y.ai code APH. Hey, Tim. How confident are you that your website tracking setup is actually giving you usable data? 00:13:30 | Tim Wilson: Well, confident is a strong word with browser changes, cookie limits, tag conflicts and messy marketing stacks. Measure pet can get complicated pretty fast. 00:13:42 | Michael Helbling: I agree. That’s why we’ve partnered with STAPE. STAPE is an all-in-one solution for server-side tagging, helping teams improve quality and the reliability of their data collection. 00:13:53 | Tim Wilson: And if your tracking is incomplete, duplicated or misconfigured, your reports may be pointing you in the wrong direction. 00:14:01 | Michael Helbling: Yeah, but you can use STAPE to clean up your server-side data collection, improve your conversion attribution, and that’ll help you make better use of your marketing budget. 00:14:12 | Tim Wilson: They also have a free website tracking checker. Enter your domain, scan your site, and in under two minutes, you get a personalized report. 00:14:20 | Michael Helbling: Yeah, I like how it has prioritized recommendations, some competitor comparisons, and AI-powered suggestions. 00:14:26 | Tim Wilson: So before the next is-the-data-wrong meeting, go get your free report. Visit STAPE.IO, that’s STAPE.IO, and use the free website tracking checker to get your personalized report. 00:14:39 | Michael Helbling: All right, and now back to the show. 00:14:44 | Tim Wilson: When you talk about, like, the organization, whether the organization is kind of geared towards learning or geared towards being static, like, do you run into the people within an organization? Like, people who are saying we should always be, we should keep adjusting and changing the way that we’re doing things versus the extreme, you know, who move my cheese, people. 00:15:09 | Val Kroll: Like, what defines, like, the organization’s nature or culture or 00:15:16 | Tim Wilson: whatever the right label is for that to say, how much is change accepted as part of something that’s always going on versus change is something that we want to minimize? How does that work? 00:15:28 | Yehonatan Schwarzmer: You were going to walk into an organization, you would have no idea. So there’s kind of the abstract part of your question, which is what is the reality of an organization? And then there’s the question of, how do we figure out what the reality of the company is so we can actually work with them? Um, I’m more comfortable with the second one because that’s the way that I typically think about it, is if we’re going to work with a group of people, let’s try to figure out where they are. 00:15:53 | Tim Wilson: So what, so I guess what are the, what are the tells there? Cause I think, I mean, a lot of people are in an organization. So I’ve never really thought about my organization through that lens. What kind of organization am I in? 00:16:03 | Yehonatan Schwarzmer: One of the topics I think that’s going to come up whenever you talk about this, either you talk about it as kind of the basic definition of organizational change, or you think about it as one of the fundamentals that you have to address is resistance. So we talk about readiness, but we also talk about resistance. And they’re not necessarily two sides of the same coin, right? Some people could be not ready because they just aren’t skilled or they don’t have the experience while others are actively resistant to a change. And so there’s, when there’s an organization called prosci, which has 00:16:38 | Val Kroll: a lot of resources related to organization change and managing the change. 00:16:44 | Yehonatan Schwarzmer: And they have a lot of models. And one of the models that’s really simple, but I found really useful is look at two spectrums to get a sense back to your question of how much management of the change do we need? One of the spectrums that we look at is the degree or the complexity of the change. Is it a significant change? Is it a small thing? And then the second one is what is the anticipated resistance? Meaning how much work will we have to do to push this change through? And that really opens up the whole question to what is the barrier to change management and how do you address that barrier? And the way that I think about it is that within change management, this is a discrete question, meaning it’s a, it’s a topic within change management, but you could easily see how this can kind of blend into every question that has to do with managing a change, which is at the end of the day. All right, what about those people that aren’t going to come along? So I’d rather think about it in the way that it is a topic within change management, because there’s a lot else that we could talk about separate from the resistance, but let’s, let’s focus on that part as a topic by itself, which is what are your barriers and how do you address them? 00:18:00 | Val Kroll: So there could be a lot of barriers, meaning when I said before that 00:18:07 | Yehonatan Schwarzmer: resistance is not necessarily the opposite to being ready. When a lot of times you see a big change being pushed at a company, you get 00:18:17 | Val Kroll: the mindset of, well, I was really clear on what we’re trying to do. 00:18:23 | Yehonatan Schwarzmer: And I had a town hall and I spoke to people about it and I have training available and I probably have office hours ready. And there’s a lot of crickets, meaning either people just aren’t interested or people are still kind of doing their own thing. Or it seems to me that they’re just waiting for this initiative to pass by just like all the other ones so they can get back to their real jobs, you know, whatever it might be. And there’s a lot of confusion. I don’t understand. Like we invested a lot. You know, we bought this new platform or we’ve got this new training or the company has to make some sort of a shift or we’ve got a new strategy or whatever it is. I expected maybe not a rallying cry, but at least people to be aware and show that they’re moving along with us. And I don’t feel that either. They’ve got their heels dug in or they’ve just kind of put their head down and they’re they’re waiting for it to pass. So what why is that? What’s what’s happening here? A lot of times when you start the conversation, it can be useful to talk about this this curve of change management. It’s it’s a model, actually. They’re trying because of you. The the Kubler-Ross model of change. Kubler-Ross model, right, which which is great. Like it’s a great, simple way to talk about here’s something that we can anticipate so that we all have our expectations pointed the same direction. Meaning if we are going to introduce a change and the change is going to be somewhat disruptive, again, we’re not changing, you know, the the the water cooler, we’re changing something that’s going to make a difference to people. And so we can expect that that’s going to affect people and the effect is likely going to mean that there is going to be number one, a decrease in productivity for a time as we shift over and we acclimate ourselves to the change and there are going to be people who need to become comfortable with the 00:20:07 | Val Kroll: change and potentially during that process of discomfort, you will see 00:20:13 | Yehonatan Schwarzmer: productivity decline or you will see ancillary things. People might look a little bit more disgruntled. They’re more whispering. You may see whatever it is. 00:20:22 | Val Kroll: So how do you what do you do about that? 00:20:25 | Yehonatan Schwarzmer: Why why is that? What do you expect? So when we think about it and the nice thing about the the model is it’s really a simple visual where you’ve got pretty much a line coming across on your time axis and then it dips down and then it slowly curves back up to where it was and then it continues up to be higher than where it was when you started and that’s the idea of what we’re trying to do here. The reason that we’re making a change is because we believe it will improve our performance. So if you look at the y-axis as performance and you say I want my performance to be higher, I’m going to go through this period of change, get my performance higher, even though that means that the cost will be for a period of time, there’s going to be a drop. There’s going to be a dip in performance and we’re going to have to work our way through that. And one way that we can think of change management is the benefit of change management is it minimizes the length and degree of the dip. So either you will be in a period of decreased productivity for a shorter period of time or it will have less of an effect. It won’t fall down as much. You know, in an extreme case, you get some people who just say, I don’t know what to do, I give up and they sort of clock out and you really get nothing for quite some time until they finally realize what’s going on or the fact that their jobs depend on it or whatever it is. And it’s time to get on board with the program. Typically, it’s not that type of thing, but the question of what will it take for someone to understand the value of this change and why we all have to get on board and what we need to do about it is where you get to getting back to the line that you were before and then getting higher. The other reason it’s useful to think about it in that model is because the person who created that model, that model was actually the second model. The original model was actually the famous model that we all know, which is the five stages of grief, which is fascinating because they came up with this idea of the stages of grief. And then after that, it evolved into a model of change management. And there’s an implication there. What does it mean that we learn from grief to what change management is? But it’s very helpful to think of it in those terms, not because change is devastating all the time, but because you have to understand that change has a real impact on people. And when you think about grief and the stages that people go through, right? So that, again, that classic model that people generally are familiar with, you’ve got your denial, anger, bargaining, depression, acceptance, right? So in change also, you’ve got your shock or denial. You’ve got your resistance, then you get maybe exploration or testing. And then finally, acceptance and then commitment. Right? 00:22:58 | Tim Wilson: Can we pause for a minute so listeners can kind of think about like the migration to do their own explore, perhaps that explains so much. There was, I mean, that was like the industry change management and the anger part. 00:23:14 | Val Kroll: Say, where are we stuck? 00:23:15 | Tim Wilson: That’s a very helpful. It could be a long process as long as it’s a process. But just to pick on and we keep coming out of like, well, there’s like a million different types of change, a million different forces, a million, I’m a little concerned. I want to kind of dive into something a little more specific. I mean, because my concern, it’s like, well, gee, it’s all over the place. So I want to push, I made the GA4 crack, but I feel like there have been platform changes. We’re moving from Domo to Power BI or moving from Power BI to Tableau or moving from Adobe Analytics to Amplitude, which there is some motivation to the change in the framing of the performance going up. 00:24:11 | Val Kroll: That’s like the change is either the licensing is getting out of hand or 00:24:17 | Tim Wilson: there’s a gap in what we can access. But I think of that dip on the Kubler-Ross curve is always somebody saying, I mean, the way you were talking through the crickets and it’s rolling out, it seems like what often is getting heard by the organization is, it’s going to be better. And then they just kind of wait until it’s there and it’s tangible. And then they look at it and they’re like, where’s that number that I used to get every Monday morning and they’re reacting to it. So in that context of this is a change is being imposed because some subset in the company made a decision, or I’m sure in Google Analytics for the 00:25:01 | Val Kroll: vendor made a decision, but the change is happening and it’s going to happen on a certain timeline. 00:25:08 | Tim Wilson: Like, what do you do when you’re thinking through what those barriers are? Like, it does feel like the organization thinks of the barriers as being the technical cutover and training and office hours are kind of, they’re like, well, this is overcoming the technical barriers and it’s going to overcome the people don’t know how to use the new tool. And it sounds like you were saying, yeah, but there’s like other barriers that those components don’t address. So like, how do you identify those barriers and then address them? 00:25:43 | Yehonatan Schwarzmer: One of the things that you said is really critical to this. The idea that I knew where my number was and now I don’t know where to find it. And it’s funny, like when I when I talk about the the concept of a change management model flowing from a grief model. And there’s like an extreme there. But in a sense, people are maybe mourning is a strong word, but they feel sad. They feel like something is missing. They feel like something important is no longer here. It might be the way that I was used to doing something was really comfortable. And now it’s gone and I have to do something uncomfortable. It may be much more fundamental than that. It may be in the old tool. I was the hero. I knew where everything was. I knew how to look good. Maybe in the new way, I won’t know how to look so good. So I’m giving up something much more fundamental, much more critical to myself. I make it about the tool. Well, this tool doesn’t have XYZ, but I’m really a little bit concerned. Number one, I may not be so good at it. Or number two, let’s say we move to a platform where everybody’s just as good. And then when everybody’s special, then no one is. I lose that there are so many things that are lost in a change that every person experiences differently. Understanding those individual changes goes a long way towards understanding what can be done about it. So it’s not just about something like, hey, here’s the training. Here’s what can be done. But it might be something like you are the expert. And the reason you’re the expert is because you’ve got all this experience. And because of that, you are primed to be the one that can take on this new tool and help everybody move along and be seen as the one that really helped us make this significant change as an example. But it’s addressing what is it that this person is losing and how do you fill something in that can give them something that build them up? 00:27:28 | Val Kroll: So two part follow up to this, because this is exactly what I hope we would be getting into, too. So one is the high level, you know, process or arc that you guys have both referenced, like whether it’s, you know, the town hall announcement followed by an email with a couple of links to FAQs on some Confluence page. A couple of dates included brown bag lunches and like, let me know if you have any questions is the most common way that I’ve seen change managed across a lot of organizations. And if you’re talking about, you know, a tool changer, something like that, you’re talking about hundreds of users. So how do you, so this is the two part, how do you address the individual and like what their resistance is or like the loss that they’re mourning at scale? And then two, how, like, who, who could an analyst recruit to help them through some of this? Because a lot of times we’re doing that program, not because we don’t care about managing the change well, but we really haven’t seen it modeled other ways necessarily. And so is there someone else in the organization that people should be tapping into, or is this another muscle that analysts really need to build themselves? 00:28:41 | Yehonatan Schwarzmer: So that’s my two for it is a challenging question. One of the reasons it’s a challenging question is because even again, going back to this organization, ProSci, who talks a lot about sort of the fundamentals, like what are the pieces that you want to make sure are in place? If you’re going to roll out a significant change, meaning you’re going to roll out a new tool and it’s going to affect a thousand people, right? Like that’s not a simple thing. It’s not something that can be solved with lunch, nor is it something could be, you know, solved by sitting down with every one of them and saying, I understand your pain. 00:29:12 | Val Kroll: What is it then that will overcome someone’s resistance? 00:29:15 | Yehonatan Schwarzmer: And within those thousand people, we have to assume we’re not going to have a thousand people resisting, but we have to know who is. 00:29:22 | Val Kroll: And so maybe starting with what is it that overcomes that resistance in the 00:29:25 | Yehonatan Schwarzmer: first place, and then how do you translate that into an actual plan for a thousand people? So when you think about overcoming the resistance, assuming that you have genuine resistance, you know, typically you’ll be able to overcome it in one of three ways. You’ll either get your, your desire, your entrained desire, you know, everybody talks about, you know, well, if it’s changed, it’s kind of a focus on the what’s in it for me, which is cliche and it’s true at the same time. 00:29:50 | Val Kroll: Right? 00:29:51 | Yehonatan Schwarzmer: Hey, with this tool, you will no longer have to deal with, right? Again, a lot of times we roll out a new tool and people immediately see, hey, this is really beneficial. That’s great. If people don’t see that, maybe you can sell them on it. Hey, let’s talk about the benefits. Let’s talk about the trade. Let’s talk about what we’re exchanging and what we’re trying to do. We’re exchanging and why we’re doing this and what benefits you get out of it and why that might be better for you. 00:30:13 | Val Kroll: There are, there is a one group that could be influenced by that because it’s 00:30:20 | Yehonatan Schwarzmer: really true for them. There is a benefit. They didn’t see it before. They may not have understood it before. And so it’s not to say that the lunches and the trainings and everything else are useless. They’re not useless. They just tend to be that we focus on the areas that they didn’t work. But for a lot of people, it does work. 00:30:34 | Val Kroll: So you’ve got, you’ve got, you’ve got two more. 00:30:37 | Tim Wilson: But I feel like even calling that out, I think there’s a tendency to one parrot. What, in a tool perspective, it just parrots what the vendor has said the benefits are, which aren’t necessarily resonant with the company. The other is the people who are implementing often get so caught up in the, I think the logistics of how is this transition going to happen that does to me sounds like a, yeah, that is a miss. Like probably repetition of these are the benefits that are meaningful for you is actually that this is the little light bulb went on that it’s, it’s easy for that. It’s like, well, they just assume everybody knows, like, sure, everybody knows why we’re doing this. We told it when we sent out the first email and it sounds like that’s something that really should be restated with an opportunity for somebody to ask follow up questions. Like you said this benefit, but what does that really mean? 00:31:34 | Yehonatan Schwarzmer: I think, right? I, I, I say yes. And actually that sort of blends into the second way, which is not such a distinctive way, but it’s the, you know, the, the FOMO approach, the, you know, the, the missing out, meaning if you do get a group of people and they are making significant progress or they do experience some real benefit, then very often that will kind of push the people on the other side to say, I didn’t realize this was, but now that I see what it actually looks like once people are doing it. And this could be because, listen, if you explain it to me at a town hall versus I see someone that’s actually doing it, totally different thing. Um, but as more people start using it, as you start building a little bit of a snowball, again, it’s funny because a lot of these things are sort of seen as cliched and we roll our eyes, but this actually is where people get influenced. When I see someone is doing something and I understand how that translates directly to me, I am much more likely to want some of that. So whether you can build that consciously or you get enough of a groundswell going, then you have a significant leg up from where you were when you were just again, having that brown bag lunch where it was very theoretical to people and most people were not thinking about how amazing this will be for them. It’s morning, what they’re losing. 00:32:48 | Tim Wilson: So, so is this one related? I’m thinking through a client that actually three of us worked on together that that is, that is picking the people who are getting on board and kind of stacking the deck for them to be not just successful themselves, but successful and elevated in public way. I mean, and it’s a little bit of getting them to be champions amongst their peers, but also it’s like, well, yeah, help them, let help them be successful with it. And they’re still going to be talking about what the job is. It’s just if whatever the change, whatever the process or technology change slips into that, then it sort of becomes the subtext. I mean, you know, oh, I thought about this stuff differently and here, look, now I’m actually saying something that’s useful and interesting and actionable. 00:33:40 | Yehonatan Schwarzmer: Yes, is the answer. And I actually, I want to address it, but I want to take one slight detour first because you’re actually, you’re also talking about the third way, which is going to transition right into this. So if the first way is, you know, what’s in it for me? And the second way is kind of the pull as people fear that they’re missing out and they see benefit and they want part of it. The third way is the push, the authority, let’s say, meaning at the end of the day, there are some people that will say, well, I don’t like it. And, you know, whatever it is. And then to have a leadership to say, this is something that is important enough for us as an organization that we will, we need to make this change for whatever reason. And so you have to make a choice. You got to get on board because this is the direction that we’re going in. Meaning there are some people who have the mindset, I’m just going to keep doing it the way that I’m doing it and it’ll probably be fine. At a certain point, that’s not going to work. And so you don’t want it to become a conflict if you can avoid it. But the reality, call it a last course, if you will, but if people aren’t going to buy into it and they’re not going to see the benefits, then they may see, all right, listen, it’s at least better for me to go along with it than the alternative, which I would rather not have. 00:34:51 | Michael Helbling: Okay, so I want to pull in AI now because a lot of organizations are getting this mandate like forcing function with very little instruction on what that actually means because no one really understands what all the parameters are. So you sort of have this existential force, right? So this externality, this forcing change, the leadership is saying, we’re going to become an AI driven organization. We got everybody report back on, I mean, in the most extreme examples, like how many tokens you’re using every day, which, wow. But like, so bringing that into the world of analytics, it’s like, oh, so what am I supposed to be doing now that there’s a dictate to make a change but no instruction for what change is exactly to make or how to make them or that kind of thing. So like that creates sort of like, I feel like a failed structure in a way of like running change management. So like, how do you sort of, how do you exist? 00:35:57 | Yehonatan Schwarzmer: This is just as fun as I thought it would be just for the record. So I want to, let me, let me do this then. I want to get to your question, but I’m going to take a detour first, which is to go on to Tim’s question, which was what the original detour was from because that’s going to get back to your question. Right. So, so Tim. Well, so, you know, you were talking about how do you get those thousand people or what does a leadership group do or how do you get beyond the lunch and learn to whatever it is. And there really are a couple of ways. And then Michael, this is going to get directly to your question, because when you talk about AI, we’re really hitting, you know, every single one of these things, right? It’s a, it’s a huge change. It’s fundamental. It potentially undermines and creates this real fear for people of what am I giving up? Am I going to be able to do it? Will other people get there before me? And what will that mean for me? I don’t even know what it looks like. We haven’t defined the vision. Nobody is even telling me what it’s supposed to look like. They’re just giving me some basic mile markers of more and no one knows what that is. How will I even know if I was successful? How will they know if I was successful, right? There’s, there’s so many areas for that concern. 00:37:09 | Val Kroll: And that’s even before we get to, you know, the complexity of AI. 00:37:14 | Yehonatan Schwarzmer: Really, this could be with any large change, but it’s only exacerbated when we talk about AI. So overall, when we talk about making a significant change and you’re trying to understand how do you, how do you connect the dots, right? I’ve got a lot of people who don’t understand. I’ve got a lot of people who are afraid and I’ve got a lot of people who are resistant. What am I supposed to do? So there’s a much longer discussion that I’m sure, you know, we can have. But in a nutshell, it really does go back to even some of the things that you were saying, which is maybe don’t hit everybody at the same time or maybe think about who it is that you want to talk to first and not think about this as one monolithic change. That could either mean something like, let’s find the early adopters, the people who are going to be excited and let’s talk to them about their role. Your, your role is not here because I want to teach you first. I want you to learn this faster than everyone else and then show everyone why you’re the star of the company of our most stinks. That’s not going to generally create the, the rainbow is feeling that you’re looking for. But if it is, listen, you, these people that I’m having a conversation with you, because I feel like you not only can be extraordinarily valuable to help our company elevate, but you can bring everyone along with you. And so the role that you’re going to play is to learn as much as you can, 00:38:31 | Val Kroll: demonstrate as much as you can, potentially be a resource for others. 00:38:35 | Yehonatan Schwarzmer: I don’t know if that means that they have to have, let’s say office hours. I’ve seen that where, you know, if you learn it well, then you can offer your services as it were to others or just let them know that your door is open. But be that person that shows everyone what the potential looks like, where the excitement comes from, why we thought this was beneficial, and then help them along. That’s one potential way you can do it. Another way you can do it is by focusing on a different group. I’ve got a thousand people. Okay, out of these thousand people, you know, I have a certain number, which are, let’s say the managers, and then everyone else is the primary users that are going to be. So let me start with the managers. Hey, listen, we’re going to make this change. It’s going to be a really significant change. People are going to feel that impact and it’s going to be jarring. So I’m talking to you first, number one, let you know where we’re coming from, how you can best prepare, how you can enable your teams, how you can have conversations with them. A lot of times the conversation from leadership is best received when it’s something about the vision or the strategy or something like that. But reality is that when I, as a, you know, I’m on the front lines, I’m doing the work, I don’t want to hear from, you know, the person who sits on a screen and is, you know, the person I watch while I’m eating my lunch. I want to hear from my manager, the person I talk to every day. So sure, the leader, lay out the vision, help me understand where the company’s going, why this is beneficial, why this is a great thing, give me confidence in the company. All right, like, let me know this. But then when it comes to how is this going to affect me and how am I going to deal with these things that I’m struggling with? I want someone who knows me. So ideally it should be the person that I report to. So if the leader first talks to those people and then they’re better prepared to have that follow-up conversation. So you have the big lunch and then immediately after that, the manager sits with their group of eight people and says, 00:40:20 | Val Kroll: all right, everybody, these are some big things coming. 00:40:24 | Yehonatan Schwarzmer: I’m sure there are some big feelings. Let’s talk about it. Or if you have questions or you need specific help, or if he said some things that weren’t so clear, or she mentioned something and you didn’t know what that means, let me, because the manager has already had the conversation with the leadership and doesn’t, it’s not the manager sitting, and a lot of times, the manager sits there just as clueless as everyone else, but they’re the manager, they’ve got no choice, they’ve got to take the hit. It’s like, okay, everybody, let’s talk about it. Those are great questions, I’ll get back to you. That is not going to instill confidence in anybody. But it means that rolling this out in not a, here’s the training that we will provide to everyone, but really being thoughtful about who can help us get everybody else on board, who is more likely to buy in, who is more likely to demonstrate the benefit, who is a manager who could really talk to people about their concerns. That’s one tactic if you’re talking about a change at scale, so that it’s not just this huge thing that you have to swallow all at once. I will say that another piece of this really does have to do with, I was trying to see if we could avoid the word culture in this entire conversation. And of course we can’t, but it’s difficult because culture means different things, different people, and it’s almost like an excuse word. I don’t want to use it as an excuse word, but the companies that have established a culture or work to establish a culture then enables these types of conversations and that orients everyone towards a very productive, aligned conversation will have a completely different experience than the ones in, let’s say, where there is no communication or everyone is fighting for themselves or there’s a lot of internal competitiveness 00:42:05 | Val Kroll: or there’s silos and turf 00:42:10 | Yehonatan Schwarzmer: and very, very different conversations. I will tell you that in this, the concept itself I think is one that we can all understand. Practically, anyone who has seen it will understand the difference night and day in creating a culture in which these things are real versus where they’re not. I’ll give you just two quick examples. I have a mentor. He’s been a mentor of mine since like 20 years. One of those people that is just giving and kind and thoughtful and wise. His name is Tony DeKemper, lives in Baltimore. And when I was just trying to figure out what I wanted to do with my life, I had some conversations with him. He was fantastic. And ever since then, he has been just incredibly guiding and he ran a company for a while. 00:42:58 | Val Kroll: And in that company, it was bizarre to me. 00:43:02 | Yehonatan Schwarzmer: But any new hire who joined that company had to join a two-day course called Fundamentals of Communication. And it was all about how do we communicate? What is communication? What is the relationship between communication and establishing relationships with other people? How do you grow those? How do you value those? How do you make sure that you’re actually communicating what you mean? How do you make sure that you’ve heard what the other person is communicating? How do you make sure that your communications are designed to build something up instead of just to make your point or whatever else it might be? It was a real commitment on the part of the company and it changed everything. Because everybody who went through that experience, first of all, you go through that experience, you understand what the company is about, you understand that they really mean when they say, we really want to hear from you or something like that. But it also levels the playing field, which is another thing that I think is critical. You all, I think, know. Noah is a guy that I worked with for quite some time. 00:44:01 | Tim Wilson: He will make an appearance later on in this episode, by the way, just so you know. 00:44:05 | Yehonatan Schwarzmer: You can only do well when he joins. But one of the things that I learned from him, and I mean, just working with him, you can learn something every day. But one of the things I learned from him is, when I joined Search Discovery, he was running teams, he was running meetings, and he always started the meetings the same way. He said, the purpose of our meetings to achieve something, and we will only be able to achieve it if all of us are guided by being open and transparent. If we are open and transparent with each other, then we can have real conversations that move things along. Otherwise, we’re going to get locked in, we’re going to get stuck. In those couple of words, that little preamble, he did so many things, and this is important for any kind of management of change. One of the things that he did is he did away with roles. In this meeting, sure, some of us have more experience, some of us have less, but we’re not coming in this in a hierarchy. We’re coming at this because we need to solve a problem. And if you have something that can contribute to solving the problem, and you keep that from us, all of us lose out because of that. You are valuable. Saying to someone that they are valuable versus telling someone they are valuable, I’m sorry, versus showing them that they’re valuable, meaning demonstrating it by saying, 00:45:19 | Val Kroll: I don’t think we have enough in this conversation 00:45:22 | Yehonatan Schwarzmer: because we haven’t heard from this person and the experience that you’re going to bring is unique and no one else has it. 00:45:27 | Val Kroll: That changes the entire feeling of the conversation. 00:45:33 | Yehonatan Schwarzmer: When we talk about overcoming the barriers to change management, I guess I’m just talking about a lot of different ways we could come at this, but the overall idea that, first of all, it doesn’t have to be one big thing that you have to crack with this entire group of as many people as it is that’s very challenging. And the other one is the norms that you establish from the beginning that you will then use to manage through the change instead of dropping it on people through the change because that is likely to end up with more resistance. And I mean, most of us have seen this, a big change is rolling out, and the company, the leadership, whatever it is, says, all of your opinions are important to us. We want to hear from all of you. What does that mean? It means that we made a little slot in the door. You could put that little paper in there, something we’ll find the key and we’ll get into the door. But for right now. 00:46:18 | Tim Wilson: We set up a special email address for you just in your questions too, so we don’t want to load a blind box. You could feel like you’re bringing people in 00:46:24 | Yehonatan Schwarzmer: and you’re drawing people in and you’re bringing people together and you’re creating the alignment, but there is a difference between the kind of paper cut ways that don’t actually do it and the ways that make people feel, I want to be part of this. And even if I am nervous, you know, the term psychological safety, I don’t like it because you can hide so much under it. But there is a principle there where people really feel like if I am uncertain or I’m concerned about something, that’s valid because that is how I will learn. And the point here is that we are all learning so we can all benefit and we can all get better and we can lift each other up. You can’t do that if you’re hiding behind things. All right. Nope. We got it. Sorry, Val. 00:47:02 | Val Kroll: It’s episode 184, 00:47:05 | Tim Wilson: psychological safety and analytics with JD Long, just to, you know, plug that, which was a fun episode. 00:47:11 | Michael Helbling: I didn’t think we would have a shortage of things to talk about and really the shortage is the amount of time we have to explore this issue. You had a 10. Yeah, so we do have to start to wrap up. You should feel bad. We didn’t even get to talk about the measurements side. Shame. Shame, shame. I should feel bad. I know. So, well, obviously that indicates that there’s much more to talk about on this topic. But yeah. 00:47:36 | Tim Wilson: And really Val’s question never really got answered. Oh, well. 00:47:40 | Yehonatan Schwarzmer: We need one minute to try to answer it. 60 seconds. Number one, if you don’t have a lot of change going on or it’s not a big group or change management is not a big concern, don’t worry about the measurement. You’ll overkill it. You’ll hammer it to death and it’s not a nail. Don’t do that. But if it is the type of thing where either you expect resistance or as big and you do need a lot of measurement, then yes, there should be change and there should be measurement of that change. How do you do that? Either you can do that integrated within the process so that if you integrate the change in the process then as you measure the project itself you’re also measuring the change or you measure the change as a parallel work stream in which there should be phases. So, in example, Kotter has a famous model. McKinsey has their 7S model. The ProSci has an ADCAR model. There are lots of models where they walk you through phases of changes. Each of those can be broken down and as you go through the change, you can actually measure how well are we approaching each of those so that we are ready for the change, we’re implementing it correctly and we’ve embedded it correctly. Those can be measured because you can break them down and then they become a separate work stream that will be measured and when you see that that going successful, the project is done. 00:48:43 | Val Kroll: Nice, yeah. 00:48:45 | Michael Helbling: That was actually pretty good. You were very close to 60 seconds on that. 00:48:49 | Val Kroll: Yeah, that was very close. 00:48:53 | Michael Helbling: Yehonatan, what a pleasure. This is so good. Interestingly, I feel like we just started on what I call the top layer of the conversation and I feel like there’s so much more exploration people can do from here. So, thank you so much for opening the book a little bit and I think there’s a lot of applicability right now for people in this conversation. So, really appreciate you coming on and doing that. One thing we do is a last call. Something go around the horn, share what might be interesting. Yonatan, you’re our guest. Do you have a last call you’d like to share? 00:49:27 | Yehonatan Schwarzmer: I will tell you that one of the things I’ve been thinking about for a long time is how technology, specifically AI now, but technology in general adds a benefit, but then very often there is a cost that comes with it that you don’t think about until later. And whether that’s a note taker, that’s really great because you have a great record, but then you didn’t do the physical work of writing something down so you just don’t remember things as well. 00:49:49 | Michael Helbling: Did Tim prep you ahead of time for this? No, actually. 00:49:52 | Yehonatan Schwarzmer: This is like a hobby horse of his right now. 00:49:54 | Tim Wilson: Okay, that might have been a moan, moantim. 00:49:59 | Yehonatan Schwarzmer: But specifically I think about it in the context of education. Education is an area where we are using a lot of technology and there’s the technology that we think is detrimental to children’s growth or students’ growth. Should they have phones in the classroom? Should they get their own Chromebooks? Things like that. But then there are also questions about what about the technology we are using to actually drive that education? And there are some companies that are coming out that want to just have education be only through technology. And I think it’s a big open debate. There’s a guy named Jared Horvath who recently gave a congressional testimony, I think it was, and he’s just recently come out with a book called The Digital Delusion. He was a teacher for quite some time and he’s talking about how education done through technology comes with a lot of benefits, but you have to be aware of what you’re giving up. And then how do you thoughtfully get ahead of those so that you don’t lose as you’re gaining? 00:50:55 | Tim Wilson: It’s so funny. He is actually in our queue as a potential… I recognize that name because he bounced around somewhere. All right. I think he’s a little controversial though, so if I’m correct, right? 00:51:10 | Yehonatan Schwarzmer: Yeah. I mean, none of this is straightforward. It’s more, like I said, it’s something I’m thinking about for a lot because the question is not clear. What are you giving up before you pick up your pitchfork? You got to be relatively clear about that. But people like picking up pitchforks. All right, Val, what about you? What’s your last call? 00:51:26 | Val Kroll: Mine is actually semi-related to the topic, so I’ll be taking those brownie points for that. It is a medium article from none other than Eric Sandosham, one of our favorites. And this one in particular is about how AI is redefining the role of analytics leadership. And he does link, as he does in a lot of his posts, to past articles that plant the seeds of some of that thought, and they’re all worth reading as well. But one of the things that I think is really interesting is he’s talking about this paradigm shift and how we’re thinking about the new chief analytics officer and what makes a good one is to redefine one of the core competencies. Instead of focusing on solving problems, it’s going to be more about the ability to problem find, problem find and problem define versus the actual solution of it. And he goes into the T-shaped skills and operating as decision scientists. And so anyways, it was a really thoughtful read, and I just found myself highlighting a bunch of excerpts in it and shooting it off to random people. So I was like, I need to share it here as well, but it was definitely a good one. 00:52:32 | Michael Helbling: Nice. Very cool. I’ll check that out. 00:52:34 | Tim Wilson: Tim, what about you? So I alluded to it earlier that our former coworker of all of us, Noah Levin, has started a sub-stack that he has posted some stuff on that is, Jonathan, as you said, he is super thoughtful. So the one that really triggered me also sort of relates to this topic. He had a piece called Let My People Code, the part of the AI debate nobody wants to say out loud, which maybe sounds a little clickbaity because he does have a marketing background. But the essence of it is he’s kind of making the case that one of the real tensions around AI, it’s not just about declining quality, it’s about experts grappling with the loss of exclusivity 00:53:21 | Val Kroll: as a way many more people can be competent. 00:53:25 | Tim Wilson: So even Jonathan, as you were talking about, the person who was the Power BI guru, and now they’re moving into Tableau and they’re losing that, and he has, it talks about Wikipedia, it talks about the printing press, and a little bit of it is a little bit of like, you know, check yourself. For me, looking at it, I’m like, I don’t want, I don’t want code, code, code, writing R because I work so hard. If everybody can write R, what am I doing? So it does force a little bit of what’s really going on when you’re resisting, when you’re much more democratizing an ability to a base level of competence. Which just to add on to that, Michael, it was two plus years ago when we were at Measure Camp New York, when you made the comment about generative AI making, getting everybody to average on stuff. Like that, that really stuck with me then. Noah’s piece is saying, people get, experts get upset that people can do average, but average is often, you know, good enough. It doesn’t mean you don’t need the expert. So, and it’s, he has all sorts of historical references and stuff. And he’s a fantastic writer. So it’s a great read. Michael. 00:54:40 | Michael Helbling: So, yeah, I, so there’s a person who frequently in my career has somehow said something much more succinctly than I’ve been able to enunciate it. And this happened again a couple of weeks ago at. Okay, it wasn’t me. Mark, you know, no, it’s not you, Tim. 00:54:59 | Yehonatan Schwarzmer: Sorry. 00:55:02 | Michael Helbling: I don’t think anybody was taking that one. Not guilty either. Yeah, I see the keyword was succinctly. Yeah, no, so I was having this conversation at marketing a little something about some of my concerns about how anthropic is kind of doing some of the things they are behind the training and the adjustment of the model and how they’re kind of giving it sort of like this sense of self kind of concept and everything. And then Mark, David McBride, pointed me to an article by Gary Angel who is that person. And it’s the articles called Can You Teach an AI Ethics? And he basically dove into specifically, there’s a Wall Street Journal article about Amanda Askell who’s their philosopher kind of training the AI on ethics and morality and he kind of got into a very interesting examination of that and I really benefited from reading it and sort of helped me kind of like bring my thinking more to a point on some of the things that were just sort of like hanging out at the edges of like, I don’t know if I really agree with that and I’m not sure why. Like Gary’s thinking has always been really helpful to me in that regard. So I highly recommend it. 00:56:19 | Tim Wilson: Gary started publishing more frequently on Medium which is always a highly recommended read. 00:56:25 | Michael Helbling: He’s fun to read since forever and so yeah. So that’s, I would highly recommend that. All right, well we’ve been chatting and I think it’s for sure. You’re going to have questions, you’re going to have thoughts, you’re going to have comments. We would love to hear from you. The best way to do that is to reach out to us. We could do that via our LinkedIn page. You could do that to the Measure Slack chat group or by email at contact at analyticshour.io and if we’re re-listen, leave a rating and review on the show as well. And if you would like stickers for your laptop because even in this day and age of AI, we still need stickers on our laptop to show what tribes we belong to, you can reach out to us on analyticshour.io. There’s a page and request form for that. All right, Yehonatan, thank you. 00:57:18 | Yehonatan Schwarzmer: Thank you so much. Thank you. It is always nice to talk to each of you and all of you together as a real treat. 00:57:24 | Val Kroll: I know, I know. 00:57:26 | Michael Helbling: This is good. This is really good. 00:57:29 | Val Kroll: All right, and I think we’re all facing change 00:57:33 | Michael Helbling: and no matter what kind you’re facing and whatever stage you’re in and change that change curve, one thing you should always do and I think I speak for both of my co-hosts when I say keep analyzing. 00:57:45 | Announcer: Thanks for listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at @analyticshour on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group. Music for the podcast by Josh Grohurst. Those smart guys wanted to fit in. So they made up a term called analytics. Analytics don’t work. 00:58:09 | Charles Barkley: Do the analytics say go for it no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 00:58:23 | Michael Helbling: I feel like I might have hit a nerve there. 00:58:28 | Tim Wilson: I was trying to be, would that come through? Would that work if I did? You’re fine. It’s totally fine. Yeah. 00:58:34 | Val Kroll: No, it’s good. 00:58:35 | Tim Wilson: Okay. I had a few other. I was going to do Kubla Ross five stages of grief or detours for the other three. 00:58:41 | Val Kroll: Yeah, I thought you were, I always try to guess you know what he’s going to say. I thought you were going to say stuck in what, what, what anger? He said, Michael, with the GA4, I think I’m such an anger. Oh yeah, I’m still stuck in anger. Yeah, I’m still stuck in. 00:58:53 | Charles Barkley: I have an immersion in anger. Still stuck in anger. 00:58:55 | Val Kroll: Yeah. Yeah. Still stuck in anger. 00:58:59 | Michael Helbling: Yeah. We’ve done a number of the things in that document. So. 00:59:08 | Tim Wilson: Well, episode number 299, take three in which Michael has followed the recommendations. 00:59:17 | Michael Helbling: Why don’t we, not all, but I’ve got all the recommendations. Oh, like I rebooted my Wi-Fi point, which has been functioning flawlessly for weeks, right up until today. And then I rebooted Chrome too. So hopefully that’s enough to get us through. Oh, we’re not going to do a full system restart. No, we don’t want to. 00:59:41 | Tim Wilson: No, we’ll say that’s our episode 300. 00:59:43 | Val Kroll: We have to have something to celebrate. 00:59:45 | Michael Helbling: Yeah, yeah. And I don’t think that’s the issue here. I’ve been putting my computer through a lot of change lately. I like it. Have any of your GPUs? 01:00:04 | Val Kroll: Yeah, just I’m doing a lot of stuff. 01:00:07 | Michael Helbling: Okay, let’s get this show on the road. Okay, here we go. Here we go in five, four, three. 01:00:28 | Tim Wilson: Rock flag and succinct refers to the expression of ideas, information, or arguments in a manner that is remarkably brief, clear, and to the point. As a succinct statement or piece of writing, I’ve always unnecessary words, fluff, and excessive detail, capturing the core essence of a message with maximum compression. The post #299: AI Can (Help) Build the Dashboard. It Can’t Build the Buy-In. appeared first on The Analytics Power Hour: Data and Analytics Podcast .

May 26, 202652 min

#298: Listener Questions Answered Live from Marketing Analytics Summit!

Picture this: four analytics professionals, one live audience, a bunch of submitted questions, and absolutely no filter when it comes to sharing their real thoughts about AI, stakeholder management, and the state of the industry. That’s what you get when the Analytics Power Hour goes live from Marketing Analytics Summit , with Michael, Moe, Tim, and Val fielding everything from, “How do I prove I’m a partner rather than just an order taker?” to “What’s your icky threshold with AI?” The conversation ping-ponged from the fundamentals—like why curiosity beats feature checklists when selecting tools—to the controversial, including a heated debate about whether AI-generated meeting notes are helpful productivity boosters or lazy crutches that strip away human editorial judgment. Along the way, they tackled data trust issues, the pressure to show AI efficiency gains, and why trying to nail down the “best” deliverable will just trigger existential musings about what a deliverable even IS! Fair warning: Tim gets triggered by AI hype, Moe calls some industry BS, and everyone agrees that being useful beats being right. This episode is brought to you by Prism from Ask-Y —your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Episode Transcript 00:00:00.00 [Announcer]: Welcome to the Analytics Power Hour. 00:00:08.92 [Announcer]: Analytics topics covered conversationally and sometimes with explicit language. 00:00:15.56 [Jim Sterne]: The Marketing Analytics Summit is pleased to present these four amazing podcasters. 00:00:24.80 [Jim Sterne]: We have Michael, the self-effacing every-man consultant who knows far more than he lets on. We have Moee, who flow all the way in from Sydney, Australia. She’s the determined, driven practitioner, the one to stand up and say, well, that’s all very well and good, but how do we actually make it work? There’s Val, who co-founded the Consultancy facts & feelings, because we have strong feelings about our facts and no facts about our feelings. And there’s this guy named Tim. Ladies and gentlemen, take it away. 00:00:56.72 [Michael Helbling]: Hi, everyone. 00:00:58.22 [Michael Helbling]: Welcome to the Analytics Power Hour, this episode 298. And we are recording live at the Marketing Analytics Summit in the beautiful Santa Barbara. You know, 21 years ago, my very first Analytics conference was right here in this city at this conference, formerly known as E-Metrics. And similar to today, it was a conference full of smart, engaging, and passionate people learning together about how to solve the problems they were facing day-to-day in the analytics industry. A lot has changed. A lot has changed. But persistent through all of that is this beautiful analytics community. And the shared learning, this podcast was actually created to encourage and celebrate. So to that end, I have my co-hosts with me. And we have put out a survey to gather your questions. We’ve been asking you during the conference. And so we have a number of them to get through, and we’ll do our best to answer them from some of the perspectives we bring to the table in our various roles. So let me introduce them. Moee Kiss. 00:02:13.44 [Michael Helbling]: Hi, folks. 00:02:14.44 [Michael Helbling]: Hi. 00:02:15.44 [Michael Helbling]: Director of Data for Product, Canva. 00:02:19.44 [Michael Helbling]: And of course, Tim Wilson, Head of Solutions at facts & feelings. 00:02:24.16 [Michael Helbling]: Hello. 00:02:25.48 [Michael Helbling]: And Val Krull, Head of Delivery at facts & feelings. 00:02:29.04 [Val Kroll]: Hello, hello. 00:02:30.04 [Michael Helbling]: And I’m Michael Helbling, my president of Stacked Analytics. 00:02:34.72 [Michael Helbling]: Okay. 00:02:35.72 [Michael Helbling]: What a privilege to be here with all of you in person. It’s been a wonderful couple of days. So let’s just dive right into it. 00:02:45.36 [Michael Helbling]: So we’ve got a question. 00:02:48.24 [Michael Helbling]: This one came from James from Namer’s Children’s Health. What’s the best approach to establishing yourself as a partner and not an order taker when working with stakeholders? 00:02:59.12 [Val Kroll]: James is the one who has the most trouble with the uncomfortable pause. So that’s why he looks the best. 00:03:09.76 [Moe Kiss]: Oh, I assumed that was your cue. Anyway, we were having a conversation earlier about taking notes over lunch. And if you don’t want to be perceived to be the person in the room taking notes, then don’t take the notes. And I think it’s really jarred with me because I have this belief, I mean, I always know that I come back to everything that Cassie Kazarkoff says about just be useful. And I am sometimes fine. I’m still like, I might be the most senior person in the room and sometimes I do still take notes. Part of that’s my brain, but part of that is also that I want to be useful and the people in the room are normally smarter than I am. And I have much more interesting things to say. So I actually kind of struggle a little bit with the question in and of itself. I think one of the things that maybe means that it’s OK to still take notes, but is also adding your voice to the conversation. 00:04:02.44 [Michael Helbling]: I think, I don’t know, I just, again, we were chatting about this at lunch. 00:04:06.20 [Moe Kiss]: When you don’t contribute to the conversation, when you’re not willing to be accountable for what you say, I think that’s when you kind of, you’re not the partner, so to speak. 00:04:15.36 [Tim Wilson]: I mean, I think it’s, I agree. 00:04:17.00 [Michael Helbling]: I think it’s don’t be an order taker. 00:04:19.12 [Tim Wilson]: I think to me it is, and there have been a lot of discussions at this conference, a lot of the discussion in the industry were at large around how are we using AI and there to be curmudgeonly and a little cranky about it. But a lot of is, how do I take more orders? How do I take more orders and produce more output? And that misses, to me, the fundamental part of becoming more of a partner is to step into their shoes, ask the questions, write it down, be genuinely curious. Don’t be trying to get to, how quickly can I turn this into a ticket? And I think that has been, AI has nothing to do with that. That has been a 20-year problem in our industry when we sit around and wring our hands and gnash our teeth about why aren’t we brought in. We produced the next dashboard like they asked us to, we gave them recommendations, but did you ever actually sit down up front and say, I just, I’m an analyst, I’m curious, I want to understand what you’re grappling with. And there have been, even at this conference, there have been some discussions around that. But I, I lay awake at night fretting about the industry being, I want to be a partner, but I behave like an order taker. And it’s like, it’s simple, just be curious, go ask them, get inside their heads and start asking them, how can I have some problems? 00:05:41.96 [Michael Helbling]: Yeah, okay. 00:05:42.96 [Moe Kiss]: I kind of want to call a bit of bullshit on it though, because I do, I get- 00:05:46.88 [Michael Helbling]: On brand? 00:05:49.72 [Moe Kiss]: I get what you’re saying, and I hadn’t really thought about like AI driving more of the order taking. I actually feel like my experience might be a little bit different. I think the thing that I just keep observing is data folks don’t want to be accountable. I had this conversation at the back of the room earlier, it’s like, they want to be like, here’s the numbers, don’t look at me, like I don’t want to be on the hook for the recommendation that I’m making. And I don’t know, I don’t think that’s an AI thing. I think that’s just a, maybe it’s a 20 year old industry thing. 00:06:19.68 [Tim Wilson]: I think it is, I mean, I actually agree that it’s like, we make recommendations and they’re not following them. And a lot of times like, where did this start? Well, we asked them what they, we asked them what their business question was. If you’re asking them, even if you’re saying, I’m asking what the business question is, there is a human element of saying, I am your partner, I am in this with you together. So I’m just pushing for moving upstream and not jumping to the data and jumping to the solutioning and jumping to, did I produce the published report that made a recommendation and then I’m upset. 00:06:53.84 [Michael Helbling]: So you didn’t fix that? 00:06:56.88 [Val Kroll]: Yeah, I’m just kidding. I would say like to get started with that, it’s not even always about like anticipating all of their needs. I think it is being curious and just thinking about the motivations and the stresses that that person has in their role, like what pressures are they facing? And I think that this has come up a couple of times too about how do we de-risk the decision that’s upcoming for them or where should they spend that next dollar and do a little bit of prep work before you walk into that room to show that you’re trying to empathize with what they’re dealing with because their job is hard too. It’s very different than yours, but starting from that place, I think is a good way to kind of say like, we’re in this together, we’re side by side, not sitting across the table and I’m just going to push something on you at some point. 00:07:38.52 [Moe Kiss]: But I think the difference there is you talked about preparation and I think a lot of the curiosity can come without necessarily absorbing stakeholder time. And I worry sometimes that the push is like, oh, well, I haven’t got enough context. My stakeholder is time poor. So, you know, that sort of thing where it’s, I actually think there’s a lot of personal responsibility that data folks need to take to be like, I have enough information available to me throughout various, various Slack channels, emails, et cetera, to have some of that curiosity before even asking the question of a stakeholder. 00:08:10.68 [Tim Wilson]: 100%. 1000%. I see your 100% and raise it. But I think if you’re thinking about it, you should be saying, this is my understanding, but here’s something that I can’t. I listened to our quarterly conference call and there was something that doesn’t make sense to me. I tried to figure it out. It doesn’t. I think you, marketing manager, might help me explain this because I can’t square that circle. So, yes, showing up and that certainly worked with analysts who say, well, what are my lists of discovery questions? What is your most challenging business problem? If that’s the way you show up, they’re going to be like, I have to explain every minute. 00:08:44.24 [Moe Kiss]: 25-year-old data slanted slightly because that’s kind of how they show up sometimes. No age. 00:08:49.84 [Michael Helbling]: No age. All right. 00:08:51.84 [Moe Kiss]: We’re going to move to our next question. This is the whole episode. 00:08:53.84 [Michael Helbling]: Sorry. And this is coming from a member who’s here in the audience, Michelle Kiss. 00:08:59.12 [Michael Helbling]: Whoo. 00:09:00.12 [Tim Wilson]: Any relation? 00:09:01.12 [Michele Kiss]: None. Completely the same person. Complete coincidence. Yes. Confuses everybody. Okay. So, first of all, my question has two parts. There’s a pre-question that is required for setting the context. 00:09:15.12 [Tim Wilson]: Yes, they’re related. 00:09:16.52 [Michele Kiss]: And so that we can interpret your answer. Okay. So, first of all, the pre-question is where do you guys stand on AI? Are you a skeptic or a fan? Okay. So, that’s the context. Then, what things have you personally found that it is useful for and reliable for? It has to be life. 00:09:42.88 [Val Kroll]: Okay. 00:09:43.88 [Michael Helbling]: I’ll go first. 00:09:44.88 [Val Kroll]: So, I would say that I’m a skeptic, but that’s just kind of like my nature. I don’t think there’s anything new or special about that. That’s just kind of who I am as a person, but I’m excited at the same time. And I think I’ve been inspired by a lot of the conversations we’ve been having over the past couple of days that have gotten my wheels turning about some different things that I can try when I get back to my desk. So, I would say more on the skeptic, but still excited, still optimistic. I don’t think, you know, world’s ending just yet. One of the things that I have been playing around, this is recency bias, but just thinking about how do I start at like day 60, day 90 with the where I’m at with an understanding of the business context for a potential prospect conversation. And so, we had been doing a lot of process effects and feelings very manually to kind of go through and listen to like most recent earnings calls or pulling out, you know, press releases or seeing what was publicly available just to kind of understand what is the context, what is the competitive nature, where are they competing and using AI and building out some gems to build that out for me. And so, I’m asking it to give me some tables and comparisons of where do they compete on what message is? Is this about pricing? Is this about quality? And so, really trying to figure out like how do I get into that context so that I’m not asking like question number one, what are your goals? It’s like, let me start with something because then I’m going to be able to ask much richer, deeper conversations. And so, again, recency bias, but that’s been the most fun one just to kind of understand outside of an analysis, what’s something I can do to show up in those conversations to kind of position myself to be a better partner. 00:11:26.88 [Moe Kiss]: Mine are not sexy. Like I don’t… That wasn’t sexy. Build me a table of your competitors. Well, it depends what you think is sexy. Yeah, I mean, I do a lot of leadership-y things and those things tend to be quite admin-y. I do think the one… Oh, I forgot your first part of the question. I’m excited and terrified. I would say I’m… I don’t… Fan feels like a strong word. I do like a lot of the potential and the personal gains in the way that it’s changed how I work, but I have this like level of fear, which I think is pretty rational and normal. 00:12:05.76 [Michael Helbling]: It’s a pretty big shift. 00:12:07.40 [Moe Kiss]: And yeah, a lot of the ways I use it are not sexy. But I do think the thing that has been really nice for me personally is feeling like code is not so far away. Like it’s been a couple of years since I wrote code very regularly, and there are things that now I feel much more comfortable. I’ll go do a quick Here’s the Queer I Want to Vibe code something. 00:12:26.56 [Michael Helbling]: And I also know enough to know if it’s not correct, which is important, but it makes 00:12:32.44 [Moe Kiss]: it feel like more approachable to a skill that’s pretty rusty. The rest of the stuff is like making my snarky comments in Slack, seem less snarky, like meeting summaries. Like, yeah, it’s not sexy. 00:12:49.48 [Tim Wilson]: So I’ll say I’m a skeptic and a fan. So I’ll punt on the first question. My skepticism comes from I’ve been developing this observation that I feel like we are as analysts, we are often saying this is the thing the tool can do. This is my hammer. So now I’m going to go figure out what the nail is, and I’m going to tell myself that the nail is going to solve a problem that has nothing to do with the technology. So I have found it to be very useful on debugging code, kind of piggybacking off of what you said, because coming from having written code, debugging it. It is very, very useful. I get very, very nervous when I have Vibe coded and yes, it brings results out. But I’ve now seen in the wild how that can go awry. I actually find that it’s very useful just from a thought clarity. And I think that hasn’t changed. That probably took me three times listening to Jim Stern, give various presentations about writing prompts for it to take. And it was the first time I heard it now, kind of everybody saying it of some of the keys with writing prompts, but prompts, I think by writing. So saying this forces me to organize my thoughts and then I’m going to trust and verify what comes back. That is much more on the help me clarify my thinking, because if you’re helping me clarify my thinking, just like a human being who I think, no, that’s wrong. That’s bullshit. You can challenge me, but you may be wrong. I am not, and which wasn’t the, where is it? Not useful, but I, anytime it’s like, it’s going to basically do glorified anomaly detection and spit out insights. Yeah, I’ll go to the mat for a while on that. And I’m pretty triggered with various claims to have it do that. Michael, what happens after you finish a GA4 analysis? 00:14:43.32 [Michael Helbling]: Oh, traditionally, I guess I paste screenshots into a doc, rename it final, final, uh, you know, do not change final, uh, lose the source query and then wait for somebody to ask, can we break this down by campaign? Terrifying. Please stop. I would love to emotionally and professionally. 00:15:05.12 [Tim Wilson]: Well, that’s why ask dash, why.ai just released the Prism Cod co-work connector. It brings the whole Prism brain to your GA4 big query data. Ooh, the whole brain analytics, agent harness, skills, memory engine, the works. 00:15:24.32 [Michael Helbling]: Wow. So co-work doesn’t just answer questions. It remembers context, uses repeatable skills, keeps analysis 00:15:30.82 [Tim Wilson]: organized, exactly. You’re, you’re picking it up. Your co-work based analyses are accessible in Prism, organized, traceable, auditable, and ready to use with your other data sets. 00:15:43.06 [Michael Helbling]: I love that. Cause currently my audit trail is mostly like, oh, I know I had a reason for doing that. I can’t remember what it is. 00:15:50.54 [Tim Wilson]: That checks out. The connector also ships with ready to run funnel and cohort skills right out of the box. 00:15:57.86 [Michael Helbling]: So I can ask for retention by acquisition channel and not immediately enter a fugue state. 00:16:02.58 [Tim Wilson]: Right. And every analysis becomes a shareable page. Prism auto-generates the dashboard page right from co-work. Oh, so the answer doesn’t die in a chat thread. That’s right. It lives as a reusable and shareable analysis. Well, that’s very rude to my old workflow, but fair. We’ll go to ask dash, why.ai. That’s ask dash the letter y.ai and sign up for the wait list. Yeah. 00:16:27.70 [Michael Helbling]: And use code APH and that’ll get you pushed to the top of that wait list. Ask dash, why.ai code APH. I like it. Co-work. 00:16:37.46 [Michael Helbling]: It’s GA for analysis, but with receipts, which was not part of the question. 00:16:43.74 [Val Kroll]: We’re only on our second question. You can’t, you can’t get triggered yet. 00:16:47.70 [Tim Wilson]: No, I welcome it. I was triggered 36 hours ago. 00:16:50.14 [Michael Helbling]: I think it’s okay. I go through life triggered. All right. We have another question from someone here in the audience. So I’ll hand it off to Jen Coons. 00:16:59.22 [Jenn Kunz]: It actually follows up with what Tim was just saying, perhaps a bit of it. Do any of the ways we promote and use AI in the industry make you feel icky? Do you have a threshold of lines that you don’t want to cross when it comes to AI? 00:17:12.98 [Tim Wilson]: Can I add, I’ll add a new one to that. I am not a fan of the note takers in meetings and doing the summaries, which I know is super controversial just inside. And because to me, I’ve watched and I’ve watched this time and time again, how much that drives laziness and not paying attention in the meeting and not editorializing. And it is flat summaries. I’ve worked with some clients where they have literally said, oh, you weren’t at the meeting, we recorded it. Here’s the Gemini summary. And it’s just not useful because it’s not telling me what the human beings in the room were thinking about. Let me throw one other, I’ll be quick. Because, Jim, when you did your closing note, you know, yesterday, which was very good, not going to be too useful for people who were listening to this, but there was a lot of talk about using AI to ramp up junior analysts and build lots of different, use different tools that kind of let them kind of do self study. And I wound up wondering about where do we teach the junior analysts, how to actually relate with people and have the creative, collaborative process. And so there’s something there, too, that makes me nervous that we’re at a conference right now, like are we on some logical trajectory where we think we’re all just going to sit at home and just have AI do everything that needs to happen. There is a very, very real part of this job that is human and communication and collaborative creativity. And I get very nervous that people are not recognizing the value of that and trying to have AI replace it. That was really deep. 00:18:56.02 [Moe Kiss]: I was going to talk about meeting notes. 00:18:58.38 [Tim Wilson]: Go for it. Do I need to move away? 00:19:01.22 [Moe Kiss]: No, but so for me, I actually find the meeting notes summaries incredibly useful. And the biggest thing for me is, like, as someone who self-declared 00:19:10.78 [Tim Wilson]: has ADHD, can you square the meeting notes with the taking notes by hand? 00:19:15.98 [Moe Kiss]: I do. So I do still often also take notes. But the taking notes for me is me, number one, it helps me pay attention in the meeting. And number two, it helps me retain the informations. Like if we’re talking about especially something complex, I won’t necessarily fully hear it. And then the next day, I sometimes do look at my own notes. I won’t necessarily look at the notes. But what I find useful is the next steps. There is always like, no, when you are trying to corral like 20 people, 00:19:47.02 [Michael Helbling]: you can be like, what is the big deal? 00:19:51.42 [Moe Kiss]: What is it? Why is that the big deal? 00:19:53.02 [Val Kroll]: Tim, just fell off the stage for those of you listening. That was a Redd Foxx impersonation for anyone who walks. 00:19:58.82 [Moe Kiss]: It’s like, this person’s going to follow up with this. By then this person’s going to follow up with it. 00:20:02.46 [Tim Wilson]: And because it’s actually fucking terrible at that, it just goes through and says, this is what it was. And you need the human editorial person saying, what are the real next steps? That’s where it’s trying to go through a discussion. 00:20:15.70 [Moe Kiss]: To get back to the question, though, of what makes me feel ick, what makes me feel ick is things being shared that have not been properly vetted and QA and meeting notes fall into that category just as much as analysis or write up does. Like that’s a bit that I get stressed about. Is it analysts are like, yes, I can pump out more stuff. I’m going to automate this report. I’ll send it out. I’m never even going to look at it. And I’m like, oh, that feels uncomfortable. 00:20:41.98 [Tim Wilson]: But isn’t the meeting notes is asking people to do a lot because they’re like, I got to get the meeting notes out promptly. And it takes an enormous amount of diligence to say, I am truly going to read through these and modify and write my own little summary and write. So it’s like, you can paint the picture, but watching what actually happens with the people I’ve worked with, all of a sudden I’m like, oh, this was just barfed out. And I’m sure they scan through it and I’m sure they told themselves, yeah, that seems about right to be fair. 00:21:13.50 [Moe Kiss]: I just take the like the here are the action items. I don’t send the whole summary. Is that worse or better? 00:21:19.86 [Tim Wilson]: I well, if you take them and you say, yeah, that looks about right. I think that’s a problem because I think there is much more often. There is the person who’s sending those out should have a responsibility to say these are the things that really need to happen. There’s a level of prioritization and wording and body language and and adding net new stuff that I know that Joe said that Joe was going to do this. But the reality is I know in our organization that Joe is going to have Mary work with him on this and putting that sort there, there is something in that maybe I’ll get off that. 00:21:56.54 [Moe Kiss]: There’s a lot of nods, so I’m interested to hear more Tim. Normally, I don’t have this kind of live feedback that suggests maybe you’re right. 00:22:02.62 [Michael Helbling]: That they’re agreeing with me. 00:22:05.66 [Michael Helbling]: I think I’d go in a little different direction, which is I get sort of this icky feeling or there’s a threshold. I don’t want to cross with AI in terms of relating to it. And what I mean by that is some of the LLM big companies put out research where they kind of take what the LLM is doing and equate it to emotion and telling you that basically if you behave around your AI a certain way, it may actually impact its performance and that’s sort of what they’re seeing. And I think it’s a really dangerous thing and I think it’s also tricky to talk about because I don’t want to advocate for being mean to your AI or whatever. But as humans, we anthropomorphize things really a lot. And so I think we very easily buy into this idea that I make my AI feel bad if I yell at it or I make it feel good if I tell it does a good job. But in reality, it feels nothing. And most importantly is human to human interactions. I modify them a great deal based on what I know of that person and the empathy and the intuition I’m getting from that conversation. 00:23:13.46 [Tim Wilson]: So if someone’s struggling, I modify empathy and modifying Tim’s going to need a definition of hang in there, 00:23:22.74 [Michael Helbling]: and so you adjust to that person like if you’re giving feedback, for instance, whereas if I’m giving feedback to an AI, I want to be as direct and succinct as possible without having to kind of couch it in a phraseology or terminology that keeps it secure in its own quote unquote emotions, which are not real. And so for me, that’s sort of a weird line that I think I’d love for us to avoid 00:23:48.58 [Michael Helbling]: as we approach AGI. 00:23:51.86 [Moe Kiss]: Can I ask a crowd question? Only if you can figure out how to get a mic to them. No, I was going to make you count the hands for a rough estimate. Well, if you each take one and we take the average of each of your estimates, maybe we’ll have a decent score. OK, creating an agent based on your stakeholder one, stakeholder two, stakeholder three, so that you can tailor your comms and have a persona built out for each of it. Like, I want to know how we feel about the ick factor of like, is that icky or is it thoughtful? Because anyway, I can finish my thoughts afterwards. 00:24:30.66 [Tim Wilson]: Sam Bert, would you like to answer that question? There was a whole session on that. 00:24:35.58 [Michael Helbling]: So here’s my take on that, because I learned about that yesterday in a session and I actually really quite liked it, because I look at AI as a tool to take information and position it in the best possible way for the audience that you’re presenting it to, much like you would present maybe a slide deck to one person and a narrative to another based on how they consume or prefer data or information to be consumed. 00:25:05.82 [Moe Kiss]: Two notes. Just to be clear, I’m not picking on Sam. That was like very persona based. I’m talking about like someone in your team creates one that’s like, this is Moee. This is how she receives information, like very personalized. I think that’s a bit different. 00:25:19.66 [Tim Wilson]: So I think and I think there’s two aspects. So I think Sam’s and yours. One, I think if it actually forces the person who’s creating it, this is where we to actually really think about like to create it, you have to think about what do I need to? What do I know about Moee? What have I seen about Moee? Can I ask Moee something? 00:25:40.18 [Moe Kiss]: So that’s like, but do you think that they would or do you think they would just be like, I’m going to upload 50,000 conversations, a bunch of Zoom transcripts, whatever of interactions with this person and you tell me what you think they’re going to like. 00:25:52.42 [Tim Wilson]: I think that’s going to be less effective. 00:25:54.06 [Moe Kiss]: Yeah, it would probably would be. 00:25:55.66 [Tim Wilson]: And then the second part of that, I completely lost what my other thought was. So on Brent. 00:26:04.74 [Michael Helbling]: Well, a lot of these questions are tailored around a bunch of information. I uploaded about you three. No, I’m just kidding. OK, we have another question coming from someone who’s unfortunately not here. Joe Domoleschi, if you were stripped of your fancies tech stack and could only provide one specific deliverable to a stakeholder to prove the value of marketing analytics, what would it be? Live dashboard, a PDF report, a slide deck, a meeting with actionable insights, etc. What would you do? 00:26:34.66 [Val Kroll]: So one specific deliverable to prove the value of marketing analytics, 00:26:38.46 [Michael Helbling]: just to clarify, that’s what it says. 00:26:41.14 [Michael Helbling]: Yeah, OK. 00:26:44.66 [Val Kroll]: The results of an A.B. 00:26:45.66 [Michael Helbling]: test. So Format can be anything. 00:26:50.42 [Val Kroll]: I mean, well, he said deliverable. OK, so I would I would do a presentation, tight narrative results of an A.B. test. That would be my that’s my no explanation. 00:27:02.30 [Tim Wilson]: I think my not might not be a prove, but might be to convince or define. I would probably go to some sort of compelling story that I was comfortable. I might have pulled data from various sources. I might have run an A.B. test, but I would actually tell a really strong narrative and maybe with slides. Maybe not. I think that would actually be more convincing. 00:27:30.10 [Val Kroll]: Like, why does said deliverable, that’s your constraint. 00:27:33.70 [Moe Kiss]: But see, OK, this is the difference when I heard deliverable. I was like, it can be anything. And it sounds like you’ve both interpreted that in like agency consulting land very different to me, because I would have said if I could pick anything, it would be an MMM. Like, I can talk about that for weeks and months. I mean, at some point, it becomes valid. But it would probably be an MMM. Like, I’d love to go through an experimentation tool, but I feel like that maybe is not in the spirit of the question. 00:28:00.90 [Tim Wilson]: I don’t know. Yeah. What is it? What is a deliverable? 00:28:03.18 [Michael Helbling]: We could get that was not the question. 00:28:06.90 [Tim Wilson]: No, it’s what is the deliverable is. So now I found sound like a consultant. But I mean, on an MMM being super compelling, and I’m sitting like 15 feet from Jim Janolio. So and having heard him talk like you can you can conduct a great MMM. You can you can build it and then you can deliver it horribly and doesn’t show anything or you can communicate it really, really effectively. So it is kind of what is a deliverable and how effectively is it created and delivered? 00:28:35.26 [Moe Kiss]: Or we could just combine all three and then we’d like be winning. If we’re perfect, you didn’t answer yourself. 00:28:43.18 [Michael Helbling]: I’m going to ask the next question. 00:28:46.10 [Michael Helbling]: From a member of our audience, Bryce Preslicka. 00:28:52.14 [Michael Helbling]: Preslicka. Preslicka. 00:28:58.58 [Brice Praslicka]: All right. So I have a client that had some major data issues previously. Once we got on to the project, we’ve cleaned things up. We’re in a much better state now. The problem is one of the clients, POC’s continues to act as though we have unreliable data. And even worse, we have a member of our team that continues to use words like discrepancy and lack of trust and keeps using words that don’t really help us accurately convey that it’s reliable now. 00:29:24.78 [Michael Helbling]: So I’ve sent memos. 00:29:27.38 [Brice Praslicka]: I have tried to coach them to not use certain words, but with both an internal team and a client that don’t trust data that is in much better spot now, how would you go about trying to regain trust? 00:29:40.26 [Val Kroll]: In the role of the client you’re supporting, are they on the business side or are they in an analytics role? 00:29:45.82 [Tim Wilson]: They’re in the business side. Well, now you have to provide an answer because he was just clear. So here, I’ll kick us off. 00:29:54.14 [Michael Helbling]: There’s no time for cash measures. Jeez, we’re in a world of AI now and we need to move quick. First thing first, stop inviting the internal person to the meetings. So they can’t screw you up. Second, take charge of those meetings and tell the client that you know what you’re talking about. And it’s time to make decisions and get off the pot. What are they afraid of? No, I don’t know if I could pull that one off. 00:30:16.50 [Michael Helbling]: But but, you know, start to form the communication 00:30:20.46 [Michael Helbling]: and put it into the positive realm so you get past that moment. 00:30:24.18 [Tim Wilson]: I I mean, this is going to sound easier than it is in practice. But to me, one of the things that AI has not helped we have struggled with for for 25 years is businesses that are looking for certainty and precision when there is actually they’re operating under conditions of uncertainty. And Jen Kunze’s presentation yesterday, like it’s like people think that, oh, the data, the data was never complete. The data was never perfect. It’s really no better, no worse. We can always point to stuff that’s not working. So that like so to me, once you’re having the discussion about is the data right? You’re losing if you’re using discrepancy. If it’s, you know what? Hey, can we reset? Can we really nail down the one or the two or the three biggest decisions you’re trying to make? Don’t worry about the data. Forget about the data. Yeah, it’s going to be involved at some point. I think a lot of times what happens, if you can really get them saying, 00:31:24.70 [Michael Helbling]: what I really want to know is, is meta delivering results? 00:31:29.62 [Tim Wilson]: And they’re like, can’t you keep crunching the data from meta? But I know there are gaps. Instead, you may say, really, let me talk to you about what a geo lift test is. So it kind of goes back to that order taker versus partner. And it’s it’s tough. They’re still working with you. So they have some level of trust. But I think we wind up fighting the fighting on the wrong ground. We were on the ground of like, no, but the data is good enough. Well, no, but this and go, I know it’s an asterisk and don’t use this word. It’s like, instead of like, we so quickly lose sight of it’s such a tangible thing to point to that the data has a problem. And we forget to say, what’s the what do we really want to know and try to elevate that conversation? I often I think it’s like, that’s actually not the right data set for it. Anyway, we keep chasing the wrong data set to most answer that question. 00:32:24.50 [Val Kroll]: I like that a lot. And the one point that you mentioned that I just expand upon is I think the really rooting yourself in like, what level of certainty is really required to answer this question? How much time do I have to turn this around? Is it something you need tomorrow? Do I have a couple of months before you’re going to make this call and really just trying to line up the various different methodologies that are at your disposal to bring that to bear to bring the right evidence to that question? You know, Paula’s presentation, like merging those different sources of evidence to really kind of paint that picture. I think can like shake people loose from like focusing on, you know, what percentage of, you know, people are opting out from whatever cookie banners and things like that, right? So I think like just not trying to play on that, like move the battle, I guess, not play on that turf and kind of say, hey, like let’s just focus on like Tim was saying, like those top those top questions that you’re really grappling with. And let’s think about how much business risk we really be introducing if it wouldn’t be perfect. So like, let’s think about the various ways we can kind of go about it. And sometimes just injecting a little creativity, if you will, into the methodologies, into that conversation can kind of get them excited about some different ways. Maybe it’s a user test. Maybe it’s, you know, it’s not going to be something we’re going to look for in a table as an example. 00:33:38.66 [Tim Wilson]: So be sure to record the meeting and send them the meeting summary. Oh, for fuck’s sake. 00:33:43.70 [Moe Kiss]: I actually stayed very quiet on that one because this is an area I think Tim generally is normally right in. Wait, why is everybody shaking their head now? But I do sometimes and I’m obviously in-house, so it’s quite different. I do find if I have a stakeholder like that, I will almost always have one-on-one time with them. And I’ll normally have some questions around like, what would have to be true for us to use this data source 00:34:10.90 [Michael Helbling]: or, you know, are there other data sources 00:34:14.34 [Moe Kiss]: that we could use to supplement the information so you’d be comfortable enough making a decision with what we have? Like kind of trying to tackle it almost one-on-one, because especially soon as you get in a meeting with a bunch of people and everyone’s like, oh, well, this data’s wrong. So, you know, we’re all stuck here and then everyone whinges about it for the rest of the meeting. It like it stops being productive. And so I would almost be trying to like really partner with that, like the biggest doubter of the group especially 00:34:35.54 [Michael Helbling]: and build up that relationship and really work with them on 00:34:39.54 [Moe Kiss]: some of the methods that Tim and Val are talking about here so that then they can also become your advocate, hopefully, over time. 00:34:46.78 [Michael Helbling]: Excellent. All right. Here’s another question we got. Everyone at my company is being tasked with showing specific efficiency improvements they’ve delivered using AI. I’m an analyst who supports marketing. What are some ideas you have that I could do for that? Don’t make me do the Hollywood Squares one again. 00:35:14.78 [Tim Wilson]: I just feel like this. I’m starting to feel like we’ve beaten this particular horse. 00:35:21.86 [Michael Helbling]: What do you mean, AI or efficiencies? 00:35:24.62 [Tim Wilson]: Well, well, yeah, I mean, the AI piece and the big, I guess the my my qualm with the efficiencies and I totally recognize the person asking the question. They don’t have control over that. That’s being pushed down and it’s an organizational challenge. But efficiency is like producing more with the same or producing more with the less or producing the same with the less whatever. And it’s like more what and and moving down the path of saying, well, we’re we’re producing more dashboards faster. We’re responding to requests faster and everybody feels resource constrained and like we can’t hire we can’t double our headcount. So AI is going to help us keep it fixed. And I think this is where I’m feeling like I’m beating a dead horse and I am the dead horse. I don’t know that that has this idea that if we its volume, volume is the issue that we just need to generate more. 00:36:24.74 [Michael Helbling]: But that volume of whatever we’re producing is going to someone. 00:36:30.38 [Tim Wilson]: Like there’s there’s value in the friction, which does not make me an AI skeptic. I just think the efficiency part is really, really tricky. You know, I think I’ve seen that in articles that that’s happening across a lot of companies are saying, we’re just trying to make this as a actually Jim talking on, I think day one was showing like this is just a chase for headcount reduction and something’s not right there. 00:36:57.26 [Michael Helbling]: So you you managed to answer that without giving this poor person any tips on how to do their job. 00:37:03.30 [Val Kroll]: They should have been a marketing analytics woman. Yeah, that’s right. 00:37:06.06 [Michael Helbling]: There’s a lot of tips there. 00:37:07.18 [Moe Kiss]: Can I jump in, though? 00:37:08.02 [Michael Helbling]: He helps. 00:37:10.26 [Moe Kiss]: I learned this recently and I’m still kind of reconciling it. So I’m going to obviously tell a bunch of people the exact advice I got. And we can all try it out, report back to me. Uh, I got some advice recently, essentially, like sometimes you just suck it up and you do it and a lot of conversations around AI at the moment are about productivity gains, not quality gains. And I find that just it gives me the ick. However, there comes a point where sometimes you’re in a position and you just suck it up and you do it and you go, oh, my God, my team saved five hours a week. Here’s all the dot points, send it up to leadership, move on with your life. And then you get your team together and say, OK, let’s have a conversation about how we improve the quality of our work. That’s what matters here. So sometimes the signal you send up doesn’t have to be the same as the signal you send down, but you better be smart about how you do it and don’t get caught. 00:37:59.30 [Tim Wilson]: But you’re setting yourself up to actually send a better signal up down the road, right? So I love that for doing both. 00:38:04.54 [Moe Kiss]: Yes, obviously, team that was the grand plan. 00:38:06.74 [Tim Wilson]: Check the box, but then separately say, but here’s the real value we got. And yeah, yeah. 00:38:11.74 [Michael Helbling]: And we’ve been battling a problem like this since forever. I mean, there used to be a time in our industry when we thought if we collected every single piece of data, we would somehow magically know more. And it sort of is a redux of a similar way of thinking. And so we have to kind of manage through it effectively. All right, we’ve got another question, and this one comes from also 00:38:34.90 [Michael Helbling]: someone in the audience, Sam Burge. 00:38:39.18 [Tim Wilson]: I think Michael surprised himself and didn’t realize he should have already been on the move. 00:38:42.34 [Michael Helbling]: So we’ll fix that in post. 00:38:48.00 [Sam Burge] How do you think analytics teams will look differently from today with AI? 00:38:54.98 [Tim Wilson]: In the future. I mean, I I hope that on the one hand, they are still spark, curious, business thinking, technical kind of have a broad set of skills. So I don’t think the teams will necessarily look a whole lot different. 00:39:16.54 [Michael Helbling]: How they work, they’re obviously going to get, I guess, efficiencies 00:39:21.90 [Tim Wilson]: and changing ways of working and become more prepared. 00:39:25.98 [Michael Helbling]: But that’s a tough one. 00:39:28.22 [Tim Wilson]: Why did I jump in and answer that? 00:39:29.54 [Michael Helbling]: I don’t think I had a good I was just buying time for one of you guys 00:39:32.38 [Tim Wilson]: to say something smart. 00:39:34.18 [Moe Kiss]: I don’t know about. OK, this is my guesstimate. 00:39:37.66 [Michael Helbling]: Yeah. And this is what I’m observing within my own team. 00:39:43.58 [Moe Kiss]: It seems like folks are kind of splitting a little bit. There are the folks that are going much more technical. I would almost, to some degree, even a little bit more specialized. And then there are the folks that it feels like the chasm between the technical and more the like business facing generalist is getting a little bit wider. I don’t necessarily see that as a bad thing. I think when folks have a particular strength in one direction, 00:40:06.02 [Michael Helbling]: like we should encourage that. 00:40:07.54 [Moe Kiss]: And that’s a great thing. I do think it makes it harder for like teams and how they work together and all of that sort of stuff. I had a massive rant a little earlier today because I am sick of reading very shitty, long documents which were based on someone’s shower thought that never should have left the shower, but now is in a four thousand word document and being flung around our organization. And then someone else comes in and writes 50 comments on it. And then someone’s like, over to you now, Moee. And I’m like, what do you want me to do with this? I am now doing all of the thinking work of having to read it. And it’s pretty like watery garbage, having to like respond, also trying to figure out what you want me to do with this because it was a shower thought bubble. And what my hope of where we get to is that it actually helps us think more critically, not less. I think we’re in the shitty stage right now. I’m optimistic. I’m going to bitch about the shitty stage we’re in right now because it is shit, reading all these documents. But I am optimistic that with time it will help us think better about what we do. Like, I don’t know, you’re creating a hypothesis, like instead of having to tap your co-worker, you can like sense check, have I got all the key components that I need to have a really strong hypothesis for this experiment? 00:41:27.82 [Tim Wilson]: No, I think even like knowledge management, the ability because AI is helping so much with unstructured data, the calling through what has happened in the past. But it feels like it is an elevated role for what a great analyst should be doing five years ago is still the same, which is being deeply embedded in the business context and the business needs. It’s been historically very hard to get the historical what have we done. So I think some of those like the preparation, the pulling this together, using the tools, but I don’t think it should change from what I think great analysts are doing, which is still having a deep connection to the business. It’s a shifting kind of tool set. 00:42:13.26 [Moe Kiss]: Can I just also add, Sam, one of the things I actually loved about your presentation was the idea also that we can change the format very quickly to suit different types of people. Like I am an audio person. I would absolutely listen to a podcast on business metrics. 00:42:28.58 [Tim Wilson]: I just remembered at the second point I was going to make back on that question. 00:42:31.22 [Michael Helbling]: All right, fine. 00:42:32.66 [Moe Kiss]: Was this like from 10 minutes ago or? It was, but it was on that. 00:42:36.02 [Tim Wilson]: It was asking, trying to figure out the best way that’s what somebody would, how to respond to them. So when you said the audio person, how often do people actually know themselves? So for all the listeners or anyone who wasn’t in Sam’s session, it was the, you know, there’s the marketing person who says, I just want to have my cup of coffee and listen to the podcast. And there’s a little trigger in me that thinks sometimes we seldom really know ourselves. So differentiating between somebody who thinks that’s what they would want 00:43:07.38 [Michael Helbling]: and someone who actually that would be effective. 00:43:13.02 [Tim Wilson]: And that had rang true from what we’ve dealt with for a hundred years. We’ve had people saying, I just need a dashboard that does X and we deliver them the exact dashboard. And they’re like, this isn’t helpful. Where’s this other thing? So there’s this other layer that I think analysts historically have needed to follow. And the same thing opening up all these different formats is great. But what somebody says would work. And I think you have to deliver it to them. But figuring out like, does that actually work? And giving them the opening, if they said, oh, I thought that would be really cool. And you tweaked and tuned the tone and the content and everything, but giving them the out to say, you know what that actually didn’t work. I don’t know that I wouldn’t have known it wasn’t gonna work until I actually tried it for a while, which we have not done in the industry very well forever. We get in sort of a whiny mode of saying, we’ve given them all these dashboards and they’re not using them. And it becomes this adversarial thing because, and then if we ask them, don’t you want these dashboards? Well, they ask for them. Of course they’re gonna say, yeah, yeah, yeah, this is really useful. We haven’t figured out how to say, no, that mechanism didn’t work and it’s okay. And we need to have the trust and we need to try something different. So we can go back and that was my other point. Maybe we should let, I’ll get a word in and twice. 00:44:35.90 [Val Kroll]: Back to how do we think teams will change? I think that there’s gonna be some new muscles that are built. I think one of the things that we had been talking about this conference is how this has given us a lot of energy and excitement. And I think that there’s been some creativity injected, which I think is just fun. Even if we’re just talking about little things that we do on the side that’s not ready for prod, but it’s just kind of like stretching us in some new ways, which I really appreciate. The other thing that we were also talking with you about, Sam or you and I were chatting about, 00:45:03.38 [Michael Helbling]: is the communication skills 00:45:06.74 [Val Kroll]: and how that’s gonna be improving. Because I think how often have you been in a conversation, even over the past couple of days, perhaps where people are just ragging on their stakeholders, like, oh, they’re so dumb, they just don’t get it, right? And it’s like, when you’re prompting and you’re talking about something you need the AI to do for you and it totally misses the boat, and you’re like, oh, geez, I’ve totally forgot to give you this piece of context. Of course you didn’t understand what I was trying to say. Think about your poor stakeholder that you were not giving that context for how many years. The video of, we were talking about this, the dad with his son and daughter about making the peanut butter and jelly sandwich, about like, take the bread out of the bag and he’s like trying to, and he like ends up like putting the knife through the whole loaf of bread, whatever, because he was just trying to follow the directions. But I think it can help us build a little empathy for our stakeholders, because like, they’re not wired the same way we are, they don’t have the same background that we do. And so I think it’s one of the byproducts that will help us become better communicators and hopefully have a little bit more empathy for people who don’t make all the immediate connections 00:46:06.82 [Michael Helbling]: that our brains do just because we’re nerds. 00:46:10.50 [Michael Helbling]: I think organizationally, I think we’ll see departments flatten out a little bit. Like over the last 15 years, we’ve become super specialized in a lot of different disciplines, because analytics is actually multidisciplinary. And I think AI will push that back together a little in a lot of organizations. And then those organizations over time will start to realize there’s still a need for some of that specialization on the fringes, and they’ll find ways to bring it back in. But I think we’ll all express experience, some compression where an analyst will go back to being able to code something and also write a data pipeline and also go access the data lake and also build a dashboard and a great visualization. And those were all things that like analytics people were attempting to do 15, 17 years ago. And then we realized we needed specialization. We needed a data engineer. We needed analytics engineer. We needed a data visualization expert. And I think we’ll begin to flatten those out with AI. 00:47:13.50 [Moe Kiss]: That doesn’t worry me a bit though. 00:47:15.54 [Michael Helbling]: I don’t say it’s good or bad. I just think that’s what will happen. 00:47:18.94 [Moe Kiss]: Like I feel like sometimes folks are over indexing on the generalization at the moment and thinking that, I don’t know, a product manager can do a data scientist job, and I mean, some product managers is doing engineering jobs. Also interesting choices. So I think we’re over indexing on the fact that folks can generalize. And I heard there’s a like- 00:47:39.06 [Michael Helbling]: Outcomes follow expertise, even with AI. Okay, we have time for one more question. And that’s our last question. And we have someone who is an audience member who’s going to ask it and it’s Jim Stern. 00:47:52.54 [Michael Helbling]: My question is, what are the first three skills 00:47:58.46 [Jim Sterne]: that analysts have mastered that are going to be successfully taken over by artificial intelligence? 00:48:08.18 [Val Kroll]: I wish we could get a question about AI. No shade, Jim. Just looking at Michael. First three skills, say it again. First three skills. Sorry, I was being an asshole. First three skills. 00:48:24.66 [Michael Helbling]: What are Tim’s favorite skills? So, probably like the, what is it? The ink to data ratio. So that probably going to be the first thing AI takes over. 00:48:37.94 [Val Kroll]: Data pixel ratio. Data pixel ratio. 00:48:39.98 [Michael Helbling]: I was paying attention, Tim. SQL, I don’t write SQL anymore. 00:48:46.82 [Tim Wilson]: I am so much more on the debugging SQL debugging R. I think debugging coming over really, really quickly. I think a second one would be QA’ing or validating or vetting the results of an analysis, having that the thing that you’re supposed to go to another analyst or try to come at it a separate way. I’m not sure what the third one is. I think it’s a lot of things that are going to be supplemental that we should be doing. Like not doing the QA, but giving me the list of, check my logic. Check the things that I, so maybe it’s not what the junior analyst is doing. I think it’s what the junior analyst ideally is working with another junior analyst or senior analyst to look over and review. There’s not a whole lot that I see a whole hard, whole hog hand in the keys over on. 00:49:42.94 [Michael Helbling]: I came up with three, but I don’t know if they fit the criteria. We’ll have to go from there, but that’s going to be where we have to wrap up. And I want to say first, a huge thank you. To Jim Stern for organizing the marketing analytics summit. 45 years. And save your applause because also to all of you for being here and bringing your energy and your questions, your insights and experiences in a world changing daily with AI, it’s the people, the community and human connection in our industry. It feels all that much more special and crucial in these changing times. And obviously there’s a huge audience also listening and we’d love to hear from you too. And you can reach us at our LinkedIn page or the measure slack chat group or by email at contact at analyticshour.io. Please leave comments, ratings and reviews on whatever platform you use to listen. We do read all of them. And Tim brings them up in meetings. 00:50:48.46 [Michael Helbling]: And I think- 00:50:50.70 [Moe Kiss]: It makes us set KPIs. 00:50:52.62 [Michael Helbling]: Yeah, it’s terrible. 00:50:55.18 [Michael Helbling]: I can’t wait till AI replaces that. All right, and I know that I speak for all of my co-hosts, Val, Tim, Moe. When I say, no matter the question or challenge 00:51:06.06 [Michael Helbling]: you’re currently solving, keep analyzing. 00:51:09.30 [Announcer]: Thanks for listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at analyticshour, on the web at analyticshour.io, our LinkedIn group and the measure chat slack group. Music for the podcast by Josh Crowhurst. 00:51:26.98 [Charles Barkley]: Those smart guys want to fit in. So they made up a term called analytics. Analytics don’t work. Do the analytics say go for it no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 00:51:46.18 [Jim Sterne]: Ladies and gentlemen, that is so much fun. Thank you so much for be adding the spark to the end of the marketing analytics summit. You guys are awesome. 00:52:02.62 [Tim Wilson]: Rock flag and AI gives me the ick. 00:52:06.42 [Jenn Kunz]: Yeah. 00:52:07.26 [Michael Helbling]: Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. The post #298: Listener Questions Answered Live from Marketing Analytics Summit! appeared first on The Analytics Power Hour: Data and Analytics Podcast .

May 12, 20261 hr 6 min

#297: Durable Wisdom in an Age of AI Slop

What do colors, soup kitchens, and mountain climbing have in common? They’re all part of the mental models that have shaped how we think about analytics, and they’re exactly the kind of durable wisdom that matters more than ever in an age of AI slop. This campfire-style conversation among the co-hosts reveals the concepts, books, and aha moments that have stuck with us across decades of analytics work. From the magic of randomization to the critical distinction between outputs and outcomes, we share the frameworks that guide our thinking whether we’re writing SQL by hand or asking Claude to do it for us. It turns out the most valuable analytics wisdom isn’t about tools or techniques—it’s about understanding how humans actually make decisions, build trust, and collaborate effectively. Some things never go out of style. This episode is brought to you by Prism from Ask-Y —your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show (Shiny App) The Magic of Randomization Illustrated with Color Outputs vs. Outcomes: this is a good resource/explanation of the core idea, but Val also addressed the concept in this Medium post , and Tim digs into it in Chapter 5 of his book (available in print, ebook, and audiobook formats!) (Book) A/B Testing: The Most Powerful Way to Turn Clicks Into Customers by Dan Siroker and Pete Koomen. And the diagram from that book that hit on local vs. global maxima through the lens of “Refinement” vs. “Exploration” (Article) Addition bias (Ingvar Kamprad / IKEA) (Book) Information Dashboard Design: Displaying Data for At-a-Glance Monitoring by Stephen Few (Book) Storytelling with Data: A Data Visualization Guide for Business Professionals by Cole Nussbaumer Knaflic (Article) Chasing Statistical Ghosts in Experimentation (Book) Field Experiments: Design, Analysis, and Interpretation by Alan S. Gerber and Donald P. Green (Book) First, Break All the Rules: What the World’s Greatest Managers Do Differently by Marcus Buckingham and the Gallup Organization (Book) Switch: How to Change Things When Change Is Hard by Chip Heath and Dan Heath (Podcast) Choiceology with Katy Milkman (Book) The Science of Storytelling: Why Stories Make Us Human and How to Tell Them Better by Will Storr (Podcast Episode) #240: Asking Better Questions with Taylor Buonocore Guthrie Photo by Joris Voeten on Unsplash Episode Transcript 00:00:00.00 [Announcer]: Welcome to the Analytics Power Hour. 00:00:08.92 [Announcer]: Analytics topics covered conversationally and sometimes with explicit language. 00:00:15.80 [Michael Helbling]: Hi everyone, welcome. It’s the Analytics Power Hour, and this is episode 297. You know, at this point, I think we’ve all been hit at some kind of output that is obviously AI generated. I mean, even it has its own term, AI slop. Oddly enough, most people don’t appreciate being hit at AI output without it being thought through by a real person, and it got us thinking. It’s not that what AI produces is always bad, but there is starting to be different categories of content based on whether AI produced it or not. And maybe there is something to be said for analytics wisdom that existed way before AI, and that will keep being true no matter how AI is involved out into the future. So we reach back to the formative moments of our own careers to share some of the hard one wisdom from our careers in analytics. Maybe this episode will lack a little AI, but I think it’ll still be worthwhile. So let’s introduce the people who make up the show. Hey, Val Crowell. 00:01:16.92 [Julie Hoyer]: Hey, Michael. 00:01:18.12 [Michael Helbling]: I’m so glad you’re here. And we’ve got Tim Wilson. Howdy, howdy. And I still put up with you somehow, so that’s good. 00:01:28.52 [Julie Hoyer]: Okay. 00:01:29.52 [Michael Helbling]: I had Moee kiss. How you going? 00:01:33.04 [Moe Kiss]: How you going? Oh, I love it. 00:01:35.04 [Michael Helbling]: Thanks. Yep. And Julie Hoyer. 00:01:38.04 [Julie Hoyer]: Welcome. 00:01:39.04 [Michael Helbling]: Hey there. And I’m Michael Helbling. So, yeah, we’ve got the whole team together. We’ve all seen so much in our careers pre-AI. And so what’s some wisdom that exists that’s going to be useful, whether AI is involved or not? Who wants to kick us off with something they’ve learned over their career? 00:01:55.12 [Julie Hoyer]: Oh, I will. 00:01:56.64 [Julie Hoyer]: Because this one, I feel like sticks with me still to this day. 00:02:01.24 [Julie Hoyer]: And this was more than over a year ago. This was three, four years ago. But it was actually back when me and Tim were working together. And I’ll never forget, we were talking about randomization and trying to figure out the best way to represent it, different ways to think about it. I think this was right around Tim when we were trying to do a talk around blocking, randomization with blocking and things. 00:02:29.84 [Julie Hoyer]: And so we could not figure out how to make it friendly to people, people who had maybe 00:02:35.88 [Julie Hoyer]: not thought about it as in-depth as we were at that moment. We were way too in the weeds. And I remember slacking Tim like the next morning, like right at the beginning of work and saying… So I had a thought. 00:02:49.68 [Tim Wilson]: Wait, it was the next morning because it was a glass of wine sitting on your couch, thought, I believe. 00:02:55.24 [Julie Hoyer]: Yes. Yeah, I was like, maybe I even slacked you that night and was like, look at this in the morning. 00:02:59.40 [Julie Hoyer]: But I was like, I just was having a glass of wine and it came to me. 00:03:02.76 [Julie Hoyer]: I was like, I feel like a good way to explain randomization would be through colors. And so I tell them how, what if each color represented a characteristic and when you randomize the population, you could see how the colors were split between test and variation and what if those colors were blended to show a group color. And then you could see that the color comes out really close because it’s randomized across the characterization. And so Tim, I nearly couldn’t resist and that’s why I told him. So I was like, that’s about as far as I can take it. You made a whole shiny app to let you choose the size of your sample, to choose how many characteristics, to choose the predictiveness of those characteristics on your outcome, the mean, all these things. And then you could actually run a simulation and see the colors blended. And then you could flip on and off blocking. And there have been multiple times, even earlier this year, when I’m thinking about certain things with randomization, I’ll still go back to that shiny app and play with it. And it helps me a lot with talk tracks or simplifying it or reminding myself of different characteristics of it. So that’s one of my favorites. 00:04:18.32 [Tim Wilson]: I like that one too because we got a little carried away because it was like, well, this work, I think it was built very quickly. And then Julie was like, what about if we, you think we could do this? What if we did a, and I was like, what if we did, and so it got a little involved. 00:04:33.96 [Julie Hoyer]: My hands never touched the keyboard. That was the best part. I got to the whole thing. 00:04:39.96 [Moe Kiss]: It was like voice control. Why do you think it’s stuck with you so much as like the antithesis of like the AI flop world? 00:04:49.56 [Julie Hoyer]: Like what was the, I don’t know, the bit that just made it really resonate or it keeps you 00:04:55.92 [Moe Kiss]: coming back. 00:04:56.92 [Julie Hoyer]: I think it’s the visualization of it. I don’t know why the, the color is what stuck with me. And sometimes like a good visual is so much easier for me to go back to. And I think the AI part, this is, I swear I’m going to loop back to your answer. But I even remember back in college starting in engineering, they just wanted to give you like an output to use. They’d say, just use this output, use this output, like a formula. And I, and I’m, no, I’m not alone this way. I always needed to understand why. So that if I forgot the perfect formula, I could like reason my way back to my understanding. And there’s something about that shiny app and the visualization of a color that just like, dang, it like hit something in my brain that when things get fuzzy and I haven’t talked about those things in a while, I haven’t thought about those things in depth. Like it kind of brings back some of that understanding. Like I can have a good starting point and like think through it again. And the AI part is like, again, they just give you output and that output is actually like the variation of an AI, you know, agent or whatever, like giving you something. It’s never exactly the same. So I like that it is steady and a starting point for me. 00:06:10.24 [Tim Wilson]: I love that. The reason I got so excited, I think it is this color piece because I will chime in that when we talk about random assignment and kind of the power of random assignment and we’d say, oh, so if you have, you know, a thousand people and you randomly split them, you’re going to have a roughly the same number of men and women in each group and roughly the same number of household income and like that idea that you’re making two groups that are effectively the same, like it’s just, it’s abstracted when you talk about all the characteristics of what’s in it. And I think when thought about it, like, I mean, it literally is kind of a color mixer that it just sort of generates a palette and then you see the average color and it may be like this lime green and it’s slightly different shades, but it’s like a simple math thing. So to me, like it just goes into my head of saying, this is what’s happening while we’re making two groups that are pretty damn close to the same. But the fact that, I mean, that was to me kind of the really useful part of that is I was trying to grapple with how to actually internalize this. And then it was very easy to say, okay, now that extrapolates to these other more nebulous characteristics of psychographic details and demographic details. 00:07:27.20 [Moe Kiss]: But it sounds like it’s solving the problem of, to quote a Canva value, making complex things simple, right? Like it’s about understandability. It’s about a way to see something so that it like clicks in someone’s brain. And it’s funny, I had someone in my team that was doing some work the other day and I went through it afterwards. So like we had a couple of different strategies and I was kind of like, we need to go through these strategies. We really need to make sure that they’re being based on the data points that we have available. And like that easily is a task, for example, that probably my instinct would have been, 00:08:04.36 [Julie Hoyer]: I’ll put it in AI and see what’s like, here are the data points, like what’s missing. 00:08:09.24 [Moe Kiss]: And this person went through it so rigorously. And the reason that the work like came back and I was like, oh, I get this, this is high quality work. And like I can really understand it is because it’s also that falling, it made it click in my brain of just like, not here are the differences, but here’s the assumption I made about why I think this is different. And here is the like, the leap that’s been made here. And I think it’s probably because of this. And I think it’s, I don’t know, when I just keep coming back to this point about like quality, it’s like when you can see the quality and it, someone finds a way to put it into a narrative that then clicks in your brain. Like that’s where I feel like the gold is right now. And it doesn’t feel like there’s a lot of it. 00:08:55.32 [Tim Wilson]: That does kind of make me want to go with one of mine because I’m seeing a direct link from that, which is also kind of just a concept or something that I find myself thinking about and talking about with clients and business partners a lot. And even with analysts is the distinction between outcomes and outputs and how we, as a, we tend to pick metrics that are outputs when we really care about our outcomes. And I traced this back 20 years at this point when I was on a United Way committee in Austin. And there was this retired social worker who we were, we were reviewing programs for funding. And so we had a lot of proposals coming in and we were in these series of committee meetings and we’d all have to like read, I don’t know, 10 proposals and then we’d meet about them. And he kept having this kind of consistent bit of feedback on multiple programs where he would say, well, these are like, you always have to say how you’re going to measure the program. And he would say, oh, well, these are, these are like output metrics and we really want outcomes. And I was 10 years into my analytics career at that point. And I was like, what are you talking about? And it slowly, he started to explain and the example, he had various ones because he’d point to them in specific programs. So the one that I come back to was talking about like food pantries and how they like, they would count like the number of meals served or like a soup kitchen number of meals served. And he was like, yeah, they can just like count the number of trays that go through the line. And that’s the number of meals served. And we know that’s good. You’re serving meals. He’s like, but really what we’re trying to do is reduce food insecurity. We’re trying to keep people from going hungry, which is related. You want your outputs to lead to an outcome. But he was like, we really want to push them to say, can they get more to an outcome oriented measure? And I’ve taken that over the course of that work, I was like, this is like profound for what I’m doing in my day to day with my colleagues at work and trying to get them to think about, this is why a click doesn’t matter. This is why the click through rate, I mean, doesn’t not matter. It’s just those are outputs and trying to guide discussions early on into outcome oriented metrics and business outcomes and then going from there and say, how close can we get to measuring that? And unfortunately, I think the guy’s name was, I am 99% sure, unless I’ve been fooling myself for years. His name was Pat Craig. He was a retired social worker. I have every four or five years, I go try to find him because I come back to that again and again and again. But it was another one that made it very tangible because it was really in the real world. People who are in need talking about outputs versus outcomes really solidified it, made it very tangible, but it then applies in the more, really, we’re not curing cancer here, we’re talking about marketing stuff, but the concept still applies. 00:12:00.60 [Julie Hoyer]: That’s a good one. I love that one. I use it all the time. Do you know how many people I’ve said that thinking, oh, they’ll know about this? And they write it down, they’re like, oh, that’s good. So I’m like, oh, thanks, Tim. 00:12:11.16 [Tim Wilson]: Well, I mean, there are times where I feel silly asking, I’m like, do we, are we, I’m sure some of you are familiar with this because I’ve thought about it for so long. To me, it’s one of those like when the light bulb goes on, you can’t stop seeing it. 00:12:29.08 [Moe Kiss]: Tim, the light bulb has just gone on for me. Just to be clear, I messaged you. It’s like a message because I was like, maybe I shouldn’t derail the whole show. But I think I’m thinking about it as like input and 00:12:43.16 [Julie Hoyer]: output. Yeah, but here we go. But anyways, or we could. No, but I’ve been thinking about it as like 00:12:50.76 [Moe Kiss]: input and output metrics. And I say, like in my mind, an output metric was an outcome. But like, I just feel like that framing is so much better. And it truly is clicking in my brain for the first time. And I’m, I’m sure I’ve heard you say this before, but sometimes like someone just said something a slightly different way, or your mind is at the right time to absorb it. Anyway, thank you. This is going to be very helpful. You’re welcome. 00:13:15.80 [Tim Wilson]: And I’ll also say this makes it sound like it’s binary. The older I get, the more I’m like, it’s not, I say like, you want to be skewed towards outcomes. You can have debates about, you know, is a monthly active user, is that an output or an outcome? And it’s kind of, it’s context dependent. And it’s not, but if you can ground it in that overall, overall pure version, I found it very, 00:13:38.60 [Val Kroll]: very good. You’re just getting soft, Tim. It’s binary. There’s no gray. 00:13:43.16 [Tim Wilson]: That’s right. You said the rules have reversed. I never ever seriously about, wow, Tim, you’re just mellowing out waiting for that to happen. 00:13:54.60 [Julie Hoyer]: So chill. Hey, Tim, have you tried out vibe analyzing yet? 00:14:03.96 [Tim Wilson]: Oh, that phrase hurts me deeply. But I did download my Strava archive to just try to like analyze my workouts. And it turned out to be like 28 different CSVs with everywhere from like two to 103 columns each. Oh, sounds messy. What’d you do? Well, I mean, for giggles, I tried Prism by Ask Why. That’s ask-the-letter why. I uploaded like all 28 files and just started asking questions through their chat interface in plain English. Started off by asking like, how many miles do I run each month? And it worked? I mean, it did. It initially it gave me results pretty quickly. It was, it was like really fast. It actually wasn’t, wasn’t perfect, but that really wasn’t a prism issue. It turned out that Strava data just refers to distance. Like that’s what the column is labeled and that distance is in kilometers. So it took a few iterations, still a little bit of the human analyst to say, wait a minute, those, I wish I was running that far, took a few iterations with the platform to get that figured out. But once we did it, it handled that conversion, not only on that query, but like automatically going forward with future queries. Oh, that’s actually pretty cool. 00:15:21.56 [Michael Helbling]: It’s just like how we want to handle mishmashes of different sources and medium names in a consistent way whenever we’re looking at working with like digital data. Exactly. And I also got to 00:15:32.52 [Tim Wilson]: kind of check out their Mingus query language, which it’s like a more readable form of SQL with the actual SQL just like one click away. And I even like built some quick visuals and some quick reports. It’s also got like a local mode for keeping all the data on your machine. I actually haven’t tried that out yet. I just did the cloud version, but it’s pretty nice feature. 00:15:54.20 [Michael Helbling]: Nice. So yeah, it sounds like something worth checking out. You can head over to ask-y.ai, join the prison beta waitlist and use the promo code APH when you sign up, and that’ll move you up to the top of the list. We can guarantee that you’ll get access faster than Tim finishes his next 10k probably. All right, back to the show. Okay, so I want to share one of mine. 00:16:23.56 [Val Kroll]: And I’ve worked with most of you and if you’re listening, I’ve worked with you as a co-worker, as my client, you will know this one of mine. In the experimentation realm and world, one of the concepts that lots of programs like to think about or keep in the back of their mind is the difference between a local and global maxima. And like the tried and true, you know, the analogy or the visual is like you climb to the top of the mountain. And now that you’re through those clouds, you see that there’s actually a secondary peak to climb. And I get that, that works. But there’s this visual that actually came from the book A.B. Testing. It was the optimizely book at the time, Dan Saroker, the CEO and Pete Kuhlman, who led their statistical arm and function. In there, there’s this visual that has, and I’ll try my best to describe it. There’s two sides 00:17:20.44 [Tim Wilson]: that are being compared. Hold it up longer. That could wind up as a YouTube short, you know, so this will drive people to the YouTube channel. Don’t you want to click through the full episode 00:17:30.68 [Val Kroll]: now? So on the left hand side, they’re describing refinement and that there’s kind of like this cone shape where like the squiggly is getting closer and closer to the star. But there’s actually a star to the side of it. And it says that that was the best solution and it was missed because they were kind of refining to this point. Whereas exploration is kind of this point that branches out and has like lots of arms to it. And so they did find the optimal solution. And the arrow is saying like, here’s where you refine from. And what I like about this image so much more than the local versus global maxima is because it gives you the visual of the consequence of not thinking big first or to not think about exploration or innovative type of thinking or testing first and going straight into like, how do we refine the micro copy on this page? And it’s like, well, was that the right page to send someone to in the first place? And so it’s about like staying curious at like a higher level because it’s not just, you know, like we’ll find these like little wins as we go, which is great, but they’re empty calories if you’re kind of missing the optimal solution. And so that visual, I bet you could find it, I have pasted that in no fewer than 50 presentations in my life. I’m quite confident because I think it’s just a really nice way to like cement the point of that concept home. 00:18:55.00 [Moe Kiss]: Well, it’s going in one of my presentations. It’s amazing. 00:19:00.28 [Val Kroll]: I like it. It’s good. 00:19:02.12 [Tim Wilson]: But is this is part of this that like that just the realities of corporate life is that it is much easier to get into is to be in the lane that you’re in and kind of refine like, oh, because to do the exploration feels riskier and it often means you’re kind of reaching more broadly with ideas. So like just like business culture drives us to say, let’s do little tweaks and refinement and and it like organize like how do organizations do organizations see this and say, you’re right, we need to push ourselves to think more broadly, take bigger or broader or more exploratory swings. 00:19:54.04 [Val Kroll]: Yeah, I totally, I mean, how many times have you been like, we’ll give a marketing context like, well, when we ran this campaign last year, we only had two versions this creative. So this year we’re going to have three. And so it’s like the smaller like, you know, I’m obviously being reductive in that example, but it’s like, based on what we did last year, we all remember that here’s one new thing we’re going to do different. And that’s like the optimization or like the refinement versus like, instead of just going direct to patients, what if we had a strategy for healthcare providers? And so like that would be the bigger swing. But to your point, Tim, I couldn’t agree more because it’s like, there’s no incentive for that because that’s more work, you know, more approvals, perhaps, you know, more budget overhead, things like that. And so I think people who are really excited about the outcomes of what that’s trying to do, and go back to yours, that those are the folks who really kind of like thrive in finding those. And you’ll notice that those are the people that lots of other people really like to work with inside of organizations, I will say, because they’re doing more exciting things in service of, you know, shared goals for the organization. But yeah, I think it’s not, it’s not the natural path, I agree. No, people want what they can control. Like in their in their lane, like you 00:21:11.64 [Julie Hoyer]: were saying, it’s hard to like look up and do that broader view, and then have to collaborate. But I think then what you said, Val, tying it back to outcomes, like, if people realize the shared outcomes, they were more focused on driving instead of their individual lane outputs, like maybe people would be more open to doing that instead of just the refinement. 00:21:30.68 [Moe Kiss]: Yeah, I was just gonna say, I don’t think the shared outcomes are always incentivized, like sometimes they are and sometimes they aren’t. But I think the one kind of the one push I would have on this framework as like, and I am a huge fan, I’m definitely definitely going to be borrowing this. I think there’s an assumption that you’ll always get to the star, like the point of refinement through exploration, and sometimes you don’t. Sometimes you explore, you do all the extra work, and it doesn’t add a significant amount of value. And I think, generally, those cases are pretty rare. But I think it does happen sometimes. They’re like, again, not binary. 00:22:10.44 [Tim Wilson]: Without any specifics, obviously. But I think of Canva as a company, like on a product level, very exploratory, like every time you turn around, they’re like, oh, yeah. And now, you know, it can make your hopes for you in the morning. So it seems like, I mean, like literally, I mean, it feels like, I sort of get updates through conversations we’re having. You’re like, well, yeah, can we get to them? Like, okay, there’s three other products that, what the hell? 00:22:40.92 [Julie Hoyer]: So it feels like that. Have there been, has Canva had like, pursuit of like specific, 00:22:48.20 [Tim Wilson]: like this is a whole new area that has gone nowhere and been shut off? Again, not asking for 00:22:55.40 [Moe Kiss]: any specifics? Yeah, I think so. I think the tension right now, though, is that we all need to lean more towards that exploration piece because of just the pace of AI products and features and how they’re shipping. Like, I think what particularly is a trap right now is if you’re in that retirement, we want to go towards this goal, we want to build this thing. Like, in today’s climate, that’s more dangerous than ever, I would say, because just the way things are changing 00:23:26.28 [Julie Hoyer]: so quickly. So I would say not necessarily like things getting completely like abandoned, 00:23:33.72 [Moe Kiss]: but more things getting refined and changed along the way. 00:23:37.96 [Tim Wilson]: I like that with AI. If you’re, if you do the kind of the MVP in multiple directions in a way that you can say, I’d rather try five wildly different things with AI in a minimal way with 00:23:53.08 [Julie Hoyer]: clarity on how I’m going to determine whether this is the best bet or not, than determine 00:24:00.52 [Tim Wilson]: we’re going to make the best chat experience using the latest LLMs ever and just like pursue that 00:24:08.28 [Michael Helbling]: and miss it. Well, I’ll share one that’s important to me. I don’t remember who first told me this, but it was about five years into my analytics career and somebody said to me, Michael, trust is hard to build and easy to break. And I think that’s more of a general statement, but applied to the world of analytics, I watched in my own career sort of people who believed when I presented an analysis and people who didn’t and losing trust with stakeholders was something I definitely experienced in the early years of my career and how much that put me in a position where I could no longer influence the business or business outcomes in certain areas. And so it really kind of hit home and I really kind of held onto that for, for the rest of my career was just sort of thinking about how do I continue to build trust when I’m working with business stakeholders when I’m talking about things. And I like, because it’s me, I don’t have any kind of format, formal like structure to that, but there are little signifiers I look for around like how do I know that trust is still there. And I use that to kind of guide how I, how I act around, you know, today my clients or stakeholders I’m working with of sort of like how is that sort of relationship which tells me then the influence I have as an analyst for that particular situation. So that one is one that has kind of always stuck with me because I love being influential. And for early in my career, I just figured if I showed you the data, it doesn’t matter who the messenger was, you would just say, okay, yeah, that’s the data and you would accept it. But the reality is, is the messenger matters quite a bit. And since I’m not Tim Wilson, you know, I had to 00:26:03.16 [Julie Hoyer]: like, you know, ramp up my skills. But this, this is like, I mean, honestly, like, 00:26:10.20 [Tim Wilson]: dear listener, if you’ll like that one, go back and listen to our last episode with Eric Friedman, because a lot of what comes up with that is that I think we think it’s, that means the data has to be perfect and the analysis has to be perfect. And I feel like what you model and what we talked about with him, a lot of it is like actually showing that you understand the environment they’re working, like you understand building trust has a lot more of the soft skill than my data is always perfect. Yeah, there’s that part of it, understanding 00:26:44.68 [Michael Helbling]: the context. And I think also not to use a bad word to you, Tim, but empathy has a lot to do with it 00:26:51.64 [Val Kroll]: as well. Don’t know what that does not compute. Yeah. No, because it’s like one of the signals 00:27:04.44 [Michael Helbling]: in like, if I’m working with other clients is if they come to me with a separate problem, that tells me I’m building trust because they are like, okay, yeah, you’re doing the project or whatever project. But if they come and say, Hey, here’s another thing that’s going on, you have any insider thoughts into this? That’s a great example to me of like, okay, we’re on a trust path together now. So that’s awesome. Let’s keep building that. So like, that’s, but yeah, you’re right. You’re absolutely right. It’s not just the data or the analysis. It’s also the context to show you understand show that you care about what they care about. And it’s challenging because as analysts, I feel like sometimes we want to not necessarily be front and center. And the reality is to influence decision making, you’ve got to be, you’ve got to be willing to kind of plant your foot and sort of be the face of the 00:27:54.04 [Moe Kiss]: data in a way. So the funny thing is, Michael, like you, you’ve been chatting about trust, and it actually makes me think, I don’t know, I know I’m obsessed with her big fan girl, but she talks about this so much. And like, she calls it work slow, which is enabled through AI. And I think like the way she articulates it is just so brilliant around like, and I feel like I’m seeing so much of this where like AI removes friction for shitty ideas, right? And so everyone just like, and I get it, I get it because I’m doing it too. Like I do, you can move faster, but it’s making us look like we’re productive, but actually we’re just producing more shitty ideas, right? And I think the bit that’s really challenging then is like being able to differentiate between the shitty ideas and the not shitty ideas, right? And so I think the thing that she really is like honing it in on, which has just been like flying through my mind. And it’s the same as the trust of the stakeholders, right? Is like AI can be this incredible tool to unlock a lot. But like, how do we really use it? How do we incentivize the quality over the velocity? Because at the moment, we’re really honing in on velocity, which is breaking that trust so deeply. And I think about it so much like, as a manager, every time you get a piece of work, as a person who’s producing work that is lower quality, like, I personally feel like you’re 00:29:22.28 [Julie Hoyer]: fragmenting trust with those around you, right? Well, but it’s, it’s, it’s got dual pressures, 00:29:27.24 [Tim Wilson]: you’ve got the pressures to use AI, that’s coming down from on high, use it, use it, use it, be efficient. When you’re delivering stuff and it’s polished and longer and grammatically correct and no typos and coherent and organized thoughts, but the person who’s getting it knows and then you’re under the gun to say, I can’t spend as much time whittling it down. Like they are competing pressures. The person who’s receiving it, I think you’re dead on, like it does, like you’re sending me like total AI slap, stupid sales pitches, but there was no trust there in the first place. It does feel like he gets sneakier when it’s with a coworker saying, oh, I, you know, here, here are the notes from the meeting. And you read through it and you’re like, this isn’t your voice and it’s a little off, but I can’t really criticize you because you did it quickly. But I also don’t feel like there’s a depth of thought. Like that’s a, but that’s the thing, right? Like I almost wish 00:30:25.32 [Moe Kiss]: there was a way to know if I have 10 docs in my pile from my reading list, which one wasn’t written by AI, because that’s the one I’m going to go and read, but there’s no way to know that, right? So then what ends up happening? Like I feel like we need to create a system that incentivizes the quality of thought and the depth and the fact that if you want something to be shorter or tighter, like you actually need, like you can’t just give it to AI. And I just, I feel like we’re 00:30:50.44 [Tim Wilson]: in this real conundrum. That can be a great idea. I’m going to, I’m going to vibe code an app to do that this weekend. That has collars. Yeah. Yeah. 00:31:01.08 [Michael Helbling]: Part of this, the thing about maintaining trust in, in this context, I think is about transparency as well. So like if you use AI, we’ll just lead with, Hey, this is AI generated. So just F, you know, so you know, or you say most of this is AI generated, but here’s my synthesis up top. So that way you can let people know the distinction. Like you don’t have to go read all this. You can, you know, it’s the same thing when you’re preparing an analysis and like you want to show all the cool things you did to the data, but you put it in the appendix because your stakeholders don’t care. They just want to know what the McKinsey title is and the, and the, the big insight. And if they trust you, they’re probably don’t need to dig much further. If they want to learn more or have more deeper interest, there’s sourcing material behind it. So a lot of times AI for me feels, feels like that where it’s like, okay, AI can pump out tons of content and, and honestly, beautiful content too. Like it’ll make a better slide than I make on average. A lot of times, you know, if I just sort of take content and plug it in, I don’t make the best slides. Okay. 00:32:07.16 [Moe Kiss]: Like I’m just being honest. I had AI make a slide deck for me the other day and it’s still the 00:32:13.72 [Michael Helbling]: work to do on the design side, I would say. No, no, no, I, I mostly don’t let it because even though it can make a pretty slide, it’s not, it doesn’t fit what I’m trying to do. So like, no, I haven’t yet been able to like really position a whole AI slide deck yet, but it’s 00:32:30.44 [Tim Wilson]: getting closer. Imagine, imagine if you just could just go on a walk with the dog and just talk, talk to the AI and then it’ll generate a deck for you. And it’s like, no, there’s, there’s value in the friction of me needing to go on the walk with the dog, think about it, stew over it, and then come down with what are the three things that I want to say. 00:32:50.44 [Moe Kiss]: Okay. I promise after this, I will get off my soapbox. I promise. But there’s an incredible, I mean, we all know I love the acquired podcast, but there’s one on IKEA, which is amazing. And Ingvar, I’m proud, I’m going to fuck up his name, I always do, but Ingvar is the guy who started IKEA, right? And he has, he had this principle, which I’ve just been thinking about, like, how do you implement this and how do you scale it? Basically, particularly as like people manager, right? Like we have addition bias. So like any time thing seem hard or tricky, we try and add to it. We try and put more on a more process, more structure, more things. And his kind of like management rule was always simplified. So if we have a problem, what’s one thing we can take away, what’s one thing we can remove. And I’m, I’m trying to like really think about that with the team, like, how do we take stuff away instead of adding? And so is it about like, and again, like my, my bias is the addition bias, I’m straight away like, okay, let’s have an experiment template, like, let’s have measurement like standards, and I’m like, I’m adding, how do I take away so that we simplify? Because especially with AI, there is a lot of things where we’re adding, we’re constantly adding, we’re adding metrics, we’re adding, like extra reports, extra things, and it’s adding to the complexity, which is not the intent that we think we’re going to have. All right. Okay. I’m 00:34:12.92 [Michael Helbling]: off the third box. I’m done. No, I like that one. It goes back to what you said before, Moe, which was sort of like, execution is going to zero. So the quality of the idea or the quality of thinking now matters more than ever, because you can go execute on a poor idea so fast, but waste everybody’s time in the process. And so like, having some thoughtfulness ahead of getting everybody rolling 00:34:40.36 [Tim Wilson]: now is sort of like even more critical. I love the language of addition bias, because I think that it is so broadly applicable. And I’m going to like, throw a quick one in, and then we’ll go back to maybe more broadly, but, and it’s a twofer, but because to me, maximizing the data pixel ratio from a data visualization and a clarity of communication, which I’ve been shouting from the rooftops for years. So information dashboard design by Steven Few, I read that like in 2006, Cole Naflik, it’s chapter three of her book is like, basically declutter the storytelling with data data visualization guide for business professionals. But that that’s in a narrow, that’s the addition bias of how do I provide, I’m going to deliver this to a stakeholder, my instinct to build more trust is to put more stuff in it. And what they really want is to remove stuff. And with AI, like with AI, when you ask it, summarize this, if you ask it to give you a two minute script, it will give you a four minute script. If you ask it, like you have to constantly tell it to do less. So it goes for processes, data visualizations, the analysis you do, let’s keep digging deeper, deeper, deeper, deeper. And it’s like, or can we stop 00:36:01.16 [Julie Hoyer]: and make a decision to move on? But it makes me think about, I was thinking about this initially, and now you saying that it makes me really want to try it. And I feel like I’ve done a little bit in passing, but when AI gives you the four minute thing, the huge long summary, I feel like a really good check on the AI is to actually ask it to give you like a four sentence summary. Because I feel like that’s where you can sniff out the BS faster. Like when I’ve asked it for a short summary, I know it’s totally missed the plot than like what I would quickly give as a four, some like four point summary or four sentence summary. Because sometimes it is like you start reading, you’re like, I guess it sounds good. Yeah, kind of. And then I think you’re more apt to just like trust it and maybe use that long format. But it’s like the old adage, sorry, it took me so long to like write you a short letter or something. I’m quoting it a little off, but it feels like that. So I do wonder, could you like stress test the AI output sometimes by asking it for the short thing and be like, ooh, really quickly, good or bad? 00:37:02.44 [Moe Kiss]: I do, I definitely do. But then I end up editing it. And I’m like, I should have just written 00:37:07.08 [Julie Hoyer]: it myself, it would have been faster. Always, always. And we thought we were going to talk about AI 00:37:12.20 [Michael Helbling]: in this episode. Yeah, well, it’s, but it is wise because it in a lot of ways in this current where we are right now with AI, it’s AI is an analytical contributor is very much in the look what I can do kind of phase. And, and it’s sort of like when you think about like coaching a junior teammate, if you want to think about AI like that, it’s sort of the same kind of thing is like, all right, strip all that out. You don’t have to say all that, you’re going way too far, you’re trying to impress because you have cool, you know, it’s like, don’t try to blind them with science, just get in, get out, say what’s important, you know, yeah. So that’s almost sort of how I feel about it. Because it’s like, yeah, it’s trying to do way too much. It’s like, look at the cool stuff I can do. And it’s like, mom, look, look at me, look at me. Like, yes, you’re very smart, 00:38:07.40 [Julie Hoyer]: shut up. Very good. 00:38:17.88 [Val Kroll]: Well, I can go with another one that’s not continuing on this like beautiful thread that we’ve been weaving. We’ll find a link, we’ll make it, we’ll ask AI to make it. 00:38:28.76 [Tim Wilson]: Wait, Tim, did you do your two books, though? Because I had cut it off. 00:38:31.96 [Julie Hoyer]: No, my two four was just the two books. The two books. 00:38:34.84 [Tim Wilson]: Colin Affleck and Steven Fugh. So yeah. 00:38:37.40 [Val Kroll]: Another experimentation heavy one, but another one that I have sent a link to these articles, maybe more than anything else. It was actually a medium post, it was a collection of medium posts, I should say, that was on Towards Data Science. And it was written by the Skyscanner engineering team. And it’s the overarching kind of umbrella title of content is Chasing Statistical Ghosts and Experimentation. And not only does it like break down some of the common like myths, but it’s the things that people still struggle to like fully understand why it’s not effective to run on it. So this is actually very similar to Julie’s first one in that it does produce a lot of visuals, although they are not interactive, to kind of illustrate exactly what the issues are with people having these like mental models. The one that I’ve sent the most, and there’s like four in the series, I believe, is the first ghost, which is it’s either significance or noise. And there’s like one quote in there, like towards the middle, and it’s about like, experiments don’t work towards significance and comparing relative significance of p values outside of the thresholds is a mistake that will lead to a lot of false positives. But the number of times and I wish that there was like a button I could press and it would like zap people in their chairs, if they said like, well, we’re trending towards significance. No, but it’s so well put, they like break it down into like its smallest little pieces, and kind of like build it all back up together. So anyways, it’s just every every part of this was so well done. But yeah, definitely kind of come back to this one quite a few times. 00:40:28.76 [Julie Hoyer]: Yeah, Val, you you turn me on to those and those are amazing. Great, like, ground yourself be like, I’m getting lost in the sauce, like let me go read my articles again for a second. 00:40:42.60 [Tim Wilson]: That’s awesome. It’s a good series. I feel like Julie, there’s a natural add on to that around 00:40:48.20 [Julie Hoyer]: experimentation that maybe you could yeah, I think I know which one you’re talking about. 00:40:56.52 [Julie Hoyer]: My well-thumbed through book with tons and tons of notes in the margin that I’ve actually read just the first five chapters about three times, I’ve done two book clubs on it. Field experiments, very simple, nice straightforward name. Really great book though, this was actually recommended to me and Tim when we worked with Joe, he was leaving at the time search discovery and we were running a randomized controlled trial with a client and it was put on me to, you know, continue it, do the analysis and run the next one and I was like, holy shit, like, what am I supposed to do? And Joe just sends us a link, he’s like, it’s fine, buy this book, read the first three chapters, you know, you guys will be good. Julie, you got this and you did. Yeah, it was 00:41:46.52 [Tim Wilson]: a little scary. I read it and I was like, I was like, wow, ooh, Julie, you got this. 00:41:52.44 [Julie Hoyer]: Yeah. Honestly, thank God. 00:41:55.80 [Tim Wilson]: You know Joe’s other one’s number, so give it a go. 00:41:57.80 [Julie Hoyer]: Yeah, thank God, he still could contact Joe. Also, thank God I was a math major and could read mathematical equations, but it is a really good book. Like for a dense book, a very informative book, again, that’s one that just set such a good, clear foundation. Like the way they write about these complex theories of running these statistical tests, randomized control trials, like again, like the magic of randomization. I mean, that shiny app I talked about at the beginning came from the same phase of life as reading this book. And it was so good and I still 00:42:36.60 [Tim Wilson]: go back. I remember that. I remember hitting that. That was like, yeah, my favorite phrase now. 00:42:41.96 [Julie Hoyer]: Yeah, really, if you need like a crash course in the foundations of randomized control trials, I really do highly recommend that book. Even the first five chapters, Joe said three. I found I needed to go at least to five. I think there’s 10 chapters total. Really good. 00:43:00.36 [Val Kroll]: As a member, former member of one of the book clubs that you ran for that book. You remember, I think I was in your second one. That was one of the densest books. It’s not a quick read, but it’s like the pieces of valuable nuggets per word is probably the highest density of any book. I’m like, oh, there was like two paragraphs and there was like three light bulbs that went off. So, and really good examples in that book too. So yeah, I like that in stories. 00:43:29.40 [Moe Kiss]: Good ones. Tim, you’ve got to talk about that. There’s one you’ve got to talk about. I’m dying 00:43:34.68 [Tim Wilson]: to hear it. Was it possibly first break all the rules? Okay. This is one where is Michael accused 00:43:44.28 [Moe Kiss]: me of not having empathy. I was surprised to see this on a list with your name. So first off, Tim, 00:43:52.60 [Michael Helbling]: it’s not an accusation. It’s just an observation. 00:44:02.76 [Tim Wilson]: Just do your fucking job. Yeah. So first break all the rules, what the world’s greatest managers do differently by Marcus Buckingham. And there were a couple of editions and other people wrote with them. And it’s Strengths Finder is what gets like all the play. And I’m not a fan of Strengths Finder, which is tied to now discover your strengths. But it was to a two book pair. First break all the rules, now discover your strengths. And I read it. It was like required reading early 2000s for managers at the company I was at. And to this day, it gave me a lot of confidence when I started working with people who just weren’t the right fit for the job they were in and getting comfortable with this idea of like skills versus talents. And you can you can teach skills, you can’t teach talents. And that doesn’t mean you can’t raise people up to get better. But if somebody is just like not good at data visualization or not good at building trust or not good at client communication or whatever it is, you can give them training and we have this tendency to say, well, that’s their deficit. That’s their deficient in that area. Let’s spend as much time as we can coaching them and training to to bring them up. And when you start to recognize that it’s like, no, that’s just not them. That doesn’t energize them. The best you can do is get them to a base level of performance. That was like one big aha from the book. The other one that like really blew my mind because the way they write about it is like so true is that our tendency, if we’re managing a team is to spend the most time with our low performers because it’s like the idea that the lowest performer is what the team is going to be judged as. And they’re the people who need to be raised up. The rock stars, you’re like, they got this. So we’ll just dump more and more stuff on them. And the book makes a really, really strong case for saying you’re you’re shooting yourself in the foot, you should be spending, you should be super charging the rock stars. And you need to support the lower performers, but it’s not really your job to try to make a square peg fit an around hole. Your job is to see if there is a role you can get them into, where they will thrive and become rock stars, but trying to coach and train when they’re just not a fit. I mean, that’s literally I read that 20 years ago. And it’s one that I multiple times, I will see it, I will be interacting with somebody, I will give honestly, even though I am kind of a jackass, like I go in with a presumption of good intentions and good capabilities. And I’ve, especially with analysts, I have gotten more and more confident over the years of somebody who’s just not, when you start redeeming their backstories, it also often was somebody who is really struggling. And they like the idea of punching buttons from the data and getting glorious insights and just have zero intuition or any of the compulsions that analysts do that make them good analysts. And I’ve got like names in my head of like, they’re never going to thrive in this. And the best thing that can happen for them is to find their way into another area. So yeah, that’s weird. That’s like a management book. But I find myself coming back to it. 00:47:36.04 [Michael Helbling]: And would you say, Tim, this has been more applicable in your work life or your podcast life? 00:47:44.52 [Tim Wilson]: Only, only so many roles I can shift people around into and I have a little control. 00:47:50.44 [Michael Helbling]: No, I’m sorry that I made a joke about it. Because actually, I see this in you, Tim, like you doing this and like how it affects how you interact with people. And I think I’ve learned some of these same lessons over the years in managing teams and people of like figuring out how to shape their role to be help them be as effective as they can be based on sort of what they’re what they’re naturally inclined towards versus sort of what you sort of want them to do. And I like that puzzle of figuring out sort of where people fit. It’s, it’s fun. Not always, you’re not always allowed in every environment to to puzzle with it as long as you’d like to. But 00:48:31.24 [Tim Wilson]: it’s the fact is spending more time with your stars is a lot more fun and energizing helps you grow as well. So like that book, giving the permission to say, you don’t feel bad that you’re like collaborating with somebody who is everyone thinks of as being like the star on your team. Like that’s here’s here’s the reasons for why that’s actually the right thing to be doing. Again, not neglecting people. It’s not like it’s a complete yeah, binary. But yeah, it’s sort of in 00:49:02.36 [Michael Helbling]: the same vein. It goes way back. I there was a radio lab episode back before I even listened to podcasts. This is like on the radio station like back in 2010. And they were discussing like how people use emotion and logic and decision making. And they had a specific case of a person who’d lost the function in their brain that allowed them to bring emotion into decision making. And so all their decision making was completely logical only. And as a result, they were actually paralyzed in every decision they would make. And it literally destroyed their life. Because they they couldn’t decide between like, Oh, I need to sign this document, should I use a blue pen or a black pen? And then it would go back and forth from the pros and cons of a blue pen or a black pen to sign and then everything they ever did good grocery store, they couldn’t pick which toothpaste to buy like, and it was really fascinating to get a window into this idea that so often emotion drives decision making more or as much as logic does and sometimes a lot more than logic does in a lot of environments. And that was another big like light switch for me. And around that same time, I was reading a book called Switch by Chip and Dan Heath, which kind of goes into some similar concepts about how to kind of be influential or make people see what you’re trying to say through different ways of kind of reasoning and and providing good examples of things and stuff. So that was sort of a time where I was sort of like, okay, yeah, how do I get people to like, engage with decision making? And also how do I this all goes back to me trying to figure out how to leverage who I was in the context of analytics, let’s just be honest about it, because like, I’m, I wouldn’t call myself a traditional analyst by any stretch. And so by kind of learning some of these lessons and watching sort of the emotional process of decision making, it gave me really good hooks into, okay, here’s how I can engage with people because I can feel the emotions coming off of people a lot of times and kind of into it, what to do. Whereas my ability to like create amazing analyses like Tim and with perfect visuals and everything is not, I have good skills, but they’re not as good as yours, Tim, let’s be honest. And so as a result, like, you know, I had to fool people into doing what I 00:51:26.36 [Tim Wilson]: wanted them to do. So it’s almost becoming like a trope or like conventional wisdom that people aren’t, they don’t make decisions based on the data and like they make it based on emotions. And then that sometimes that gets used as like weaponized against analytics, like how do you square that in a way that’s healthy because it gets, it gets treated like so throw away like, well, then what the hell are we even doing here? It’s such a balancing act. 00:52:06.36 [Michael Helbling]: I feel like there, people get a hold of this information and do seek to manipulate it. And I feel like that’s, at some point, like that kind of behavior is going to get caught and it’s going to have a limiting function. And so you shouldn’t do that. It’s sort of like if you’re a company 00:52:27.08 [Tim Wilson]: that’s in the public. People being being manipulative and playing to emotions. Yeah, yeah, yeah. Isn’t okay. Gotcha. So that’s like the whole point of traceology, right? And that’s why I love 00:52:37.08 [Moe Kiss]: that show so deeply is because like, each episode is going into one of those biases and how it shows up and the decisions that we’re making. And like, and it’s, I mean, Katie Milken is just so good at explaining what sometimes they’re like, quite technical, scientific studies in such a relatable way. But I just, I don’t know if I see it being weaponized. I’m more, I actually sometimes feel, I see it the other way, which is that I see data people being like, data decisions should totally be based on logic and what the data is saying. And I, I have very much come to this piece of like, intuition matters more. That is a different piece of data that is all your past experiences. And that also is worthy of consideration when you’re making a decision. There are other quantifiable data points that you also include, but like the idea that someone’s only going to make a decision based on those without other factors like intuition, what you’re like, I don’t know, maybe. Is that, is that like when you kind of intuition together? 00:53:40.84 [Tim Wilson]: You should be bringing together like facts & feelings. 00:53:45.32 [Michael Helbling]: Somebody should run with that. Somebody should run with that. But yeah, that was the other word, Moe, that I was going to bring into it as well. Intuition sort of runs this path as well, because people will rely on that more than they will rely on external facts and figures. And like, there’s that famous Jeff Bezos quote of like, oh, if the numbers look different to you than your intuition, then probably trust your intuition, you know, that kind of thing. And, and in reality, it’s true. Like our, our, and in the way our minds work as humans, there is a really good book on storytelling by, I think we’ll store and one of the quotes out of there was basically like, we look to fit patterns. So if the data fits our model, then we’ll be more willing to accept it. And if it doesn’t, or more, more, less likely to onboard it, just because like the way our human minds work, we’re trying to fit things in. And so if it’s sort of like counter to what our intuition or, or what we think is true of the world, but also when you look at even your own decision making process when you have different emotional states. So if you have high cortisol levels and anxiety, your decision making is actually impaired as compared to like, lower levels of anxiety or, or more calm feelings. Like it’s just true. Like you, you make different decisions and those don’t smirk. All of you are laughing and smirking because you know it’s true. It’s just hitting home for me. I’m like, yeah, yeah. And we think about that. So, so think about that for like a CMO who is like literally these next few campaigns literally mean their job. And you were coming in with some analysis that need them to make decisions and they are so high strong. Like how do you even pull some of the action out of the room so they can get to a good place to make good decisions sometimes? It’s so tall. It’s like you end up in a therapy role practically when you’re trying to provide analysis. And it’s, it’s not like you should be doing that, but it’s sort of like a reality of the work environment sometimes. So we don’t always do a good job. But like there’s all this data that, you know, Google did that huge project about sort of how people perform maximally in their jobs. And like when you provide the right amount of psychological safety and those kinds of things like people’s performance improves, like there’s so much data to support all of this stuff. And so the idea that emotion shouldn’t have a place or isn’t involved in decision making, like you really kind of like miss that at your own peril. And it’s not just sort of like, how do I feel about you? It’s like, how is everything around me happening? And it could be nothing to do with you walking in somebody’s office with like an amazing analysis that you’ve really thought through and prepared very well. And the meeting might go super poorly and just has a zoolcho to do with you. It’s because they got screamed at in the meeting before this, and they’re still coming down off of that. And there’s really nothing you can do about it except maybe come back Moenday and talk about it again and hope that they’re in a little better place. 00:56:45.32 [Tim Wilson]: When there’s like a, like another corollary, which is separate from that is like recognizing 00:56:50.92 [Julie Hoyer]: fear and shame, like negative emotions that the goal, like if you can get them emotionally on 00:57:00.28 [Tim Wilson]: as a partnership, it goes to kind of the trust, it goes to all of the pieces of this, any idea that you’re going to walk in and like put up a chart that makes somebody look bad or surprises them in a negative way. You know, even if you’re like, oh, I’d make these four people really happy, but this person, it makes them look like I’m finally calling bullshit on them. And that’s going to be great because my manager and their manager are going to love me because I’m going to help them win this battle. Like that is a short term win. And you just, like much, much 00:57:32.76 [Michael Helbling]: better going to be gunning for you every single time from here on out. Even if they’re a great 00:57:37.88 [Julie Hoyer]: person who doesn’t want like much, much better to have them along on the journey so that they too 00:57:45.16 [Tim Wilson]: are vested in like, it’s like asking the question like, what would you need to see in order to make this decision? Or what would you like to bring them on the journey where they are, I don’t know if that doesn’t go, that’s trying to play to the emotions, marrying it with the data to say, let’s think about how would we feel if we saw that? How would we feel if we saw that take a little bit of the sting out of the, what does this mean for my self-worth and my professional career and make it more about, hey, we’re doing something good for the company. Wow. Damn, that’s me. I’m sorry. That’s too much lip service to soft skills. I love it. I mean, maybe what we’re 00:58:27.08 [Michael Helbling]: finding out, Tim, is like, all this stuff really matters in a world of AI. There’s not a word I’ve 00:58:32.44 [Tim Wilson]: said that I haven’t just got straight from Claude. Like, I was just like, and now they’re talking about emotions. What should I say? It actually bugs me. Oh, go ahead. Well, no, go ahead, Michael. 00:58:44.84 [Michael Helbling]: No, it’s a sort of a tangent, but like lately, Anthropic has been releasing sort of these things about how Claude has like these emotions that it expresses. And it really rose me the wrong way that they even frame it like that, because I’m like, you can’t be teaching people to like emotionally support in AI. Like, it’s not a good way to deal with LLMs in my view. But again, you know, whatever, what do I know? 00:59:15.80 [Val Kroll]: Well, you made me think of another quick one, making sure someone’s ready to hear a message, especially if it’s a difficult one, episode 240, Taylor Bonacore Guthrie. The questions, that’s one that I definitely reach back for a lot. On the scale of one to 10, where, you know, one is a dumpster fire, 10 is the best day of your life, where are you? And so if they say, four, because I just got my ass chewed by, well, you’re like, well, we’re gonna save me for Moenday. And we’re gonna go ahead and come back. Need a couple more days. But yeah, no, that’s a, that was, I learned a lot from, from that episode from Taylor and definitely have returned to that one. So that, that’s another one I should have put on there for 01:00:01.56 [Michael Helbling]: this episode. That’s a good one. All right, we do have to start to wrap up. This has been a really fun conversation. I think all of you, there’s been some really amazing insights. So trying to empathetically put myself in the position of a listener, I think we had some good stuff there. So hopefully it’s helpful. You know, we’ll, we’ll wait and check the 01:00:24.84 [Tim Wilson]: median, median listen length and let the data tell us whether this was good or not. 01:00:29.40 [Michael Helbling]: Yeah, that’s right. The algorithm will tell us. Anyway, as you’ve been listening, I bet you have things that you found in your career that are super helpful and guide you even in an AI kind of environment. We’d love to hear from you. So yeah, please reach out to us. The best way to do that, you can do that through our website. You can also do that on Measure Slack or on LinkedIn, or via email at contact at analyticshour.io. And as you listen to the show, leave us a rating or review on whatever platform you use us on. We’d love to hear those. We’d love to see them. It helps us out. And if you love to rock a sticker on your laptop or your phone case or something like that, we’ve got analytics power hour stickers and you can order those as well on our website at analyticshour.io. So yeah, thanks again, everybody. Moe, Julie, Tim, Val, this is really nice. Yeah, I’m feeling really nice. I feel good. All right, this kind of felt like a hug. Yeah, I feel more prepared to face the future. And I think no matter what the future holds, I think I speak for all my co-hosts when I say 01:01:41.88 [Announcer]: keep analyzing. Thanks for listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at at analyticshour, on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group. Music for the podcast by Josh Crohurst. Those smart guys wanted to fit in. So they made up a term called analytics. Analytics don’t work. 01:02:08.84 [Julie Hoyer]: Do the analytics say go for it no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning 01:02:20.28 [Julie Hoyer]: in competition. Okay, one of my fallback ones I have to say to that point, Michael, I was trying to remember, I always like reference this one idea. I was like, I knew I listened to it on a podcast and I was trying so hard to remember what it was. It was from this podcast. So I put it in there as a fallback, but it was this podcast that I heard. I mean, please do bring that one up. 01:02:43.80 [Julie Hoyer]: I was like, that’s maybe a little awkward, but I’ll throw it in there just in case. 01:02:51.56 [Michael Helbling]: I would even like that episode very much. So I’m glad you got something good. Yeah, I didn’t mean it either. 01:03:00.68 [Julie Hoyer]: Which is funny because, yeah, we don’t have to talk about it now. We don’t know. We can talk about it. It’s fine. 01:03:07.40 [Tim Wilson]: That’s East like Denison. Oh, okay. Yeah, that sounds really tough. Yeah. 01:03:12.92 [Michael Helbling]: Shut up, Val. Columbus is not that big, but 45 minute commute in Columbus. That’s a big deal. 01:03:19.56 [Val Kroll]: No, you said South side of Columbus. It just sounded like… Oh, that’s not you. You come from Chicago. 01:03:27.72 [Michael Helbling]: No, South side of Chicago, yeah, has a rep. 01:03:32.36 [Julie Hoyer]: See, I feel like I should be drinking a glass of wine to tell that one. Do I? Okay. Parameters here before I go off and don’t do the format of the show, right? Also, I feel like I’ve picked up a slight Southern twang from being in Mississippi for two days. So if you hear it… Oh, this really is a… I’m not actually background. 01:03:53.48 [Tim Wilson]: Dude, you become vaguely racist and also misogynistic. Hey, easy. I’m not trying to defend Mississippi. 01:04:02.20 [Julie Hoyer]: But do we want to give the background of where it came from or how you first… 01:04:07.56 [Julie Hoyer]: Yeah, of course. …came across it and then how you used it? Is that kind of the format or am I misogynistic? 01:04:12.92 [Moe Kiss]: Just go with the vibe. 01:04:15.08 [Julie Hoyer]: But I’m not saying the vibe. And my freaking comment. 01:04:18.12 [Michael Helbling]: Hold on. No, you’re not freaking everyone out. Just testing my distance from the microphone. 01:04:21.64 [Julie Hoyer]: Do I need… Is my volume okay? Your volume is fine. That’s my hot mic. Your volume’s great. And you don’t have to be quiet. 01:04:27.96 [Julie Hoyer]: Yeah, I’m sorry. 01:04:30.36 [Julie Hoyer]: True. See, that’s the thing. Like, I tend to try to be quiet because my mic was so bad. 01:04:33.88 [Val Kroll]: I can tell by the way you move your mouth that you’re trying to be quieter. But also, I’m weirdly observant about things like that. No pressure. 01:04:45.00 [Moe Kiss]: Fuck my life. One second. I’m so sorry. 01:04:47.40 [Michael Helbling]: All right, perfect. I didn’t want to start right now. Okay, wait. 01:04:50.52 [Julie Hoyer]: Do we get to tell the story about Moee and her saying, fuck my life in our last episode? Oh, it’s in the outtakes. 01:04:56.28 [Tim Wilson]: It is totally in the outtakes. Oh, my God. 01:04:57.96 [Julie Hoyer]: Wait. 01:04:58.12 [Tim Wilson]: So good. 01:04:59.72 [Julie Hoyer]: Oh, that’s Moee’s tagline. 01:05:04.44 [Tim Wilson]: Yeah, Moee had to like, as we were losing co-hosts sporadically for various crises, and Moee left and thought that we would have finished wrapping up by then. So she just came back in with the hot fuck my life. And we just went with it. In the middle of the wrap up. Can I finish the wrap up now? 01:05:26.04 [Julie Hoyer]: And Moee was just going in the back. He’s like, can I finish wrapping? 01:05:29.72 [Julie Hoyer]: And she was like, oh, my God. I was like, yes. 01:05:33.88 [Moe Kiss]: I think I actually said fuck my life again. 01:05:36.28 [Michael Helbling]: Yeah, you did. You did. Yeah. 01:05:38.60 [Moe Kiss]: Oh, jeez. Anyway. Okay, good to go. It was great. 01:05:40.52 [Michael Helbling]: All right. 01:05:41.56 [Tim Wilson]: Here we go. All right. We’ll get started in five. I’d like to say right now, Tony, this is going to be the perfect. There will be what you went through on the last one. This one is smooth as silk. Rock flag and do your fucking job. The post #297: Durable Wisdom in an Age of AI Slop appeared first on The Analytics Power Hour: Data and Analytics Podcast .

April 28, 20261 hr 4 min

#296: Avoiding Major Oopsies: Twyman’s Law, Intuition, and Valuing Accuracy Over Precision

What do diamond ring shopping, Uber pricing psychology, and active user metrics gone wrong have in common? They all highlight our complicated relationship with precision versus accuracy—and how that relationship can either build or destroy trust in our data. Arik Friedman from Atlassian joins us to unpack why being “about right” often beats being “exactly wrong,” and why your nagging feeling that something’s off might be a useful insight in and of itself. From the discipline of documenting assumptions to the art of knowing when to round your numbers, we tackle the very human challenge of working with data that’s supposed to be objective but rarely is. Plus, we explore Twyman’s Law (if data looks too good to be true, it probably is) and why sometimes your intuition is your last line of defense against embarrassing mistakes. This episode is brought to you by Prism from Ask-Y —your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show (Free Books: separate R & Python versions) An Introduction to Statistical Learning (Video) Hamel Husain | The Revenge of the Data Scientist | PyAI Conf 2026 (X Article) The Revenge of the Data Scientist (Video) Lenny’s Podcast: Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar The Monarch App (Article) Vibe Coding Will Bite You. Here’s Exactly Where… by Cassie Kozyrkov A Checklist for Making Better Decisions: A Glossary of All the Topics We’ve Featured on Choiceology Photo by Sarah Kilian on Unsplash Episode Transcript 00:00:00.00 [Tim Wilson]: Welcome to the Analytics Power Hour. 00:00:08.92 [Announcer]: Analytics topics covered conversationally and sometimes with explicit language. 00:00:15.20 [Tim Wilson]: Hi everyone, welcome to the Analytics Power Hour. This is episode number 290, oh wait, no, wait, let me check, this is episode number 296. 00:00:26.04 [Tim Wilson]: That was a close one. 00:00:28.32 [Tim Wilson]: I mean, how embarrassing would that have been? We’re all set to have a discussion about data accuracy and getting business partners to trust the data and I almost whiffed on something as simple as the episode number. Already though, perhaps I’ve undermined your trust in me, Tim Wilson, sitting in the host chair that is usually occupied by Michael Helbling. Perhaps I have. Let’s ask my co-hosts. Moe Kiss from Canva, welcome to the show. What do you think? Did my little gaff destroy my credibility? 00:00:57.68 [Moe Kiss]: Yeah, trust is completely lost. No, I’m joking, you nailed it. 00:01:02.44 [Tim Wilson]: It’s been lost, it was lost years ago with you, so, and Julie Hoyer, from further, have I already destroyed our audience’s confidence in me? 00:01:14.16 [Julie Hoyer]: What’s a mistake among friends, right? I’m just your pre-read. 00:01:17.20 [Tim Wilson]: Okay, I like it. Well, that’s the topic for this episode. I mean, sort of. All data can be digitized and an overwhelming amount of it is and digitized data breaks down into like discrete little ones and zeros somewhere down the chain. It should be cold and objective and to use it effectively, we need to get it to be as accurate and precise as possible, right? I mean, well, maybe. So joining us for a discussion on just that topic is Arik Friedman. 00:01:48.88 [Tim Wilson]: Arik started his career as a software engineer, then took a turn and did a PhD in computer 00:01:54.28 [Tim Wilson]: science focused on privacy-preserving data mining, which is a tongue twister, then went on to work a few years as a program manager for Microsoft R&D, then he popped back over to research for a while, and now he’s been at Atlassian for over 10 years where he is currently a senior principal data scientist focused on product analytics. And today he is our guest. 00:02:18.40 [Tim Wilson]: So welcome to the show, Arik. 00:02:21.52 [Arik Friedman]: Thank you very much. Long time listener. Happy to be here. And I assure you, like all of these career moves made a lot of sense, like it, it made a lot of sense and at the time, it was a logical progression at the time. Absolutely. 00:02:36.34 [Tim Wilson]: I went architecture, technical writing, marcom analytics. So I, I sympathize and I actually, I don’t know that any of those made any sense. 00:02:45.84 [Tim Wilson]: But that’ll be something for me to ponder on my deathbed, but so Arik, you presented 00:02:53.40 [Tim Wilson]: at the last measure camp in Sydney and in that presentation, I think you had a, a fun or maybe it’s relatable at least, or maybe it’s a tragic story about an active users metric that looked like almost too good. Like the results were brilliant. Everyone was thrilled, but you had this like nagging feeling that something was off and you couldn’t really nail down the specific problem. So maybe that’s a good way to sort of start the show. Cause I think it gets to something really human about our relationship with data. So maybe we’ll kick things off by having this, having you walk us through kind of what happened 00:03:28.88 [Tim Wilson]: there. 00:03:29.88 [Arik Friedman]: Sure. So like this is a story from a while back, but you know, at the time the product team was working on a product feature, a big change in the user interface and a lot of investment went into that. So we put a lot of work on actually testing things, you know, A, B testing. I worked with another data scientist to make sure that we got things right. And I remember back then, you know, us standing in the boardroom with all the PMS and it’s looked like really a big change. Like it moved the needle quite significantly. So that was, that was a big deal. 00:04:11.24 [Tim Wilson]: Everybody were happy. And then as we went on and rolled out this feature, we saw what we expected to see. Active users went up. 00:04:22.96 [Arik Friedman]: But then, you know, over time, I’m starting to develop this, you know, old feeling, you know, you look at data over time, you get a sense of how it looks, how growth looks 00:04:34.28 [Tim Wilson]: like. 00:04:36.04 [Arik Friedman]: And I was looking at the curse and they looked a bit suspicious, right? Like it’s not supposed to go like that. And I started looking into that, okay? Because it felt a bit odd. And I remember, you know, trying to go down to specific trace logs, trying to find what went wrong, I found like everything checked. So you know, after putting some time into that, I said everything was good. We actually have a reason to believe that, you know, active users goes up, all is fine, right? I cannot say anything about this. I don’t have a smoking gun. Fast forward, sometime later, another data scientist and a software engineer, of course, we keep looking into this and they actually found a bug. 00:05:23.32 [Tim Wilson]: It turned out that because of a bug, it caused inflation of active usage monitoring. 00:05:30.00 [Arik Friedman]: And that was an unpleasant surprise for the product team. So for me, it definitely caused a lot of thinking, you know, like, how come I didn’t find that? 00:05:40.24 [Tim Wilson]: What could I have done different? 00:05:43.00 [Arik Friedman]: And that, you know, that causes a lot of reflection. 00:05:47.60 [Tim Wilson]: Well, did you go tell them, like, immediately, abruptly, did you ease it out? Like how did you actually communicate, like, what’s the rest of the story on that? 00:06:01.08 [Tim Wilson]: Yeah, so I think that’s probably the biggest mistake I made at the time because, you know, 00:06:08.36 [Arik Friedman]: I didn’t find an issue. And you know, I, at least back then, I thought, you know, we are the data people, we are the evidence people. And if we don’t have the data to back up what we say, we should shut up. And what I learned from that is actually, you know, our intuition are, you know, that’s part of our expert opinion. 00:06:31.08 [Tim Wilson]: And we should sometimes just go with it. 00:06:36.12 [Arik Friedman]: And I think there are a lot of things we can do ahead of time, you know, to prevent mistakes or to check things. But at least for me, like, for this story, like, when I look, you know, what could I’ve done different, I actually knew to do all the checks. And eventually, when your intuition is your last line of defense, and sometimes you just have to go with that. 00:06:58.04 [Moe Kiss]: So sometimes, I don’t know, I, firstly, I just want to say thank you. I’m really grateful that you’re sharing, you know, straight off the bat, an experience of where you made a mistake that’s incredibly humble and wonderful for folks to be able to learn from. So a very big thank you. And I suppose I just, I’m curious to understand this intuition piece, right? Like I feel we all have it. And I know you and I have had conversations previously about like when we’re making decisions, you know, we have data, we have intuition, we have, you know, maybe previous experience, we have all these different factors and part of our role is helping pull those things together. But I’m curious to understand. So how has this changed how you would show up? Like, let’s say this happens next time, you can’t find a smoking gun, you can’t find a data point, but you had your, you know, your intuition in your gut that’s like something’s not right here. Like, do you think this time you would say something and how would you frame it? 00:07:49.92 [Arik Friedman]: Yeah. 00:07:50.92 [Tim Wilson]: I think, first of all, it’s about being more opinionated. 00:07:56.88 [Arik Friedman]: And in the, like over time, that’s part of my growth journey. I think in the past I would, I tended to be a bit more, you know, impartial, right? We let the data speak. Like we’re just there to give the data its voice. And I think over time I feel a bit more confident, you know, to be opinionated, like I have my own opinion of things. And I think we’re actually expected to be opinionated. So I think that’s, that’s one thing, just like be more confident in our own expertise. And then I think it’s probably, I mean, the opinion on its own is not enough, right? Like a probably, even if we have just this intuition, we will still be expected to, you know, okay, dig into that. Okay, can you actually find the evidence, but I think it’s a start of a conversation, right? 00:08:50.52 [Tim Wilson]: Like this is what I think. Maybe the data looks a bit odd and we maybe want to dig a bit more into that. 00:08:59.16 [Arik Friedman]: And then the business can decide, you know what, no, like we did all our checks and balances. It’s good. We did our due diligence. 00:09:05.20 [Tim Wilson]: Let’s go on. 00:09:06.36 [Arik Friedman]: Or they could say, you know, let’s, let’s take more time and actually give it more space 00:09:10.24 [Tim Wilson]: to dig into that. 00:09:13.00 [Arik Friedman]: And at the time I basically made my own decision and said, you know, I looked enough into that and I kept it with myself. And this is something where we need to be more open and just share what we think with a business partners about that. 00:09:28.08 [Tim Wilson]: So I will sometimes find myself not doing this necessarily with a ton of structure and rigor, but trying to think what I expect to see before I actually look at the data, or sometimes asking if I really don’t think I have context and my business partner says, I think it’ll go up. I’m like, what is, like, what does up mean for this metric or what is, what would that mean? Like I don’t give me, use it as an opportunity to try to get a little more context about their intuition about their business, like there’s, there’s some psychological trick or something where you, you sort of force yourself to set what you think you’re going to see as opposed to waiting until you see it and then you immediately rationalize why it makes sense. Like I had an example from years ago where a product marketing manager came racing in because he had already told the whole company that this tiny little change he’d made to a webpage had like drastically driven the track traffic up, which made no sense. And that was one where I dug in and found that it was, it was a bot, there was a company that was selling us software and they pointed literally at his page to gather some data for the pitch in, in a few days. But it was one where it was like, he saw it, he was a good news, kind of like the active users. But if I’d gone back, if I’d known he was making that change, I would say, what do you 00:10:59.28 [Tim Wilson]: realistically think this is going to do? 00:11:03.24 [Tim Wilson]: Because if it goes way past that, maybe we want to, you know, apply a little bit of climate law or, you know, something to it. But I think there’s, I don’t know, I feel like that’s a whole, it’s a challenge, especially for people just entering a field or the space to have that intuition and to have the confidence on top of it. Like you said that a few times that it’s like, as you get more experience, the more faith you have in your intuition and your conviction. And that’s like, how do you, how do you develop that if other than letting time pass? 00:11:42.00 [Arik Friedman]: I think a lot of this is getting to live a bit in your business partner’s world, like getting to speak the language and, you know, getting to know what they care about, what they’re after. I think that adds a lot of context. And for example, I remember at some point, you know, I started hearing PMs talk all 00:12:01.48 [Tim Wilson]: the time about like JTBD, JTBD this, JTBD that, like jobs to be done. 00:12:06.72 [Arik Friedman]: I had no idea what it was all about. And, you know, I started digging into that, reading a bit about that. And, and, you know, what I found, like, I didn’t really get what they were talking about. So getting into the world, learning about that, I think helps create this common language and thinking about it from their perspective. So I think that definitely helps us, you know, to get, so to speak, like in their head. 00:12:34.20 [Tim Wilson]: When that was one where you actually, if I, I had this happen with OKRs once where I was working with someone who was, and I knew what OKRs were, I’d lived in kind of 00:12:43.72 [Tim Wilson]: an OKRs training, but it was like, oh, this client is really into it. 00:12:49.12 [Tim Wilson]: I’m going to go get a book. And didn’t you do that? Didn’t you wind up going and like read, like whatever the JTBD Bible is? Like, OK, if we’re going to use this, I want to understand it. I mean, you probably found out that they actually weren’t using it correctly. It’s like Agile or any other number of things that use the acronym, but maybe aren’t applying the process. 00:13:10.36 [Arik Friedman]: Yeah. So I went on and I actually read about jobs to be done. And I, it was actually weird because like what I was reading and what I was hearing from the PMs were not exactly the same things. And at the time, like in our confluence, like internal wiki, I wrote a blog post. Hey, like, are we doing JTBD wrong? 00:13:32.04 [Tim Wilson]: And it was a bit of a clickbait, but it started a conversation 00:13:36.96 [Arik Friedman]: because I mean, then I started like through the conversation with them. I started to find out, oh, actually there are different kinds of definitions of JTBD out there and having this conversation and trying to understand, you know, what they really mean when they talk about that, that really helped. And I did it from the perspective of, you know, how it can be more useful as a data scientist and what is it that they actually after when they talk about understanding the JTBD. So I think it was first, you know, just to catch up with them, but also to see, you know, how can I answer their questions and how can I better understand the question and, you know, improve myself as a data scientist 00:14:18.68 [Tim Wilson]: that helps them. Michael, I have news. 00:14:23.48 [Tim Wilson]: The AI analyst is emerging. 00:14:27.08 [Tim Wilson]: Oh, that’s big coming from the quintessential analyst. What do you mean, like a cryptid? Well, I mean, more like a more like a job promotion, but, you know, 00:14:35.32 [Tim Wilson]: with more existential dread, you know, how foundation models created the AI engineer role. Yeah. Developers got all these cool titles and analysts got. Can you pull this by in today? Exactly. But now we’re watching the birth of the AI analyst, someone who uses LOMs to multiply their capabilities without, you know, multiplying their stress rash. Nice. So an analyst, but with superpowers and fewer open tabs. Exactly. And the tool for that is Prism by ask-y.ai. 00:15:07.92 [Tim Wilson]: Yeah. Prism is basically the interface between what I mean and the 900 steps I don’t want to do. You ask in plain English and it helps you get from question to analysis really fast. 00:15:19.72 [Tim Wilson]: And it doesn’t forget your world. Prism’s jam memory as J.A.M. their jam memory remembers your definition like what your means by conversion. Which table is the source of truth? And that July data, don’t forget, it’s, you know, cursed. 00:15:37.64 [Tim Wilson]: Yes. Thank you. I’m tired of explaining our metrics like it’s the extended 00:15:43.92 [Tim Wilson]: MCU universe. Plus, you can capture like repeatable workflows as skills, portable expertise, like, you know, clean UTMs or fixed GA for channel grouping or standardized campaign naming and reuse those across different data 00:16:01.84 [Tim Wilson]: sets. I like that because I do a lot of copy paste. This is can’t continue type of feeling. And I feel like it would be nice to be like run skill, look smart, drink water. 00:16:15.64 [Tim Wilson]: Nice. Want to become the AI analyst before your coworker does? Go to ask-y.ai and join the wait list. 00:16:24.16 [Tim Wilson]: Yeah. And use code APH and ask why it pushes you to the top of the wait list. That’s ask-y, the letter Y.ai and use code APH. Yeah. The AI analyst is here. This product is in beta, but you can get in on the ground floor. And it’s coming for your busy work, not your job. 00:16:44.64 [Moe Kiss]: So, you know, relax, chill out. So, Arik, you and I have spent a lot of time talking about this whole like accuracy versus precision playoff. And it’s something that has just really resonated with me because I would say I always lean to kind of like the best possible answer with the time that we have in the business to help make a decision. Like, I suppose I would say I’m like quite pragmatic, but a lot of what I hear coming from data scientists is this number is wrong. This one’s right. We can’t do it this way. This, you know, we have to do it this way because this is the right way. And I guess I just wanted to hear your framing about this like playoff between accuracy and precision. And like, I don’t know, you have such an elevated way of thinking about it. 00:17:31.40 [Arik Friedman]: Yes. So I think like straight from primary school, the way we’re taught things is that accurate and precision are like being accurate and precise are the same things, right? So at school, you get like this big equation. And like in the data science world, you can think about a business question, 00:17:51.20 [Tim Wilson]: like, you know, what, what kind of features correlate with user satisfaction 00:17:55.96 [Arik Friedman]: or something like that? Or how can we predict those kind of things in parallel? 00:18:00.60 [Tim Wilson]: Like, like at school, you might get a question, like, you know, 00:18:06.24 [Arik Friedman]: how much is like 2,124 times 3,926? Like you get this big equation. And what you’re taught is that you need to go through the methods, right? Like you have to multiply this digit by that digit, carry over. If you get anything off, like any single digit is off, you lose all your marks. So if your answer is not precise, it’s not accurate. You lose all your points. And I think we kind of like carry on this mindset with us into our jobs. And in the business domain, actually precision and accuracy are not the same thing because, you know, because, oh, like these big numbers, it’s about, you know, 2k times 4k, it’s about 8 million. And like about 8 million is accurate. It’s not precise, but it’s accurate. And I think that also, like when we get, you know, business questions, there are many ways we can go and approach and solve them. You know, we can throw them, I don’t know, like hidden Markov models or, you know, clustering algorithms. So there’s all this arsenal of like all these methodologies that we learned. And like, you know, that’s kind of like even part of our pride in the craft, right? Like we want to show that we know to do all those things. But sometimes you can get a quite a good answer just by writing like a very simple SQL query. And, you know, it’s maybe not the best answer, but it’s good enough and it’s faster. And there is this quote from John Tukey from like his, he had like this paper about the future of data analysis. 00:19:43.44 [Tim Wilson]: And like this is from like the sixties. 00:19:46.44 [Arik Friedman]: And he says, like far better and approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise. 00:19:56.92 [Tim Wilson]: And I think it really captures well that, you know, it’s first of all, 00:20:02.28 [Arik Friedman]: it’s better to answer the right questions. And I think that part of our job is to help get to those right questions. But also when we answer, it’s not just about precision. It’s, you know, the answer can be accurate without being precise. 00:20:18.24 [Tim Wilson]: And I think that’s a way for us to be more fast and effective 00:20:23.24 [Arik Friedman]: and be focused on business impact. 00:20:24.84 [Tim Wilson]: But there’s there’s a there’s a challenge, right? 00:20:27.56 [Tim Wilson]: That that our business partners accessing tools and data, they’re working with spreadsheets, they’re working with dashboards because they’re in this digital world and processing and multiplication is is is cheap. Like they are conditioned to see things that are precise. I mean, I love the like the the accuracy versus precision. Like a dark archery thing that sort of shows accurate the grid, the two by two of accurate and precise, accurate, not precise, precise, not accurate, not accurate, not precise. And it shows like the spread around the target. 00:21:07.80 [Tim Wilson]: And I think that’s good for us to be aware of. And part of it is, I mean, can that be handled a little bit with the language 00:21:17.72 [Tim Wilson]: of saying, try to get yourself out of the boat of providing. Precise answers, like get comfortable with even if you have a precise answer, better to kind of back off the precision of it a little bit so that somebody isn’t looking at you’re not arming them with ways to. To find another person, it may be too reasonably accurate, 00:21:44.64 [Tim Wilson]: but the precision means that the the first three digits kind of differ. 00:21:49.92 [Tim Wilson]: Like there’s there’s a challenge with us understanding that and our business partners thinking that way. And the levers we can pull to try to help them think in language of. Of this is this is accurate, you know, it’s about it’s about eight million, 00:22:08.88 [Tim Wilson]: you know, or whatever the answer is. 00:22:11.92 [Tim Wilson]: And they’re probably not going to object to that. They’re not going to come back and say, what do you mean about a million? 00:22:16.08 [Tim Wilson]: I need that down to the to the first digit, right? I mean, it’s the final line to walk. 00:22:22.76 [Arik Friedman]: Yeah, and I think it really depends on the context and the question being cast because in some contexts, like if you deal with financial data, you definitely want to be precise. It’s important, but most like a lot of the business questions are more direct directional in nature. And when you deal with directional questions, it doesn’t matter as much. You know, if you’re precise and actually precision can kind of draw the attention to the wrong thing because you don’t care about the position necessarily in these kind of questions. So this is about getting to the bottom of what are the decisions that they are about to take 00:22:58.80 [Tim Wilson]: and what really matters for answering those decisions. 00:23:02.36 [Moe Kiss]: So one of the things I wanted to like challenge and have a discussion on. So I’m trying to find a specific article. And of course, I can’t. But there was some research that was done with pricing and Uber. And I often quote it when I’m counselling people on buying engagement rings. Buying a two-carat diamond ring is more expensive than a one point nine nine nine carat ring, right? Like we have this bias and heuristic to think that the two carat, the rounded number is better. So hold on. 00:23:30.16 [Tim Wilson]: Just wait, wait a minute. How many people are you counselling on buying rings? Like, what is your world? 00:23:37.52 [Moe Kiss]: I my team are quite young. And so a lot of them are going through that life stage. 00:23:43.76 [Tim Wilson]: Those are just one on ones. They’re like, hey, what’s going on? I could really do some help. 00:23:48.36 [Moe Kiss]: Anyway, OK, carry on. So also, I don’t know who’s buying your two-carat diamond ring. 00:23:54.52 [Tim Wilson]: It’s a it’s a anyway. 00:23:59.32 [Moe Kiss]: Is that that’s a that’s a big one? Three carat is quite big. Yes, that that’s not for America, but for Australia. Americans tend to buy much bigger rings. Anyway, back to the point of the story. Well, and now the lab, you should see these rings. 00:24:12.16 [Julie Hoyer]: People are thrown around. 00:24:13.76 [Tim Wilson]: Sorry. You guys are living in a different world. 00:24:18.44 [Moe Kiss]: But what I was going to back to my point, one of my concerns is like, I hear you and I agree with everything that you’re saying, right? But especially, OK, the same thing happens when you’re trying to counsel people on like a day right. Sometimes like people will intentionally like make a number, not a round number, because people are more inclined to believe it, that you’ve put more work into it, because we have this bias that say if you say $1,600 a day, that like it’s not it’s you you’ve just picked a number from the sky versus having put thought into it. So like I’m just trying to think about how that plays off with our discussion here about not necessarily always wanting that precision. So you’re saying if we don’t go precise enough, 00:25:04.72 [Julie Hoyer]: they won’t trust the number. 00:25:06.48 [Moe Kiss]: Yes. So if we say, oh, the number is eight million versus eight million two thousand and ninety seven, will people just be like, is that bias going to come to life? Where they’re like, oh, maybe that number is not right. And like I can kind of disregard it because it’s come from nowhere. Or like, do you see what I’m saying about like, does the precision give any extra credibility or build more trust? 00:25:29.84 [Arik Friedman]: Right. So in those cases, like precision sometimes is kind of like the signal that you did the work. Right. And again, I think these kind of things are context dependent because 00:25:42.84 [Tim Wilson]: ideally, if you have established a good like 00:25:47.72 [Arik Friedman]: trust trust relationship with your business partner, then they trust your judgment and said you made the call the right call about what’s a good methodology to approach this. And in some cases, yeah, maybe precision does matter. Like maybe it’s a signal that’s important to them. And in that case, yeah, maybe you do need to put a bit more work 00:26:11.44 [Tim Wilson]: and and give that kind of signal. 00:26:15.60 [Arik Friedman]: So that definitely is context dependent. 00:26:18.28 [Julie Hoyer]: And part of the context, I’m just thinking like, Tim, like back to some of our discussions we’ve had with clients when they have really low volume, right? The difference of one is greater than if you have a large volume. The difference of 10 is like minuscule or, you know, like you can kind of ignore it. So I do feel like, too, it’s like, are you speaking of a really small end? Because then you don’t want to round compared to when you’re speaking large volumes, you can round to eight million and it’s probably OK. The other thing with precision that I think is interesting as we get sucked into a lot is when the stakeholders want to compare systems, numbers from systems, which we know are not going to give you the exact same one. Like directionally, they should be close enough. But I do feel like that is a scenario where they really get stuck on. Like, why is this one percent different? And it’s like, well, it’s OK. 00:27:16.12 [Tim Wilson]: It’s like eight million. 00:27:18.56 [Julie Hoyer]: It’s all right. And we’re not talking dollars and you know, which one’s your source of truth. So we should be OK to continue to use, you know, the other number in our decision making. But that’s really hard, I think, for stakeholders and even for analysts 00:27:33.68 [Tim Wilson]: to navigate. But I think we tend to make, I mean, if it’s like, you’re going to err on one side or the other. I mean, I look at somebody makes a line chart to show the results and they label every single data point down to the first value. And that’s not so like there’s there there are choices made in the presentation of the information where I think you can say there’s precision precision there. I’m showing a dashboard, but this is rounded to the nearest million or the nearest thousand. It’s not I’m not giving you eight zero zero zero zero zero zero. I’m saying eight million, you know, or eight point one million. You know, give them a give them a decimal place. So I think there are choices. I mean, certainly are to your point that like the context and actually Julia point as well, like the context does matter. When does it matter? But I I feel like there’s like the trap we can fall into on the analytics side is we have the precision precision might as well show it. You know, our P value is not only is a less than point oh five. It’s every model spits out. It’s the P values point oh oh oh one three four seven two. Like please don’t put that, you know, just kind of paste. Yeah. P values less than point oh five. 00:28:51.64 [Arik Friedman]: And Julia, I think, by the way, it’s a great point because it matters a lot like why the numbers are different. And, you know, scenario one, the numbers are different. We have no idea why. OK, that’s that’s not a good place to be. And that’s that’s a place where you probably do want to ask, you know, questions why why do these things don’t match versus a different scenario where, you know, we actually expect those numbers not to be perfectly the same because maybe we measure slightly different things or, you know, there are known reasons why they should mismatch. So I think usually it’s more about the confidence. Like we know what’s going what’s going on here and team to your point. Yeah, like maybe if we think that this will just distract and raise, you know, questions that are not relevant, then yeah, maybe it’s better off to reduce the position to just avoid that altogether. 00:29:44.04 [Tim Wilson]: So I think the other I think business our business partners and then data teams get sucked into it when it’s the different systems aren’t going to reconcile and I think to your point, like you should understand not down to the perfect we can net out everything, but say these are the the big movers of the differences. A lot of times there’s the ability to say over time, look, these move in the same direction, I watch companies say, well, we just need to pick the system of record for that metric, which feels like my the hairs on the back of my neck go up. Like that’s a cool idea that somebody in the C-suite or one level down said, we have the solution. We just pick our system of record. But the reality is all those systems exist for different reasons and it gets back to context. So I think there’s a part that says we need to put this to rest at some point by doing a little bit of an analysis of saying these differ. These are the main reasons they differ. Let us show you that they generally move in the same direction over time. And now we’re just going to have our standard little footnote that these, you know, move differently. But I think also we’re going to go look at the system that gives us the most what we need because it’s often kind of like upstream system in some process has this data, has this data, hands off to another system, which goes downstream from that. So you have to pick the system where you can get the slice. I mean, I’m thinking in a CRM digital analytics CRM sales world, you can necessarily track the marketing channel all the way through to a sale. And in the middle, you’ve got something like a lead that the downstream ability to follow it comes out of one system. The upstream to follow it comes from another system and heading off and understanding, and maybe this gets back to your point earlier. You’re like, you really need to be figuring out is the question being asked very clear and precise and let that drive everything as to where you look, what level of precision, how you pull it, what the intuition is. And now it just went on a bit of a tirade. 00:32:05.88 [Julie Hoyer]: Well, no, but you bring up a point that, you know, maybe this is me still like wrapping my head totally around the accuracy versus precision. But I do keep thinking of what you were calling out like the target. So we were talking about people being most comfortable with precision of systems matching, but we just discussed, you know, thoroughly, like why that’s, we know it doesn’t happen. But to your point, Tim, if they are, if we can explain in at least broad strokes, Arik, as you said, like why these things are differing. And if we then can say, well, as long as we’re pointing this at the right like target, the right question, the right problem to be focused on, then can we be comfortable with accuracy without precision between the tools? Like, am I taking this way off? But you’re saying, you know, like, if they’re always about 1% off, they’re 00:33:03.28 [Tim Wilson]: all hitting the right target, maybe not closely bunched together, that would 00:33:08.16 [Julie Hoyer]: make people happy with accuracy and precision. 00:33:14.08 [Arik Friedman]: Yeah. So I think that data science teams usually have a good opportunity to collaborate and aim to standardize measurements. And, you know, one example, like I’ve seen in the past where, you know, growth teams and product teams had different measurements of impact. 00:33:31.52 [Tim Wilson]: And that can be very confusing because, you know, when one team says, oh, 00:33:35.68 [Arik Friedman]: like we had that impact, and then the other team kind of like interprets it in their own language. So I think definitely for data scientists, we can be in a role where we, you know, coordinate between our teams to see, you know, can we actually standardize and measure things the same way and ensure that we actually get the same numbers all over the place. But as you said, like sometimes it’s not possible, right? Sometimes, you know, there are actually very good reasons why things should be different. 00:34:08.60 [Tim Wilson]: And maybe then we should just use different names for them. 00:34:11.96 [Arik Friedman]: So it’s very clear that, you know, when this team talks about something, it’s not the same thing that the other team talks about. So I think we definitely have a role to play with this. 00:34:20.88 [Tim Wilson]: So can we shift a little bit, because some of this talking about kind of getting trust in the data, and it feels like there is a, there’s a big one we covered, which is like, we’ll get people to trust in the accuracy. We’ve covered two things. If I’m going to mid-show recap, which we don’t really do recaps and I don’t know what the hell I’m doing. But we’ve got kind of the develop and trust your intuition. We’ve got the think about accuracy and precision differently. We haven’t actually covered like, how do you not push bad info out? And the intuition is kind of like one hook into that, but I like the frenetic pace everybody in modern business is working in. There is a drive to get the task completed off my desk over the wall and into the hands, which just like begs for mistakes to be made. And the cost of making a mistake with the query, with the report is of damaging like trust. So like how, what other like ways do we have? We, we, we briefly referenced like time is law, which really goes to the intuition and doing checks, but like, what are other ways to maintain trust in the data from our business partners? How do we prevent ourselves from undermining trust? It’s like a long ramble with a broad question. 00:36:01.00 [Arik Friedman]: Yeah. 00:36:01.36 [Tim Wilson]: I think first thing and something that I definitely would like to see people do 00:36:08.16 [Arik Friedman]: more is just stop and ask, does this make sense? And I don’t think people do that enough. And it’s like at every stage, right? Like, you know, you look at the raw data, does it make sense? Is there anything off there? If you apply methodology, does it make sense in this business context? You get a certain outcome. Does this outcome make sense? So I think this is kind of like a first line of defense. You know, does it make sense? 00:36:38.60 [Tim Wilson]: But aren’t there, there are times, there are times where it makes sense, but it’s actually wrong. Like if you’re like, oh, it made sense. Therefore, I’m going to go full steam ahead. And then you find out later that. Oops. Just because it passed that hurdle, it was plausible, but. 00:36:57.44 [Moe Kiss]: But you’re making the best decision you have with the information available at the time. I feel like the dust doesn’t make sense. Actually, it’s just a like, it’s a spot check, right? I actually think the thing that we need to do better at is. Does it make sense with our stakeholders? Like actually bringing them into some of those checkpoints versus kind of like, does it make sense to me individually? 00:37:21.44 [Arik Friedman]: Absolutely. I mean, I guess like this is the first check that you need to do with yourself before you engage with others. And I think that’s already a checkpoint that I think can probably spot some of the issues. Beyond that, yeah, I mean. Involving other people in this thought process. And like in general, you want to do peer review process for everything. And by the way, I’m coming from a computer science background. My brain is wired as a software engineer. So basically any time that I do anything that involves statistical modeling or math, I will always pull someone else that has like, you know, strong. Math jobs and statistical modeling, you know, hey, check by work. Like, did I actually apply the methodology correctly? Because I know that they think about this in a different way than I do. And they can spot things that I want. 00:38:17.40 [Tim Wilson]: Will the question come up in that context? Because you said like, does the raw data make sense? And from like the EDA of saying, I’ve made my initial query. 00:38:30.12 [Tim Wilson]: Now I’ve got a table of data that has say 25 columns and 150,000 rows. 00:38:40.72 [Tim Wilson]: Do have I gone through and checked that one is 150,000 rows seem like the right number of rows? Like that’s like a saying to like, is the, does the size of the data make sense? But going kind of checking for gaps, distributions, the values, like checking the columns, doing kind of a, doing that step of saying, this should be the right data, but have I actually done, have I checked all the values, all the variables to see if the distributions and values are reasonable? Like not just trust the query, because that adds time and it’s probably a judgment. 00:39:25.36 [Tim Wilson]: It depends on how solid and simple the SQL is. 00:39:29.24 [Tim Wilson]: Does that make sense? And would somebody, if you’re having, if you’re running it by, is this the right model, how likely are they to say, did you double check that the data that you’re 00:39:38.64 [Tim Wilson]: feeding into this model is the data that you think it is? 00:39:43.80 [Arik Friedman]: Yeah. And I think it’s probably both those things. And it depends also if it’s like, is this the first time I’m answering this kind of question or not? And I’ll give us an example. Like there were cases, you know, at some point we worked on, say, internal developer effectiveness metrics, and we tried to understand, you know, flow of work and things like that. And this was the first time we went to calculate these kind of things. So, you know, I worked on this, there was another data scientist 00:40:15.16 [Tim Wilson]: calculating the same metric in parallel, and we got completely different results. 00:40:22.32 [Arik Friedman]: And the thing is that none of us was wrong. I mean, technically, none of us, we didn’t make any mistakes or technical mistakes in the process, but we started working through the calculations step by step and really as we realized that, oh, we actually made different assumptions about what the data meant, and we made different assumptions about what we wanted to measure. So just by working through this process until the numbers matched, it allowed us to align on the assumption and it gave us the confidence that we were actually measuring the right thing. And so, like, it’s definitely not something you will do with any metric or any calculation that you do, but definitely, like, the first time that you do something, it’s good to have, like, this cross check. And once you got these right ones, okay, the next time you already know what you’re doing and you don’t need the same level of, you know, due diligence. 00:41:18.56 [Tim Wilson]: So Julie’s, like, lighting up on the, like, oh, yeah, yeah, assumptions. Like, that’s another thing we haven’t, we’ve talked about it in the past, but the discipline of documenting the assumptions that you made, and it may be, like, the list of assumptions I made were these 20. If I’m having another analyst review it, I’m going to show them all 20. When I present this to my business partner, maybe I even pre-read it, I’m like, these are the three or four that I want to make sure they understood that I made these assumptions. Like, we’ve talked about that with, as a practice of analytics of you are making these decisions and just, like, making an assumption and moving ahead is, like, pretty dangerous actually having in the process a place to write down those assumptions, one from a repeatability, somebody checking your work, but also to actually go back to the business and actually include that in the results. Not the exhaustive, here’s how we made the sausage, but, hey, I just want you to know, going into this, and if that’s your opening slide and they say, well, that’s a bad assumption, then you actually screwed up because you should have done a pre-read with, hey, before we present this, these are like the assumptions we made and here’s why we made them. But how much, how much do you think the business do that? 00:42:41.20 [Moe Kiss]: That they actually, like, I feel like assumptions are included, they’re often, like, at the bottom of the slide, in the text somewhere, and, like, the business just glosses over them. Like, I would love a business stakeholder to be like, and I’ve had that at previous companies where, like, I’ve had very senior folks, like the founders or CEO, be like, I don’t agree with this assumption, like, let’s, let’s debate this, let’s talk it out. But I don’t see, and I would love to see more of that. 00:43:06.72 [Tim Wilson]: I mean, I find it as a way to, you know, on the building trust to not have that in the delivery, but it’s like, I mean, say there’s an executive you’re presenting to, but there’s this business partner that you’re actually collaborating with, it’s a great way to throw in slack, say, hey, is it safe to assume this and have them say, yes, is it safe to assume that? 00:43:29.20 [Tim Wilson]: And then I, I sometimes will put those assumptions, like, front and center. Like, I want to be confident that, and, but, but I probably wouldn’t say I made 00:43:39.36 [Tim Wilson]: the assumption, I would say we made the assumptions. And hey, just so you know, we assumed, you know, that, that holiday has no impact on our bottom line yet. No, that’s, that would be a bad assumption to make. But I mean, it’s not, it’s not just like a CYA of like, throw this in the, in the footnotes, it’s, I think it’s as much of a show that you’re trying to understand the context and the operating environment, which is building trust upstream, which also means that there will be more trust in the output. No? 00:44:14.60 [Julie Hoyer]: Honestly, though, I feel like sometimes that type of documentation and assumptions are like, definition stuff too. I’m with Yuma, like it’s refreshing and encouraging when you are stakeholders key in on it and understand the value of it or want to debate it and can call it out if it’s wrong. But like, honestly, I think a lot of times it is more useful for yourself or others to replicate the work later on, or if you have to build on the current work. 00:44:42.92 [Tim Wilson]: Like, I think sometimes if you’re not kind of obsessed with like documenting 00:44:48.88 [Julie Hoyer]: all of those things, I mean, I’ve been the analyst that’s been given a spreadsheet with random numbers, supposedly where they pulled it. And I have no other definitions or assumptions. And they’re like, oh, recreate this and do it again for the new time period. And you’re like, F my life, I’m never going to get this. And the person that made it originally is gone. So all that to say, like, sometimes if it’s not building trust directly with stakeholders, I do think it helps like maybe the analytics team broadly or the data scientist team broadly, more for the replicability is how I see it. 00:45:26.32 [Arik Friedman]: Yeah, I think definitely like documenting the analysis or the assumptions. And like, that’s probably not the document that you’re going to eventually show as your final result, but it’s probably good to always have like, here’s the technical document with all the details, all the assumptions. So it’s available there. And so you can reproduce this. And yeah, but like referring to most point, I think you have this conversation with your business partner is part of the process, right? Like you get a question. Like, here is how I understand it. This is my interpretation. This is how it translates to the methodology. So you definitely want to be sure that, you know, you’re on the same page and you’re actually answering the same question. 00:46:10.32 [Julie Hoyer]: The other thing I think of too, like if it does get put in the footnotes 00:46:14.92 [Tim Wilson]: of the document or this the presentation, like sent to your business stakeholders 00:46:20.76 [Julie Hoyer]: mode, like I have spent so much time talking to people and on my team or being so worried about myself, what happens when I give this deck to my stakeholders and then they pass it to their stakeholders and they pass it to their stakeholders and then they start asking follow up questions and nobody knows to ask me for the clarification. So like, honestly, I think my safeguard and anxiety is what makes me put a lot in the final documentation in the appendix or in the footnotes. Because I’m like, hey, if this gets passed like three degrees away from me, which again, that’s a win if your work gets passed that far along. 00:46:54.84 [Moe Kiss]: I see this happen all the time with experiment results where like it, you know, a data scientist has pulled something together and then someone like summarizes it and someone summarizes it for a slack message and like these steps further and further away, which look, do I want stakeholders to be able to interpret experiments and communicate it? Absolutely. Lutely. But I think sometimes that’s like just the the the understanding of like the core tradeoffs kind of gets watered down or there’s like different incentives and it just gets really tricky. Arik, I know like you’ve done a lot of thinking about this, about like once the analysis is out, the experiment result is out and you kind of lose control of the narrative, so to speak, like how do you approach that? 00:47:41.76 [Arik Friedman]: Yeah, so it definitely happens like sometimes you you get a chart or something that gets copy pasted and then someone puts it on a slide deck and it’s completely out of context and you lost that. And like one one thing that I try is again, like if you have a documentation 00:48:02.40 [Tim Wilson]: of your analysis, like you should definitely have a link to that. 00:48:06.28 [Arik Friedman]: So if it’s copy pasted, at least the link to the document is still there. And, you know, there’s the footnote in the chart where you can highlight the main details, so it’s definitely a question of a balance, right? Like you don’t want to overload your visuals with all the assumptions and all the details, it’s distracting, but at least have this kind of, you know, selective context or pointer to where the information is. So it kind of like travels together with the visuals. And, you know, sometimes, you know, you just lose control and you have nothing to do about that. But at least you can take some mitigating steps. So, you know, the evidence is there. Like if anyone is really like will want to find this information, there is a way for them to get there. So that’s the least we can do to at least control that. 00:48:59.24 [Tim Wilson]: Yeah. I mean, I think like pointing to like having the footnote of like, what was the what was the right with the source? I mean, you’re well stated that it’s like you don’t want to overload it with all the assumptions, but you want to provide the breadcrumb to say 00:49:12.88 [Tim Wilson]: and the more you can put that proximate to the chart. So they’re not likely, I mean, that’s pretty gross misbehavior 00:49:21.92 [Tim Wilson]: if somebody says, oh, there are footnotes on this slide, but I’m going to pull just the chart and drop it in an email. There’s a little bit of that’s on them if it gets its own legs. If you’re giving it like, hey, this is probably important reference information that should go along with it. I think that’s good. But this, I feel like we could go on for multiple hours, but unfortunately, we need to head to rap or we will lose trust with our audience by having a two hour episode, which when you’re called the analytics power hour, that would be a disconnect between data sources. No, that’s a that’s a stretch. But before we leave, the last thing we like to do on the show always is go around the horn and share a last call, something that might be of interest to our users. And Arc, you’re our first guest. Do you have a last call or maybe a couple last calls you’d like to share? 00:50:26.00 [Arik Friedman]: The first one is, I guess, a classic, the ICLR, like the introduction to statistical learning book, which is actually a free resource. And today, there’s also a version for Python introduction to statistical learning in Python. And, you know, at times when, you know, AI sucks all the attention, I think that actually going back to the basics, to the foundations is, you know, just as important as ever, if not more so. And I know that at least from my experience, even just going over the first chapters of the books, you know, linear regression, it’s like a very practical oriented book. So, yeah, that’s that’s a good recommendation that I usually provide. Like it landed for me. 00:51:12.96 [Tim Wilson]: They have an R version and a Python version with, like, examples on it. 00:51:16.40 [Arik Friedman]: Is that right? Yeah. So there are two versions of the book. The original one was with R. Oh, it’s ISLR and ISL. 00:51:22.44 [Tim Wilson]: And ISLB. Introduction to stuff. OK, gotcha. 00:51:24.68 [Arik Friedman]: And about the available at www.startlearning.com. And like I noted, at least for me, it landed a lot of concepts that I didn’t really get before. So I really recommend that. 00:51:36.80 [Tim Wilson]: And if I’m allowed a second and a second call, 00:51:41.84 [Arik Friedman]: which is maybe more related. So I saw recently an article from Hamel Kussein called Revenge of the Data Scientist, and it’s actually both available as a YouTube talk. It’s a Pi AI talk from March. And he also posted it as a Twitter article. And he actually talks about, I think it’s about, specifically about mindsets, you know, we have the mindset that you go with to, you know, in this agent-first world. And I think that actually his point is that the data scientists approach, their mindset, their core skills, like a exploratory data analysis, metric design, model evaluation, all these things are as critical as ever and try to translate really well to this world. So I definitely recommend giving this a read. And also Hamel Kussein and Trash and Carr had a terrific episode about AI evils in Lenny’s podcast. So that’s a great resource as well. 00:52:41.64 [Tim Wilson]: Awesome. So that was three, really, sort of. But those all sound amazing. I feel like I’ve read the first three chapters of like more. Those are the books I’m most likely to abandon, but still get a lot of value because it’s like chapter four, where I start to I’m like, what, we’re at the area under the curve. I’m like, oh, boy, oh, boy, the equations are getting pretty in-depth here. So I kind of want to check those out. Julie, what’s your last call? 00:53:12.40 [Julie Hoyer]: Well, my last call is accurate and precise. And I’m feeling like maybe I have called this one out before. But you know what? It’s such a good one that I’m just going to say it again. It’s called the Moenarch app. And I’ve actually been using it now for quite a while for my own family budgeting and financial like tool for the family spenders. And it’s really nice because you can connect everything directly into it. So on top of having all your different credit cards, bank accounts, you get really nice cash flow visuals. You can recategorize like any of your expenses that come through. And it will notify you like, hey, this is a recurring one or hey, this is one that’s not categorized. You want to come in and quickly do it. But budgeting is not easy to do on your own. And I feel like this is the first app and different thing I’ve tried that I can actually keep up with it. I can quickly get to it. You know, there’s no delay like I spent. It shows there by paycheck hits. It shows there. So it’s been awesome. You can also set a lot of your budgets and goals in the app as well. So like for all your different categories, you can customize categories or not. And then you can say, you know, I’m looking to spend X amount each month in each category. And it just, I feel like makes it a lot easier to actually live out the financial plan and budget that we’re going for. So check it out if you’re needing a tool. 00:54:44.20 [Tim Wilson]: Have you figured out a way to turn off the turn off the net worth? The words because because there are periods where I want to not look at that 00:54:51.48 [Tim Wilson]: when the yeah, I’m a monarch user. 00:54:56.44 [Tim Wilson]: So I’m a fan. 00:54:57.48 [Julie Hoyer]: But you are. So you guys, it must be good if Tim’s into it. If he approves of the visuals and the tools. 00:55:03.24 [Tim Wilson]: Yeah, judge, judge your trust in the source. Moe, what’s your last call? 00:55:11.20 [Moe Kiss]: I know we talk about her a lot. But Cassie, Cassie has her cough has a really great new. I mean, it’s just her regular newsletter, but she’s doing a couple of back to back newsletters on why the vibe coding will bite you. And here is exactly where and she’s talked through a few like text scenarios of where things have gone really wrong. Like prod systems being completely white, things like that. So definitely go have a listen to that. But I think the thing like the real takeaway is that the stories are all about the same thing, which is just like misplaced trust and the speed at which it computes. And so it’s not like the nobody got hacked. AI didn’t go rogue. It’s just like people let their guard down. And I think the thing that’s been on my mind the most, which she’s so she’s just so damn articulate, but it’s like expertise won’t save you guardrails might. And I think guardrails is the topic that just keeps looping on my brain at the moment. And so, yeah, I’ve really enjoyed that newsletter and she’s got a couple more coming out on the same topic. So check it out. 00:56:17.64 [Tim Wilson]: I think that series like motivated me to dive into some new vibe coding project specifically. So far, I’m safe. I haven’t crashed the podcast because that’s where most of them happen. Starting to wonder if Riverside, our podcast recording app might actually be leaning a little too much on vibe coding is it’s having some of the joys it’s been bringing us of late. But so I’m going to pander to mo here a little bit because with the new season of choiceology that Katie milkman came out with. Um, one of the things she has is this checklist, which is this mapping of now she said she had a couple of undergraduate students do it. And that means it’s like in a PDF for some insane reason, but she’s basically gone through and looked at like, what are the different topics like attribution bias or Dunning Kruger or left digit bias, like the stuff that her episodes cover, this kind of reverses it. And it’s a guide to broken down by these are all the topics of kind of 00:57:20.56 [Tim Wilson]: cognitive biases, um, then which are the episodes you can actually listen to. 00:57:26.52 [Tim Wilson]: So if you’re not a, um, Katie milkman choiceology completionist, like I am, but you’re like, I wonder if she’s ever had an episode about, um, you know, mean reversion neglect, then this little guide will pop you to it. It is comically in a PDF, which I’m like, this is great. You guys really work to format this thing to one page. And she’s about to start a new season, which means this, this of all things that should be a vibe coded website where it gets updated and maintained. This is it, but you know what the undergrads, they’ll learn. That’s how it was scoped academia. They’re going to do their little thing. So, um, with that, um, our thanks so much for coming on. I feel like this is a case where we actually have like the show prep documents that have like even more gold in it that we were not able to get to. So, so we will have a lot of fun with that content our own ourselves. We may figure out how to bring you back for more of that. So thanks so much for coming on. 00:58:31.64 [Arik Friedman]: Thank you very much. Awesome. 00:58:33.76 [Tim Wilson]: So if you, uh, listeners, um, we love to hear from you. So if you’d love to have you leave a review or rating on whatever platform you listen to us on, if you’d like a free sticker of the podcast, uh, you can go to analytics hour.io and request one. I will, if you’ve gotten this far in the episode and you think, wow, that was a smooth conversation and these guys are professional, I will just call out now that we have dealt with, uh, tornado warning. They led to a power outage and two young children and a dog sheltering in place and with one co-host, we’ve dealt with a busted internet, um, that has been busted for the entire episode. But of course the repair team showed up for that during the episode. Um, and we’ve dealt with various cases of people dropping off and returning and not even realizing that we were still recording the show. So I encourage you to stick around for the outtakes because there might be some real doozies in those. Um, I would like to Tony, please leave this in. Thank you so much. 00:59:39.92 [Tim Wilson]: Cause if you pulled this thing together, you’re a good on your mate. 00:59:45.28 [Tim Wilson]: Uh, so it’s been fun. It’s been a fun discussion. Uh, we’ve been at this for four hours to get this one hour pulled together. No, it hadn’t been quite that bad, but we would love to hear from you. We’d love to hear if you thought that the edit was pretty smooth. If you’ve got your own thoughts on how to build, maintain, recover, trust, uh, you can reach out to us on LinkedIn. You can reach out to us on the measure slack. You can just send us an email at contact at analyticshour.io. So for Julie, for Moe, for all of the conspiring mother nature and construction projects that tried to not allow us to record this show about trust and accuracy 01:00:31.72 [Tim Wilson]: and precision, keep analyzing. 01:00:34.80 [Announcer]: Thanks for listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at, at analytics hour on the web at analyticshour.io, our LinkedIn 01:00:46.24 [Tim Wilson]: group and the measure chat slack group music for the podcast by Josh Crowhurst. 01:00:52.36 [Announcer]: Those smart guys wanted to fit in. So they made up a term called analytics. Analytics don’t work. 01:00:59.04 [Charles Barkley]: Do the analytics say go for it? No matter who’s going for it. So if you and I want to feel the analytics, they go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. Fuck my life. 01:01:15.48 [Moe Kiss]: So the work that’s being done out the front of my house. Knocked the NBN cable out. 01:01:24.48 [Tim Wilson]: I’m a rap now. 01:01:26.12 [Moe Kiss]: Oh, shit. Were we still recording? Oh, fuck my life. 01:01:33.84 [Tim Wilson]: I had to smoke this, but I was, I was trying to make so many notes that I, uh, on other aspects that I doubted. 01:01:40.56 [Moe Kiss]: There’s a lot going on today. Oh, my God. Well, you gave me so much. 01:01:46.56 [Tim Wilson]: I just, there was no writing. 01:01:48.88 [Julie Hoyer]: Nobody has the heart to tell you if I’m like, well, then the second one, one is like, so can I wrap? Oh, guys. All right, guys. 01:02:03.32 [Arik Friedman]: I definitely want to take Tony in advance because like, yeah, like, that’s probably some work to do here. Yeah. 01:02:16.12 [Julie Hoyer]: My hot mic was the least of our concerns. I don’t know if you would have thought, Jesus. 01:02:21.08 [Tim Wilson]: Yeah, but I mean, I think it’s, I think it’s going to come together well. 01:02:26.04 [Tim Wilson]: And my mid-show recap was like me organizing my thoughts because I was like, 01:02:29.76 [Moe Kiss]: I think we’re actually hitting on some, I actually feel like Arik, we need like another two hours with you because there’s so much stuff here. It’s such gold. 01:02:39.40 [Julie Hoyer]: Seriously. Yeah. You had so much in the show prep doc that I was like, oh, I want to talk about that. 01:02:45.32 [Moe Kiss]: Oh, and this is why I was like, I knew that the two of you would really like 01:02:48.56 [Tim Wilson]: Arik because like you’re the same with the very good at prep and 01:02:53.80 [Moe Kiss]: organization and all those things. I literally bought a new microphone and it’s still. Why have I still got this shitty one that doesn’t even have a proper stand and it still works great and everyone keeps buying all these fancy ones. It sucks. 01:03:08.16 [Julie Hoyer]: I got to return my hundred dollar one, I guess, and try a twenty dollar one. Maybe I need it to be less sensitive. 01:03:14.64 [Tim Wilson]: I think the show might need to buy you an audio interface. I think it’s I don’t don’t don’t don’t make any moves. Don’t do anything drastic. 01:03:25.12 [Moe Kiss]: Yeah, the audio interfaces and changing microphones. 01:03:33.04 [Tim Wilson]: It is because it’s moving where the preamp is. It moves it into its separate. It moves it from a little thing, crappy thing in the microphone. It then takes it out and puts it in a dedicated box. 01:03:44.40 [Tim Wilson]: OK. No, this is so above my head. 01:03:48.20 [Julie Hoyer]: I just want to buy a microphone that works. 01:03:59.24 [Tim Wilson]: Rock flag and accuracy versus precision. The post #296: Avoiding Major Oopsies: Twyman’s Law, Intuition, and Valuing Accuracy Over Precision appeared first on The Analytics Power Hour: Data and Analytics Podcast .

April 14, 20261 hr 9 min

#295: Research and Analytics: the Peanut Butter and Chocolate of Data?

Research and analytics: are they more like peanut butter and chocolate, or more like oil and water? On this episode, we dig into the surprisingly common (and surprisingly unfortunate) divide between these two disciplines with Stefanie Zammit , Global Director of Analytics and Insights at Bang & Olufsen. Stefanie has spent her career bridging the qual and quant worlds, and she makes a compelling case that the best insights come from putting both methodologies to work on the same business problems. From the “never ask a survey question you already have the answer to” rule to why personas are usually terrible (spoiler: it’s not the clustering, it’s the storytelling), we explore how organizations can break down the silos between research and analytics teams. Turns out, the fear of the unknown and a bunch of fancy terminology might be keeping us from some pretty powerful insights. Also, apparently 100% soundproof rooms are absolutely terrifying. This episode is brought to you by Prism from Ask-Y —your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show The Practice of Market Research: An Introduction Training crucial leadership skills through serious games The Bang & Olufsen Factory Tour Women in Research (WIRe) Women in Product How Quantum Computing Works How to Make Sense of AI Photo by Vardan Papikyan on Unsplash Episode Transcript 00:00:00.00 [Announcer]: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. 00:00:13.24 [Michael Helbling]: Hi everybody, welcome to the Analytics Power Hour and this is episode 295. You know a common phrase we hear in our industry is that the data tells us what happened, but not necessarily why. And by and large that’s true. We keep getting better at inference and some patterns of data are pretty well understood in terms of their meaning, but still there is simply something so compelling about observing how people interact with the things we’ve built, websites, products, etc. Personally I remember what an eye-opening experience it was for me 15 years ago, the first time I was sitting in a usability lab watching from behind a one-way mirror as people use the website that I measured every day with my digital data. So we wanted to talk about it, get into the topic bridging between research and more traditional analytics. And I want to introduce my co-hosts, Val Kroll. Welcome. 00:01:12.56 [Val Kroll]: Hello. 00:01:13.56 [Announcer]: Hello. 00:01:14.56 [Michael Helbling]: And I know this is a special topic for you because if you’re background into customer research. 00:01:20.08 [Val Kroll]: Oh my gosh, doing back flips. Very excited for this. 00:01:22.08 [Michael Helbling]: Yeah, I’m excited too. And Julie Hoyer, welcome. Hello. Hi. Have you done much like market research or customer research? 00:01:32.04 [Julie Hoyer]: Not myself, but I have gotten a chance to like utilize the outputs of some of those studies, which has been nice. So I’m really excited to talk about it more. 00:01:40.36 [Michael Helbling]: Yeah. Excellent. All right. And I’m Michael Helbling. And to bring additional expertise to this topic, I’m pleased to introduce our guest, Stefanie Zammit. She is the Global Director of Analytics and Insights at Bang & Olufsen. Prior to that, she led research and analytics teams at companies like Starbucks and Marks and Spencer’s. She’s worked both as a consultant in the space for many years as well. And today she is our guest. Welcome to the show, Stefanie. 00:02:05.24 [Stefanie Zammit]: Hello. 00:02:06.24 [Michael Helbling]: Happy to be here. Awesome. We’re so glad to have you. And I think this is a topic that while we cover sort of like data and analytics, research is one of those things that we really like. And so we’re really excited when we met you to sort of dig into this topic more. But to kind of catch everybody up to speed, I thought it’d be great to kick off with just you explain a little bit about your background and career and kind of how it bridged these two things and sort of, you know, what your journey has been across research and analytics. 00:02:40.40 [Stefanie Zammit]: Yeah. Absolutely. I’m very much from a pure hardcore research background. That’s where I started my career too many years ago. I started as, actually, my first job was at university. I didn’t even know what research was. I took a part-time job doing the national student survey. That’s like a thing here in the UK. It’s how the university rankings are put together. And when I graduated, I think a lot of researchers would have a similar story to this, that they sort of ended up accidentally in the field. And so I graduated during a recession, dark times, and I was desperate. And I thought back to that part-time job I had at university and like, what is that? Like, is that an industry? Is that a thing I could do forever? So I did some research and I was in the UK at the time where, luckily, there’s an amazing research industry here. There’s so many great consultancies. It’s a thriving industry. I took my first job at a company called Quadrangle, which was a management consultancy that leaned very heavily on their own in-house research function. And it was amazing, an eye-opening, and I immediately fell in love with so many aspects of it. I then went to Ipsos, which is one of the big five. That’s where I was like, I need some hardcore research skills. I need to learn about statistics and hit me with the heavy quant stuff, go to a big powerhouse. So I was there for a couple of years. And then after that, I built my own. I co-founded a research consultancy at this time I was in the Middle East, was a company called Intelligence Qatar, which is still very much there today, although sadly, I don’t get to play a part in it anymore. And I think that was where previously at research consultancies, they really divide the teams up. So you had your market researchers, and then you had your analytics department with all the smart people, the statisticians were all there, then you’d have your fieldwork teams, your data processing teams. And it was very siloed, and even market research was pretty siloed. You’d have your qual team, and then your quant team, and something that was really hard for me as I progressed agency side, you’d have recruiters say to you, like, do you want a qual role or a quant role? And I struggled so hard to give them the answer because I genuinely loved both. And my favorite projects were the multi-phase projects where you’d get the best of both. But I was also super kind of looking over the shoulder of my analytics colleagues and statistician colleagues, what are you guys up to? What are you doing? So I always naturally was interested in the entire spectrum. And then when I had my own agency, again, really investing in the analytics side, as well as the research side just brought me closer to those worlds. And then finally went client side, I thought, all right, I had enough of this consulting game, joined Marks and Spencer’s. And I was really lucky that they had research analytics together in one department. That’s all I’ve known because it was like that in that first company. And we had a great leader at the time that was very adamant that the best deliverables are worked on by both of these teams. So I took that to Starbucks as well, where again, the organization is huge in Starbucks, you’re looking at around 250 people, but they are all in the same department together, research and analytics, if subteams, at least it’s still in the same family. So it’s something that I’ve been so passionate about today, and I think really set me up for success to do what I do now, which is to lead both the analytics function and the market research function in one team. 00:06:09.76 [Val Kroll]: I love that. The one thing that you mentioned there about not being able to pick which you liked better, the qual or the quant in your favorite, where the multi-phase projects that start with the qual and then lead into quant, I feel like that’s like a little bit novel, like a little bit inside baseball for researchers. Can you describe what that is because I have a follow up question after you talk about that a little bit, but just kind of describe like why someone would have like a multi-phased approach to like their research project? 00:06:40.60 [Announcer]: Absolutely. 00:06:42.20 [Stefanie Zammit]: So every methodology has its benefit. Qualitative research is where you start when you don’t know much about a subject and you need to explore. It’s very exploratory. You are using it to figure out where your hypotheses even are. You might have one or two hypotheses, but they’re fluffy and you need to explore the topic more. So qualitative is where you do that in this kind of limitless way of a very open data generation phase. And then you need to validate because you’ve spoken to like, I don’t know, max 40 people if you’re doing a really big qualitative project. So you want to validate that and you need some statistics and some numbers behind it. So you want to do your quant. So I have my hypotheses now, I’ll run a survey to measure and size those truths and see how statistically significant they are. And methodologically, these require two different expertise because qualitatively you’re trained in moderation, projective techniques, how to read between the lines, how to read people’s faces and emotions and hear what they’re not saying as well as what they are saying. There’s also a much more kind of deep psychology to interpreting those insights because, again, you’re reading between the lines. Quantitative, you need to know about driver’s analysis and cluster analysis. In fact, you need to know all of the statistical models that give you the derived insights that are so, so valuable from a survey. So they are usually kept separate. And I think that’s a shame because the best projects are the ones that do both of these things for a really strong final insight. 00:08:21.36 [Val Kroll]: Yeah, that was very well explained. And I remember in my market research days, a lot of times, if you think about if you were to do a survey, you have to, a quantitative survey, and you’re picking the list of options that someone is going to select that is the right answer to the question for them, sometimes that list isn’t always as clear, like what should belong in it, even if it’s a list of competitors. A lot of times, clients will think about in-category competitors, about who your competitors are for an alcohol brand. But that same dollar could be spent on other things that are out of category competition. And so you can use QAL to help explore to figure out what even is the list, because you could miss so much if you start just with a quant without going broad first, to be exploratory, like exactly, except Stefanie, to figure out what your hypothesis is. And the reason I want to dig into this a little bit is because this is one of the ways that I love helping illustrate or describe, especially to people describe, to people in the analytics, some of the value of bringing these worlds together. Because it’s not just using one single methodology or tool. It helps you illuminate a different part of your question or your process. And so how those two things come together very naturally inside of research is one of the ways you can kind of illustrate the coming together beyond just the research or the direct consumer or B2B context. 00:09:49.28 [Stefanie Zammit]: Exactly. And I think from a research-only perspective, I was already at a very early stage so powered up by the idea that if you put these two together, you’re getting better insights because you’re starting broad and then you’re getting specific with the quant. And I took that into as I gained more seniority in my career and started working, especially in-house, where you have colleagues in other disciplines, which is all data and insight. I took that into that as well to say, well, how can analytics be part of this? And why are we asking questions and surveys that we already have the answer to? That seems like a huge waste of time. How can we be using all these disciplines to get the best insight? And all of this ultimately comes down to a passion to chase the best insight, right? Methodology should be irrelevant to it’s not about the journey, it’s about where you end up. And I think having that crystallized in my mind from the very beginning really helped me see the world in the way that I see it now, that who cares what you’re trained in. At the end of the day, we use the best method to get the best insight. And it doesn’t matter whether that’s in this team or that team. 00:11:01.92 [Julie Hoyer]: And I feel like you are one of the more rare data leaders out there that recognize it has to be problem focused instead of leaders coming to their teams with ordering a solution. They have a problem, but they’re not always communicating that to the teams that are going to help service them. They’re kind of coming with, well, I want you to pull these numbers or ask these questions type thing instead of to your point just anchoring on, I don’t care how you do it, you guys are the experts in that part. But what I’m facing is problem X and I’m looking for ways to solve it. And then letting people get creative with how do they maybe partner together to get them the best solution? I feel like we just run into that at kind of like all levels of the business. We’ve all had different experiences with like trying to overcome that hurdle. So it’s really refreshing to hear you so eloquently like talk about it in the way you frame it. 00:12:06.64 [Stefanie Zammit]: And 100%, that’s so normal everywhere. I’ve experienced it everywhere that you have stakeholders coming to you. We want to do some quality. We want to run a survey. I need a dashboard. I need they they’re very prescriptive. And I think it’s part of our job as insights folks, irrespective of our training to say, whoa, whoa, hold on there. Let me understand what you’re trying to do. What decision are you trying to make? The answer might actually not be in a dashboard. It might be in a piece of custom analytics or it might be. So our consulting work is a big part of this job to find the best methodology for the best answer. We can’t expect our stakeholders to know what that is, although they’re very welcome to make suggestions, of course. But it’s a broad broad spectrum of tools that we could use. 00:12:54.40 [Julie Hoyer]: Yeah, absolutely. And one of the questions I’ve been dying to ask you is, why do you think, and Val, I’d be interested in your take, too, because you’ve kind of lived in both of these worlds as well. Like, historically, why don’t research teams and analytics teams always play together at all? Or if they’re playing together, they don’t always play nicely together? Good question. Get into it. 00:13:20.64 [Val Kroll]: Stefanie, you have to go first. 00:13:22.72 [Stefanie Zammit]: It’s a great question. And I think the answer is fear. I think that there is the fear of the unknown. And there is an assumption that the other world is so mysterious and so different to our world. We don’t understand each other. It’s literally a form of othering within teams, within departments. And it’s a fear I myself had. I could never keep up with these data scientists and, oh, they’re just so smart. And they understand all these things in a way that I never could. And then you start working together. And you see that you actually have more in common than you do have differences. And they see it, too. They understand, they can learn from the research process and understand, oh, hey, I thought research was just qual. I don’t know if you’ve ever heard that, any of you guys, especially Val, that the assumption that research equals qual, that’s the qual work. And so I just did a survey with 8,000 people in this hardcore conjoint statistical model. You can’t call that qual. But there isn’t an understanding. There isn’t enough knowledge. And everyone assumes that the other side of the coin is so different. It’s a whole different world. And I think it comes from the way that agencies are set up and departments are set up to separate these skills when actually they’re stronger together. 00:14:43.36 [Tim Wilson]: Michael, why does every quick question come with a 20-minute origin story? Well, that’s because our metrics have, I don’t know, lore. 00:14:53.76 [Michael Helbling]: Conversions might mean three different things depending on who’s presenting and how close we are to the next quarterly board meeting. 00:15:00.72 [Tim Wilson]: And I mean, every time you switch tools, you have to re-explain the lore like you’re residing ancient prophecy. 00:15:07.20 [Michael Helbling]: On the seventh day of Q3, the trekking broke and lo, that metric was doubled for July. 00:15:12.80 [Tim Wilson]: That’s why we’re excited about askwide.ai in Prism, because it has memory that actually remembers. You don’t have to repeat the lore. 00:15:21.76 [Michael Helbling]: Yeah, the jam system keeps context across sessions what your org means by revenue 00:15:27.84 [Announcer]: or conversions, which tables the source of truth. 00:15:31.36 [Michael Helbling]: And the weird exception, you’ll always forget until it’s too late. 00:15:34.56 [Tim Wilson]: Plus, you can save your best workflows as skills. Portable expertise you can actually reuse like a human. 00:15:42.88 [Michael Helbling]: So I don’t have to manually normalize UTMs or fix or tweak that GA4 channel grouping or deduplicate leads without breaking down into tears. 00:15:57.04 [Tim Wilson]: Yeah, so instead of rebuilding the same process like every week. 00:16:01.28 [Michael Helbling]: Yeah, I guess I just run the skill and go about the rest of my day. Kind of happy. 00:16:07.84 [Tim Wilson]: So one end, go to ask-wide.ai and join the waitlist. It’s in beta, but you can get in on the ground floor. 00:16:14.96 [Michael Helbling]: Yeah, and if you use code APH, you’ll get pushed to the top of the waitlist. That’s ask-the-letter-why.ai and use code APH. 00:16:26.16 [Announcer]: All right, let’s get back to the show. 00:16:28.56 [Val Kroll]: And where you landed on that, I think is one of the drivers that I see. The drivers that I see in my mind is if you think about how these two disciplines grew up in the world, research used to live within marketing. It used to be marketing research. And so we would sit within marketing teams. All of my clients back in the day were CMOs or reported up to the CMO. My first web analytics job, I was in IT and it was very technology heavy. It was about the tools and the way the data was collected. And like surveys, people think like, oh, like pen and paper, like mailing surveys or someone chasing you down at a mall with a clipboard. Like, would you like to take a survey? There’s so much more to that, like with panels and different ways that you can contact people nowadays. So I think- Limes have moved on. Yes, yes. There has been an evolution, yes. And so I think it’s just kind of been just how it grew up, 00:17:25.04 [Announcer]: was kind of thought of differently, budgeted for differently, like research, 00:17:29.52 [Val Kroll]: I think has always has this like, wrap a little bit. I’m interested in your thoughts on this too, Stefanie. That like a lot of times it can be like costly, like not just dollars, but time. So that it takes a long time to, you know, every, you know, we only do the brand tracking study once a year because it’s such a big, you know, piece of research or like it’s actually not valuable to keep track of that on a more frequent basis where a lot of the costs from some of the other analytics practices or areas are some more hidden costs because they’re in the technology or actually the human solutions. And so especially when we’re talking about like in-house, I think that it’s just budgeted for differently. And so people aren’t really like connecting the dots. 00:18:08.16 [Announcer]: But I think the organizations where you can break down those silos, 00:18:11.36 [Val Kroll]: because I’ve actually never worked for a client that’s had the setup that you’re talking about, Stefanie, which would be like, oh, Nirvana to have them like coming together. But I always found ourselves making the suggestion about like, did you talk to that team? And they’re like, who? And they’re like, well, I was scrolling through your active directory and I found this person with this title, like you should reach out to them to see if they can help us. But yeah, so that’s kind of what I think is like part of the rationale that I think that I do hope that this is like a movement though, like this evolution towards thinking more flexibly about the methodologies and what’s the right fit for the question at hand and what’s going to serve the business best. 00:18:50.80 [Stefanie Zammit]: Yes. And I should add that the Nirvana, I may be making it sound more Nirvana-like than it actually is. I mean, again, when you look at huge organizations like Starbucks, which it’s just such a math, there’s thousands of people at head office. Although we were all in a department together, the reality is that the silos were still very strong. At the time when I first joined Starbucks, I was in service to the loyalty team. So the rewards program and the app, you know, and how our customers use the app, et cetera. And I realized that I was trying to serve these stakeholders with insights about app usage and loyalty program behaviors and all the rest of it. Meanwhile, you had folks in analytics who were also answering questions for the same stakeholders. And there was just such a clear overlap in, we’re both talking about behavioral data, we’re both talking about what the client wants, the customer wants and needs. And so what we did was we formed a little, within the department, a little community of people who are in service to this stakeholder group, irrespective of where in the mega data analytics and insights department you are, we come together and we talk about all our projects. We formed a little, I don’t want to call it a Sierco because I hate the word Sierco, very anti-allergic. Oh my God. But just like a little forum of round robin, what’s everyone working on? And then we can say, oh, you’re doing that. I’ve got a survey coming up which directly overlaps with the objectives of what you’re looking at. But I can tell you why and you’re measuring what. So why don’t we put them together and, hey, our stakeholder will get something that’s more complete and less confusing rather than 10 different reports that all overlap, but the actionability is lost because we’re pointing in different directions on similar 00:20:43.12 [Michael Helbling]: and yet not quite the same topics. That’s crazy because I literally have built something almost identical but coming from the data side to the research side at a company I used to work at, which was forming this little team that we met and we’re saying, okay, let’s get together and get all of us together so we have a coherent story. And it’s always stuck with me that why is the organization having to be managed sort of bottom up in that regard when the reality is the structure or the layout of the org should be thought through to enable that kind of capability from the very top. And it’s just one of those things that sort of sticks out as a sore thumb. And I don’t know if I have prescriptions for that, but I will say it’s like Val, you mentioned we walk into a client, you’re kind of looking around like where’s your research team and why aren’t we talking to them too, which I think is really apt. But also like certain companies like you’re going to do analytics work and there is no research team or nothing named as such. And it’s sort of like a lost function or a missing function. And it might be kind of Julie to your earlier point, the leaders of the org just sort of think they’ve got it figured out so they don’t need someone to sort of like think about what the customer actually thinks because they’re thinking for the customer, if you will, not a great plan. But anyways, I’m just curious, Stefanie, because you’ve done some consulting in this space as well, like how do companies sort of jump the chasm first from just not even having a concept for doing research like this? Because everybody’s got analytics, like we’ve all got snowflake and data bricks or something running the back end of all of our data, but a lot of companies have zero going on in terms of like either customer or market research. 00:22:45.76 [Stefanie Zammit]: And if they do, they outsource it to agencies. So they would do a one-off project that an external company will run. And this is where you write it as complexity, because it’s rare that a company would have a full function in-house research team, because the manpower that you need on a per project level is huge, right? We’re talking thousands of people like, okay, not thousands, but at least 50 going out, interviewing in different countries and then your data processing and then your statistics. And then so the per project value for money of that much manpower is just not worth it. So they outsource to agencies who have economies of scale. Those agencies then in turn don’t think to ask, hey, do you have a data team? Do you have an analytics team? They sell analytics. Why would they say, hey, we’re going to build a segmentation for you. We’ll do all the research, but we’ll hand over to your in-house analytics and they can do the clusters. Like they’re never going to say that. They’ll be like, yeah, we do end to end, you know. But so I think that’s the challenge. And what I would advise organizations to do, to mitigate against that, is to have even just one person. At Bang & Offsen, we have one person. He’s a superhero. He’s a one-man research department. One person to coordinate all research projects and you can still use vendors, but you have an internal knowledge bank being built and internal consistency even. And then that person knows to work with analytics and to blend the two together while also getting the economies of scale from the agencies. 00:24:21.92 [Val Kroll]: Closing the chasm, I want to spend a little bit more time on this and your thought of making sure you have someone to represent that perspective versus like, does anyone want to add any new questions to this year’s tracker? Like it has to be so much deeper than that for it to be meaningful. But my first boss, when I was in market research, she grew up, Lynn Bartos, if you’re listening, she grew up at Burke. And so she was like hardcore, like you were saying, like having those skills. And one of the things that she always talked about was that she spent two weeks touring all the different departments, like someone who sat within, you know, data processing, someone who sat within coding for all the open-ended responses to get an appreciation for the operations and like how the sausage was made, because that makes you smarter when you request your banners or when you think about like, you know, how do I develop the questions to get me to a driver’s analysis or things like that. And so we did that ourselves on her team. And I really had an appreciation for that. Do you think that the closing the chasm is cross-training people so that they have like a better appreciation for each other’s skills, like when they do both exist in-house? Or is it more, you know, pizza lunches, like dog and pony show of like results? Or because I love what you talked about of having the little non-steerco-steerco, that’s aligned to a stakeholder group, like I would have never thought of that. But I’m just wondering if there’s some other things, unfortunately, like you were also saying, Michael bottoms up that people could do to kind of close this chasm between the team. Like what things would you recommend to listeners for who have an interest on the other side? Absolutely, 00:26:03.12 [Stefanie Zammit]: yes. So for anyone listening who’s not managing a team but is in one of these roles, whether you’re in data science, you know, data engineering, research, whatever it might be that touches on data, I’d really recommend the best that that person can do for their own career is to have a natural curiosity for the methodologies and the training that the others in the department have. And I always say to people in my team, if you find yourself in a conversation where you have no idea what people are talking about, or it feels like scary or just very different to what your expertise is, that is where you will learn that you should lean into those conversations. We should go out of our way to understand, not in an annoying way, like, hey, dude, what are you doing every day? But just to have a natural curiosity of how does your work fit with my work? And that’s not even just for analytics and insights or for data. That should be for everyone in a corporate job working for a brand where we serve our customers. We should all be understanding how do our worlds fit together for the customer. But especially within data, because data is the customer, we represent the customer, have that curiosity. And for anyone who’s listening who’s in a leadership role, then yes, I really recommend fostering that within your teams, 00:27:24.16 [Announcer]: that natural curiosity, and getting people to take a moment to question if anyone else in a data 00:27:32.24 [Stefanie Zammit]: related role can contribute to the project to add further insights. So would the data science team know anything here? Maybe they don’t have a deliverable, but maybe from their investigations or their work, they would have context that would add valuable insight to my work. Would the research team maybe have had a project about this? Maybe not, but again, the data folks are in, you know, they’re hands dirty in the data every day. The amount of information that they process and that they gain exposure to, which is not reported ever, is huge, right? We could never report all the facts, but it’s there, it’s in their heads and in their experience. So it might not be reported anywhere, but that’s a good person to talk to because they would have context. Same with the researchers, you know, they’re out conducting intercepts, ethnography focus groups, not everything makes it into the final report. But if you sit down and talk to each other, you realize, oh, yeah, I know something about that. I remember seeing something about that. I have a good quote that brings what you’re doing to life, whatever it might be. See, I really encourage curiosity. My first job, I started in QUAL, actually, which is like the furthest away from data science and analytics. And I was so scared to move toward QUANT. And I got reassigned to a tracker. So I went from like the QUAL team, you know, ultra open to a tracker, like 100% only ever working on this one tracker. And at the time, I was so grumpy about it. I was just like, this sucks. Like, 00:29:03.20 [Announcer]: where’s the creativity? You know, where’s the art? But actually, it was the best thing that 00:29:09.52 [Stefanie Zammit]: could have happened to me, because I didn’t know anything about tracking. And I, you know, I would have been pigeonholed if I’d have just followed my natural heart. It forced me to learn about a different world. And then I realized, actually, this is interesting. Actually, QUANT. QUANT is interesting. Look at that. Who would have thought? Tracking is actually interesting. There’s insights here. But unless you’re kind of forced into it, or at least when you’re young, you need to be forced a little bit, it’s difficult to just naturally expect that you would find these other worlds interesting. But I guarantee if you really, you know, scratch at that or, you know, peek into those boxes, you will find a lot of very interesting things that will help your own role. 00:29:50.88 [Julie Hoyer]: You touched on this a little bit earlier too, Stefanie. And I’m curious. So when you’re leading your team and you have this great point of view on how these things can work 00:30:01.84 [Announcer]: together. And obviously, we talked about trying to encourage the curiosity of everyone on your team. 00:30:08.00 [Julie Hoyer]: But are there some more like formal processes that you’ve also put in place for your team, or how you guys pick up projects or execute projects with your stakeholders that really help these come together in the best way possible? Because you mentioned earlier, like the, you know, starting with the, the qual and then you get a hypothesis, then you follow up with quant. So I wanted to like dive a little deeper in that area and hear what some actual harder, like boundaries or processes you utilize to help. 00:30:39.12 [Stefanie Zammit]: Yeah, 100%. And there’s so many examples, I’m going to try to stay focused here. So first rule of thumb, no survey asks a question that we already have the answer to. And the only way to know that is to go and talk to the data team and know what we have the answers to. So just as a rule of thumb, and even from our customer experiences, right, customers shouldn’t be telling us like their demographics, if they’re signed up to us and, you know, we should know who they are. Second rule of thumb, every research project, a sub sample, even if it’s not in your objectives to interview clients, even if you’re, say, you’re doing a new customer acquisition piece. And, you know, you want to go out to like a purely external sample, you don’t want any internal sample. Even if that’s the case, a subsection of the survey respondees should be from internal customer, known customer sample. And especially if you have a segmentation or you have certain key questions or certain key profile, you know, data points that you want, you want to continue building that knowledge internally, use your surveys in that way. So for example, we’re doing segmentation work now, we’ve designed, we’re starting, we actually started with data. So what do we know about our client? To what extent can we profile them before it becomes a mystery? Right. That’s the point at which now we take it to research, we fill in the blanks with research, but we, we interview as much of our own customers as we can. So that then once the survey is complete, we bring all that data enrichment back in house, it’s tagged to non customers. And we can use both worlds to create the segmentation. So you have your attitudinal stuff, you have your profiling that you would never be able to know just from data, and you have your behavioral data as well. And we have this amazing analytics team, they can do the segmentation. We don’t have to use an agency for the entire thing. So we’ve saved money, you know, we know where we can stop the agency to make use of our internal skills. Now we’ve got money left over for a different project. Great. Let’s go do some call with these segments and get to know them, bring them to life, do ethnography, take video. Now when we go out to our stakeholders, we’ve got our segmentation, it’s attitudinal and, and behavioral. We’ve got all this great qualitative bringing them to life. Right. So, and this goes for every project. So you’re doing a piece of work with drivers analysis. Okay, great. We might do a survey, you know, you do your usual like drivers analysis. But then let’s say, okay, how can we, some of that survey was internal customer sample, how do we bring that back to the analytics team and say, well, we want to grow. So let’s identify these people and do an internal business drivers analysis based on the learnings from the survey to see if we can replicate those drivers in our data. Lo and behold, you’ve got an internal business drivers analysis that you can now track because it’s in our data. So how if we move the needle, did we actually grow, we can actually say, yes, that was that insight was successful. And we did actually grow. There’s many more examples, but wherever we can put both worlds together in a single project, I absolutely recommend we do. The other thing is communities. If, if you’re a business that’s lucky enough to have a customer community, which is a qualitative research tool, it’s like a panel of customers that you can do quick polls and surveys, it’s like a social media for your customers. And they, you get so much qualitative insight. Those communities, the best versions of those are built on top of internal data lakes. So you can follow the strings down to who these people are and how they are transacting. Every project you run in your community now has a behavioral data trail to look at, okay, we’ve got insights, the business made a decision. Now you can measure the impact of that decision because we can track these people. Did they actually spend more? Did they actually convert? So it’s, the possibilities are endless. And it honestly all comes from recognizing that we’re all after the same goal here. And, and especially research quant, research quant have analytics teams. They’re doing, they have very similar backgrounds to in-house analytics teams. Like I said, there’s more similarities than there are differences, but the in-house analytics teams might not naturally be tasked with, we’re going to run a con joint or we’re going to do, you know, things like Max Diff, which is research analytics. It’s not really as well known methodology in in-house analytics, but they can learn why, you know, why shouldn’t we bring those tools to in-house analytics. And then it’s interesting for the analytics teams to learn these methodologies as well. Over time, you save money because you’re spending less on external agencies. That’s awesome. I love those. And you have better data. Yeah, I love the rule of thumb. Yeah, I feel 00:35:22.80 [Julie Hoyer]: like it’s an accelerator the way you’re talking about. I kind of hate the word flywheel, but it 00:35:27.60 [Stefanie Zammit]: makes me think of a flywheel. But the researchers also need discipline. Like they, they need to be really close to in-house data analytics reporting. Like it’s amazing how many researchers I’ve met that have never used like the dashboards, you know, they’re not in the BI at all. And like, why wouldn’t you be, again, as a rule of thumb, if you want to conduct good research, you need your sample to be representative of your customer base. How do you know what that looks like? Well, there’s BI reporting that shows, you know, you should be feeding that to the vendors to build the sample plan and the waiting plan. So it’s all connected. It’s all one thing. And I think when 00:36:07.12 [Val Kroll]: you’re talking about this connectivity too, you can be smarter about like, you know, even breaking it up like an example, especially if you have the panel or if it’s a known population, instead of asking like, amazing, like how likely are you to buy this again over the next six months or how often, you know, I remember working on advertising awareness research for a cruise line. And they would always ask like, how likely are you just to plan a cruise for you and your family over the next year? I’m like, over the next year, like these people, like they don’t know what they’re doing, they don’t know what they’re having for lunch. Like, why are you asking over the next year? Like, let’s look, there’s got to be other data for this. But in the same way, like there’s so many people who will be building out a fallout report in Adobe Analytics, like looking at them, like trying to discover the friction points and like coming up with the why like on their own, like, oh, they couldn’t make it to this next step because and it’s like, well, did you ask them if that was part of the friction because usability labs, like, you know, pop up surveys, there’s like so many different tools or ways that we can connect with the customer nowadays that, you know, not trying to fill in the blanks, like there’s a lot of different ways that we could just find out 00:37:18.08 [Stefanie Zammit]: directly. And that is another of my rules of thumbs that I didn’t mention earlier is the power of derived research versus stated, honestly, you’re wasting your money on stated surveys, they’re just no one knows humans do not know why we behave the way we do, right? It’s all like deep psyche. We’re super weird creatures. We have all these quirks that we don’t understand. So there, yeah, there’s you’re wasting your dollars on how likely are you to book a cruise? Like, whatever, it’s bullshit. It’s not sorry. Oh, we can swear on this. Yeah. Oh, we’re explicitly ready to send it. Let’s encourage. Yeah. Exactly. Absolutely. Like that, that needs to be the best quant research has analytics. If you’re running quant without analytics in your surveys, I don’t know what you’re spending your money on. Honestly, it’s yeah, it’s just not good value. And that again, 00:38:10.32 [Michael Helbling]: that ties then to internal analytics. This is so good. So I’m sitting here from a data practitioner standpoint and just loving the conversation. At the same time, I’m going to admit to you that like you’re throwing out certain terminology that I vaguely familiar with, but don’t necessarily know. Like what are there resources that could help someone kind of like level up and get better understanding of just sort of topics, structures, stuff like that, any like good overall books or 00:38:39.36 [Stefanie Zammit]: resources online, like anything you might recommend? Yes. And this is a really important point. I think another reason for the othering that happens in these fields is purely language. And so you’re right. I’m using, I’m trained in research terminology, which is fine. I just am like 00:38:56.08 [Michael Helbling]: admitting that I don’t know all the words you used. So yeah, but the, the, a lot of the words 00:39:00.88 [Stefanie Zammit]: I’m using, there’s, there’s a analytic or data equivalent of it. It’s, it’s just that different term terminologies used. So, you know, data might talk about addressable audience, which is like sample plans. The best advice I can give is research is grounded in academia. It came from, you know, like scientific studies or social studies. And so for anyone who, who was at university doing research projects as part of their university degree, that is the foundation of modern commercial research, but there absolutely are some great tools. There’s a really great book that I recommend. It’s, it’s one book. It’s the only one you need. What’s it called? I think it’s called, it’s the market research society’s main, main book. I think it’s called intro to market research. And that’s, yeah, that’s your one reference of just looking up these words. And you’ll be amazed how many of them you look up that you’ll recognize as not actually that unusual. So like an attribution model, how different is that from a customer journey? A research team would run a customer journey study, a data science team or analytics team would run an attribution model or call it customer journey, but you know, depending on what, where your journey 00:40:07.04 [Michael Helbling]: is. It’s a, it’s a customer journey has been used all over the place for all kinds of stuff. 00:40:14.48 [Julie Hoyer]: Really, whatever you want. Just like use case. Oh God. 00:40:20.64 [Michael Helbling]: The journey of customer journey is a little bit tricky. 00:40:24.64 [Val Kroll]: That’s, that’s, that’s a, that’s a cartoon strip right there. Well, so there actually is, so to your point, Michael, there is one, because you were just starting to talk about this about the difference between like stated versus like derived importance, which gets to max diff, which I, 00:40:40.88 [Announcer]: I literally couldn’t love anything more than studies where we get to do that. Because there’s 00:40:45.68 [Val Kroll]: so many, there’s so many different applications. But when I was, I worked on the telecom vertical is my first job out of college. And we would use that to figure out like back in the day, like cable internet TV, what should be involved in that packaging and at what price points. And people would say things like, Oh yeah, I need access to like 600 channels. When we asked like, what’s most important to you, but when we actually did like the force ranking, or there’s like different techniques, that that actually is one of the things that fell to the bottom, that it was about, you know, how long is it going to take for the technician to install the, the cable box. And there was like all these other things that were like, not something that someone would say necessarily, but it really, when it comes out in the wash. But anyways, that’s just like one example. But could you, especially if you have an example that you can pull upon to talk a little bit about state of versus drive importance or one of your favorite examples of that? It’s a really fun one. 00:41:40.40 [Stefanie Zammit]: I mean, in a nutshell, the difference is asking someone, yeah, how likely are you to do this? And with, whereas with derive, you basically give them an exercise and then you observe behavior. So for me, derived research is the same as what would be happening in a data team where you’re observing the behaviors, right? Because in data, there is no stated, like you’re not stating it, you’re just watching people behave and you’re tracking their data. So it’s the research version of that, that we give them different exercises or force response between many, like many, many, many choices, again, and again, and again, mixing them up again and again, so they could never remember. Yeah, like you could never remember the pattern. And through the continuous, like, it’s a choice between these four things or a choice between these and again and again, different scenarios, again and again, you can derive what the true behavior is or will be. So it could be used predictively to say, when this is true, this is the behavior that we want. Similar to building a predictive model based on behavioral data, that you can, you know, take all your data and map it over time and start to say, you know, just like run correlations to 00:42:54.56 [Announcer]: sort of say, when this is true, this is more likely to happen. So again, very similar outcomes, 00:42:59.60 [Michael Helbling]: but just completely different methodologies. All right, I’ve got another question that’s probably going to reveal how much I don’t know about this topic, but I want to ask it. Why are personas so bad most of the time? Segmentation makes my skin itch. If people are 00:43:18.64 [Julie Hoyer]: like, well, let’s look at the segments. I’m like, can we not? Because I honestly think, 00:43:23.68 [Michael Helbling]: I honestly think this is also a driver of the divide in a lot of ways. Like as a data guy, like I see people come up with these persona studies and stuff, and they’re dog shit. Like they’re really like terrible. And I’m like, scrap your personas. It’s all behavioral based at all. Like, you don’t because what I observe people do is they just make up who they like their customer to be. And then they’re like, this is this is Sally. And she’s a hip mother of three. And she drives a van. And but she’s got this cool thing that we like about our brand. And so that’s one of our personas. And it’s like, Sally doesn’t exist in our database anywhere. That’s not our customer. And the people who buy the products you’re talking about don’t look like her at all. Like it’s no correlation. Anyway, sorry, I’m now getting into my rants. But 00:44:18.16 [Stefanie Zammit]: what’s happening there? Like, why is it so bad? I love it. And I think the word segmentation or persona in themselves can mean so many things that these are words that are overused and not necessarily always used in the right way. And I don’t think there is actually a fixed definition. I think it just depends internally on what definition you choose. But why are they so bad? I have run uncountable number of segmentations, mostly from my market research background, where I have more years of experience. And I think that the win or lose of a segmentation is in how it is translated or brought to life for the business. So you cannot have a good segmentation without really solid underlying data, you know, hardcore, like, like a good, the factor analysis and the clustering, like, and all that has to be and you had the right variables and you had the right ingredients, all of that is really important. But that’s not actually matters or has impact. That’s just designing the output. It’s like a BI report. You can have like, you know, the BI report with 1000 million, like every possible data point, it’s amazing. But unless it’s, you know, user friendly, then it just doesn’t have impact. It’s the same with the segmentation. So without, if you only had a segment descriptively, this is Sally, you know, and Sally does this and Sally does that without knowing why or without finding Sally in the data and like saying, look, this is Sally, like specifically look, we’re going to, we know what Sally wants. So we’re going to, we’re going to send her a, we’re going to do a CRM strategy around Sally, we’re going to sell to her. And now we found Sally in her data, we can actually say, look at her changing her behavior. That’s when a segmentation is really powerful. And, and I think the best segmentation is to get to that level, you need both your behavioral data and your, your research, because research gets to how does Sally think, like what matters to Sally, you’re never going to get that from just observing her in data. You need to get into her psyche. So you put the two together. Now the marketing team have a strategy on like, who is Sally in terms of her psychology, what’s going to get Sally really freaking excited and get her to behave the way we want to do, but it’s underpinned by existing data. So you can actually see her, you can maybe test with her and, and over time see the impact of your segmentation and video, video, like I cannot overstate the power of a customer inside video, just like Sally walking down the street, like you can read, this is her life. It makes such a difference stakeholders in understanding who that person is 00:46:58.80 [Julie Hoyer]: beyond her being a data point. You make it sound like so, I mean, it is so ideal, but it makes it sound so like, duh, if you just did this, you know, you’d get everything you want because then we go where I go work with clients and it’s like, they’re just so far from that 00:47:13.36 [Announcer]: point. And it feels like such an uphill battle to try to help them fit together the two worlds that 00:47:19.20 [Julie Hoyer]: you’re talking about research and analytics, you know, segmentation is all based off of outcomes. And it’s like, but you want them to change behavior to drive outcomes, but now you’ve split them by outcome already. And then you ask them questions about outcome, it just feels like something’s been lost in a lot of the situation. And it’s, it’s sad to see because they would have to, I really think like start from the ground up to get it to where they’re utilizing analytics and research in the right way to get the benefits you’re talking about. Research hurts because every 00:47:52.16 [Stefanie Zammit]: time you do a study, you need to pay money, especially because like I said, no one has in house research teams, right? Not like fully intense. That’s not a thing. I think maybe Sky TV have one, but like most companies don’t. And so you’d have to make a business case to say, why should I spend money doing something that the analytics team can do internally using this data? Why is that not good enough for stakeholders to understand that without having ever seen what good looks like is really difficult. And it’s something I find really challenging in my job actually, just explaining the value of something without having it to hand. So it’s like hypothetical to a stakeholder, right? And this is where research teams are lucky if you have good relationships with agencies that will send you case studies and they feel sort of safe enough to send you examples that you can use to build your business cases. But when you’re talking in hypotheticals, it’s very difficult to get the budget and it’s not cheap. Market research is expensive. It’s slow. It’s a big investment for any company to make. But what I do find is once you start investing in it and putting the two together, showing the impact of that, the stakeholders will then understand like, wow, I get it so much more now because I’ve got my data, but I’ve also got my why and like my… I get how this person thinks. Put those two together and it suddenly you think, how did I ever make a decision without knowing this full picture? And then that’s where you will wet the appetite and it snowballs from there. But it is very difficult to do the very first one and hopefully agencies can help with those case studies. No, I have kind of a random question. So I’ll save it to Michael when he wants 00:49:30.72 [Val Kroll]: to wrap. Okay. I love this. So I actually had a very… The opposite experience of you, Stefanie. I started on trackers and I wanted to kind of branch out of that. And so I got thrown on the iHuts, which are in-home usage product testing, which is like could not be further from the tracking world. But I love that. Like when you said the power of video, Gillette was one of our clients and they sent out these new like beard trimmers to like men and they asked them to do videos of them shaving. And it was just so funny that like they were watching like, oh, like why in God’s name are they putting that clip? Don’t hold it like that. You’re going to cut their nose off. Like, oh my gosh, we need to change our directions. But it was so funny, including like testimonials, like the voice of customer, like in some of those reports that could be like leveraged 100 different places. We also had Haynes as a client and they were trying to… There was testing all the tag lists back in the day when everyone was switching. So it was like t-shirts and underwear. So we were sending out boxes and boxes of like whitey tighties and asking people like, tell us about… Did the tag scratch your butt? Like that was like the question. But the quote that we got, I’m like, I really wish I could see this used in like the 00:50:43.60 [Announcer]: internal decks, like where this went. But anyways, to your point about like it’s an investment in 00:50:51.12 [Val Kroll]: the first one, if you only think about it as like, I’m going to send out a survey and I wonder what they’re going to respond on like likelihood to agree to a certain attitude or statement versus like thinking more creatively about the different ways that you can interact with the customer, I think you can get people really excited about ways it can be injected in. So if you take one thing away, don’t think myopically about what research is and the ways it could be applied because it can actually be pretty fun and pretty enlightening. So… All right, Julie, 00:51:23.20 [Michael Helbling]: lightning round. Random question time. Lightning round. Because we have to start to wrap up here 00:51:27.68 [Julie Hoyer]: actually. Fine, Michael, we have to start to wrap. Okay, my question, because you were saying 00:51:35.20 [Announcer]: research is not fast and it is not cheap. But nowadays people like fast and people like cheap 00:51:41.52 [Julie Hoyer]: and people like AI because AI is… Oh, that was literally my question, Julie. So I have a slight spin. Let’s see if you went this far too, Michael, because maybe we were totally, totally parallel 00:51:51.60 [Announcer]: thinking, which I love. Because there were the two things, the faster and the cheaper. So 00:51:58.64 [Julie Hoyer]: we’ve had an episode in the past about synthetic data. I was curious like your thoughts on using synthetic data in this space, maybe some pros and cons. But then it immediately my brain jumped to, well AI in general is the fast and cheap option. And both of those things feel like people are very quickly going to grab for them to fill the gaps of classic research. But we’ve spent this episode saying that we’re weird creatures. And like to actually figure out the why, you can’t just ask them why, you got to take the time to observe. And those things are just at 00:52:29.68 [Stefanie Zammit]: such opposite ends of the spectrum. So it depends who you are as a company, how relevant or how useful synthetic data would be for you. So for example, at Bang and Alson, we work in small data, our transaction volumes are relatively low. You know, we’re lucky if we get a data set of, 00:52:52.24 [Announcer]: I don’t know, a couple of hundred thousand rows. So our client is so niche and so 00:53:02.88 [Stefanie Zammit]: under, I’m so misunderstood, or so not yet understood, because we are a luxury brand in the consumer electronic space. We can’t learn from other consumer electronics behaviors, but we can’t learn from other luxury behaviors. So synthetic data is just never going to be relevant for us as a brand. There isn’t enough volume and there isn’t enough lookalike profiles. And we’re still exploring the category, you know, being the pioneers of the category. If you’re a CPG company, and you sell in, you know, the typical supermarket or grocery store, 00:53:36.40 [Announcer]: then yes, absolutely. That makes sense. I would say though that there is a watch out that 00:53:43.44 [Stefanie Zammit]: we’re living in a changing world. So my rule of thumb when it comes to any kind of behavioral insights work is that they have a shelf life of around three to five years. But that’s been my rule of thumb since pre-COVID. And I do think that the world is changing more quickly now post-COVID than it was pre-COVID. So you have to think culturally, is the world the same enough for me to rely on synthetic data, which might go back, it depends where your cutoff is, right, where you start your data set from. So I would warn against caution to think about that. If there’s a huge world event, you’re probably going to need to go in with fresh questions or fresh exploration. And yeah, just the world we live in right now, it is so turbulent that, yeah, it’s not, the three to five year shelf life thing might not 00:54:31.60 [Michael Helbling]: be applicable anymore. Oh, that’s really good insight. And yeah, that was basically Julie, the question I was going to ask is about AI and its place in this, because I’ve seen startups going around, you know, being like, we can create a 100 personal digital twin research panel for you on the fly with AI and you can do your pre-research research with it and stuff like that. And I think there might be a place for it. But like, like you said, Stefanie, you have to kind of like think through the applicability. And I like the way you specified, like, hey, for our brand, we understand how unique we are. So a group of averages is not going to get us to an insight that we could use, which is, I think, very, very relevant. That’s really good. All right, we’ve got to start to wrap up. This is so fascinating. So thank you so much, Stefanie, for joining us. And it’s very good, educational, and really fun to talk about. And I know Val, you probably also are loving this episode. So, okay, what we’ve got to do last calls, something we do every show, we just go around the horn, share something that might be of interest to our listeners. Stefanie, you’re our guest. Do you have a last call or a couple you’d 00:55:45.68 [Stefanie Zammit]: like to share? I do. I have two last calls. And you know what, of all the prep I did for this episode, this was the thing that stressed me out the most, because you guys’s last calls are so good, as I got it coming with something good. It can’t just be any old thing. I did lose some sleep over these. But I think I got two good ones for you. So the first one is it’s actually a game. I’m a gamer. I love any type of game, board game, video game. And I attended a leadership training that was organized by our amazing HR team at Bang and Olson. And we played, it’s essentially a simulation game. You’re given a group of employees and they have to deliver a project together. And you know, it’s a bit like Moenopoly, you get chance cards and things go wrong. And it’s like, oh, the project, you know, somebody went on stress leave, what are you going to do? How are you going to keep to the time and the budget? Uh-oh, your main stakeholder has suddenly decided that they forgot what this was all about. What are you going to do? 00:56:42.72 [Julie Hoyer]: That sounds traumatic. I was like, this is giving me like stress. 00:56:51.44 [Michael Helbling]: It was so fun. I play that game every day, Stefanie. What are you talking about? 00:56:58.72 [Stefanie Zammit]: Sorry. No, but you’re, you’re sorry. We do play that game every day, but the fact of having the safe space where you could have these, oh shit, like everything’s going wrong in my project moments, but you’re learning how to deal with those in the safe space so that when it comes to your real life game, you’re prepared. I thought it was a great idea. It’s the game that we played was called the Playmakers game. It’s made by a company called the Works, Works with a Z. And I think they have a bunch of other sort of professional world simulation games as well. Super recommended. 00:57:31.12 [Announcer]: Yeah, next onsite. And it was a great way to like build rapport with your stakeholders as well, 00:57:37.44 [Stefanie Zammit]: right? Cause safe space and you play the game with a team of actual colleagues. So it was good 00:57:42.88 [Val Kroll]: for the bonding. Oh my gosh. You should like reverse roles. Like I get to be the stakeholder this time. Yeah. That would actually be hilarious. What should I do with all this power? 00:57:53.60 [Michael Helbling]: My problem would be like, really? You’re going to do that? Like no. 00:57:59.28 [Stefanie Zammit]: You have to all agree on the decision. That’s the game. Like how are we all happy? All going to do it. Oh no, we lost the team to stress leave. Damn it. Like we, so yeah, it was 00:58:11.36 [Announcer]: that’s awesome. That’s very cool. What else? Well, I have measured my second one just because I 00:58:17.20 [Stefanie Zammit]: am really excited that this is like hot off the press. It’s literally just gone live. I think two weeks ago, as you know, I worked for Bang and Offsend and we’ve just opened the factories up for anybody who’s interested in audio to go experience the manufacturing of our products. And honestly, if you’re a sound nerd or an audiophile, it’s a incredible experience. Like a really just sort of once in a lifetime immersion into the world of audio in a very beautiful part of the world. So yeah, that was my second one. Fun. Wait, Stefanie, I did see your note. You have to say the freaky part. Oh, the freaky part. Yes. So it’s a tour all through the, you know, like how the tonmeisters are called, how they find the perfect sound. And there’s a lot of different rooms that are created in a way for you to experience sound in different ways, which is how the products are developed. And one of the rooms is the 100% noise-proofed room. And it is the scariest place. Like honestly, I couldn’t stay in there with the door closed. You wouldn’t believe how scary 100% soundproof is. You can hear your blood flowing through your 00:59:27.44 [Julie Hoyer]: veins. It’s terrifying. Insane. I was like trying to imagine that. And I was like, 00:59:34.96 [Stefanie Zammit]: that’s just like breaking my brain. Honestly, 10 minutes and you’re like, get me out of here. Like I’m going crazy. Yeah, I bet. Okay, now I need to go do this. 00:59:45.68 [Michael Helbling]: I know. Stefanie, great job. You’ve upheld your end to the Les calls by far. Yeah, 10 out of 10. Absolutely. Yeah. Who wants to follow that? Val, what’s your Les call? 00:59:59.28 [Val Kroll]: So mine is going to be a little research-related. So I thought that one of the things we might discuss, and I do think we spent a good amount of time on it, is how to be curious in how to get broader in your understanding of these different methodologies or teams that might exist inside your own organization. And so my recommendation is to look out for some different communities that you could be a part of or join. The one that I’m still a part of today is the women in research, the wire group actually started in Chicago a long time ago. But I have benefited so much from my different mentorship conversations. I still stay in touch with the mentor I was assigned with, I think 13 years ago now, 14 years ago, Sheri Binky, shout out, I know she’s a listener. But there’s like, I’m a part of like women in product groups, and it doesn’t have to be like women only groups too, but they have so many different great events where you can get out and talk to people. So get out, touch some grass, talk to people, learn about how people are putting some of these different ideas to use inside of organizations, because that can totally be an inspiration for the way that you bring it to your own work. 01:01:12.96 [Michael Helbling]: Outstanding. All right, Julie, what about you? What’s your last call? 01:01:17.04 [Julie Hoyer]: Fine, it’s a little bit random. But honestly, this is something I’ve definitely heard mentioned before, quantum computing. And I do think this is like the next leap probably after AI, if my naive take on it is anywhere close to true. I was reading a newsletter that I always get, and there was this one mention of like the next big leap, like leap, I think it was called. And so I clicked on it, and it was all about quantum computing. It was an infographic about I’m like, well, I’ve heard it mentioned, I don’t know what it is, like, sure, I’ll take a look at infographic. And it, it was really good. And the way they broke it down within like not a very long read, I totally have a new appreciation of what this means and why so many companies are going after it. Pretty much they’re saying like, instead of using electrons for zeros and ones, like, we’re going to use a subatomic particle that is like super finicky to keep stable. But pretty much we go from being able to compute things one at a time linearly to doing things what’s it called simultaneously. So all these computations simultaneously that you could do and you think about how much computing like AI is doing or different industries like finance or supply chain even, and they, they walked through like a supply chain example. And it was amazing to think that you could go from something that would take a normal supercomputer, like they even said like a trillion years in their example to doing it so much quicker with quantum computing. And they said that some of this quantum computing power could actually happen in the next two to five years. And some of these people working at companies were like, I was told I wouldn’t see some of the milestones we have hit recently in my lifetime and like they have hit them. So it was one super interesting. I finally feel like I kind of understand what it is. I was a really quick read too. So it got me kind of freaked out and excited. And I just felt like, Oh, I learned something. 01:03:16.32 [Announcer]: Sounds good. I like it. Yeah. What about you, Helbs? 01:03:20.56 [Michael Helbling]: Well, I, as per usual, love everything I read on CommonCog.com, Cedric Chin, and he wrote an article recently about how to make sense of everything that’s happening at AI, because it just sort of feels overwhelming most of the time. And it actually sort of ties back to the episode in a way, because one of the points he was making was sort of like, don’t listen to what people say about AI, watch what they’re doing with it in the real world, to use that as a guidepost for how you should be responding to AI, which kind of goes back to sort of like user research. So anyways, the article is really good, but it’s very practical in terms of just better sense making around, like, okay, there’s sort of this hype and concern and all these other things. But like, look at actual detailed examples of how people are actually using it, and then ask some questions from there, like, what other outcomes are possible? What actions could I take? What matters most in my context? Those kinds of things. So anyways, really good article, just to like, take some of the pressure out of what I think a lot of us are feeling about AI, like, half the time it’s like, is it going to take my job? And the other half of time, this is so cool. I can’t believe I don’t, you know, I’m just doing everything with AI now. So there’s some balance where you have to find a balance. Otherwise, we’re going to blow up. Anyway, so that’s my best call. Blow up. All right. Yeah. Tune in. What’s the old, whatever, I won’t try to remember what the hippies used to say. Stefanie, thank you so much for coming on the show. This has been so fun. 01:05:01.12 [Stefanie Zammit]: Thank you. Yeah. Thank you for having me. It’s awesome to get to talk to you guys and talk about 01:05:06.40 [Michael Helbling]: nerdy topics that I love. So thank you. Yeah. No, it’s been great. And I’m sure as you’ve been listening to the show, you might have questions or you might have ideas. We’d love to hear from you. The best way to reach out to us is through the major Slack chat group or LinkedIn or via email at contact at analyticshour.io. And please feel free to reach out and leave us a review on the platform that you listen to us on, whether that’s Apple or Spotify or whatever, you know, whatever one we’d love to hear from you. We love getting feedback on the show. So definitely do that. And we’re still asking you to give us some questions just a couple of weeks to go until we’re going to be recording a show live at Marketing Analytics Summit on April 29th in Santa Barbara, sunny California. And we’ve got a survey which is out on the show notes page. Go fill it out. I mean, perfect example. Hopefully we did a good survey. I’m pretty sure we probably did. I know, right? Although again, I just want to caveat each time I had nothing to do with the art at the end of that survey. You’ll have to fill the survey out to see what I’m talking about. But it was I had no editorial control whatsoever. But if you have a question, we want to gather lots of great questions from listeners, either if you’re be there or not, we’re going to let answer them live on the show when we record it there at Marketing Analytics Summit. So 01:06:29.92 [Announcer]: looking forward to that is just a couple of weeks away. All right. Great show. Very fun. 01:06:37.76 [Michael Helbling]: And I know I speak for both of my co-hosts, Val and Julie. And I say, no matter what your 01:06:42.96 [Announcer]: market research says, keep analyzing. Thanks for listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at At Analytics Hour, on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst. Those smart guys wanted to fit in. So they made up a term called analytics. Analytics don’t work. Do the analytics say go for it, no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 01:07:24.72 [Val Kroll]: Julie has this mic that like, she could be across the room and she’d be like, 01:07:28.88 [Announcer]: it’s like so loud. 01:07:35.28 [Michael Helbling]: Julie inadvertently must have bought one of those ASMR mics or something. 01:07:39.92 [Julie Hoyer]: Literally, I have like the game turned as far down. It’s like more than on arms length away for me. I’m like speaking, you know, I’m trying to speak on the softer side and I’ve turned on my volume in reverse side. No, I think it sounds good. And I’ve made sure my humidifiers are off. So hopefully no background humming. I unplugged my wine fridge, like all the things. Yeah. 01:08:03.28 [Michael Helbling]: And the worst part though, Stefanie, is we have this really great engineer, Tony, who goes through and does like all the audio editing and he gives very specific feedback about who’s audio quality was terrible. And so we’re going to use notes back from Tim of like, oh, Michael’s awful this episode. And we’re like, oh, thanks. That’s so great. So that was like so cautious. You sound a little soft, but you sound okay. Yeah. Anyways, no, it’s just one of those things where you’re like, you think we’d have it nailed down after so many years, but we’re still like 01:08:38.00 [Val Kroll]: every episode we’re sort of like fine tuning. You know what y’all need? 01:08:42.32 [Julie Hoyer]: Carabangan Olsson’s just saying. Yeah. That’s right. Yeah, you’re the person to ask about that. I 01:08:48.80 [Announcer]: should go look. Well, I’ll wait for Val to stop typing. I know this. Usually it’s Moee and she’s 01:08:58.16 [Val Kroll]: like keyboard cat, like you’re like Moee. Yeah. We’re a very serious professional podcast. 01:09:06.48 [Michael Helbling]: That’s right. Bringing it all together. Here we go. All right, I’ll give us a five count and 01:09:10.80 [Announcer]: we’ll get started. We’ll go in five, four, three, rock flag and two worlds, one family. The post #295: Research and Analytics: the Peanut Butter and Chocolate of Data? appeared first on The Analytics Power Hour: Data and Analytics Podcast .

Is this your show?

Claim this listing to keep it up to date, reach guests who want to pitch you, and manage bookings with Guestify.

Claim this listing

More Business podcasts