Find partners
The Analytics Power Hour

The Analytics Power Hour

Hosted by Michael Helbling, Tim Wilson, Moe Kiss, Val Kroll, and Julie Hoyer

BusinessManagementInterviews guests

Episodes

10

Latest episode

Jun 2026

Language

EN

About the show

Attend any conference for any topic and you will hear people saying after that the best and most informative discussions happened in the bar after the show. Read any business magazine and you will find an article saying something along the lines of “Business Analytics is the hottest job category out there, and there is a significant lack of people, process and best practice.”In this case the conference was eMetrics, the bar was….multiple, and the attendees were Michael Helbling, Tim Wilson and Jim Cain (Co-Host Emeritus). After a few pints and a few hours of discussion about the cutting edge of digital analytics, they realized they might have something to contribute back to the community. This podcast is one of those contributions. Each episode is a closed topic and an open forum – the goal is for listeners to enjoy listening to Julie, Val, Michael, Tim, and Moe share their thoughts and experiences and, hopefully, take away something to try at work the next day.

Listen to episodes

10 recent
June 9, 20261 hr 0 min

#299: AI Can (Help) Build the Dashboard. It Can’t Build the Buy-In.

There are roughly a thousand ways to roll out a new analytics platform, a BI tool migration, or an AI initiative to your organization. Most of them involve a town hall, an email with a link to some training materials, and the quiet hope that everyone figures it out. Most of them also don’t really work. On this episode, Yehonatan Schwarzmer joined Michael, Val, and Tim to bring some long-overdue organizational change management thinking into the analytics conversation. Yehonatan has the unusual combination of real-world experience in both change management consulting and data leadership, which makes him exactly the right person to explain why the technical rollout is the easy part. The harder part is understanding that when someone says “this tool doesn’t have what I need,” they might really be saying “I was the hero in the old system and I don’t know who I’ll be in the new one.” The Kübler-Ross grief model shows up. Psychological safety shows up (reluctantly). And Val’s question about who analysts should recruit to help them manage change at scale almost gets answered. This episode is brought to you by Prism from Ask-Y—your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. This episode is also brought to you by Stape, your all-in-one solution for server-side tagging. Links to Resources Mentioned in the Show Prosci Kübler-Ross Change Curve Five Stages of Grief #184: Psychological Safety and Analytics with J.D. Long McKinsey 7S Framework Kotter’s Change Management Theory The Digital Delusion by Jared Horvath How AI is Redefining the Role of Analytics Leadership by Eric Sandosham Let My People Code by Noah Omri Levin Marketing Analytics Summit Can You Teach an AI Ethics? by Gary Angel Photo by Brad Starkey on Unsplash Episode Transcript00:00:00 | Announcer: Welcome to the Analytics Power Hour. 00:00:08 | Announcer: Analytics topics covered conversationally and sometimes with explicit language. 00:00:12 | Michael Helbling: Hi, everybody. 00:00:15 | Michael Helbling: Welcome. This is the Analytics Power Hour, and this is Episode 299. You know, in the movie Godfather, Michael Corleone takes over the family in a time of crisis. And he quickly moves Tom, his adopted brother, out of his role as consigliere. His reason for this was because Tom wasn’t a wartime consigliere. In times of great change, I guess it’s important to recognize the need for different styles of leadership and approaches to our work. And, oh, I mean, AI is kind of putting every company on a, quote, wartime footing, I guess, in that survival, our ability to predict and normal cycles of growth, even our own understanding of the levers of business are kind of being transformed. And how do you manage it all? All of this filters down to analytics because AI is making some things much easier, but potentially creating some downstream challenges that could be very, very damaging. So it’s time to put on your wartime consigliere hat and embrace the change. I mean, it’s already underway. So let me introduce you to my co-hosts, Val, “Ground Truth” Kroll. How are you doing? 00:01:25 | Val Kroll: I’m good. Now that I’ve Googled consigliere and those roles from the Godfather. Yeah. 00:01:31 | Michael Helbling: Just keep doing it the Chicago way. That’s right. Got it. 00:01:35 | Val Kroll: Got it. And, of course, Tim, the last word, Wilson, how are you doing? 00:01:42 | Tim Wilson: Doing quite fine. 00:01:43 | Michael Helbling: And I’m, my goal, the diplomat, Helbley. All right. We needed a guest. Somebody could walk us through what’s happening, how to react to it, and I think we found a great one. Yehonatan Schwarzmer is the director of business intelligence at Networx. He’s also held analytics leadership positions at Search Discovery now further. He has done organizational development consulting and he has a master’s degree in industrial and organizational psychology from Columbia University. And today he is our guest. Welcome to the show, Yehonatan. Thank you. 00:02:15 | Val Kroll: I’m very happy to be here. 00:02:17 | Michael Helbling: Great to have you. 00:02:19 | Val Kroll: Okay. 00:02:20 | Michael Helbling: So I think there’s a lot to get into here, but I think, you know, in your world, you’ve kind of lived across two different distinct worlds in your career. One is sort of this idea of change management as its own practice and then analytics as its own sort of discipline within the business. So like, as you see those two things sort of come together in this moment, like what are the things you’re noticing in the community that you’re in? 00:02:46 | Yehonatan Schwarzmer: The idea of change management is one that we all focus on as a topic by itself. You know, we have an issue or we see that there’s a company or organization struggling with shifting or a change that they know is coming and the feelings that come along with that. And so there’s a lot to talk about with change management. Some people think about it like organizational therapy or some other label that we stick on it, but separate from the question of what is change management, it also has to do with what it is that you’re managing a change of. And specifically when you talk about the management of change in the world of analytics, it really focuses on a lot of the big questions that come up that you typically explore in the world of data and analytics, just with an eye towards what is happening on the people side of these changes? How are the people being affected? How are they going to approach it? How do we think about that so that the change is seamless and gets us to where we want to be and we’ve gotten everything we want out of it? You know, when I shifted, kind of talk about two worlds, so when I shifted into the world of analytics from the world of pure change management, right? Pure change management. So we were doing consulting to companies and talking about, you know, leadership development. There was like conflict resolution, especially when you have those like family companies and you know, lots of the issues that come up with those types of things. And then you shifted to the world of analytics and it starts with what’s the number, talk about the numbers, just the numbers, but then really, really quickly, you get to the understanding that it’s not just about the numbers. 00:04:22 | Val Kroll: And if you only focus on the numbers, you’re going to miss a big part of the value that 00:04:26 | Yehonatan Schwarzmer: can be delivered. You really have to understand the people involved in it. And that’s where the cross between the two topics became very exciting. 00:04:35 | Tim Wilson: It does seem like you said something in that that the one change, figuring out how to get where you’re trying to go to. And I guess, is it fair to say in the world of data and analytics, it seems like often there is where we want to go to is being data driven or being it’s kind of like this squishy aspirational, we want to use the data to make decisions. But that’s not really like the future state hasn’t been fully formed. It’s kind of more of this aspirational idea. Does that not bring challenge? Because if you haven’t really defined the destination, which I guess is also a temporal thing, because the destination is always going to keep moving. Is that a particular challenge if you’re doing change management and you’re not acknowledging how the future should look different from today? Yes, of course. 00:05:34 | Val Kroll: There’s a couple of the typical or the classic issues that you’re going to bump 00:05:41 | Yehonatan Schwarzmer: into either when you are doing change management or that come up that make you realize we should really be managing this change. So one of them for sure is we don’t really have a vision for what this change is supposed to be. Of course, we don’t have people on the bus. Nobody knows where the bus is going or we have 10 different buses and they’re going in 10 different directions and people have to decide which ones they want to get on. So for sure, defining it is one of the critical components. There’s a number of critical components. You talk about things like defining the vision. You talk about things like making sure that you have buy-in that you’ve dealt 00:06:12 | Val Kroll: with resistance, that you have a organizational sponsorship. 00:06:16 | Yehonatan Schwarzmer: Let’s say a primary sponsor, someone who’s really going to make sure that this thing is managed the way that it needs to be. So there’s a bunch of them. But for sure, having a vision or where it is that we’re supposed to be is one of the fundamentals. 00:06:28 | Val Kroll: I love the way you put some of that introduction. It has me thinking like whether I should adjust my definition or how I think about it a little bit. That is something that happens separate. So I guess there’s the concept of like changes happening, whether you’re managing it or not. And so you have to make the choice for it to be an active part of whatever is in scope. But do you think of it as like if it were one of several work streams, if you will, on a project, is that missing the boat because it’s not like interwoven with the other activities on the project? Or is that healthy because then you can kind of give it its own time and attention and like the capital letters of the name of the work stream? Or like how do you kind of consider that in the scope of or in the context, I should say, of like a larger initiative or project? 00:07:17 | Yehonatan Schwarzmer: I think that as with most of these types of questions, the answer is going to be it depends, which is only because there are so many pieces that come into it. You know, we talk about a change. So you think about a small startup that’s quickly shifting, you know, every month, it seems to focus on something new and different. You talk about a very large bureaucracy where someone has come in and says that the way that we’ve been doing things for the past 20 years just isn’t working. You talk about a company that’s turned around and said, oh, my goodness, we’re falling behind in the AI bandwagon and we better get everybody on board as quickly as we can. You know, these the changes that we’re talking about, they there’s not one type. And because there’s not one type, there really isn’t one answer. People who love change management sometimes tend to think of it as a hammer and everything is a nail. And it’s a terrible hammer because there there are so many things that happen separate from the people side and the people’s dealing of the change. You know, we say change management is so important because if you don’t deal with the people, then you’re going to really struggle. That’s true. But if you’re just dealing with the people and let’s say you’ve forgotten about the strategy, for example, or the vision, then you’re not really much better off. So when I say it depends, I think of really a couple of different axes that you’d you’d want to think about that. It depends factor on, so, for example, what type of change is it an incremental change, something small changing, probably doesn’t need a lot of management. Is it a transition or shifting something underlying or much more critical to our regular operations? Or on the other end of the spectrum, is this a huge fundamental shift? And we have to redefine our identity, let’s say we have to become something that we weren’t before. So that’s one way to think about it. What’s the motivation? Where does it come from? You know, you have these, especially now, you know, you get some less startup that pops up and they’re very excited and, you know, they’re running in one direction and they know that they’re working in an uncertain world and then things change around and they’re like, this is amazing, a new challenge. And they turn around and they go in that direction. And then they, you know, and it’s a group of, you know, six or seven or eight people and they’re charging in this direction, they’re charging in their interaction and we can talk about whether that’s good or not good. But the management of the change and the willingness of the people to go along is 100% as opposed to the next level up, which is a change imposed, but it’s imposed internally. So let’s say you have a new leadership and they come in and they say, from now on, we’re going to do it this way. I’ve looked at how we’re doing it. It’s not going to work. We got to do something new or I found a new capability. And so there are some people pushing for it inside and some people have to kind of get along with that. And then the third kind is you’ve got an externally driven change that the people on the inside never really asked for, but they don’t have a choice. So all three of those categories are going to react to the change in a very different way. And so the, the management of it, how necessary that is, really does defend and depend, and then there’s the, the organization itself, you know, you’ve got some organizations that are oriented towards learning. They love learning. They love conversations. They love that free flow of what’s working. What’s not working. Let’s be honest with each other. Let’s move the ideas forward, et cetera. You have others that are a little bit more calcified and not as willing to move things along. Um, and then you have some that are just, let’s take it as it goes. You know, we, we, we don’t necessarily point to those as the, uh, the way things are supposed to be, but there are some places where management is, is kind of a committee and decisions are made by committee. And we all know what that looks like. So there are lots of ways that a change could be imposed and lots of ways to respond to it. And so depending on that determines the level at which you have to manage change as a separate thing from your project or as integrated within the project itself. I’m going to stop there and see if that makes sense. I could, I could talk about that for far too long. 00:11:23 | Val Kroll: So I’m going to stop there. Well, we have a whole episode to dive in. 00:11:28 | Tim Wilson: Michael, exciting news. Claude, co-work is here for prism. 00:11:33 | Michael Helbling: Finally, because I’ve got some GA for BigQuery exports and frankly, they look like somebody shredded a website and taught the various pieces 00:11:40 | Tim Wilson: accounting. That’s why ask dash y.ai built the prism, co-work connector, which is quite a literative. Your GA for BigQuery now has a smart analyst that actually speaks GA for. That includes having funnel and cohort analysis baked in ready to run right out of the box. 00:11:59 | Michael Helbling: Oh, that’s huge because I mean, half the time GA for speaks in riddles, event parameters and frankly, emotional damage. 00:12:07 | Tim Wilson: I don’t think that’s what it has, but co-work ships with sessionization built on GA for his own methodology as a baked in skill. 00:12:16 | Michael Helbling: Okay, I do love that because my session start final actual real query can finally be sent to the farm upstate. 00:12:24 | Tim Wilson: And every analysis becomes a shareable page. Prism auto generates a dashboard page directly from within co-work. 00:12:32 | Michael Helbling: Oh, no copy paste, no screenshot archeology, no, please ignore the column 00:12:36 | Val Kroll: named temp fix to nope. 00:12:39 | Tim Wilson: And co-work brings the whole prism brain, the analytics agent, harness skills, memory engine, everything. 00:12:46 | Michael Helbling: I like an organized analysis and it sounds like it provides that traceable, audible and ready to use with your other data sets. 00:12:54 | Tim Wilson: Exactly. You might have to do some adjusting to your personality and standard way of working. Well, I do feel threatened, but you know, in a growth mindset way. All right. We’ll go to ask-y.ai. That’s ask-the-letter y.ai and sign up for the waitlist. Use code APH to jump to the top. 00:13:13 | Michael Helbling: Yeah, that’s ask-y.ai code APH. Hey, Tim. How confident are you that your website tracking setup is actually giving you usable data? 00:13:30 | Tim Wilson: Well, confident is a strong word with browser changes, cookie limits, tag conflicts and messy marketing stacks. Measure pet can get complicated pretty fast. 00:13:42 | Michael Helbling: I agree. That’s why we’ve partnered with STAPE. STAPE is an all-in-one solution for server-side tagging, helping teams improve quality and the reliability of their data collection. 00:13:53 | Tim Wilson: And if your tracking is incomplete, duplicated or misconfigured, your reports may be pointing you in the wrong direction. 00:14:01 | Michael Helbling: Yeah, but you can use STAPE to clean up your server-side data collection, improve your conversion attribution, and that’ll help you make better use of your marketing budget. 00:14:12 | Tim Wilson: They also have a free website tracking checker. Enter your domain, scan your site, and in under two minutes, you get a personalized report. 00:14:20 | Michael Helbling: Yeah, I like how it has prioritized recommendations, some competitor comparisons, and AI-powered suggestions. 00:14:26 | Tim Wilson: So before the next is-the-data-wrong meeting, go get your free report. Visit STAPE.IO, that’s STAPE.IO, and use the free website tracking checker to get your personalized report. 00:14:39 | Michael Helbling: All right, and now back to the show. 00:14:44 | Tim Wilson: When you talk about, like, the organization, whether the organization is kind of geared towards learning or geared towards being static, like, do you run into the people within an organization? Like, people who are saying we should always be, we should keep adjusting and changing the way that we’re doing things versus the extreme, you know, who move my cheese, people. 00:15:09 | Val Kroll: Like, what defines, like, the organization’s nature or culture or 00:15:16 | Tim Wilson: whatever the right label is for that to say, how much is change accepted as part of something that’s always going on versus change is something that we want to minimize? How does that work? 00:15:28 | Yehonatan Schwarzmer: You were going to walk into an organization, you would have no idea. So there’s kind of the abstract part of your question, which is what is the reality of an organization? And then there’s the question of, how do we figure out what the reality of the company is so we can actually work with them? Um, I’m more comfortable with the second one because that’s the way that I typically think about it, is if we’re going to work with a group of people, let’s try to figure out where they are. 00:15:53 | Tim Wilson: So what, so I guess what are the, what are the tells there? Cause I think, I mean, a lot of people are in an organization. So I’ve never really thought about my organization through that lens. What kind of organization am I in? 00:16:03 | Yehonatan Schwarzmer: One of the topics I think that’s going to come up whenever you talk about this, either you talk about it as kind of the basic definition of organizational change, or you think about it as one of the fundamentals that you have to address is resistance. So we talk about readiness, but we also talk about resistance. And they’re not necessarily two sides of the same coin, right? Some people could be not ready because they just aren’t skilled or they don’t have the experience while others are actively resistant to a change. And so there’s, when there’s an organization called prosci, which has 00:16:38 | Val Kroll: a lot of resources related to organization change and managing the change. 00:16:44 | Yehonatan Schwarzmer: And they have a lot of models. And one of the models that’s really simple, but I found really useful is look at two spectrums to get a sense back to your question of how much management of the change do we need? One of the spectrums that we look at is the degree or the complexity of the change. Is it a significant change? Is it a small thing? And then the second one is what is the anticipated resistance? Meaning how much work will we have to do to push this change through? And that really opens up the whole question to what is the barrier to change management and how do you address that barrier? And the way that I think about it is that within change management, this is a discrete question, meaning it’s a, it’s a topic within change management, but you could easily see how this can kind of blend into every question that has to do with managing a change, which is at the end of the day. All right, what about those people that aren’t going to come along? So I’d rather think about it in the way that it is a topic within change management, because there’s a lot else that we could talk about separate from the resistance, but let’s, let’s focus on that part as a topic by itself, which is what are your barriers and how do you address them? 00:18:00 | Val Kroll: So there could be a lot of barriers, meaning when I said before that 00:18:07 | Yehonatan Schwarzmer: resistance is not necessarily the opposite to being ready. When a lot of times you see a big change being pushed at a company, you get 00:18:17 | Val Kroll: the mindset of, well, I was really clear on what we’re trying to do. 00:18:23 | Yehonatan Schwarzmer: And I had a town hall and I spoke to people about it and I have training available and I probably have office hours ready. And there’s a lot of crickets, meaning either people just aren’t interested or people are still kind of doing their own thing. Or it seems to me that they’re just waiting for this initiative to pass by just like all the other ones so they can get back to their real jobs, you know, whatever it might be. And there’s a lot of confusion. I don’t understand. Like we invested a lot. You know, we bought this new platform or we’ve got this new training or the company has to make some sort of a shift or we’ve got a new strategy or whatever it is. I expected maybe not a rallying cry, but at least people to be aware and show that they’re moving along with us. And I don’t feel that either. They’ve got their heels dug in or they’ve just kind of put their head down and they’re they’re waiting for it to pass. So what why is that? What’s what’s happening here? A lot of times when you start the conversation, it can be useful to talk about this this curve of change management. It’s it’s a model, actually. They’re trying because of you. The the Kubler-Ross model of change. Kubler-Ross model, right, which which is great. Like it’s a great, simple way to talk about here’s something that we can anticipate so that we all have our expectations pointed the same direction. Meaning if we are going to introduce a change and the change is going to be somewhat disruptive, again, we’re not changing, you know, the the the water cooler, we’re changing something that’s going to make a difference to people. And so we can expect that that’s going to affect people and the effect is likely going to mean that there is going to be number one, a decrease in productivity for a time as we shift over and we acclimate ourselves to the change and there are going to be people who need to become comfortable with the 00:20:07 | Val Kroll: change and potentially during that process of discomfort, you will see 00:20:13 | Yehonatan Schwarzmer: productivity decline or you will see ancillary things. People might look a little bit more disgruntled. They’re more whispering. You may see whatever it is. 00:20:22 | Val Kroll: So how do you what do you do about that? 00:20:25 | Yehonatan Schwarzmer: Why why is that? What do you expect? So when we think about it and the nice thing about the the model is it’s really a simple visual where you’ve got pretty much a line coming across on your time axis and then it dips down and then it slowly curves back up to where it was and then it continues up to be higher than where it was when you started and that’s the idea of what we’re trying to do here. The reason that we’re making a change is because we believe it will improve our performance. So if you look at the y-axis as performance and you say I want my performance to be higher, I’m going to go through this period of change, get my performance higher, even though that means that the cost will be for a period of time, there’s going to be a drop. There’s going to be a dip in performance and we’re going to have to work our way through that. And one way that we can think of change management is the benefit of change management is it minimizes the length and degree of the dip. So either you will be in a period of decreased productivity for a shorter period of time or it will have less of an effect. It won’t fall down as much. You know, in an extreme case, you get some people who just say, I don’t know what to do, I give up and they sort of clock out and you really get nothing for quite some time until they finally realize what’s going on or the fact that their jobs depend on it or whatever it is. And it’s time to get on board with the program. Typically, it’s not that type of thing, but the question of what will it take for someone to understand the value of this change and why we all have to get on board and what we need to do about it is where you get to getting back to the line that you were before and then getting higher. The other reason it’s useful to think about it in that model is because the person who created that model, that model was actually the second model. The original model was actually the famous model that we all know, which is the five stages of grief, which is fascinating because they came up with this idea of the stages of grief. And then after that, it evolved into a model of change management. And there’s an implication there. What does it mean that we learn from grief to what change management is? But it’s very helpful to think of it in those terms, not because change is devastating all the time, but because you have to understand that change has a real impact on people. And when you think about grief and the stages that people go through, right? So that, again, that classic model that people generally are familiar with, you’ve got your denial, anger, bargaining, depression, acceptance, right? So in change also, you’ve got your shock or denial. You’ve got your resistance, then you get maybe exploration or testing. And then finally, acceptance and then commitment. Right? 00:22:58 | Tim Wilson: Can we pause for a minute so listeners can kind of think about like the migration to do their own explore, perhaps that explains so much. There was, I mean, that was like the industry change management and the anger part. 00:23:14 | Val Kroll: Say, where are we stuck? 00:23:15 | Tim Wilson: That’s a very helpful. It could be a long process as long as it’s a process. But just to pick on and we keep coming out of like, well, there’s like a million different types of change, a million different forces, a million, I’m a little concerned. I want to kind of dive into something a little more specific. I mean, because my concern, it’s like, well, gee, it’s all over the place. So I want to push, I made the GA4 crack, but I feel like there have been platform changes. We’re moving from Domo to Power BI or moving from Power BI to Tableau or moving from Adobe Analytics to Amplitude, which there is some motivation to the change in the framing of the performance going up. 00:24:11 | Val Kroll: That’s like the change is either the licensing is getting out of hand or 00:24:17 | Tim Wilson: there’s a gap in what we can access. But I think of that dip on the Kubler-Ross curve is always somebody saying, I mean, the way you were talking through the crickets and it’s rolling out, it seems like what often is getting heard by the organization is, it’s going to be better. And then they just kind of wait until it’s there and it’s tangible. And then they look at it and they’re like, where’s that number that I used to get every Monday morning and they’re reacting to it. So in that context of this is a change is being imposed because some subset in the company made a decision, or I’m sure in Google Analytics for the 00:25:01 | Val Kroll: vendor made a decision, but the change is happening and it’s going to happen on a certain timeline. 00:25:08 | Tim Wilson: Like, what do you do when you’re thinking through what those barriers are? Like, it does feel like the organization thinks of the barriers as being the technical cutover and training and office hours are kind of, they’re like, well, this is overcoming the technical barriers and it’s going to overcome the people don’t know how to use the new tool. And it sounds like you were saying, yeah, but there’s like other barriers that those components don’t address. So like, how do you identify those barriers and then address them? 00:25:43 | Yehonatan Schwarzmer: One of the things that you said is really critical to this. The idea that I knew where my number was and now I don’t know where to find it. And it’s funny, like when I when I talk about the the concept of a change management model flowing from a grief model. And there’s like an extreme there. But in a sense, people are maybe mourning is a strong word, but they feel sad. They feel like something is missing. They feel like something important is no longer here. It might be the way that I was used to doing something was really comfortable. And now it’s gone and I have to do something uncomfortable. It may be much more fundamental than that. It may be in the old tool. I was the hero. I knew where everything was. I knew how to look good. Maybe in the new way, I won’t know how to look so good. So I’m giving up something much more fundamental, much more critical to myself. I make it about the tool. Well, this tool doesn’t have XYZ, but I’m really a little bit concerned. Number one, I may not be so good at it. Or number two, let’s say we move to a platform where everybody’s just as good. And then when everybody’s special, then no one is. I lose that there are so many things that are lost in a change that every person experiences differently. Understanding those individual changes goes a long way towards understanding what can be done about it. So it’s not just about something like, hey, here’s the training. Here’s what can be done. But it might be something like you are the expert. And the reason you’re the expert is because you’ve got all this experience. And because of that, you are primed to be the one that can take on this new tool and help everybody move along and be seen as the one that really helped us make this significant change as an example. But it’s addressing what is it that this person is losing and how do you fill something in that can give them something that build them up? 00:27:28 | Val Kroll: So two part follow up to this, because this is exactly what I hope we would be getting into, too. So one is the high level, you know, process or arc that you guys have both referenced, like whether it’s, you know, the town hall announcement followed by an email with a couple of links to FAQs on some Confluence page. A couple of dates included brown bag lunches and like, let me know if you have any questions is the most common way that I’ve seen change managed across a lot of organizations. And if you’re talking about, you know, a tool changer, something like that, you’re talking about hundreds of users. So how do you, so this is the two part, how do you address the individual and like what their resistance is or like the loss that they’re mourning at scale? And then two, how, like, who, who could an analyst recruit to help them through some of this? Because a lot of times we’re doing that program, not because we don’t care about managing the change well, but we really haven’t seen it modeled other ways necessarily. And so is there someone else in the organization that people should be tapping into, or is this another muscle that analysts really need to build themselves? 00:28:41 | Yehonatan Schwarzmer: So that’s my two for it is a challenging question. One of the reasons it’s a challenging question is because even again, going back to this organization, ProSci, who talks a lot about sort of the fundamentals, like what are the pieces that you want to make sure are in place? If you’re going to roll out a significant change, meaning you’re going to roll out a new tool and it’s going to affect a thousand people, right? Like that’s not a simple thing. It’s not something that can be solved with lunch, nor is it something could be, you know, solved by sitting down with every one of them and saying, I understand your pain. 00:29:12 | Val Kroll: What is it then that will overcome someone’s resistance? 00:29:15 | Yehonatan Schwarzmer: And within those thousand people, we have to assume we’re not going to have a thousand people resisting, but we have to know who is. 00:29:22 | Val Kroll: And so maybe starting with what is it that overcomes that resistance in the 00:29:25 | Yehonatan Schwarzmer: first place, and then how do you translate that into an actual plan for a thousand people? So when you think about overcoming the resistance, assuming that you have genuine resistance, you know, typically you’ll be able to overcome it in one of three ways. You’ll either get your, your desire, your entrained desire, you know, everybody talks about, you know, well, if it’s changed, it’s kind of a focus on the what’s in it for me, which is cliche and it’s true at the same time. 00:29:50 | Val Kroll: Right? 00:29:51 | Yehonatan Schwarzmer: Hey, with this tool, you will no longer have to deal with, right? Again, a lot of times we roll out a new tool and people immediately see, hey, this is really beneficial. That’s great. If people don’t see that, maybe you can sell them on it. Hey, let’s talk about the benefits. Let’s talk about the trade. Let’s talk about what we’re exchanging and what we’re trying to do. We’re exchanging and why we’re doing this and what benefits you get out of it and why that might be better for you. 00:30:13 | Val Kroll: There are, there is a one group that could be influenced by that because it’s 00:30:20 | Yehonatan Schwarzmer: really true for them. There is a benefit. They didn’t see it before. They may not have understood it before. And so it’s not to say that the lunches and the trainings and everything else are useless. They’re not useless. They just tend to be that we focus on the areas that they didn’t work. But for a lot of people, it does work. 00:30:34 | Val Kroll: So you’ve got, you’ve got, you’ve got two more. 00:30:37 | Tim Wilson: But I feel like even calling that out, I think there’s a tendency to one parrot. What, in a tool perspective, it just parrots what the vendor has said the benefits are, which aren’t necessarily resonant with the company. The other is the people who are implementing often get so caught up in the, I think the logistics of how is this transition going to happen that does to me sounds like a, yeah, that is a miss. Like probably repetition of these are the benefits that are meaningful for you is actually that this is the little light bulb went on that it’s, it’s easy for that. It’s like, well, they just assume everybody knows, like, sure, everybody knows why we’re doing this. We told it when we sent out the first email and it sounds like that’s something that really should be restated with an opportunity for somebody to ask follow up questions. Like you said this benefit, but what does that really mean? 00:31:34 | Yehonatan Schwarzmer: I think, right? I, I, I say yes. And actually that sort of blends into the second way, which is not such a distinctive way, but it’s the, you know, the, the FOMO approach, the, you know, the, the missing out, meaning if you do get a group of people and they are making significant progress or they do experience some real benefit, then very often that will kind of push the people on the other side to say, I didn’t realize this was, but now that I see what it actually looks like once people are doing it. And this could be because, listen, if you explain it to me at a town hall versus I see someone that’s actually doing it, totally different thing. Um, but as more people start using it, as you start building a little bit of a snowball, again, it’s funny because a lot of these things are sort of seen as cliched and we roll our eyes, but this actually is where people get influenced. When I see someone is doing something and I understand how that translates directly to me, I am much more likely to want some of that. So whether you can build that consciously or you get enough of a groundswell going, then you have a significant leg up from where you were when you were just again, having that brown bag lunch where it was very theoretical to people and most people were not thinking about how amazing this will be for them. It’s morning, what they’re losing. 00:32:48 | Tim Wilson: So, so is this one related? I’m thinking through a client that actually three of us worked on together that that is, that is picking the people who are getting on board and kind of stacking the deck for them to be not just successful themselves, but successful and elevated in public way. I mean, and it’s a little bit of getting them to be champions amongst their peers, but also it’s like, well, yeah, help them, let help them be successful with it. And they’re still going to be talking about what the job is. It’s just if whatever the change, whatever the process or technology change slips into that, then it sort of becomes the subtext. I mean, you know, oh, I thought about this stuff differently and here, look, now I’m actually saying something that’s useful and interesting and actionable. 00:33:40 | Yehonatan Schwarzmer: Yes, is the answer. And I actually, I want to address it, but I want to take one slight detour first because you’re actually, you’re also talking about the third way, which is going to transition right into this. So if the first way is, you know, what’s in it for me? And the second way is kind of the pull as people fear that they’re missing out and they see benefit and they want part of it. The third way is the push, the authority, let’s say, meaning at the end of the day, there are some people that will say, well, I don’t like it. And, you know, whatever it is. And then to have a leadership to say, this is something that is important enough for us as an organization that we will, we need to make this change for whatever reason. And so you have to make a choice. You got to get on board because this is the direction that we’re going in. Meaning there are some people who have the mindset, I’m just going to keep doing it the way that I’m doing it and it’ll probably be fine. At a certain point, that’s not going to work. And so you don’t want it to become a conflict if you can avoid it. But the reality, call it a last course, if you will, but if people aren’t going to buy into it and they’re not going to see the benefits, then they may see, all right, listen, it’s at least better for me to go along with it than the alternative, which I would rather not have. 00:34:51 | Michael Helbling: Okay, so I want to pull in AI now because a lot of organizations are getting this mandate like forcing function with very little instruction on what that actually means because no one really understands what all the parameters are. So you sort of have this existential force, right? So this externality, this forcing change, the leadership is saying, we’re going to become an AI driven organization. We got everybody report back on, I mean, in the most extreme examples, like how many tokens you’re using every day, which, wow. But like, so bringing that into the world of analytics, it’s like, oh, so what am I supposed to be doing now that there’s a dictate to make a change but no instruction for what change is exactly to make or how to make them or that kind of thing. So like that creates sort of like, I feel like a failed structure in a way of like running change management. So like, how do you sort of, how do you exist? 00:35:57 | Yehonatan Schwarzmer: This is just as fun as I thought it would be just for the record. So I want to, let me, let me do this then. I want to get to your question, but I’m going to take a detour first, which is to go on to Tim’s question, which was what the original detour was from because that’s going to get back to your question. Right. So, so Tim. Well, so, you know, you were talking about how do you get those thousand people or what does a leadership group do or how do you get beyond the lunch and learn to whatever it is. And there really are a couple of ways. And then Michael, this is going to get directly to your question, because when you talk about AI, we’re really hitting, you know, every single one of these things, right? It’s a, it’s a huge change. It’s fundamental. It potentially undermines and creates this real fear for people of what am I giving up? Am I going to be able to do it? Will other people get there before me? And what will that mean for me? I don’t even know what it looks like. We haven’t defined the vision. Nobody is even telling me what it’s supposed to look like. They’re just giving me some basic mile markers of more and no one knows what that is. How will I even know if I was successful? How will they know if I was successful, right? There’s, there’s so many areas for that concern. 00:37:09 | Val Kroll: And that’s even before we get to, you know, the complexity of AI. 00:37:14 | Yehonatan Schwarzmer: Really, this could be with any large change, but it’s only exacerbated when we talk about AI. So overall, when we talk about making a significant change and you’re trying to understand how do you, how do you connect the dots, right? I’ve got a lot of people who don’t understand. I’ve got a lot of people who are afraid and I’ve got a lot of people who are resistant. What am I supposed to do? So there’s a much longer discussion that I’m sure, you know, we can have. But in a nutshell, it really does go back to even some of the things that you were saying, which is maybe don’t hit everybody at the same time or maybe think about who it is that you want to talk to first and not think about this as one monolithic change. That could either mean something like, let’s find the early adopters, the people who are going to be excited and let’s talk to them about their role. Your, your role is not here because I want to teach you first. I want you to learn this faster than everyone else and then show everyone why you’re the star of the company of our most stinks. That’s not going to generally create the, the rainbow is feeling that you’re looking for. But if it is, listen, you, these people that I’m having a conversation with you, because I feel like you not only can be extraordinarily valuable to help our company elevate, but you can bring everyone along with you. And so the role that you’re going to play is to learn as much as you can, 00:38:31 | Val Kroll: demonstrate as much as you can, potentially be a resource for others. 00:38:35 | Yehonatan Schwarzmer: I don’t know if that means that they have to have, let’s say office hours. I’ve seen that where, you know, if you learn it well, then you can offer your services as it were to others or just let them know that your door is open. But be that person that shows everyone what the potential looks like, where the excitement comes from, why we thought this was beneficial, and then help them along. That’s one potential way you can do it. Another way you can do it is by focusing on a different group. I’ve got a thousand people. Okay, out of these thousand people, you know, I have a certain number, which are, let’s say the managers, and then everyone else is the primary users that are going to be. So let me start with the managers. Hey, listen, we’re going to make this change. It’s going to be a really significant change. People are going to feel that impact and it’s going to be jarring. So I’m talking to you first, number one, let you know where we’re coming from, how you can best prepare, how you can enable your teams, how you can have conversations with them. A lot of times the conversation from leadership is best received when it’s something about the vision or the strategy or something like that. But reality is that when I, as a, you know, I’m on the front lines, I’m doing the work, I don’t want to hear from, you know, the person who sits on a screen and is, you know, the person I watch while I’m eating my lunch. I want to hear from my manager, the person I talk to every day. So sure, the leader, lay out the vision, help me understand where the company’s going, why this is beneficial, why this is a great thing, give me confidence in the company. All right, like, let me know this. But then when it comes to how is this going to affect me and how am I going to deal with these things that I’m struggling with? I want someone who knows me. So ideally it should be the person that I report to. So if the leader first talks to those people and then they’re better prepared to have that follow-up conversation. So you have the big lunch and then immediately after that, the manager sits with their group of eight people and says, 00:40:20 | Val Kroll: all right, everybody, these are some big things coming. 00:40:24 | Yehonatan Schwarzmer: I’m sure there are some big feelings. Let’s talk about it. Or if you have questions or you need specific help, or if he said some things that weren’t so clear, or she mentioned something and you didn’t know what that means, let me, because the manager has already had the conversation with the leadership and doesn’t, it’s not the manager sitting, and a lot of times, the manager sits there just as clueless as everyone else, but they’re the manager, they’ve got no choice, they’ve got to take the hit. It’s like, okay, everybody, let’s talk about it. Those are great questions, I’ll get back to you. That is not going to instill confidence in anybody. But it means that rolling this out in not a, here’s the training that we will provide to everyone, but really being thoughtful about who can help us get everybody else on board, who is more likely to buy in, who is more likely to demonstrate the benefit, who is a manager who could really talk to people about their concerns. That’s one tactic if you’re talking about a change at scale, so that it’s not just this huge thing that you have to swallow all at once. I will say that another piece of this really does have to do with, I was trying to see if we could avoid the word culture in this entire conversation. And of course we can’t, but it’s difficult because culture means different things, different people, and it’s almost like an excuse word. I don’t want to use it as an excuse word, but the companies that have established a culture or work to establish a culture then enables these types of conversations and that orients everyone towards a very productive, aligned conversation will have a completely different experience than the ones in, let’s say, where there is no communication or everyone is fighting for themselves or there’s a lot of internal competitiveness 00:42:05 | Val Kroll: or there’s silos and turf 00:42:10 | Yehonatan Schwarzmer: and very, very different conversations. I will tell you that in this, the concept itself I think is one that we can all understand. Practically, anyone who has seen it will understand the difference night and day in creating a culture in which these things are real versus where they’re not. I’ll give you just two quick examples. I have a mentor. He’s been a mentor of mine since like 20 years. One of those people that is just giving and kind and thoughtful and wise. His name is Tony DeKemper, lives in Baltimore. And when I was just trying to figure out what I wanted to do with my life, I had some conversations with him. He was fantastic. And ever since then, he has been just incredibly guiding and he ran a company for a while. 00:42:58 | Val Kroll: And in that company, it was bizarre to me. 00:43:02 | Yehonatan Schwarzmer: But any new hire who joined that company had to join a two-day course called Fundamentals of Communication. And it was all about how do we communicate? What is communication? What is the relationship between communication and establishing relationships with other people? How do you grow those? How do you value those? How do you make sure that you’re actually communicating what you mean? How do you make sure that you’ve heard what the other person is communicating? How do you make sure that your communications are designed to build something up instead of just to make your point or whatever else it might be? It was a real commitment on the part of the company and it changed everything. Because everybody who went through that experience, first of all, you go through that experience, you understand what the company is about, you understand that they really mean when they say, we really want to hear from you or something like that. But it also levels the playing field, which is another thing that I think is critical. You all, I think, know. Noah is a guy that I worked with for quite some time. 00:44:01 | Tim Wilson: He will make an appearance later on in this episode, by the way, just so you know. 00:44:05 | Yehonatan Schwarzmer: You can only do well when he joins. But one of the things that I learned from him, and I mean, just working with him, you can learn something every day. But one of the things I learned from him is, when I joined Search Discovery, he was running teams, he was running meetings, and he always started the meetings the same way. He said, the purpose of our meetings to achieve something, and we will only be able to achieve it if all of us are guided by being open and transparent. If we are open and transparent with each other, then we can have real conversations that move things along. Otherwise, we’re going to get locked in, we’re going to get stuck. In those couple of words, that little preamble, he did so many things, and this is important for any kind of management of change. One of the things that he did is he did away with roles. In this meeting, sure, some of us have more experience, some of us have less, but we’re not coming in this in a hierarchy. We’re coming at this because we need to solve a problem. And if you have something that can contribute to solving the problem, and you keep that from us, all of us lose out because of that. You are valuable. Saying to someone that they are valuable versus telling someone they are valuable, I’m sorry, versus showing them that they’re valuable, meaning demonstrating it by saying, 00:45:19 | Val Kroll: I don’t think we have enough in this conversation 00:45:22 | Yehonatan Schwarzmer: because we haven’t heard from this person and the experience that you’re going to bring is unique and no one else has it. 00:45:27 | Val Kroll: That changes the entire feeling of the conversation. 00:45:33 | Yehonatan Schwarzmer: When we talk about overcoming the barriers to change management, I guess I’m just talking about a lot of different ways we could come at this, but the overall idea that, first of all, it doesn’t have to be one big thing that you have to crack with this entire group of as many people as it is that’s very challenging. And the other one is the norms that you establish from the beginning that you will then use to manage through the change instead of dropping it on people through the change because that is likely to end up with more resistance. And I mean, most of us have seen this, a big change is rolling out, and the company, the leadership, whatever it is, says, all of your opinions are important to us. We want to hear from all of you. What does that mean? It means that we made a little slot in the door. You could put that little paper in there, something we’ll find the key and we’ll get into the door. But for right now. 00:46:18 | Tim Wilson: We set up a special email address for you just in your questions too, so we don’t want to load a blind box. You could feel like you’re bringing people in 00:46:24 | Yehonatan Schwarzmer: and you’re drawing people in and you’re bringing people together and you’re creating the alignment, but there is a difference between the kind of paper cut ways that don’t actually do it and the ways that make people feel, I want to be part of this. And even if I am nervous, you know, the term psychological safety, I don’t like it because you can hide so much under it. But there is a principle there where people really feel like if I am uncertain or I’m concerned about something, that’s valid because that is how I will learn. And the point here is that we are all learning so we can all benefit and we can all get better and we can lift each other up. You can’t do that if you’re hiding behind things. All right. Nope. We got it. Sorry, Val. 00:47:02 | Val Kroll: It’s episode 184, 00:47:05 | Tim Wilson: psychological safety and analytics with JD Long, just to, you know, plug that, which was a fun episode. 00:47:11 | Michael Helbling: I didn’t think we would have a shortage of things to talk about and really the shortage is the amount of time we have to explore this issue. You had a 10. Yeah, so we do have to start to wrap up. You should feel bad. We didn’t even get to talk about the measurements side. Shame. Shame, shame. I should feel bad. I know. So, well, obviously that indicates that there’s much more to talk about on this topic. But yeah. 00:47:36 | Tim Wilson: And really Val’s question never really got answered. Oh, well. 00:47:40 | Yehonatan Schwarzmer: We need one minute to try to answer it. 60 seconds. Number one, if you don’t have a lot of change going on or it’s not a big group or change management is not a big concern, don’t worry about the measurement. You’ll overkill it. You’ll hammer it to death and it’s not a nail. Don’t do that. But if it is the type of thing where either you expect resistance or as big and you do need a lot of measurement, then yes, there should be change and there should be measurement of that change. How do you do that? Either you can do that integrated within the process so that if you integrate the change in the process then as you measure the project itself you’re also measuring the change or you measure the change as a parallel work stream in which there should be phases. So, in example, Kotter has a famous model. McKinsey has their 7S model. The ProSci has an ADCAR model. There are lots of models where they walk you through phases of changes. Each of those can be broken down and as you go through the change, you can actually measure how well are we approaching each of those so that we are ready for the change, we’re implementing it correctly and we’ve embedded it correctly. Those can be measured because you can break them down and then they become a separate work stream that will be measured and when you see that that going successful, the project is done. 00:48:43 | Val Kroll: Nice, yeah. 00:48:45 | Michael Helbling: That was actually pretty good. You were very close to 60 seconds on that. 00:48:49 | Val Kroll: Yeah, that was very close. 00:48:53 | Michael Helbling: Yehonatan, what a pleasure. This is so good. Interestingly, I feel like we just started on what I call the top layer of the conversation and I feel like there’s so much more exploration people can do from here. So, thank you so much for opening the book a little bit and I think there’s a lot of applicability right now for people in this conversation. So, really appreciate you coming on and doing that. One thing we do is a last call. Something go around the horn, share what might be interesting. Yonatan, you’re our guest. Do you have a last call you’d like to share? 00:49:27 | Yehonatan Schwarzmer: I will tell you that one of the things I’ve been thinking about for a long time is how technology, specifically AI now, but technology in general adds a benefit, but then very often there is a cost that comes with it that you don’t think about until later. And whether that’s a note taker, that’s really great because you have a great record, but then you didn’t do the physical work of writing something down so you just don’t remember things as well. 00:49:49 | Michael Helbling: Did Tim prep you ahead of time for this? No, actually. 00:49:52 | Yehonatan Schwarzmer: This is like a hobby horse of his right now. 00:49:54 | Tim Wilson: Okay, that might have been a moan, moantim. 00:49:59 | Yehonatan Schwarzmer: But specifically I think about it in the context of education. Education is an area where we are using a lot of technology and there’s the technology that we think is detrimental to children’s growth or students’ growth. Should they have phones in the classroom? Should they get their own Chromebooks? Things like that. But then there are also questions about what about the technology we are using to actually drive that education? And there are some companies that are coming out that want to just have education be only through technology. And I think it’s a big open debate. There’s a guy named Jared Horvath who recently gave a congressional testimony, I think it was, and he’s just recently come out with a book called The Digital Delusion. He was a teacher for quite some time and he’s talking about how education done through technology comes with a lot of benefits, but you have to be aware of what you’re giving up. And then how do you thoughtfully get ahead of those so that you don’t lose as you’re gaining? 00:50:55 | Tim Wilson: It’s so funny. He is actually in our queue as a potential… I recognize that name because he bounced around somewhere. All right. I think he’s a little controversial though, so if I’m correct, right? 00:51:10 | Yehonatan Schwarzmer: Yeah. I mean, none of this is straightforward. It’s more, like I said, it’s something I’m thinking about for a lot because the question is not clear. What are you giving up before you pick up your pitchfork? You got to be relatively clear about that. But people like picking up pitchforks. All right, Val, what about you? What’s your last call? 00:51:26 | Val Kroll: Mine is actually semi-related to the topic, so I’ll be taking those brownie points for that. It is a medium article from none other than Eric Sandosham, one of our favorites. And this one in particular is about how AI is redefining the role of analytics leadership. And he does link, as he does in a lot of his posts, to past articles that plant the seeds of some of that thought, and they’re all worth reading as well. But one of the things that I think is really interesting is he’s talking about this paradigm shift and how we’re thinking about the new chief analytics officer and what makes a good one is to redefine one of the core competencies. Instead of focusing on solving problems, it’s going to be more about the ability to problem find, problem find and problem define versus the actual solution of it. And he goes into the T-shaped skills and operating as decision scientists. And so anyways, it was a really thoughtful read, and I just found myself highlighting a bunch of excerpts in it and shooting it off to random people. So I was like, I need to share it here as well, but it was definitely a good one. 00:52:32 | Michael Helbling: Nice. Very cool. I’ll check that out. 00:52:34 | Tim Wilson: Tim, what about you? So I alluded to it earlier that our former coworker of all of us, Noah Levin, has started a sub-stack that he has posted some stuff on that is, Jonathan, as you said, he is super thoughtful. So the one that really triggered me also sort of relates to this topic. He had a piece called Let My People Code, the part of the AI debate nobody wants to say out loud, which maybe sounds a little clickbaity because he does have a marketing background. But the essence of it is he’s kind of making the case that one of the real tensions around AI, it’s not just about declining quality, it’s about experts grappling with the loss of exclusivity 00:53:21 | Val Kroll: as a way many more people can be competent. 00:53:25 | Tim Wilson: So even Jonathan, as you were talking about, the person who was the Power BI guru, and now they’re moving into Tableau and they’re losing that, and he has, it talks about Wikipedia, it talks about the printing press, and a little bit of it is a little bit of like, you know, check yourself. For me, looking at it, I’m like, I don’t want, I don’t want code, code, code, writing R because I work so hard. If everybody can write R, what am I doing? So it does force a little bit of what’s really going on when you’re resisting, when you’re much more democratizing an ability to a base level of competence. Which just to add on to that, Michael, it was two plus years ago when we were at Measure Camp New York, when you made the comment about generative AI making, getting everybody to average on stuff. Like that, that really stuck with me then. Noah’s piece is saying, people get, experts get upset that people can do average, but average is often, you know, good enough. It doesn’t mean you don’t need the expert. So, and it’s, he has all sorts of historical references and stuff. And he’s a fantastic writer. So it’s a great read. Michael. 00:54:40 | Michael Helbling: So, yeah, I, so there’s a person who frequently in my career has somehow said something much more succinctly than I’ve been able to enunciate it. And this happened again a couple of weeks ago at. Okay, it wasn’t me. Mark, you know, no, it’s not you, Tim. 00:54:59 | Yehonatan Schwarzmer: Sorry. 00:55:02 | Michael Helbling: I don’t think anybody was taking that one. Not guilty either. Yeah, I see the keyword was succinctly. Yeah, no, so I was having this conversation at marketing a little something about some of my concerns about how anthropic is kind of doing some of the things they are behind the training and the adjustment of the model and how they’re kind of giving it sort of like this sense of self kind of concept and everything. And then Mark, David McBride, pointed me to an article by Gary Angel who is that person. And it’s the articles called Can You Teach an AI Ethics? And he basically dove into specifically, there’s a Wall Street Journal article about Amanda Askell who’s their philosopher kind of training the AI on ethics and morality and he kind of got into a very interesting examination of that and I really benefited from reading it and sort of helped me kind of like bring my thinking more to a point on some of the things that were just sort of like hanging out at the edges of like, I don’t know if I really agree with that and I’m not sure why. Like Gary’s thinking has always been really helpful to me in that regard. So I highly recommend it. 00:56:19 | Tim Wilson: Gary started publishing more frequently on Medium which is always a highly recommended read. 00:56:25 | Michael Helbling: He’s fun to read since forever and so yeah. So that’s, I would highly recommend that. All right, well we’ve been chatting and I think it’s for sure. You’re going to have questions, you’re going to have thoughts, you’re going to have comments. We would love to hear from you. The best way to do that is to reach out to us. We could do that via our LinkedIn page. You could do that to the Measure Slack chat group or by email at contact at analyticshour.io and if we’re re-listen, leave a rating and review on the show as well. And if you would like stickers for your laptop because even in this day and age of AI, we still need stickers on our laptop to show what tribes we belong to, you can reach out to us on analyticshour.io. There’s a page and request form for that. All right, Yehonatan, thank you. 00:57:18 | Yehonatan Schwarzmer: Thank you so much. Thank you. It is always nice to talk to each of you and all of you together as a real treat. 00:57:24 | Val Kroll: I know, I know. 00:57:26 | Michael Helbling: This is good. This is really good. 00:57:29 | Val Kroll: All right, and I think we’re all facing change 00:57:33 | Michael Helbling: and no matter what kind you’re facing and whatever stage you’re in and change that change curve, one thing you should always do and I think I speak for both of my co-hosts when I say keep analyzing. 00:57:45 | Announcer: Thanks for listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at @analyticshour on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group. Music for the podcast by Josh Grohurst. Those smart guys wanted to fit in. So they made up a term called analytics. Analytics don’t work. 00:58:09 | Charles Barkley: Do the analytics say go for it no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 00:58:23 | Michael Helbling: I feel like I might have hit a nerve there. 00:58:28 | Tim Wilson: I was trying to be, would that come through? Would that work if I did? You’re fine. It’s totally fine. Yeah. 00:58:34 | Val Kroll: No, it’s good. 00:58:35 | Tim Wilson: Okay. I had a few other. I was going to do Kubla Ross five stages of grief or detours for the other three. 00:58:41 | Val Kroll: Yeah, I thought you were, I always try to guess you know what he’s going to say. I thought you were going to say stuck in what, what, what anger? He said, Michael, with the GA4, I think I’m such an anger. Oh yeah, I’m still stuck in anger. Yeah, I’m still stuck in. 00:58:53 | Charles Barkley: I have an immersion in anger. Still stuck in anger. 00:58:55 | Val Kroll: Yeah. Yeah. Still stuck in anger. 00:58:59 | Michael Helbling: Yeah. We’ve done a number of the things in that document. So. 00:59:08 | Tim Wilson: Well, episode number 299, take three in which Michael has followed the recommendations. 00:59:17 | Michael Helbling: Why don’t we, not all, but I’ve got all the recommendations. Oh, like I rebooted my Wi-Fi point, which has been functioning flawlessly for weeks, right up until today. And then I rebooted Chrome too. So hopefully that’s enough to get us through. Oh, we’re not going to do a full system restart. No, we don’t want to. 00:59:41 | Tim Wilson: No, we’ll say that’s our episode 300. 00:59:43 | Val Kroll: We have to have something to celebrate. 00:59:45 | Michael Helbling: Yeah, yeah. And I don’t think that’s the issue here. I’ve been putting my computer through a lot of change lately. I like it. Have any of your GPUs? 01:00:04 | Val Kroll: Yeah, just I’m doing a lot of stuff. 01:00:07 | Michael Helbling: Okay, let’s get this show on the road. Okay, here we go. Here we go in five, four, three. 01:00:28 | Tim Wilson: Rock flag and succinct refers to the expression of ideas, information, or arguments in a manner that is remarkably brief, clear, and to the point. As a succinct statement or piece of writing, I’ve always unnecessary words, fluff, and excessive detail, capturing the core essence of a message with maximum compression. The post #299: AI Can (Help) Build the Dashboard. It Can’t Build the Buy-In. appeared first on The Analytics Power Hour: Data and Analytics Podcast.

May 26, 202652 min

#298: Listener Questions Answered Live from Marketing Analytics Summit!

Picture this: four analytics professionals, one live audience, a bunch of submitted questions, and absolutely no filter when it comes to sharing their real thoughts about AI, stakeholder management, and the state of the industry. That’s what you get when the Analytics Power Hour goes live from Marketing Analytics Summit, with Michael, Moe, Tim, and Val fielding everything from, “How do I prove I’m a partner rather than just an order taker?” to “What’s your icky threshold with AI?” The conversation ping-ponged from the fundamentals—like why curiosity beats feature checklists when selecting tools—to the controversial, including a heated debate about whether AI-generated meeting notes are helpful productivity boosters or lazy crutches that strip away human editorial judgment. Along the way, they tackled data trust issues, the pressure to show AI efficiency gains, and why trying to nail down the “best” deliverable will just trigger existential musings about what a deliverable even IS! Fair warning: Tim gets triggered by AI hype, Moe calls some industry BS, and everyone agrees that being useful beats being right. This episode is brought to you by Prism from Ask-Y—your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Episode Transcript00:00:00.00 [Announcer]: Welcome to the Analytics Power Hour. 00:00:08.92 [Announcer]: Analytics topics covered conversationally and sometimes with explicit language. 00:00:15.56 [Jim Sterne]: The Marketing Analytics Summit is pleased to present these four amazing podcasters. 00:00:24.80 [Jim Sterne]: We have Michael, the self-effacing every-man consultant who knows far more than he lets on. We have Moee, who flow all the way in from Sydney, Australia. She’s the determined, driven practitioner, the one to stand up and say, well, that’s all very well and good, but how do we actually make it work? There’s Val, who co-founded the Consultancy facts & feelings, because we have strong feelings about our facts and no facts about our feelings. And there’s this guy named Tim. Ladies and gentlemen, take it away. 00:00:56.72 [Michael Helbling]: Hi, everyone. 00:00:58.22 [Michael Helbling]: Welcome to the Analytics Power Hour, this episode 298. And we are recording live at the Marketing Analytics Summit in the beautiful Santa Barbara. You know, 21 years ago, my very first Analytics conference was right here in this city at this conference, formerly known as E-Metrics. And similar to today, it was a conference full of smart, engaging, and passionate people learning together about how to solve the problems they were facing day-to-day in the analytics industry. A lot has changed. A lot has changed. But persistent through all of that is this beautiful analytics community. And the shared learning, this podcast was actually created to encourage and celebrate. So to that end, I have my co-hosts with me. And we have put out a survey to gather your questions. We’ve been asking you during the conference. And so we have a number of them to get through, and we’ll do our best to answer them from some of the perspectives we bring to the table in our various roles. So let me introduce them. Moee Kiss. 00:02:13.44 [Michael Helbling]: Hi, folks. 00:02:14.44 [Michael Helbling]: Hi. 00:02:15.44 [Michael Helbling]: Director of Data for Product, Canva. 00:02:19.44 [Michael Helbling]: And of course, Tim Wilson, Head of Solutions at facts & feelings. 00:02:24.16 [Michael Helbling]: Hello. 00:02:25.48 [Michael Helbling]: And Val Krull, Head of Delivery at facts & feelings. 00:02:29.04 [Val Kroll]: Hello, hello. 00:02:30.04 [Michael Helbling]: And I’m Michael Helbling, my president of Stacked Analytics. 00:02:34.72 [Michael Helbling]: Okay. 00:02:35.72 [Michael Helbling]: What a privilege to be here with all of you in person. It’s been a wonderful couple of days. So let’s just dive right into it. 00:02:45.36 [Michael Helbling]: So we’ve got a question. 00:02:48.24 [Michael Helbling]: This one came from James from Namer’s Children’s Health. What’s the best approach to establishing yourself as a partner and not an order taker when working with stakeholders? 00:02:59.12 [Val Kroll]: James is the one who has the most trouble with the uncomfortable pause. So that’s why he looks the best. 00:03:09.76 [Moe Kiss]: Oh, I assumed that was your cue. Anyway, we were having a conversation earlier about taking notes over lunch. And if you don’t want to be perceived to be the person in the room taking notes, then don’t take the notes. And I think it’s really jarred with me because I have this belief, I mean, I always know that I come back to everything that Cassie Kazarkoff says about just be useful. And I am sometimes fine. I’m still like, I might be the most senior person in the room and sometimes I do still take notes. Part of that’s my brain, but part of that is also that I want to be useful and the people in the room are normally smarter than I am. And I have much more interesting things to say. So I actually kind of struggle a little bit with the question in and of itself. I think one of the things that maybe means that it’s OK to still take notes, but is also adding your voice to the conversation. 00:04:02.44 [Michael Helbling]: I think, I don’t know, I just, again, we were chatting about this at lunch. 00:04:06.20 [Moe Kiss]: When you don’t contribute to the conversation, when you’re not willing to be accountable for what you say, I think that’s when you kind of, you’re not the partner, so to speak. 00:04:15.36 [Tim Wilson]: I mean, I think it’s, I agree. 00:04:17.00 [Michael Helbling]: I think it’s don’t be an order taker. 00:04:19.12 [Tim Wilson]: I think to me it is, and there have been a lot of discussions at this conference, a lot of the discussion in the industry were at large around how are we using AI and there to be curmudgeonly and a little cranky about it. But a lot of is, how do I take more orders? How do I take more orders and produce more output? And that misses, to me, the fundamental part of becoming more of a partner is to step into their shoes, ask the questions, write it down, be genuinely curious. Don’t be trying to get to, how quickly can I turn this into a ticket? And I think that has been, AI has nothing to do with that. That has been a 20-year problem in our industry when we sit around and wring our hands and gnash our teeth about why aren’t we brought in. We produced the next dashboard like they asked us to, we gave them recommendations, but did you ever actually sit down up front and say, I just, I’m an analyst, I’m curious, I want to understand what you’re grappling with. And there have been, even at this conference, there have been some discussions around that. But I, I lay awake at night fretting about the industry being, I want to be a partner, but I behave like an order taker. And it’s like, it’s simple, just be curious, go ask them, get inside their heads and start asking them, how can I have some problems? 00:05:41.96 [Michael Helbling]: Yeah, okay. 00:05:42.96 [Moe Kiss]: I kind of want to call a bit of bullshit on it though, because I do, I get- 00:05:46.88 [Michael Helbling]: On brand? 00:05:49.72 [Moe Kiss]: I get what you’re saying, and I hadn’t really thought about like AI driving more of the order taking. I actually feel like my experience might be a little bit different. I think the thing that I just keep observing is data folks don’t want to be accountable. I had this conversation at the back of the room earlier, it’s like, they want to be like, here’s the numbers, don’t look at me, like I don’t want to be on the hook for the recommendation that I’m making. And I don’t know, I don’t think that’s an AI thing. I think that’s just a, maybe it’s a 20 year old industry thing. 00:06:19.68 [Tim Wilson]: I think it is, I mean, I actually agree that it’s like, we make recommendations and they’re not following them. And a lot of times like, where did this start? Well, we asked them what they, we asked them what their business question was. If you’re asking them, even if you’re saying, I’m asking what the business question is, there is a human element of saying, I am your partner, I am in this with you together. So I’m just pushing for moving upstream and not jumping to the data and jumping to the solutioning and jumping to, did I produce the published report that made a recommendation and then I’m upset. 00:06:53.84 [Michael Helbling]: So you didn’t fix that? 00:06:56.88 [Val Kroll]: Yeah, I’m just kidding. I would say like to get started with that, it’s not even always about like anticipating all of their needs. I think it is being curious and just thinking about the motivations and the stresses that that person has in their role, like what pressures are they facing? And I think that this has come up a couple of times too about how do we de-risk the decision that’s upcoming for them or where should they spend that next dollar and do a little bit of prep work before you walk into that room to show that you’re trying to empathize with what they’re dealing with because their job is hard too. It’s very different than yours, but starting from that place, I think is a good way to kind of say like, we’re in this together, we’re side by side, not sitting across the table and I’m just going to push something on you at some point. 00:07:38.52 [Moe Kiss]: But I think the difference there is you talked about preparation and I think a lot of the curiosity can come without necessarily absorbing stakeholder time. And I worry sometimes that the push is like, oh, well, I haven’t got enough context. My stakeholder is time poor. So, you know, that sort of thing where it’s, I actually think there’s a lot of personal responsibility that data folks need to take to be like, I have enough information available to me throughout various, various Slack channels, emails, et cetera, to have some of that curiosity before even asking the question of a stakeholder. 00:08:10.68 [Tim Wilson]: 100%. 1000%. I see your 100% and raise it. But I think if you’re thinking about it, you should be saying, this is my understanding, but here’s something that I can’t. I listened to our quarterly conference call and there was something that doesn’t make sense to me. I tried to figure it out. It doesn’t. I think you, marketing manager, might help me explain this because I can’t square that circle. So, yes, showing up and that certainly worked with analysts who say, well, what are my lists of discovery questions? What is your most challenging business problem? If that’s the way you show up, they’re going to be like, I have to explain every minute. 00:08:44.24 [Moe Kiss]: 25-year-old data slanted slightly because that’s kind of how they show up sometimes. No age. 00:08:49.84 [Michael Helbling]: No age. All right. 00:08:51.84 [Moe Kiss]: We’re going to move to our next question. This is the whole episode. 00:08:53.84 [Michael Helbling]: Sorry. And this is coming from a member who’s here in the audience, Michelle Kiss. 00:08:59.12 [Michael Helbling]: Whoo. 00:09:00.12 [Tim Wilson]: Any relation? 00:09:01.12 [Michele Kiss]: None. Completely the same person. Complete coincidence. Yes. Confuses everybody. Okay. So, first of all, my question has two parts. There’s a pre-question that is required for setting the context. 00:09:15.12 [Tim Wilson]: Yes, they’re related. 00:09:16.52 [Michele Kiss]: And so that we can interpret your answer. Okay. So, first of all, the pre-question is where do you guys stand on AI? Are you a skeptic or a fan? Okay. So, that’s the context. Then, what things have you personally found that it is useful for and reliable for? It has to be life. 00:09:42.88 [Val Kroll]: Okay. 00:09:43.88 [Michael Helbling]: I’ll go first. 00:09:44.88 [Val Kroll]: So, I would say that I’m a skeptic, but that’s just kind of like my nature. I don’t think there’s anything new or special about that. That’s just kind of who I am as a person, but I’m excited at the same time. And I think I’ve been inspired by a lot of the conversations we’ve been having over the past couple of days that have gotten my wheels turning about some different things that I can try when I get back to my desk. So, I would say more on the skeptic, but still excited, still optimistic. I don’t think, you know, world’s ending just yet. One of the things that I have been playing around, this is recency bias, but just thinking about how do I start at like day 60, day 90 with the where I’m at with an understanding of the business context for a potential prospect conversation. And so, we had been doing a lot of process effects and feelings very manually to kind of go through and listen to like most recent earnings calls or pulling out, you know, press releases or seeing what was publicly available just to kind of understand what is the context, what is the competitive nature, where are they competing and using AI and building out some gems to build that out for me. And so, I’m asking it to give me some tables and comparisons of where do they compete on what message is? Is this about pricing? Is this about quality? And so, really trying to figure out like how do I get into that context so that I’m not asking like question number one, what are your goals? It’s like, let me start with something because then I’m going to be able to ask much richer, deeper conversations. And so, again, recency bias, but that’s been the most fun one just to kind of understand outside of an analysis, what’s something I can do to show up in those conversations to kind of position myself to be a better partner. 00:11:26.88 [Moe Kiss]: Mine are not sexy. Like I don’t… That wasn’t sexy. Build me a table of your competitors. Well, it depends what you think is sexy. Yeah, I mean, I do a lot of leadership-y things and those things tend to be quite admin-y. I do think the one… Oh, I forgot your first part of the question. I’m excited and terrified. I would say I’m… I don’t… Fan feels like a strong word. I do like a lot of the potential and the personal gains in the way that it’s changed how I work, but I have this like level of fear, which I think is pretty rational and normal. 00:12:05.76 [Michael Helbling]: It’s a pretty big shift. 00:12:07.40 [Moe Kiss]: And yeah, a lot of the ways I use it are not sexy. But I do think the thing that has been really nice for me personally is feeling like code is not so far away. Like it’s been a couple of years since I wrote code very regularly, and there are things that now I feel much more comfortable. I’ll go do a quick Here’s the Queer I Want to Vibe code something. 00:12:26.56 [Michael Helbling]: And I also know enough to know if it’s not correct, which is important, but it makes 00:12:32.44 [Moe Kiss]: it feel like more approachable to a skill that’s pretty rusty. The rest of the stuff is like making my snarky comments in Slack, seem less snarky, like meeting summaries. Like, yeah, it’s not sexy. 00:12:49.48 [Tim Wilson]: So I’ll say I’m a skeptic and a fan. So I’ll punt on the first question. My skepticism comes from I’ve been developing this observation that I feel like we are as analysts, we are often saying this is the thing the tool can do. This is my hammer. So now I’m going to go figure out what the nail is, and I’m going to tell myself that the nail is going to solve a problem that has nothing to do with the technology. So I have found it to be very useful on debugging code, kind of piggybacking off of what you said, because coming from having written code, debugging it. It is very, very useful. I get very, very nervous when I have Vibe coded and yes, it brings results out. But I’ve now seen in the wild how that can go awry. I actually find that it’s very useful just from a thought clarity. And I think that hasn’t changed. That probably took me three times listening to Jim Stern, give various presentations about writing prompts for it to take. And it was the first time I heard it now, kind of everybody saying it of some of the keys with writing prompts, but prompts, I think by writing. So saying this forces me to organize my thoughts and then I’m going to trust and verify what comes back. That is much more on the help me clarify my thinking, because if you’re helping me clarify my thinking, just like a human being who I think, no, that’s wrong. That’s bullshit. You can challenge me, but you may be wrong. I am not, and which wasn’t the, where is it? Not useful, but I, anytime it’s like, it’s going to basically do glorified anomaly detection and spit out insights. Yeah, I’ll go to the mat for a while on that. And I’m pretty triggered with various claims to have it do that. Michael, what happens after you finish a GA4 analysis? 00:14:43.32 [Michael Helbling]: Oh, traditionally, I guess I paste screenshots into a doc, rename it final, final, uh, you know, do not change final, uh, lose the source query and then wait for somebody to ask, can we break this down by campaign? Terrifying. Please stop. I would love to emotionally and professionally. 00:15:05.12 [Tim Wilson]: Well, that’s why ask dash, why.ai just released the Prism Cod co-work connector. It brings the whole Prism brain to your GA4 big query data. Ooh, the whole brain analytics, agent harness, skills, memory engine, the works. 00:15:24.32 [Michael Helbling]: Wow. So co-work doesn’t just answer questions. It remembers context, uses repeatable skills, keeps analysis 00:15:30.82 [Tim Wilson]: organized, exactly. You’re, you’re picking it up. Your co-work based analyses are accessible in Prism, organized, traceable, auditable, and ready to use with your other data sets. 00:15:43.06 [Michael Helbling]: I love that. Cause currently my audit trail is mostly like, oh, I know I had a reason for doing that. I can’t remember what it is. 00:15:50.54 [Tim Wilson]: That checks out. The connector also ships with ready to run funnel and cohort skills right out of the box. 00:15:57.86 [Michael Helbling]: So I can ask for retention by acquisition channel and not immediately enter a fugue state. 00:16:02.58 [Tim Wilson]: Right. And every analysis becomes a shareable page. Prism auto-generates the dashboard page right from co-work. Oh, so the answer doesn’t die in a chat thread. That’s right. It lives as a reusable and shareable analysis. Well, that’s very rude to my old workflow, but fair. We’ll go to ask dash, why.ai. That’s ask dash the letter y.ai and sign up for the wait list. Yeah. 00:16:27.70 [Michael Helbling]: And use code APH and that’ll get you pushed to the top of that wait list. Ask dash, why.ai code APH. I like it. Co-work. 00:16:37.46 [Michael Helbling]: It’s GA for analysis, but with receipts, which was not part of the question. 00:16:43.74 [Val Kroll]: We’re only on our second question. You can’t, you can’t get triggered yet. 00:16:47.70 [Tim Wilson]: No, I welcome it. I was triggered 36 hours ago. 00:16:50.14 [Michael Helbling]: I think it’s okay. I go through life triggered. All right. We have another question from someone here in the audience. So I’ll hand it off to Jen Coons. 00:16:59.22 [Jenn Kunz]: It actually follows up with what Tim was just saying, perhaps a bit of it. Do any of the ways we promote and use AI in the industry make you feel icky? Do you have a threshold of lines that you don’t want to cross when it comes to AI? 00:17:12.98 [Tim Wilson]: Can I add, I’ll add a new one to that. I am not a fan of the note takers in meetings and doing the summaries, which I know is super controversial just inside. And because to me, I’ve watched and I’ve watched this time and time again, how much that drives laziness and not paying attention in the meeting and not editorializing. And it is flat summaries. I’ve worked with some clients where they have literally said, oh, you weren’t at the meeting, we recorded it. Here’s the Gemini summary. And it’s just not useful because it’s not telling me what the human beings in the room were thinking about. Let me throw one other, I’ll be quick. Because, Jim, when you did your closing note, you know, yesterday, which was very good, not going to be too useful for people who were listening to this, but there was a lot of talk about using AI to ramp up junior analysts and build lots of different, use different tools that kind of let them kind of do self study. And I wound up wondering about where do we teach the junior analysts, how to actually relate with people and have the creative, collaborative process. And so there’s something there, too, that makes me nervous that we’re at a conference right now, like are we on some logical trajectory where we think we’re all just going to sit at home and just have AI do everything that needs to happen. There is a very, very real part of this job that is human and communication and collaborative creativity. And I get very nervous that people are not recognizing the value of that and trying to have AI replace it. That was really deep. 00:18:56.02 [Moe Kiss]: I was going to talk about meeting notes. 00:18:58.38 [Tim Wilson]: Go for it. Do I need to move away? 00:19:01.22 [Moe Kiss]: No, but so for me, I actually find the meeting notes summaries incredibly useful. And the biggest thing for me is, like, as someone who self-declared 00:19:10.78 [Tim Wilson]: has ADHD, can you square the meeting notes with the taking notes by hand? 00:19:15.98 [Moe Kiss]: I do. So I do still often also take notes. But the taking notes for me is me, number one, it helps me pay attention in the meeting. And number two, it helps me retain the informations. Like if we’re talking about especially something complex, I won’t necessarily fully hear it. And then the next day, I sometimes do look at my own notes. I won’t necessarily look at the notes. But what I find useful is the next steps. There is always like, no, when you are trying to corral like 20 people, 00:19:47.02 [Michael Helbling]: you can be like, what is the big deal? 00:19:51.42 [Moe Kiss]: What is it? Why is that the big deal? 00:19:53.02 [Val Kroll]: Tim, just fell off the stage for those of you listening. That was a Redd Foxx impersonation for anyone who walks. 00:19:58.82 [Moe Kiss]: It’s like, this person’s going to follow up with this. By then this person’s going to follow up with it. 00:20:02.46 [Tim Wilson]: And because it’s actually fucking terrible at that, it just goes through and says, this is what it was. And you need the human editorial person saying, what are the real next steps? That’s where it’s trying to go through a discussion. 00:20:15.70 [Moe Kiss]: To get back to the question, though, of what makes me feel ick, what makes me feel ick is things being shared that have not been properly vetted and QA and meeting notes fall into that category just as much as analysis or write up does. Like that’s a bit that I get stressed about. Is it analysts are like, yes, I can pump out more stuff. I’m going to automate this report. I’ll send it out. I’m never even going to look at it. And I’m like, oh, that feels uncomfortable. 00:20:41.98 [Tim Wilson]: But isn’t the meeting notes is asking people to do a lot because they’re like, I got to get the meeting notes out promptly. And it takes an enormous amount of diligence to say, I am truly going to read through these and modify and write my own little summary and write. So it’s like, you can paint the picture, but watching what actually happens with the people I’ve worked with, all of a sudden I’m like, oh, this was just barfed out. And I’m sure they scan through it and I’m sure they told themselves, yeah, that seems about right to be fair. 00:21:13.50 [Moe Kiss]: I just take the like the here are the action items. I don’t send the whole summary. Is that worse or better? 00:21:19.86 [Tim Wilson]: I well, if you take them and you say, yeah, that looks about right. I think that’s a problem because I think there is much more often. There is the person who’s sending those out should have a responsibility to say these are the things that really need to happen. There’s a level of prioritization and wording and body language and and adding net new stuff that I know that Joe said that Joe was going to do this. But the reality is I know in our organization that Joe is going to have Mary work with him on this and putting that sort there, there is something in that maybe I’ll get off that. 00:21:56.54 [Moe Kiss]: There’s a lot of nods, so I’m interested to hear more Tim. Normally, I don’t have this kind of live feedback that suggests maybe you’re right. 00:22:02.62 [Michael Helbling]: That they’re agreeing with me. 00:22:05.66 [Michael Helbling]: I think I’d go in a little different direction, which is I get sort of this icky feeling or there’s a threshold. I don’t want to cross with AI in terms of relating to it. And what I mean by that is some of the LLM big companies put out research where they kind of take what the LLM is doing and equate it to emotion and telling you that basically if you behave around your AI a certain way, it may actually impact its performance and that’s sort of what they’re seeing. And I think it’s a really dangerous thing and I think it’s also tricky to talk about because I don’t want to advocate for being mean to your AI or whatever. But as humans, we anthropomorphize things really a lot. And so I think we very easily buy into this idea that I make my AI feel bad if I yell at it or I make it feel good if I tell it does a good job. But in reality, it feels nothing. And most importantly is human to human interactions. I modify them a great deal based on what I know of that person and the empathy and the intuition I’m getting from that conversation. 00:23:13.46 [Tim Wilson]: So if someone’s struggling, I modify empathy and modifying Tim’s going to need a definition of hang in there, 00:23:22.74 [Michael Helbling]: and so you adjust to that person like if you’re giving feedback, for instance, whereas if I’m giving feedback to an AI, I want to be as direct and succinct as possible without having to kind of couch it in a phraseology or terminology that keeps it secure in its own quote unquote emotions, which are not real. And so for me, that’s sort of a weird line that I think I’d love for us to avoid 00:23:48.58 [Michael Helbling]: as we approach AGI. 00:23:51.86 [Moe Kiss]: Can I ask a crowd question? Only if you can figure out how to get a mic to them. No, I was going to make you count the hands for a rough estimate. Well, if you each take one and we take the average of each of your estimates, maybe we’ll have a decent score. OK, creating an agent based on your stakeholder one, stakeholder two, stakeholder three, so that you can tailor your comms and have a persona built out for each of it. Like, I want to know how we feel about the ick factor of like, is that icky or is it thoughtful? Because anyway, I can finish my thoughts afterwards. 00:24:30.66 [Tim Wilson]: Sam Bert, would you like to answer that question? There was a whole session on that. 00:24:35.58 [Michael Helbling]: So here’s my take on that, because I learned about that yesterday in a session and I actually really quite liked it, because I look at AI as a tool to take information and position it in the best possible way for the audience that you’re presenting it to, much like you would present maybe a slide deck to one person and a narrative to another based on how they consume or prefer data or information to be consumed. 00:25:05.82 [Moe Kiss]: Two notes. Just to be clear, I’m not picking on Sam. That was like very persona based. I’m talking about like someone in your team creates one that’s like, this is Moee. This is how she receives information, like very personalized. I think that’s a bit different. 00:25:19.66 [Tim Wilson]: So I think and I think there’s two aspects. So I think Sam’s and yours. One, I think if it actually forces the person who’s creating it, this is where we to actually really think about like to create it, you have to think about what do I need to? What do I know about Moee? What have I seen about Moee? Can I ask Moee something? 00:25:40.18 [Moe Kiss]: So that’s like, but do you think that they would or do you think they would just be like, I’m going to upload 50,000 conversations, a bunch of Zoom transcripts, whatever of interactions with this person and you tell me what you think they’re going to like. 00:25:52.42 [Tim Wilson]: I think that’s going to be less effective. 00:25:54.06 [Moe Kiss]: Yeah, it would probably would be. 00:25:55.66 [Tim Wilson]: And then the second part of that, I completely lost what my other thought was. So on Brent. 00:26:04.74 [Michael Helbling]: Well, a lot of these questions are tailored around a bunch of information. I uploaded about you three. No, I’m just kidding. OK, we have another question coming from someone who’s unfortunately not here. Joe Domoleschi, if you were stripped of your fancies tech stack and could only provide one specific deliverable to a stakeholder to prove the value of marketing analytics, what would it be? Live dashboard, a PDF report, a slide deck, a meeting with actionable insights, etc. What would you do? 00:26:34.66 [Val Kroll]: So one specific deliverable to prove the value of marketing analytics, 00:26:38.46 [Michael Helbling]: just to clarify, that’s what it says. 00:26:41.14 [Michael Helbling]: Yeah, OK. 00:26:44.66 [Val Kroll]: The results of an A.B. 00:26:45.66 [Michael Helbling]: test. So Format can be anything. 00:26:50.42 [Val Kroll]: I mean, well, he said deliverable. OK, so I would I would do a presentation, tight narrative results of an A.B. test. That would be my that’s my no explanation. 00:27:02.30 [Tim Wilson]: I think my not might not be a prove, but might be to convince or define. I would probably go to some sort of compelling story that I was comfortable. I might have pulled data from various sources. I might have run an A.B. test, but I would actually tell a really strong narrative and maybe with slides. Maybe not. I think that would actually be more convincing. 00:27:30.10 [Val Kroll]: Like, why does said deliverable, that’s your constraint. 00:27:33.70 [Moe Kiss]: But see, OK, this is the difference when I heard deliverable. I was like, it can be anything. And it sounds like you’ve both interpreted that in like agency consulting land very different to me, because I would have said if I could pick anything, it would be an MMM. Like, I can talk about that for weeks and months. I mean, at some point, it becomes valid. But it would probably be an MMM. Like, I’d love to go through an experimentation tool, but I feel like that maybe is not in the spirit of the question. 00:28:00.90 [Tim Wilson]: I don’t know. Yeah. What is it? What is a deliverable? 00:28:03.18 [Michael Helbling]: We could get that was not the question. 00:28:06.90 [Tim Wilson]: No, it’s what is the deliverable is. So now I found sound like a consultant. But I mean, on an MMM being super compelling, and I’m sitting like 15 feet from Jim Janolio. So and having heard him talk like you can you can conduct a great MMM. You can you can build it and then you can deliver it horribly and doesn’t show anything or you can communicate it really, really effectively. So it is kind of what is a deliverable and how effectively is it created and delivered? 00:28:35.26 [Moe Kiss]: Or we could just combine all three and then we’d like be winning. If we’re perfect, you didn’t answer yourself. 00:28:43.18 [Michael Helbling]: I’m going to ask the next question. 00:28:46.10 [Michael Helbling]: From a member of our audience, Bryce Preslicka. 00:28:52.14 [Michael Helbling]: Preslicka. Preslicka. 00:28:58.58 [Brice Praslicka]: All right. So I have a client that had some major data issues previously. Once we got on to the project, we’ve cleaned things up. We’re in a much better state now. The problem is one of the clients, POC’s continues to act as though we have unreliable data. And even worse, we have a member of our team that continues to use words like discrepancy and lack of trust and keeps using words that don’t really help us accurately convey that it’s reliable now. 00:29:24.78 [Michael Helbling]: So I’ve sent memos. 00:29:27.38 [Brice Praslicka]: I have tried to coach them to not use certain words, but with both an internal team and a client that don’t trust data that is in much better spot now, how would you go about trying to regain trust? 00:29:40.26 [Val Kroll]: In the role of the client you’re supporting, are they on the business side or are they in an analytics role? 00:29:45.82 [Tim Wilson]: They’re in the business side. Well, now you have to provide an answer because he was just clear. So here, I’ll kick us off. 00:29:54.14 [Michael Helbling]: There’s no time for cash measures. Jeez, we’re in a world of AI now and we need to move quick. First thing first, stop inviting the internal person to the meetings. So they can’t screw you up. Second, take charge of those meetings and tell the client that you know what you’re talking about. And it’s time to make decisions and get off the pot. What are they afraid of? No, I don’t know if I could pull that one off. 00:30:16.50 [Michael Helbling]: But but, you know, start to form the communication 00:30:20.46 [Michael Helbling]: and put it into the positive realm so you get past that moment. 00:30:24.18 [Tim Wilson]: I I mean, this is going to sound easier than it is in practice. But to me, one of the things that AI has not helped we have struggled with for for 25 years is businesses that are looking for certainty and precision when there is actually they’re operating under conditions of uncertainty. And Jen Kunze’s presentation yesterday, like it’s like people think that, oh, the data, the data was never complete. The data was never perfect. It’s really no better, no worse. We can always point to stuff that’s not working. So that like so to me, once you’re having the discussion about is the data right? You’re losing if you’re using discrepancy. If it’s, you know what? Hey, can we reset? Can we really nail down the one or the two or the three biggest decisions you’re trying to make? Don’t worry about the data. Forget about the data. Yeah, it’s going to be involved at some point. I think a lot of times what happens, if you can really get them saying, 00:31:24.70 [Michael Helbling]: what I really want to know is, is meta delivering results? 00:31:29.62 [Tim Wilson]: And they’re like, can’t you keep crunching the data from meta? But I know there are gaps. Instead, you may say, really, let me talk to you about what a geo lift test is. So it kind of goes back to that order taker versus partner. And it’s it’s tough. They’re still working with you. So they have some level of trust. But I think we wind up fighting the fighting on the wrong ground. We were on the ground of like, no, but the data is good enough. Well, no, but this and go, I know it’s an asterisk and don’t use this word. It’s like, instead of like, we so quickly lose sight of it’s such a tangible thing to point to that the data has a problem. And we forget to say, what’s the what do we really want to know and try to elevate that conversation? I often I think it’s like, that’s actually not the right data set for it. Anyway, we keep chasing the wrong data set to most answer that question. 00:32:24.50 [Val Kroll]: I like that a lot. And the one point that you mentioned that I just expand upon is I think the really rooting yourself in like, what level of certainty is really required to answer this question? How much time do I have to turn this around? Is it something you need tomorrow? Do I have a couple of months before you’re going to make this call and really just trying to line up the various different methodologies that are at your disposal to bring that to bear to bring the right evidence to that question? You know, Paula’s presentation, like merging those different sources of evidence to really kind of paint that picture. I think can like shake people loose from like focusing on, you know, what percentage of, you know, people are opting out from whatever cookie banners and things like that, right? So I think like just not trying to play on that, like move the battle, I guess, not play on that turf and kind of say, hey, like let’s just focus on like Tim was saying, like those top those top questions that you’re really grappling with. And let’s think about how much business risk we really be introducing if it wouldn’t be perfect. So like, let’s think about the various ways we can kind of go about it. And sometimes just injecting a little creativity, if you will, into the methodologies, into that conversation can kind of get them excited about some different ways. Maybe it’s a user test. Maybe it’s, you know, it’s not going to be something we’re going to look for in a table as an example. 00:33:38.66 [Tim Wilson]: So be sure to record the meeting and send them the meeting summary. Oh, for fuck’s sake. 00:33:43.70 [Moe Kiss]: I actually stayed very quiet on that one because this is an area I think Tim generally is normally right in. Wait, why is everybody shaking their head now? But I do sometimes and I’m obviously in-house, so it’s quite different. I do find if I have a stakeholder like that, I will almost always have one-on-one time with them. And I’ll normally have some questions around like, what would have to be true for us to use this data source 00:34:10.90 [Michael Helbling]: or, you know, are there other data sources 00:34:14.34 [Moe Kiss]: that we could use to supplement the information so you’d be comfortable enough making a decision with what we have? Like kind of trying to tackle it almost one-on-one, because especially soon as you get in a meeting with a bunch of people and everyone’s like, oh, well, this data’s wrong. So, you know, we’re all stuck here and then everyone whinges about it for the rest of the meeting. It like it stops being productive. And so I would almost be trying to like really partner with that, like the biggest doubter of the group especially 00:34:35.54 [Michael Helbling]: and build up that relationship and really work with them on 00:34:39.54 [Moe Kiss]: some of the methods that Tim and Val are talking about here so that then they can also become your advocate, hopefully, over time. 00:34:46.78 [Michael Helbling]: Excellent. All right. Here’s another question we got. Everyone at my company is being tasked with showing specific efficiency improvements they’ve delivered using AI. I’m an analyst who supports marketing. What are some ideas you have that I could do for that? Don’t make me do the Hollywood Squares one again. 00:35:14.78 [Tim Wilson]: I just feel like this. I’m starting to feel like we’ve beaten this particular horse. 00:35:21.86 [Michael Helbling]: What do you mean, AI or efficiencies? 00:35:24.62 [Tim Wilson]: Well, well, yeah, I mean, the AI piece and the big, I guess the my my qualm with the efficiencies and I totally recognize the person asking the question. They don’t have control over that. That’s being pushed down and it’s an organizational challenge. But efficiency is like producing more with the same or producing more with the less or producing the same with the less whatever. And it’s like more what and and moving down the path of saying, well, we’re we’re producing more dashboards faster. We’re responding to requests faster and everybody feels resource constrained and like we can’t hire we can’t double our headcount. So AI is going to help us keep it fixed. And I think this is where I’m feeling like I’m beating a dead horse and I am the dead horse. I don’t know that that has this idea that if we its volume, volume is the issue that we just need to generate more. 00:36:24.74 [Michael Helbling]: But that volume of whatever we’re producing is going to someone. 00:36:30.38 [Tim Wilson]: Like there’s there’s value in the friction, which does not make me an AI skeptic. I just think the efficiency part is really, really tricky. You know, I think I’ve seen that in articles that that’s happening across a lot of companies are saying, we’re just trying to make this as a actually Jim talking on, I think day one was showing like this is just a chase for headcount reduction and something’s not right there. 00:36:57.26 [Michael Helbling]: So you you managed to answer that without giving this poor person any tips on how to do their job. 00:37:03.30 [Val Kroll]: They should have been a marketing analytics woman. Yeah, that’s right. 00:37:06.06 [Michael Helbling]: There’s a lot of tips there. 00:37:07.18 [Moe Kiss]: Can I jump in, though? 00:37:08.02 [Michael Helbling]: He helps. 00:37:10.26 [Moe Kiss]: I learned this recently and I’m still kind of reconciling it. So I’m going to obviously tell a bunch of people the exact advice I got. And we can all try it out, report back to me. Uh, I got some advice recently, essentially, like sometimes you just suck it up and you do it and a lot of conversations around AI at the moment are about productivity gains, not quality gains. And I find that just it gives me the ick. However, there comes a point where sometimes you’re in a position and you just suck it up and you do it and you go, oh, my God, my team saved five hours a week. Here’s all the dot points, send it up to leadership, move on with your life. And then you get your team together and say, OK, let’s have a conversation about how we improve the quality of our work. That’s what matters here. So sometimes the signal you send up doesn’t have to be the same as the signal you send down, but you better be smart about how you do it and don’t get caught. 00:37:59.30 [Tim Wilson]: But you’re setting yourself up to actually send a better signal up down the road, right? So I love that for doing both. 00:38:04.54 [Moe Kiss]: Yes, obviously, team that was the grand plan. 00:38:06.74 [Tim Wilson]: Check the box, but then separately say, but here’s the real value we got. And yeah, yeah. 00:38:11.74 [Michael Helbling]: And we’ve been battling a problem like this since forever. I mean, there used to be a time in our industry when we thought if we collected every single piece of data, we would somehow magically know more. And it sort of is a redux of a similar way of thinking. And so we have to kind of manage through it effectively. All right, we’ve got another question, and this one comes from also 00:38:34.90 [Michael Helbling]: someone in the audience, Sam Burge. 00:38:39.18 [Tim Wilson]: I think Michael surprised himself and didn’t realize he should have already been on the move. 00:38:42.34 [Michael Helbling]: So we’ll fix that in post. 00:38:48.00 [Sam Burge] How do you think analytics teams will look differently from today with AI? 00:38:54.98 [Tim Wilson]: In the future. I mean, I I hope that on the one hand, they are still spark, curious, business thinking, technical kind of have a broad set of skills. So I don’t think the teams will necessarily look a whole lot different. 00:39:16.54 [Michael Helbling]: How they work, they’re obviously going to get, I guess, efficiencies 00:39:21.90 [Tim Wilson]: and changing ways of working and become more prepared. 00:39:25.98 [Michael Helbling]: But that’s a tough one. 00:39:28.22 [Tim Wilson]: Why did I jump in and answer that? 00:39:29.54 [Michael Helbling]: I don’t think I had a good I was just buying time for one of you guys 00:39:32.38 [Tim Wilson]: to say something smart. 00:39:34.18 [Moe Kiss]: I don’t know about. OK, this is my guesstimate. 00:39:37.66 [Michael Helbling]: Yeah. And this is what I’m observing within my own team. 00:39:43.58 [Moe Kiss]: It seems like folks are kind of splitting a little bit. There are the folks that are going much more technical. I would almost, to some degree, even a little bit more specialized. And then there are the folks that it feels like the chasm between the technical and more the like business facing generalist is getting a little bit wider. I don’t necessarily see that as a bad thing. I think when folks have a particular strength in one direction, 00:40:06.02 [Michael Helbling]: like we should encourage that. 00:40:07.54 [Moe Kiss]: And that’s a great thing. I do think it makes it harder for like teams and how they work together and all of that sort of stuff. I had a massive rant a little earlier today because I am sick of reading very shitty, long documents which were based on someone’s shower thought that never should have left the shower, but now is in a four thousand word document and being flung around our organization. And then someone else comes in and writes 50 comments on it. And then someone’s like, over to you now, Moee. And I’m like, what do you want me to do with this? I am now doing all of the thinking work of having to read it. And it’s pretty like watery garbage, having to like respond, also trying to figure out what you want me to do with this because it was a shower thought bubble. And what my hope of where we get to is that it actually helps us think more critically, not less. I think we’re in the shitty stage right now. I’m optimistic. I’m going to bitch about the shitty stage we’re in right now because it is shit, reading all these documents. But I am optimistic that with time it will help us think better about what we do. Like, I don’t know, you’re creating a hypothesis, like instead of having to tap your co-worker, you can like sense check, have I got all the key components that I need to have a really strong hypothesis for this experiment? 00:41:27.82 [Tim Wilson]: No, I think even like knowledge management, the ability because AI is helping so much with unstructured data, the calling through what has happened in the past. But it feels like it is an elevated role for what a great analyst should be doing five years ago is still the same, which is being deeply embedded in the business context and the business needs. It’s been historically very hard to get the historical what have we done. So I think some of those like the preparation, the pulling this together, using the tools, but I don’t think it should change from what I think great analysts are doing, which is still having a deep connection to the business. It’s a shifting kind of tool set. 00:42:13.26 [Moe Kiss]: Can I just also add, Sam, one of the things I actually loved about your presentation was the idea also that we can change the format very quickly to suit different types of people. Like I am an audio person. I would absolutely listen to a podcast on business metrics. 00:42:28.58 [Tim Wilson]: I just remembered at the second point I was going to make back on that question. 00:42:31.22 [Michael Helbling]: All right, fine. 00:42:32.66 [Moe Kiss]: Was this like from 10 minutes ago or? It was, but it was on that. 00:42:36.02 [Tim Wilson]: It was asking, trying to figure out the best way that’s what somebody would, how to respond to them. So when you said the audio person, how often do people actually know themselves? So for all the listeners or anyone who wasn’t in Sam’s session, it was the, you know, there’s the marketing person who says, I just want to have my cup of coffee and listen to the podcast. And there’s a little trigger in me that thinks sometimes we seldom really know ourselves. So differentiating between somebody who thinks that’s what they would want 00:43:07.38 [Michael Helbling]: and someone who actually that would be effective. 00:43:13.02 [Tim Wilson]: And that had rang true from what we’ve dealt with for a hundred years. We’ve had people saying, I just need a dashboard that does X and we deliver them the exact dashboard. And they’re like, this isn’t helpful. Where’s this other thing? So there’s this other layer that I think analysts historically have needed to follow. And the same thing opening up all these different formats is great. But what somebody says would work. And I think you have to deliver it to them. But figuring out like, does that actually work? And giving them the opening, if they said, oh, I thought that would be really cool. And you tweaked and tuned the tone and the content and everything, but giving them the out to say, you know what that actually didn’t work. I don’t know that I wouldn’t have known it wasn’t gonna work until I actually tried it for a while, which we have not done in the industry very well forever. We get in sort of a whiny mode of saying, we’ve given them all these dashboards and they’re not using them. And it becomes this adversarial thing because, and then if we ask them, don’t you want these dashboards? Well, they ask for them. Of course they’re gonna say, yeah, yeah, yeah, this is really useful. We haven’t figured out how to say, no, that mechanism didn’t work and it’s okay. And we need to have the trust and we need to try something different. So we can go back and that was my other point. Maybe we should let, I’ll get a word in and twice. 00:44:35.90 [Val Kroll]: Back to how do we think teams will change? I think that there’s gonna be some new muscles that are built. I think one of the things that we had been talking about this conference is how this has given us a lot of energy and excitement. And I think that there’s been some creativity injected, which I think is just fun. Even if we’re just talking about little things that we do on the side that’s not ready for prod, but it’s just kind of like stretching us in some new ways, which I really appreciate. The other thing that we were also talking with you about, Sam or you and I were chatting about, 00:45:03.38 [Michael Helbling]: is the communication skills 00:45:06.74 [Val Kroll]: and how that’s gonna be improving. Because I think how often have you been in a conversation, even over the past couple of days, perhaps where people are just ragging on their stakeholders, like, oh, they’re so dumb, they just don’t get it, right? And it’s like, when you’re prompting and you’re talking about something you need the AI to do for you and it totally misses the boat, and you’re like, oh, geez, I’ve totally forgot to give you this piece of context. Of course you didn’t understand what I was trying to say. Think about your poor stakeholder that you were not giving that context for how many years. The video of, we were talking about this, the dad with his son and daughter about making the peanut butter and jelly sandwich, about like, take the bread out of the bag and he’s like trying to, and he like ends up like putting the knife through the whole loaf of bread, whatever, because he was just trying to follow the directions. But I think it can help us build a little empathy for our stakeholders, because like, they’re not wired the same way we are, they don’t have the same background that we do. And so I think it’s one of the byproducts that will help us become better communicators and hopefully have a little bit more empathy for people who don’t make all the immediate connections 00:46:06.82 [Michael Helbling]: that our brains do just because we’re nerds. 00:46:10.50 [Michael Helbling]: I think organizationally, I think we’ll see departments flatten out a little bit. Like over the last 15 years, we’ve become super specialized in a lot of different disciplines, because analytics is actually multidisciplinary. And I think AI will push that back together a little in a lot of organizations. And then those organizations over time will start to realize there’s still a need for some of that specialization on the fringes, and they’ll find ways to bring it back in. But I think we’ll all express experience, some compression where an analyst will go back to being able to code something and also write a data pipeline and also go access the data lake and also build a dashboard and a great visualization. And those were all things that like analytics people were attempting to do 15, 17 years ago. And then we realized we needed specialization. We needed a data engineer. We needed analytics engineer. We needed a data visualization expert. And I think we’ll begin to flatten those out with AI. 00:47:13.50 [Moe Kiss]: That doesn’t worry me a bit though. 00:47:15.54 [Michael Helbling]: I don’t say it’s good or bad. I just think that’s what will happen. 00:47:18.94 [Moe Kiss]: Like I feel like sometimes folks are over indexing on the generalization at the moment and thinking that, I don’t know, a product manager can do a data scientist job, and I mean, some product managers is doing engineering jobs. Also interesting choices. So I think we’re over indexing on the fact that folks can generalize. And I heard there’s a like- 00:47:39.06 [Michael Helbling]: Outcomes follow expertise, even with AI. Okay, we have time for one more question. And that’s our last question. And we have someone who is an audience member who’s going to ask it and it’s Jim Stern. 00:47:52.54 [Michael Helbling]: My question is, what are the first three skills 00:47:58.46 [Jim Sterne]: that analysts have mastered that are going to be successfully taken over by artificial intelligence? 00:48:08.18 [Val Kroll]: I wish we could get a question about AI. No shade, Jim. Just looking at Michael. First three skills, say it again. First three skills. Sorry, I was being an asshole. First three skills. 00:48:24.66 [Michael Helbling]: What are Tim’s favorite skills? So, probably like the, what is it? The ink to data ratio. So that probably going to be the first thing AI takes over. 00:48:37.94 [Val Kroll]: Data pixel ratio. Data pixel ratio. 00:48:39.98 [Michael Helbling]: I was paying attention, Tim. SQL, I don’t write SQL anymore. 00:48:46.82 [Tim Wilson]: I am so much more on the debugging SQL debugging R. I think debugging coming over really, really quickly. I think a second one would be QA’ing or validating or vetting the results of an analysis, having that the thing that you’re supposed to go to another analyst or try to come at it a separate way. I’m not sure what the third one is. I think it’s a lot of things that are going to be supplemental that we should be doing. Like not doing the QA, but giving me the list of, check my logic. Check the things that I, so maybe it’s not what the junior analyst is doing. I think it’s what the junior analyst ideally is working with another junior analyst or senior analyst to look over and review. There’s not a whole lot that I see a whole hard, whole hog hand in the keys over on. 00:49:42.94 [Michael Helbling]: I came up with three, but I don’t know if they fit the criteria. We’ll have to go from there, but that’s going to be where we have to wrap up. And I want to say first, a huge thank you. To Jim Stern for organizing the marketing analytics summit. 45 years. And save your applause because also to all of you for being here and bringing your energy and your questions, your insights and experiences in a world changing daily with AI, it’s the people, the community and human connection in our industry. It feels all that much more special and crucial in these changing times. And obviously there’s a huge audience also listening and we’d love to hear from you too. And you can reach us at our LinkedIn page or the measure slack chat group or by email at contact at analyticshour.io. Please leave comments, ratings and reviews on whatever platform you use to listen. We do read all of them. And Tim brings them up in meetings. 00:50:48.46 [Michael Helbling]: And I think- 00:50:50.70 [Moe Kiss]: It makes us set KPIs. 00:50:52.62 [Michael Helbling]: Yeah, it’s terrible. 00:50:55.18 [Michael Helbling]: I can’t wait till AI replaces that. All right, and I know that I speak for all of my co-hosts, Val, Tim, Moe. When I say, no matter the question or challenge 00:51:06.06 [Michael Helbling]: you’re currently solving, keep analyzing. 00:51:09.30 [Announcer]: Thanks for listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at analyticshour, on the web at analyticshour.io, our LinkedIn group and the measure chat slack group. Music for the podcast by Josh Crowhurst. 00:51:26.98 [Charles Barkley]: Those smart guys want to fit in. So they made up a term called analytics. Analytics don’t work. Do the analytics say go for it no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 00:51:46.18 [Jim Sterne]: Ladies and gentlemen, that is so much fun. Thank you so much for be adding the spark to the end of the marketing analytics summit. You guys are awesome. 00:52:02.62 [Tim Wilson]: Rock flag and AI gives me the ick. 00:52:06.42 [Jenn Kunz]: Yeah. 00:52:07.26 [Michael Helbling]: Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. The post #298: Listener Questions Answered Live from Marketing Analytics Summit! appeared first on The Analytics Power Hour: Data and Analytics Podcast.

May 12, 20261 hr 6 min

#297: Durable Wisdom in an Age of AI Slop

What do colors, soup kitchens, and mountain climbing have in common? They’re all part of the mental models that have shaped how we think about analytics, and they’re exactly the kind of durable wisdom that matters more than ever in an age of AI slop. This campfire-style conversation among the co-hosts reveals the concepts, books, and aha moments that have stuck with us across decades of analytics work. From the magic of randomization to the critical distinction between outputs and outcomes, we share the frameworks that guide our thinking whether we’re writing SQL by hand or asking Claude to do it for us. It turns out the most valuable analytics wisdom isn’t about tools or techniques—it’s about understanding how humans actually make decisions, build trust, and collaborate effectively. Some things never go out of style. This episode is brought to you by Prism from Ask-Y—your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show (Shiny App) The Magic of Randomization Illustrated with Color Outputs vs. Outcomes: this is a good resource/explanation of the core idea, but Val also addressed the concept in this Medium post, and Tim digs into it in Chapter 5 of his book (available in print, ebook, and audiobook formats!) (Book) A/B Testing: The Most Powerful Way to Turn Clicks Into Customers by Dan Siroker and Pete Koomen. And the diagram from that book that hit on local vs. global maxima through the lens of “Refinement” vs. “Exploration” (Article) Addition bias (Ingvar Kamprad / IKEA) (Book) Information Dashboard Design: Displaying Data for At-a-Glance Monitoring by Stephen Few (Book) Storytelling with Data: A Data Visualization Guide for Business Professionals by Cole Nussbaumer Knaflic (Article) Chasing Statistical Ghosts in Experimentation (Book) Field Experiments: Design, Analysis, and Interpretation by Alan S. Gerber and Donald P. Green (Book) First, Break All the Rules: What the World’s Greatest Managers Do Differently by Marcus Buckingham and the Gallup Organization (Book) Switch: How to Change Things When Change Is Hard by Chip Heath and Dan Heath (Podcast) Choiceology with Katy Milkman (Book) The Science of Storytelling: Why Stories Make Us Human and How to Tell Them Better by Will Storr (Podcast Episode) #240: Asking Better Questions with Taylor Buonocore Guthrie Photo by Joris Voeten on Unsplash Episode Transcript00:00:00.00 [Announcer]: Welcome to the Analytics Power Hour. 00:00:08.92 [Announcer]: Analytics topics covered conversationally and sometimes with explicit language. 00:00:15.80 [Michael Helbling]: Hi everyone, welcome. It’s the Analytics Power Hour, and this is episode 297. You know, at this point, I think we’ve all been hit at some kind of output that is obviously AI generated. I mean, even it has its own term, AI slop. Oddly enough, most people don’t appreciate being hit at AI output without it being thought through by a real person, and it got us thinking. It’s not that what AI produces is always bad, but there is starting to be different categories of content based on whether AI produced it or not. And maybe there is something to be said for analytics wisdom that existed way before AI, and that will keep being true no matter how AI is involved out into the future. So we reach back to the formative moments of our own careers to share some of the hard one wisdom from our careers in analytics. Maybe this episode will lack a little AI, but I think it’ll still be worthwhile. So let’s introduce the people who make up the show. Hey, Val Crowell. 00:01:16.92 [Julie Hoyer]: Hey, Michael. 00:01:18.12 [Michael Helbling]: I’m so glad you’re here. And we’ve got Tim Wilson. Howdy, howdy. And I still put up with you somehow, so that’s good. 00:01:28.52 [Julie Hoyer]: Okay. 00:01:29.52 [Michael Helbling]: I had Moee kiss. How you going? 00:01:33.04 [Moe Kiss]: How you going? Oh, I love it. 00:01:35.04 [Michael Helbling]: Thanks. Yep. And Julie Hoyer. 00:01:38.04 [Julie Hoyer]: Welcome. 00:01:39.04 [Michael Helbling]: Hey there. And I’m Michael Helbling. So, yeah, we’ve got the whole team together. We’ve all seen so much in our careers pre-AI. And so what’s some wisdom that exists that’s going to be useful, whether AI is involved or not? Who wants to kick us off with something they’ve learned over their career? 00:01:55.12 [Julie Hoyer]: Oh, I will. 00:01:56.64 [Julie Hoyer]: Because this one, I feel like sticks with me still to this day. 00:02:01.24 [Julie Hoyer]: And this was more than over a year ago. This was three, four years ago. But it was actually back when me and Tim were working together. And I’ll never forget, we were talking about randomization and trying to figure out the best way to represent it, different ways to think about it. I think this was right around Tim when we were trying to do a talk around blocking, randomization with blocking and things. 00:02:29.84 [Julie Hoyer]: And so we could not figure out how to make it friendly to people, people who had maybe 00:02:35.88 [Julie Hoyer]: not thought about it as in-depth as we were at that moment. We were way too in the weeds. And I remember slacking Tim like the next morning, like right at the beginning of work and saying… So I had a thought. 00:02:49.68 [Tim Wilson]: Wait, it was the next morning because it was a glass of wine sitting on your couch, thought, I believe. 00:02:55.24 [Julie Hoyer]: Yes. Yeah, I was like, maybe I even slacked you that night and was like, look at this in the morning. 00:02:59.40 [Julie Hoyer]: But I was like, I just was having a glass of wine and it came to me. 00:03:02.76 [Julie Hoyer]: I was like, I feel like a good way to explain randomization would be through colors. And so I tell them how, what if each color represented a characteristic and when you randomize the population, you could see how the colors were split between test and variation and what if those colors were blended to show a group color. And then you could see that the color comes out really close because it’s randomized across the characterization. And so Tim, I nearly couldn’t resist and that’s why I told him. So I was like, that’s about as far as I can take it. You made a whole shiny app to let you choose the size of your sample, to choose how many characteristics, to choose the predictiveness of those characteristics on your outcome, the mean, all these things. And then you could actually run a simulation and see the colors blended. And then you could flip on and off blocking. And there have been multiple times, even earlier this year, when I’m thinking about certain things with randomization, I’ll still go back to that shiny app and play with it. And it helps me a lot with talk tracks or simplifying it or reminding myself of different characteristics of it. So that’s one of my favorites. 00:04:18.32 [Tim Wilson]: I like that one too because we got a little carried away because it was like, well, this work, I think it was built very quickly. And then Julie was like, what about if we, you think we could do this? What if we did a, and I was like, what if we did, and so it got a little involved. 00:04:33.96 [Julie Hoyer]: My hands never touched the keyboard. That was the best part. I got to the whole thing. 00:04:39.96 [Moe Kiss]: It was like voice control. Why do you think it’s stuck with you so much as like the antithesis of like the AI flop world? 00:04:49.56 [Julie Hoyer]: Like what was the, I don’t know, the bit that just made it really resonate or it keeps you 00:04:55.92 [Moe Kiss]: coming back. 00:04:56.92 [Julie Hoyer]: I think it’s the visualization of it. I don’t know why the, the color is what stuck with me. And sometimes like a good visual is so much easier for me to go back to. And I think the AI part, this is, I swear I’m going to loop back to your answer. But I even remember back in college starting in engineering, they just wanted to give you like an output to use. They’d say, just use this output, use this output, like a formula. And I, and I’m, no, I’m not alone this way. I always needed to understand why. So that if I forgot the perfect formula, I could like reason my way back to my understanding. And there’s something about that shiny app and the visualization of a color that just like, dang, it like hit something in my brain that when things get fuzzy and I haven’t talked about those things in a while, I haven’t thought about those things in depth. Like it kind of brings back some of that understanding. Like I can have a good starting point and like think through it again. And the AI part is like, again, they just give you output and that output is actually like the variation of an AI, you know, agent or whatever, like giving you something. It’s never exactly the same. So I like that it is steady and a starting point for me. 00:06:10.24 [Tim Wilson]: I love that. The reason I got so excited, I think it is this color piece because I will chime in that when we talk about random assignment and kind of the power of random assignment and we’d say, oh, so if you have, you know, a thousand people and you randomly split them, you’re going to have a roughly the same number of men and women in each group and roughly the same number of household income and like that idea that you’re making two groups that are effectively the same, like it’s just, it’s abstracted when you talk about all the characteristics of what’s in it. And I think when thought about it, like, I mean, it literally is kind of a color mixer that it just sort of generates a palette and then you see the average color and it may be like this lime green and it’s slightly different shades, but it’s like a simple math thing. So to me, like it just goes into my head of saying, this is what’s happening while we’re making two groups that are pretty damn close to the same. But the fact that, I mean, that was to me kind of the really useful part of that is I was trying to grapple with how to actually internalize this. And then it was very easy to say, okay, now that extrapolates to these other more nebulous characteristics of psychographic details and demographic details. 00:07:27.20 [Moe Kiss]: But it sounds like it’s solving the problem of, to quote a Canva value, making complex things simple, right? Like it’s about understandability. It’s about a way to see something so that it like clicks in someone’s brain. And it’s funny, I had someone in my team that was doing some work the other day and I went through it afterwards. So like we had a couple of different strategies and I was kind of like, we need to go through these strategies. We really need to make sure that they’re being based on the data points that we have available. And like that easily is a task, for example, that probably my instinct would have been, 00:08:04.36 [Julie Hoyer]: I’ll put it in AI and see what’s like, here are the data points, like what’s missing. 00:08:09.24 [Moe Kiss]: And this person went through it so rigorously. And the reason that the work like came back and I was like, oh, I get this, this is high quality work. And like I can really understand it is because it’s also that falling, it made it click in my brain of just like, not here are the differences, but here’s the assumption I made about why I think this is different. And here is the like, the leap that’s been made here. And I think it’s probably because of this. And I think it’s, I don’t know, when I just keep coming back to this point about like quality, it’s like when you can see the quality and it, someone finds a way to put it into a narrative that then clicks in your brain. Like that’s where I feel like the gold is right now. And it doesn’t feel like there’s a lot of it. 00:08:55.32 [Tim Wilson]: That does kind of make me want to go with one of mine because I’m seeing a direct link from that, which is also kind of just a concept or something that I find myself thinking about and talking about with clients and business partners a lot. And even with analysts is the distinction between outcomes and outputs and how we, as a, we tend to pick metrics that are outputs when we really care about our outcomes. And I traced this back 20 years at this point when I was on a United Way committee in Austin. And there was this retired social worker who we were, we were reviewing programs for funding. And so we had a lot of proposals coming in and we were in these series of committee meetings and we’d all have to like read, I don’t know, 10 proposals and then we’d meet about them. And he kept having this kind of consistent bit of feedback on multiple programs where he would say, well, these are like, you always have to say how you’re going to measure the program. And he would say, oh, well, these are, these are like output metrics and we really want outcomes. And I was 10 years into my analytics career at that point. And I was like, what are you talking about? And it slowly, he started to explain and the example, he had various ones because he’d point to them in specific programs. So the one that I come back to was talking about like food pantries and how they like, they would count like the number of meals served or like a soup kitchen number of meals served. And he was like, yeah, they can just like count the number of trays that go through the line. And that’s the number of meals served. And we know that’s good. You’re serving meals. He’s like, but really what we’re trying to do is reduce food insecurity. We’re trying to keep people from going hungry, which is related. You want your outputs to lead to an outcome. But he was like, we really want to push them to say, can they get more to an outcome oriented measure? And I’ve taken that over the course of that work, I was like, this is like profound for what I’m doing in my day to day with my colleagues at work and trying to get them to think about, this is why a click doesn’t matter. This is why the click through rate, I mean, doesn’t not matter. It’s just those are outputs and trying to guide discussions early on into outcome oriented metrics and business outcomes and then going from there and say, how close can we get to measuring that? And unfortunately, I think the guy’s name was, I am 99% sure, unless I’ve been fooling myself for years. His name was Pat Craig. He was a retired social worker. I have every four or five years, I go try to find him because I come back to that again and again and again. But it was another one that made it very tangible because it was really in the real world. People who are in need talking about outputs versus outcomes really solidified it, made it very tangible, but it then applies in the more, really, we’re not curing cancer here, we’re talking about marketing stuff, but the concept still applies. 00:12:00.60 [Julie Hoyer]: That’s a good one. I love that one. I use it all the time. Do you know how many people I’ve said that thinking, oh, they’ll know about this? And they write it down, they’re like, oh, that’s good. So I’m like, oh, thanks, Tim. 00:12:11.16 [Tim Wilson]: Well, I mean, there are times where I feel silly asking, I’m like, do we, are we, I’m sure some of you are familiar with this because I’ve thought about it for so long. To me, it’s one of those like when the light bulb goes on, you can’t stop seeing it. 00:12:29.08 [Moe Kiss]: Tim, the light bulb has just gone on for me. Just to be clear, I messaged you. It’s like a message because I was like, maybe I shouldn’t derail the whole show. But I think I’m thinking about it as like input and 00:12:43.16 [Julie Hoyer]: output. Yeah, but here we go. But anyways, or we could. No, but I’ve been thinking about it as like 00:12:50.76 [Moe Kiss]: input and output metrics. And I say, like in my mind, an output metric was an outcome. But like, I just feel like that framing is so much better. And it truly is clicking in my brain for the first time. And I’m, I’m sure I’ve heard you say this before, but sometimes like someone just said something a slightly different way, or your mind is at the right time to absorb it. Anyway, thank you. This is going to be very helpful. You’re welcome. 00:13:15.80 [Tim Wilson]: And I’ll also say this makes it sound like it’s binary. The older I get, the more I’m like, it’s not, I say like, you want to be skewed towards outcomes. You can have debates about, you know, is a monthly active user, is that an output or an outcome? And it’s kind of, it’s context dependent. And it’s not, but if you can ground it in that overall, overall pure version, I found it very, 00:13:38.60 [Val Kroll]: very good. You’re just getting soft, Tim. It’s binary. There’s no gray. 00:13:43.16 [Tim Wilson]: That’s right. You said the rules have reversed. I never ever seriously about, wow, Tim, you’re just mellowing out waiting for that to happen. 00:13:54.60 [Julie Hoyer]: So chill. Hey, Tim, have you tried out vibe analyzing yet? 00:14:03.96 [Tim Wilson]: Oh, that phrase hurts me deeply. But I did download my Strava archive to just try to like analyze my workouts. And it turned out to be like 28 different CSVs with everywhere from like two to 103 columns each. Oh, sounds messy. What’d you do? Well, I mean, for giggles, I tried Prism by Ask Why. That’s ask-the-letter why. I uploaded like all 28 files and just started asking questions through their chat interface in plain English. Started off by asking like, how many miles do I run each month? And it worked? I mean, it did. It initially it gave me results pretty quickly. It was, it was like really fast. It actually wasn’t, wasn’t perfect, but that really wasn’t a prism issue. It turned out that Strava data just refers to distance. Like that’s what the column is labeled and that distance is in kilometers. So it took a few iterations, still a little bit of the human analyst to say, wait a minute, those, I wish I was running that far, took a few iterations with the platform to get that figured out. But once we did it, it handled that conversion, not only on that query, but like automatically going forward with future queries. Oh, that’s actually pretty cool. 00:15:21.56 [Michael Helbling]: It’s just like how we want to handle mishmashes of different sources and medium names in a consistent way whenever we’re looking at working with like digital data. Exactly. And I also got to 00:15:32.52 [Tim Wilson]: kind of check out their Mingus query language, which it’s like a more readable form of SQL with the actual SQL just like one click away. And I even like built some quick visuals and some quick reports. It’s also got like a local mode for keeping all the data on your machine. I actually haven’t tried that out yet. I just did the cloud version, but it’s pretty nice feature. 00:15:54.20 [Michael Helbling]: Nice. So yeah, it sounds like something worth checking out. You can head over to ask-y.ai, join the prison beta waitlist and use the promo code APH when you sign up, and that’ll move you up to the top of the list. We can guarantee that you’ll get access faster than Tim finishes his next 10k probably. All right, back to the show. Okay, so I want to share one of mine. 00:16:23.56 [Val Kroll]: And I’ve worked with most of you and if you’re listening, I’ve worked with you as a co-worker, as my client, you will know this one of mine. In the experimentation realm and world, one of the concepts that lots of programs like to think about or keep in the back of their mind is the difference between a local and global maxima. And like the tried and true, you know, the analogy or the visual is like you climb to the top of the mountain. And now that you’re through those clouds, you see that there’s actually a secondary peak to climb. And I get that, that works. But there’s this visual that actually came from the book A.B. Testing. It was the optimizely book at the time, Dan Saroker, the CEO and Pete Kuhlman, who led their statistical arm and function. In there, there’s this visual that has, and I’ll try my best to describe it. There’s two sides 00:17:20.44 [Tim Wilson]: that are being compared. Hold it up longer. That could wind up as a YouTube short, you know, so this will drive people to the YouTube channel. Don’t you want to click through the full episode 00:17:30.68 [Val Kroll]: now? So on the left hand side, they’re describing refinement and that there’s kind of like this cone shape where like the squiggly is getting closer and closer to the star. But there’s actually a star to the side of it. And it says that that was the best solution and it was missed because they were kind of refining to this point. Whereas exploration is kind of this point that branches out and has like lots of arms to it. And so they did find the optimal solution. And the arrow is saying like, here’s where you refine from. And what I like about this image so much more than the local versus global maxima is because it gives you the visual of the consequence of not thinking big first or to not think about exploration or innovative type of thinking or testing first and going straight into like, how do we refine the micro copy on this page? And it’s like, well, was that the right page to send someone to in the first place? And so it’s about like staying curious at like a higher level because it’s not just, you know, like we’ll find these like little wins as we go, which is great, but they’re empty calories if you’re kind of missing the optimal solution. And so that visual, I bet you could find it, I have pasted that in no fewer than 50 presentations in my life. I’m quite confident because I think it’s just a really nice way to like cement the point of that concept home. 00:18:55.00 [Moe Kiss]: Well, it’s going in one of my presentations. It’s amazing. 00:19:00.28 [Val Kroll]: I like it. It’s good. 00:19:02.12 [Tim Wilson]: But is this is part of this that like that just the realities of corporate life is that it is much easier to get into is to be in the lane that you’re in and kind of refine like, oh, because to do the exploration feels riskier and it often means you’re kind of reaching more broadly with ideas. So like just like business culture drives us to say, let’s do little tweaks and refinement and and it like organize like how do organizations do organizations see this and say, you’re right, we need to push ourselves to think more broadly, take bigger or broader or more exploratory swings. 00:19:54.04 [Val Kroll]: Yeah, I totally, I mean, how many times have you been like, we’ll give a marketing context like, well, when we ran this campaign last year, we only had two versions this creative. So this year we’re going to have three. And so it’s like the smaller like, you know, I’m obviously being reductive in that example, but it’s like, based on what we did last year, we all remember that here’s one new thing we’re going to do different. And that’s like the optimization or like the refinement versus like, instead of just going direct to patients, what if we had a strategy for healthcare providers? And so like that would be the bigger swing. But to your point, Tim, I couldn’t agree more because it’s like, there’s no incentive for that because that’s more work, you know, more approvals, perhaps, you know, more budget overhead, things like that. And so I think people who are really excited about the outcomes of what that’s trying to do, and go back to yours, that those are the folks who really kind of like thrive in finding those. And you’ll notice that those are the people that lots of other people really like to work with inside of organizations, I will say, because they’re doing more exciting things in service of, you know, shared goals for the organization. But yeah, I think it’s not, it’s not the natural path, I agree. No, people want what they can control. Like in their in their lane, like you 00:21:11.64 [Julie Hoyer]: were saying, it’s hard to like look up and do that broader view, and then have to collaborate. But I think then what you said, Val, tying it back to outcomes, like, if people realize the shared outcomes, they were more focused on driving instead of their individual lane outputs, like maybe people would be more open to doing that instead of just the refinement. 00:21:30.68 [Moe Kiss]: Yeah, I was just gonna say, I don’t think the shared outcomes are always incentivized, like sometimes they are and sometimes they aren’t. But I think the one kind of the one push I would have on this framework as like, and I am a huge fan, I’m definitely definitely going to be borrowing this. I think there’s an assumption that you’ll always get to the star, like the point of refinement through exploration, and sometimes you don’t. Sometimes you explore, you do all the extra work, and it doesn’t add a significant amount of value. And I think, generally, those cases are pretty rare. But I think it does happen sometimes. They’re like, again, not binary. 00:22:10.44 [Tim Wilson]: Without any specifics, obviously. But I think of Canva as a company, like on a product level, very exploratory, like every time you turn around, they’re like, oh, yeah. And now, you know, it can make your hopes for you in the morning. So it seems like, I mean, like literally, I mean, it feels like, I sort of get updates through conversations we’re having. You’re like, well, yeah, can we get to them? Like, okay, there’s three other products that, what the hell? 00:22:40.92 [Julie Hoyer]: So it feels like that. Have there been, has Canva had like, pursuit of like specific, 00:22:48.20 [Tim Wilson]: like this is a whole new area that has gone nowhere and been shut off? Again, not asking for 00:22:55.40 [Moe Kiss]: any specifics? Yeah, I think so. I think the tension right now, though, is that we all need to lean more towards that exploration piece because of just the pace of AI products and features and how they’re shipping. Like, I think what particularly is a trap right now is if you’re in that retirement, we want to go towards this goal, we want to build this thing. Like, in today’s climate, that’s more dangerous than ever, I would say, because just the way things are changing 00:23:26.28 [Julie Hoyer]: so quickly. So I would say not necessarily like things getting completely like abandoned, 00:23:33.72 [Moe Kiss]: but more things getting refined and changed along the way. 00:23:37.96 [Tim Wilson]: I like that with AI. If you’re, if you do the kind of the MVP in multiple directions in a way that you can say, I’d rather try five wildly different things with AI in a minimal way with 00:23:53.08 [Julie Hoyer]: clarity on how I’m going to determine whether this is the best bet or not, than determine 00:24:00.52 [Tim Wilson]: we’re going to make the best chat experience using the latest LLMs ever and just like pursue that 00:24:08.28 [Michael Helbling]: and miss it. Well, I’ll share one that’s important to me. I don’t remember who first told me this, but it was about five years into my analytics career and somebody said to me, Michael, trust is hard to build and easy to break. And I think that’s more of a general statement, but applied to the world of analytics, I watched in my own career sort of people who believed when I presented an analysis and people who didn’t and losing trust with stakeholders was something I definitely experienced in the early years of my career and how much that put me in a position where I could no longer influence the business or business outcomes in certain areas. And so it really kind of hit home and I really kind of held onto that for, for the rest of my career was just sort of thinking about how do I continue to build trust when I’m working with business stakeholders when I’m talking about things. And I like, because it’s me, I don’t have any kind of format, formal like structure to that, but there are little signifiers I look for around like how do I know that trust is still there. And I use that to kind of guide how I, how I act around, you know, today my clients or stakeholders I’m working with of sort of like how is that sort of relationship which tells me then the influence I have as an analyst for that particular situation. So that one is one that has kind of always stuck with me because I love being influential. And for early in my career, I just figured if I showed you the data, it doesn’t matter who the messenger was, you would just say, okay, yeah, that’s the data and you would accept it. But the reality is, is the messenger matters quite a bit. And since I’m not Tim Wilson, you know, I had to 00:26:03.16 [Julie Hoyer]: like, you know, ramp up my skills. But this, this is like, I mean, honestly, like, 00:26:10.20 [Tim Wilson]: dear listener, if you’ll like that one, go back and listen to our last episode with Eric Friedman, because a lot of what comes up with that is that I think we think it’s, that means the data has to be perfect and the analysis has to be perfect. And I feel like what you model and what we talked about with him, a lot of it is like actually showing that you understand the environment they’re working, like you understand building trust has a lot more of the soft skill than my data is always perfect. Yeah, there’s that part of it, understanding 00:26:44.68 [Michael Helbling]: the context. And I think also not to use a bad word to you, Tim, but empathy has a lot to do with it 00:26:51.64 [Val Kroll]: as well. Don’t know what that does not compute. Yeah. No, because it’s like one of the signals 00:27:04.44 [Michael Helbling]: in like, if I’m working with other clients is if they come to me with a separate problem, that tells me I’m building trust because they are like, okay, yeah, you’re doing the project or whatever project. But if they come and say, Hey, here’s another thing that’s going on, you have any insider thoughts into this? That’s a great example to me of like, okay, we’re on a trust path together now. So that’s awesome. Let’s keep building that. So like, that’s, but yeah, you’re right. You’re absolutely right. It’s not just the data or the analysis. It’s also the context to show you understand show that you care about what they care about. And it’s challenging because as analysts, I feel like sometimes we want to not necessarily be front and center. And the reality is to influence decision making, you’ve got to be, you’ve got to be willing to kind of plant your foot and sort of be the face of the 00:27:54.04 [Moe Kiss]: data in a way. So the funny thing is, Michael, like you, you’ve been chatting about trust, and it actually makes me think, I don’t know, I know I’m obsessed with her big fan girl, but she talks about this so much. And like, she calls it work slow, which is enabled through AI. And I think like the way she articulates it is just so brilliant around like, and I feel like I’m seeing so much of this where like AI removes friction for shitty ideas, right? And so everyone just like, and I get it, I get it because I’m doing it too. Like I do, you can move faster, but it’s making us look like we’re productive, but actually we’re just producing more shitty ideas, right? And I think the bit that’s really challenging then is like being able to differentiate between the shitty ideas and the not shitty ideas, right? And so I think the thing that she really is like honing it in on, which has just been like flying through my mind. And it’s the same as the trust of the stakeholders, right? Is like AI can be this incredible tool to unlock a lot. But like, how do we really use it? How do we incentivize the quality over the velocity? Because at the moment, we’re really honing in on velocity, which is breaking that trust so deeply. And I think about it so much like, as a manager, every time you get a piece of work, as a person who’s producing work that is lower quality, like, I personally feel like you’re 00:29:22.28 [Julie Hoyer]: fragmenting trust with those around you, right? Well, but it’s, it’s, it’s got dual pressures, 00:29:27.24 [Tim Wilson]: you’ve got the pressures to use AI, that’s coming down from on high, use it, use it, use it, be efficient. When you’re delivering stuff and it’s polished and longer and grammatically correct and no typos and coherent and organized thoughts, but the person who’s getting it knows and then you’re under the gun to say, I can’t spend as much time whittling it down. Like they are competing pressures. The person who’s receiving it, I think you’re dead on, like it does, like you’re sending me like total AI slap, stupid sales pitches, but there was no trust there in the first place. It does feel like he gets sneakier when it’s with a coworker saying, oh, I, you know, here, here are the notes from the meeting. And you read through it and you’re like, this isn’t your voice and it’s a little off, but I can’t really criticize you because you did it quickly. But I also don’t feel like there’s a depth of thought. Like that’s a, but that’s the thing, right? Like I almost wish 00:30:25.32 [Moe Kiss]: there was a way to know if I have 10 docs in my pile from my reading list, which one wasn’t written by AI, because that’s the one I’m going to go and read, but there’s no way to know that, right? So then what ends up happening? Like I feel like we need to create a system that incentivizes the quality of thought and the depth and the fact that if you want something to be shorter or tighter, like you actually need, like you can’t just give it to AI. And I just, I feel like we’re 00:30:50.44 [Tim Wilson]: in this real conundrum. That can be a great idea. I’m going to, I’m going to vibe code an app to do that this weekend. That has collars. Yeah. Yeah. 00:31:01.08 [Michael Helbling]: Part of this, the thing about maintaining trust in, in this context, I think is about transparency as well. So like if you use AI, we’ll just lead with, Hey, this is AI generated. So just F, you know, so you know, or you say most of this is AI generated, but here’s my synthesis up top. So that way you can let people know the distinction. Like you don’t have to go read all this. You can, you know, it’s the same thing when you’re preparing an analysis and like you want to show all the cool things you did to the data, but you put it in the appendix because your stakeholders don’t care. They just want to know what the McKinsey title is and the, and the, the big insight. And if they trust you, they’re probably don’t need to dig much further. If they want to learn more or have more deeper interest, there’s sourcing material behind it. So a lot of times AI for me feels, feels like that where it’s like, okay, AI can pump out tons of content and, and honestly, beautiful content too. Like it’ll make a better slide than I make on average. A lot of times, you know, if I just sort of take content and plug it in, I don’t make the best slides. Okay. 00:32:07.16 [Moe Kiss]: Like I’m just being honest. I had AI make a slide deck for me the other day and it’s still the 00:32:13.72 [Michael Helbling]: work to do on the design side, I would say. No, no, no, I, I mostly don’t let it because even though it can make a pretty slide, it’s not, it doesn’t fit what I’m trying to do. So like, no, I haven’t yet been able to like really position a whole AI slide deck yet, but it’s 00:32:30.44 [Tim Wilson]: getting closer. Imagine, imagine if you just could just go on a walk with the dog and just talk, talk to the AI and then it’ll generate a deck for you. And it’s like, no, there’s, there’s value in the friction of me needing to go on the walk with the dog, think about it, stew over it, and then come down with what are the three things that I want to say. 00:32:50.44 [Moe Kiss]: Okay. I promise after this, I will get off my soapbox. I promise. But there’s an incredible, I mean, we all know I love the acquired podcast, but there’s one on IKEA, which is amazing. And Ingvar, I’m proud, I’m going to fuck up his name, I always do, but Ingvar is the guy who started IKEA, right? And he has, he had this principle, which I’ve just been thinking about, like, how do you implement this and how do you scale it? Basically, particularly as like people manager, right? Like we have addition bias. So like any time thing seem hard or tricky, we try and add to it. We try and put more on a more process, more structure, more things. And his kind of like management rule was always simplified. So if we have a problem, what’s one thing we can take away, what’s one thing we can remove. And I’m, I’m trying to like really think about that with the team, like, how do we take stuff away instead of adding? And so is it about like, and again, like my, my bias is the addition bias, I’m straight away like, okay, let’s have an experiment template, like, let’s have measurement like standards, and I’m like, I’m adding, how do I take away so that we simplify? Because especially with AI, there is a lot of things where we’re adding, we’re constantly adding, we’re adding metrics, we’re adding, like extra reports, extra things, and it’s adding to the complexity, which is not the intent that we think we’re going to have. All right. Okay. I’m 00:34:12.92 [Michael Helbling]: off the third box. I’m done. No, I like that one. It goes back to what you said before, Moe, which was sort of like, execution is going to zero. So the quality of the idea or the quality of thinking now matters more than ever, because you can go execute on a poor idea so fast, but waste everybody’s time in the process. And so like, having some thoughtfulness ahead of getting everybody rolling 00:34:40.36 [Tim Wilson]: now is sort of like even more critical. I love the language of addition bias, because I think that it is so broadly applicable. And I’m going to like, throw a quick one in, and then we’ll go back to maybe more broadly, but, and it’s a twofer, but because to me, maximizing the data pixel ratio from a data visualization and a clarity of communication, which I’ve been shouting from the rooftops for years. So information dashboard design by Steven Few, I read that like in 2006, Cole Naflik, it’s chapter three of her book is like, basically declutter the storytelling with data data visualization guide for business professionals. But that that’s in a narrow, that’s the addition bias of how do I provide, I’m going to deliver this to a stakeholder, my instinct to build more trust is to put more stuff in it. And what they really want is to remove stuff. And with AI, like with AI, when you ask it, summarize this, if you ask it to give you a two minute script, it will give you a four minute script. If you ask it, like you have to constantly tell it to do less. So it goes for processes, data visualizations, the analysis you do, let’s keep digging deeper, deeper, deeper, deeper. And it’s like, or can we stop 00:36:01.16 [Julie Hoyer]: and make a decision to move on? But it makes me think about, I was thinking about this initially, and now you saying that it makes me really want to try it. And I feel like I’ve done a little bit in passing, but when AI gives you the four minute thing, the huge long summary, I feel like a really good check on the AI is to actually ask it to give you like a four sentence summary. Because I feel like that’s where you can sniff out the BS faster. Like when I’ve asked it for a short summary, I know it’s totally missed the plot than like what I would quickly give as a four, some like four point summary or four sentence summary. Because sometimes it is like you start reading, you’re like, I guess it sounds good. Yeah, kind of. And then I think you’re more apt to just like trust it and maybe use that long format. But it’s like the old adage, sorry, it took me so long to like write you a short letter or something. I’m quoting it a little off, but it feels like that. So I do wonder, could you like stress test the AI output sometimes by asking it for the short thing and be like, ooh, really quickly, good or bad? 00:37:02.44 [Moe Kiss]: I do, I definitely do. But then I end up editing it. And I’m like, I should have just written 00:37:07.08 [Julie Hoyer]: it myself, it would have been faster. Always, always. And we thought we were going to talk about AI 00:37:12.20 [Michael Helbling]: in this episode. Yeah, well, it’s, but it is wise because it in a lot of ways in this current where we are right now with AI, it’s AI is an analytical contributor is very much in the look what I can do kind of phase. And, and it’s sort of like when you think about like coaching a junior teammate, if you want to think about AI like that, it’s sort of the same kind of thing is like, all right, strip all that out. You don’t have to say all that, you’re going way too far, you’re trying to impress because you have cool, you know, it’s like, don’t try to blind them with science, just get in, get out, say what’s important, you know, yeah. So that’s almost sort of how I feel about it. Because it’s like, yeah, it’s trying to do way too much. It’s like, look at the cool stuff I can do. And it’s like, mom, look, look at me, look at me. Like, yes, you’re very smart, 00:38:07.40 [Julie Hoyer]: shut up. Very good. 00:38:17.88 [Val Kroll]: Well, I can go with another one that’s not continuing on this like beautiful thread that we’ve been weaving. We’ll find a link, we’ll make it, we’ll ask AI to make it. 00:38:28.76 [Tim Wilson]: Wait, Tim, did you do your two books, though? Because I had cut it off. 00:38:31.96 [Julie Hoyer]: No, my two four was just the two books. The two books. 00:38:34.84 [Tim Wilson]: Colin Affleck and Steven Fugh. So yeah. 00:38:37.40 [Val Kroll]: Another experimentation heavy one, but another one that I have sent a link to these articles, maybe more than anything else. It was actually a medium post, it was a collection of medium posts, I should say, that was on Towards Data Science. And it was written by the Skyscanner engineering team. And it’s the overarching kind of umbrella title of content is Chasing Statistical Ghosts and Experimentation. And not only does it like break down some of the common like myths, but it’s the things that people still struggle to like fully understand why it’s not effective to run on it. So this is actually very similar to Julie’s first one in that it does produce a lot of visuals, although they are not interactive, to kind of illustrate exactly what the issues are with people having these like mental models. The one that I’ve sent the most, and there’s like four in the series, I believe, is the first ghost, which is it’s either significance or noise. And there’s like one quote in there, like towards the middle, and it’s about like, experiments don’t work towards significance and comparing relative significance of p values outside of the thresholds is a mistake that will lead to a lot of false positives. But the number of times and I wish that there was like a button I could press and it would like zap people in their chairs, if they said like, well, we’re trending towards significance. No, but it’s so well put, they like break it down into like its smallest little pieces, and kind of like build it all back up together. So anyways, it’s just every every part of this was so well done. But yeah, definitely kind of come back to this one quite a few times. 00:40:28.76 [Julie Hoyer]: Yeah, Val, you you turn me on to those and those are amazing. Great, like, ground yourself be like, I’m getting lost in the sauce, like let me go read my articles again for a second. 00:40:42.60 [Tim Wilson]: That’s awesome. It’s a good series. I feel like Julie, there’s a natural add on to that around 00:40:48.20 [Julie Hoyer]: experimentation that maybe you could yeah, I think I know which one you’re talking about. 00:40:56.52 [Julie Hoyer]: My well-thumbed through book with tons and tons of notes in the margin that I’ve actually read just the first five chapters about three times, I’ve done two book clubs on it. Field experiments, very simple, nice straightforward name. Really great book though, this was actually recommended to me and Tim when we worked with Joe, he was leaving at the time search discovery and we were running a randomized controlled trial with a client and it was put on me to, you know, continue it, do the analysis and run the next one and I was like, holy shit, like, what am I supposed to do? And Joe just sends us a link, he’s like, it’s fine, buy this book, read the first three chapters, you know, you guys will be good. Julie, you got this and you did. Yeah, it was 00:41:46.52 [Tim Wilson]: a little scary. I read it and I was like, I was like, wow, ooh, Julie, you got this. 00:41:52.44 [Julie Hoyer]: Yeah. Honestly, thank God. 00:41:55.80 [Tim Wilson]: You know Joe’s other one’s number, so give it a go. 00:41:57.80 [Julie Hoyer]: Yeah, thank God, he still could contact Joe. Also, thank God I was a math major and could read mathematical equations, but it is a really good book. Like for a dense book, a very informative book, again, that’s one that just set such a good, clear foundation. Like the way they write about these complex theories of running these statistical tests, randomized control trials, like again, like the magic of randomization. I mean, that shiny app I talked about at the beginning came from the same phase of life as reading this book. And it was so good and I still 00:42:36.60 [Tim Wilson]: go back. I remember that. I remember hitting that. That was like, yeah, my favorite phrase now. 00:42:41.96 [Julie Hoyer]: Yeah, really, if you need like a crash course in the foundations of randomized control trials, I really do highly recommend that book. Even the first five chapters, Joe said three. I found I needed to go at least to five. I think there’s 10 chapters total. Really good. 00:43:00.36 [Val Kroll]: As a member, former member of one of the book clubs that you ran for that book. You remember, I think I was in your second one. That was one of the densest books. It’s not a quick read, but it’s like the pieces of valuable nuggets per word is probably the highest density of any book. I’m like, oh, there was like two paragraphs and there was like three light bulbs that went off. So, and really good examples in that book too. So yeah, I like that in stories. 00:43:29.40 [Moe Kiss]: Good ones. Tim, you’ve got to talk about that. There’s one you’ve got to talk about. I’m dying 00:43:34.68 [Tim Wilson]: to hear it. Was it possibly first break all the rules? Okay. This is one where is Michael accused 00:43:44.28 [Moe Kiss]: me of not having empathy. I was surprised to see this on a list with your name. So first off, Tim, 00:43:52.60 [Michael Helbling]: it’s not an accusation. It’s just an observation. 00:44:02.76 [Tim Wilson]: Just do your fucking job. Yeah. So first break all the rules, what the world’s greatest managers do differently by Marcus Buckingham. And there were a couple of editions and other people wrote with them. And it’s Strengths Finder is what gets like all the play. And I’m not a fan of Strengths Finder, which is tied to now discover your strengths. But it was to a two book pair. First break all the rules, now discover your strengths. And I read it. It was like required reading early 2000s for managers at the company I was at. And to this day, it gave me a lot of confidence when I started working with people who just weren’t the right fit for the job they were in and getting comfortable with this idea of like skills versus talents. And you can you can teach skills, you can’t teach talents. And that doesn’t mean you can’t raise people up to get better. But if somebody is just like not good at data visualization or not good at building trust or not good at client communication or whatever it is, you can give them training and we have this tendency to say, well, that’s their deficit. That’s their deficient in that area. Let’s spend as much time as we can coaching them and training to to bring them up. And when you start to recognize that it’s like, no, that’s just not them. That doesn’t energize them. The best you can do is get them to a base level of performance. That was like one big aha from the book. The other one that like really blew my mind because the way they write about it is like so true is that our tendency, if we’re managing a team is to spend the most time with our low performers because it’s like the idea that the lowest performer is what the team is going to be judged as. And they’re the people who need to be raised up. The rock stars, you’re like, they got this. So we’ll just dump more and more stuff on them. And the book makes a really, really strong case for saying you’re you’re shooting yourself in the foot, you should be spending, you should be super charging the rock stars. And you need to support the lower performers, but it’s not really your job to try to make a square peg fit an around hole. Your job is to see if there is a role you can get them into, where they will thrive and become rock stars, but trying to coach and train when they’re just not a fit. I mean, that’s literally I read that 20 years ago. And it’s one that I multiple times, I will see it, I will be interacting with somebody, I will give honestly, even though I am kind of a jackass, like I go in with a presumption of good intentions and good capabilities. And I’ve, especially with analysts, I have gotten more and more confident over the years of somebody who’s just not, when you start redeeming their backstories, it also often was somebody who is really struggling. And they like the idea of punching buttons from the data and getting glorious insights and just have zero intuition or any of the compulsions that analysts do that make them good analysts. And I’ve got like names in my head of like, they’re never going to thrive in this. And the best thing that can happen for them is to find their way into another area. So yeah, that’s weird. That’s like a management book. But I find myself coming back to it. 00:47:36.04 [Michael Helbling]: And would you say, Tim, this has been more applicable in your work life or your podcast life? 00:47:44.52 [Tim Wilson]: Only, only so many roles I can shift people around into and I have a little control. 00:47:50.44 [Michael Helbling]: No, I’m sorry that I made a joke about it. Because actually, I see this in you, Tim, like you doing this and like how it affects how you interact with people. And I think I’ve learned some of these same lessons over the years in managing teams and people of like figuring out how to shape their role to be help them be as effective as they can be based on sort of what they’re what they’re naturally inclined towards versus sort of what you sort of want them to do. And I like that puzzle of figuring out sort of where people fit. It’s, it’s fun. Not always, you’re not always allowed in every environment to to puzzle with it as long as you’d like to. But 00:48:31.24 [Tim Wilson]: it’s the fact is spending more time with your stars is a lot more fun and energizing helps you grow as well. So like that book, giving the permission to say, you don’t feel bad that you’re like collaborating with somebody who is everyone thinks of as being like the star on your team. Like that’s here’s here’s the reasons for why that’s actually the right thing to be doing. Again, not neglecting people. It’s not like it’s a complete yeah, binary. But yeah, it’s sort of in 00:49:02.36 [Michael Helbling]: the same vein. It goes way back. I there was a radio lab episode back before I even listened to podcasts. This is like on the radio station like back in 2010. And they were discussing like how people use emotion and logic and decision making. And they had a specific case of a person who’d lost the function in their brain that allowed them to bring emotion into decision making. And so all their decision making was completely logical only. And as a result, they were actually paralyzed in every decision they would make. And it literally destroyed their life. Because they they couldn’t decide between like, Oh, I need to sign this document, should I use a blue pen or a black pen? And then it would go back and forth from the pros and cons of a blue pen or a black pen to sign and then everything they ever did good grocery store, they couldn’t pick which toothpaste to buy like, and it was really fascinating to get a window into this idea that so often emotion drives decision making more or as much as logic does and sometimes a lot more than logic does in a lot of environments. And that was another big like light switch for me. And around that same time, I was reading a book called Switch by Chip and Dan Heath, which kind of goes into some similar concepts about how to kind of be influential or make people see what you’re trying to say through different ways of kind of reasoning and and providing good examples of things and stuff. So that was sort of a time where I was sort of like, okay, yeah, how do I get people to like, engage with decision making? And also how do I this all goes back to me trying to figure out how to leverage who I was in the context of analytics, let’s just be honest about it, because like, I’m, I wouldn’t call myself a traditional analyst by any stretch. And so by kind of learning some of these lessons and watching sort of the emotional process of decision making, it gave me really good hooks into, okay, here’s how I can engage with people because I can feel the emotions coming off of people a lot of times and kind of into it, what to do. Whereas my ability to like create amazing analyses like Tim and with perfect visuals and everything is not, I have good skills, but they’re not as good as yours, Tim, let’s be honest. And so as a result, like, you know, I had to fool people into doing what I 00:51:26.36 [Tim Wilson]: wanted them to do. So it’s almost becoming like a trope or like conventional wisdom that people aren’t, they don’t make decisions based on the data and like they make it based on emotions. And then that sometimes that gets used as like weaponized against analytics, like how do you square that in a way that’s healthy because it gets, it gets treated like so throw away like, well, then what the hell are we even doing here? It’s such a balancing act. 00:52:06.36 [Michael Helbling]: I feel like there, people get a hold of this information and do seek to manipulate it. And I feel like that’s, at some point, like that kind of behavior is going to get caught and it’s going to have a limiting function. And so you shouldn’t do that. It’s sort of like if you’re a company 00:52:27.08 [Tim Wilson]: that’s in the public. People being being manipulative and playing to emotions. Yeah, yeah, yeah. Isn’t okay. Gotcha. So that’s like the whole point of traceology, right? And that’s why I love 00:52:37.08 [Moe Kiss]: that show so deeply is because like, each episode is going into one of those biases and how it shows up and the decisions that we’re making. And like, and it’s, I mean, Katie Milken is just so good at explaining what sometimes they’re like, quite technical, scientific studies in such a relatable way. But I just, I don’t know if I see it being weaponized. I’m more, I actually sometimes feel, I see it the other way, which is that I see data people being like, data decisions should totally be based on logic and what the data is saying. And I, I have very much come to this piece of like, intuition matters more. That is a different piece of data that is all your past experiences. And that also is worthy of consideration when you’re making a decision. There are other quantifiable data points that you also include, but like the idea that someone’s only going to make a decision based on those without other factors like intuition, what you’re like, I don’t know, maybe. Is that, is that like when you kind of intuition together? 00:53:40.84 [Tim Wilson]: You should be bringing together like facts & feelings. 00:53:45.32 [Michael Helbling]: Somebody should run with that. Somebody should run with that. But yeah, that was the other word, Moe, that I was going to bring into it as well. Intuition sort of runs this path as well, because people will rely on that more than they will rely on external facts and figures. And like, there’s that famous Jeff Bezos quote of like, oh, if the numbers look different to you than your intuition, then probably trust your intuition, you know, that kind of thing. And, and in reality, it’s true. Like our, our, and in the way our minds work as humans, there is a really good book on storytelling by, I think we’ll store and one of the quotes out of there was basically like, we look to fit patterns. So if the data fits our model, then we’ll be more willing to accept it. And if it doesn’t, or more, more, less likely to onboard it, just because like the way our human minds work, we’re trying to fit things in. And so if it’s sort of like counter to what our intuition or, or what we think is true of the world, but also when you look at even your own decision making process when you have different emotional states. So if you have high cortisol levels and anxiety, your decision making is actually impaired as compared to like, lower levels of anxiety or, or more calm feelings. Like it’s just true. Like you, you make different decisions and those don’t smirk. All of you are laughing and smirking because you know it’s true. It’s just hitting home for me. I’m like, yeah, yeah. And we think about that. So, so think about that for like a CMO who is like literally these next few campaigns literally mean their job. And you were coming in with some analysis that need them to make decisions and they are so high strong. Like how do you even pull some of the action out of the room so they can get to a good place to make good decisions sometimes? It’s so tall. It’s like you end up in a therapy role practically when you’re trying to provide analysis. And it’s, it’s not like you should be doing that, but it’s sort of like a reality of the work environment sometimes. So we don’t always do a good job. But like there’s all this data that, you know, Google did that huge project about sort of how people perform maximally in their jobs. And like when you provide the right amount of psychological safety and those kinds of things like people’s performance improves, like there’s so much data to support all of this stuff. And so the idea that emotion shouldn’t have a place or isn’t involved in decision making, like you really kind of like miss that at your own peril. And it’s not just sort of like, how do I feel about you? It’s like, how is everything around me happening? And it could be nothing to do with you walking in somebody’s office with like an amazing analysis that you’ve really thought through and prepared very well. And the meeting might go super poorly and just has a zoolcho to do with you. It’s because they got screamed at in the meeting before this, and they’re still coming down off of that. And there’s really nothing you can do about it except maybe come back Moenday and talk about it again and hope that they’re in a little better place. 00:56:45.32 [Tim Wilson]: When there’s like a, like another corollary, which is separate from that is like recognizing 00:56:50.92 [Julie Hoyer]: fear and shame, like negative emotions that the goal, like if you can get them emotionally on 00:57:00.28 [Tim Wilson]: as a partnership, it goes to kind of the trust, it goes to all of the pieces of this, any idea that you’re going to walk in and like put up a chart that makes somebody look bad or surprises them in a negative way. You know, even if you’re like, oh, I’d make these four people really happy, but this person, it makes them look like I’m finally calling bullshit on them. And that’s going to be great because my manager and their manager are going to love me because I’m going to help them win this battle. Like that is a short term win. And you just, like much, much 00:57:32.76 [Michael Helbling]: better going to be gunning for you every single time from here on out. Even if they’re a great 00:57:37.88 [Julie Hoyer]: person who doesn’t want like much, much better to have them along on the journey so that they too 00:57:45.16 [Tim Wilson]: are vested in like, it’s like asking the question like, what would you need to see in order to make this decision? Or what would you like to bring them on the journey where they are, I don’t know if that doesn’t go, that’s trying to play to the emotions, marrying it with the data to say, let’s think about how would we feel if we saw that? How would we feel if we saw that take a little bit of the sting out of the, what does this mean for my self-worth and my professional career and make it more about, hey, we’re doing something good for the company. Wow. Damn, that’s me. I’m sorry. That’s too much lip service to soft skills. I love it. I mean, maybe what we’re 00:58:27.08 [Michael Helbling]: finding out, Tim, is like, all this stuff really matters in a world of AI. There’s not a word I’ve 00:58:32.44 [Tim Wilson]: said that I haven’t just got straight from Claude. Like, I was just like, and now they’re talking about emotions. What should I say? It actually bugs me. Oh, go ahead. Well, no, go ahead, Michael. 00:58:44.84 [Michael Helbling]: No, it’s a sort of a tangent, but like lately, Anthropic has been releasing sort of these things about how Claude has like these emotions that it expresses. And it really rose me the wrong way that they even frame it like that, because I’m like, you can’t be teaching people to like emotionally support in AI. Like, it’s not a good way to deal with LLMs in my view. But again, you know, whatever, what do I know? 00:59:15.80 [Val Kroll]: Well, you made me think of another quick one, making sure someone’s ready to hear a message, especially if it’s a difficult one, episode 240, Taylor Bonacore Guthrie. The questions, that’s one that I definitely reach back for a lot. On the scale of one to 10, where, you know, one is a dumpster fire, 10 is the best day of your life, where are you? And so if they say, four, because I just got my ass chewed by, well, you’re like, well, we’re gonna save me for Moenday. And we’re gonna go ahead and come back. Need a couple more days. But yeah, no, that’s a, that was, I learned a lot from, from that episode from Taylor and definitely have returned to that one. So that, that’s another one I should have put on there for 01:00:01.56 [Michael Helbling]: this episode. That’s a good one. All right, we do have to start to wrap up. This has been a really fun conversation. I think all of you, there’s been some really amazing insights. So trying to empathetically put myself in the position of a listener, I think we had some good stuff there. So hopefully it’s helpful. You know, we’ll, we’ll wait and check the 01:00:24.84 [Tim Wilson]: median, median listen length and let the data tell us whether this was good or not. 01:00:29.40 [Michael Helbling]: Yeah, that’s right. The algorithm will tell us. Anyway, as you’ve been listening, I bet you have things that you found in your career that are super helpful and guide you even in an AI kind of environment. We’d love to hear from you. So yeah, please reach out to us. The best way to do that, you can do that through our website. You can also do that on Measure Slack or on LinkedIn, or via email at contact at analyticshour.io. And as you listen to the show, leave us a rating or review on whatever platform you use us on. We’d love to hear those. We’d love to see them. It helps us out. And if you love to rock a sticker on your laptop or your phone case or something like that, we’ve got analytics power hour stickers and you can order those as well on our website at analyticshour.io. So yeah, thanks again, everybody. Moe, Julie, Tim, Val, this is really nice. Yeah, I’m feeling really nice. I feel good. All right, this kind of felt like a hug. Yeah, I feel more prepared to face the future. And I think no matter what the future holds, I think I speak for all my co-hosts when I say 01:01:41.88 [Announcer]: keep analyzing. Thanks for listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at at analyticshour, on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group. Music for the podcast by Josh Crohurst. Those smart guys wanted to fit in. So they made up a term called analytics. Analytics don’t work. 01:02:08.84 [Julie Hoyer]: Do the analytics say go for it no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning 01:02:20.28 [Julie Hoyer]: in competition. Okay, one of my fallback ones I have to say to that point, Michael, I was trying to remember, I always like reference this one idea. I was like, I knew I listened to it on a podcast and I was trying so hard to remember what it was. It was from this podcast. So I put it in there as a fallback, but it was this podcast that I heard. I mean, please do bring that one up. 01:02:43.80 [Julie Hoyer]: I was like, that’s maybe a little awkward, but I’ll throw it in there just in case. 01:02:51.56 [Michael Helbling]: I would even like that episode very much. So I’m glad you got something good. Yeah, I didn’t mean it either. 01:03:00.68 [Julie Hoyer]: Which is funny because, yeah, we don’t have to talk about it now. We don’t know. We can talk about it. It’s fine. 01:03:07.40 [Tim Wilson]: That’s East like Denison. Oh, okay. Yeah, that sounds really tough. Yeah. 01:03:12.92 [Michael Helbling]: Shut up, Val. Columbus is not that big, but 45 minute commute in Columbus. That’s a big deal. 01:03:19.56 [Val Kroll]: No, you said South side of Columbus. It just sounded like… Oh, that’s not you. You come from Chicago. 01:03:27.72 [Michael Helbling]: No, South side of Chicago, yeah, has a rep. 01:03:32.36 [Julie Hoyer]: See, I feel like I should be drinking a glass of wine to tell that one. Do I? Okay. Parameters here before I go off and don’t do the format of the show, right? Also, I feel like I’ve picked up a slight Southern twang from being in Mississippi for two days. So if you hear it… Oh, this really is a… I’m not actually background. 01:03:53.48 [Tim Wilson]: Dude, you become vaguely racist and also misogynistic. Hey, easy. I’m not trying to defend Mississippi. 01:04:02.20 [Julie Hoyer]: But do we want to give the background of where it came from or how you first… 01:04:07.56 [Julie Hoyer]: Yeah, of course. …came across it and then how you used it? Is that kind of the format or am I misogynistic? 01:04:12.92 [Moe Kiss]: Just go with the vibe. 01:04:15.08 [Julie Hoyer]: But I’m not saying the vibe. And my freaking comment. 01:04:18.12 [Michael Helbling]: Hold on. No, you’re not freaking everyone out. Just testing my distance from the microphone. 01:04:21.64 [Julie Hoyer]: Do I need… Is my volume okay? Your volume is fine. That’s my hot mic. Your volume’s great. And you don’t have to be quiet. 01:04:27.96 [Julie Hoyer]: Yeah, I’m sorry. 01:04:30.36 [Julie Hoyer]: True. See, that’s the thing. Like, I tend to try to be quiet because my mic was so bad. 01:04:33.88 [Val Kroll]: I can tell by the way you move your mouth that you’re trying to be quieter. But also, I’m weirdly observant about things like that. No pressure. 01:04:45.00 [Moe Kiss]: Fuck my life. One second. I’m so sorry. 01:04:47.40 [Michael Helbling]: All right, perfect. I didn’t want to start right now. Okay, wait. 01:04:50.52 [Julie Hoyer]: Do we get to tell the story about Moee and her saying, fuck my life in our last episode? Oh, it’s in the outtakes. 01:04:56.28 [Tim Wilson]: It is totally in the outtakes. Oh, my God. 01:04:57.96 [Julie Hoyer]: Wait. 01:04:58.12 [Tim Wilson]: So good. 01:04:59.72 [Julie Hoyer]: Oh, that’s Moee’s tagline. 01:05:04.44 [Tim Wilson]: Yeah, Moee had to like, as we were losing co-hosts sporadically for various crises, and Moee left and thought that we would have finished wrapping up by then. So she just came back in with the hot fuck my life. And we just went with it. In the middle of the wrap up. Can I finish the wrap up now? 01:05:26.04 [Julie Hoyer]: And Moee was just going in the back. He’s like, can I finish wrapping? 01:05:29.72 [Julie Hoyer]: And she was like, oh, my God. I was like, yes. 01:05:33.88 [Moe Kiss]: I think I actually said fuck my life again. 01:05:36.28 [Michael Helbling]: Yeah, you did. You did. Yeah. 01:05:38.60 [Moe Kiss]: Oh, jeez. Anyway. Okay, good to go. It was great. 01:05:40.52 [Michael Helbling]: All right. 01:05:41.56 [Tim Wilson]: Here we go. All right. We’ll get started in five. I’d like to say right now, Tony, this is going to be the perfect. There will be what you went through on the last one. This one is smooth as silk. Rock flag and do your fucking job. The post #297: Durable Wisdom in an Age of AI Slop appeared first on The Analytics Power Hour: Data and Analytics Podcast.

April 28, 20261 hr 4 min

#296: Avoiding Major Oopsies: Twyman’s Law, Intuition, and Valuing Accuracy Over Precision

What do diamond ring shopping, Uber pricing psychology, and active user metrics gone wrong have in common? They all highlight our complicated relationship with precision versus accuracy—and how that relationship can either build or destroy trust in our data. Arik Friedman from Atlassian joins us to unpack why being “about right” often beats being “exactly wrong,” and why your nagging feeling that something’s off might be a useful insight in and of itself. From the discipline of documenting assumptions to the art of knowing when to round your numbers, we tackle the very human challenge of working with data that’s supposed to be objective but rarely is. Plus, we explore Twyman’s Law (if data looks too good to be true, it probably is) and why sometimes your intuition is your last line of defense against embarrassing mistakes. This episode is brought to you by Prism from Ask-Y—your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show (Free Books: separate R & Python versions) An Introduction to Statistical Learning (Video) Hamel Husain | The Revenge of the Data Scientist | PyAI Conf 2026 (X Article) The Revenge of the Data Scientist (Video) Lenny’s Podcast: Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar The Monarch App (Article) Vibe Coding Will Bite You. Here’s Exactly Where… by Cassie Kozyrkov A Checklist for Making Better Decisions: A Glossary of All the Topics We’ve Featured on Choiceology Photo by Sarah Kilian on Unsplash Episode Transcript00:00:00.00 [Tim Wilson]: Welcome to the Analytics Power Hour. 00:00:08.92 [Announcer]: Analytics topics covered conversationally and sometimes with explicit language. 00:00:15.20 [Tim Wilson]: Hi everyone, welcome to the Analytics Power Hour. This is episode number 290, oh wait, no, wait, let me check, this is episode number 296. 00:00:26.04 [Tim Wilson]: That was a close one. 00:00:28.32 [Tim Wilson]: I mean, how embarrassing would that have been? We’re all set to have a discussion about data accuracy and getting business partners to trust the data and I almost whiffed on something as simple as the episode number. Already though, perhaps I’ve undermined your trust in me, Tim Wilson, sitting in the host chair that is usually occupied by Michael Helbling. Perhaps I have. Let’s ask my co-hosts. Moe Kiss from Canva, welcome to the show. What do you think? Did my little gaff destroy my credibility? 00:00:57.68 [Moe Kiss]: Yeah, trust is completely lost. No, I’m joking, you nailed it. 00:01:02.44 [Tim Wilson]: It’s been lost, it was lost years ago with you, so, and Julie Hoyer, from further, have I already destroyed our audience’s confidence in me? 00:01:14.16 [Julie Hoyer]: What’s a mistake among friends, right? I’m just your pre-read. 00:01:17.20 [Tim Wilson]: Okay, I like it. Well, that’s the topic for this episode. I mean, sort of. All data can be digitized and an overwhelming amount of it is and digitized data breaks down into like discrete little ones and zeros somewhere down the chain. It should be cold and objective and to use it effectively, we need to get it to be as accurate and precise as possible, right? I mean, well, maybe. So joining us for a discussion on just that topic is Arik Friedman. 00:01:48.88 [Tim Wilson]: Arik started his career as a software engineer, then took a turn and did a PhD in computer 00:01:54.28 [Tim Wilson]: science focused on privacy-preserving data mining, which is a tongue twister, then went on to work a few years as a program manager for Microsoft R&D, then he popped back over to research for a while, and now he’s been at Atlassian for over 10 years where he is currently a senior principal data scientist focused on product analytics. And today he is our guest. 00:02:18.40 [Tim Wilson]: So welcome to the show, Arik. 00:02:21.52 [Arik Friedman]: Thank you very much. Long time listener. Happy to be here. And I assure you, like all of these career moves made a lot of sense, like it, it made a lot of sense and at the time, it was a logical progression at the time. Absolutely. 00:02:36.34 [Tim Wilson]: I went architecture, technical writing, marcom analytics. So I, I sympathize and I actually, I don’t know that any of those made any sense. 00:02:45.84 [Tim Wilson]: But that’ll be something for me to ponder on my deathbed, but so Arik, you presented 00:02:53.40 [Tim Wilson]: at the last measure camp in Sydney and in that presentation, I think you had a, a fun or maybe it’s relatable at least, or maybe it’s a tragic story about an active users metric that looked like almost too good. Like the results were brilliant. Everyone was thrilled, but you had this like nagging feeling that something was off and you couldn’t really nail down the specific problem. So maybe that’s a good way to sort of start the show. Cause I think it gets to something really human about our relationship with data. So maybe we’ll kick things off by having this, having you walk us through kind of what happened 00:03:28.88 [Tim Wilson]: there. 00:03:29.88 [Arik Friedman]: Sure. So like this is a story from a while back, but you know, at the time the product team was working on a product feature, a big change in the user interface and a lot of investment went into that. So we put a lot of work on actually testing things, you know, A, B testing. I worked with another data scientist to make sure that we got things right. And I remember back then, you know, us standing in the boardroom with all the PMS and it’s looked like really a big change. Like it moved the needle quite significantly. So that was, that was a big deal. 00:04:11.24 [Tim Wilson]: Everybody were happy. And then as we went on and rolled out this feature, we saw what we expected to see. Active users went up. 00:04:22.96 [Arik Friedman]: But then, you know, over time, I’m starting to develop this, you know, old feeling, you know, you look at data over time, you get a sense of how it looks, how growth looks 00:04:34.28 [Tim Wilson]: like. 00:04:36.04 [Arik Friedman]: And I was looking at the curse and they looked a bit suspicious, right? Like it’s not supposed to go like that. And I started looking into that, okay? Because it felt a bit odd. And I remember, you know, trying to go down to specific trace logs, trying to find what went wrong, I found like everything checked. So you know, after putting some time into that, I said everything was good. We actually have a reason to believe that, you know, active users goes up, all is fine, right? I cannot say anything about this. I don’t have a smoking gun. Fast forward, sometime later, another data scientist and a software engineer, of course, we keep looking into this and they actually found a bug. 00:05:23.32 [Tim Wilson]: It turned out that because of a bug, it caused inflation of active usage monitoring. 00:05:30.00 [Arik Friedman]: And that was an unpleasant surprise for the product team. So for me, it definitely caused a lot of thinking, you know, like, how come I didn’t find that? 00:05:40.24 [Tim Wilson]: What could I have done different? 00:05:43.00 [Arik Friedman]: And that, you know, that causes a lot of reflection. 00:05:47.60 [Tim Wilson]: Well, did you go tell them, like, immediately, abruptly, did you ease it out? Like how did you actually communicate, like, what’s the rest of the story on that? 00:06:01.08 [Tim Wilson]: Yeah, so I think that’s probably the biggest mistake I made at the time because, you know, 00:06:08.36 [Arik Friedman]: I didn’t find an issue. And you know, I, at least back then, I thought, you know, we are the data people, we are the evidence people. And if we don’t have the data to back up what we say, we should shut up. And what I learned from that is actually, you know, our intuition are, you know, that’s part of our expert opinion. 00:06:31.08 [Tim Wilson]: And we should sometimes just go with it. 00:06:36.12 [Arik Friedman]: And I think there are a lot of things we can do ahead of time, you know, to prevent mistakes or to check things. But at least for me, like, for this story, like, when I look, you know, what could I’ve done different, I actually knew to do all the checks. And eventually, when your intuition is your last line of defense, and sometimes you just have to go with that. 00:06:58.04 [Moe Kiss]: So sometimes, I don’t know, I, firstly, I just want to say thank you. I’m really grateful that you’re sharing, you know, straight off the bat, an experience of where you made a mistake that’s incredibly humble and wonderful for folks to be able to learn from. So a very big thank you. And I suppose I just, I’m curious to understand this intuition piece, right? Like I feel we all have it. And I know you and I have had conversations previously about like when we’re making decisions, you know, we have data, we have intuition, we have, you know, maybe previous experience, we have all these different factors and part of our role is helping pull those things together. But I’m curious to understand. So how has this changed how you would show up? Like, let’s say this happens next time, you can’t find a smoking gun, you can’t find a data point, but you had your, you know, your intuition in your gut that’s like something’s not right here. Like, do you think this time you would say something and how would you frame it? 00:07:49.92 [Arik Friedman]: Yeah. 00:07:50.92 [Tim Wilson]: I think, first of all, it’s about being more opinionated. 00:07:56.88 [Arik Friedman]: And in the, like over time, that’s part of my growth journey. I think in the past I would, I tended to be a bit more, you know, impartial, right? We let the data speak. Like we’re just there to give the data its voice. And I think over time I feel a bit more confident, you know, to be opinionated, like I have my own opinion of things. And I think we’re actually expected to be opinionated. So I think that’s, that’s one thing, just like be more confident in our own expertise. And then I think it’s probably, I mean, the opinion on its own is not enough, right? Like a probably, even if we have just this intuition, we will still be expected to, you know, okay, dig into that. Okay, can you actually find the evidence, but I think it’s a start of a conversation, right? 00:08:50.52 [Tim Wilson]: Like this is what I think. Maybe the data looks a bit odd and we maybe want to dig a bit more into that. 00:08:59.16 [Arik Friedman]: And then the business can decide, you know what, no, like we did all our checks and balances. It’s good. We did our due diligence. 00:09:05.20 [Tim Wilson]: Let’s go on. 00:09:06.36 [Arik Friedman]: Or they could say, you know, let’s, let’s take more time and actually give it more space 00:09:10.24 [Tim Wilson]: to dig into that. 00:09:13.00 [Arik Friedman]: And at the time I basically made my own decision and said, you know, I looked enough into that and I kept it with myself. And this is something where we need to be more open and just share what we think with a business partners about that. 00:09:28.08 [Tim Wilson]: So I will sometimes find myself not doing this necessarily with a ton of structure and rigor, but trying to think what I expect to see before I actually look at the data, or sometimes asking if I really don’t think I have context and my business partner says, I think it’ll go up. I’m like, what is, like, what does up mean for this metric or what is, what would that mean? Like I don’t give me, use it as an opportunity to try to get a little more context about their intuition about their business, like there’s, there’s some psychological trick or something where you, you sort of force yourself to set what you think you’re going to see as opposed to waiting until you see it and then you immediately rationalize why it makes sense. Like I had an example from years ago where a product marketing manager came racing in because he had already told the whole company that this tiny little change he’d made to a webpage had like drastically driven the track traffic up, which made no sense. And that was one where I dug in and found that it was, it was a bot, there was a company that was selling us software and they pointed literally at his page to gather some data for the pitch in, in a few days. But it was one where it was like, he saw it, he was a good news, kind of like the active users. But if I’d gone back, if I’d known he was making that change, I would say, what do you 00:10:59.28 [Tim Wilson]: realistically think this is going to do? 00:11:03.24 [Tim Wilson]: Because if it goes way past that, maybe we want to, you know, apply a little bit of climate law or, you know, something to it. But I think there’s, I don’t know, I feel like that’s a whole, it’s a challenge, especially for people just entering a field or the space to have that intuition and to have the confidence on top of it. Like you said that a few times that it’s like, as you get more experience, the more faith you have in your intuition and your conviction. And that’s like, how do you, how do you develop that if other than letting time pass? 00:11:42.00 [Arik Friedman]: I think a lot of this is getting to live a bit in your business partner’s world, like getting to speak the language and, you know, getting to know what they care about, what they’re after. I think that adds a lot of context. And for example, I remember at some point, you know, I started hearing PMs talk all 00:12:01.48 [Tim Wilson]: the time about like JTBD, JTBD this, JTBD that, like jobs to be done. 00:12:06.72 [Arik Friedman]: I had no idea what it was all about. And, you know, I started digging into that, reading a bit about that. And, and, you know, what I found, like, I didn’t really get what they were talking about. So getting into the world, learning about that, I think helps create this common language and thinking about it from their perspective. So I think that definitely helps us, you know, to get, so to speak, like in their head. 00:12:34.20 [Tim Wilson]: When that was one where you actually, if I, I had this happen with OKRs once where I was working with someone who was, and I knew what OKRs were, I’d lived in kind of 00:12:43.72 [Tim Wilson]: an OKRs training, but it was like, oh, this client is really into it. 00:12:49.12 [Tim Wilson]: I’m going to go get a book. And didn’t you do that? Didn’t you wind up going and like read, like whatever the JTBD Bible is? Like, OK, if we’re going to use this, I want to understand it. I mean, you probably found out that they actually weren’t using it correctly. It’s like Agile or any other number of things that use the acronym, but maybe aren’t applying the process. 00:13:10.36 [Arik Friedman]: Yeah. So I went on and I actually read about jobs to be done. And I, it was actually weird because like what I was reading and what I was hearing from the PMs were not exactly the same things. And at the time, like in our confluence, like internal wiki, I wrote a blog post. Hey, like, are we doing JTBD wrong? 00:13:32.04 [Tim Wilson]: And it was a bit of a clickbait, but it started a conversation 00:13:36.96 [Arik Friedman]: because I mean, then I started like through the conversation with them. I started to find out, oh, actually there are different kinds of definitions of JTBD out there and having this conversation and trying to understand, you know, what they really mean when they talk about that, that really helped. And I did it from the perspective of, you know, how it can be more useful as a data scientist and what is it that they actually after when they talk about understanding the JTBD. So I think it was first, you know, just to catch up with them, but also to see, you know, how can I answer their questions and how can I better understand the question and, you know, improve myself as a data scientist 00:14:18.68 [Tim Wilson]: that helps them. Michael, I have news. 00:14:23.48 [Tim Wilson]: The AI analyst is emerging. 00:14:27.08 [Tim Wilson]: Oh, that’s big coming from the quintessential analyst. What do you mean, like a cryptid? Well, I mean, more like a more like a job promotion, but, you know, 00:14:35.32 [Tim Wilson]: with more existential dread, you know, how foundation models created the AI engineer role. Yeah. Developers got all these cool titles and analysts got. Can you pull this by in today? Exactly. But now we’re watching the birth of the AI analyst, someone who uses LOMs to multiply their capabilities without, you know, multiplying their stress rash. Nice. So an analyst, but with superpowers and fewer open tabs. Exactly. And the tool for that is Prism by ask-y.ai. 00:15:07.92 [Tim Wilson]: Yeah. Prism is basically the interface between what I mean and the 900 steps I don’t want to do. You ask in plain English and it helps you get from question to analysis really fast. 00:15:19.72 [Tim Wilson]: And it doesn’t forget your world. Prism’s jam memory as J.A.M. their jam memory remembers your definition like what your means by conversion. Which table is the source of truth? And that July data, don’t forget, it’s, you know, cursed. 00:15:37.64 [Tim Wilson]: Yes. Thank you. I’m tired of explaining our metrics like it’s the extended 00:15:43.92 [Tim Wilson]: MCU universe. Plus, you can capture like repeatable workflows as skills, portable expertise, like, you know, clean UTMs or fixed GA for channel grouping or standardized campaign naming and reuse those across different data 00:16:01.84 [Tim Wilson]: sets. I like that because I do a lot of copy paste. This is can’t continue type of feeling. And I feel like it would be nice to be like run skill, look smart, drink water. 00:16:15.64 [Tim Wilson]: Nice. Want to become the AI analyst before your coworker does? Go to ask-y.ai and join the wait list. 00:16:24.16 [Tim Wilson]: Yeah. And use code APH and ask why it pushes you to the top of the wait list. That’s ask-y, the letter Y.ai and use code APH. Yeah. The AI analyst is here. This product is in beta, but you can get in on the ground floor. And it’s coming for your busy work, not your job. 00:16:44.64 [Moe Kiss]: So, you know, relax, chill out. So, Arik, you and I have spent a lot of time talking about this whole like accuracy versus precision playoff. And it’s something that has just really resonated with me because I would say I always lean to kind of like the best possible answer with the time that we have in the business to help make a decision. Like, I suppose I would say I’m like quite pragmatic, but a lot of what I hear coming from data scientists is this number is wrong. This one’s right. We can’t do it this way. This, you know, we have to do it this way because this is the right way. And I guess I just wanted to hear your framing about this like playoff between accuracy and precision. And like, I don’t know, you have such an elevated way of thinking about it. 00:17:31.40 [Arik Friedman]: Yes. So I think like straight from primary school, the way we’re taught things is that accurate and precision are like being accurate and precise are the same things, right? So at school, you get like this big equation. And like in the data science world, you can think about a business question, 00:17:51.20 [Tim Wilson]: like, you know, what, what kind of features correlate with user satisfaction 00:17:55.96 [Arik Friedman]: or something like that? Or how can we predict those kind of things in parallel? 00:18:00.60 [Tim Wilson]: Like, like at school, you might get a question, like, you know, 00:18:06.24 [Arik Friedman]: how much is like 2,124 times 3,926? Like you get this big equation. And what you’re taught is that you need to go through the methods, right? Like you have to multiply this digit by that digit, carry over. If you get anything off, like any single digit is off, you lose all your marks. So if your answer is not precise, it’s not accurate. You lose all your points. And I think we kind of like carry on this mindset with us into our jobs. And in the business domain, actually precision and accuracy are not the same thing because, you know, because, oh, like these big numbers, it’s about, you know, 2k times 4k, it’s about 8 million. And like about 8 million is accurate. It’s not precise, but it’s accurate. And I think that also, like when we get, you know, business questions, there are many ways we can go and approach and solve them. You know, we can throw them, I don’t know, like hidden Markov models or, you know, clustering algorithms. So there’s all this arsenal of like all these methodologies that we learned. And like, you know, that’s kind of like even part of our pride in the craft, right? Like we want to show that we know to do all those things. But sometimes you can get a quite a good answer just by writing like a very simple SQL query. And, you know, it’s maybe not the best answer, but it’s good enough and it’s faster. And there is this quote from John Tukey from like his, he had like this paper about the future of data analysis. 00:19:43.44 [Tim Wilson]: And like this is from like the sixties. 00:19:46.44 [Arik Friedman]: And he says, like far better and approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise. 00:19:56.92 [Tim Wilson]: And I think it really captures well that, you know, it’s first of all, 00:20:02.28 [Arik Friedman]: it’s better to answer the right questions. And I think that part of our job is to help get to those right questions. But also when we answer, it’s not just about precision. It’s, you know, the answer can be accurate without being precise. 00:20:18.24 [Tim Wilson]: And I think that’s a way for us to be more fast and effective 00:20:23.24 [Arik Friedman]: and be focused on business impact. 00:20:24.84 [Tim Wilson]: But there’s there’s a there’s a challenge, right? 00:20:27.56 [Tim Wilson]: That that our business partners accessing tools and data, they’re working with spreadsheets, they’re working with dashboards because they’re in this digital world and processing and multiplication is is is cheap. Like they are conditioned to see things that are precise. I mean, I love the like the the accuracy versus precision. Like a dark archery thing that sort of shows accurate the grid, the two by two of accurate and precise, accurate, not precise, precise, not accurate, not accurate, not precise. And it shows like the spread around the target. 00:21:07.80 [Tim Wilson]: And I think that’s good for us to be aware of. And part of it is, I mean, can that be handled a little bit with the language 00:21:17.72 [Tim Wilson]: of saying, try to get yourself out of the boat of providing. Precise answers, like get comfortable with even if you have a precise answer, better to kind of back off the precision of it a little bit so that somebody isn’t looking at you’re not arming them with ways to. To find another person, it may be too reasonably accurate, 00:21:44.64 [Tim Wilson]: but the precision means that the the first three digits kind of differ. 00:21:49.92 [Tim Wilson]: Like there’s there’s a challenge with us understanding that and our business partners thinking that way. And the levers we can pull to try to help them think in language of. Of this is this is accurate, you know, it’s about it’s about eight million, 00:22:08.88 [Tim Wilson]: you know, or whatever the answer is. 00:22:11.92 [Tim Wilson]: And they’re probably not going to object to that. They’re not going to come back and say, what do you mean about a million? 00:22:16.08 [Tim Wilson]: I need that down to the to the first digit, right? I mean, it’s the final line to walk. 00:22:22.76 [Arik Friedman]: Yeah, and I think it really depends on the context and the question being cast because in some contexts, like if you deal with financial data, you definitely want to be precise. It’s important, but most like a lot of the business questions are more direct directional in nature. And when you deal with directional questions, it doesn’t matter as much. You know, if you’re precise and actually precision can kind of draw the attention to the wrong thing because you don’t care about the position necessarily in these kind of questions. So this is about getting to the bottom of what are the decisions that they are about to take 00:22:58.80 [Tim Wilson]: and what really matters for answering those decisions. 00:23:02.36 [Moe Kiss]: So one of the things I wanted to like challenge and have a discussion on. So I’m trying to find a specific article. And of course, I can’t. But there was some research that was done with pricing and Uber. And I often quote it when I’m counselling people on buying engagement rings. Buying a two-carat diamond ring is more expensive than a one point nine nine nine carat ring, right? Like we have this bias and heuristic to think that the two carat, the rounded number is better. So hold on. 00:23:30.16 [Tim Wilson]: Just wait, wait a minute. How many people are you counselling on buying rings? Like, what is your world? 00:23:37.52 [Moe Kiss]: I my team are quite young. And so a lot of them are going through that life stage. 00:23:43.76 [Tim Wilson]: Those are just one on ones. They’re like, hey, what’s going on? I could really do some help. 00:23:48.36 [Moe Kiss]: Anyway, OK, carry on. So also, I don’t know who’s buying your two-carat diamond ring. 00:23:54.52 [Tim Wilson]: It’s a it’s a anyway. 00:23:59.32 [Moe Kiss]: Is that that’s a that’s a big one? Three carat is quite big. Yes, that that’s not for America, but for Australia. Americans tend to buy much bigger rings. Anyway, back to the point of the story. Well, and now the lab, you should see these rings. 00:24:12.16 [Julie Hoyer]: People are thrown around. 00:24:13.76 [Tim Wilson]: Sorry. You guys are living in a different world. 00:24:18.44 [Moe Kiss]: But what I was going to back to my point, one of my concerns is like, I hear you and I agree with everything that you’re saying, right? But especially, OK, the same thing happens when you’re trying to counsel people on like a day right. Sometimes like people will intentionally like make a number, not a round number, because people are more inclined to believe it, that you’ve put more work into it, because we have this bias that say if you say $1,600 a day, that like it’s not it’s you you’ve just picked a number from the sky versus having put thought into it. So like I’m just trying to think about how that plays off with our discussion here about not necessarily always wanting that precision. So you’re saying if we don’t go precise enough, 00:25:04.72 [Julie Hoyer]: they won’t trust the number. 00:25:06.48 [Moe Kiss]: Yes. So if we say, oh, the number is eight million versus eight million two thousand and ninety seven, will people just be like, is that bias going to come to life? Where they’re like, oh, maybe that number is not right. And like I can kind of disregard it because it’s come from nowhere. Or like, do you see what I’m saying about like, does the precision give any extra credibility or build more trust? 00:25:29.84 [Arik Friedman]: Right. So in those cases, like precision sometimes is kind of like the signal that you did the work. Right. And again, I think these kind of things are context dependent because 00:25:42.84 [Tim Wilson]: ideally, if you have established a good like 00:25:47.72 [Arik Friedman]: trust trust relationship with your business partner, then they trust your judgment and said you made the call the right call about what’s a good methodology to approach this. And in some cases, yeah, maybe precision does matter. Like maybe it’s a signal that’s important to them. And in that case, yeah, maybe you do need to put a bit more work 00:26:11.44 [Tim Wilson]: and and give that kind of signal. 00:26:15.60 [Arik Friedman]: So that definitely is context dependent. 00:26:18.28 [Julie Hoyer]: And part of the context, I’m just thinking like, Tim, like back to some of our discussions we’ve had with clients when they have really low volume, right? The difference of one is greater than if you have a large volume. The difference of 10 is like minuscule or, you know, like you can kind of ignore it. So I do feel like, too, it’s like, are you speaking of a really small end? Because then you don’t want to round compared to when you’re speaking large volumes, you can round to eight million and it’s probably OK. The other thing with precision that I think is interesting as we get sucked into a lot is when the stakeholders want to compare systems, numbers from systems, which we know are not going to give you the exact same one. Like directionally, they should be close enough. But I do feel like that is a scenario where they really get stuck on. Like, why is this one percent different? And it’s like, well, it’s OK. 00:27:16.12 [Tim Wilson]: It’s like eight million. 00:27:18.56 [Julie Hoyer]: It’s all right. And we’re not talking dollars and you know, which one’s your source of truth. So we should be OK to continue to use, you know, the other number in our decision making. But that’s really hard, I think, for stakeholders and even for analysts 00:27:33.68 [Tim Wilson]: to navigate. But I think we tend to make, I mean, if it’s like, you’re going to err on one side or the other. I mean, I look at somebody makes a line chart to show the results and they label every single data point down to the first value. And that’s not so like there’s there there are choices made in the presentation of the information where I think you can say there’s precision precision there. I’m showing a dashboard, but this is rounded to the nearest million or the nearest thousand. It’s not I’m not giving you eight zero zero zero zero zero zero. I’m saying eight million, you know, or eight point one million. You know, give them a give them a decimal place. So I think there are choices. I mean, certainly are to your point that like the context and actually Julia point as well, like the context does matter. When does it matter? But I I feel like there’s like the trap we can fall into on the analytics side is we have the precision precision might as well show it. You know, our P value is not only is a less than point oh five. It’s every model spits out. It’s the P values point oh oh oh one three four seven two. Like please don’t put that, you know, just kind of paste. Yeah. P values less than point oh five. 00:28:51.64 [Arik Friedman]: And Julia, I think, by the way, it’s a great point because it matters a lot like why the numbers are different. And, you know, scenario one, the numbers are different. We have no idea why. OK, that’s that’s not a good place to be. And that’s that’s a place where you probably do want to ask, you know, questions why why do these things don’t match versus a different scenario where, you know, we actually expect those numbers not to be perfectly the same because maybe we measure slightly different things or, you know, there are known reasons why they should mismatch. So I think usually it’s more about the confidence. Like we know what’s going what’s going on here and team to your point. Yeah, like maybe if we think that this will just distract and raise, you know, questions that are not relevant, then yeah, maybe it’s better off to reduce the position to just avoid that altogether. 00:29:44.04 [Tim Wilson]: So I think the other I think business our business partners and then data teams get sucked into it when it’s the different systems aren’t going to reconcile and I think to your point, like you should understand not down to the perfect we can net out everything, but say these are the the big movers of the differences. A lot of times there’s the ability to say over time, look, these move in the same direction, I watch companies say, well, we just need to pick the system of record for that metric, which feels like my the hairs on the back of my neck go up. Like that’s a cool idea that somebody in the C-suite or one level down said, we have the solution. We just pick our system of record. But the reality is all those systems exist for different reasons and it gets back to context. So I think there’s a part that says we need to put this to rest at some point by doing a little bit of an analysis of saying these differ. These are the main reasons they differ. Let us show you that they generally move in the same direction over time. And now we’re just going to have our standard little footnote that these, you know, move differently. But I think also we’re going to go look at the system that gives us the most what we need because it’s often kind of like upstream system in some process has this data, has this data, hands off to another system, which goes downstream from that. So you have to pick the system where you can get the slice. I mean, I’m thinking in a CRM digital analytics CRM sales world, you can necessarily track the marketing channel all the way through to a sale. And in the middle, you’ve got something like a lead that the downstream ability to follow it comes out of one system. The upstream to follow it comes from another system and heading off and understanding, and maybe this gets back to your point earlier. You’re like, you really need to be figuring out is the question being asked very clear and precise and let that drive everything as to where you look, what level of precision, how you pull it, what the intuition is. And now it just went on a bit of a tirade. 00:32:05.88 [Julie Hoyer]: Well, no, but you bring up a point that, you know, maybe this is me still like wrapping my head totally around the accuracy versus precision. But I do keep thinking of what you were calling out like the target. So we were talking about people being most comfortable with precision of systems matching, but we just discussed, you know, thoroughly, like why that’s, we know it doesn’t happen. But to your point, Tim, if they are, if we can explain in at least broad strokes, Arik, as you said, like why these things are differing. And if we then can say, well, as long as we’re pointing this at the right like target, the right question, the right problem to be focused on, then can we be comfortable with accuracy without precision between the tools? Like, am I taking this way off? But you’re saying, you know, like, if they’re always about 1% off, they’re 00:33:03.28 [Tim Wilson]: all hitting the right target, maybe not closely bunched together, that would 00:33:08.16 [Julie Hoyer]: make people happy with accuracy and precision. 00:33:14.08 [Arik Friedman]: Yeah. So I think that data science teams usually have a good opportunity to collaborate and aim to standardize measurements. And, you know, one example, like I’ve seen in the past where, you know, growth teams and product teams had different measurements of impact. 00:33:31.52 [Tim Wilson]: And that can be very confusing because, you know, when one team says, oh, 00:33:35.68 [Arik Friedman]: like we had that impact, and then the other team kind of like interprets it in their own language. So I think definitely for data scientists, we can be in a role where we, you know, coordinate between our teams to see, you know, can we actually standardize and measure things the same way and ensure that we actually get the same numbers all over the place. But as you said, like sometimes it’s not possible, right? Sometimes, you know, there are actually very good reasons why things should be different. 00:34:08.60 [Tim Wilson]: And maybe then we should just use different names for them. 00:34:11.96 [Arik Friedman]: So it’s very clear that, you know, when this team talks about something, it’s not the same thing that the other team talks about. So I think we definitely have a role to play with this. 00:34:20.88 [Tim Wilson]: So can we shift a little bit, because some of this talking about kind of getting trust in the data, and it feels like there is a, there’s a big one we covered, which is like, we’ll get people to trust in the accuracy. We’ve covered two things. If I’m going to mid-show recap, which we don’t really do recaps and I don’t know what the hell I’m doing. But we’ve got kind of the develop and trust your intuition. We’ve got the think about accuracy and precision differently. We haven’t actually covered like, how do you not push bad info out? And the intuition is kind of like one hook into that, but I like the frenetic pace everybody in modern business is working in. There is a drive to get the task completed off my desk over the wall and into the hands, which just like begs for mistakes to be made. And the cost of making a mistake with the query, with the report is of damaging like trust. So like how, what other like ways do we have? We, we, we briefly referenced like time is law, which really goes to the intuition and doing checks, but like, what are other ways to maintain trust in the data from our business partners? How do we prevent ourselves from undermining trust? It’s like a long ramble with a broad question. 00:36:01.00 [Arik Friedman]: Yeah. 00:36:01.36 [Tim Wilson]: I think first thing and something that I definitely would like to see people do 00:36:08.16 [Arik Friedman]: more is just stop and ask, does this make sense? And I don’t think people do that enough. And it’s like at every stage, right? Like, you know, you look at the raw data, does it make sense? Is there anything off there? If you apply methodology, does it make sense in this business context? You get a certain outcome. Does this outcome make sense? So I think this is kind of like a first line of defense. You know, does it make sense? 00:36:38.60 [Tim Wilson]: But aren’t there, there are times, there are times where it makes sense, but it’s actually wrong. Like if you’re like, oh, it made sense. Therefore, I’m going to go full steam ahead. And then you find out later that. Oops. Just because it passed that hurdle, it was plausible, but. 00:36:57.44 [Moe Kiss]: But you’re making the best decision you have with the information available at the time. I feel like the dust doesn’t make sense. Actually, it’s just a like, it’s a spot check, right? I actually think the thing that we need to do better at is. Does it make sense with our stakeholders? Like actually bringing them into some of those checkpoints versus kind of like, does it make sense to me individually? 00:37:21.44 [Arik Friedman]: Absolutely. I mean, I guess like this is the first check that you need to do with yourself before you engage with others. And I think that’s already a checkpoint that I think can probably spot some of the issues. Beyond that, yeah, I mean. Involving other people in this thought process. And like in general, you want to do peer review process for everything. And by the way, I’m coming from a computer science background. My brain is wired as a software engineer. So basically any time that I do anything that involves statistical modeling or math, I will always pull someone else that has like, you know, strong. Math jobs and statistical modeling, you know, hey, check by work. Like, did I actually apply the methodology correctly? Because I know that they think about this in a different way than I do. And they can spot things that I want. 00:38:17.40 [Tim Wilson]: Will the question come up in that context? Because you said like, does the raw data make sense? And from like the EDA of saying, I’ve made my initial query. 00:38:30.12 [Tim Wilson]: Now I’ve got a table of data that has say 25 columns and 150,000 rows. 00:38:40.72 [Tim Wilson]: Do have I gone through and checked that one is 150,000 rows seem like the right number of rows? Like that’s like a saying to like, is the, does the size of the data make sense? But going kind of checking for gaps, distributions, the values, like checking the columns, doing kind of a, doing that step of saying, this should be the right data, but have I actually done, have I checked all the values, all the variables to see if the distributions and values are reasonable? Like not just trust the query, because that adds time and it’s probably a judgment. 00:39:25.36 [Tim Wilson]: It depends on how solid and simple the SQL is. 00:39:29.24 [Tim Wilson]: Does that make sense? And would somebody, if you’re having, if you’re running it by, is this the right model, how likely are they to say, did you double check that the data that you’re 00:39:38.64 [Tim Wilson]: feeding into this model is the data that you think it is? 00:39:43.80 [Arik Friedman]: Yeah. And I think it’s probably both those things. And it depends also if it’s like, is this the first time I’m answering this kind of question or not? And I’ll give us an example. Like there were cases, you know, at some point we worked on, say, internal developer effectiveness metrics, and we tried to understand, you know, flow of work and things like that. And this was the first time we went to calculate these kind of things. So, you know, I worked on this, there was another data scientist 00:40:15.16 [Tim Wilson]: calculating the same metric in parallel, and we got completely different results. 00:40:22.32 [Arik Friedman]: And the thing is that none of us was wrong. I mean, technically, none of us, we didn’t make any mistakes or technical mistakes in the process, but we started working through the calculations step by step and really as we realized that, oh, we actually made different assumptions about what the data meant, and we made different assumptions about what we wanted to measure. So just by working through this process until the numbers matched, it allowed us to align on the assumption and it gave us the confidence that we were actually measuring the right thing. And so, like, it’s definitely not something you will do with any metric or any calculation that you do, but definitely, like, the first time that you do something, it’s good to have, like, this cross check. And once you got these right ones, okay, the next time you already know what you’re doing and you don’t need the same level of, you know, due diligence. 00:41:18.56 [Tim Wilson]: So Julie’s, like, lighting up on the, like, oh, yeah, yeah, assumptions. Like, that’s another thing we haven’t, we’ve talked about it in the past, but the discipline of documenting the assumptions that you made, and it may be, like, the list of assumptions I made were these 20. If I’m having another analyst review it, I’m going to show them all 20. When I present this to my business partner, maybe I even pre-read it, I’m like, these are the three or four that I want to make sure they understood that I made these assumptions. Like, we’ve talked about that with, as a practice of analytics of you are making these decisions and just, like, making an assumption and moving ahead is, like, pretty dangerous actually having in the process a place to write down those assumptions, one from a repeatability, somebody checking your work, but also to actually go back to the business and actually include that in the results. Not the exhaustive, here’s how we made the sausage, but, hey, I just want you to know, going into this, and if that’s your opening slide and they say, well, that’s a bad assumption, then you actually screwed up because you should have done a pre-read with, hey, before we present this, these are like the assumptions we made and here’s why we made them. But how much, how much do you think the business do that? 00:42:41.20 [Moe Kiss]: That they actually, like, I feel like assumptions are included, they’re often, like, at the bottom of the slide, in the text somewhere, and, like, the business just glosses over them. Like, I would love a business stakeholder to be like, and I’ve had that at previous companies where, like, I’ve had very senior folks, like the founders or CEO, be like, I don’t agree with this assumption, like, let’s, let’s debate this, let’s talk it out. But I don’t see, and I would love to see more of that. 00:43:06.72 [Tim Wilson]: I mean, I find it as a way to, you know, on the building trust to not have that in the delivery, but it’s like, I mean, say there’s an executive you’re presenting to, but there’s this business partner that you’re actually collaborating with, it’s a great way to throw in slack, say, hey, is it safe to assume this and have them say, yes, is it safe to assume that? 00:43:29.20 [Tim Wilson]: And then I, I sometimes will put those assumptions, like, front and center. Like, I want to be confident that, and, but, but I probably wouldn’t say I made 00:43:39.36 [Tim Wilson]: the assumption, I would say we made the assumptions. And hey, just so you know, we assumed, you know, that, that holiday has no impact on our bottom line yet. No, that’s, that would be a bad assumption to make. But I mean, it’s not, it’s not just like a CYA of like, throw this in the, in the footnotes, it’s, I think it’s as much of a show that you’re trying to understand the context and the operating environment, which is building trust upstream, which also means that there will be more trust in the output. No? 00:44:14.60 [Julie Hoyer]: Honestly, though, I feel like sometimes that type of documentation and assumptions are like, definition stuff too. I’m with Yuma, like it’s refreshing and encouraging when you are stakeholders key in on it and understand the value of it or want to debate it and can call it out if it’s wrong. But like, honestly, I think a lot of times it is more useful for yourself or others to replicate the work later on, or if you have to build on the current work. 00:44:42.92 [Tim Wilson]: Like, I think sometimes if you’re not kind of obsessed with like documenting 00:44:48.88 [Julie Hoyer]: all of those things, I mean, I’ve been the analyst that’s been given a spreadsheet with random numbers, supposedly where they pulled it. And I have no other definitions or assumptions. And they’re like, oh, recreate this and do it again for the new time period. And you’re like, F my life, I’m never going to get this. And the person that made it originally is gone. So all that to say, like, sometimes if it’s not building trust directly with stakeholders, I do think it helps like maybe the analytics team broadly or the data scientist team broadly, more for the replicability is how I see it. 00:45:26.32 [Arik Friedman]: Yeah, I think definitely like documenting the analysis or the assumptions. And like, that’s probably not the document that you’re going to eventually show as your final result, but it’s probably good to always have like, here’s the technical document with all the details, all the assumptions. So it’s available there. And so you can reproduce this. And yeah, but like referring to most point, I think you have this conversation with your business partner is part of the process, right? Like you get a question. Like, here is how I understand it. This is my interpretation. This is how it translates to the methodology. So you definitely want to be sure that, you know, you’re on the same page and you’re actually answering the same question. 00:46:10.32 [Julie Hoyer]: The other thing I think of too, like if it does get put in the footnotes 00:46:14.92 [Tim Wilson]: of the document or this the presentation, like sent to your business stakeholders 00:46:20.76 [Julie Hoyer]: mode, like I have spent so much time talking to people and on my team or being so worried about myself, what happens when I give this deck to my stakeholders and then they pass it to their stakeholders and they pass it to their stakeholders and then they start asking follow up questions and nobody knows to ask me for the clarification. So like, honestly, I think my safeguard and anxiety is what makes me put a lot in the final documentation in the appendix or in the footnotes. Because I’m like, hey, if this gets passed like three degrees away from me, which again, that’s a win if your work gets passed that far along. 00:46:54.84 [Moe Kiss]: I see this happen all the time with experiment results where like it, you know, a data scientist has pulled something together and then someone like summarizes it and someone summarizes it for a slack message and like these steps further and further away, which look, do I want stakeholders to be able to interpret experiments and communicate it? Absolutely. Lutely. But I think sometimes that’s like just the the the understanding of like the core tradeoffs kind of gets watered down or there’s like different incentives and it just gets really tricky. Arik, I know like you’ve done a lot of thinking about this, about like once the analysis is out, the experiment result is out and you kind of lose control of the narrative, so to speak, like how do you approach that? 00:47:41.76 [Arik Friedman]: Yeah, so it definitely happens like sometimes you you get a chart or something that gets copy pasted and then someone puts it on a slide deck and it’s completely out of context and you lost that. And like one one thing that I try is again, like if you have a documentation 00:48:02.40 [Tim Wilson]: of your analysis, like you should definitely have a link to that. 00:48:06.28 [Arik Friedman]: So if it’s copy pasted, at least the link to the document is still there. And, you know, there’s the footnote in the chart where you can highlight the main details, so it’s definitely a question of a balance, right? Like you don’t want to overload your visuals with all the assumptions and all the details, it’s distracting, but at least have this kind of, you know, selective context or pointer to where the information is. So it kind of like travels together with the visuals. And, you know, sometimes, you know, you just lose control and you have nothing to do about that. But at least you can take some mitigating steps. So, you know, the evidence is there. Like if anyone is really like will want to find this information, there is a way for them to get there. So that’s the least we can do to at least control that. 00:48:59.24 [Tim Wilson]: Yeah. I mean, I think like pointing to like having the footnote of like, what was the what was the right with the source? I mean, you’re well stated that it’s like you don’t want to overload it with all the assumptions, but you want to provide the breadcrumb to say 00:49:12.88 [Tim Wilson]: and the more you can put that proximate to the chart. So they’re not likely, I mean, that’s pretty gross misbehavior 00:49:21.92 [Tim Wilson]: if somebody says, oh, there are footnotes on this slide, but I’m going to pull just the chart and drop it in an email. There’s a little bit of that’s on them if it gets its own legs. If you’re giving it like, hey, this is probably important reference information that should go along with it. I think that’s good. But this, I feel like we could go on for multiple hours, but unfortunately, we need to head to rap or we will lose trust with our audience by having a two hour episode, which when you’re called the analytics power hour, that would be a disconnect between data sources. No, that’s a that’s a stretch. But before we leave, the last thing we like to do on the show always is go around the horn and share a last call, something that might be of interest to our users. And Arc, you’re our first guest. Do you have a last call or maybe a couple last calls you’d like to share? 00:50:26.00 [Arik Friedman]: The first one is, I guess, a classic, the ICLR, like the introduction to statistical learning book, which is actually a free resource. And today, there’s also a version for Python introduction to statistical learning in Python. And, you know, at times when, you know, AI sucks all the attention, I think that actually going back to the basics, to the foundations is, you know, just as important as ever, if not more so. And I know that at least from my experience, even just going over the first chapters of the books, you know, linear regression, it’s like a very practical oriented book. So, yeah, that’s that’s a good recommendation that I usually provide. Like it landed for me. 00:51:12.96 [Tim Wilson]: They have an R version and a Python version with, like, examples on it. 00:51:16.40 [Arik Friedman]: Is that right? Yeah. So there are two versions of the book. The original one was with R. Oh, it’s ISLR and ISL. 00:51:22.44 [Tim Wilson]: And ISLB. Introduction to stuff. OK, gotcha. 00:51:24.68 [Arik Friedman]: And about the available at www.startlearning.com. And like I noted, at least for me, it landed a lot of concepts that I didn’t really get before. So I really recommend that. 00:51:36.80 [Tim Wilson]: And if I’m allowed a second and a second call, 00:51:41.84 [Arik Friedman]: which is maybe more related. So I saw recently an article from Hamel Kussein called Revenge of the Data Scientist, and it’s actually both available as a YouTube talk. It’s a Pi AI talk from March. And he also posted it as a Twitter article. And he actually talks about, I think it’s about, specifically about mindsets, you know, we have the mindset that you go with to, you know, in this agent-first world. And I think that actually his point is that the data scientists approach, their mindset, their core skills, like a exploratory data analysis, metric design, model evaluation, all these things are as critical as ever and try to translate really well to this world. So I definitely recommend giving this a read. And also Hamel Kussein and Trash and Carr had a terrific episode about AI evils in Lenny’s podcast. So that’s a great resource as well. 00:52:41.64 [Tim Wilson]: Awesome. So that was three, really, sort of. But those all sound amazing. I feel like I’ve read the first three chapters of like more. Those are the books I’m most likely to abandon, but still get a lot of value because it’s like chapter four, where I start to I’m like, what, we’re at the area under the curve. I’m like, oh, boy, oh, boy, the equations are getting pretty in-depth here. So I kind of want to check those out. Julie, what’s your last call? 00:53:12.40 [Julie Hoyer]: Well, my last call is accurate and precise. And I’m feeling like maybe I have called this one out before. But you know what? It’s such a good one that I’m just going to say it again. It’s called the Moenarch app. And I’ve actually been using it now for quite a while for my own family budgeting and financial like tool for the family spenders. And it’s really nice because you can connect everything directly into it. So on top of having all your different credit cards, bank accounts, you get really nice cash flow visuals. You can recategorize like any of your expenses that come through. And it will notify you like, hey, this is a recurring one or hey, this is one that’s not categorized. You want to come in and quickly do it. But budgeting is not easy to do on your own. And I feel like this is the first app and different thing I’ve tried that I can actually keep up with it. I can quickly get to it. You know, there’s no delay like I spent. It shows there by paycheck hits. It shows there. So it’s been awesome. You can also set a lot of your budgets and goals in the app as well. So like for all your different categories, you can customize categories or not. And then you can say, you know, I’m looking to spend X amount each month in each category. And it just, I feel like makes it a lot easier to actually live out the financial plan and budget that we’re going for. So check it out if you’re needing a tool. 00:54:44.20 [Tim Wilson]: Have you figured out a way to turn off the turn off the net worth? The words because because there are periods where I want to not look at that 00:54:51.48 [Tim Wilson]: when the yeah, I’m a monarch user. 00:54:56.44 [Tim Wilson]: So I’m a fan. 00:54:57.48 [Julie Hoyer]: But you are. So you guys, it must be good if Tim’s into it. If he approves of the visuals and the tools. 00:55:03.24 [Tim Wilson]: Yeah, judge, judge your trust in the source. Moe, what’s your last call? 00:55:11.20 [Moe Kiss]: I know we talk about her a lot. But Cassie, Cassie has her cough has a really great new. I mean, it’s just her regular newsletter, but she’s doing a couple of back to back newsletters on why the vibe coding will bite you. And here is exactly where and she’s talked through a few like text scenarios of where things have gone really wrong. Like prod systems being completely white, things like that. So definitely go have a listen to that. But I think the thing like the real takeaway is that the stories are all about the same thing, which is just like misplaced trust and the speed at which it computes. And so it’s not like the nobody got hacked. AI didn’t go rogue. It’s just like people let their guard down. And I think the thing that’s been on my mind the most, which she’s so she’s just so damn articulate, but it’s like expertise won’t save you guardrails might. And I think guardrails is the topic that just keeps looping on my brain at the moment. And so, yeah, I’ve really enjoyed that newsletter and she’s got a couple more coming out on the same topic. So check it out. 00:56:17.64 [Tim Wilson]: I think that series like motivated me to dive into some new vibe coding project specifically. So far, I’m safe. I haven’t crashed the podcast because that’s where most of them happen. Starting to wonder if Riverside, our podcast recording app might actually be leaning a little too much on vibe coding is it’s having some of the joys it’s been bringing us of late. But so I’m going to pander to mo here a little bit because with the new season of choiceology that Katie milkman came out with. Um, one of the things she has is this checklist, which is this mapping of now she said she had a couple of undergraduate students do it. And that means it’s like in a PDF for some insane reason, but she’s basically gone through and looked at like, what are the different topics like attribution bias or Dunning Kruger or left digit bias, like the stuff that her episodes cover, this kind of reverses it. And it’s a guide to broken down by these are all the topics of kind of 00:57:20.56 [Tim Wilson]: cognitive biases, um, then which are the episodes you can actually listen to. 00:57:26.52 [Tim Wilson]: So if you’re not a, um, Katie milkman choiceology completionist, like I am, but you’re like, I wonder if she’s ever had an episode about, um, you know, mean reversion neglect, then this little guide will pop you to it. It is comically in a PDF, which I’m like, this is great. You guys really work to format this thing to one page. And she’s about to start a new season, which means this, this of all things that should be a vibe coded website where it gets updated and maintained. This is it, but you know what the undergrads, they’ll learn. That’s how it was scoped academia. They’re going to do their little thing. So, um, with that, um, our thanks so much for coming on. I feel like this is a case where we actually have like the show prep documents that have like even more gold in it that we were not able to get to. So, so we will have a lot of fun with that content our own ourselves. We may figure out how to bring you back for more of that. So thanks so much for coming on. 00:58:31.64 [Arik Friedman]: Thank you very much. Awesome. 00:58:33.76 [Tim Wilson]: So if you, uh, listeners, um, we love to hear from you. So if you’d love to have you leave a review or rating on whatever platform you listen to us on, if you’d like a free sticker of the podcast, uh, you can go to analytics hour.io and request one. I will, if you’ve gotten this far in the episode and you think, wow, that was a smooth conversation and these guys are professional, I will just call out now that we have dealt with, uh, tornado warning. They led to a power outage and two young children and a dog sheltering in place and with one co-host, we’ve dealt with a busted internet, um, that has been busted for the entire episode. But of course the repair team showed up for that during the episode. Um, and we’ve dealt with various cases of people dropping off and returning and not even realizing that we were still recording the show. So I encourage you to stick around for the outtakes because there might be some real doozies in those. Um, I would like to Tony, please leave this in. Thank you so much. 00:59:39.92 [Tim Wilson]: Cause if you pulled this thing together, you’re a good on your mate. 00:59:45.28 [Tim Wilson]: Uh, so it’s been fun. It’s been a fun discussion. Uh, we’ve been at this for four hours to get this one hour pulled together. No, it hadn’t been quite that bad, but we would love to hear from you. We’d love to hear if you thought that the edit was pretty smooth. If you’ve got your own thoughts on how to build, maintain, recover, trust, uh, you can reach out to us on LinkedIn. You can reach out to us on the measure slack. You can just send us an email at contact at analyticshour.io. So for Julie, for Moe, for all of the conspiring mother nature and construction projects that tried to not allow us to record this show about trust and accuracy 01:00:31.72 [Tim Wilson]: and precision, keep analyzing. 01:00:34.80 [Announcer]: Thanks for listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at, at analytics hour on the web at analyticshour.io, our LinkedIn 01:00:46.24 [Tim Wilson]: group and the measure chat slack group music for the podcast by Josh Crowhurst. 01:00:52.36 [Announcer]: Those smart guys wanted to fit in. So they made up a term called analytics. Analytics don’t work. 01:00:59.04 [Charles Barkley]: Do the analytics say go for it? No matter who’s going for it. So if you and I want to feel the analytics, they go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. Fuck my life. 01:01:15.48 [Moe Kiss]: So the work that’s being done out the front of my house. Knocked the NBN cable out. 01:01:24.48 [Tim Wilson]: I’m a rap now. 01:01:26.12 [Moe Kiss]: Oh, shit. Were we still recording? Oh, fuck my life. 01:01:33.84 [Tim Wilson]: I had to smoke this, but I was, I was trying to make so many notes that I, uh, on other aspects that I doubted. 01:01:40.56 [Moe Kiss]: There’s a lot going on today. Oh, my God. Well, you gave me so much. 01:01:46.56 [Tim Wilson]: I just, there was no writing. 01:01:48.88 [Julie Hoyer]: Nobody has the heart to tell you if I’m like, well, then the second one, one is like, so can I wrap? Oh, guys. All right, guys. 01:02:03.32 [Arik Friedman]: I definitely want to take Tony in advance because like, yeah, like, that’s probably some work to do here. Yeah. 01:02:16.12 [Julie Hoyer]: My hot mic was the least of our concerns. I don’t know if you would have thought, Jesus. 01:02:21.08 [Tim Wilson]: Yeah, but I mean, I think it’s, I think it’s going to come together well. 01:02:26.04 [Tim Wilson]: And my mid-show recap was like me organizing my thoughts because I was like, 01:02:29.76 [Moe Kiss]: I think we’re actually hitting on some, I actually feel like Arik, we need like another two hours with you because there’s so much stuff here. It’s such gold. 01:02:39.40 [Julie Hoyer]: Seriously. Yeah. You had so much in the show prep doc that I was like, oh, I want to talk about that. 01:02:45.32 [Moe Kiss]: Oh, and this is why I was like, I knew that the two of you would really like 01:02:48.56 [Tim Wilson]: Arik because like you’re the same with the very good at prep and 01:02:53.80 [Moe Kiss]: organization and all those things. I literally bought a new microphone and it’s still. Why have I still got this shitty one that doesn’t even have a proper stand and it still works great and everyone keeps buying all these fancy ones. It sucks. 01:03:08.16 [Julie Hoyer]: I got to return my hundred dollar one, I guess, and try a twenty dollar one. Maybe I need it to be less sensitive. 01:03:14.64 [Tim Wilson]: I think the show might need to buy you an audio interface. I think it’s I don’t don’t don’t don’t make any moves. Don’t do anything drastic. 01:03:25.12 [Moe Kiss]: Yeah, the audio interfaces and changing microphones. 01:03:33.04 [Tim Wilson]: It is because it’s moving where the preamp is. It moves it into its separate. It moves it from a little thing, crappy thing in the microphone. It then takes it out and puts it in a dedicated box. 01:03:44.40 [Tim Wilson]: OK. No, this is so above my head. 01:03:48.20 [Julie Hoyer]: I just want to buy a microphone that works. 01:03:59.24 [Tim Wilson]: Rock flag and accuracy versus precision. The post #296: Avoiding Major Oopsies: Twyman’s Law, Intuition, and Valuing Accuracy Over Precision appeared first on The Analytics Power Hour: Data and Analytics Podcast.

April 14, 20261 hr 9 min

#295: Research and Analytics: the Peanut Butter and Chocolate of Data?

Research and analytics: are they more like peanut butter and chocolate, or more like oil and water? On this episode, we dig into the surprisingly common (and surprisingly unfortunate) divide between these two disciplines with Stefanie Zammit, Global Director of Analytics and Insights at Bang & Olufsen. Stefanie has spent her career bridging the qual and quant worlds, and she makes a compelling case that the best insights come from putting both methodologies to work on the same business problems. From the “never ask a survey question you already have the answer to” rule to why personas are usually terrible (spoiler: it’s not the clustering, it’s the storytelling), we explore how organizations can break down the silos between research and analytics teams. Turns out, the fear of the unknown and a bunch of fancy terminology might be keeping us from some pretty powerful insights. Also, apparently 100% soundproof rooms are absolutely terrifying. This episode is brought to you by Prism from Ask-Y—your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show The Practice of Market Research: An Introduction Training crucial leadership skills through serious games The Bang & Olufsen Factory Tour Women in Research (WIRe) Women in Product How Quantum Computing Works How to Make Sense of AI Photo by Vardan Papikyan on Unsplash Episode Transcript00:00:00.00 [Announcer]: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. 00:00:13.24 [Michael Helbling]: Hi everybody, welcome to the Analytics Power Hour and this is episode 295. You know a common phrase we hear in our industry is that the data tells us what happened, but not necessarily why. And by and large that’s true. We keep getting better at inference and some patterns of data are pretty well understood in terms of their meaning, but still there is simply something so compelling about observing how people interact with the things we’ve built, websites, products, etc. Personally I remember what an eye-opening experience it was for me 15 years ago, the first time I was sitting in a usability lab watching from behind a one-way mirror as people use the website that I measured every day with my digital data. So we wanted to talk about it, get into the topic bridging between research and more traditional analytics. And I want to introduce my co-hosts, Val Kroll. Welcome. 00:01:12.56 [Val Kroll]: Hello. 00:01:13.56 [Announcer]: Hello. 00:01:14.56 [Michael Helbling]: And I know this is a special topic for you because if you’re background into customer research. 00:01:20.08 [Val Kroll]: Oh my gosh, doing back flips. Very excited for this. 00:01:22.08 [Michael Helbling]: Yeah, I’m excited too. And Julie Hoyer, welcome. Hello. Hi. Have you done much like market research or customer research? 00:01:32.04 [Julie Hoyer]: Not myself, but I have gotten a chance to like utilize the outputs of some of those studies, which has been nice. So I’m really excited to talk about it more. 00:01:40.36 [Michael Helbling]: Yeah. Excellent. All right. And I’m Michael Helbling. And to bring additional expertise to this topic, I’m pleased to introduce our guest, Stefanie Zammit. She is the Global Director of Analytics and Insights at Bang & Olufsen. Prior to that, she led research and analytics teams at companies like Starbucks and Marks and Spencer’s. She’s worked both as a consultant in the space for many years as well. And today she is our guest. Welcome to the show, Stefanie. 00:02:05.24 [Stefanie Zammit]: Hello. 00:02:06.24 [Michael Helbling]: Happy to be here. Awesome. We’re so glad to have you. And I think this is a topic that while we cover sort of like data and analytics, research is one of those things that we really like. And so we’re really excited when we met you to sort of dig into this topic more. But to kind of catch everybody up to speed, I thought it’d be great to kick off with just you explain a little bit about your background and career and kind of how it bridged these two things and sort of, you know, what your journey has been across research and analytics. 00:02:40.40 [Stefanie Zammit]: Yeah. Absolutely. I’m very much from a pure hardcore research background. That’s where I started my career too many years ago. I started as, actually, my first job was at university. I didn’t even know what research was. I took a part-time job doing the national student survey. That’s like a thing here in the UK. It’s how the university rankings are put together. And when I graduated, I think a lot of researchers would have a similar story to this, that they sort of ended up accidentally in the field. And so I graduated during a recession, dark times, and I was desperate. And I thought back to that part-time job I had at university and like, what is that? Like, is that an industry? Is that a thing I could do forever? So I did some research and I was in the UK at the time where, luckily, there’s an amazing research industry here. There’s so many great consultancies. It’s a thriving industry. I took my first job at a company called Quadrangle, which was a management consultancy that leaned very heavily on their own in-house research function. And it was amazing, an eye-opening, and I immediately fell in love with so many aspects of it. I then went to Ipsos, which is one of the big five. That’s where I was like, I need some hardcore research skills. I need to learn about statistics and hit me with the heavy quant stuff, go to a big powerhouse. So I was there for a couple of years. And then after that, I built my own. I co-founded a research consultancy at this time I was in the Middle East, was a company called Intelligence Qatar, which is still very much there today, although sadly, I don’t get to play a part in it anymore. And I think that was where previously at research consultancies, they really divide the teams up. So you had your market researchers, and then you had your analytics department with all the smart people, the statisticians were all there, then you’d have your fieldwork teams, your data processing teams. And it was very siloed, and even market research was pretty siloed. You’d have your qual team, and then your quant team, and something that was really hard for me as I progressed agency side, you’d have recruiters say to you, like, do you want a qual role or a quant role? And I struggled so hard to give them the answer because I genuinely loved both. And my favorite projects were the multi-phase projects where you’d get the best of both. But I was also super kind of looking over the shoulder of my analytics colleagues and statistician colleagues, what are you guys up to? What are you doing? So I always naturally was interested in the entire spectrum. And then when I had my own agency, again, really investing in the analytics side, as well as the research side just brought me closer to those worlds. And then finally went client side, I thought, all right, I had enough of this consulting game, joined Marks and Spencer’s. And I was really lucky that they had research analytics together in one department. That’s all I’ve known because it was like that in that first company. And we had a great leader at the time that was very adamant that the best deliverables are worked on by both of these teams. So I took that to Starbucks as well, where again, the organization is huge in Starbucks, you’re looking at around 250 people, but they are all in the same department together, research and analytics, if subteams, at least it’s still in the same family. So it’s something that I’ve been so passionate about today, and I think really set me up for success to do what I do now, which is to lead both the analytics function and the market research function in one team. 00:06:09.76 [Val Kroll]: I love that. The one thing that you mentioned there about not being able to pick which you liked better, the qual or the quant in your favorite, where the multi-phase projects that start with the qual and then lead into quant, I feel like that’s like a little bit novel, like a little bit inside baseball for researchers. Can you describe what that is because I have a follow up question after you talk about that a little bit, but just kind of describe like why someone would have like a multi-phased approach to like their research project? 00:06:40.60 [Announcer]: Absolutely. 00:06:42.20 [Stefanie Zammit]: So every methodology has its benefit. Qualitative research is where you start when you don’t know much about a subject and you need to explore. It’s very exploratory. You are using it to figure out where your hypotheses even are. You might have one or two hypotheses, but they’re fluffy and you need to explore the topic more. So qualitative is where you do that in this kind of limitless way of a very open data generation phase. And then you need to validate because you’ve spoken to like, I don’t know, max 40 people if you’re doing a really big qualitative project. So you want to validate that and you need some statistics and some numbers behind it. So you want to do your quant. So I have my hypotheses now, I’ll run a survey to measure and size those truths and see how statistically significant they are. And methodologically, these require two different expertise because qualitatively you’re trained in moderation, projective techniques, how to read between the lines, how to read people’s faces and emotions and hear what they’re not saying as well as what they are saying. There’s also a much more kind of deep psychology to interpreting those insights because, again, you’re reading between the lines. Quantitative, you need to know about driver’s analysis and cluster analysis. In fact, you need to know all of the statistical models that give you the derived insights that are so, so valuable from a survey. So they are usually kept separate. And I think that’s a shame because the best projects are the ones that do both of these things for a really strong final insight. 00:08:21.36 [Val Kroll]: Yeah, that was very well explained. And I remember in my market research days, a lot of times, if you think about if you were to do a survey, you have to, a quantitative survey, and you’re picking the list of options that someone is going to select that is the right answer to the question for them, sometimes that list isn’t always as clear, like what should belong in it, even if it’s a list of competitors. A lot of times, clients will think about in-category competitors, about who your competitors are for an alcohol brand. But that same dollar could be spent on other things that are out of category competition. And so you can use QAL to help explore to figure out what even is the list, because you could miss so much if you start just with a quant without going broad first, to be exploratory, like exactly, except Stefanie, to figure out what your hypothesis is. And the reason I want to dig into this a little bit is because this is one of the ways that I love helping illustrate or describe, especially to people describe, to people in the analytics, some of the value of bringing these worlds together. Because it’s not just using one single methodology or tool. It helps you illuminate a different part of your question or your process. And so how those two things come together very naturally inside of research is one of the ways you can kind of illustrate the coming together beyond just the research or the direct consumer or B2B context. 00:09:49.28 [Stefanie Zammit]: Exactly. And I think from a research-only perspective, I was already at a very early stage so powered up by the idea that if you put these two together, you’re getting better insights because you’re starting broad and then you’re getting specific with the quant. And I took that into as I gained more seniority in my career and started working, especially in-house, where you have colleagues in other disciplines, which is all data and insight. I took that into that as well to say, well, how can analytics be part of this? And why are we asking questions and surveys that we already have the answer to? That seems like a huge waste of time. How can we be using all these disciplines to get the best insight? And all of this ultimately comes down to a passion to chase the best insight, right? Methodology should be irrelevant to it’s not about the journey, it’s about where you end up. And I think having that crystallized in my mind from the very beginning really helped me see the world in the way that I see it now, that who cares what you’re trained in. At the end of the day, we use the best method to get the best insight. And it doesn’t matter whether that’s in this team or that team. 00:11:01.92 [Julie Hoyer]: And I feel like you are one of the more rare data leaders out there that recognize it has to be problem focused instead of leaders coming to their teams with ordering a solution. They have a problem, but they’re not always communicating that to the teams that are going to help service them. They’re kind of coming with, well, I want you to pull these numbers or ask these questions type thing instead of to your point just anchoring on, I don’t care how you do it, you guys are the experts in that part. But what I’m facing is problem X and I’m looking for ways to solve it. And then letting people get creative with how do they maybe partner together to get them the best solution? I feel like we just run into that at kind of like all levels of the business. We’ve all had different experiences with like trying to overcome that hurdle. So it’s really refreshing to hear you so eloquently like talk about it in the way you frame it. 00:12:06.64 [Stefanie Zammit]: And 100%, that’s so normal everywhere. I’ve experienced it everywhere that you have stakeholders coming to you. We want to do some quality. We want to run a survey. I need a dashboard. I need they they’re very prescriptive. And I think it’s part of our job as insights folks, irrespective of our training to say, whoa, whoa, hold on there. Let me understand what you’re trying to do. What decision are you trying to make? The answer might actually not be in a dashboard. It might be in a piece of custom analytics or it might be. So our consulting work is a big part of this job to find the best methodology for the best answer. We can’t expect our stakeholders to know what that is, although they’re very welcome to make suggestions, of course. But it’s a broad broad spectrum of tools that we could use. 00:12:54.40 [Julie Hoyer]: Yeah, absolutely. And one of the questions I’ve been dying to ask you is, why do you think, and Val, I’d be interested in your take, too, because you’ve kind of lived in both of these worlds as well. Like, historically, why don’t research teams and analytics teams always play together at all? Or if they’re playing together, they don’t always play nicely together? Good question. Get into it. 00:13:20.64 [Val Kroll]: Stefanie, you have to go first. 00:13:22.72 [Stefanie Zammit]: It’s a great question. And I think the answer is fear. I think that there is the fear of the unknown. And there is an assumption that the other world is so mysterious and so different to our world. We don’t understand each other. It’s literally a form of othering within teams, within departments. And it’s a fear I myself had. I could never keep up with these data scientists and, oh, they’re just so smart. And they understand all these things in a way that I never could. And then you start working together. And you see that you actually have more in common than you do have differences. And they see it, too. They understand, they can learn from the research process and understand, oh, hey, I thought research was just qual. I don’t know if you’ve ever heard that, any of you guys, especially Val, that the assumption that research equals qual, that’s the qual work. And so I just did a survey with 8,000 people in this hardcore conjoint statistical model. You can’t call that qual. But there isn’t an understanding. There isn’t enough knowledge. And everyone assumes that the other side of the coin is so different. It’s a whole different world. And I think it comes from the way that agencies are set up and departments are set up to separate these skills when actually they’re stronger together. 00:14:43.36 [Tim Wilson]: Michael, why does every quick question come with a 20-minute origin story? Well, that’s because our metrics have, I don’t know, lore. 00:14:53.76 [Michael Helbling]: Conversions might mean three different things depending on who’s presenting and how close we are to the next quarterly board meeting. 00:15:00.72 [Tim Wilson]: And I mean, every time you switch tools, you have to re-explain the lore like you’re residing ancient prophecy. 00:15:07.20 [Michael Helbling]: On the seventh day of Q3, the trekking broke and lo, that metric was doubled for July. 00:15:12.80 [Tim Wilson]: That’s why we’re excited about askwide.ai in Prism, because it has memory that actually remembers. You don’t have to repeat the lore. 00:15:21.76 [Michael Helbling]: Yeah, the jam system keeps context across sessions what your org means by revenue 00:15:27.84 [Announcer]: or conversions, which tables the source of truth. 00:15:31.36 [Michael Helbling]: And the weird exception, you’ll always forget until it’s too late. 00:15:34.56 [Tim Wilson]: Plus, you can save your best workflows as skills. Portable expertise you can actually reuse like a human. 00:15:42.88 [Michael Helbling]: So I don’t have to manually normalize UTMs or fix or tweak that GA4 channel grouping or deduplicate leads without breaking down into tears. 00:15:57.04 [Tim Wilson]: Yeah, so instead of rebuilding the same process like every week. 00:16:01.28 [Michael Helbling]: Yeah, I guess I just run the skill and go about the rest of my day. Kind of happy. 00:16:07.84 [Tim Wilson]: So one end, go to ask-wide.ai and join the waitlist. It’s in beta, but you can get in on the ground floor. 00:16:14.96 [Michael Helbling]: Yeah, and if you use code APH, you’ll get pushed to the top of the waitlist. That’s ask-the-letter-why.ai and use code APH. 00:16:26.16 [Announcer]: All right, let’s get back to the show. 00:16:28.56 [Val Kroll]: And where you landed on that, I think is one of the drivers that I see. The drivers that I see in my mind is if you think about how these two disciplines grew up in the world, research used to live within marketing. It used to be marketing research. And so we would sit within marketing teams. All of my clients back in the day were CMOs or reported up to the CMO. My first web analytics job, I was in IT and it was very technology heavy. It was about the tools and the way the data was collected. And like surveys, people think like, oh, like pen and paper, like mailing surveys or someone chasing you down at a mall with a clipboard. Like, would you like to take a survey? There’s so much more to that, like with panels and different ways that you can contact people nowadays. So I think- Limes have moved on. Yes, yes. There has been an evolution, yes. And so I think it’s just kind of been just how it grew up, 00:17:25.04 [Announcer]: was kind of thought of differently, budgeted for differently, like research, 00:17:29.52 [Val Kroll]: I think has always has this like, wrap a little bit. I’m interested in your thoughts on this too, Stefanie. That like a lot of times it can be like costly, like not just dollars, but time. So that it takes a long time to, you know, every, you know, we only do the brand tracking study once a year because it’s such a big, you know, piece of research or like it’s actually not valuable to keep track of that on a more frequent basis where a lot of the costs from some of the other analytics practices or areas are some more hidden costs because they’re in the technology or actually the human solutions. And so especially when we’re talking about like in-house, I think that it’s just budgeted for differently. And so people aren’t really like connecting the dots. 00:18:08.16 [Announcer]: But I think the organizations where you can break down those silos, 00:18:11.36 [Val Kroll]: because I’ve actually never worked for a client that’s had the setup that you’re talking about, Stefanie, which would be like, oh, Nirvana to have them like coming together. But I always found ourselves making the suggestion about like, did you talk to that team? And they’re like, who? And they’re like, well, I was scrolling through your active directory and I found this person with this title, like you should reach out to them to see if they can help us. But yeah, so that’s kind of what I think is like part of the rationale that I think that I do hope that this is like a movement though, like this evolution towards thinking more flexibly about the methodologies and what’s the right fit for the question at hand and what’s going to serve the business best. 00:18:50.80 [Stefanie Zammit]: Yes. And I should add that the Nirvana, I may be making it sound more Nirvana-like than it actually is. I mean, again, when you look at huge organizations like Starbucks, which it’s just such a math, there’s thousands of people at head office. Although we were all in a department together, the reality is that the silos were still very strong. At the time when I first joined Starbucks, I was in service to the loyalty team. So the rewards program and the app, you know, and how our customers use the app, et cetera. And I realized that I was trying to serve these stakeholders with insights about app usage and loyalty program behaviors and all the rest of it. Meanwhile, you had folks in analytics who were also answering questions for the same stakeholders. And there was just such a clear overlap in, we’re both talking about behavioral data, we’re both talking about what the client wants, the customer wants and needs. And so what we did was we formed a little, within the department, a little community of people who are in service to this stakeholder group, irrespective of where in the mega data analytics and insights department you are, we come together and we talk about all our projects. We formed a little, I don’t want to call it a Sierco because I hate the word Sierco, very anti-allergic. Oh my God. But just like a little forum of round robin, what’s everyone working on? And then we can say, oh, you’re doing that. I’ve got a survey coming up which directly overlaps with the objectives of what you’re looking at. But I can tell you why and you’re measuring what. So why don’t we put them together and, hey, our stakeholder will get something that’s more complete and less confusing rather than 10 different reports that all overlap, but the actionability is lost because we’re pointing in different directions on similar 00:20:43.12 [Michael Helbling]: and yet not quite the same topics. That’s crazy because I literally have built something almost identical but coming from the data side to the research side at a company I used to work at, which was forming this little team that we met and we’re saying, okay, let’s get together and get all of us together so we have a coherent story. And it’s always stuck with me that why is the organization having to be managed sort of bottom up in that regard when the reality is the structure or the layout of the org should be thought through to enable that kind of capability from the very top. And it’s just one of those things that sort of sticks out as a sore thumb. And I don’t know if I have prescriptions for that, but I will say it’s like Val, you mentioned we walk into a client, you’re kind of looking around like where’s your research team and why aren’t we talking to them too, which I think is really apt. But also like certain companies like you’re going to do analytics work and there is no research team or nothing named as such. And it’s sort of like a lost function or a missing function. And it might be kind of Julie to your earlier point, the leaders of the org just sort of think they’ve got it figured out so they don’t need someone to sort of like think about what the customer actually thinks because they’re thinking for the customer, if you will, not a great plan. But anyways, I’m just curious, Stefanie, because you’ve done some consulting in this space as well, like how do companies sort of jump the chasm first from just not even having a concept for doing research like this? Because everybody’s got analytics, like we’ve all got snowflake and data bricks or something running the back end of all of our data, but a lot of companies have zero going on in terms of like either customer or market research. 00:22:45.76 [Stefanie Zammit]: And if they do, they outsource it to agencies. So they would do a one-off project that an external company will run. And this is where you write it as complexity, because it’s rare that a company would have a full function in-house research team, because the manpower that you need on a per project level is huge, right? We’re talking thousands of people like, okay, not thousands, but at least 50 going out, interviewing in different countries and then your data processing and then your statistics. And then so the per project value for money of that much manpower is just not worth it. So they outsource to agencies who have economies of scale. Those agencies then in turn don’t think to ask, hey, do you have a data team? Do you have an analytics team? They sell analytics. Why would they say, hey, we’re going to build a segmentation for you. We’ll do all the research, but we’ll hand over to your in-house analytics and they can do the clusters. Like they’re never going to say that. They’ll be like, yeah, we do end to end, you know. But so I think that’s the challenge. And what I would advise organizations to do, to mitigate against that, is to have even just one person. At Bang & Offsen, we have one person. He’s a superhero. He’s a one-man research department. One person to coordinate all research projects and you can still use vendors, but you have an internal knowledge bank being built and internal consistency even. And then that person knows to work with analytics and to blend the two together while also getting the economies of scale from the agencies. 00:24:21.92 [Val Kroll]: Closing the chasm, I want to spend a little bit more time on this and your thought of making sure you have someone to represent that perspective versus like, does anyone want to add any new questions to this year’s tracker? Like it has to be so much deeper than that for it to be meaningful. But my first boss, when I was in market research, she grew up, Lynn Bartos, if you’re listening, she grew up at Burke. And so she was like hardcore, like you were saying, like having those skills. And one of the things that she always talked about was that she spent two weeks touring all the different departments, like someone who sat within, you know, data processing, someone who sat within coding for all the open-ended responses to get an appreciation for the operations and like how the sausage was made, because that makes you smarter when you request your banners or when you think about like, you know, how do I develop the questions to get me to a driver’s analysis or things like that. And so we did that ourselves on her team. And I really had an appreciation for that. Do you think that the closing the chasm is cross-training people so that they have like a better appreciation for each other’s skills, like when they do both exist in-house? Or is it more, you know, pizza lunches, like dog and pony show of like results? Or because I love what you talked about of having the little non-steerco-steerco, that’s aligned to a stakeholder group, like I would have never thought of that. But I’m just wondering if there’s some other things, unfortunately, like you were also saying, Michael bottoms up that people could do to kind of close this chasm between the team. Like what things would you recommend to listeners for who have an interest on the other side? Absolutely, 00:26:03.12 [Stefanie Zammit]: yes. So for anyone listening who’s not managing a team but is in one of these roles, whether you’re in data science, you know, data engineering, research, whatever it might be that touches on data, I’d really recommend the best that that person can do for their own career is to have a natural curiosity for the methodologies and the training that the others in the department have. And I always say to people in my team, if you find yourself in a conversation where you have no idea what people are talking about, or it feels like scary or just very different to what your expertise is, that is where you will learn that you should lean into those conversations. We should go out of our way to understand, not in an annoying way, like, hey, dude, what are you doing every day? But just to have a natural curiosity of how does your work fit with my work? And that’s not even just for analytics and insights or for data. That should be for everyone in a corporate job working for a brand where we serve our customers. We should all be understanding how do our worlds fit together for the customer. But especially within data, because data is the customer, we represent the customer, have that curiosity. And for anyone who’s listening who’s in a leadership role, then yes, I really recommend fostering that within your teams, 00:27:24.16 [Announcer]: that natural curiosity, and getting people to take a moment to question if anyone else in a data 00:27:32.24 [Stefanie Zammit]: related role can contribute to the project to add further insights. So would the data science team know anything here? Maybe they don’t have a deliverable, but maybe from their investigations or their work, they would have context that would add valuable insight to my work. Would the research team maybe have had a project about this? Maybe not, but again, the data folks are in, you know, they’re hands dirty in the data every day. The amount of information that they process and that they gain exposure to, which is not reported ever, is huge, right? We could never report all the facts, but it’s there, it’s in their heads and in their experience. So it might not be reported anywhere, but that’s a good person to talk to because they would have context. Same with the researchers, you know, they’re out conducting intercepts, ethnography focus groups, not everything makes it into the final report. But if you sit down and talk to each other, you realize, oh, yeah, I know something about that. I remember seeing something about that. I have a good quote that brings what you’re doing to life, whatever it might be. See, I really encourage curiosity. My first job, I started in QUAL, actually, which is like the furthest away from data science and analytics. And I was so scared to move toward QUANT. And I got reassigned to a tracker. So I went from like the QUAL team, you know, ultra open to a tracker, like 100% only ever working on this one tracker. And at the time, I was so grumpy about it. I was just like, this sucks. Like, 00:29:03.20 [Announcer]: where’s the creativity? You know, where’s the art? But actually, it was the best thing that 00:29:09.52 [Stefanie Zammit]: could have happened to me, because I didn’t know anything about tracking. And I, you know, I would have been pigeonholed if I’d have just followed my natural heart. It forced me to learn about a different world. And then I realized, actually, this is interesting. Actually, QUANT. QUANT is interesting. Look at that. Who would have thought? Tracking is actually interesting. There’s insights here. But unless you’re kind of forced into it, or at least when you’re young, you need to be forced a little bit, it’s difficult to just naturally expect that you would find these other worlds interesting. But I guarantee if you really, you know, scratch at that or, you know, peek into those boxes, you will find a lot of very interesting things that will help your own role. 00:29:50.88 [Julie Hoyer]: You touched on this a little bit earlier too, Stefanie. And I’m curious. So when you’re leading your team and you have this great point of view on how these things can work 00:30:01.84 [Announcer]: together. And obviously, we talked about trying to encourage the curiosity of everyone on your team. 00:30:08.00 [Julie Hoyer]: But are there some more like formal processes that you’ve also put in place for your team, or how you guys pick up projects or execute projects with your stakeholders that really help these come together in the best way possible? Because you mentioned earlier, like the, you know, starting with the, the qual and then you get a hypothesis, then you follow up with quant. So I wanted to like dive a little deeper in that area and hear what some actual harder, like boundaries or processes you utilize to help. 00:30:39.12 [Stefanie Zammit]: Yeah, 100%. And there’s so many examples, I’m going to try to stay focused here. So first rule of thumb, no survey asks a question that we already have the answer to. And the only way to know that is to go and talk to the data team and know what we have the answers to. So just as a rule of thumb, and even from our customer experiences, right, customers shouldn’t be telling us like their demographics, if they’re signed up to us and, you know, we should know who they are. Second rule of thumb, every research project, a sub sample, even if it’s not in your objectives to interview clients, even if you’re, say, you’re doing a new customer acquisition piece. And, you know, you want to go out to like a purely external sample, you don’t want any internal sample. Even if that’s the case, a subsection of the survey respondees should be from internal customer, known customer sample. And especially if you have a segmentation or you have certain key questions or certain key profile, you know, data points that you want, you want to continue building that knowledge internally, use your surveys in that way. So for example, we’re doing segmentation work now, we’ve designed, we’re starting, we actually started with data. So what do we know about our client? To what extent can we profile them before it becomes a mystery? Right. That’s the point at which now we take it to research, we fill in the blanks with research, but we, we interview as much of our own customers as we can. So that then once the survey is complete, we bring all that data enrichment back in house, it’s tagged to non customers. And we can use both worlds to create the segmentation. So you have your attitudinal stuff, you have your profiling that you would never be able to know just from data, and you have your behavioral data as well. And we have this amazing analytics team, they can do the segmentation. We don’t have to use an agency for the entire thing. So we’ve saved money, you know, we know where we can stop the agency to make use of our internal skills. Now we’ve got money left over for a different project. Great. Let’s go do some call with these segments and get to know them, bring them to life, do ethnography, take video. Now when we go out to our stakeholders, we’ve got our segmentation, it’s attitudinal and, and behavioral. We’ve got all this great qualitative bringing them to life. Right. So, and this goes for every project. So you’re doing a piece of work with drivers analysis. Okay, great. We might do a survey, you know, you do your usual like drivers analysis. But then let’s say, okay, how can we, some of that survey was internal customer sample, how do we bring that back to the analytics team and say, well, we want to grow. So let’s identify these people and do an internal business drivers analysis based on the learnings from the survey to see if we can replicate those drivers in our data. Lo and behold, you’ve got an internal business drivers analysis that you can now track because it’s in our data. So how if we move the needle, did we actually grow, we can actually say, yes, that was that insight was successful. And we did actually grow. There’s many more examples, but wherever we can put both worlds together in a single project, I absolutely recommend we do. The other thing is communities. If, if you’re a business that’s lucky enough to have a customer community, which is a qualitative research tool, it’s like a panel of customers that you can do quick polls and surveys, it’s like a social media for your customers. And they, you get so much qualitative insight. Those communities, the best versions of those are built on top of internal data lakes. So you can follow the strings down to who these people are and how they are transacting. Every project you run in your community now has a behavioral data trail to look at, okay, we’ve got insights, the business made a decision. Now you can measure the impact of that decision because we can track these people. Did they actually spend more? Did they actually convert? So it’s, the possibilities are endless. And it honestly all comes from recognizing that we’re all after the same goal here. And, and especially research quant, research quant have analytics teams. They’re doing, they have very similar backgrounds to in-house analytics teams. Like I said, there’s more similarities than there are differences, but the in-house analytics teams might not naturally be tasked with, we’re going to run a con joint or we’re going to do, you know, things like Max Diff, which is research analytics. It’s not really as well known methodology in in-house analytics, but they can learn why, you know, why shouldn’t we bring those tools to in-house analytics. And then it’s interesting for the analytics teams to learn these methodologies as well. Over time, you save money because you’re spending less on external agencies. That’s awesome. I love those. And you have better data. Yeah, I love the rule of thumb. Yeah, I feel 00:35:22.80 [Julie Hoyer]: like it’s an accelerator the way you’re talking about. I kind of hate the word flywheel, but it 00:35:27.60 [Stefanie Zammit]: makes me think of a flywheel. But the researchers also need discipline. Like they, they need to be really close to in-house data analytics reporting. Like it’s amazing how many researchers I’ve met that have never used like the dashboards, you know, they’re not in the BI at all. And like, why wouldn’t you be, again, as a rule of thumb, if you want to conduct good research, you need your sample to be representative of your customer base. How do you know what that looks like? Well, there’s BI reporting that shows, you know, you should be feeding that to the vendors to build the sample plan and the waiting plan. So it’s all connected. It’s all one thing. And I think when 00:36:07.12 [Val Kroll]: you’re talking about this connectivity too, you can be smarter about like, you know, even breaking it up like an example, especially if you have the panel or if it’s a known population, instead of asking like, amazing, like how likely are you to buy this again over the next six months or how often, you know, I remember working on advertising awareness research for a cruise line. And they would always ask like, how likely are you just to plan a cruise for you and your family over the next year? I’m like, over the next year, like these people, like they don’t know what they’re doing, they don’t know what they’re having for lunch. Like, why are you asking over the next year? Like, let’s look, there’s got to be other data for this. But in the same way, like there’s so many people who will be building out a fallout report in Adobe Analytics, like looking at them, like trying to discover the friction points and like coming up with the why like on their own, like, oh, they couldn’t make it to this next step because and it’s like, well, did you ask them if that was part of the friction because usability labs, like, you know, pop up surveys, there’s like so many different tools or ways that we can connect with the customer nowadays that, you know, not trying to fill in the blanks, like there’s a lot of different ways that we could just find out 00:37:18.08 [Stefanie Zammit]: directly. And that is another of my rules of thumbs that I didn’t mention earlier is the power of derived research versus stated, honestly, you’re wasting your money on stated surveys, they’re just no one knows humans do not know why we behave the way we do, right? It’s all like deep psyche. We’re super weird creatures. We have all these quirks that we don’t understand. So there, yeah, there’s you’re wasting your dollars on how likely are you to book a cruise? Like, whatever, it’s bullshit. It’s not sorry. Oh, we can swear on this. Yeah. Oh, we’re explicitly ready to send it. Let’s encourage. Yeah. Exactly. Absolutely. Like that, that needs to be the best quant research has analytics. If you’re running quant without analytics in your surveys, I don’t know what you’re spending your money on. Honestly, it’s yeah, it’s just not good value. And that again, 00:38:10.32 [Michael Helbling]: that ties then to internal analytics. This is so good. So I’m sitting here from a data practitioner standpoint and just loving the conversation. At the same time, I’m going to admit to you that like you’re throwing out certain terminology that I vaguely familiar with, but don’t necessarily know. Like what are there resources that could help someone kind of like level up and get better understanding of just sort of topics, structures, stuff like that, any like good overall books or 00:38:39.36 [Stefanie Zammit]: resources online, like anything you might recommend? Yes. And this is a really important point. I think another reason for the othering that happens in these fields is purely language. And so you’re right. I’m using, I’m trained in research terminology, which is fine. I just am like 00:38:56.08 [Michael Helbling]: admitting that I don’t know all the words you used. So yeah, but the, the, a lot of the words 00:39:00.88 [Stefanie Zammit]: I’m using, there’s, there’s a analytic or data equivalent of it. It’s, it’s just that different term terminologies used. So, you know, data might talk about addressable audience, which is like sample plans. The best advice I can give is research is grounded in academia. It came from, you know, like scientific studies or social studies. And so for anyone who, who was at university doing research projects as part of their university degree, that is the foundation of modern commercial research, but there absolutely are some great tools. There’s a really great book that I recommend. It’s, it’s one book. It’s the only one you need. What’s it called? I think it’s called, it’s the market research society’s main, main book. I think it’s called intro to market research. And that’s, yeah, that’s your one reference of just looking up these words. And you’ll be amazed how many of them you look up that you’ll recognize as not actually that unusual. So like an attribution model, how different is that from a customer journey? A research team would run a customer journey study, a data science team or analytics team would run an attribution model or call it customer journey, but you know, depending on what, where your journey 00:40:07.04 [Michael Helbling]: is. It’s a, it’s a customer journey has been used all over the place for all kinds of stuff. 00:40:14.48 [Julie Hoyer]: Really, whatever you want. Just like use case. Oh God. 00:40:20.64 [Michael Helbling]: The journey of customer journey is a little bit tricky. 00:40:24.64 [Val Kroll]: That’s, that’s, that’s a, that’s a cartoon strip right there. Well, so there actually is, so to your point, Michael, there is one, because you were just starting to talk about this about the difference between like stated versus like derived importance, which gets to max diff, which I, 00:40:40.88 [Announcer]: I literally couldn’t love anything more than studies where we get to do that. Because there’s 00:40:45.68 [Val Kroll]: so many, there’s so many different applications. But when I was, I worked on the telecom vertical is my first job out of college. And we would use that to figure out like back in the day, like cable internet TV, what should be involved in that packaging and at what price points. And people would say things like, Oh yeah, I need access to like 600 channels. When we asked like, what’s most important to you, but when we actually did like the force ranking, or there’s like different techniques, that that actually is one of the things that fell to the bottom, that it was about, you know, how long is it going to take for the technician to install the, the cable box. And there was like all these other things that were like, not something that someone would say necessarily, but it really, when it comes out in the wash. But anyways, that’s just like one example. But could you, especially if you have an example that you can pull upon to talk a little bit about state of versus drive importance or one of your favorite examples of that? It’s a really fun one. 00:41:40.40 [Stefanie Zammit]: I mean, in a nutshell, the difference is asking someone, yeah, how likely are you to do this? And with, whereas with derive, you basically give them an exercise and then you observe behavior. So for me, derived research is the same as what would be happening in a data team where you’re observing the behaviors, right? Because in data, there is no stated, like you’re not stating it, you’re just watching people behave and you’re tracking their data. So it’s the research version of that, that we give them different exercises or force response between many, like many, many, many choices, again, and again, and again, mixing them up again and again, so they could never remember. Yeah, like you could never remember the pattern. And through the continuous, like, it’s a choice between these four things or a choice between these and again and again, different scenarios, again and again, you can derive what the true behavior is or will be. So it could be used predictively to say, when this is true, this is the behavior that we want. Similar to building a predictive model based on behavioral data, that you can, you know, take all your data and map it over time and start to say, you know, just like run correlations to 00:42:54.56 [Announcer]: sort of say, when this is true, this is more likely to happen. So again, very similar outcomes, 00:42:59.60 [Michael Helbling]: but just completely different methodologies. All right, I’ve got another question that’s probably going to reveal how much I don’t know about this topic, but I want to ask it. Why are personas so bad most of the time? Segmentation makes my skin itch. If people are 00:43:18.64 [Julie Hoyer]: like, well, let’s look at the segments. I’m like, can we not? Because I honestly think, 00:43:23.68 [Michael Helbling]: I honestly think this is also a driver of the divide in a lot of ways. Like as a data guy, like I see people come up with these persona studies and stuff, and they’re dog shit. Like they’re really like terrible. And I’m like, scrap your personas. It’s all behavioral based at all. Like, you don’t because what I observe people do is they just make up who they like their customer to be. And then they’re like, this is this is Sally. And she’s a hip mother of three. And she drives a van. And but she’s got this cool thing that we like about our brand. And so that’s one of our personas. And it’s like, Sally doesn’t exist in our database anywhere. That’s not our customer. And the people who buy the products you’re talking about don’t look like her at all. Like it’s no correlation. Anyway, sorry, I’m now getting into my rants. But 00:44:18.16 [Stefanie Zammit]: what’s happening there? Like, why is it so bad? I love it. And I think the word segmentation or persona in themselves can mean so many things that these are words that are overused and not necessarily always used in the right way. And I don’t think there is actually a fixed definition. I think it just depends internally on what definition you choose. But why are they so bad? I have run uncountable number of segmentations, mostly from my market research background, where I have more years of experience. And I think that the win or lose of a segmentation is in how it is translated or brought to life for the business. So you cannot have a good segmentation without really solid underlying data, you know, hardcore, like, like a good, the factor analysis and the clustering, like, and all that has to be and you had the right variables and you had the right ingredients, all of that is really important. But that’s not actually matters or has impact. That’s just designing the output. It’s like a BI report. You can have like, you know, the BI report with 1000 million, like every possible data point, it’s amazing. But unless it’s, you know, user friendly, then it just doesn’t have impact. It’s the same with the segmentation. So without, if you only had a segment descriptively, this is Sally, you know, and Sally does this and Sally does that without knowing why or without finding Sally in the data and like saying, look, this is Sally, like specifically look, we’re going to, we know what Sally wants. So we’re going to, we’re going to send her a, we’re going to do a CRM strategy around Sally, we’re going to sell to her. And now we found Sally in her data, we can actually say, look at her changing her behavior. That’s when a segmentation is really powerful. And, and I think the best segmentation is to get to that level, you need both your behavioral data and your, your research, because research gets to how does Sally think, like what matters to Sally, you’re never going to get that from just observing her in data. You need to get into her psyche. So you put the two together. Now the marketing team have a strategy on like, who is Sally in terms of her psychology, what’s going to get Sally really freaking excited and get her to behave the way we want to do, but it’s underpinned by existing data. So you can actually see her, you can maybe test with her and, and over time see the impact of your segmentation and video, video, like I cannot overstate the power of a customer inside video, just like Sally walking down the street, like you can read, this is her life. It makes such a difference stakeholders in understanding who that person is 00:46:58.80 [Julie Hoyer]: beyond her being a data point. You make it sound like so, I mean, it is so ideal, but it makes it sound so like, duh, if you just did this, you know, you’d get everything you want because then we go where I go work with clients and it’s like, they’re just so far from that 00:47:13.36 [Announcer]: point. And it feels like such an uphill battle to try to help them fit together the two worlds that 00:47:19.20 [Julie Hoyer]: you’re talking about research and analytics, you know, segmentation is all based off of outcomes. And it’s like, but you want them to change behavior to drive outcomes, but now you’ve split them by outcome already. And then you ask them questions about outcome, it just feels like something’s been lost in a lot of the situation. And it’s, it’s sad to see because they would have to, I really think like start from the ground up to get it to where they’re utilizing analytics and research in the right way to get the benefits you’re talking about. Research hurts because every 00:47:52.16 [Stefanie Zammit]: time you do a study, you need to pay money, especially because like I said, no one has in house research teams, right? Not like fully intense. That’s not a thing. I think maybe Sky TV have one, but like most companies don’t. And so you’d have to make a business case to say, why should I spend money doing something that the analytics team can do internally using this data? Why is that not good enough for stakeholders to understand that without having ever seen what good looks like is really difficult. And it’s something I find really challenging in my job actually, just explaining the value of something without having it to hand. So it’s like hypothetical to a stakeholder, right? And this is where research teams are lucky if you have good relationships with agencies that will send you case studies and they feel sort of safe enough to send you examples that you can use to build your business cases. But when you’re talking in hypotheticals, it’s very difficult to get the budget and it’s not cheap. Market research is expensive. It’s slow. It’s a big investment for any company to make. But what I do find is once you start investing in it and putting the two together, showing the impact of that, the stakeholders will then understand like, wow, I get it so much more now because I’ve got my data, but I’ve also got my why and like my… I get how this person thinks. Put those two together and it suddenly you think, how did I ever make a decision without knowing this full picture? And then that’s where you will wet the appetite and it snowballs from there. But it is very difficult to do the very first one and hopefully agencies can help with those case studies. No, I have kind of a random question. So I’ll save it to Michael when he wants 00:49:30.72 [Val Kroll]: to wrap. Okay. I love this. So I actually had a very… The opposite experience of you, Stefanie. I started on trackers and I wanted to kind of branch out of that. And so I got thrown on the iHuts, which are in-home usage product testing, which is like could not be further from the tracking world. But I love that. Like when you said the power of video, Gillette was one of our clients and they sent out these new like beard trimmers to like men and they asked them to do videos of them shaving. And it was just so funny that like they were watching like, oh, like why in God’s name are they putting that clip? Don’t hold it like that. You’re going to cut their nose off. Like, oh my gosh, we need to change our directions. But it was so funny, including like testimonials, like the voice of customer, like in some of those reports that could be like leveraged 100 different places. We also had Haynes as a client and they were trying to… There was testing all the tag lists back in the day when everyone was switching. So it was like t-shirts and underwear. So we were sending out boxes and boxes of like whitey tighties and asking people like, tell us about… Did the tag scratch your butt? Like that was like the question. But the quote that we got, I’m like, I really wish I could see this used in like the 00:50:43.60 [Announcer]: internal decks, like where this went. But anyways, to your point about like it’s an investment in 00:50:51.12 [Val Kroll]: the first one, if you only think about it as like, I’m going to send out a survey and I wonder what they’re going to respond on like likelihood to agree to a certain attitude or statement versus like thinking more creatively about the different ways that you can interact with the customer, I think you can get people really excited about ways it can be injected in. So if you take one thing away, don’t think myopically about what research is and the ways it could be applied because it can actually be pretty fun and pretty enlightening. So… All right, Julie, 00:51:23.20 [Michael Helbling]: lightning round. Random question time. Lightning round. Because we have to start to wrap up here 00:51:27.68 [Julie Hoyer]: actually. Fine, Michael, we have to start to wrap. Okay, my question, because you were saying 00:51:35.20 [Announcer]: research is not fast and it is not cheap. But nowadays people like fast and people like cheap 00:51:41.52 [Julie Hoyer]: and people like AI because AI is… Oh, that was literally my question, Julie. So I have a slight spin. Let’s see if you went this far too, Michael, because maybe we were totally, totally parallel 00:51:51.60 [Announcer]: thinking, which I love. Because there were the two things, the faster and the cheaper. So 00:51:58.64 [Julie Hoyer]: we’ve had an episode in the past about synthetic data. I was curious like your thoughts on using synthetic data in this space, maybe some pros and cons. But then it immediately my brain jumped to, well AI in general is the fast and cheap option. And both of those things feel like people are very quickly going to grab for them to fill the gaps of classic research. But we’ve spent this episode saying that we’re weird creatures. And like to actually figure out the why, you can’t just ask them why, you got to take the time to observe. And those things are just at 00:52:29.68 [Stefanie Zammit]: such opposite ends of the spectrum. So it depends who you are as a company, how relevant or how useful synthetic data would be for you. So for example, at Bang and Alson, we work in small data, our transaction volumes are relatively low. You know, we’re lucky if we get a data set of, 00:52:52.24 [Announcer]: I don’t know, a couple of hundred thousand rows. So our client is so niche and so 00:53:02.88 [Stefanie Zammit]: under, I’m so misunderstood, or so not yet understood, because we are a luxury brand in the consumer electronic space. We can’t learn from other consumer electronics behaviors, but we can’t learn from other luxury behaviors. So synthetic data is just never going to be relevant for us as a brand. There isn’t enough volume and there isn’t enough lookalike profiles. And we’re still exploring the category, you know, being the pioneers of the category. If you’re a CPG company, and you sell in, you know, the typical supermarket or grocery store, 00:53:36.40 [Announcer]: then yes, absolutely. That makes sense. I would say though that there is a watch out that 00:53:43.44 [Stefanie Zammit]: we’re living in a changing world. So my rule of thumb when it comes to any kind of behavioral insights work is that they have a shelf life of around three to five years. But that’s been my rule of thumb since pre-COVID. And I do think that the world is changing more quickly now post-COVID than it was pre-COVID. So you have to think culturally, is the world the same enough for me to rely on synthetic data, which might go back, it depends where your cutoff is, right, where you start your data set from. So I would warn against caution to think about that. If there’s a huge world event, you’re probably going to need to go in with fresh questions or fresh exploration. And yeah, just the world we live in right now, it is so turbulent that, yeah, it’s not, the three to five year shelf life thing might not 00:54:31.60 [Michael Helbling]: be applicable anymore. Oh, that’s really good insight. And yeah, that was basically Julie, the question I was going to ask is about AI and its place in this, because I’ve seen startups going around, you know, being like, we can create a 100 personal digital twin research panel for you on the fly with AI and you can do your pre-research research with it and stuff like that. And I think there might be a place for it. But like, like you said, Stefanie, you have to kind of like think through the applicability. And I like the way you specified, like, hey, for our brand, we understand how unique we are. So a group of averages is not going to get us to an insight that we could use, which is, I think, very, very relevant. That’s really good. All right, we’ve got to start to wrap up. This is so fascinating. So thank you so much, Stefanie, for joining us. And it’s very good, educational, and really fun to talk about. And I know Val, you probably also are loving this episode. So, okay, what we’ve got to do last calls, something we do every show, we just go around the horn, share something that might be of interest to our listeners. Stefanie, you’re our guest. Do you have a last call or a couple you’d 00:55:45.68 [Stefanie Zammit]: like to share? I do. I have two last calls. And you know what, of all the prep I did for this episode, this was the thing that stressed me out the most, because you guys’s last calls are so good, as I got it coming with something good. It can’t just be any old thing. I did lose some sleep over these. But I think I got two good ones for you. So the first one is it’s actually a game. I’m a gamer. I love any type of game, board game, video game. And I attended a leadership training that was organized by our amazing HR team at Bang and Olson. And we played, it’s essentially a simulation game. You’re given a group of employees and they have to deliver a project together. And you know, it’s a bit like Moenopoly, you get chance cards and things go wrong. And it’s like, oh, the project, you know, somebody went on stress leave, what are you going to do? How are you going to keep to the time and the budget? Uh-oh, your main stakeholder has suddenly decided that they forgot what this was all about. What are you going to do? 00:56:42.72 [Julie Hoyer]: That sounds traumatic. I was like, this is giving me like stress. 00:56:51.44 [Michael Helbling]: It was so fun. I play that game every day, Stefanie. What are you talking about? 00:56:58.72 [Stefanie Zammit]: Sorry. No, but you’re, you’re sorry. We do play that game every day, but the fact of having the safe space where you could have these, oh shit, like everything’s going wrong in my project moments, but you’re learning how to deal with those in the safe space so that when it comes to your real life game, you’re prepared. I thought it was a great idea. It’s the game that we played was called the Playmakers game. It’s made by a company called the Works, Works with a Z. And I think they have a bunch of other sort of professional world simulation games as well. Super recommended. 00:57:31.12 [Announcer]: Yeah, next onsite. And it was a great way to like build rapport with your stakeholders as well, 00:57:37.44 [Stefanie Zammit]: right? Cause safe space and you play the game with a team of actual colleagues. So it was good 00:57:42.88 [Val Kroll]: for the bonding. Oh my gosh. You should like reverse roles. Like I get to be the stakeholder this time. Yeah. That would actually be hilarious. What should I do with all this power? 00:57:53.60 [Michael Helbling]: My problem would be like, really? You’re going to do that? Like no. 00:57:59.28 [Stefanie Zammit]: You have to all agree on the decision. That’s the game. Like how are we all happy? All going to do it. Oh no, we lost the team to stress leave. Damn it. Like we, so yeah, it was 00:58:11.36 [Announcer]: that’s awesome. That’s very cool. What else? Well, I have measured my second one just because I 00:58:17.20 [Stefanie Zammit]: am really excited that this is like hot off the press. It’s literally just gone live. I think two weeks ago, as you know, I worked for Bang and Offsend and we’ve just opened the factories up for anybody who’s interested in audio to go experience the manufacturing of our products. And honestly, if you’re a sound nerd or an audiophile, it’s a incredible experience. Like a really just sort of once in a lifetime immersion into the world of audio in a very beautiful part of the world. So yeah, that was my second one. Fun. Wait, Stefanie, I did see your note. You have to say the freaky part. Oh, the freaky part. Yes. So it’s a tour all through the, you know, like how the tonmeisters are called, how they find the perfect sound. And there’s a lot of different rooms that are created in a way for you to experience sound in different ways, which is how the products are developed. And one of the rooms is the 100% noise-proofed room. And it is the scariest place. Like honestly, I couldn’t stay in there with the door closed. You wouldn’t believe how scary 100% soundproof is. You can hear your blood flowing through your 00:59:27.44 [Julie Hoyer]: veins. It’s terrifying. Insane. I was like trying to imagine that. And I was like, 00:59:34.96 [Stefanie Zammit]: that’s just like breaking my brain. Honestly, 10 minutes and you’re like, get me out of here. Like I’m going crazy. Yeah, I bet. Okay, now I need to go do this. 00:59:45.68 [Michael Helbling]: I know. Stefanie, great job. You’ve upheld your end to the Les calls by far. Yeah, 10 out of 10. Absolutely. Yeah. Who wants to follow that? Val, what’s your Les call? 00:59:59.28 [Val Kroll]: So mine is going to be a little research-related. So I thought that one of the things we might discuss, and I do think we spent a good amount of time on it, is how to be curious in how to get broader in your understanding of these different methodologies or teams that might exist inside your own organization. And so my recommendation is to look out for some different communities that you could be a part of or join. The one that I’m still a part of today is the women in research, the wire group actually started in Chicago a long time ago. But I have benefited so much from my different mentorship conversations. I still stay in touch with the mentor I was assigned with, I think 13 years ago now, 14 years ago, Sheri Binky, shout out, I know she’s a listener. But there’s like, I’m a part of like women in product groups, and it doesn’t have to be like women only groups too, but they have so many different great events where you can get out and talk to people. So get out, touch some grass, talk to people, learn about how people are putting some of these different ideas to use inside of organizations, because that can totally be an inspiration for the way that you bring it to your own work. 01:01:12.96 [Michael Helbling]: Outstanding. All right, Julie, what about you? What’s your last call? 01:01:17.04 [Julie Hoyer]: Fine, it’s a little bit random. But honestly, this is something I’ve definitely heard mentioned before, quantum computing. And I do think this is like the next leap probably after AI, if my naive take on it is anywhere close to true. I was reading a newsletter that I always get, and there was this one mention of like the next big leap, like leap, I think it was called. And so I clicked on it, and it was all about quantum computing. It was an infographic about I’m like, well, I’ve heard it mentioned, I don’t know what it is, like, sure, I’ll take a look at infographic. And it, it was really good. And the way they broke it down within like not a very long read, I totally have a new appreciation of what this means and why so many companies are going after it. Pretty much they’re saying like, instead of using electrons for zeros and ones, like, we’re going to use a subatomic particle that is like super finicky to keep stable. But pretty much we go from being able to compute things one at a time linearly to doing things what’s it called simultaneously. So all these computations simultaneously that you could do and you think about how much computing like AI is doing or different industries like finance or supply chain even, and they, they walked through like a supply chain example. And it was amazing to think that you could go from something that would take a normal supercomputer, like they even said like a trillion years in their example to doing it so much quicker with quantum computing. And they said that some of this quantum computing power could actually happen in the next two to five years. And some of these people working at companies were like, I was told I wouldn’t see some of the milestones we have hit recently in my lifetime and like they have hit them. So it was one super interesting. I finally feel like I kind of understand what it is. I was a really quick read too. So it got me kind of freaked out and excited. And I just felt like, Oh, I learned something. 01:03:16.32 [Announcer]: Sounds good. I like it. Yeah. What about you, Helbs? 01:03:20.56 [Michael Helbling]: Well, I, as per usual, love everything I read on CommonCog.com, Cedric Chin, and he wrote an article recently about how to make sense of everything that’s happening at AI, because it just sort of feels overwhelming most of the time. And it actually sort of ties back to the episode in a way, because one of the points he was making was sort of like, don’t listen to what people say about AI, watch what they’re doing with it in the real world, to use that as a guidepost for how you should be responding to AI, which kind of goes back to sort of like user research. So anyways, the article is really good, but it’s very practical in terms of just better sense making around, like, okay, there’s sort of this hype and concern and all these other things. But like, look at actual detailed examples of how people are actually using it, and then ask some questions from there, like, what other outcomes are possible? What actions could I take? What matters most in my context? Those kinds of things. So anyways, really good article, just to like, take some of the pressure out of what I think a lot of us are feeling about AI, like, half the time it’s like, is it going to take my job? And the other half of time, this is so cool. I can’t believe I don’t, you know, I’m just doing everything with AI now. So there’s some balance where you have to find a balance. Otherwise, we’re going to blow up. Anyway, so that’s my best call. Blow up. All right. Yeah. Tune in. What’s the old, whatever, I won’t try to remember what the hippies used to say. Stefanie, thank you so much for coming on the show. This has been so fun. 01:05:01.12 [Stefanie Zammit]: Thank you. Yeah. Thank you for having me. It’s awesome to get to talk to you guys and talk about 01:05:06.40 [Michael Helbling]: nerdy topics that I love. So thank you. Yeah. No, it’s been great. And I’m sure as you’ve been listening to the show, you might have questions or you might have ideas. We’d love to hear from you. The best way to reach out to us is through the major Slack chat group or LinkedIn or via email at contact at analyticshour.io. And please feel free to reach out and leave us a review on the platform that you listen to us on, whether that’s Apple or Spotify or whatever, you know, whatever one we’d love to hear from you. We love getting feedback on the show. So definitely do that. And we’re still asking you to give us some questions just a couple of weeks to go until we’re going to be recording a show live at Marketing Analytics Summit on April 29th in Santa Barbara, sunny California. And we’ve got a survey which is out on the show notes page. Go fill it out. I mean, perfect example. Hopefully we did a good survey. I’m pretty sure we probably did. I know, right? Although again, I just want to caveat each time I had nothing to do with the art at the end of that survey. You’ll have to fill the survey out to see what I’m talking about. But it was I had no editorial control whatsoever. But if you have a question, we want to gather lots of great questions from listeners, either if you’re be there or not, we’re going to let answer them live on the show when we record it there at Marketing Analytics Summit. So 01:06:29.92 [Announcer]: looking forward to that is just a couple of weeks away. All right. Great show. Very fun. 01:06:37.76 [Michael Helbling]: And I know I speak for both of my co-hosts, Val and Julie. And I say, no matter what your 01:06:42.96 [Announcer]: market research says, keep analyzing. Thanks for listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at At Analytics Hour, on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst. Those smart guys wanted to fit in. So they made up a term called analytics. Analytics don’t work. Do the analytics say go for it, no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 01:07:24.72 [Val Kroll]: Julie has this mic that like, she could be across the room and she’d be like, 01:07:28.88 [Announcer]: it’s like so loud. 01:07:35.28 [Michael Helbling]: Julie inadvertently must have bought one of those ASMR mics or something. 01:07:39.92 [Julie Hoyer]: Literally, I have like the game turned as far down. It’s like more than on arms length away for me. I’m like speaking, you know, I’m trying to speak on the softer side and I’ve turned on my volume in reverse side. No, I think it sounds good. And I’ve made sure my humidifiers are off. So hopefully no background humming. I unplugged my wine fridge, like all the things. Yeah. 01:08:03.28 [Michael Helbling]: And the worst part though, Stefanie, is we have this really great engineer, Tony, who goes through and does like all the audio editing and he gives very specific feedback about who’s audio quality was terrible. And so we’re going to use notes back from Tim of like, oh, Michael’s awful this episode. And we’re like, oh, thanks. That’s so great. So that was like so cautious. You sound a little soft, but you sound okay. Yeah. Anyways, no, it’s just one of those things where you’re like, you think we’d have it nailed down after so many years, but we’re still like 01:08:38.00 [Val Kroll]: every episode we’re sort of like fine tuning. You know what y’all need? 01:08:42.32 [Julie Hoyer]: Carabangan Olsson’s just saying. Yeah. That’s right. Yeah, you’re the person to ask about that. I 01:08:48.80 [Announcer]: should go look. Well, I’ll wait for Val to stop typing. I know this. Usually it’s Moee and she’s 01:08:58.16 [Val Kroll]: like keyboard cat, like you’re like Moee. Yeah. We’re a very serious professional podcast. 01:09:06.48 [Michael Helbling]: That’s right. Bringing it all together. Here we go. All right, I’ll give us a five count and 01:09:10.80 [Announcer]: we’ll get started. We’ll go in five, four, three, rock flag and two worlds, one family. The post #295: Research and Analytics: the Peanut Butter and Chocolate of Data? appeared first on The Analytics Power Hour: Data and Analytics Podcast.

March 31, 20261 hr 8 min

#294: Adapting an Analytics Team to an AI World

AI is moving fast. But so is life. AI is widely recognized as a must-adopt technology, but how and where are data workers expected to find the time for that?! Organizations are struggling to find effective ways to productively drive healthy adoption of AI: What is it they expect their workers to do with AI? Is it purely an efficiency driver, or should they expect other avenues of value creation to be pursued? What guardrails need to be in place? What incentive structures are (and are not) effective when it comes encouraging team members to take the AI plunge? One tactic that is definitely effective is to have leaders who are excited, engaged, and transparent as they get their hands dirty. And, boy, did the algorithm deliver one of those to us in the form of John Lovett, VP of Analytics at SEER Interactive, for this discussion! This episode is brought to you by Prism from Ask-Y—your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show (Book) The *NEW* Big Book of KPIs: (Key Performance Indicators) by John Lovett Wil Reynolds AI Operating Manual: A Step-by-Step Guide to Teaching AI Systems How You Actually Think Twyman’s Law (Markdown file) Twyman’s Law Data Quality Module For LLM Analytics Integration & MCP Servers (Blog) Olympics analysis w/ AI by SEER (not published at time of recording, so we intended to track it down before this went live; if you’re reading this, Tim’s system failed and that did not happen!) (Blog) Live GEO Olympics Winter 2026 Results (Book) The AI Marketing Canvas, Second Edition: A Five-Step AI Plan for Marketers by Rajkumar Venkatsen and Jim Lecinski The diagram that Moe referenced from the above book (also called out two episodes back!): (Podcast) The Artificial Intelligence Show – Paul Ritzer and Mike Kaput (Conference) MAICON: The Marketing Artificial Intelligence Conference – Oct. 13-15, 2026 in Cleveland, OH LinkedIn GEO Community (Substack) From Data to Product by Eric Weber (Book)  Code Name Hélène: A Novel by Ariel Lawhon Data Kids Visualization Contest for Children (Conference) Marketing Analytics Summit – April 28-29, 2026 in Santa Barbara, CA Go to analyticshour.io/listener to submit a question for us to (potentially) answer when we record at Marketing Analytics Summit! Photo by Maximalfocus on Unsplash Episode Transcript00:00:05.76 [Announcer]: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. 00:00:15.87 [Michael Helbling]: Hi everybody, welcome. It’s the Analytics Power Hour. This is episode 294. Yeah, probably right after the old RTO policies got rolled out, you probably got another executive communication about going AI first or whatever the hell that means. And let’s be honest, I think we’re all making pretty heavy use of AI now in some capacity. But what does that even mean exactly? And specifically in our world of data? I mean, should I just be uploading all my data to Claude and letting it come up with my analysis? Or is it in planning where you are constantly having to tell chat GPT not to jump the gun and start writing SQL when you’re just exploring some concepts or ideas? And I don’t know. So welcome to the AI Analytics Power Hour, I guess, this time. We’re going to talk about it, not just using it yourself, but how to think about rolling AI out across your team or your organization. As always, I’m joined by my co-hosts, Julie Hoyer. Welcome. 00:01:16.73 [Julie Hoyer]: I’m back. 00:01:17.39 [Michael Helbling]: Actually, welcome back. Glad to be here. Oh my gosh, yeah, of course. It’s like you’ve never been here. It’s been a while. It has been a while. And Moee Kiss, how you going? I’m going pretty good. Outstanding. And I’m Michael Hellblink, and I’m excited for our returning guest. John Lovett is the VP of Analytics at SEER Interactive. He previously held leadership positions at Further and Web Analytics to Mystified as well as many other companies. He’s the author of at least a couple of books, the most recent one being The New Big Book of KPIs, Key Performance Indicators. And today he is our guest. Welcome back to the show, John. Thank you so much. It’s great to be here. Awesome. Well, it’s great to have you back. Finally, it’s been a long time. So it’s a long time coming. But I think what drove us to this was specifically how it seems like you and all the team at Seer are really diving headfirst into using AI in significant ways across your organization and across the teams to do your work, to bring new ideas to life. Just talk a little bit about what it’s like inside your four walls and how AI is impacting your work. 00:02:27.94 [John Lovett]: Yeah, yeah. Well, I’ll start by saying, so I’ve been at SEER now for a little over three and a half years. And the last 18 to 24 months of that, I’ve just been immersed in it. And I’m not like talking as an AI enthusiast. I leave my analytics division, I hold a P&L, and basically I’m making hard bets. And as the kids say, I’ve got some receipts to show for it. So I do want to say, though, that At our company, I’m obviously going to talk a lot about my experiences and how I rolled it out to my team. But the first thing I really want to say is that getting your hands dirty with AI isn’t optional anymore. You said this in your preference, Michael. But doing it without accountability is really how you lose credibility. Every story I’m going to tell today is about building infrastructure that makes AI trustworthy. And not just fast and not just do more, but this is a little trite, but I developed this when I first got to SEER. The tagline for my division is we build trust in data and we empower people to use it, to use their data. And that hasn’t changed with AI. AIs come along, but that’s really fundamental is trust in data and really being able to use it. So that’s kind of the first thing that I’ll say about that. And if you indulge me and let me ramble a little more, just talking about SEER as a whole, like I couldn’t, well, what I’ve been able to accomplish, starts at the top and shout out to Will Reynolds, the man, the myth, the legend. He’s the CEO. In fact, he doesn’t like to call himself CEO of Sear, but he founded Sear and he basically said, guys, listen, we are all in on AI and everybody here needs to get on board or. You don’t need to, but there’s the door. And made it such that we had a big pivot. I want to go back to December 2025. It was a mandate for every single person in our company. I think we’re about 175. We’re closer to 200 people now. Had to take an AI training course and get certified in AI. So it was like mandatory requisite. Everybody gets trained. And then we rolled around to summer of 2025 and Will said, listen, everyone at this organization, from associate all the way up to executive, senior VP has agency to be able to take license to develop with AI, build the prototypes, show them to your clients as prototypes, get feedback, and let’s get them into production if people like them. So the company as a whole basically said, we’re all in on this. We’re going to give you the tools. We’re going to give you some training and let you loose on this and really make it a part of our culture at the company and make it such that it was a part of everybody’s job. I’ll pause there. I’ve got more on that, but I’ll pause there for a minute. 00:05:36.80 [Julie Hoyer]: Even with the leadership mandates, that is pretty big to hear such big pivots and saying they wanted it company-wide. That’s awesome to hear because it sounds like it gives you and your team a lot of opportunities and the invitation to go try things out with AI, see what works, get ahead of some of these trends or figure out what it could be best used for. But I am curious, even smaller, before the leadership mandates or right afterwards, what was your first thought around it, John? Or what was your first even small step? Because you were running a whole team. How did you then as a leader of an entire delivery team of analysts, go about that. 00:06:22.67 [John Lovett]: Yeah. You know, honestly, it was kind of like staring at a blank piece of paper like, what do I do with this? I was just like everybody else. And I think my first experiences, which I do encourage people to try, are just use it for your personal life. Like I would literally open my refrigerator door, take a picture with chat GPT and say, what can I make for dinner? And it would look at it and say like, oh, you’ve got pasta sauce there. I see some tomatoes and I see a bell pepper. Like here’s a recipe for you. And then I would just start using it and talking to it. My family and I took a trip to Ireland last Thanksgiving, back in 2000. 24. And basically, I had to build our itinerary. And so we’re driving around Ireland. And of course, I had to put the chat to PT Irish accent on my on the talker. And so my wife called my girlfriend. It was like, that was like, where should we go today? What restaurant should we go to in Galway? And it was like, it was giving me recommendations. And they were great. It told me where to stay. The funny things that started happening when I would talk to it and it tell the things You know, I said like, hey, I’ve got three teenage boys. Here’s the things we like to do. This is what we want to try to do on our own. It was like a month later and I’m in the kitchen and I’m asking it something probably what to cook because I rely on that a lot. It thinks I thought I liked to cook and I say, I only ask you because I hate to cook. I just need the ideas. And it said to me, how are the boys doing? What would they like for dinner? And that for me was like, Like, and of course, like, this was my… Oh, see that? 00:08:00.48 [Julie Hoyer]: I feel like that would scare the shit out of me. 00:08:02.62 [John Lovett]: It totally scared me because I was like, what? And I almost dropped the phone because I was like, how do you even know that? And then the voice said back, well, you told me that you, I know that you went to Ireland with your boys and you talk about them a lot. And it brought that up in conversation. And that for me was like… like blew my mind in terms of what the possibilities were. And that was all just like my personal life before I really even started getting into it with work. 00:08:28.23 [Moe Kiss]: Can I ask a little bit? I’ve had a similar experience where there’s been a lot of support leadership down in terms of everyone at the company has been given access to every single tool that they could possibly want, enterprise grade. I do personally think that is an absolute game changer. I’m not going to tell you which tool to use, you can use any tool. But talk to me about the team experience, because I do feel there are those that are like, yeah, I’m going to roll my sleeves up. I’m going to get my hands dirty, that sort of stuff. And there are folks that are really scared. And they have that, it’s going to take my job. How did you navigate that team dynamic of the folks that are super keen versus the ones that are very apprehensive? 00:09:15.83 [John Lovett]: Yeah. Yeah. associated with that. There were people on my team that were like, let me add it, I’m all in. Others that are like, I don’t think so. I’m skeptical. One of the things that we did as an organization that I think helped open the door a lot, and this was early last year. So actually, many of you have even been early 2025. We, as a company, every division looked at every single deliverable that we had. What do we regularly produce for clients? What are our workflows? How do we work? And then built this huge list of, if we do all these things, where can we start AI that would help us to do these things faster, more efficiently, better? And we came up with a list at that time of 15 different workflows, deliverables, processes that we did on a regular basis. Then we said, okay, we’re going to prioritize this subset in Horizon 1. We call them, we have horizons of work. Horizon 1 is going to be this first build. We had a series of them in Horizon 1, then we go to Horizon 2, and Horizon 3 is bigger thinking, stacking them all on top of each other. But that really gave us the opportunity to think about the day-to-day work that we do, and how AI can be a part of that. I reiterate it at every team meeting, at every dead meeting. I’m going to hold that for you guys, I don’t know if you’ll see it, but I keep a sticky note. You probably can’t read this, but on my monitor that says, how can AI help me do this? And I genuinely think about that. It’s right here on my monitor. I have to look at it every minute because I encourage my team, like, you’re going to do something. Think about if AI can help. And there’s plenty of things that AI cannot do for you, but there’s a lot that it can do. So that was sort of a door opener for people to say, oh, I do this every day. One quick example I’ll give you is like, we’re an agency, right? And despite me trying to squash this as much as I can, we live and die by the billable hour. So we have to track time cards, right? And I’ve got team members that do it religiously and team members that just don’t. And they’re not part of their workflow, not part of their habit. We built an agent that said like, hey, connect. In our case, it’s a Claude instance to my calendar, to my rake, which is our project management. We’ve got connections to all of our platforms, but I can say, hey, look at my calendar and see everything that’s on there and populate my timecard for today or this week. Just that alone, I spent maybe less than an hour on that every day, but now I’m like, boom, I got it and I can do it. That same tool that I built to do that, Now, every morning I get in and I say, what’s my morning briefing? What is the most important thing I have to do this day? And it goes to my inbox. It goes to my Slack. It goes to all these different tools that I use every day. And it can say like, oh, you’ve got a podcast tonight with the with the greatest podcasters on the planet, you need a prep for that. And so it will tell me kind of the most important things to do. And that for me has been like such an eye-opener to say, like, you know, I would sit at my desk on Moenday, it’s like, oh, crap, what do I have to do this week? And I would write it out with a pencil on a piece of paper. And now I can just ask my AI partners, like, how do I, you know, what’s important to me? And that’s been a big change as well. 00:12:40.11 [Julie Hoyer]: Did you? Okay. This is a little nitty gritty, but I’m curious. Some people still love like, I still love to write like a to-do list. Obviously that’s not AI friendly by any means, unless I put it somewhere digitally. Um, you do. Okay. So I’m, I’m curious, like, do you then to help with that workflow of utilizing AI to surface things to you? Are you actually like snapping a picture of your physical to-do lists? Are you retyping them? Are you dropping slacks to yourself so that your AI will read it? Like, 00:13:09.07 [John Lovett]: A lot of times, and maybe this is me, old guy, old school, I take notes when I’m on a call with a client and they’re just riffing and talking about stuff and I’m either trying to capture requirements or figure out what’s important to them. I want to be in that moment. And for me, even though Whenever I can, I have a inscriber that’s recording the call so I can have the whole transcript and we can talk about that later. But that’s critical for me going back to, but I’m still taking those notes because when I go to write a scope of work or proposal or just update a report for a client or whatever I’m trying to do. That to me is like, okay, my brain interpreted this, I wrote it down, I know I need to get this done. I generally don’t take pictures of that and put them into Slack. A lot of times I will send myself Slack messages for reminders, but more often than not, those are links and different things that I want to come back to. But I do rely on head and paper for my thinking process to help me do things. And I still haven’t gotten away from the satisfaction of checking something off the list. I give that to myself. That’s so good. 00:14:21.81 [Tim Wilson]: Michael, quick question. How many times have you solved this same analytics problem this month? 00:14:29.56 [Michael Helbling]: Oh, probably enough times that I’m considering invoicing myself. Step one, fix GA4 source medium. Step two, lose the will to live. 00:14:39.21 [Tim Wilson]: Cool, so let’s stop living like that. Prism by Ask Why lets you save your best workflows as skills, portable expertise you can reuse across datasets and tools. 00:14:50.95 [Michael Helbling]: Okay, so like normalize UTMs, dedubleeds, merge Facebook and Google spend, maybe rename 37 versions of newsletter into one civilized channel. 00:15:01.69 [Tim Wilson]: Exactly. You build it once, then you run it again, instead of recreating it like it’s Groundhog Day, but with more spreadsheets and less Bill Murray. I mean, I like Bill Murray, but I do like fewer tabs. Plus, you know, there was Andy McDowell as well, but really plus jam, jam memory. It remembers context across sessions like your org’s definition of active user, which table is the source of truth, and that one cursed date range where tracking alas broke. 00:15:31.61 [Michael Helbling]: So I don’t have to start every meeting with before we begin. 00:15:35.52 [Tim Wilson]: Here’s the lore of our metrics Yeah, or or we explain that revenue means net of refunds here not whatever looks best on the slide Okay, well my dashboard has now been personally attacked Well, that’s good. My mission, my mission is complete. But skills plus memory means prism gets smarter about your world over time, your processes, your definitions, your shortcuts. 00:16:03.31 [Michael Helbling]: So it’s like an assistant that actually remembers my preferences, like I’m not getting from most of my streaming apps. 00:16:10.86 [Tim Wilson]: Exactly. So if you want early access, you can go to ask.y and join the waitlist. Speaking of waitlist, use code APH when you sign up and you will be bumped right to the top of that list. 00:16:22.93 [Michael Helbling]: OK, good deal. 00:16:23.93 [Tim Wilson]: I’m already on the website. Well, we’ve been doing this ad spot for a while. I hope you, Michael, have already gone to the website and signed up. But for anyone else, that’s Ask the letter “y” dot A I and use code APH. Yeah, I’m putting myself in the place of the user. 00:16:40.47 [Michael Helbling]: It’s called Empathy Tim. Oh, well, I don’t understand that. And I’m already over here saving time and my sanity using these skills. Yeah. It’s too good. I take my notes online, but, but same thing where I’ve got the transcript plus the notes. And sometimes I can push them together into the prompt and use both. But yeah. 00:17:01.92 [Julie Hoyer]: Okay. Small tangent. Well, I’m just curious. So John, you have your three boys. Um, and I have a few wonderful people in my life that are, you know, their teenage years getting ready to go to college. And I fell into a very interesting conversation. And so I’m curious if like what you tell your boys, like going into college, maybe they’re already in college, out of college, but like in the AI times, I was talking to someone and they were telling me that like they aren’t great at taking notes. And I kind of panicked for them thinking, you’re about to go to college. Like you have to get really good at taking notes. And they said, yeah, but there’s AI. But as we just talked about, like there still is this analyst skill of hearing certain things from a stakeholder or still having your own like mental filter right of like what you think is important or you really want to reiterate on or you want to build a story later so to be able to jot it down and it is a skill I think to actively listen. you know, take your notes, whether you’re typing or writing. And I suddenly got a little worried and I didn’t want to like harp on them saying like, well, you really need to like learn how to do it. But it had me thinking. What are your thoughts? 00:18:13.43 [Moe Kiss]: I still take notes as well, even though there’s like a transcribe function. For me, I wouldn’t say it’s necessarily, sometimes it is like, what is the key points that I’m taking away that I really want to follow up, but I actually think it’s how I listen. Like for me, how I absorb the information, if I’m not taking notes, I will, my brain will probably go off on five different things. So I wonder if it’s like more a style thing than a like AI, not AI thing. 00:18:42.26 [Michael Helbling]: I have noticed in meetings, I will summarize and repeat in the meeting sometimes so that the AI picks up on it better. And that’s a change I’ve noticed just for the transcript. I’ll be like, OK, so to summarize, we’re probably going to make sure we do this, this, and this. And then I know that the AI then will come through the meeting transcript and be like, oh, I’ll pull that out as it to do. So even my style of conducting the meeting is shifting a little bit behind it because I know that then I don’t have to write that down. I get the AI will surface that as my to-do. But yeah, it’s crazy how we’re adapting. 00:19:14.79 [John Lovett]: We joke about that. I do that too where we’re like, hey, transcribe or remember this piece and we kind of joke. I did literally just take out a pen and paper because I don’t want to forget the questions here. So Julie, starting with the question, I definitely worry about the future for kids. My oldest is about to turn 21, which is a frightening thing in and of itself. He’s off at college in North Carolina. was a very, he helped me may listen to this because such a proud dad, he changed his major. He was a finance major. He changed his major and I said, which change to? And he’s like, dad, I entered the business analytics program in the business school. And I said, What? Shut the front door. And I said, you do know that’s what I do, right? Because I never got a notice. I mean, he’s like, yeah, dad, I know that’s what you do. So, like, such a proud dad moment. But he’s never been, I actually wrote a blog post about this. He’s had dyslexia. He’s had learning challenges. He switched high schools because he wasn’t getting the support that he needed. To your point, His handwriting, even as an adult, was horrific. He just didn’t read like anybody else. He didn’t do things like everybody else. When he did math, he did it all in his head. He didn’t write out the problems. And so he just thinks differently. And so he has been a very early adopter of AI. And for good or for bad, he’s also as a high school student and now a college student figured out how do I use tools like chat tpt and what have you and then not have my teachers think that I wrote it with AI. So he’s got the anti-AI tools to figure that out, which honestly, like I’m like, buddy, I’m going to pay for that subscription. I’m going to pay for your chat tpt subscriptions or whatever he needs because I want my kids, when they get out of college, to have this as part of their skill set. My middle son is a junior right now in high school, and we’re looking at colleges. We went to Syracuse a couple of weekends ago, and those are my questions. I want to know from universities, how are you guys going to teach AI? Because the university that says to me, oh, it’s off limits, they can’t use it, I’m going to be like, you know what, you’re not going to prepare McTin for the workforce. that might not be the way I want to go. And I just think it’s a matter of, like, we took notes, we listen and think with our brains and our hands and record things. My kids, like, they’re on their phones. They’re, you know, they don’t actually know how to talk on a phone. They text and I don’t know if you guys have younger kids, but like, when you call a kid on the phone these days, they’re like, they don’t even say hello. They don’t even say what. It’s weird. They know how to. 00:22:00.98 [Moe Kiss]: No, but they know it. I mean, my three-year-old knows how to like pinch and zoom and swipe and you’re like, what the? 00:22:07.63 [John Lovett]: It’s wild. So I just think about learning style. I’m seeing with my own children shifts and I wanna, and I tell them like, you may not plagiarize, don’t just take it and copy and paste it. You have to put your brain into this. It’s gonna give you an output, but the output it gives you is generic stop. And until you put your voice into it and teach it and put your brain into it, that’s when it becomes a partnership, not just a, dictation machine or an answer engine, it’s really when you start to leverage the value of it. And there’s little tricks that I do, we can talk about later about like, like I teach AI my voice. I said like, I uploaded my books, I uploaded my blog posts. I said, I uploaded like email examples. I said, this is how I write, this is how I talk. And I want you when you’re writing on my behalf to mirror this, to use this, to add this to your knowledge so that when you’re generating something, For me, it sounds like me, and it’s legitimate like me. And all of my agents and tools, no M-dashes, I sign off cheers, so my email messages, I have all these little quirky things that I wanna say. I’m like, no jargon, don’t use buzzwords. I put things that I am like, no pie charts, stuff like that. I’ll put those into my instructions. And in fact, if anybody wants, this. I can give this as a resource, but I built, we were first to choose at our company. I haven’t made the choice yet, but it’s like, do you want to go with chat GPT or Claude? And I was like, well, if I give up one, how do I take all that teaching and learning that I taught it and bring it to the other. And so I developed a guide to be able to take out of the model and say, what is my personality? What do you know about me? How do I talk? How do I think? How do I act? And then give me those instructions that I can upload to my next agent so that it teaches it what I am like. And I found that to be like a transferable thing that I could say like, oh, if I suddenly lose access to one of these tools, how do I not lose all that history with what I’ve told it, darling? 00:24:20.10 [Moe Kiss]: Okay, John, you’re brilliant. And yes, we want all the resources, absolutely, because I’m literally doing that at the moment. Like no comment on politics at the moment, but yes, migrating from one tool to another. But okay, what you’re talking about is like fundamentally leading from example, putting in the time and effort to do it well. Back to your team. I am sure there are people that are dabbling and just producing utter shit. And as someone on the receiving end of reading lots of shit, it’s like that exploration period. I guess I just like I want to better understand is like you have gotten to the value point. I’m I think probably quicker than most. How do you how do you help your team get to that value point more quickly? Because The feedback is always, I’m busy, I don’t have time, and I’m the first to say all of those things. How did you create the time for both yourself and then your team to get to value faster? 00:25:17.97 [John Lovett]: Yeah, yeah. So time is, you still have to do your day job. So for me, I do end up working more. I try to tell my team not to do that. But sometimes it happens. But putting that aside for a second, One of the things I did early on last year, I’m a huge believer in conversational analytics. I think it’s coming. I think it’s coming for us all. And so I built a conversational analytics, which is how do you get an MCP to communicate with whatever LLM you choose, chat, GPT, quad, whatever you want. I started with GA4 because that’s what we had most access to, and I expanded it to BigQuery. And I said, everybody on the team has to make these connections. Use the MCP, follow the instructions, set it up so that you have the ability to talk to your data with these tools. And so, some begrudgingly, as we started, some jumped right into it, some were slower to act. I think it was not even a week, it was the first few days of doing this exercise. One of the analysts on my team sent out on our company-wide Slack, oh my gosh, our blog post just went viral. We had a huge spike in traffic. It’s amazing. This blog has never seen this much traffic. Will, our CEO, he’s got all the agents and the MCP connected as well. He’s a big runner, right? He’s a marathoner. He talks to it when he’s running through his headphones and using Claude, as he’s running, and he’s like, hey, so-and-so just posted this post, so we had this big viral spike, and he started asking questions. He’s like, I’m just curious, where did this traffic come from? And the agent responded, and it’s talking to him, oh, it looks like 99% is from China. And then he goes deeper and he goes, oh, really? Like, what kind of traffic is this? Did they bounce right away? What did they engage? And they were like, no, all the visits were sub one second or whatever it was. Come to find out it was a bot. You know, so a bot was hammering our site. My analyst team was like, hey, we made this great discovery. Look at me. I got this new conversational analytics thing to work. And that was a like screech hit the brakes, like needle off the record moment for me. I was like, wait a minute, I got to put some controls on this. So it was that same week, I had seen, actually, I think Tim Wilson reposted, oh, and I’m gonna forget his name, Twyman’s Law. I’ll come back, we’ll get this in the show notes, who gave me the reference. I wrote a lengthy post about it. But Twyman’s law, for those who don’t know, is essentially, if any number looks too good to be true, it probably is. And Twyman never really published this. It was like word of mouth, and it got around. It has become this marketing staple. And so I was like, hey, that would really work for my conversational agents. Why don’t I build that in to say, if you’re seeing this huge spike in traffic and it’s anomalous and it doesn’t match any of the other patterns and the data doesn’t match, question it and dig in. And that was really a groundbreaking moment for me to say these things a lot of you. You know, they’re gonna tell you, if anybody’s used chat GPT, I definitely call that the yes man in my arsenal or my toolkit, if you will, because it’s always like, John, you look great today. That shirt looks awesome on you. You cooked the best dinner ever. Like it just, it always gives me props and like, yes. And that’s what he was doing. It was like, you found an amazing insight. Look at this. And then you put something like the guardrails on it, like Twyman’s Law, to be able to say, you can’t just throw out a number like that. You need to verify it. And since then, I’ve actually adapted it to This isn’t too technical, but all of these models that we use, or most of them, are all probabilistic. You ask GPT question, Claude, Gemini, whomever you want, and it’s going to give you probably the answer. It’s doing word by word, and what’s the next logical word, and how does it go? If you ask it, what’s two plus two, they say, I’ve seen this enough, that’s probably four. But when you’re asking it to do analysis and the metrics and dimensions and all these different things, it sometimes with a thousand percent confidence tells you, you had this massive traffic spike, you got a viral sensation on your hands, and it thinks it’s true. And so I figured out a way to be able to take those probabilistic models and built into my instructions, deterministic instructions. So I say, never give me a number. Every number that you generate, I want to see the SQL query, I want to see the math behind it, and I want to know the logic. I want to know what’s missing from the data, and I want to know what you can’t show me reliably. And that has also helped me to provide those guardrails. So I started with Twyman’s Law, but I’ve evolved it to have this deterministic layer to say, like, don’t just give me a number. I want you to perform the calculation. And I actually have SQL built into my instructions that says, like, how do you do this? And how do you get at it? And that for me is really up the reliability of the answers that I get, because it will give you garbage and junk and mislead you out of the gate unless you put those filters on it. 00:30:33.63 [Moe Kiss]: So what’s your perspective then on, you’ve applied that to your own instances and Will is a phenomenal leader who gets the data side, right? But there are lots of stakeholders who don’t, who won’t build that. So is your thinking then that you need to find a way to apply those guardrails for everyone in your company? Or how do you scale that? 00:30:57.05 [John Lovett]: Yeah, so great question. Remember, I mentioned that we’ve all had agency to build our own agents and experiment and do things. I am the vibe coding master. In fact, I was up till 2 AM last night just vibe coding because I was having so much fun. And so I just build these prototypes, and I build them for me, and I build them, and I play with them, and I see what works. I iterate on them. We just had our first release day. So release day was, we had, I believe it was, 11 agents that got released to the company. So these are somebody’s vibe coded that went to our engineering team, they stress test them, they productionalized them, they made them ready for the whole company to use. And that is where I get things like Twyman’s Law, the data guardrails into production for everybody. As a company, the engineering team has built out, we call it the CRMCP. The CRMCP has connections to our data sets. It’s got connections to all of our sales transcripts. It’s got connections to every one of Will’s presentations. It’s got connections to every one of our town halls. You can get so much information from this one. CR MCP, that that is how we productionalize things. We bring them as an organization to once they’ve been viped and tested and tried, we productionalize them by having it go through a relatively specific process. But everybody’s encouraged to bring the ideas kind of back to the original question is like, for my team, I built an AI innovation lab. And we’ve got weekly meetings, they’re optional. I usually have them on Fridays. And I’ve got a core team that prioritizes ideas and puts things on our roadmap for delivery. But everybody is allowed to bring ideas. Everybody brings challenges. And this is something I’m playing with, or this is how I’m trying to work through this issue. And so that just opens up the the door to possibilities and everybody trying things and everybody getting excited about what they can possibly do with AI that can help them. So that’s been a big lift for us as well. 00:33:01.01 [Julie Hoyer]: I’m curious because I’ve seen it so many times. I’m sure all of us have a new technology. It’s very exciting to use it. John, I’m curious how did you, because it sounds like you guys have gotten to the point where you’re very focused on problem solving, but like what you’re saying with your Friday meetings, right? Of like bringing ideas or problems, things they’re trying to work on. Like how do you keep your team or in the beginning, how did you get your team to really think of using AI as a tool to solve specific problems? Like is that a fight you’re still fighting? Do you feel like you guys are pretty mature in that thinking? I’m curious how you got there, if you are. 00:33:34.26 [John Lovett]: Yeah. So it is definitely a tough one. I’ll give you another real-world example. So it was three weeks before the Olympics were about to start. I’m a huge Olympics fan. And I think it was a Saturday morning, whatever, scrolling on my phone. And I said, hey, what if I ran the world’s largest geotest to test a bunch of prompts and see what kind of data I can get back about the Olympics, that LLM model. So this is you typing in a prompt to chat GPT, perplexity, like all the different models and seeing what responses come back. And I developed five hypotheses. So like narrative persistence is one, like how long does a narrative stick before it changes? Temporal velocity, how soon when an event happens or let’s say somebody is awarded a medal, do the AIs pick up on that and see, I looked at social proof, does social presence make a difference with which how frequently athletes showed up in LLM responses. So I had these five hypotheses. And I basically, I wrote a blog post about it. I was so excited at all the state of collecting. And I put out a general thing to the team. I was like, hey guys, I’m the only person working on this project right now. Everybody’s invited. Come on, jump in. Like, give me some help. And it was crickets. Like everybody. Oh, I was going to be like, no one responded, right? Because like, nobody, everybody’s like, I got my day job. I got all this stuff to do. Like, and, and again, like this is sort of that, uh, I don’t want to call it apathy, but like, uh, I’m afraid of it. I don’t know how to do this. I don’t know what you’re asking me to do. I’ve never done that before. How do I do this? And so the Olympics started, I had three ways, pre-Olympics, during Olympics and post-Olympics. And obviously we’re in that post-Olympics phase right now. During the Olympics phase, Nobody responded to me. It had been three weeks, almost a month. And I said, you know what? I can’t have this. I went to four of my people on my team and I said, I need you to be a leader here. I need you to, here’s the hypothesis, it’s framed. All you have to do, I built an agent to be able to analyze the data. It had all my guardrails in it. It had the hypotheses. It had what we were testing and it was showing data and I was doing preliminary results. I said, I need you guys to log in here and either prove or disprove these hypotheses with the data. And every single one of the four people I asked was like, I will do that. And again, I had to think about how I was going to write it. I used my agents to help me, like what’s a persuasive message that people with not a lot of time are going to want to do this and adapt to this. And, you know, so I was thoughtful about it. Do you think part of it was like the just a bystander effect like you kind of ask everyone and everyone’s like and soon as you went to people individually prop could very well be in and again that’s part of you know I’ve I’ve got a big team out see everybody every day and so I’m reaching out to people that I see on zoom maybe once a week or just at big meetings and me reaching out and saying like, hey, I need your help on this. This is an important project for us. This is going to help us know what to test with regard to GEO. It’s going to tell us about how the models think and how we can use that to build tests and experiments and really understand what’s going on. Everybody’s on board. We’ve got to give all hands tomorrow and we’re not even done the analysis. I’m the fifth person. There’s five hypotheses and We’re all going to do five minutes on what are we learning so far? What have we found? Like, how is this going? And it was that little nudge, that personal touch, that reaching out directly that helped me get my team on board. Because everybody, you know, when you do ask that bigger question, there’s a lot of like, that’s, he’s not asking me. I can kind of shirk off into the shadows. and see who else will step up first. So that was a big one for me. And I’m super excited about the results. Like I’ve already found so many fascinating things just through this research already. So that’s something that maybe by the time this actually probably will be by the time this is published, you can check out the GEO results. 00:37:44.14 [Michael Helbling]: Yeah, I like that. I find that for certain people, they just jump in and start doing their own AI process. And other people need AI defined into the process for them. And I think that’s sort of where I’ve seen, like I was talking to somebody recently, and they were like, I need my team to be doing this. And I was like, well, why don’t we set up a process where you take them through these steps? And one of the steps is you go do this thing with AI. And now you’re putting an AI enablement step into the process, just making it part of the standard process for them. And it was like, oh, yeah, that’ll totally work. And so some people just need you to give them delay out the how do I do these steps even though like a lot of us because we’re the way we are with analytics and in curiosity and asking why we’ll go into the AI and be like I want to set up a series this I want to set up a process or so we’ll just start with the AI and work through. and learn as we go. And then other people need like, I need you to tell me exactly how to do it, but AI can be part of it as part of that. So it’s very interesting because I’m watching adoption like this too. And I’m sort of like, yeah, not everybody is just going to jump in with both feet. So how do I get them active? And that was one of the ways that we, we kind of thought through about, about that was just sort of like, okay, well, one of the steps is you go to the AI and you do this, this, this and this with it. And that’s how you get through the process. 00:39:09.28 [John Lovett]: I love that example, Michael. One of the things that sort of runs in parallel with that, when I use AI to do something, if I like, let’s say it’s an analysis, I’m just like, hey, I’m trying to understand why does this brand get a bunch of citations, but not a bunch of mentions? And I do an analysis, I get a good output that I like, I usually go back and forth with the AI, and I’m a, Maybe I’m a beneficiary of having lots of tools, but I’ll take a Claude output and throw it over to ChatGPT and I’ll say, what do you think of this? And it says like, oh, that’s pretty good, but you forgot these three things. And then I’ll read it and edit it and I’ll put it back in Claude. And I was like, hey, how about this? And I was like, wow, that’s really smart. Those are good ads. So I play them off one another. But the one thing that I always do, and this is maybe a limitation of tools in 2026s, I use Ninja Cat, I use Claude, I use Gemini, I use basically use everything, but I am so worried about losing my work. And so it’s like that whole thing like I didn’t even save your file and your computer stopped and you lost it. I tell it. So I hit context windows. We have like you hit your limit for the company this month. You can’t log in for 24 hours. As soon as I think that’s going to happen or as soon as I get an output that I like, I tell the agent, write the instructions so that I can replicate this and give those instructions to another agent and teach another agent how to do what we just did. That for me, I can then pick up and say, hey guys, I show my team, I built this analysis, here’s something that we do every day. Here’s an agent that will help you do this. you can ask it any question and it’s gonna guide you down this path of using the right information, asking you questions to be able to get to the right outputs that are gonna produce something that’s relatively consistent. And for me, that’s been a huge unlock because those people who are like, I don’t know what to do with this thing. I don’t know how to build an agent. They can certainly use an agent and they can certainly use something like that to help guide them through an analysis or really any type of workflow. 00:41:12.45 [Julie Hoyer]: How have your analysts felt? Because Moe, you and I have talked about this on previous episodes of the shift in work of looking at a blank page and you’re creating your own thing, your own work, you’re on thinking your own analysis, right? Compared to if you are using AI to give you something to react to, it’s a completely different process in your brain. So I’m curious, John, along those lines, What has been the reaction or the feedback from your team? If they’re using AI in these tools for an analysis, how have they liked, disliked, you know, pros and cons of using AI to start an analysis and they’re like checking the work. They are confirming what AI has found, things like that. 00:42:00.86 [John Lovett]: Yeah, I think it definitely happens. I would say that the first reaction of the team is, this is just slowing me down. I have to ask it all the questions, do all the things I would do the analysis for, and I have to go make sure it didn’t hallucinate or give me bogus answers. it does slow you down at first. And it does make you go a little bit slower to say, I’m going to question that. I’m going to be curious about it. The whole part of AI, is it going to do the job of the analyst? It can help to surface insights and get things. But if you don’t have the curiosity, if you don’t have a spidey sense for that number ain’t right, it will just give you junk. I would say that initially it does take longer to do things. And this might take us on a tangent, but what we’re doing for that today is My team still builds dashboards, right? We still have reports, and that is our source of truth. We get our data, we pump it through our tools, we use tools like Funnel, and we pump it to BigQuery, and we can do queries out of there. I find out the specific regex for the queries, and I replicate that in my tools in my agents so that when I’m doing an analysis, I can look at my dashboard and say, okay, LLM visibility rate is 42%, share voice is whatever. What does it say in the agent? And if they match, then I feel good about that. And I’m like, okay, this matches my source of truth. If it’s way off, I’m like, okay, why was it off? What was going on here? What was happening? We’ve tried to build in those things where we can say, let’s have a source of truth. It used to be for me, I would ask it a question and then I would go to GA4 or Adobe Analytics and like, all right, let me dig up this number. Honestly, like I’m so far out of those tools from the day-to-day perspective, I’d be like clunking around and be like, oh, how do I find, I don’t even know what explorer to build to get through this number versus having the conversational agent when I could just ask it things. My team was great at that. They would do an analysis and then they would verify a GA or whatever platform so we could see those two things. But having that source of truth and having that dashboard, I’m still gonna, we will still rely on those. AI isn’t going to kill the dashboard just yet, but I think it’s an important resource to have for that validity, for the data quality, for the ability to make sure it’s not, you know, you’re not getting AI slop. 00:44:37.28 [Julie Hoyer]: Have you guys then turned the corner where you’re seeing efficiency gains from your new process of using AI like in your day to day work. And then second part of the question, I’m going to hit you with two before I forget my second part of my question. On our International Women’s Day episode, Moe, you guys were talking about how maybe efficiency gains is like not the only outcome or great part that could come from AI. But right now, that’s what people are most focused on. So I’m curious, John, have you guys found the efficiency gains? And are efficiency gains the only positive that have come out of you guys integrating AI into what you do, or have you found other great things coming from? 00:45:20.29 [John Lovett]: Definitely, the efficiency gains are a big thing. For us, it’s been a lot about, hey, we’ve got this process that we do. It’s part of the workflow. Now when we do it, we can repeat it across client to client to client. That’s agency life, right? It’s like we’re repeating these things. We’ve got similar analysis, different data sets. That has definitely helped us move faster through these things. The structured prompts, the way that we build methodologies and the way I tend to take, hey, I built this once, give me the instructions to build it over and over again. That has gained us a lot of efficiency. And then the ability to upscale employees, right? So it’s like, I get a new team member, I get a contractor on my team. And I can say like, Hey, here’s an agent that’s already built, you can get up to speed much more quickly using this. So those have definitely been the case for efficiency. I think, I think the other thing, the second question, if I’m right, It was like, what else besides just the efficiency is that? 00:46:29.17 [Moe Kiss]: So, John, there’s a concept we talked about a couple of episodes and I keep it. It’s funny, Julie, that you mentioned it because I was going to bring up the exact same point. So, Jim Lysinki, I think is how you pronounce his name. He wrote a book, The AI Marketing Canvas, and he has this like quadrant thing and, you know, talks about internal productivity and that’s really where everyone’s focused. But the other quadrants are like internal growth. So, like, using tools to accelerate your workforce. Then there’s external productivity, which is a lot more of that customer service, how you can use it to have productivity gains that are for your users. Then there’s the fourth quadrant, which is really external growth, using AI to completely unlock new revenue streams. My observation is that everyone’s really stuck in that internal productivity quadrant. From what you’ve shared, it sounds like you’re also using it for internal growth. Is that a thing that you’re seeing play out where the productivity gains just seem to be the thing everyone’s so anchored on? The thing I also then want to understand is if you are having productivity gains, how are you measuring that? 00:47:39.66 [John Lovett]: So with, I mentioned earlier the horizon builds that we’re doing, every horizon build has assigned productivity metrics. Like how much time did it save? How much money did it generate? We have KPIs that we build. You guys know I like KPIs. So we got KPIs that we build around each one of those things. So we are measuring productivity in a number of different ways. I think with the growth, this is an interesting one because it is sort of a creeper. It moves more slowly than just the productivity gains. But one example I’ll give to you. So I mentioned that we record all our calls. We ask our clients, can we record these calls when they allow us to? We do. So we’ve got all these transcripts. And so my My head of BD comes to me and says, hey, I’ve been talking to this prospect actually since October. And we’ve had a dozen conversations. I built a notebook LM that contains all the transcripts, contains what we talked about. And now, here we are in January or February, and they just asked for, we think we need to include analytics in the scope. And so everybody’s been talking about this project. We’ve got pricing calculators. We’ve got scopes of work. We’ve got all these things. And he basically said, I need you to get up to speed on this. And so I was able to use all of the resources, the transcripts, what the client wanted. I did get on one call with the client and talked to them and got to ask my very specific questions. But immediately after that call with having no prior knowledge, I was able to write a scope of work and my BD guy came back to me and he’s like, holy shit, John, you nailed it. Like, I can’t believe that you got that figured out in such a short amount of time. And we didn’t even talk about it. Like he just gave me the resources, but I was able to plug in and use my tools to be able to say, what does the client need? How does that match up, match up with my products and services? And then what can we offer them that’s going to fit what I heard in our conversation? And so that for me was a growth moment where I could say like, that really not only did it save me time, like it would have taken me months to get up to speed, but I was able to turn that around in like 24 hours and get something that was so spot on that my BDO was like, that’s amazing. And hopefully fingers crossed that, you know, that deal comes through, but it was just a good growth moment. That’s one example that I can think of there. I think the other thing I’ll just mention, you know, productivity is, Obviously where you wanna go, the part of this as being analysts, there is a whole new discipline and we call it geo, but it’s AI search, right? So it’s like, hey, we got all these models, people have questions, we’re moving toward this zero click world where it used to be, you’re ranked on a, on Google or Bing or wherever, somebody saw your link at the top and they clicked you, and they got a visit to your website. Now, your brand is getting surfaced via these LLM models. They see your brand, and they may not get cited. And they’re like, OK, I’m narrowing my list down. I’m seeing these things, but I’m not even clicking through. And I’ll just type in direct to get to that brand’s website. And so for me, part of this being that curious analyst and I’m collecting all this data and writing prompts and developing prompt methodologies, I’m like finding wild stuff. One example was, hey, in Claude, we see mentions, which is like your brand is mentioned, and then citations, which is the link to, you know, whether it’s a podcast episode or your resources, whatever it is, like all of a sudden in Claude, all the citations dropped off December 1st. And I’m like, what happened? And just because I’m looking at all this data across all my clients, I was like, oh, Claude really did stop using citations at that point in time and just cut it off. And then the other example, I’m analyzing data. I’m trying to build a report for a client. And I see the data went back to December 15th, 2025. It was the week that ChatGPT announced they were gonna start having ads in their free accounts, right? So you won’t have them on enterprise or paid accounts, but the free accounts are gonna start getting served ads. And I actually looked at the data and I was like, what is going on here? I started to see the nature of the responses change to purchase intent-driven responses. they were actually preceding the way that chatGPT responses to same questions we were asking like months ago, they were changing this dynamic. And I saw all these changes in the responses. And I was like, something’s going on here. And I connected it to the fact that they just introduced ads. They are prepping, they have been prepping for like a month and a half, you get ready for this. So I mentioned that because There is so much to learn about how these models operate. And it’s kind of like going back to like search days when you’re trying to understand the algorithm, like we’ve got this whole new field of who knows what the hell’s going on within these models is up to us to, and if anybody tells you they do, like I’m calling bullshit on that because All we can do right now is experiment, test things, try things and see what works. Honestly, in my 20 years in analytics, this is the most fun I’ve had because I’m learning new stuff, I’m playing with new tools, I’m getting to see all these things. For me, that’s growth. I am growing as a professional because my tool gets expanding, I’m learning all these new things. I can tell clients, if I can surprise and delight a client by saying, look at something I found about you. I’ll just give you one example. My client said, hey, this blog post just popped. Over the summer, it started ramping up and ramping up. We get all this traffic to it. It’s amazing. We’re so stoked on this. And on the call with the client, this was like a regular status call, I looked up their prompts and their geo-reporting, and I found the URL that they had referenced. I said, wow, this is amazing. The last two weeks, you guys have gotten 102 citations on this particular blog post. But yet all of those responses, nobody mentions you. You’re never mentioned in this set. And the category was like risk management. They had become this authority on risk management that enterprises were citing, brands were citing, you know, forums were citing, all these people were citing, but their name never got associated with that because it just wasn’t in there. And we developed a simple task for like, hey, just don’t do it in a pitchy, salesy way, but just insert your brand name into You know, here’s a description of what this is, and by the way, our products solve for this. In your FAQs, I had a big FAQ section in the blog post, I said, hey, just enter your brand name as, you know, when you’re closing it out, say like, we do this, and here’s the product that delivers this. Within two days of that test, mentioned started showing up in AI Overviews, which is one of the fastest to pick up on changes to websites. And I was like, boom, proof point right there. I got the first signal where it was like that change that they made to their content pages produced a mention which had never been seen before out of like six months of testing. And so, you know, that was only like a couple of days into the process. So when I can show a client like that, and then I did another analysis yesterday morning and I’m like, okay, signal strong. And it led me down this whole other rabbit hole of like understanding mentions and citations. And I learned about ghost citations, which I won’t get into. But like, this is the fun stuff where it’s like, we’re curious analysts, we’re trying to figure stuff out. And never before has there been this playground of like so much data, so much information. that we can just dive into and show clients things they’ve never seen before. And that for me is, I think that’s the growth that, you know, it’s gratifying, it’s fun, it’s exciting, and it’s definitely keeping me going. 00:55:25.19 [Moe Kiss]: Oh my God, John, this is like positively infectious. I love it. 00:55:28.76 [Michael Helbling]: Yeah, I know. I’m trying to remember a time when I’ve seen you like this fired up, John, honestly, like I’ve known you for a long time. This is great. 00:55:37.35 [Moe Kiss]: But so like you’ve had, I guess, I’m going to say the privilege of approaching this as a leader who then is trying to like bring your team along. There are lots of folks, I would say, who probably have the same level of enthusiasm as you, but might not be in a leadership role. What advice would you give that person, that mid-level analyst who’s doing all this playing, they’re having lots of fun, but how can they have a ripple effect on their business if they’re not in a leadership role? 00:56:05.44 [John Lovett]: Build something cool, show it to somebody. Show it to your manager, show it to your boss’s boss. If nobody picks up on it and you think it’s brilliant, share it on LinkedIn, share it on Measure Slack, share it somewhere, get some feedback on it. And if you get that positive feedback where people are like, this is cool, we’ve actually done this with our blog posts. at SEER, we’re almost not allowed to write a blog post anymore until we’ve seen something on LinkedIn, like see any of your reacts, see if you get any comments, see if you get any mentions on it. So test it, like play with it, put it out there, see what you get as a response. You know, and this may be harsh, but if you’re an organization and you built something that you know is productive, adds growth, is clever, is adding to what you do, and your leadership doesn’t recognize that. I’d be time to look for new leadership, but it is hard. I would just encourage people to experiment with things, build things within your boundaries that you’re allowed to do, and then share. put them out to the world. And if your leadership won’t listen, take it to LinkedIn, take it to Measure Slack, take it somewhere that you can find an audience that thinks that’s cool, and you’ll grow your brand that way, you’ll be able to find your people, I guess. 00:57:21.29 [Michael Helbling]: Yeah, that’s great. All right, we do have to start to wrap up, unfortunately. This is so good though. All right, well, one thing we love to do is go around, share a last call. AI is never going to change that. Well, maybe it will, I don’t know. But John, you’re our guest. Do you have a last call you want to share? 00:57:39.20 [John Lovett]: Well, I have two quick ones, but I guess I need to ask permission. Am I allowed to offer another podcast? Yeah, of course. 00:57:45.93 [Michael Helbling]: Yeah, come on. Do you think we follow rules around here? 00:57:49.53 [John Lovett]: I would bring it anyway, but the artificial intelligence show is a podcast. It is run by, I want to make sure I get their names right. Paul. Paul Ritzer and Mike Kaput. And every Tuesday they put out a podcast and they aggregate all the most recent AI news. And it’s brilliant. My wife actually loves listening to it with me in the car. We listen to it a lot. And she’ll talk, she’ll be like, oh, their voices are so soothing. But just great intel, great information. This is also the company. I think their company is changing brands, SmarterX. They were on the MACON conference in Cleveland, I wanna say. Cleveland, yeah. Yeah. Yeah. And Julie, you need to get there because it’s all in Cleveland. Yeah, it’s a great conference. But that is also the training that everybody at SEER was required to take was piloting AI. So great resource they’ve got. They do a free one-on-one training on a bunch of different things once a week where you can tune in and ask questions. But just a fabulous podcast, a fabulous resource, definitely worth checking out. And then my second one, a very quick hit. I encourage everybody on LinkedIn. There was a community that started, and I just happened to see it that was called the Geo Community. And Geo stands for Generative Engine Optimization. Some people call it AI Search. Some people call it all sorts of different stuff. I just happened to be, I saw it. I was like, that seems cool. And the first couple posts, were very intriguing to me, and I started commenting on it, and all of a sudden, I want to get a Rohit thing as the founder, and he’s like, hey, would you want to be an admin on this and join me in kind of managing this community? So I think we’re only a couple of hundred people strong, but if you’re curious about Geo and all that stuff, I got super excited about learning. Check out the LinkedIn Geo community, the Geo community, good resource to get up to speed. 00:59:48.66 [Michael Helbling]: Nice. Excellent. Okay, Moe, what about you? What’s your last call? 00:59:53.42 [Moe Kiss]: Well, I am quite excited. Probably not as excited as John, but my good friend Eric Weber is… back writing, and I’m super, super pumped about it. So he has a great blog from Data to Product On Substack, and I get it via email. The latest one was the Conundrum on Buy versus Build, which is something I always am super interested to read about. 01:00:18.51 [Michael Helbling]: Awesome. All right. Julie, what about you? What’s your last call? 01:00:22.57 [Julie Hoyer]: Okay, my last call is totally not AI industry related at all. My life the past few months, you know, I’m just trying to keep my eyes open in the middle of the night with a little baby. So I’ve been doing a lot of reading. So my last call is I went down a path of reading some historical fiction books. And I read one that was really good. So if anyone’s looking for some new reading material, a little break from AI news, maybe, you know, switch it up. It was, and I know this is popular, but it was codename Helene. And it’s by Ariel Lohan, if I’m saying her last name right, but either way, really great book. It is about a British spy going to France near the end of World War. So it was a really like interesting take, a different storyline that I had not really read about. And it was just an awesome breed. 01:01:16.18 [Moe Kiss]: Julia, I’m going to sidebar you after this and send you the name of an author who’s written like six books very similar to this. I’m going to read this one, but I’ll send you mine too. 01:01:22.79 [Michael Helbling]: We got a whole other podcast going here. Yes, I have a last call. So we heard about this and we’re kind of think it’s really cool. There’s a new visual data visualization contest, but it’s for children. So if you have a kid between the ages of seven and 12, there’s two different age groups. I know. So not everybody’s kids fit into that category. 01:01:47.26 [Moe Kiss]: I can fake, he’s tall, I can fake his age. 01:01:49.93 [Michael Helbling]: Yeah, whatever you wanna do. It’s like Aussie age, you know, it’s different. It’s, yeah, the conversion. All right, anyways, we think it’s a really cool idea. There’s some really great advisors behind it, but they’re doing a data for kids visualization contest. And it opens, the contest opened literally yesterday before the show comes out. So there’s still time right now. You can go jump on their site, we’ll put it in the show notes and you can check it out. But if you have a kid in that age group that’s really different age brackets, I think between seven and nine and 10 and 12. And so you can kind of work with your son or daughter and just come up with a cool data viz together and might be a fun little project. So anyway, that was my last call. All right, John, what a pleasure. Thank you so much for coming back on the show. It’s so good to talk to you. It’s been fun. It’s been great talking to you all. It’s, yeah, and I know we’re gonna see you at Marketing Analytics Summit, right, in April, so. Can I do a team look forward? Am I allowed to do that? Yes, of course, yes. 01:02:53.57 [John Lovett]: Absolutely. The extra day till Thursday, I am doing a half-day workshop on conversational analytics and how you can connect your GapGPT LLM of choice with BigQuery or Google Analytics, and so you’ll see it live there. Nice. At the Marketing Analytics Summit in Santa Barbara. 01:03:10.15 [Michael Helbling]: It’s funny, John, your blog post inspired me to create my own conversational analytics integration with Google that I built myself. Because I was like, hey, I should try to build something like this because, you know, I read your blog post and I was like, yeah, this was pretty, pretty cool. And I made some cool things out of it. Anyways, so I want to kill you. And it didn’t kill me. And I’m okay. I did stay up until two o’clock in the morning one time working on it, but that’s the fun part, I guess, you know. No, but you don’t have to stay up until two o’clock in the morning to come to Marketing Analytics Summit. And there’s a couple of really important things about that. One is it’ll be April 28th and 29th. So John will be there, we’ll be there. And we want your questions. We actually have a survey live right now. You can go to analyticshour.com. IO slash listener. Did somebody get that right? Yes, listener. And take our survey. And then you can submit questions that we will answer on the show, hopefully. So that’s kind of out there right now. We’d love to hear from you what questions you have to answer live at Marketing Analytics Summit. So that’s coming up. And so don’t miss that. We also love to hear from you every other witch away too. So please reach out to us. If you’re doing cool things with AI, if you’re inspired by some of the stuff you’re hearing, of course we’d like to hear from you. Obviously, when talking to John, it sounds like John, you’re pretty active on LinkedIn. So that’s a great place to find you and follow what you’re doing and interact with you there. And then also in the Measure Slack chat group. And we also love to hear from you via email contact at analyticshour.io. So please reach out. And we have stickers and Tim loves sending them out. So you can ask for stickers too. So just send us a little note. All right, I know that I speak for both of my co-hosts when I say, no matter how AI is changing your work and no matter how you’re getting your processes rolled up, hopefully it’s being both efficient, driving efficiency and increasing productivity. But remember, keep analyzing. 01:05:20.12 [Announcer]: Thanks for listening. Let’s keep the conversation going with your comments, suggestions, and questions on Twitter at @analyticshour on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst. Those smart guys wanted to fit in so they made up a term called analytics. Analytics don’t work. 01:05:44.69 [Charles Barkley]: Do the analytics say go for it, no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 01:05:57.91 [Tim Wilson]: Tony? No. None of this in the outtakes. None of this. None of this. 01:06:01.92 [Michael Helbling]: None of this. Yeah, that’s fine. It’s yeah, that’s fine. 01:06:07.96 [Moe Kiss]: That’s my hopes. That’s my thinking face. Like, what do you want me to do with that? 01:06:12.49 [Michael Helbling]: Moee, I’m just it’s fine. And people know who we are now. If our listeners are like, I can’t believe they didn’t look engaged enough in this short video that they put on their website. I’ll be like, you know what? That’s why we can’t have lights things. 01:06:28.68 [John Lovett]: I love the images you guys are putting out there. They’ve been fun to watch. 01:06:34.74 [Michael Helbling]: Oh yeah, thanks AI Studio, Google AI Studio, Nano Banana Pro. I just, it’s, what’s hilarious is like, I don’t even have good pictures of all of us. I just grab random headshots and throw them in there and be like, make a picture of this. That’s pretty good. I don’t know if John, if Tim shared the video I created with Vio of him and I, Crip Walking, but we’re not going to put that on social media. 01:07:01.19 [Tim Wilson]: That’s in the, that’s in the slide channel. 01:07:03.87 [Michael Helbling]: Yeah, that’s in the slide channel. 01:07:05.41 [Julie Hoyer]: That’s the only reason I joined the, the Mass Life channel. Cause I agreed that there was some fun happening. 01:07:10.52 [Michael Helbling]: And I created an analytics power hour brain at 40 else that I’m holding while we’re doing it. So nice. 01:07:18.87 [John Lovett]: Nice. Oh my God. 01:07:19.29 [Michael Helbling]: It’s you, it’s like imagery wizard. Oh, oh no, John, you don’t understand like, I’m fully AI enabled at this point. Like, it’s a problem. 01:07:32.12 [John Lovett]: AI enabled the dangers. 01:07:33.34 [Michael Helbling]: It’s not good. 01:07:42.54 [Moe Kiss]: Rock flag and review your workflows first. The post #294: Adapting an Analytics Team to an AI World appeared first on The Analytics Power Hour: Data and Analytics Podcast.

March 17, 20261 hr 5 min

#293: Tool Selection and the Unhelpfulness of Feature Comparisons

The one rule about the Analytics Power Hour is that we don’t talk about specific tools. But that doesn’t mean we won’t talk about tool SELECTION! Jason Packer recently released the second edition of Google Analytics Alternatives, (also available on Amazon) and his approach in the book is very much not an RFP-like “check which features your tool offers” system. And his rationale for that seems just as applicable (to us, at least!) for any data platform selection, be it a digital/product analytics platform, a BI tool, database or storage infrastructure, or, well, you name it! Ultimately, the challenge is how to go about getting a reasonably strong understanding of the philosophy and historical roots of each platform being considered and then marrying that up with the foundational priorities and needs of the organization. Is that a lot harder than a feature checklist? Yes. But them’s the breaks. This episode is brought to you by Prism from Ask-Y—your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show (eBook) Google Analytics Alternatives, Second Edition by Jason Packer (use “APH” as a discount code!) (Paperback / audio book) Google Analytics Alternatives, Second Edition by Jason Packer The music league: join the Measure Chat Slack (join.measure.chat) and then join the #measure-music channel (Rock opera) User Journey – The Rock Opera (aka, “Universal Sunset”) (Podcast) Pivot (Article) The AI Analyst Hype Cycle by Marc Dupuis (Video) Why You Should Fail 15% of the Time by David Epstein (Article) What If We Don’t Need the Semantic Layer? by Jacob Matson MeasureCamp NYC Go to analyticshour.io/listener to submit a question for us to (potentially) answer when we record at Marketing Analytics Summit! Photo by Alexander Schimmeck on Unsplash Episode Transcript00:00:05.76 [Announcer]: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. 00:00:17.05 [Michael Helbling]: Hi everybody, welcome. It’s the Analytics Power Hour. This is episode 293. Okay, listen, we draw a hard line of this show. We don’t talk about tools, but we never said anything about tool selection. And let’s be honest, we have all been there trying to figure out which vendor to go with after putting in tons of effort into our carefully crafted spreadsheet with all the selection criteria, which somehow every vendor says, yes, they can absolutely do all the stuff on there. It’s enough to make a person cynical. And we analysts don’t need help with that. So take a pause from reading the cold sales emails from the latest analytics AI SAS vendor. And let’s talk about the ins and outs of selecting a tool. But first, let me introduce my co-hosts, Tim Wilson. Or as I like to call you, Tim, tool selection, Wilson. No. How are you doing, Tim? 00:01:11.97 [Tim Wilson]: I’m just about ready to select a new podcast recording platform. Oh, perfect. 00:01:18.30 [Michael Helbling]: That’s going to probably trigger a bunch of inbound emails. All right. Moee Kiss, how are you going? I know you do a lot of vendor evaluation and selection in your role. 00:01:31.25 [Moe Kiss]: I certainly do. I’m very pumped to talk about this. I think the only thing you missed is the like, oh, don’t worry. If we can’t do it yet, it’s on our roadmap. Oh, yeah. 00:01:40.86 [Tim Wilson]: That happened yesterday to a client. They turned to the vendor and they’re like, yeah, we can’t do that. And the response from the client, which was a very large company was like, well, is it on your roadmap? It’s on our QN plus one roadmap. 00:01:57.02 [Moe Kiss]: Yeah. 00:01:59.38 [Michael Helbling]: All right, and I’m Michael Helbling, and we wanted to bring on a guest, and we found a great one. Jason Packer is the founder of Quantable Analytics. It’s an analytics consultancy focused on analytics engineering and implementation. He’s also the author of the book, Google Analytics Alternatives, now in its second edition. And the genius behind the Measure Music channel on the Measure Chat Slack group. And now he is our guest. Welcome to the show, Jason. 00:02:24.65 [Jason Packer]: Thanks, Michael. I’m really happy to be here. It’s a bucket list item to finally make it on the podcast. 00:02:30.44 [Michael Helbling]: Well, it’s awesome to have you. All right, so maybe to kick things off, Jason, maybe just walk us through sort of what brought up the idea and was behind the idea of writing the book in the first place. 00:02:42.55 [Jason Packer]: Yeah, so I’ve always been really interested in evaluating software and knowing what’s out there, even back to my early days as a Unix administrator and software developer. I liked looking at all the different tools and back in the era when the Google Universal Analytics Universal Sunset was coming up. There was a lot of people that were asking these questions. There were a lot of people asking me these questions. And so I thought, well, I may as well start doing this research. Seems like a fun thing to do. And I started out thinking, well, maybe I’ll write a series of blog posts. And then someone at Columbus at the time, Web Analytics Wednesday, said, well, why don’t you just write a book, Jason? And that seemed like a good idea to me. And so I did it. And now a few years later, there’s some, you know, things have changed. There’s some new tools I wanted to look at. And I thought I would just, you know, make the same mistake again. So that’s here. Here we are. 00:03:47.46 [Tim Wilson]: Wait, who was it? Who was it? It didn’t last Wednesday. Who said that? 00:03:51.94 [Jason Packer]: It was Ahmad. Ahmad. Oh, OK. Nice. Which I think I credit him for in the first book, at least for the for the idea. 00:04:02.15 [Tim Wilson]: Look, I read it. I didn’t memorize the acknowledgments. Jeez. Come on, Tim. 00:04:08.73 [Moe Kiss]: But it sounds Jason like a big part of your process and like understanding the capabilities of the tool is like really playing with it, right? And I think one of the things that I’m often thinking about is like, I see folks trying to evaluate tools without getting their hands dirty. And so like, Do you think that’s what everyone should be doing, or is that just the thing that’s always worked for you? 00:04:32.16 [Jason Packer]: Well, I think everybody loves to have an opinion about a tool, and it’s very easy to form an opinion. You get in there, you see how it looks and how it feels, and that’s fine. I have opinions about that too, but you really have to balance that against really learning what the tool is about. And for me, the way to do that is to use it and to use it with real data, not to use it, not to watch videos about it, not to be walked through a demo by somebody, but to install it on a website, even if it’s just a trivial website. install it and use it. And that’s how I learn best. That’s how I learn most quickly. And do you think that using it with real data? 00:05:17.98 [Moe Kiss]: The bit that I’m taking away from that is it helps you understand it. But how do you think it changes the evaluation process itself? 00:05:28.04 [Jason Packer]: I think using real data will show you a lot more about where the issues are. For example, if you’re working with a vendor and they walk you through it, they’re going to show you the highlights. They’re going to show you the things that work well. They’re going to show you a tool that’s completely, perfectly set up. and we all know. That’s not how it is. In the book, everything that I evaluate, I used on real websites with real user data. For example, one of the issues with those real websites is one of them had a terrible bot problem. It was a site that I bought on the secondary market. I didn’t make the website, I just bought it. you know, it had some real traffic, but it was just like, you know, littered with bots. And so the traffic looked really weird. And like, there was all kinds of strange hits to pages that weren’t there. But that led me to learn a lot about how, you know, these different tools worked in the cases where there’s a bunch of 404s or there’s huge amounts of bot traffic. So like, that’s the difference between no vendor, whatever, like, show you a demo where 90% of the traffic was bots. That’d be crazy. And in some ways, it can be challenging to do that because that might not be your use case. So a lot of the things I talk about in the book is use case match. That’s the challenge as a tool evaluator is to match your constraints of your use case to the best match of a tool. Like I said, opinions, everybody’s got them. And there are in some ways in which some tools are more technically advanced than others, or some tools are faster than others or whatever. But it’s really about matching use case to tool through the lens of those constraints. 00:07:30.86 [Tim Wilson]: Back to the using the actual data, so the book was kind of digital analytics, product analytics stuff. I would put BI platforms in there, put data warehouse platforms. All of those when it’s like, you want to try it with your data. I mean, a really high bar or a real challenge seems to be, we want to do a bake-off or we want to do a proof of concept. We want to try it out. I’ve gone through processes where it’s like, we’re going to do the RFPs, we’re going to select some finalists, we’re then going to do a bake-off. And that does mean you’re fundamentally doing some sort of mini implementation and trying to draw the line of, you know, and that can include getting through some compliance hurdles to say, yeah, we’re using our real data, or do you say, well, we’re going to dummy up or we’re going to do an effort to make it’s kind of like our data, but it’s been anonymized to the point that it’s not our data, but it’s still mimics our data enough that we could actually try it in this platform. It does seem like companies To me, that’s what motivates a lot of the not wanting to go through that process. 00:08:48.93 [Jason Packer]: Ideally, it would be great to use your actual data and to do, like you say, a real mini implementation, but that’s just not feasible in a lot of cases. 00:08:56.60 [Tim Wilson]: I mean, Moe, have you done that? 00:08:59.32 [Moe Kiss]: Yeah. I’m not going to bait around the bush. I do a lot of… like analysis of different vendors and different tools and that sort of stuff. I would say I definitely lean towards the, we should do multiple POCs. Like the last major tool selection we did, I think I wanted to do maybe four POCs. And obviously like that’s a negotiation with the business and capacity and things like that. We ended up agreeing on two. But I think the thing that I found really hard is like, often the folks doing the evaluation and the assessment and those sort of things. I don’t know if the incentives are always there to do multiple POCs. I find that hard to reconcile with because it is. It’s really hard to understand how good a feature is or a particular capability that you’re looking for without stress testing it. And yeah, I don’t know if I just air too far on the POC side maybe. I think folks internally would probably say I do. 00:10:04.27 [Jason Packer]: I think that’s really challenging, right? Because a POC is great, but even before you want to get to that POC, you want to feel like you’ve narrowed it down to something that’s worth the effort there. And for me, part of that can be not even doing a real POC, but doing a toy test. Oh, let me do it with my podcast website. Let me do it with my… a personal website or whatever. That’s part of the reason also why I’m a big proponent of free tier, even on enterprise tools. That can be a challenge, right? Not everybody can offer that. Sometimes, if you’re talking about a huge BI platform or something, what would a free tier even mean if it’s even doing a simple example, implementation means putting in 100 hours of work or something. But the ability to get a little bit into the product before you really start talking about committing company resources to it, I think, because I do love the POC approach and the more, the better. But it can be hard to get those resources for sure. 00:11:23.11 [Moe Kiss]: Also, just getting it through security is a really big step. You’re basically doing a procurement process for something that you’re running a PRC on. It takes a lot of time and energy, but I obviously am very biased here because I lean strongly on the side that that’s worth it. Yeah, that’s my lived experience, but yeah. 00:11:46.77 [Tim Wilson]: Well, Bill, have you run into, because I can see the doubt. Say it’s only two, you get down to two tools and you get in and you’ve got multiple people who are all trying it and they all have different things they most care about. And then you get to the end of that, and you’re like, all we’ve done is allowed people to dig their heels in further on their preferred tools, because now they have hard evidence that that other tool doesn’t do this thing that I think is really important, and it does this thing. Like, do you wind up saying, well, this is supposed, we’re hoping that we arrive at a clear winner, but even if you do a POC of four tools, They’re still not the one clear winner and you’re still in kind of a negotiating phase. And you’re also setting up the people who didn’t back the ultimate winner to be able to say, see, we did the POC and I told you we shouldn’t have that one. Sorry, that’s just depressing me. No, no, no. 00:12:45.80 [Moe Kiss]: I can still remember like a few years ago, we were doing a BI tool selection. It must have been like five years ago and all the data analysts got in a room and we like this, this was the absolute worst way to do it. I would never ever do this. But we were like, how important is this thing to you when everyone would go to one side of the room or the other side of the room? And almost every time I was on the side of the room on my own. And I think, so it’s suffice to say we did not pick the tool that I wanted to get, but it is what it is. I think the thing that I find so difficult about data tools in particularly, and I know we had Colin on previously, from Omni talking about how especially BI tools, you’re trying to be many things to many different people. And I think what’s so challenging about data tools is data folks have very strong opinions about the things that they do and don’t want to work with. But also their opinions are normally representing what is best for them and not always what is best for the business. And that’s human nature, right? You think about what’s going to make your own job easier. And so I think I I often come with this perspective of a data tool is actually for our stakeholders. So even if it’s a little bit trickier or a little bit harder for us in our day-to-day, is it going to help our stakeholders in their relationship with data be better? Because I will up wait that. But I don’t think that’s the common. I’m not sure that’s necessarily a common view. 00:14:13.11 [Tim Wilson]: Michael, what’s your relationship status with SQL? 00:14:20.42 [Michael Helbling]: Oh, I think you know it’s complicated. It keeps gaslighting me with a syntax error near from, like, I don’t know where from lives. 00:14:30.46 [Tim Wilson]: Well, here’s a healthier relationship. Prism by Ask Why. You ask in plain English? Prism writes the sequel. 00:14:37.02 [Michael Helbling]: Ooh, like Revenue by Channel week over week, excluding refunds. And instead of me crafting a 47-line query and a three-line apology, Prism just does it? 00:14:46.36 [Tim Wilson]: That’s right. The best part? It doesn’t forget everything the moment you close the tab. Prism’s jam of memory remembers your reality, your definitions, your quirks. I mean, not your personality ones, but, you know, your coding quirks. 00:15:00.86 [Michael Helbling]: Well, but like the BigQuery table is the source of truth and conversion means this and not whatever gets decided by somebody like mid-meeting somewhere. 00:15:11.01 [Tim Wilson]: Exactly. So you don’t have to re-explain your business context like it’s a bedtime story for robots. 00:15:17.69 [Michael Helbling]: Yeah, I have to admit I’m a little tired of starting every session with previously on analytics. 00:15:24.28 [Tim Wilson]: And when Prism generates SQL, you get traceability. You can track changes, see what was created, and follow the logic. 00:15:31.25 [Michael Helbling]: I like that, because when somebody asks me where this number come from, I can stop saying, well, from the number tree. 00:15:38.33 [Tim Wilson]: It’s like version control for your analytics brain. 00:15:41.34 [Michael Helbling]: I like it. A little bit of accountability, but it’s convenient. 00:15:45.58 [Tim Wilson]: That’s right, so do you want in? Go to asky.ai and join the waitlist. That’s ask-the-letter-y.ai and use code APH to go to the top of that waitlist. 00:15:58.78 [Michael Helbling]: I like the idea of letting AI write some of the SQL. 00:16:01.56 [Tim Wilson]: And let your memory do literally anything else. 00:16:06.29 [Jason Packer]: No, I think it’s not. And I think, you know, everybody also wants to work with the cool that’s good for them. Like personally, well, like, right, like this idea of sort of like you’re implying that like, hey, I want to work with the new tool. I want to work with the cool tool. I want to work with a tool that’s good for my career. I want to work with a tool that my LinkedIn posts are going to be, you know, go with. And, you know, that, that. A lot of times that’s not the right fit. It’s really about the whole organization, not just the analysts, but a lot of times the analysts isn’t even really the one. 00:16:44.96 [Michael Helbling]: Flip it around and people want to work with a tool they’re familiar with. I used this in my last job, so I want to use it here. 00:16:51.57 [Tim Wilson]: Which was good when GA4 came out and Universal Analytics got sunset, then it was like, well, nobody’s familiar with it. So reset, yeah. 00:16:59.78 [Jason Packer]: Yeah, that’s what I was going to say, too, is that a lot of times a tool switch is not the right answer. We all like to think, hey, there’s a tool out there. The perfect tool out there that’s going to fix my problems is going to make my personal life better, my company do better, et cetera, et cetera. But there’s no perfect tool. There’s no Grasses looks greener, but a lot of times the tool you have now just isn’t implemented correctly. The new one you get isn’t going to be implemented correctly either. That can be a real challenge too, especially if you’re like, hey, I want to do these Hey, we’re going to do two POCs and put in all these resources. In the end, we’re going to say, oh, well, actually, I think the answer is that we stick with what we got and we just spend a little more time trying to improve our reputation. Nobody wants that answer. 00:17:52.95 [Moe Kiss]: In your experience, talk me through when There are trade-offs, right? We’ve all said no tool is going to meet the brief perfectly. How have you approached balancing those trade-offs? What’s your thinking? And how do you, when you’re working with businesses, convince them of the trade-offs they should make versus shouldn’t? 00:18:11.59 [Jason Packer]: Yeah, it’s really difficult because how I evaluate the tools from the book is a totally different mindset than how I think when I’m talking to an organization. A lot of times, I won’t even really be talking about the same things. In the book, I talk about the underlying tracking structure of different tools, the databases that different tools use, how they work with consent, things like that. And when I’m talking to a particular business, I listen for what their real pain points are. Is this an organization that they just need to get off of GA because of compliance issues? And that’s like, Then I focused their selection on solving those pain points as directly as possible, but also trying to not get into the weeds with them about the details of the tools that the people listening to this might find interesting because they’re not going to find that interesting. 00:19:20.08 [Tim Wilson]: I think you just kind of mixed it because part of what you did and maybe it’s worth having you What I loved about both editions, because the structure stayed the same, is that the tool by tool, blow by blow, and it’s not a feature by feature, but the tool by tool kind of write ups are the second half of the book. The first half of the book is you got to have kind of a framework of what matters to you. You admitted throughout, you’re like, there is no perfect categorization, but you just talked about one of those was the tracking methods. I could see for the right company, they would say, we’ve been getting burned by our current tracking method and we have got to find something. You’re like, cool, well, let’s then think about the philosophical difference from the different tools. If somebody else says, we just need something super cheap, it’s like, okay, well, then let’s talk about the nature of your digital experience in the different pricing models. If somebody says, we just got to get off a GA because it’s compliance. We actually love everything about it. Our compliance team has said we have to get off of it. I would say in the example you just gave, it was how you approach the book. It’s just where you’re going deeper in that understanding what attributes truly matter and then going deeper, right? 00:20:51.82 [Jason Packer]: Yeah, I think actually that’s fair. One of the things I’ve talked about is how things like that’s all about constraints and how price is a constraint. Price is a real important thing for organizations. It’s not the coolest thing to talk about when it comes to tooling. Similarly, it’s just a question of how you’re engaging with the decision makers, I guess. are in that first half of the book are just a long list of the things that I think about. I might think about a bunch of those when talking to a particular organization about a tool. I might not be talking to them about all those things, but I’m certainly thinking about a lot of them. I think it’s important to understand them to a certain degree. For example, in the new edition, there’s a chapter on server side. Obviously, I’m not going to teach someone everything about server side analytics in a chapter, a 3,000-word chapter of my book that’s not primarily about that. understanding at least enough about that to know if you’re talking to a vendor when they say, oh, yeah, we support server side. It’s easy. This is what you do to be able to understand, interpret what they’re saying, to know like, oh, well, really kind of like anybody could do server side. It’s not really about the tool, it’s more about the deployment about, oh, are you using server-side GTM to deploy that? And if you are, then this, and perhaps the real underlying problem is tracker blockers or something like that. And then your lens for viewing that is different. So that’s why I think that the The first half of the book, the guide part of it, rather than the product evaluations, is the lens in which I look at all product evaluations, and I’m trying to share that viewpoint in the first half. Tim liked it at least. 00:23:08.83 [Tim Wilson]: Can I ask, and this is probably also a question for multiple people like you, and you said it in kind of some of the earlier discussion that you explicitly did not talk to the vendors, even though they were, especially after the first edition, they knew you were doing the second edition. And they’re like, come on, just let our sales engineer help you out. You know, once you just understand, and I think you did that to say, I want a level playing field and I need to finish this book at some point. And if doing 15 POCs is tough, letting their sales teams get their hooks into you would be absolutely impossible. Whereas, yeah, and so whereas Moe, I feel like if you’re down to a couple, the where does sales play? So I don’t know, maybe you can talk through that. 00:23:56.68 [Jason Packer]: I mean, yeah, that’s sort of an unusual choice that I make in the book is to like, I mean, I definitely have talked and I know a lot of really great people at a lot of these vendors, like, especially after the first edition, you know, I’ve talked to a lot of these, these people and there’s a lot of them told you what you got wrong. Not as many as some, some, yeah. But it’s important to me that I was really, really fair more than I was particularly making any value judgments or anything like that. But the not engaging with them is about loving the playing field to some degree. It also fits in well with how I learn, like describing the learning from doing. Again, getting a demo account or some kind of account where I can use the product is the fastest way for me to learn rather than being on sales and engineering calls. But I think that was my case for writing the book. That’s different than most case with engaging with vendors from a large org that has specific needs. I think that a lot like engaging with vendor reps can be really, really helpful, but it also gives you an idea too of the culture fit between the product and your organization, which is a real thing. Something that when I started the first edition of the book, I didn’t expect to be so important, but is, I think, quite important. 00:25:36.33 [Michael Helbling]: Do you hear that, Tim? Culture is very important. I just wanted to reiterate that point really quickly. Sorry, you’re going to mo. 00:25:45.99 [Moe Kiss]: Just to add to that, I have personally found that engaging with sales, engineering support, whatever, is like a really big part of the process because I want to make sure that we can learn from their expertise that we’re not facing challenges that are very easily fixed. And I think part of then, even in the playing field, is making sure that you get that with all the companies that you’re PGO seeing. It’s not a favorites game. And you’re so right, Jason. Such a big part of it is about the culture or the ways of working that you then get to explore with that other company. And very transparently, I’ve talked about our relationship with Snowflake quite a bit and a big, big part of our success. I will rail on about implementation for years to come. But a big part of it is we’ve had really close relationships with their product teams, with their product managers, their tech leads, we will have calls like testing out new features and new functionality and being able to influence a roadmap. That is a huge, hugely important thing for us when we’re doing vendor selection because we want to make sure that in a year’s time, we have the kind of relationship where we can push their product if we need to. And so I think that letting those folks in the room so that we can stress test each other is a big part of the evaluation for me. 00:27:12.97 [Jason Packer]: Yeah, I agree with that. Again, it depends on your organization and why you’re buying the thing to start with. If you’re a tiny startup and you’re not really going to if you’re the thing that I hate is like you’re you’re a tiny startup and you’re you’re talking to an enterprise software provider and you know you get to the point where okay we’re ready to actually like talk some real prices and like okay well you know start for for your volume data we’re starting out with $65,000 a month and you’re like that’s my what are you talking about that’s like my entire yearly budget for all of my analytics so you know it I love transparency. I make that pretty clear book. And I think that like, you know, that’s just a great thing to get people on the same page as quickly as possible, because that’s super important. And I think that like, When you are engaging with the vendors, being transparent with them helps everybody. Nobody wants to seem like a dummy when they’re talking to a vendor, but if it’s a new tool, I don’t know the tool. They know the tool. They know they’re immediate competitors far better than I will. I try to be very direct about, hey, the budget is this and here’s my seemingly very stupid question. When you gave me answer, I didn’t understand, I’m going to just ask that stupid question again because it’s important to everybody that we find the best fit in the most direct way possible. 00:29:03.33 [Moe Kiss]: And I do think, obviously, I come from a place of absolute tech privilege. I think a lot about what are the hills. We say that all the time. We’re like, well. Well, anyway, I just want to be conscious of other folks have very different budget constraints when it comes to tool selection and things like that. But there are things I will die on a hill for. And one of them is, I do think, obviously, budget is incredibly important. But if there is a very good tool and it is not 10x, like other options, but it is a good fit. I personally think that is a fight worth having with the business, like getting support for that extra budget to make the right tool decision versus being so constrained by it that you make a really, really shitty choice. And again, everyone’s not in that position, but that situation you just described, Jason. Knowing the prices much earlier in the process is absolutely something that folks should be doing. You can’t wait till you’ve done a POC to start getting an idea of their pricing because if it is way out of the realm of possibility, you don’t want to waste your time and energy on it. 00:30:12.54 [Jason Packer]: I guess I haven’t thrown Google under the bus yet. There we go. Here we go. Yeah. One of the things that I think Google really made hard for is that they made a, with Universal, they made a pretty darn good product, and they made it free to just an incredible degree. It was technically free to 10 million hits a month, and in reality, it was quite a bit higher than that. And with GA4, of course, there’s no hard event limit. the one million per day export limit to BigQuery is probably the thing that people hit first. But they’re giving away so much for free. And that’s really caused people in the industry to think that analytics should be basically free, that the software should be free. And it’s very distorting. things really hard for new tools to come out there and to get a foothold in the market. It makes what Moee you’re saying as far as, hey, we need to understand that even if this tool is a little bit more money, think of the cost in people, the cost in data decisions in the organization. you’re really undervaluing analytics, and part of undervaluing analytics started with VA being free. And that’s still happening. 00:31:53.35 [Moe Kiss]: Oh, Jason, I feel like we could sit around and like… chat for hours because I think fundamentally one of the biggest mistakes I see is, yes, folks want to work on cool shit to put on their resume or LinkedIn or whatever, but it’s also the open source fallacy or the free fallacy, which is like, oh, this is open source or it’s free, it’s not going to cost us anything. I will push pretty heavily on like, that does not mean it’s free. We need to actually think like, we’re talking about a solution here that has five full-time engineers supporting it. That is not free to me. That is actually a huge cost to the business. And if we want to do that because we think that’s the right decision, that’s okay. But that needs to be a line item in our decision as well, not just the like on paper cost of the tool. 00:32:39.25 [Tim Wilson]: That also then extended if we’re going to have to support and we have those five engineers and one of those engineers leaves, what’s the size of the pool of candidates that we’re going to have to replace it, which is one of those where Market leaders and whatever tend to have a leg up, and it’s a legitimate leg up. They’ve achieved some critical mass. Nobody got fired for buying Tableau or Power BI. Part of that is because everybody’s been exposed and is familiar, but it’s also legitimate saying, well, if I need a Power BI developer, That’s a much larger pool to draw from, right? 00:33:20.45 [Michael Helbling]: Yeah. When Moee excitingly get ready for a new wave of that with AI, because now people are going to be like, it’s free. We can just build it with AI. It’s a question to Jason, sort of from your perspective, because obviously, we’ve all kind of been through vendor selection processes and we kind of touched on how Google Analytics is free. So obviously, as people are sort of jumping on the AI bandwagon and seeing how easy it is to prototype things, not necessarily build full-on products yet, but we’re moving in that direction, I would say. Do you think that’s going to be something that will enter the process of the build versus buy debate certainly changes a lot in the future? 00:34:08.82 [Jason Packer]: Yeah, I think so. I think that’s already happening. I think that it’s happened not exactly with AI, but the simplified realm of these tools like the In the book, I call them simplified web analytics tools, including things like plausible and fathom. Um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um And there are so many of those tools out there now. Every few months, a new one comes out. Some of them are quite good. It’s not a hard thing to prototype, and it’s even easier with AI. I could go in and especially if you start with the ones that are open source, you can be like, hey, build me an Umami clone. Here’s the Umami GitHub repo about it. And you could build yourself something like that pretty quickly. And some people are doing that. I think you run into some of the problems that Tim was talking about as far as having expertise with the tool. If it’s some internal tool, then the internal people are going to be the only ones that have the experience with it. And also, when it comes to some of the more complicated underlying database things, that’s I think far beyond the complexity of what AI can do a good job with. And so, you know, like, and also things like schema AI doesn’t do great job with, I mean, it can do okay, but it needs a lot of like, you know, human hand holding. So I think that like, ultimately, it’s like, a mistake for most organizations to try to think that they can build their own when there are so many really great platforms already out there. I think it’s maybe fine to think like, oh, we’re going to extend. We’re using whatever. We’re using Postog. And Postog is an open source tool. It’s one of the widest tools in the market. They have 34 different apps built into the tool. of whether you want session recording or feature flags or whatever, or LLM analytics, they’ve got it probably. And if there was something in there that you need to add to that, then using AI on top of that, I think to extend it, it makes sense. But trying to build a foundation to your analytics platform, your analytics practice, without really having that real strong attention to detail that these platforms that have been out there and tested have. It doesn’t make sense whether it’s, you know, humans building it. or it’s even worse if AI is building it. But I think it makes a lot more sense to build on some of the great tools that are already out there. 00:37:26.55 [Michael Helbling]: That gave me a cool weekend idea though. The mommy club coming out. 00:37:30.72 [Jason Packer]: Yeah. 00:37:31.04 [Michael Helbling]: Yeah. Just because you can. I don’t know. 00:37:34.58 [Tim Wilson]: But as you brought up post-hoc, because that’s one that I’m not familiar with, but you mentioned them because they kind of, well, actually, I think I sat and watched you talking to a different vendor and you brought them up a lot as being kind of like, who was the tool built by and for? Posthog is like, of by and for the developer. Google Analytics, Universal Analytics was kind of intended to be for the more casual initially, and they tried to stick with it. They’re like, this is easy. This is for the marketer. BI platforms seem like they’re similar. If you’re in the Power BI, you’re in the Microsoft stack, you’re like, this is built for the enterprise that wants to have the complete ecosystem and progression of tools. What do you think you’ve said here? You definitely said it in the book. What is the philosophy? What is at the What’s in the DNA of the company that’s building it? Who do they feel is the user that has primacy? Is that a fair… Absolutely. 00:39:01.22 [Jason Packer]: You and I have talked about that. Product philosophy and outlook is more important than any sort of feature comparison. Probably some people have heard me complain about this before, but future comparisons are helpful in some ways, but they also are already outdated. By the time you posted your future comparison list, it’s already out of date. The one checklist item that it says it does X, There’s so much to unpack underneath that, that their version of X might not be what you really think that you’re getting when you get that feature. And people want more features, like I just talked about postdoc, they’ve got every feature under the sun, they’ve got, you know, They’ve got session capture, but maybe you already have session capture. You’re already running Microsoft Clarity or HotJar or something like that. So while that looks like a great thing on a future comparison list, that’s not something you need, or that’s not something that’s going to help you. It’s going to add confusion to the product. The more features you add onto a product, the harder it can be to use. That’s just the way these suites work. But it can be really hard at the same time to understand, to peel back that layer of marketing a little bit and be like, well, what is really this product philosophy? Who is this for? you know, when you’re in that job, when you’re, you know, when you’re an analyst and you get, I don’t know, whether you get in front of like Adobe Analytics workspace, you’re like, oh, okay, I get it. This is, for me, this was written by people that have been listening to people like me. This makes sense to me and is useful to my use case. That It’s clear and a lot of times once you get in and use the tool like I was saying before, but when you’re just looking at the marketing, it can be like, well, both platforms say they have the ability to customize reports like done, like they’re the same when they can be wildly different. 00:41:10.19 [Moe Kiss]: Talk to me about this philosophy piece because I find this really interesting, the philosophy of the platforms. I think maybe I was alluding to a similar idea before, but how do people figure that out? Is the philosophy who the user is or the direction they want to take it? 00:41:28.25 [Jason Packer]: I think it’s not easy because, in a lot of ways, I think the vendors themselves don’t know a lot of times. It’s a product of the history of the company. It’s a product of what the target market of that tool is, and it’s a product of the people that built it and who they were listening to when they built it. Let’s take, we’re talking about possible on some of the simplified tools, they have a clear philosophy of this is a simple tool. There’s not going to be vast ability to customize reporting. Everything is going to be on one screen or pretty much everything is going to be on one screen. We’re not going to have a drown you in configuration options. This is designed to be simple and part of that is in that as privacy as well, is that it’s you’re giving up some amount of complexity to make it easier and to perhaps make it more private as well. That’s a philosophy. And I think that philosophy for them is, can be pretty clear, right? When you look at their marketing, you look at the sample, you know, you try out a similar product. It can be pretty easy to understand their philosophy when they communicate it well. And it’s not, you know, a sprawling platform with 27 different components or whatever. versus some of the more complicated tools like I talk about piano in my comprehensive category along with tools like Adobe. If you’re looking at either of those tools, they offer so many different features and functionality. And there’s a much more complicated onboarding process that it can be really hard to understand what that philosophy is until you get much further along in the process. I do think that talking to the vendor and engaging with them before you get too far along can help you understand that, but it also can confuse the process too. I don’t know that I have a real great answer. 00:43:37.60 [Tim Wilson]: I like that because you said you have to sort of where the company like looking at the roots of the tool and I don’t have a million examples, but I look at like in the BI space, you had Tableau, which was like one of the second generation of tools that clearly was like The shit should be drag and drop, and we should be able to customize it to conform to things that Steven Fugh would give a 10 out of 10 to. They were coming at it saying, it’s got to be a drag and drop, WYSIWYG interface that you can have highly customized to be very, very clean visuals. So there were a BI tool philosophically, I think it was forward on the quality of the visualization. Contrast that with DOMO comes along a number of years later. And I would say DOMO was saying, no, no, no, it’s all about the ease of connecting to all of your data sources. And they kind of led with the connectors. Now, they’re competing with each other, so over time, their sales teams are saying, we’re losing Domo, we’re losing deals because our visualizations are shitty, and Tableau’s getting pushback of saying, we’re losing deals because we’re not easy to connect to all these different things. not necessarily permanently handicapped, but it is one that I would say both of those tools. That’s where their various strengths versus weaknesses are, which means if I’m looking at a BI platform and those two are in the consideration set, I may be thinking, do I have stuff going pretty tightly into most of my stuff is going to go into a data warehouse and occasionally it’ll be pretty normalized and I want to hook into it and occasionally I might want to hook into something else or are we going to just live in a chaotic world where I’m always going to be needing to hook into a gazillion different data sources that are all going to be messy and I’m going to need to be able to do transformation within it. I think it does take a lot of and maturity or wisdom or thought to try to map where is my company’s kind of, which philosophical or historical underpinnings are most aligned with my needs and then stand up and say, and guess what? That means our visualizations will never be as good as what the perfect idea would have because that’s a lower, 00:46:09.08 [Jason Packer]: Yeah, that’s where I think I talk a lot about understanding fundamentals, how that can be really helpful to close. Some of that you’re talking about is the gap between the marketing that you see from the vendor and the reality. Part of closing that gap and understanding really what a tool is all about can be understanding the fundamentals of how particular things work. If we’re talking about you know, databases, right? If we’re talking about this product uses MySQL and this product uses Snowflake and this product uses Postgres and this product uses Clickhouse. Knowing just a little bit about the differences between those two tools is going to tell you a lot about the product, you know, like if we’re talking about We’re talking about, say, we’re comparing PivotPro and Matomo. PivotPro uses Clickhouse as the database underlying their product, and Matomo uses MySQL. On the surface, they’re pretty similar products, but they end up working quite differently because of that difference in the underlying database. MySQL is a simpler database. It’s something that’s easy to self-host. It’s something that’s easy to see the raw data from. It’s something that’s not super performant in a lot of more complicated analytical queries. And all those things surface in the products. And if you know that background and you know that It’s not like you need to know how to use those tools, but just knowing a little bit. The same is true for like tracking methods like cookies, tracking with cookies versus tracking with this IP plus user agent method or tracking with browser fingerprinting or whatever. Just knowing a little allows you to sort of see like, oh, the vendor says X. Oh, I think what they mean is this. There’s not as much as the vendor might say, it’s not like there’s a million new things and a million new ways to do things. There’s a limited number of ways. 00:48:12.40 [Michael Helbling]: Before I wrap up, I’m going to give Moe the opportunity to jump in one last time, but we do have to start to wrap up soon. But yeah, go ahead, Moe. I know you want to ask one more. 00:48:23.25 [Moe Kiss]: Jason, we’ve talked about a lot of different concepts and things you need to think about in this whole tooling decision space. If I’m sitting at my desk and I just take like your one like absolute, this is the thing that should be most top of mind from all the things we’ve chatted about today. What would be like the one thing that you would say, just if you pay attention to this, then you’ll probably make a slightly better decision. 00:48:49.96 [Jason Packer]: Oh, that’s a tough question. I might actually say price. So I’m disappointed. It is. I’m disappointed. I’d like to say something cool like the fundamental database schema or something like that. It’s a shortcut to a lot of putting you in the right area. I don’t want to do, but that’s where I would go. 00:49:27.05 [Tim Wilson]: Price is one input to a total cost of ownership. I mean, that’s, again, maybe another one. Have you ever come at it that way, Moee, with any of your… That’s a better, you know, total cost of ownership. 00:49:38.75 [Jason Packer]: Let’s just say that. Say I said total cost of ownership of price. That’s what I meant. 00:49:42.51 [Moe Kiss]: There you go. That needs to be in version three of your book, because I like that framing. Total cost of ownership sounds way better than… I think I do use total cost of ownership. 00:49:52.13 [Tim Wilson]: I don’t know if… Maybe it’s 10… I mean, I think it makes sense if you’re going through it differently. Philosophically, how much am I going to have to invest in added tooling to work around a limitation in their tracking or something? It could be, but yeah. 00:50:07.46 [Michael Helbling]: All right. Well, we do have to start to wrap up. This is awesome conversation and honestly so It’s a good conversation, because I think everybody deals with this in some capacity in their analyst career. So Jason, thank you so much for coming on the show and being our guest today. One thing we like to do is go around the horn, share last call. It could be any topic, anything at all, just something that might be of interest to our listeners. Jason, you’re our guest. Do you have a last call you’d like to share? 00:50:38.79 [Jason Packer]: So my last call is something that you already mentioned, Michael, which is Music League. Nice. Michael and I, and I’m how I believe your sister is a part of this as well. Music League is a, like, It’s a competition sort of, it’s a friendly competition where every week somebody, like there’s a theme, like this week’s theme in the music league that I’m part of is Beatles covers. So everybody picks a Beatles cover that they like, then a playlist is made automatically from that, whatever, 20 songs. and everybody votes in the ones that they like and fun as hell. It’s not complicated. It’s fun to do with your peers, your friend group, your work. We’ve been doing it on the measure slack for what, three years now or something like that? It’s, I mean, it’s a lot of fun. 00:51:40.10 [Tim Wilson]: Is it in the measure? What? For somebody who’s interested, they have to be in the measure slack and then in the measure music channel, they can find it. 00:51:47.46 [Jason Packer]: Yeah, that’s where the conversation happens. You don’t technically have to be part of that. But anybody can start musically too. And there’s also like free 00:51:57.09 [Tim Wilson]: You know, like, yeah, but we like to do stuff around, you know, us. Don’t just get people to go out and do their own thing. 00:52:05.14 [Michael Helbling]: They got to be part of the measure slack to do this. So join that first. Obviously, top tier. Yeah. Yeah. And the group is amazing. Like, that’s also great. We have tons of cool, fun, music-based conversations with all your peers in analytics and, um, It’s a lot of fun. So for my own personal experience, it’s it’s a great time. And I’ve got a great idea, Jason, because, you know, we’ve been growing as we grow and then we get the big power hour bump on this now. We can start like different levels of leagues. So there could be like a Premier League with relegation and a championship league like like British soccer, you know. 00:52:44.59 [Jason Packer]: I think I would be relegated. I’m not sure I would like that. I do not. 00:52:47.60 [Michael Helbling]: Well, I probably would be too. I don’t often score very well, but I have a lot of fun. Anyways, it’s also really cool to get a new playlist every couple weeks or so of songs you might not have ever heard or genres you’re not that into. So it’s nice. I like it. 00:53:02.80 [Tim Wilson]: So we do occasionally get comments from people who are like, you guys mentioned the measure slack where it’s like, if you literally go to measure.chat and then you join.measure.chat. And we’ll also have it on the show notes page. So if anybody’s like, you guys keep mentioning it and you, and it’s in our outro and we don’t have instructions for how to find it. 00:53:22.46 [Michael Helbling]: So listen, if you’re committed, you’ll find your way in. All right. No, thank you. Yeah, that’s awesome. And Jason, thank you for kind of being the oomph behind that, as I know it’s a ton of work on the back end to make it work. 00:53:36.24 [Jason Packer]: The official commissioner. 00:53:38.54 [Michael Helbling]: Yeah, the commissioner. The ska loving commissioner of the Measure Music channel. All right, Moe, what about you? What’s your last call? 00:53:50.95 [Moe Kiss]: Okay, so my husband has been listening to a podcast for a long time that folks will probably be familiar with. I have noticed it indexes highly to men. I know a lot of men that listen to it. I don’t know a lot of women. 00:54:03.93 [Tim Wilson]: Joe, you’re wrong. 00:54:08.85 [Moe Kiss]: Sorry. the Pivot podcast. One of the, my husband listens to it on like loud speaker around the house and it like really like drives me nuts. And I have not been the biggest fan of Scott Galloway. However, I have had my opinion changed very significantly. I am now a listener of Pivot. I have been incredibly impressed with how they’ve talked about I mean, AI and like tech over the last few months, but particularly the coverage on the Epstein files is something that I just really, like it really impressed me and that’s why I’ve become a really big listener. Scott also last month did this like resist and unsubscribe initiative, which folks might have seen in the media, which was really cool, which was like encouraging folks to basically use our economic power to let tech companies know that we’re not happy with how they’re supporting the administration. I felt like they were using their voice to share their perspective on something in a really meaningful way. Also, just for everyone out there, checking on the women in your life, the last few months have been like shaken us to the core. And so just to just check in on your, your wives, your mums, your daughters, all the women. 00:55:38.89 [Michael Helbling]: Nice. Yeah. Great. 00:55:40.63 [Moe Kiss]: All right. 00:55:41.50 [Tim Wilson]: Yeah, great. Yeah, Tim, what’s your last call? What have you got? Well, there was this episode of the Rogan. No. So, I’m going to do two. They’ll be quick. One, David Epstein, who I’m a big fan of like his books, like his videos, but he did a 15 minute video called why you should fail 15% of the time. And he talks about desirable difficulties, which is a phrase I don’t think I knew, but he kind of breaks down the value of doing hard things the hard way and specifically what that does for you, which in the world of vibe being shit, there are a lot of people grappling with it, but he’s just a well done video and he’s delightful to listen to. And then maybe kind of adjacent to that, there was just an article, I don’t know, it was the metadata weekly, Mark Dupuis, the AI analyst hype cycle. And I just, there were some quotes in it that were just, I thought were gems, like quote, if AI can only answer questions that have been preconfigured by the data team in a semantic layer, what have we actually built? An expensive natural language interface to existing dashboards, which And he kind of makes the case of where is this all going? It’s narrowing down to where what you actually get is maybe not that great. But they also had the analysts who thrive will be those who can translate business problems into the right questions, validate AI output, build the context systems that make AI useful and provide the judgment. and recommendations that AI cannot, which I think a lot of people are saying, but that’s kind of like a cheap throwaway thing to say when I look what people are then also saying, I did this thing. It often kind of skips those components of it. The AI analyst hype cycle by Mark Dupuis is my second one. Michael, what’s your last call? 00:57:43.19 [Michael Helbling]: Well, we did an episode a while back talking about semantic layers with Cindy Hausen from Thought Spot, which was awesome. And we also did an episode about AI that I remembered something Moee said about how So letting AIs leverage how the queries are being used in the organization is also a way of training the AI to do that. And I read an article recently from Jacob Mattson at Moether Duck about rethinking the semantic layer and kind of challenging the idea that a semantic layer is kind of the only way to go. And I just thought it was a cool counterpoint. I don’t know that I’ve got a strong opinion one way or the other. I very much respect the conversation we had with Cindy and I really thought it was really powerful. But there’s some interesting research and discovery going on as well on sort of like letting the AI consume all your SQL queries and using that to help it understand some of the context behind where and how your data is getting pulled together. So anyway, it’s a good read, good to kind of think through those things. I don’t think we’ve solved it for our industry. So I think it’s early days on all this. So yeah. Oh, and what’s this breaking news? I’m getting word now straight from our correspondent. There is a book out there that Jason Packer has written called. What’s the name of the book again? Hold on. I haven’t written down Google Analytics alternatives. And for listeners of the analytics power hour, he’s going to give you a 20% discount. So that’s pretty sweet. If you haven’t already bought the book, that is the incentive to do so. Discount code APH. So there you go. We’ll put the link to that in the show notes as well. All right. Well, Jason, once again, thank you so much for coming on the show. This has been a lot of fun and a really good conversation. Appreciate all the work you’ve done. It’s a labor of love, I’m sure, just to do all this. And so very much appreciate it on behalf of a vendor weary industry, I think you’re doing us all a big service. So thank you. 00:59:50.85 [Jason Packer]: Thank you. You’re welcome. Yes. 00:59:52.19 [Michael Helbling]: Great time. All right. Well, we’d love to hear from you too, because you’ve been listening and you probably have questions or you’ve got thoughts. And so reach out to us and you can do that on the Measure Slack chat group, which we’ve spoken about on the show. as well as our LinkedIn page or via email at contact at analyticshour.io. And we also love to get your comments and ratings on whatever podcast platform you listen to. Please feel free to do that as well. And I think I speak for both of my co-hosts. 01:00:23.72 [Tim Wilson]: Boy, have you listened to a few things that are a little important. If only our show prep had it. So one, just know that Michael, you and Jason and I will all be at Measure Camp New York on the 28th of March, so if you want to see us. 01:00:39.95 [Michael Helbling]: That is true, we will. 01:00:41.09 [Tim Wilson]: But even more important. 01:00:41.97 [Michael Helbling]: I didn’t expect by now there’d be tickets left, so I was leaving that out because it’s too late, you probably can’t make it. Well, there’s something that’s available. 01:00:50.97 [Tim Wilson]: If you can get a ticket. Moere important from an operational perspective, the marketing analytics summit that we’ll be at on April 29th, Yeah. Okay, now you’re… Yeah, I did skip that. 01:01:04.63 [Michael Helbling]: I did skip that, yeah. Moere breaking news, I’m getting… Yeah, we’re going to be a marketing analytics summit and we need your help. We want your questions. We’ve got a very cool survey of which there’s an Easter egg at the end that I had no part of. And we’ll have to take the survey and ask a question to see it. But yeah, go to analyticshour.io slash listener and submit a question. We’ll be recording at the marketing analytics summit at on April 29th in Santa Barbara, California. And we hope to see you there. But if you can’t make it there, we can still ask a question and we may answer it on the podcast. So please do that if you want to ask us a question. And even if you don’t want to, push yourself a little bit and ask what anyway. I highly preference questions that make Tim feel uncomfortable. So like, you know, asking emotional questions about, you know, the best manager he ever had or 01:02:09.06 [Tim Wilson]: Yeah, luckily, we have not figured out how we’re sharing access to all the questions with all the co-hosts. 01:02:14.85 [Michael Helbling]: Oh, yeah, that’s the little tricky part of that. All right, well, before I forget anything else about the show wrap-up, let me just say thanks once again, Jason. And I think I speak for both of my co-hosts, Moe and Tim, when I say, no matter what vendor you need to pick, just keep analyzing. 01:02:37.26 [Announcer]: listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at @analyticshour on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst. Those smart guys wanted to fit in so they made up a term called analytics. Analytics don’t work. 01:03:01.49 [Charles Barkley]: Do the analytics say go for it, no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 01:03:15.95 [Michael Helbling]: All right. Well, we do have an editor who we’ve been talking so fondly about. So we can stop and start as needed. 01:03:31.43 [Tim Wilson]: Well, without further ado. Well, actually before, so just like Moee, are there any, because I mean, you’re kind of often in the midst of vendor selection stuff. So you’re comfortable. There’ll be anything you talk about. You can yourself edit for whatever, named and unnamed. 01:03:49.58 [Moe Kiss]: Yeah. So like we just signed a new BI tool, which I probably can’t. say, but I will just say I’ve been involved in multiple BI tool selections and stuff like that. Okay. 01:04:02.06 [Michael Helbling]: Yeah. All right. All right. Let’s start clackin’ the keyboard and record this thing. 01:04:14.42 [Moe Kiss]: I think I got it. You got it? I think so. I was like, I better do it before you start, because if I do it half a year, it’ll be like… 01:04:23.24 [Michael Helbling]: That was great timing actually. 01:04:26.59 [Moe Kiss]: Pretty sure I got it right. 01:04:33.76 [Michael Helbling]: Here we go in five four 01:04:47.01 [Tim Wilson]: Rock flag and an instrumental rock flag rendition by our guest. 01:05:13.64 [Michael Helbling]: Oh my gosh. That’s the permanent one at the end of every show now. That’s incredible. 01:05:24.53 [Tim Wilson]: I don’t know why I was showing that it was going to play that one, and instead it just played like Transition 2, so it’s good. 01:05:32.12 [Moe Kiss]: Fucking Rostat. The post #293: Tool Selection and the Unhelpfulness of Feature Comparisons appeared first on The Analytics Power Hour: Data and Analytics Podcast.

March 3, 20261 hr 4 min

#292: AI Without Adult Supervision with Aubrey Blanche

As Kevin McCallister once taught us: just because the house is still standing doesn’t mean everything’s under control. Everyone’s racing to adopt AI, but has anyone actually read the fine print? For this year’s International Women’s Day episode, we are joined by Aubrey Blanche to unpack the hype, the hidden tradeoffs, and the quiet ways teams are giving up agency in the name of “productivity.” We explore how data and tech teams are uniquely prepared and positioned to ask better questions, measure what really matters, and avoid letting the AI teenager run the house. Learn more about “phantom value” and why faster isn’t always better… or even cheaper! This episode is brought to you by Prism from Ask-Y—your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show The Adolescence of Technology: Confronting and Overcoming the Risks of Powerful AI Exposed Moltbook Database Let Anyone Take Control of Any AI Agent on the Site Disempowerment patterns in real-world AI usage AI safety is not a model property: Trying to make an AI model that can’t be misused is like trying to make a computer that can’t be used for bad things The AI Marketing Canvas, Second Edition: A Five-Step AI Plan for Marketers Should your AI notetaker be in the room? Heated Rivalry I Don’t Care What You Build (And Neither Should You) And the diagram Moe referenced: Photo by Johanneke Kroesbergen-Kamps on Unsplash Episode Transcript00:00:05.75 [Moe Kiss]: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. 00:00:14.99 [Moe Kiss]: Hey, everyone. Welcome to the Analytics Power Hour. This is Episode 292. The world of AI is moving lightning fast, and I think it’s fair to say that most of us are struggling to keep up. I know I am. There’s new tools, new capabilities, new risks, new headlines, and they seem to land pretty much every week. It’s getting harder to separate what actually matters from the hype. So for this episode, we want to chat deeply about what all of this means, especially when it comes to ethical AI and the real world conundrums that we’re all facing in tech right now. If we’re honest, it feels a little bit like we’ve left the teenager home alone and that teenager is AI. The house is still standing right now, but maybe it needs a bit more supervision. But before we get into it, let me introduce my co-host, Val Kroll. Great to have you here. Hey, Moe, excited to be here. I know. And this is actually our special International Women’s Day episode. So we have an all women class today, which is awesome. But it’s also really fitting because today we’re going to welcome back a guest who joined us many years ago for another important conversation about creating balanced teams and avoiding group think. So we’re really thrilled to have Aubrey back on the show. Since her last appearance, Aubrey has remained a sharp and influential voice at the intersection of technology, power, incentives, and human impact. She’s held senior leadership roles, including as a director at the Ethics Centre, a VP for equitable operations at Culture Amp, and as the global head of diversity and belonging at Atlassian. She’s also served on lots of influential boards and advisor roles in the tech community. She’s currently completing a master’s of AI ethics in society at the University of Cambridge. So, with that amazing wrap, so far to say, she’s been spending a lot of time thinking about AI, about agency, about risk, about what responsibility actually looks like for the people building and deploying these systems. And I’m so excited to dig into what’s been on her mind, what she’s been reading and what she’s been thinking about. So, Aubrey, massive welcome back to the Analytics Power Hour. To kick us off, what the current state of AI. Let’s just open it up. What feels like the most important bit to get stuck into? 00:02:34.48 [Aubrey Blanche]: Oh, my gosh. I love the metaphor you’ve used about the teenager. I have to say, there is no greater joy for a neurodivergent person than someone asking about our special interests. And I think it’s mine, so I’m so happy to be here today. But for me, I think the headline of what I’m trying to convey is actually about working with more intention in AI. So right now, there’s an incredible amount of hype, and in that hype, there are narratives that are just fundamentally untrue. So, there’s this narrative about inevitability. Now, the reality is the teenager exists. Now is not the time to debate whether we should have a teenager because the teenagers are in the house. But I think that the idea that AI is going to do XYZQ is actually not written yet. Yes, there are structural incentives that make certain things more likely. But that doesn’t mean inevitable. And so I think if we all embrace this idea that the way things have gone is not the way they have to go is a really powerful counter narrative to what folks who quite frankly, you know, the commas in their bank accounts depend on you believing that AI is inevitable and going to create value and things like that. And I’m not sitting here saying that can’t happen, but for my professional expertise, having scaled some of the most influential tech companies in the world, I actually think the way people at multiple levels running tech companies, certainly legislators in Australia are thinking about this, actually decreases the likelihood that we get to the good stuff and increases the likelihood that we get to the bad stuff. But again, we can choose differently. 00:04:12.80 [Moe Kiss]: Okay, so what you’re saying is that at the moment, people think it’s inevitable that AI is going to be very influential, have all these amazing outcomes. But the way we’re approaching it right now actually increases the likelihood of the less good outcomes, but not necessarily the good stuff. Is that what you’re saying? 00:04:32.20 [Aubrey Blanche]: Yeah, so let’s take like there’s this idea around that AI makes us more efficient and productive. Okay, so that’s not like on its face completely false, but it’s actually incredibly dumb way to conceptualize the goal of AI. So if the goal is efficiency, what you’re saying is, I’m wedded to the status quo, I just want to do it faster. And it takes a very particular person to think the status quo is sufficient is the way we want to run the world, right? Like you have to be having a pretty good time. Lots of those comments. Lots of those commas or you just like aren’t likely to get killed by police walking down the street. And so, but I think what if we flipped it on Ted and we said we actually believe that AI can be used for increased innovation and value. Right. And efficiency might be one of the tactics that we use in certain scenarios to achieve that, but efficiency itself is a bullshit objective. And so when I see, you know, in the case of like a national plan around AI that’s specifically focused around productivity, what I don’t see is consideration around the questions of what are we producing? Can we produce new things that actually are of higher net benefit to a broader set of society? Like we should be having those questions, but because I think there is such a gap in understanding how this technology works and the implications between folks who are sort of running companies that are building it and those that are trying to regulate and govern it, we’re not having the productive conversation we could have. And at the end of the day, I think it means we’re not going to get access to a lot of the benefits that are possible. And so part of my prescription for solving that, because I’m not seeing leaders, quite frankly, act in a way I’d like to, is that we each take on our own little sphere of influence and say, how do I make principled decisions about the use of this technology in the sphere that I operate in every day? Because that actually does make a difference. 00:06:42.23 [Val Kroll]: So I’m curious if the reset on the objective, is that one of the ways that we can be more intentional upfront? Is that one of the things you’re thinking about in that space? 00:06:52.26 [Aubrey Blanche]: Yeah, I think so. Because I think if we agree that the objective is innovation or human flourishing, then we can then say, oh, it’s not about I’m going to throw AI on everything. It’s about saying, what’s the class of problems for which AI is an excellent tool? and what is the best way to use that technology in that use case to achieve that objective. It completely changes the analytical frame. It doesn’t mean we run away from the technology, but it does mean that we probably aren’t yet chasing phantom value that there isn’t a lot of empirical rigor to suggest that it actually exists. That’s the thing that shits me a little bit is everyone’s like, oh, we can reduce our workforce. Okay, maybe that technology will work, but it lies to you a lot. And so like maybe you don’t want to get rid of the humans quite yet, even if it looks good for your P&L and your ASX quarterly results. 00:07:52.95 [Moe Kiss]: One of the things I read recently, and I can pop something about it in the show notes, it was by Jim Lysinki. And he had kind of like a framework for thinking about this, which is like a quadrant. And it’s like basically like in the bottom left quadrant, he’s got like internal productivity. So quick wins, repetitive tasks. And like in the top quadrant, is really about external growth. It’s about entirely new revenue streams, addressing new customer problems. Really innovation is the way that I would think about it, that bucket. And one of the things, it was a very marketable way of framing it. But I really like the way that he thinks about, we need to get from that bottom quadrant to that top quadrant. And I feel like there’s a lot of commonality with what you’re saying here as well about, We’re doing the dumb shit with AI right now. 00:08:49.19 [Aubrey Blanche]: Yeah, and I’m not saying that we can’t, because I’m happy to share this, but I just wrote a piece about the ethics of using AI note takers, because I tortured myself about it for a while. And then I started using it, and I was like, wow, this is really great for my brain. And so that is an example of where something that’s basically become automated is actually a value add for me, because I’m doing more interesting stuff with it. Yeah, like that’s not an innovative, like I’m not creating something new with that. And so yeah, but I haven’t seen that, but I would agree that I think there is just more that we can do. Now, I have a classmate at Cambridge. She’s incredible. Her name’s Oya. And she talks about this, like the idea of co-intelligence. is that so many people think about AI as replacing a human, which is this very capitalistic, I’m just trying to reduce my operating costs question. But she does the most interesting, amazing stuff, but her contribution is that she talks about co-intelligence, is that looking at the way that humans think and the way that machines operate and working together actually creates more value for organizations, So in that way, these ideas I think are just not being talked about because people are so focused on the short-term returns that they’re getting. But I think if we start to optimize over longer time horizons, these ideas around experimentation and innovation and value creation potential actually expands those possibilities. 00:10:26.53 [Val Kroll]: Does it seem like, I guess my perception, I shouldn’t say it doesn’t seem like, my perception is that lots of organizations focus on some more of those productivity or replacing the headcount. not only for the P&L purposes because it feels safer because it’s like, oh, it’s behind closed doors. It’s not like I’m throwing a chat bot out there that’s going to do something dangerous to my customers or make promises or hurt my business in any way. It feels like a safer way to test it out to expand into those areas. First of all, I question if it’s actually air quotes safer to be doing any of that because it’s still playing with resources and people, but is that why organizations start there or what’s the common thread between being that bottom quadrant before they could start to enable more of the air quotes good stuff? 00:11:16.90 [Aubrey Blanche]: Yeah, I think part of it is a lack of appetite for risk taking, or risk taking of a particular type, I should say, because the way that risk is thought of. So there’s some research, and I’m sorry, I can’t remember the citation, but they were talking about how 80% of corporate leaders felt that they were behind on AI. But when you looked at the data about where their companies were on AI adoption, they were either averaged to slightly ahead. And so people’s beliefs about what’s happening and what’s actually happening are quite divergent when it comes to the use of AI and organizations. And so I think there’s a particular kind of organizational risk management where, yes, of course, no one wants to put in a chatbot that starts identifying as Mecca Hitler, which you can Google, and the answer is Grock. But yes, so I think there’s a particular corporate risk of saying, because the reality is once you get into that innovation quadrant, like the percentage of things that fail goes up and they fail in unpredictable ways. So this is something when we’re talking about generative AI in particular is a known property of these models. like large AI models fail in unpredictable ways. And so the level of risk that an organization is taking is high. Now, I would say there are certain types of risks that aren’t necessarily being managed in the same way. So I rarely see corporate leaders, unless they’ve specifically engaged me to talk about this, considering the risk of Earth’s dwindling, you know, freshwater supplies, like when they’re thinking about AI adoption in their organizations. And so that’s something I would say is, again, I just think there’s a conservatism in organizations as holding them back from achieving the benefits and also having principled and hard boundaries about places we shouldn’t be using this technology or shouldn’t yet be using this technology. 00:13:15.09 [Moe Kiss]: How like, I don’t know, I think one of the things is like I was prepping for this episode. A lot of what was rolling through my mind is like it felt a lot like the privacy debate of a few years ago, where like individuals would give up their their privacy for like little personal wins. But if you’re a big corporate, maybe you have to be more stringent. And this feels like a similar space where people are willing to accept a shittier output for something like low value, but high value. Actually, the stakes are higher. I guess what I’m trying to really conceptualize. When you’re in a technology business, how do you think about those higher fidelity problems and what the guideline should be of where it’s acceptable to use it here? How do businesses do that other than just paying you lots of money, which I highly endorse is a good decision? 00:14:10.12 [Aubrey Blanche]: Oh, thank you. I mean, one of the things, I was just chatting to a pro bono client yesterday, and they’re a particular organization in that kind of an off the shelf like AI governance framework and like decision making principles like is not appropriate, like we have to do something fully custom. But in my mind, the way to like get to this is first to like craft an organizational perspective on AI use. So whether you call that acceptable AI use and AI policy, But that should detail kind of the vision and the general beliefs that you have about how this technology is used, probably have a set of principles that guide particular decisions. Then I think you should have pre-worked through at least a handful of anticipated scenarios that are going to come up. But then you also need to do enablement for employees, not just on how to technically use the tools, which I think is important, which has to include safety and responsibility behaviors, But also, actually, most importantly, teaching the individuals within the organization how to apply the decision-making framework that you’ve made. It should be grounded in your values, the particular positioning of your organization. And that’s something that I love. I obviously do this in my consulting with AI, but at the Ethics Center, one of the reasons that I joined was because when I was chatting to Simon, the executive director, one of the things that really struck me was he emphasized that at the ethics center, our mission is to bring ethics to the center of everyday life, but we teach people how to think, not what to think. And I think that we need to take that principle into AI because the reality is so many of the problems and the challenges that people face around this technology is because they actually haven’t been given a framework of how to make decisions within their scope. And I think there’s a special risk because most of us have not grown up actually being caught ethical decision making in particular. And so there’s a skills gap in the workforce to actually be able to, and there are ways to Think about ethics and responsibility in a structured way, but most people haven’t been exposed to a framework or a process to be able to do that for themselves. 00:16:22.21 [Val Kroll]: Michael, how many tabs do you have open right now? 00:16:25.62 [Michael Helbling]: Oh, I’d say enough to qualify as a distributed system and probably a cry for help. 00:16:31.69 [Val Kroll]: Well, same. If you’re an analyst, you’re basically full-stack now. Excel, BigQuery, SQL, dashboards, plus explaining conversion like it’s a bedtime story. 00:16:44.14 [Michael Helbling]: Yeah, and every tool wants the same context over again. Which table? What’s revenue? Why is July doubled? Okay, sure. Whatever. I guess I just live here forever now. 00:16:57.34 [Val Kroll]: Well, that’s why we’re hyped about Prism by Ask Why, the AI analyst moment. You ask in plain English and Prism orchestrates across your stack, queries, views, charts, all without the constant tool hopping. 00:17:10.04 [Michael Helbling]: Yeah, it’s context-focused too. It remembers your definitions with the jam memory. I mean, it will literally hang onto what does conversion mean in your world. 00:17:19.36 [Val Kroll]: And you can save your best workflows as skills, portable expertise you reuse anywhere, like clean GA4 medium field variations so you don’t reinvent the same duct tape logic weekly. 00:17:31.39 [Michael Helbling]: Yeah, and there’s community skills. Stuff other analysts have already proven works. So you don’t end up debugging a formula. Sometimes it looks like some kind of ancient ritual or something. 00:17:43.37 [Val Kroll]: Prism also does SQL views with version control, so you can save, commit, and rollback changes like a responsible adult. 00:17:51.69 [Michael Helbling]: It sounds amazing. It’s built by analytics practitioners, and there’s free early access while they continue to refine and build the product. 00:18:00.60 [Val Kroll]: So to go see for yourself, go to ask-y.ai and join the waitlist. 00:18:06.12 [Michael Helbling]: And the best thing about it is use code APH and that will jump you to the top of the waitlist. That’s ask-y.ai with code APH. Just think about it. Fewer tabs, more answers, same chaos, but this time more organized. 00:18:25.10 [Val Kroll]: But how does an org, I guess like, you know, outside of bringing in people from the outside, I’m just curious about who inside of an organization is best poised or could mobilize to think about the valuation of those types of risks. I like how you said in scope. Everyone’s role probably has a different amount of risk that they should be allowed to take or comfortable taking, but how do you start to think about assessing that risk? 00:18:52.71 [Aubrey Blanche]: So I think it’s, and there’s so much debate in kind of academic circles about like where accountability sits and how governance structures work. And so I think it really, but for my, I hate to say the answer is a committee, like in general. But I kind of think that in that you need a set of people to think about these risks. You need folks who are actual risk management professionals who understand those processes, but you also need an ethicist who understands the use cases and market that you’re in. because the risks of AI are so unique to how it’s being used. There’s a side note that AI doesn’t mean anything. It’s like a giant bundle of technologies doing a bunch of stuff. You need an ethicist who is qualified to speak to you about those particular issues. You also need someone to represent the customer or the external face of the company because there are major reputational risks and considerations in this. as well. And then you probably need some technical folks who can reign us in when they get a line about what’s actually achievable. So if we’ve decided philosophically this, we’ve made an operational decision, but what does that actually look like in terms of developing or delivering a product or in putting a tool in the hands of our employees? I recently learned about a company that’s based in the UK, that they are incredibly rigid about their responsible AI approach to the point where any employee at any time can raise an ethical issue about something they’re doing with AI that can actually be deferred to an ethics council for debate and dissolution. Wow. So really, really cool. And so I offer that as an example that like If they wanted to, they would in the sense that like, yes, we can’t mitigate our risk. I’m not trying to say that, but we could do much better than we currently are if companies had the will. And as someone who spent a lot of my career in DEI, so diversity, equity, inclusion, doing kind of anti-discrimination and social justice work across tech, like the number one factor that I have seen in whether programs that are about responsibility and ethics and social justice, et cetera, does the CEO care enough to keep it funded when every other incentive in the company is to shut it down? 00:21:17.94 [Moe Kiss]: I want to push you a little bit because I feel like folks will be like, listen, this episode will be like, yes, I want to do this. I want to go into my organization. And someone is going to say this. And I’ve sat around with you and debated many a time. And I know I’m going to say Aubrey is probably like the best, best person ever to have a discussion with because she’s so good at like reframing things. So sorry, I’ll turn down the fan girl right about now. But so someone in the organization will be like, yeah, cool. We can build some frameworks and guidelines. But AI is moving so fast. By the time we build the guidelines, they won’t be relevant anymore. So how would you handle that conversation? 00:21:58.42 [Aubrey Blanche]: Okay, I kind of think it’s silly. I wouldn’t say that if I was in an actual debate because I care about influence and changing people’s minds. But no, so I think that’s true. But that’s why I think for me, and there is debate about this, so I don’t want to act like it. But like, for me, principle-based frameworks actually solve some of that problem. because the idea is if you get into a framework where it’s like this is in this is out and you have a laundry list. Yes, that’s going to get stale really quickly because the way the technology works is going to change fundamentally or or the way it’s being deployed is going to change really quickly. But the idea of For example, a company could make a decision that says, we don’t deploy technology that makes decisions about humans into the market without having done thorough impact assessments measured for the potential of bias. and also developed a process for someone to alert us if something has gone wrong. You can decide that, and the underlying function of the technology changing actually doesn’t change that as a governance structure. The way you achieve those things may change, and so you need to be flexible and always willing to update your processes. But so yes, I do think it moves fast, but the idea that like, oh, it moves so fast, we just can’t do the right thing is like the bullshit that the tech elite has been selling us for decades, because it’s more convenient for them and because it maximizes their profits. And I want to say something really specifically. There is a difference between believing Profits should always be maximized as the primary goal and like we can maybe give up a little bit of that to not destroy civilization So there’s often this binary of like oh you like you hate money or like you want to make all the money in the world like no We could make principal decisions that yes may actually have some potential like marginal impact on profit but like I sometimes push leaders to say, are you standing behind the behavior that maximizing your profit is more important than the welfare of your employees or customers? And would you be willing to say that to the media? Because that’s the implication of your decision. And so I’d put that to folks to say, if you believe that, there’s probably nothing I can do to help you. But if that’s not what you mean, we can actually take different actions to align those values and beliefs. in a way that supports business, supports growth, but also balances the kind of risks that come off. So like the middle way is possible. And so I just want to call that out is like, it’s not one or the other. There’s a giant spectrum in the middle. 00:24:33.72 [Val Kroll]: I think that a lot of, especially thinking about analysts working inside of organizations are feeling disconnected from those larger implications when they’re deciding which note picker am I going to send to the meeting to pull on that strain from earlier. But I guess, is there anything that you would offer or suggest for someone inside of an organization that has access to use those tools internally, but maybe hasn’t been given a lot of guidance, but wants to be a good actor in all this, that maybe they’re not going to be the one to run up the flagpole to the CEO, that we need to be doing all these things, but is there anything in the middle for them that you would suggest they keep in mind? 00:25:16.80 [Aubrey Blanche]: Yeah, I think it’s actually the same kind of advice I would give to anyone who wanted to be kind of an advocate or an activist within an organization is look at who you are. So what’s your position in the world? And then what power do you have in the organization? So people often think of power as like formal power, like I can hire you, fire you, promote you. But things like, do you have relationships? Do people trust your judgment? That’s the type of influential power. And to say like, number one, make your decisions for you. So we’ll use the note-taker example, because I’m like all about it. 00:25:53.79 [Moe Kiss]: I’m going to ask you to like walk us through how you made those ethical decisions. 00:25:58.99 [Aubrey Blanche]: Sorry to distract, but. No, and I have a whole article that you can put in the show notes, but basically the like, Let’s walk through how someone says, okay, I can’t control this, but X tool has been white listed for or allow listed for note taking in my organization. I’m going to decide how I’m going to use it. So I’ve decided that there’s utility benefit to me, like there’s an obvious benefit, but there are harms in terms of potential privacy of data leakage issues if they’re training on my data or depending on where that data is stored. And so for me, when I walked through that, I said first, like, one, are they using my data to train? I don’t ethically stand behind companies using my data to train their models. My economic argument is that’s a resource they’re not paying for. I’m paying them for the service, and then they’re extracting value from me, like that doesn’t feel like equitable value exchange. It also exposes myself and the people that I have meetings with to potential security issues. So a lot of these companies are newer. They don’t have the robust security architectures that you would expect of enterprise tools. And also the use of AI creates security vulnerabilities that are often unanticipated and there’s a huge rise in like AI assisted cyber attacks. So data leakage risks are just higher. So for me, that meant, okay, I’m turning off data training in the tool that I use. It’s important to note that companies have an incentive to keep that turned on by default. So you have to go and turn it off. I think that’s an unethical design choice. I think the default should be off and people can opt in if they want to. It’s a dark pattern. Also thinking about, have I done my due diligence about the security practices of this organization? So I want to look to see if they have a SOC2 type one or type two certification, if they have ISO 27001. And then ideally, so those are like standard security control certifications. There’s also a new standard called ISO 42001, which is the AI management system standard. So this is something I would want to see, but recognizing that I think somewhere slightly north of 50 organizations in the world, it is believed have the certification at this time. I do want to give a shout out to Culture Amp. because they are one of the companies that has that certification. And I love that about them. But so those are the things I would look at. And then I’m thinking about how I’m gathering consent to record. So depending on where people are in the meeting, that might be a legal issue. But it’s also an ethical one that people need to be able to opt in. And so for me, the qualities that generate fully informed consent are one, everyone knows they’re being recorded. They understand the risks that they’re taking with their data and how it’s being stored. And also they feel full agency to opt out. And because the note taker I use doesn’t call into the meeting, so no one can see that it’s there, I explicitly start the meeting by saying, hey, I really like to use an AI note taker so I can be more present in meetings. I want to be thoughtful that this does not train on your data, but the data is stored in the cloud on AWS. So recognizing that if you’re boycotting Amazon, that might not work for you. If you have any problem with this, I’m happy to take notes by hand. So I start my meetings where I choose to use that, but I also only use note takers in meetings where there isn’t sensitive or confidential information being shared, so I don’t take any risk with that data leaking. But that process, yes, I’m literally an ethicist, so I do that, but you can follow people who do that pre-work for you. So you have control over whether you use that note taker and the costs and benefits but I would say like for me I curate my social media ecosystem with a lot of people smarter than me so that I for a lot of things can. kind of skip the deliberation process because they’ll walk through their thinking and I can go oh you shortcutted me and I figure out the ethical decision I wanted to make and that’s something that a lot of my online content does is I try to talk through how I’ve thought through a problem so that people can decide if like that’s the lane they want to go in also. 00:30:10.88 [Moe Kiss]: I love this. I love also how much personal responsibility you’re demonstrating because I think sometimes it can be really easy to be like, well, someone should give me guidelines and someone should sort it out. And like, I can’t do anything within my sphere of influence. And I really love to like stand some personal responsibility. I’m curious to hear your thoughts specifically as it pertains to data practitioners. Like, I think there’s another layer on top of that, which is we’re often privy to user data, first-party data. There’s a lot of ways that it’s unlocking incredible value for data folks. But do you think there’s another layer of dimensionality on top of that we should be thinking about, particularly in the data space? 00:30:56.04 [Aubrey Blanche]: Yeah, so I would say like the closer you are to sensitive or confidential information, the closer you are to harms. And so I think the level of personal responsibility goes up. But one thing that I’m really encouraged about by especially folks in the in the data space is that We’ve practiced for this before. When GDPR came into effect, the hygiene behaviors around privacy, the bar got raised in major ways. Obviously, if you’re not operating in Europe or on European citizens, but I think that the norms and practices around privacy, this isn’t actually fundamentally different. I hope that people would take a bit of hope in that and that they actually already have a lot of the skills needed to do this well. So this isn’t some, it’s easy to be like, oh, it’s an alien species. Sure, it’s kind of weird, but it’s not fundamentally different. And there’s a whole academic literature debate about whether AI is fundamentally special or it’s actually just a normal technology that mostly works faster. I tend to believe it’s more of a normal technology. And so to me, what that says is the skills and frameworks that we already have are useful. for governing and managing the risks associated with this technology. But I do think from a data professional perspective, the most powerful thing you can do is be open about asking what could go wrong and what do we need to do to prevent that. And I think if we just got in the habit of before we do take a moment of consideration to say, like, what’s the worst case scenario? Now, one thing that I will say that concerns me is that there’s kind of two specific issues that make answering that question correctly really difficult. One is that there’s a huge number of people who do not understand how AI works. I think data professionals, that is less of a risk. They just tend to understand the technology more. But the second piece, and I know this is now going on a half a decade of talking about this, but a lot of people in the data and text space don’t have the lived experience to accurately answer that question because the worst case scenario doesn’t happen to them. Wait, say more. So like, and this is an oversimplification, but like how many like Rooms full of data people are like a bunch of white dudes with no disabilities who like make over six figures. And I’m not critiquing them for those qualities, although I could. It’s that the likelihood that they have, for example, read a bunch of black feminist theory is quite low. And we know that black women are uniquely at risk of being harmed by poor deployment of these technologies. So it’s great that you build the muscle. to ask what could go wrong. But you also need to critically question your own ability to answer that question in a way that’s universal. So Lucy Suchman, who’s a feminist science and technology studies scholar, talks about the idea that people with a lot of power or privilege build things in their own image and assume that their experiences are universal. And so that’s something I want to talk about is we still need to talk about who’s in the room and what qualifications they have. to make those decisions. There’s also some interesting research being done by someone a year ahead of me and my master is looking at the demographic distribution of people in the AI ethics versus like more technical spaces, because AI ethics is actually much more female, much queer, much browner than like the technical things. And so there’s, again, this is why I say It sounds a little self-serving, but you need an ethicist in the room because the likelihood that the room is qualified to answer the ethical questions is quite low without them. 00:34:58.64 [Val Kroll]: I like that a lot. Can I ask another data, bring it to the data crew specific one? Cause that was, that was awesome. You mentioned something earlier about, um, phantom value. And I wonder if you could expound upon that a little bit is I think I know what you mean by that, but I would love to make sure that I, I understand. Is it just like a perception versus a reality or a lack of measurement to objectively say whether or not, you know, use case was valuable to the organization or yeah. 00:35:29.09 [Aubrey Blanche]: D, all of the above. So all of the above. So there is a couple of particular threads that are all contributing to that belief that I have, which is one, there’s not a ton of research. And part of it is because we only have a couple of years of LLMs rewriting the world, although AI as a concept has existed for a long time. depending on how you define that, which is, again, a very specific thing. That’s another episode. Yeah, like, people with PhDs debating what artificial intelligence means. And so I think there’s, number one, like, there isn’t a lot of hard evidence that, like, the primary benefit of using AI is, like, increased financial return specifically. Like, the data is just pretty thin that you can draw a direct line between, like, throwing AI at a problem and I’m suddenly making more money. Again, if you lay off 20% of your workforce, that’s probably going to happen, but you’re going to likely incur a bunch of other problems that are more expensive than whatever game. And not to call out specific companies because they’re not the only ones doing it. So I think it’s really important that this is a broad issue. But like Klarna got rid of a bunch of their customer success staff, and then a year later hired the team back because they realized that the technology couldn’t do what they somehow decided it could do without any proof. So I think there’s that. And then I think there’s also just the Yeah, the reality that this tool may or may not return the kind of returns that we’re thinking about. And so I don’t think we should be going all in on something that’s so untested because right now like entire markets are responding to like PR talking points written by people who have an incentive for you to believe that and don’t really have any accountability structures to tell the truth. 00:37:23.30 [Moe Kiss]: I think one of the things that I’ve been chatting to a few friends about who own small businesses and whatnot is a lot of the pressure that’s on them is that investors or clients are basically expecting the returns from AI to reduce prices or increase revenue streams, but it’s not actually performing at that level yet. Some of the smaller businesses are really in a crunch position because they’ve got these clients who are like, well, I expect that you’re going to charge me less. But it’s not actually providing that kind of value to our business yet. Now you’re just asking. That’s a really difficult position to be in. 00:38:02.47 [Aubrey Blanche]: Yeah, I mean, you know, perhaps a little bit more radical philosophy than like the average listener of this pod is on. But like, yes, so in general, there’s a ton of academic discussion about the fact that like the logic of AI as it is currently being built and deployed is like extractive and capitalistic in that it is inherently being used to devalue labor and expertise. I cannot remember who said it, so I feel really bad about this, but AI is kind of at a point right now where it can write a really good facsimile of a PhD level paper, but you wouldn’t trust it to make decisions about your kids. And so I think that Again, we just need to be a bit more deliberative about this. I think we are a bit as a society drunk on the marketing hype and we’re not making principal decisions. I think there’s also a question in that with a small business. So I’m thinking like professional services, right? Like a consulting business. It’s like, oh, well that took you less time to do. And it’s like, okay, so you are assuming that the cost to you is based on the time it took me to execute that as opposed to the quality of the work, which speaks to an underlying belief about how we value expertise and labor, which doesn’t make sense. The story that I think illustrates this, well, and I don’t know if it’s actually real, but it’s like floating on the internet, so Pablo Picasso is sitting at a bar, and some dude six months later realizes he’s Pablo Picasso. and says, oh my God, could you draw me a thing and doodles on a napkin? And he says, that’ll be $30,000. And the guy says, but that took you five seconds. He said, it took me 30 years to be able to do that in five seconds. 00:39:48.45 [Moe Kiss]: Oh, I love that analogy. 00:39:52.40 [Aubrey Blanche]: And so I think that part of getting away from that is actually equipping small businesses to explain the source of the value that they’re providing to a client. And then also recognizing that some clients just only care about the bottom line and that sucks. But I think we need to equip them to say, but also move to more fixed fee. project structures. So there are ways to structure your pricing that can deal with that in a way that avoids those conversations. But again, I don’t know who is enabling SMEs who are already stretched them to figure out how to cope with that. That’s definitely something I worry about is corporate consolidation, noting that in Australia in particular, I saw a stat that something like over 99% of businesses in Australia are SMEs. like the corporates we’re talking about actually make up a vanishingly small amount of the overall economic ecosystem here. So yeah, just something to think about. Interesting. 00:40:54.44 [Val Kroll]: So is the measurement piece, because it feels to me like analysts are uniquely poised to measure cause and effect. And so if this could be one of the other areas that we could have a little rallying cry to the analytics community to say, hey, if you’re going to fire the CS team, a customer success team at Klarna, we’ll pick on them again for this example. Let’s, oh, I don’t know, think about what we intend that to achieve and let’s measure that. And if it doesn’t, then let’s figure out what the next plan is or whatever. But is that like another area where you think analysts could jump in and step up to help with understanding how this is impacting organizations versus just like, oh, there’s 10 less people here now. So that must be better. 00:41:42.51 [Aubrey Blanche]: Right. So I would say yes. And I would go a click deeper. There’s something that analysts can bring to this that folks outside of the field might not, which is it’s not just what could go wrong. But what is the leading indicator that would tell us it is? And what does our data infrastructure allow us to measure? So that, to me, is the really exciting thing, is that an analyst, because they’re deep in the systems, they understand the data, they’re actually able to translate this idea of harm as a theoretical thing into a set of monitoring procedures that would actually tell you if something’s going wrong. Because the impression I don’t want to get is like everything is terrible all the time. like I operate from a slightly different frame is everything could go wrong all the time. And like, if that is my baseline belief, then I personally am motivated to do things to reduce that risk. And so it’s not meant to be doom and gloom, it’s meant to actually just be responsible to say like, So that’s what I would say is like, okay, define the bad. Like in the case of Klarna, it could be something as simple as like, okay, well, we need to track customer satisfaction with individual interactions or successful resolution of issues, right? It doesn’t need to be like a social justice coded metric. You know, in my head, I’m like, are different customers getting different quality of experiences because they’re having different types of problems, et cetera. But like, we’ll start at a baseline of like, do we see a decline in customer satisfaction? But also the analysts can say, hey, given that we’re not sure about this impact, maybe we just do 10% of the objective for two or three months to actually measure that before we make a decision about whether this is a broader kind of initiative, whether that’s workforce reduction or redeployment or implementing technologies. customer service chatbot as an example. So I think that’s where analysts have a really special and unique role and quite frankly really powerful to lead their organizations to be more responsible. And there’s also a lot of debate about like making the ethical case versus the business case. The reality is they’re both tools and which one works will depend on the organization you’re in. I’ve worked with organizations that go full in on the ethics case and the leaders get upset when you talk about financial returns. And I’ve worked with organizations on the other side that are like, this is about like the board reports every quarter. And I’m like, cool, if I need to explain this to you in money, it’s fine as long as we get to the outcome that we all agree is good, which is you know, creating customer value, which is smooth operations, which is avoiding screwing people over. Like as long as we end up in the same place, the path we take is kind of real. 00:44:25.31 [Moe Kiss]: I think that’s like literally a one-on-one in how data practitioners work, which is always like figuring out what does the person making the decision care about, and then how do you frame your analysis and your recommendation in a way that speaks to the thing they care about. 00:44:39.02 [Aubrey Blanche]: Totally. But you’ve just proved the point that I made earlier, which is that We already have most of the skills to do this well. 00:44:47.38 [Moe Kiss]: Oh, burn. Look at you full circle. 00:44:50.64 [Aubrey Blanche]: Yeah, not to be like, I was right, but I think that’s actually more everybody else is already capable of being right. 00:44:57.13 [Moe Kiss]: So one thing we haven’t talked like, I feel like we haven’t gotten into the actual nitty-gritty, like I could spend another five hours talking to you. But one of the things that we did chat about as we were like preparing for the show was about agent to AI and giving up agency. And I would really love to hear your thoughts about those trade-offs about how people give up agency and what folks are willing to give up in terms of speed. Is the sacrifice worth it? And I feel like we’ve touched on that, but maybe not as deeply, particularly with the agent agai example. 00:45:34.65 [Aubrey Blanche]: Yeah. This is a bit of a conjecture, but I feel strongly about it. But if anyone wants to yell at me in the comments, I’m happy to be proven wrong. I’m thinking about the study that Anthropoc put out called Disempowerment Patterns, and what it showed is that people very often gave up agency to AI. which I find really concerning because there’s other research that shows that the majority of people don’t actually understand how LLMs work. If you don’t know LLMs, they just predict what the next word most likely is. They’re really good at producing things that look like language, but they don’t actually know anything. There’s no conscious, there’s no intention behind it. It’s literally just like, I think that people give up agency because they don’t actually understand the problem that they’re faced with. I think there’s also research that shows that the people who are most likely to give up agency are the least skilled at the thing that AI is doing. So there’s this concept called like, I think it’s AI acceptance, which is like the rate at which someone accepts the output of the AI versus challenges it or changes it. And there’s a strong correlation between the expertise that someone has in the domain and their likelihood to challenge AI. So someone with more expertise rejects AI because they have the ability to evaluate the quality of the output. Whereas if you’re not an expert, you actually don’t have the underlying knowledge to understand whether the output is valid or not. And so you tend to defer to the model. Contrary to what’s happening, the ideal behavior is that you only use AI in places you have the expertise to evaluate the output. But that’s not what happens. 00:47:19.77 [Moe Kiss]: The total example that’s coming to mind is using AI for data analysis, right? Because I absolutely will go back and forth. But like you said, I probably have a stronger threshold of AI acceptance because I’m an expert in that area. So I know when something’s not right or something’s off or I have that intuition and that experience the 30 years built up, maybe not 30, I’m not that old. But I had that experience built up. Whereas for someone else who’s trying to use an LLM to do data analysis, they’re much more likely to accept it on face value. And therefore, the risk increases because they don’t have the expertise. 00:48:00.62 [Aubrey Blanche]: Oh, I never knew that’s what that was called. Yeah. And you think about the idea that even some basic practices that you would practice in data science or kind of analytics is that you run, let’s say, I don’t know, the last time I programmed and did analysis, it was in R. So I don’t know how out of date that is. I’m still, I’m probably starting like I’m coming from the 1800s. But like, You run your code, you still do some cursory checks to make sure nothing weird or unexpected has gone on. So that goes back to my point that like data folks have the skills to deal with this, which is like, I never trust an AI output. So I’d use it for research and editing and all these things. But I’m always asking AI to produce links. I always go read the original source of anything that’s being analyzed or presented to me with an LLM. but it speeds up my acquisition of stuff on a certain topic so I can spend more time analyzing and less time searching. So that’s like an example where I’m expert enough to know that LLMs bullshit me. And so I always have just a dot of skepticism that what I’m reading is true. And so again, that’s an attitude that you can build, which is trust is earned. And I have not seen evidence that this technology is deserving of our full trust. But again, I really think teaching AI literacy has to become a basic skill that’s taught in primary schools. I think it was in Oakland. This is a bit of a tangent, but I promise I’ll come back. There was a focus around public health in Oakland, and so they did a really interesting community-based thing where they found that teeth brushing was really highly correlated with a bunch of other positive health outcomes. I don’t know the science behind it, but clean mouth, much better body function. And so they actually made it so that they ran these community programs or became normalized for community members to teach each other about three facts about teeth brushing that were shown to promote better brushing behavior. And I think that’s actually what we need. We need to think about AI literacy as a public health problem. and to say like there was a baseline of competence that we want everyone to have. And I think to your earlier point because of how fast the technology is moving, my personal belief is that organizations have a higher ethical responsibility to teach their employees safety behaviors because the reality is the government can and does not move fast enough to achieve those things on a scale that we want. And I don’t think it has to be like, you don’t need to spend $250,000 on responsible AI training. literally run it for free where you’re like, here are the three AI behaviors that we encourage. One, like always check out puts. Two, be careful with sensitive and confidential information. Don’t put it in tools that aren’t locked out. Like you can teach people that in 10 minutes and reinforce it over time. Again, corporate leaders have the skills to pull off setting expectations for their business. This is not something that’s outside their realm of the capability of anyone who’s getting paid to lead an organization. 00:51:15.04 [Val Kroll]: I like that. So much to think about. 00:51:20.76 [Aubrey Blanche]: But I think it goes back to, like, you talked about personal responsibility. I think we all have a responsibility. And what that responsibility is changes depending on the power we have access to, the systems we have access to, the work that we’re doing. But I hope that’s an empowering message, which means that you can do a lot. Because think about it can be really easy to go, oh, this is all big and structural and scary and whatever. But imagine if each of us did one slightly more responsible thing every week. That’s actually fundamental systems change, and it doesn’t actually require enormous sacrifices and changes on behalf of any one person, but that takes us getting a collective lens on what it means to achieve safety and responsibility with this tech. 00:52:06.68 [Moe Kiss]: I have a very weird one that has… I have not fully processed this, and so I’m just going to talk about it out loud because I want to get your thoughts because that’s what I use the podcast for. Okay. One of the things that An amazing one on the team, Jennifer, was talking to me about planning and how important planning is. Basically, the analogy she gave me is like, Moe, we need to know we’re going from Melbourne to Sydney. We don’t need to know that we’re catching a plane, a bus, or driving a car, but we need to know we’re going from Melbourne to Sydney. I was like, that is excellent. That’s a framework I’ve used now for how we think about planning. because we might have a path, I promise, I’ll come back. But we have a probable way we’re going to get there, but it might change as we learn new things. I was trying to think about this in the context of AI product development the other week. What was bubbling up in my mind is, Maybe it’s that we don’t know that we’re going from Melbourne to Sydney but we know we want to go from Melbourne to the beach. We just don’t know which beach we want to go to and so what might be different about that process is we need to also figure out how we’re going to get there and then we need to figure out which beach. But then I think the bit that’s been rolling around in my mind is, am I treating AI product development as different to other product development because I’m giving this, what’s the word, uncertainty to it that maybe doesn’t exist? I’m curious, Aubrey, you’ve said a couple of times about AI just being a stack of other technologies and we’ve seen this all before. It’s nothing actually that transformative and a lot of it is hype. I guess I’m just processing live. Am I thinking about it with the hype layer on and actually we’re just going from Melbourne to Sydney or is it slightly different and we do need to have that different mindset? 00:54:06.41 [Aubrey Blanche]: So I think it is slightly different. Like I think most of the things we think about kind of standard software development apply like basic, you know, don’t put your code on the internet unless you’re intentionally open sourcing, etc. Like those principles. But the reality is like traditional software development for the most part, software does what you tell it to do. The unique risk posed by AI is AI sometimes does stuff you didn’t tell it to do. not always. And so the way to deal with that uncertainty is, and what I see happening is some people go, Oh, well, it’s uncertain. So we can’t do anything through our hands up. And like, I think that’s quite silly. We can say, okay, I know that there is a degree of uncertainty and any risk professional will tell you that uncertainty is a fact of the universe. And there are actually very good ways to manage it. And from my perspective, one of the things I’m often telling companies that are saying, you know, I know that I want to go to the beach, but I’m not sure which beach. I’m going to, is to say, but there are certain things you can do to prepare. So for example, you need a mode of transportation. You need to know the rules about how to operate that mode of transportation safely. Do you want to take a bus and get a ticket? Do you need to know how fast you can legally go in New South Wales without getting a ticket on your license? So there’s like things that are knowable that you can plan for and you should do that. But then you also have to operate with an understanding of something will go wrong. and I may or may not detect it with AI. And so the question is, great, what path will I find out that thing has gone wrong? And how do I plan to respond to something unknown going wrong? So like an example from corporate, like corporates have crisis communications frameworks. They don’t know what’s gonna blow up, but they know who they’re going to call when that thing does blow up. what the roles and responsibilities are to responding to that thing. So again, Moe, I think you’re actually right. I think we’re just, we need to borrow from more fields of expertise to manage these things. But a lot of the skills and frameworks that are needed to manage them are not things that need to be invented. 00:56:15.31 [Moe Kiss]: Oh, I love this. But the thing that, okay, the thing that I took away from what you just said is like, If we apply this intentionality as well, it might also help us figure out a better path to get there that minimizes the risk. So like we might figure out taking a bus will mitigate certain risks that driving a car won’t. And so therefore that’s the, oh, okay. I love an analogy. 00:56:40.53 [Aubrey Blanche]: I love how well you played with it too. That was, I really liked your beach analogy. I was like, you just come up and seem cold, but. I don’t know if I want to go to that one. But yeah, I think that’s it. And again, there’s this like underlying thing that happens with people who have expertise in one field. And I would say I tend to see it manifest among certain demographics more than others. Take that how you want. Look at my social media and you’ll figure it out. But people tend to think that because you have expertise in one domain that you therefore make good expert level judgments in other domains. And I think in tech, As an industry, we have so deified engineers to be like, oh, they’re amazing. And let me be clear, engineering is hard. It’s amazing that people can do that. I stand behind that. But because you are amazing at building technology does not necessarily mean that you are qualified to evaluate and manage ethical risk or operational risk. It’s that different people have different expertise, and we need to recognize that the people building often have not been trained or exposed to the types of problems they would need to be able to make expert level decisions about these things, which goes back to my point about committees, which I know is really exciting. But my point is that I do not believe that safety and goodness can be achieved without getting a lot of the right bits of expertise in the room. And someone’s going to tell me they’re going to like, go on chat GPT and like program a suite of agents to function that way. And I’m not really sure if I believe that that’s sufficient. One for the particular reason that like, you know, we’re talking about novel problems and extrapolating outside of samples is a problem every data person knows well. So that’s what I would say is like we need to be really careful and part of it is the underlying value of expertise that’s non-technical. 00:58:33.98 [Moe Kiss]: That feels like an incredible place to wrap, which is impossible because I swear I could sit here all day and just keep chatting about this. But the last thing we do on the show is we go around the horn and we share what’s called a last call. And something that might be of interest to our listeners, not our users, our listeners. Just something you’ve read, you’ve watched that’s interest you. We will, of course, share all the links to Aubrey’s amazing reading list in the show notes. But Aubrey, do you want to go first since you’re our guest? 00:59:04.06 [Aubrey Blanche]: Oh yeah, I’m like fully alone, so if anyone does anything, just like drop everything and watch Heated Rivalry. When it’s good, like come for the hot guys, stay for all of like the completely renorming of like queer media. but also get in on the discourse online. The ethics and the quality and the values that the people who are engaged in creating this show are exhibiting is, I think, transformational in terms of the way media comes into the world and what it does. One, it’s just really fun, but if you’re into more critical analysis and things like that, it’s this rich well of things to think about. For me, ultimately, the way we could do things differently and better. 00:59:46.54 [Moe Kiss]: I love it. Nice. 00:59:47.30 [Val Kroll]: I love that. What about you, Val? This is a medium article. I subscribe to a lot of engineering and product and design content to get the diversity of perspective, not just all analytical content. I actually clicked on this one. I started reading it thinking I was going to hate it. I thought this was rage bait for the analytics crew, but I ended up really liking it. So it’s called, I don’t care what you build and neither should you by Joel Dickinson. And he’s talking a lot about like he has quotes about like, you know, Ronnie Kohavi saying that, you know, only 10 to 30% of experiments or product features actually add any impact. So like, why should we care? And like, who cares about the target? And I’m like, like clutching my pearls. But then he starts talking about the framework that he thinks works. And he was saying, I found that good leadership in engineering boils down to asking relentlessly, how will we know? So not what you will build or what technology we’ll use, don’t show me the architecture, just how will we know if the problem is solved? And I was like, okay, you got me. I love that whole thinking about the problems frameworks and things like that. Anyways, definitely not from an analyst perspective, but I really enjoyed it. It was a good read. It was a fun one. 01:01:02.13 [Moe Kiss]: I have a bit of a weird one today. Normally, it’s something I’ve read or looked at, but this time, I’m going to crowdsource some help. I’ve been thinking a lot about a measurement of AI products. And obviously there is like a wealth of information, but I think the angle particularly that I’m thinking about is as it relates to like user engagement and users like having a successful experience and how that potentially differs from like an AI product versus like a more traditional product and like what that intersectionality is. So I’m not like talking about like evaluating a model, I’m more about like How do we understand if a user has multiple designs that are generated from an AI output? Does that count as someone doing something creative or is that like, I’m just like really trying to wrestle with some of these concepts and I don’t have a firm view yet. So it’s more of a shout out that if folks are coming across interesting articles or perspectives on this, I would love you to share them with me because it’s definitely top of mind, especially like, when you have like AI products and non-AI products in your stack and really wanting to be able to paint a holistic picture. So anyway, that’s just, I thought I’d share my conundrum at the moment. I like it. Okay, so this has been such a wonderful conversation. Epically huge thank you, Aubrey. Like, just love having you on the show. We’re so appreciative. Oh my gosh. 01:02:35.92 [Aubrey Blanche]: Literally call me anytime. I’ll move my calendar to show up. Right. I’m sorry, my diary. My diary. I’m trying to assembly here. 01:02:43.51 [Moe Kiss]: Nice. So what we would love after the show is for you to please come and leave us a review or a rating on your listening app of choice and feel free also to request a sticker on the analyticshour.io link. You can also reach out to us on LinkedIn and the measure Slack, and also through our email contact at analyticshour.io. So that’s a wrap on AI the teenager. A very big thank you once again. And for all of my co-hosts, who today is just Val, keep analyzing. 01:03:23.06 [Moe Kiss]: Thanks for listening. Let’s keep the conversation going with your comments, suggestions, and questions on Twitter at @analyticshour on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst. 01:03:41.02 [Charles Barkley]: Those smart guys wanted to fit in, so they made up a term called analytics. Analytics don’t work. Do the analytics say go for it, no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 01:04:08.49 [Val Kroll]: Rock flag and let’s get intentional. The post #292: AI Without Adult Supervision with Aubrey Blanche appeared first on The Analytics Power Hour: Data and Analytics Podcast.

February 17, 20261 hr 2 min

#291: The Data Work that Lives in the Shadows

We know what the work of the data practitioner is, right? It’s everything from managing data ingestion to data governance to report development to experimental design to basic and advanced analytics. It’s writing (or vibe-writing?) SQL or Python or R while also being adept at whatever data stack—no matter how modern—is at hand. Of course, it’s a lot more, too! And that’s the topic of this episode: the unofficial, often unheralded, but often quite important “shadow work” of the analyst—the myriad tasks required to effectively glue together all the data work that occurs out in broad daylight to enable the data to truly be useful at driving the business forward. This episode is brought to you by Prism from Ask-Y—your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show DataTune Conference in Nashville: March 6-7, 2026 MeasureCamp New York: March 28, 2026 Marketing Analytics Summit in Santa Barbara, CA: April 28-29, 2026 Photo by Darwin Boaventura on Unsplash Episode Transcript00:00:05.78 [Announcer]: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. 00:00:14.91 [Michael Helbling]: Hey everybody, welcome to the Analytics Power Hour. This is episode 291. Who knows what evil lurks in the heart of men? The Shadow knows. Moest of our listeners probably don’t know that callback to the extremely famous radio drama The Shadow, but what they probably will recognize is the work that data and analytics people do that lurks in the shadows of our day to day. That’s not really the job description. It usually doesn’t get recognized, but you do it anyway. Maybe some days you feel more like a janitor cleaning up ugly data or a therapist listening to stakeholders’ frustrations or some sort of data marketer just trying to sell your wares internally. I think we should talk about it. Let me introduce my co-hosts. Moee Kisss. How you going? 00:01:03.93 [Moe Kiss]: I’m going great. Thanks for checking in. 00:01:06.77 [Michael Helbling]: Have you ever heard of The Shadow? The radio show, The Shadow? 00:01:10.88 [Moe Kiss]: It was like from the early… No, but I’m deeply familiar with the sentiment. 00:01:14.45 [Michael Helbling]: Oh, okay. Yeah, yeah. And Val Kroll, welcome. Thank you. Bye, everyone. Go Bears. Yeah. And… Hey, it was close. Tim Wilson, probably the only person that got. 00:01:36.40 [Michael Helbling]: I was going to ask you if you remember. I think first up, maybe let’s talk about what kinds of shadow work have you found yourself getting into in your career? Like what are some of the categories or the types of things you’ve gotten into? And then as we sort of get into that discussion, maybe figure out if we thought it was necessary or not or whether it was good or not. So who wants to start us off with some of the stuff you’ve run into? 00:02:16.22 [Moe Kiss]: Oh, I mean, the one that starts with a capital A, admin. And I think this is potentially more on the internal side. I’m going to be curious to hear reflections. But I feel like there ends up being a lot of cadences in a business. And I think I’ve gotten to a point now where I kind of see it. And I’m like, if as a data team, you start to pick up, I don’t know if admin’s the right word or project management or heckling people to be like, you need to fill out this spreadsheet. Have you done this bit of this deck? All of that. And some people might think that that’s fair. But in a space where you have admin support and folks who are meant to have that as part of their role, feel like I see data, people end up having to fill that gap a lot just to keep momentum moving forward. And it’s almost like once you assume responsibility for it, it’s almost impossible to ever roll it back. 00:03:22.03 [Tim Wilson]: I’ve thought, I mean, there’s one specific part of that. There’s like the input, I need to do admin to get stuff. And then when you first said admin, I was thinking like, user governance, like, oh, somebody needs access to whatever. I feel like there’s an admin part that I think is good for the analyst when An analysis is delivered or something is delivered that is supposed to lead to a decision and an action that for a long time, I’ve felt that the analyst does kind of need to own that because it’s pretty easy for somebody to say, yeah, that’s awesome, but they don’t really necessarily have an incentive, direct incentive to take the action as was prescribed. as an accountability mechanism for the analysts to say, oh, I’m going to be here because I know how to set recurring reminders. I’m going to set a reminder to come back and say, hey, you said that was great. In the next release, you were going to do X or Y. Did you do it? I don’t think that’s admin though. 00:04:32.28 [Moe Kiss]: That’s not admin. What’s the word? That’s checking back to be like, if we said there was going to be some outcome, did we achieve that outcome? I would see that almost as being accountable for measurement and making sure that we hit the success bar and making sure that other people in the business are accountable. I think it’s more when you’re like, I know, Tim, you’re going to have strong views on this. But when you think of monthly reports and cadences like that, and it ends up being about getting people to fill out their section, not, hey, I’m doing the data bit and I’m going to partner with my stakeholder on the commentary or whatever it is, it’s like heckling and following up people and making sure people have done their bit. because ultimately like a data person might be responsible for making sure the reports are not or whatever. I think there’s a difference between like ownership and making sure you’re accountable and like Following up people to make sure they do their job. Oh, this is going to be like a trigger point, Tim. 00:05:37.58 [Michael Helbling]: Well, it’s interesting because I’ve definitely found in my career mode where we would go to the business and we would have like a recommendation or insight from the data, which was all part of our job. A couple weeks later, we’d be in a meeting with the IT department to explain what we wanted to change on the website as a result of that. We’re riding shotgun with the project now. It’s like, wait a second, when do we stop doing the analysis and start being the project managers for the implementation of this? That was when I was like, wait a second, what job do I actually have here? Because you’re kind of like, I’m not now not doing data analytics. I’m now running sort of like an integration task force, if you will. So I don’t know if that’s more like in the line of what you’re talking about. 00:06:22.30 [Moe Kiss]: It’s such a fine line though, right? 00:06:25.18 [Michael Helbling]: Because if you want to see your insight go live, you know, yeah. 00:06:30.08 [Moe Kiss]: And it’s something that I do worry sometimes like data folks are like, here, I’ve got a recommendation. I’m going to throw it over the fence. It’s your choice if you do it. And like not taking ownership. I think part of being a strategic partner is taking ownership and being like, I’ve made this recommendation. We’ve agreed on it. I like, I want to see it forward. And I’m, I’m part of this. I’m accountable to it too, because I’ve made this recommendation. So it is such a fine line between picking up too much of the behind the scenes stuff and what you actually need to do to like see the project or recommendation move forward from a business perspective. 00:07:05.72 [Tim Wilson]: Some of it gets down to just recognizing that if it’s kind of Michael, to your example, it’s when everybody agrees that should happen. I mean, that’s kind of like business 101. If it’s like, well, everybody agrees, but no one actually assigned, there was no ownership assigned. If you can do that in the moment, then a lot of times it’s like, well, who should be doing this? If I wait and we haven’t got it, then it shouldn’t be the analyst. But if everybody leaves and the analyst is saying, well, nobody’s gonna do it unless I step up and do it, That’s a little bit of a shame on the organization, shame on the analyst, but there is that part of like the full life cycle is, does need to go all the way through. So what is the next milestone? Who’s going to do what by when? And then looking at that person and being like, are they going to do it or is somebody going to need to babysit them? Which isn’t, I mean, that’s kind of a reality of business as much as the analyst role, I guess. 00:07:59.75 [Val Kroll]: As you were talking through the admin stuff, Moe, I think the consultancy equivalent of some of the admin work is, can you send me that thing that you told me you were going to send me? Can you send me that thing? Or can I have access to that? Especially if it’s like I need one of your other partners or other agencies to send me or give me access to something. The number of times, like, top of a call, like, okay, moving around a lot. Did you get approval for that one thing? Are we good to move forward with that thing? Which is, like, a lot less connected to meaningful stuff. 00:08:36.49 [Michael Helbling]: Explaining another agency’s data analytics to the client. That’s some shadow work right there. I’ll even just say, listen, I don’t think you want to pay me to explain this to you, so let’s find a different way to do it. Not that I don’t want to help you, but I’ve had many experiences where they’re like, okay, we got this from this. Maybe it’s a different agency that runs a specific program for them, like media or SEO or something, and they’re pulling their own reports. They’re like, how did they get these numbers? And I’m like, okay, so now you need me to go reverse engineer how they pulled these numbers together. And it’s like, oh boy. 00:09:26.22 [Tim Wilson]: That’s not a bad part of shadow work, getting poorly documented, regardless of where it comes from. Somebody wants me to take it forward. The first thing I have to do is basically replicate what was done so that I know what I’m carrying forward, which is just not… Some of that can be addressed by documentation, but that’s like this. There can be this expectation like, well, here’s the number and it links to this dashboard so surely you know everything you need to know. You’re at the starting line. It’s like, well, no, no, I’m still actually back in the locker room trying to get ready to come out to the starting line. 00:10:04.00 [Michael Helbling]: That’s a good point. And getting coordinated so that everyone’s kind of using the same data and everyone trusts the data that’s being presented, whether it’s internal or external, goes to that sort of like, that work in preparation I think is very much a part of what I consider like a data and analytics role to be doing. But sometimes it falls in your lap in a weird way, maybe. 00:10:29.36 [Moe Kiss]: Okay. And I think the thing that comes to mind is the word alignment. So like not all shadow work is shit. Some shadow work is actually very valuable. It’s just the fact that the business doesn’t understand like how consuming it is or how important it is. And alignment I think is one of those things where it really is often about like this business unit thinks this or this client thinks this and this area thinks this and like making sure that everyone is speaking the same language, whether it’s about the metric definition, whether it’s about the outcome of the work or like that alignment pace, I think is incredibly important. But I don’t think it’s always something that the business understands that it’s such a big part of a data practitioners role. 00:11:14.50 [Tim Wilson]: I second that. I mean, I think even the alignment, what is it we ran this campaign? What was it supposed to do? And then the fact that the analysts are like, well, I need to be in the meeting up front like that. We need to make sure everybody’s on the same page of what we’re trying to accomplish. It’s not run it. And then the analyst gets involved because the data now exists so they can pull it and they can provide the answers. that upfront, which I mean, some would say I co-created a consultancy that is geared a lot more around trying to get multiple parties on the same page, so that the analytics work or the experimentation work can be productive and successful is a huge part. 00:12:01.60 [Val Kroll]: I’ll third that motion on sometimes the shadow work is really important to move forward when we first started talking about this topic, the first thing that came up for me and granted I do have very much of a recency experimentation bend. is the culture of experimentation work, how that’s become more prominent, especially on LinkedIn in the zeitgeist about how to be successful with experimentation. But if you think how many other roles around a business have to make space for everything that they’re supposed to be doing after the job description was approved and you were hired. It really is all about like building consensus and getting people excited and a little dose of education, a little dose of this is why you should care about what I do kind of a stuff. 00:12:49.38 [Tim Wilson]: And I feel like sometimes that’s a little… The culture of finance, the culture of accounting. Famous. 00:12:56.70 [Val Kroll]: Famous for going around to get people on board. Well, maybe during budgeting season, but just to go back to the point that it’s not that it’s not important, but it’s usually not the first thing you think of when you’re like, oh yeah, I lead an experimentation team inside of an organization. It’s not that it’s not important, but it’s usually not the first thing that comes to mind. 00:13:17.46 [Tim Wilson]: It does seem like maybe to bridge from that to explaining the realities of the data, which kind of takes two angles. there’s always going to be a presumption that the data is cleaner, more accessible, less ambiguous, which is like, no, our data is a company. It is always wildly more complicated than any kind of new person to it thinks it is. And then there’s the other part of that that is what the data can and can’t deliver. Like the data is the objective truth. So there’s a data fluency component where it does sometimes feel like in analytics, and maybe this is the grass is always greener on the other side. If you’re talking about finance, somebody’s in a financial analyst, somebody would expect that they’re an expert around finance and they can go to them and defer to their expertise. I feel like in marketing and product and digital analytics, sometimes it’s like There’s not a presumption of knowledge of complexity. The shadow work is building trust, building the relationship, walking them at the appropriate pace through why a diff and diff is not appropriate in this situation. educating of the business partners that does feel like it’s a proportionally heavier lift than many other roles. Does that count as shadow work? 00:14:57.94 [Michael Helbling]: Tim, when someone asks, which channel has the highest ROI adjusted for LTV, how long does that take you? 00:15:06.30 [Tim Wilson]: Pull GA4, then export to Excel, write SQL for BigQuery, find my LTV formula. I don’t know, let’s say about three hours in a couple of existential crises. At least two. 00:15:20.99 [Michael Helbling]: This is why ask-wise full-stack approach works. Ask in plain English, prism orchestrates across your stack and applies your saved calculations. 00:15:30.06 [Tim Wilson]: So I’m not manually stitching together five tools like some kind of data Frankenstein? 00:15:35.83 [Michael Helbling]: Nope, everything’s traceable, not a black box. DataState secure, semantic layer, generated code, runs locally. It’s all set up for you. 00:15:46.98 [Tim Wilson]: So for a product with a name that makes you think of a title or asking why, repeatedly, this is pretty sophisticated. 00:15:54.65 [Michael Helbling]: I’m not sure making fun of our sponsor’s name is the move here, Tim. Wait, I did say pretty sophisticated. That’s a compliment. All right, fair enough. Well, go to ask-y.ai, that’s ask-y.ai, and use code APH for priority beta access. Join the rise of the AI analyst. 00:16:19.41 [Val Kroll]: 100%. Yeah, I think so. Or even some of that same concept, the explanation to like backend developers, like you were talking about the business partner audience, but that was one, I think we were talking about this a little bit too much. I know that you have scars, but the story that comes to mind for me, when I worked at the American Medical Association, we were working off of the free version of GA at the time, and we had just gotten an analytics canvas license. to overcome the sampling. So it would like hit like every 30 minutes or every hour or whatever it was so that we could extract air quotes, all the data. And I remember it like there was some, some backend developers were like, Oh, perfect. Now we can just, you can just give us all of your GA data every night and we’ll just throw it into the membership cube. And I was like, it doesn’t work like that. Also, like, what do you mean everything? Like, do you even, but like so many conversations, conversations that got escalated, my boss had to pull me into it. And it was like, you guys, This is not like, I don’t, maybe this is on me at this point for not being able to explain this, but this is a little bit of a nightmare. But the other thing is that membership cube, the ID, the key was the membership ID. And I was like, do you think that the only people who visit our website are members and that they’re authenticating at least once every 30 days? Like you are off your rocker, but it was like, at least, at least three months of my life spent on that topic, if not longer. 00:17:45.90 [Michael Helbling]: And a lot of times shadow work is just cleaning up or trying to clean up a data warehouse you inherited from a previous team or something like that. You know, you walk into an Oregon, they’re like, oh, we want to do this, this and this amazing thing. And you’re like, well, the snowflake instance we have is not going to do any of that till we really clean up a bunch of it. 00:18:05.27 [Moe Kiss]: And you’re helps. 00:18:07.76 [Michael Helbling]: Yeah. 00:18:08.24 [Moe Kiss]: Is this like one of those times where you read my exact life situation that is going on right now around to a huge rebuild of our entire data warehouse for a very specific, like very similar reason, right? Like the data wasn’t structured in a way that we can answer the business questions of today. And so, and I think the thing that’s so hard about projects like this is they’re often huge and very time intensive and unlock the heap of value, but people don’t see the value until like months. 00:18:36.97 [Michael Helbling]: Yes, it’s a long time and it’s hard to go pitch those because it’s not very sexy or very exciting to be like it’s not doing anything but setting up a potential for a future as opposed to delivering a business result. It’s so much nicer to go in and say, hey, here’s this analysis where we can make $100 million more this year if we do X, Y, and Z versus, hey, we need to spend a bunch of money redoing stuff we already have because it’s not doing this, this, and this. Eventually, you can write the business case to show where the value will come from, but man, it’s an uphill battle. I don’t know if that’s shadow work exactly. 00:19:15.27 [Tim Wilson]: I think there’s often, I mean, I will see that example and raise it one with wait for a year. I lived this scenario many times, but the most horrifying one, I think, traumatic one, working with a large pharma company that was using Adobe Analytics, and they said, we’re going to get everything into a Azure you know, data store of some sort. And so many requests, they’d say, oh, we don’t have that yet, but it’s all going in. And they were just locked into these backend developers said, we’re going to take the Adobe’s horribly weird and never really thought through, gotta take the Viz high and Viz low, like stitching like messy, messy, messy data feed data. And they were saying, we’re just gonna pump it in at that raw level. And then we’ll just kind of write SQL queries that people can use. I’m like, the SQL query just to answer how many users came to this page is kind of a beast. But we couldn’t get an audience with them because they were just convinced, which seems very common with developers. I feel like it’s maybe less of an issue if you’re taking an event-driven product analytics perspective, but anytime you’re going to something where you’ve got this de-duping sessionization, developers think of event. They don’t think of the need for stuff to be deduplicated by something. So this idea that, well, we’ll just pump all the raw data in, and then you’ll be set. You’ll just have to write SQL, which then becomes a case of needing to maintain SQL libraries, I think. I don’t know whether, Moee, you’re like, that really doesn’t happen if you’ve done it right, or whether you’re thinking, yeah, no, that happens. Oh, or yeah. 00:21:09.32 [Michael Helbling]: Well, I mean, there’s tools that help with that, like, you know, um, dbt or data form or stuff like that that helps you kind of maintain your sequel and repositories and use it effectively. 00:21:19.79 [Tim Wilson]: But sometimes that’s to me, you’re like, you’ve gone with this, like, let’s Let’s get the full ocean, and then we’re going to add layers on top of it. 00:21:30.99 [Michael Helbling]: And then the downstream is the next question that comes from the business user requires yet another SQL query to be written to build out the next reportlet or whatever. So you put yourself in a pretty challenging chain of events just to get answers to data, which AI will totally solve. So don’t worry. 00:21:51.87 [Moe Kiss]: Literally, that’s about to be my comment. I think the biggest challenge right now is that everyone thinks that you can overcome a shitty data architecture with AI, which is just so fucked and hard to manage because you’re literally that’d be broken unless we have the right data architecture. The same way that we’d need to write a bespoke SQL query or you don’t even know where to point the question because of the way we’ve structured the data. That’s the problem that we need to solve. And yeah, it’s not sexy. Like getting the buying is incredibly difficult for this stuff. And it probably is the hardest. I would say one of the hardest parts of my role right now. 00:22:34.80 [Tim Wilson]: So that is deep because the business partners who ultimately want to get value from it, it’s not going to maintain their attention or technical depth, but the analyst is supposed to be engaging with them and serving them. So the analyst becomes the proxy for the business and is now dealing with the backend. And so they become subject matter experts in an area that has Nothing to do with running analyses or validating hypotheses. It’s just they’re living in that middle tier and there’s just no one else. The shadow has to serve it because there is no one. That’s all there is. That’s the best you got. 00:23:20.60 [Moe Kiss]: spot on and then you end up with like one or two people in the air and the business who know one area and no one else can do it because it’s so complex and there are all these like gotchas so even if you’re going to write a bespoke fickle query it has to go through this one or two people because they’re the ones that know those tables know how to to use it well and like that, then you’ve created your own bottleneck, right? And it’s not an intentional thing. I think often the systems were created with the intent to have a lot of flexibility, but then by having flexibility, you don’t have enough standardization and like, yeah, it’s a chicken and egg. But I would say that is one of the hardest shadow tasks for sure. 00:23:59.33 [Tim Wilson]: There does seem like there’s like a macro thought, this whole topic of the show that it’s like the I feel like I’ve worked with analysts who take the attitude, well, that’s, that shouldn’t be my job. So it’s not my job. So I’m not going to do it. And then it kind of falls through the cracks and doesn’t happen. 00:24:20.80 [Michael Helbling]: So on that meta thing, like there’s something to the idea that like some people by personality are going to be more suited to generalist types of roles versus specialist ones or more drawn to them. And so like, I’m definitely much more of a generalist. So when I find myself running further afield of doing the actual data work and the analysis, it doesn’t bug me at all. It’s actually kind of fun to see something different and do something different for a little while. I sometimes will think about, is this really truly serving our purpose? Are we getting done? We need to get done. But generally speaking, doing those tasks, not a big deal. I feel great about it. But I absolutely think there are people who Like that is much more disconcerting to step outside of the role to do those things and less of something that plays to their strengths and much more plays to like the things they definitely do not want to do. And so like that’s the other issue is just sort of like the person kind of matters a little bit to this too. 00:25:19.40 [Val Kroll]: Yeah. And I don’t think, I mean, at least from my personal experience, it hasn’t been like a conscious choice of like, whether I’m going to step outside or, you know, get in someone else’s lane, but it always feels like I’m tugging on a thread of something that in the moment feels necessary for me to understand what I’m analyzing or to understand root cause of like why that had been a problem. I mean, and a lot of times I personally just get fascinated by like, you know, authentication handshakes and like, you know, all the different nuances in that space. But it always ends up feeling like it’s adding to this like mosaic of my understanding, which always feels like it pays dividends in the future too. So I’ve never, I’ve never tried to quiet that voice. Also, I’m just really nosy. 00:26:10.98 [Tim Wilson]: This reminds me of me going overboard on it, where there were webinars in a company that we think we know what webinars. You have a registration and attendance. This was in a business model where it was not that at all, and it was like bonkers how salespeople would sometimes go into an office and sit and watch the webinars, and there were two or three systems involved. The more I pulled on that thread, it definitely was interesting, but it was like, oh, wow, I was looking at this one table of data and interpreting that attendees meant the number of people who attended the webinar, and that was completely wrong. I wound up writing up It was probably a 10 or 12 page document very, very clearly written because there were all these parties in different places and I thought, nobody has put all this together. I have done the most glorious, valuable. This is so useful. I’m pretty sure not even the webinar business owner. read it. I got probably 25% of the information from her, but I was like, oh, she was excited to explain to me the nuances of the complexity, but I kept digging further and further and saying, aha, look what I, the external consultant, has done to really help you understand what’s going on here. And there was kind of no interest. So that was one where I’m like, it was useful for me. It should have been useful downstream. In today’s world now, boy, I’d be throwing that into an LLM somewhere and saying, that’s really helpful data potentially. But I’m pretty sure that document, I was like, I became the domain expert on something that People cared about webinars, they did not care to hear how messy it was to interpret any of the data that was captured. 00:27:57.26 [Moe Kiss]: The thing that’s resonating with me a lot right now, one of the values that I, I do have leadership values, it’s a weird corny thing. But one of them is be unwaveringly useful. Does anyone, pop quiz, anyone remember where that comes from? 00:28:15.39 [Tim Wilson]: I don’t think so. 00:28:16.75 [Moe Kiss]: Oh, from being useful would be… A good friend, Cassie. Yep. I got it. Ding, ding, ding. Yeah. Yeah. She put it in one of her blog articles and it’s always resonated with me. And I’m completely contradicting myself now because at the start I was like, don’t pick up the admin work. But I’m the first person to be like, if someone’s not doing something and I can add value or move something forward, I’ll normally just end up doing it. So like I am, yeah, a walking contradiction. But I do think there is part of that. responsibility of data folk like I tend to get really frustrated when a data person is like, well, that’s not my job. And I’m like, your job is to help the business make better decisions. So if there’s something you can do to be useful to help the business make better decisions, that is your job. Yeah, I don’t know. That’s just the thing that’s bubbling around in my mind at the moment as we’re, I mean, not relevant to Tim’s example, but more broadly about this area of like sometimes it is about getting the domain expertise. Sometimes it is about documenting something that no one in the business has written down. It’s like, sometimes those things are less useful, but a lot of the times are really useful. 00:29:20.48 [Michael Helbling]: just to give some people who might be listening a chance to sort of be like, well, maybe, Moe, I can’t do that thing or I’m not good at that thing. Is it necessarily that you have to go personally be the one in charge of that as much as be part of helping solve it in some way, see that it gets done? So it’s more of like the ownership taking versus the taking on the role and doing it yourself, just so that people who are very specialized or don’t Yeah, because I have a ton of empathy for people who are like, Michael, I just can’t get up in front of people and talk. Cause like I analyze data and that’s what I like to do. And I very stressed out every time I have to go present something. And it’s like, okay, well then has someone else can do that part, but like you just need to make sure you’re a facilitating it up to the moment where it, where it happens. So it doesn’t have to be you necessarily taking on that role. I don’t know if I agree. So don’t pick presenting something then, something else, like managing the project or something like that. But the point being, like, not every person fits every single role. Like, you don’t have to be a polyglot, if you will. Or a polymath. 00:30:27.29 [Tim Wilson]: What that, I mean, if you… Poly PM. First break all the rules, like the precursor to the now discover your strengths, strengths finder, which… But first break all the rules. I’ve always, to that same point, identifying what needs to happen. I think, Moe, that’s the brilliant way to frame it. What is your job? It is not to write SQL. It is not to develop reports. It is not to deliver results. It is to move the organization forward by helping them make decisions. If you say, well, That means that somebody every Tuesday morning needs to reach out to this one person and ask them a question. Like it can be frustrating. It can suck. But you know what? There’s somebody who’s actually super sociable, who loves to ping people or whatever. Like building up that list is kind of, these are the discrete tasks. Not that somebody’s going to love and relish doing every one of them, but it does at a team level. help start to shift around, like, oh, somebody needs to document these database tables, or somebody needs to ask why Guru, they need to know how that tool works really well, figuring out who gravitates to it. I do think there’s, and I think I was cringing similarly with Michael grabbing a random example, there is a fine line between what is a complete analyst need to be able to do and do even if they’re outside of their comfort zone. So it’s, it gets a little squishy. Which of this is shadow work that like somebody’s got to do it, this person gravitates to it. Which of this is going to be a really ineffective handoff because someone just doesn’t, doesn’t want to write sequel. I mean, they’ll use that example. Somebody, I don’t want to, I’m just not the kind of analyst who’s going to learn to write code. It’s like, cool, then you’re not the kind of analyst who’s going to progress particularly far in your career. So, cool. We got it. 00:32:32.88 [Michael Helbling]: Hey, I’ve gotten pretty far. So, you know, no, now you can’t do it. You can’t. 00:32:39.32 [Moe Kiss]: I don’t want to get into team dynamics too much, but I do think a big part of figuring out the shadow work as a team is figuring out who had strengths for different parts of it and we’re making sure people lean in. I know in my previous team, we had a really big gap of, we didn’t really have someone who was really good at the I would say leadership team documenting stuff, pushing it forward, hyper-organized, being like, hey, Moe, these are all the things we have coming up in this time frame. We very intentionally hired someone that was really strong in that space to complement our team. I think that we really need to be thoughtful of what are all those things, especially the shadow work, because if you put someone on something and that’s their strength, it’s so much easier for everyone. They feel like they’re adding value, that the balance feels better. And to be fair, there are some things that no one particularly wants to do, and then it just comes about making sure everyone takes a turn. 00:33:43.61 [Tim Wilson]: Can we hit on that stuff a little bit? And maybe this administrative work, maybe more broadly, because I think that is the danger there. And I do think I’ve seen stuff written that women are much more likely to get screwed on this one, is that this thing needs to happen. And they’re like, oh, well, it’s admin work, like the latent misogyny Maybe not intentional is, well, Moee’s really good at that, but it’s absolute shit work, and she’s not going to speak up. I think there is that the shadow work that needs to be done that has value, that is being done as efficiently as possible, and there can be some gravitate to it. Shadow work that is has to be done. There is value. No one wants to do it and making sure that that doesn’t fall to the passive nice person because because that can spin out where wait now half of your job is unseen shadow work and you can’t advance in your career. Even though everybody’s like, well, this all needs to be done. But good old Jane is, you know, always there for it, you know, but it’s in the shadows. It’s not getting 00:34:59.87 [Michael Helbling]: Yeah. That’s not visible. So this is actually kind of an interesting pivot, Tim, because as you turn into a leader in your space or leading teams and those kinds of things, your job becomes taking the work out of the shadows for some of the exact reasons you just said, because it needs to be recognized. What’s being done, the people doing it need to be recognized. And then who should be doing it, should be much more strategically thought out as opposed to quote, fallen into just because, oh, so-and-so is more agreeable, so they just take it on without fighting too much, which is just a terrible solution to the problem. So anyways, I thought that was a really great point, Tim. And I think that’s sort of the thing that maybe take away is like, when you turn from an individual practitioner or individual contributor into a leader, you know, when you’re just an IC sitting at your desk, you’re like, wow, do all the shadow work. When you’re a leader, you’re like, we need to take the shadow work and expose it to the light. 00:35:55.99 [Tim Wilson]: That sounds hard. That’s why I’m not going to, I’m not striving to be a leader. 00:35:59.48 [Moe Kiss]: I do think though it also is about like recognition. And like one of the things that I would say like, and I’m thinking of this particular person, like I know at the moment their rating would be very good or like they’re like an assessment of their performance, right? Because I value that work. And so I think where the challenge is is like, when there’s that tension where someone’s like picking up a lot of shadow work, that then is not valued or not given the value that it’s deserved. Whereas I see it as like being incredibly essential. And if you do that shit well, like you can unlock a lot for your your team or the business. And so like, I want to make sure that that’s rewarded and reflected. So it there’s a lot of new ones, though, like, obviously, it’s very dependent on specifically like what paths we’re talking about. And yeah, and many factors. 00:36:50.66 [Tim Wilson]: I’d just like to say to all of Moee’s team who’s listening to this podcast, she’s talking about you. 00:36:54.83 [Moe Kiss]: She values you. 00:36:56.86 [Tim Wilson]: Oh, wow. She gave us the name off Mike and it was your name. So good job. 00:37:00.83 [Moe Kiss]: Stop it. You were so cruel. 00:37:03.66 [Val Kroll]: The other thing that this is making me think about is that when any in-house role that I’ve had, I’ve never reported to an analyst. It’s always been, you know, ahead of digital or someone else who it was really hard to message up not only for myself when I was the IC, but then when I grew my team about all the things that takes like, I’ll say, do you think we just sit there and like convey your belt? Just like analyze, analyze. Like that’s so not all that the job is, right? So there’s a lot more. education in that scenario, whereas I was thinking about your comment, Michael, like with the elevation of analysts and to those leadership roles that there’s a lot more visibility and line of sight. So I agree with you on the accountability we’re going to put on any listener to bring that work out of the shadows and acknowledge and like what you were talking about both. So that’s a really good point. 00:37:55.11 [Michael Helbling]: I think what we’re finding out is that the work has value. Whether we should be doing it or not as analytics people isn’t necessarily all the story. Sometimes you should go back and say, workflow-wise, the solution should be to take this group and pull them into this piece of work. rearrange it and come up with a strategy. My early example, Tim, you pointed out, we exposed basically an organizational workflow flaw when we came up with an insight and then had to go drive the insight through the org. What we exposed was no one had thought about, hey, what if we have an optimization, we want to make a reality? How does that get done in our company? Well, somebody should have probably thought about that, and so that was the work that had to be done was to figure out and create a machine that would take care of that. But it’s the same thing with all the rest of it. It’s sort of like, okay, well, what are the parts that need to move into the right places to get it done? Not necessarily you, the data analyst should do it, but that it gets done because it is valuable work at the end of the day, especially if it’s actually driving impact or decision making in the organization using data, which is sort of like the thing that makes me smile anytime I get a chance to be part of something like that. 00:39:13.85 [Moe Kiss]: Can we talk about data quality? We have not touched on that at all. 00:39:17.96 [Michael Helbling]: It’s usually pretty good. Yeah. I mean, just kind of automatically. Yeah. What do you mean? What was there to talk about? So I’m pretty sure. 00:39:25.89 [Moe Kiss]: I think it’s going. 00:39:26.37 [Michael Helbling]: I think it’s going. 00:39:27.19 [Moe Kiss]: Oh my God, stop. Everyone stop triggering me. 00:39:30.04 [Michael Helbling]: Sorry. Sorry, well. 00:39:33.50 [Moe Kiss]: Just come on. I think the one that I’m specifically comes to mind is Bend sent from a media agency. And I just get so frustrated or from a finance team. 00:39:56.95 [Michael Helbling]: Talk about the highly formatted Excel files you might be receiving. 00:40:03.09 [Tim Wilson]: In wide format when they should be in a long format. 00:40:08.30 [Moe Kiss]: Of course. I’m glad you could all laugh about it. I am not at the laughing stage. 00:40:14.39 [Charles Barkley]: Sorry, well, this is probably a whole episode we need to do on stuff like this. 00:40:20.95 [Moe Kiss]: But it just, I think what’s so fucking hard is that your stakeholder will be like, especially the one that owns the relationship with the media agency. I didn’t get it. They sent a spreadsheet over on Moenday. Like, you’ve got the data. What’s the problem? Like, why is it going to take you a week? And you’re like, Do you know that every single city that they run media in is in a completely different format and we then need to sense check it with our record? No, that is a huge lift. And fuck. Anyway, and then you’ve got some very senior, brilliant data scientist that is spending their time basically QAing data. It’s really frustrating. 00:41:08.94 [Tim Wilson]: That is one of those cases where that’s another shadow that the analysts can fall into where they’re the bridge between the data creation. That data may be created out of some contractual necessity, but doesn’t have any real incentive or stake outside of what’s in an agreement. It’s like, oh, we’ll send you data. We’ll send you data. Check the box. And this is going after media agencies pretty hard, that a lot of times they don’t really under, they’re like, whatever the platforms, you know, trade desk spits this data out or runs into our data warehouse and we’ll just give you a feed. And the analyst is the one who winds up having to explain their data to them. So it’s like another version of that. That particularly is another version of what you were talking about earlier, Michael, where you have to be like, Yeah, how can this possibly be zeros across here? It’s like, wait a minute, I’m now having to reach out to… Everybody seems to assume that it’s coming in fine, but I have to set up time to go three levels deep with some partner to get them to agree that it’s actually a problem or explain to me why it’s not a problem. 00:42:27.06 [Michael Helbling]: I’m literally in a situation like that right now. I ran into a situation just this past week where a company is like, yeah, we’re pretty sure the quality of the data in this system is great, and so I get my hands on it and immediately see three things I’m pretty sure making their data quality really bad. And so you’re literally starting out with sort of like, okay, well, our first conversation is gonna be, guess what? The data you thought was really good? Not good. And there’s a number of fixes we’re gonna need to do before we even start on the things we wanna get further along. And it’s frustrating but real, right? So it’s sort of like, yeah. And then the other one that gets me sometimes is sort of like alerts and notifications, anomaly detection and those kinds of things. That is a part of data, but it’s not really what an analyst does necessarily. 00:43:15.77 [Tim Wilson]: Well, the analyst gets blamed if the data all of a sudden it’s found that something wasn’t there for weeks. They’re like, what were you doing as an analyst? How did you not notice? 00:43:24.44 [Michael Helbling]: Raise your hand if you’re the only one that’s had your own secret dashboard so you don’t get caught up in one of those things. So you have advanced warning of something that’s happening. 00:43:34.23 [Moe Kiss]: I think anomalies is part of our job, but you will keep saying analyst, and I think of data practitioners, whether it’s a data analyst, analytics engineer, data scientist. For example, if there is something in our B2B pipeline that breaks our leads coming through that is absolutely data quality and normally detection and I would expect an analytics engineer to go in and solve that. Absolutely. When we’re doing at the complete other end of like a metric goes up, a metric goes down, that sort of stuff, again, I would expect a data person to go in and kind of debug that. It might, they might not be responsible for the complete like up level, you know, challenge of why that thing is or isn’t working anymore. But like, I would expect someone to be pretty across that if we saw like a number tank or something like that or a number skyrocket. 00:44:28.59 [Tim Wilson]: But, but that’s the, I mean, the way you just framed it, not to, I mean, you’re just speaking off the cuff that a, There is a perception that, yeah, yeah, yeah, they need to catch if a number of tanks are a number of skyrockets. In practice, every time I’ve had a system where it’s like trying to tune where, like there’s not a threshold, then there will be platforms out there that say, look, you can set this at a 95% threshold, set up 100 alerts. I’m like, cool, I’ll get on average five alerts a day. 00:44:58.94 [Moe Kiss]: I’m not necessarily expecting a data person to catch them all. I think that’s a really hard thing. It’s so difficult, right? Because if you have a stakeholder who comes to you and is like, hey, this number declined and you’re the data person who’s like, what? I had no idea. That’s shit. It’s hard for trust. But at the same token, expecting a data person to be able to be ahead of the game on every anomaly is also not an expectation I have. But I would I would basically be like, okay, something has gone wrong here. I’m going to reach out to my stakeholders. I’m responsible for letting them know. I’m responsible for letting them know what we’re doing to investigate, how we’re going to solve it, keep them updated. That, absolutely, I do think is a data person’s role. 00:45:47.72 [Tim Wilson]: I still have the alert turned on for a certain tax preparation company that you and I worked on years ago and like January 12th, their home page was down from Seattle because I just never turned it off. But that was one where they were having sporadic Issues and it was like somebody should be monitoring this and I can go set something up And I had to set up on like my personal account and I just never turned it off. 00:46:12.89 [Michael Helbling]: So literally Michael knows the brand I know the brand it was down for about 35 minutes Yeah You need to do some account access cleanup that’s some shadow work that a lot of consultants have to do Get yourself off of those old GA accounts or Adobe accounts that you’ve been on for years and years that you no longer work with. 00:46:36.08 [Tim Wilson]: No, this was using Site 24 by 7. I was doing like a ping tracker that I set up, so I had set it up. So a third-party tool. 00:46:47.24 [Michael Helbling]: You’re doing third-party data collection. I was using a third-party tool. And they’re probably like, why is our website getting crawled by this website? 00:46:55.54 [Tim Wilson]: But that was, they were sometimes saying like, the tool is down. And I’m like, no, like why is this anomaly in the data? Cause your fucking site went down. Like, that’s not a, cause I think I set up a ping for the footer as well. Cause based on where they had the tagging track, but I think it started with them saying your digital analytics, your web analytics data is bad. And I was like, yeah, that’s weird. What’s going on? It’s like, well, no, the whole site went down. 00:47:21.07 [Moe Kiss]: No, I didn’t once find, though I was working somewhere there was like an issue that I couldn’t figure out like why this number had gone weird or whatever. And then like a month after I left, I figured out why. And it was like completely tangential. I was just working on something different. And I did reach out to let them know. I was like, Hey, this is probably what this was. You should fix it. Here is how to fix it. You’re welcome. I’m not a shit human. I want everyone to have the best data they can. 00:47:48.07 [Tim Wilson]: I also get the, it’s backup. So every time I’ve seen it, it’s come back up quickly. So there hasn’t been a point. 00:47:54.94 [Val Kroll]: Okay. So before Michael wraps, cause you got that look in your eyes. I would love to hear. love to hear people’s thoughts on shadow work, not shadow work for like data fluency, data literacy. We’ll call it, we’ll call it, cause data literacy programs I think are one of the more common ways people talk about it because it is like a whole category of work. Yeah. I like data fluency. I think it’s less obnoxious than data literacy. 00:48:24.33 [Michael Helbling]: Everybody can read and write. 00:48:25.89 [Val Kroll]: Yes. So is it shadow work, not shadow work? I think it’s shadow work, but I think it’s important shadow work. 00:48:31.99 [Michael Helbling]: Yeah, I think it goes back to that sort of like what do you need to do to help the organization take a step forward with data, make decisions, use the data, be effective with the data. And a lot of times that’s building up data fluency in an org or helping people build up their data fluency. 00:48:48.85 [Tim Wilson]: But that’s one where if you try to bring it out of the shadows and say, oh, why don’t we just solve this once and for all and send everybody through a data fluency program, pretty ineffective. So it’s the thing that needs to be in the shadows that is a I mean, not that there’s not the opportunity for some of that training. I feel like I’ve been learning how much, I mean, it’s not, it’s the reality of a short attention span that the more you can have like in the moment, like, let me come up with, let me show you this now. Let me explain this little thing now. Let’s talk about, oh, you know what? When you all people say correlation is not causation, this is like the perfect example. Let’s talk about that for five minutes because that’s a trap you’re falling into. 00:49:30.55 [Moe Kiss]: Tim, it actually makes me think about gender bias training and all the research on that, where lots of companies do gender bias training. It doesn’t necessarily result in any differences in behaviour or attitudes or anything, but it’s a tick the box thing. When we start talking about like data fluency or training or education or whatever it is in the data space, I think what happens when we sometimes roll out those programs with really good intent, it’s a tick the box thing. But again, like those in the moment. 00:50:01.93 [Tim Wilson]: That sounds like the sort of observational woman would make, by the way. 00:50:05.21 [Moe Kiss]: We’re going to send Tim back to training. Those in the moment discussions are actually what I think is makes it so hard because it is shadow work because it’s not like I built a program, I’ve shipped it, I’ve ticked it, it’s done. It’s like every time I talk to the stakeholder, I’m trying to help them get a little bit further in how they think and understand data. And that is like you’re never done, you’ve never ticked the box. And so it does have like a very heavy cognitive load, but it’s incredibly important and probably leads to the best outcome I say without a data informed opinion on that at all. Just like that. 00:50:51.84 [Val Kroll]: I’m actually surprised that you guys are all on the same page. I don’t think it’s shadow work at all, whether it’s bite-sized or a big part of it, because even some of the criteria we were talking about using earlier, if your role is to help the business make smarter decisions, like making sure that you’re connecting what you’re finding, what you saw, what you observed, what you validated, what your recommendations are with like what the business can actually be doing with that information. It feels like it’s a, I don’t know, to put it another way, there was a leader who I worked for at UBS who like the four D’s of product development, like the defined design, develop, deploy. He always said there was a fifth like shadow. not actually a fifth one of adoption. Until you understand how people are using that or if this is a data product or whatever, then you’re not done. The work isn’t done when you ship it. The work is done when you understand and create the feedback loops. I feel like it’s very much in the same vein of how to make sure that your work continues. Michael, the work almost similar to what you were saying, creating the processes so that the team knew how to take advantage of those recommendations. I don’t know how I just feel like it’s not Like you’re not done just when the analysis is complete, or it’s not. 00:52:09.76 [Tim Wilson]: Yeah, shadow is optional. Like it has to happen, it’s just not identified as something. 00:52:13.93 [Val Kroll]: It feels like it’s squarely in the court of, I would expect it to be in a job description. Like, that’s what makes me feel like it’s not shadow work. 00:52:21.44 [Michael Helbling]: Again, that’s where I think some of this work should rise up out of shadow work. But again, it’s about recognition. The importance of it, I completely agree. But Tim’s point, I think, was, will you see it in a job description? Probably not. Or if you do, it’ll be run a once-in-a-quarter training and call it done. And we all know that’s not going to be effective. But it’s spending that time, like I’m realizing this episode that like 90% of what I do is shadow work sometimes. It’s so hard to pin down. Michael works the shadows. That or I just don’t do anything. I don’t know. But I remember I had a very specific instance where I had a review and my boss at the time was like, you’re not spending your time the way that it should be spent. And I had to actually walk him through. If I spent the time the way that they wanted me to, it would lose the company money. And I walked him through step by step. If I actually did it the way you said, the company would lose money as a result of the effort. So what you’re telling me is that you would like the company to lose this much revenue by changing what I do day to day, are you sure that’s what you want? And so it was a really interesting conversation because I was able to enunciate exactly where the value lied in each of those things that I was doing. I could show the outputs of those things. But it was a very interesting conversation because it was like, oh yeah. Now, in that case, I had actually prepped that person ahead of time by showing them exactly how I was going to spend my time. They just ignored it and came back with the template. 00:54:01.66 [Moe Kiss]: What was the outcome, though? Like, what was the end of the story? Did you get off to change it? Or did they, like, be like, oh, I see the value of what you’re doing. 00:54:09.14 [Michael Helbling]: I kept going. Yeah. No, I was, uh, that was a role in which firing me probably wasn’t an option. Probably they felt like it in the moment. I sent an email to the head of HR ahead of that meeting being like, I’m about to chew up my boss. Um, But it worked, and I still had a good relationship with that person afterwards. But it was a situation where they were like, oh, OK, well, never mind then. And I just kept going with what I was doing, since it was made sense. 00:54:36.06 [Tim Wilson]: I will claim to the job description that I think when we read job descriptions and say, well, this is looking for a unicorn that’s ridiculous. Or when we read a job description and say, wow, that looks really good, I bet, I’m thinking through some that I’ve seen, the ones that actually have the shadow work articulated as part of the responsibility is collaborating with the business partners to how to ask questions in an informed way. That actually may be, it would be fascinating to look through some job descriptions that when people say this is garbage and say, is any of the shadow work captured? Hey, this one looked, because you’ve had that reaction where you look at one and you’re like, oh, they get it. Like they actually, they’re describing a realistic and practical role. And I bet that that is because there are nuggets of what you’ll be expected to do, include some of the shadow work we’ve talked about. 00:55:30.36 [Moe Kiss]: Okay, but just a push on that I do agree. I think the challenge though is like In my role we write we write the job descriptions for data people data people are writing job descriptions for data people So you can still have a mismatch with the stakeholder of what they think a data person should be doing like so I’m just saying it’s not like bulletproof 00:55:52.79 [Tim Wilson]: Yeah, but I think if that’s recognized, it’s like, hey, we’ve got a bunch of really difficult, unrealistic stakeholder. We should have in the job description that part of this is collaborating with, not you don’t say collaborating with assholes, but you’re like, You know, collaborating with, educating, informing, iterating with, so I think he can still be captured. 00:56:18.53 [Val Kroll]: He’s ambiguous in challenging circumstances. That’s right. Yeah, exactly. Oh, yeah, there we go. 00:56:24.88 [Michael Helbling]: Sell starter, able to juggle multiple priorities simultaneously. It’s like, ugh. 00:56:32.90 [Tim Wilson]: Often when the hiring manager isn’t an analyst, then that’s why that job description doesn’t have the shadow work in it. And that does some of the. 00:56:39.96 [Michael Helbling]: And it comes out ringing false. Yeah. Well, some of my shadow work is trying to get the show wrapped up on time. So let’s go to do that. 00:56:49.96 [Tim Wilson]: We got to find somebody who’s good at it. 00:56:51.36 [Michael Helbling]: All right. Let’s hand that off to somebody else. All right. Well, listen, Moe and Val and Tim, thank you so much. This is, I think, a really interesting topic. And I appreciate your insights on the show. A lot of work is really important, but doesn’t necessarily get recognized for what it is. And I think that’s sort of where this discussion took us today. So thank you for that. You know, as you’re listening, I imagine you’re thinking some of the thoughts yourself. We’d love to hear from you and you can reach out to us. You can reach out to us on LinkedIn or the Measure Slack chat group or through email at contact at analyticshour.io. And if you’re listening to this on Apple podcasts or Spotify or whatever platform you listen to it, give us a review or a rating or a comment. We’d love to see it, love to hear it, love to hear from you. And of course, a couple other things. We’re not doing less calls, but a couple of things where you can find us coming up this year. is at a couple of few conferences and actually coming up really quickly. So, I know Tim and Val, you all will be at the Datatune conference in Nashville. Is that right? You want to talk about it? 00:58:09.09 [Tim Wilson]: Yeah. It’s a little, it’s a Friday is workshops and Saturday, it’s a Conference, it’s a pretty low cost, low three-figures conference all day. It looks kind of not measure campy from an unconference perspective, but from a enthusiasm and critical people, a lot of people, critical mass of people showing up pretty interesting topics. 00:58:32.32 [Michael Helbling]: What are the dates? 00:58:33.02 [Tim Wilson]: Oh, that would be important. Yeah. 00:58:37.65 [Michael Helbling]: I’m here for you. 00:58:38.81 [Tim Wilson]: I’m here for you. 00:58:40.11 [Michael Helbling]: Talk about shadow work. 00:58:43.88 [Tim Wilson]: What is it? 00:58:46.78 [Michael Helbling]: That’s awesome. And then, of course, Measure Camp New York will be in March 28th in New York City. It’s officially in New York City, not New Jersey this year. 00:58:56.14 [Val Kroll]: Very exciting. Very exciting stuff. 00:58:58.06 [Michael Helbling]: Yeah, it’s going to be a great… Measure Camp is always a great time. Obviously, Val’s super involved with Measure Camp Chicago. Moe with Measure Camp Sydney. Tim with Measure Camp Columbus. Me with not being involved with Measure Camp in any official capacity, but I love going to them. And I think right now Tim and I are planning to be at that one, and that’s March 28th in New York City. And then finally, April 28th and 29th, the whole Analytics Power Hour, or a lot of the Analytics Power Hour folks will be at the Marketing Analytics Summit in Santa Barbara, California, which sunshine on the West Coast. Hello. Get there. We love to. 00:59:35.50 [Tim Wilson]: We got some exciting plans for that. Stay tuned to future episodes. 00:59:39.92 [Michael Helbling]: What’s the drink that you have in Santa Barbara? What’s like a good cocktail for that? 00:59:45.15 [Tim Wilson]: I’m sure it’s some fruity California liberal. 00:59:48.22 [Michael Helbling]: Wine. Exactly. Wine. White wine or rosé on the beach or in the sunshine. 00:59:54.24 [Moe Kiss]: Love this. 00:59:54.70 [Michael Helbling]: Love this from me. I don’t know. I’m terrible at picking out drinks. All right. That’s the show. We’re excited to have brought it to you. And I think I speak for all my co-hosts when I say, no matter whether the work is in the shadows or way out in the open, keep analyzing. 01:00:15.26 [Announcer]: Thanks for listening. Let’s keep the conversation going with your comments, suggestions, and questions on Twitter at @analyticshour on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst. 01:00:33.22 [Charles Barkley]: Those smart guys wanted to fit in, so they made up a term called analytics. Analytics don’t work. 01:00:39.83 [Charles Barkley]: Do the analytics say go for it, no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 01:00:53.06 [Michael Helbling]: Lacy Fusion Productions. Lacy Fusion. 01:00:58.51 [Tim Wilson]: That’s our production studio’s sister organization on Southern Hemisphere covering Lacy Fusion Media. 01:01:05.83 [Michael Helbling]: 4th floor productions, Lacyfusion Media. 01:01:08.74 [Val Kroll]: Known for expanding into Australia. 01:01:12.26 [Michael Helbling]: Ken Riverside. And the Lacyfusion Media. Present a 4th floor production. 01:01:21.68 [Val Kroll]: Okay, well screw your green bars. You sound like you’re in this building with a paper cup and a string. All right, love you. 01:01:43.82 [Michael Helbling]: Your temperature, Matt. 01:01:50.70 [Moe Kiss]: She is so cute. 01:01:52.76 [Michael Helbling]: I know. It’s ridiculous. 01:01:55.49 [Moe Kiss]: So cute. 01:02:02.79 [Tim Wilson]: Rock flag and who knows what insights lurk in the tables of our databases. The shadow analyst knows. 01:02:15.79 [Michael Helbling]: Nice. That’s actually pretty close. 01:02:20.54 [Moe Kiss]: I’m like, Damn, you got the voice. 01:02:23.81 [Tim Wilson]: That’s got something. The post #291: The Data Work that Lives in the Shadows appeared first on The Analytics Power Hour: Data and Analytics Podcast.

February 3, 20261 hr 6 min

#290: Always Be Learning

From a professional development perspective, you should always be learning: listening to podcasts, reading books, connecting with internal colleagues, following useful people on Medium and LinkedIn, and so on. Did we mention listening to podcasts? Well, THIS episode of THIS podcast is not really about that kind of learning. It’s more about the sort of organizational learning that experimentation and analytics is supposed to deliver. How does a brand stay ahead of their competitors? One surefire way is to get smarter about their customers at a faster rate than their competitors do. But what does that even mean? Is it a learning to discover that the MVP of a hot new feature…doesn’t look to be moving the needle at all? Our guest, Mårten Schultzberg from Spotify, makes a compelling case that it is! And the co-hosts agree. But it’s tricky. This episode is brought to you by Prism from Ask-Y—your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show (Article) Beyond Winning: Spotify’s Experiments with Learning Framework (Article) Two Questions Every Experiment Should Answer (Platform) Confidence by Spotify (Article) Choosing a Sequential Testing Framework — Comparisons and Discussions (Article) Bringing Sequential Testing to Experiments with Longitudinal Data (Part 1): The Peeking Problem 2.0 (Article) Bringing Sequential Testing to Experiments with Longitudinal Data (Part 2): Sequential Testing (Article) Risk-Aware Product Decisions in A/B Tests with Multiple Metrics (YouTube Channel) 3blue1brown by Grant Sanderson (Article) Escaping the AI sludge…why MVPs should be delightful (Conference) DataTune in Nashville: March 6-7, 2026 (Conference) Marketing Analytics Summit in Santa Barbara: April 28-29 (Article) The next data bottleneck by Katie Bauer Photo by Jason Dent on Unsplash Episode Transcript00:00:05.75 [Announcer]: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. 00:00:15.90 [Tim Wilson]: Hi, everyone. Welcome to the Analytics Power Hour. This is episode 290. I’m Tim Wilson, and I’m joined for this episode by Val Kroll. How’s it going, Val? 00:00:25.38 [Val Kroll]: Fantastic. Excited for today. 00:00:28.80 [Tim Wilson]: Outstanding. Unfortunately, we were supposed to also be joined by Michael Helbling for this show, but he’s gone all on brand for the winner and gotten the flu. Luckily, as we’re into our 11th year of doing this show now, we’ve learned a thing or two about rolling with the punches. And as it turns out, learning is the topic for today’s show. I mean, it’s implicit in all forms of working with data. We’re looking at analysis or research or experimentation results and hoping, just hoping that we come out of the experience with a deeper knowledge of something. I mean, and hopefully it’s something useful, more knowledge than we had before. It’s a simple idea. Sometimes though, it’s a little harder to execute in practice. That’s why we perked up when we came across an article from some folks at Spotify called Beyond Winning, Spotify’s experiments with learning framework. We’re excited to welcome one of the co-authors of that piece to today’s show. Mårten Schultzberg is a product manager and staff’s data scientist at Spotify. He has a deep background in experimentation and statistics, including actually teaching advanced statistics in a prior role for a number of years. So who better to chat with about learning? Welcome to the show, Mårten. Thank you so much. Excited to be here. Oh, right. It’s a borderline giddy about the topic as we were diving into our excitement before we hit the record button. Yeah. 00:01:59.34 [Val Kroll]: We definitely fought over who got to be on this one. 00:02:05.69 [Tim Wilson]: Mårten, in the article that I referenced in the opening, which we’re definitely going to link to in the show notes, it’s a great read, you and your co-authors make the distinction between a win rate and a learning rate for experimentation. That’s the premise of the article is this win rate. this learning rate as a proposed metrics or a metric that’s actually in use. That seems like a good place to start. Maybe can you explain what you were seeing as a drawback to too much focus on win rate as a metric for experimentation programs? 00:02:43.90 [Mårten Schultzberg]: Yes. I think it needs to take a little step back. I think it started with When we rolled experimentation out at Spotify properly, like at scale 2019-2020, we quite quickly realized that one of the biggest wins that we made over and over again was to detect bad things early and being able to avoid them. So using it as a sort of dodge-bullets type of mechanism. And we have used it like that since. It’s one of the biggest reasons why we run so many experiments. We want to avoid shipping bad things that happens, you know, unintentionally. Side effects and stuff like that. And at the same time, I’ve seen over the years a lot of blog posts and papers published about win rates from other companies. Win rates as in the rate of experiments where you find a variant that is better than the previous variant and you ship it. So a clear winner. And so I just felt that it was sort of under celebrated all of the other types of wins that you can make besides finding something that was better than the current version. And I also think that it doesn’t really reflect how most companies, at least the companies I’m familiar with, are actually using experimentation. They’re using experimentation partly to optimize things. So to find winners, to continuously improve something and optimize it. But that’s only one part of that puzzle. The other part of using it as a mechanism for safety and safety net is something that wasn’t, I think, talked about enough. And so that’s sort of where this sprung from. 00:04:22.92 [Val Kroll]: I love that. And the one thing though that I think is, I would love for you to talk a little bit about more, that I think even if an organization was like, yes, like in spirit, I completely agree with that premise, Mårten. It seems like using a metric like learning rate seems squishy. Like win rate is objective. We can tally that in a column and calculate that percentage. But can you talk a little bit about how you thought about the criteria for determining how you say, yes, we learned something from this experiment or how it’s defined. 00:04:56.01 [Mårten Schultzberg]: Yeah, and so yeah, I want to firstly call out that this was a team effort. It was a lot of people involved. It was driven by the central experimentation team at Spotify, but there was also a lot of other data scientists that are actually doing product work that was involved in this discussion. We had a lot of really good discussions actually about what learning means and when you actually get value from an experiment. And so I just want to call that out. And I think We see it as there are essentially three ways that you can learn from a Navy test. One is that you find an obvious winner. So what other people refer to as win rate. So you find a version that is better than the current version. The other one is that you find that the current version was worse somehow, that you detect something bad, that you detect the regression or Often that can be, you know, not only that users didn’t like it, but more that maybe something went wrong with some integration somewhere, so you get latencies increasing or crashes increasing. And so those are quite obvious wins, so the finding better stuff and avoiding worse stuff. And then there is the middle one, which is more nuanced, which is when you run a well-planned experiment and you find nothing. So a neutral experiment, which is, I guess, vague. But what we count there as a win is an experiment that actually had a sample size calculation before that did a proper power analysis and said, hey, I want to have a certain power of finding an effect if it exists. And then they ran that experiment according to that plan, and they found nothing. We also view that as a learning, because at that point, they can actually, with the certainty that they hoped for, say, no, there was no effect from this change. So the neutrality in that case is informative, because you can say, hey, maybe this is not worth pursuing, because we actually ran a proper experiment. If there was an effect of the size that we were interested in, or that we hypothesized, we would have found it. So there are those three cases. And obviously that middle one, the neutral one, is a little bit more complicated. It’s more complicated to implement or to instrument because you need to know what sample size calculations were run and if the experiment actually met the planned sample size and all of those things. Fortunately for us, in our tool, it’s fairly easy to do. But yeah, take some thinking to get that right. 00:07:27.62 [Val Kroll]: I’m literally writing those because there’s so many things I want to dig into. But before going to the 5,000-foot view, I guess I’m just curious about the culture change internally. with so many people with access to run experiments and this appetite for experiments, what was it like to get them to shift away from the win rate to this other new metric that you rolled out? I’m just very curious what that experience was like if there was resistance, if there was excitement, or some people were really questioning it. 00:08:01.99 [Mårten Schultzberg]: There’s always people questioning everything at Spotify, which is one of the things that I love about Spotify. So that’s a constant. But yeah, I think because of the fact that we so early realized that experiment was such a powerful tool to avoid mistakes and to detect bad things early, I think that the sort of common definition of learning was already incorporating that aspect of experimentation. I think a lot of people has sort of over the years learned to, I should not use the word learned, come to appreciate that, yeah exactly, come to appreciate that avoiding something bad is a great learning and something that is super valuable for product development. So I think that part was not so controversial when we developed this metric. I think the neutral one is trickier and there’s also It’s a much more room there for discussions about what should count, should you be super strict about that it should be exactly powered, should you allow some wiggle room, there’s a lot of things that you can discuss there. We were eager to get a very clear and explicit definition out and we were also eager actually to write about it externally because we were hoping that other companies would, and I guess this podcast is a good example of that too, that we could have this discussion because I think it’s been I’m really curious how other people think about this. I’m not convinced that our definition of learning is like the ultimate one or the final one or anything, but I think it’s a good first step away from the more naive, only wins count definition. 00:09:53.50 [Tim Wilson]: The raging cynic in me would be, well, gee, if people realized that’s a way to game the metric would be to run really inconsequential small tests, which at the same time, the analyst in me thinks that, yeah, that happens with analytics a lot, that you’re kind of digging in and trying to find something. You’re like, well, somebody thought there would be some relationship here and we’re just not seeing it. And that can be equally unsatisfying for the analyst. So like, how do you think about neutral being, we were trying something that did have a legitimate chance of being meaningful. And maybe this kind of bridges to another article that you wrote, which is, you know, like, how do you say neutral, but not have neutral become a crutch for, yeah, we’re essentially doing AA tests and, you know, giving ourselves two thumbs up on the learning rate. 00:11:01.44 [Mårten Schultzberg]: That’s a great question. I think we’ve been thinking a lot about what a healthy distribution should look like. A healthy distribution of different types of winds and also the proportion of neutral experiments. And I think that’s actually a super interesting topic. I think depending on what kind of strategy you have here from a product side, you can want to have different distributions. So for example, if you take the If we wait with the neutral one, because it’s maybe a trickier one, but if we think about how many experiments should you find regressions in that you dodge versus the win rate, how should that distribution look? Well, that will depend on a lot of things. But if you’re a company that has everything to win and little to lose, then maybe you can afford to have a high rate of just trying stuff. Because whenever you find a win, it’s going to be quite big because you’re still in early stages, whereas if you’re If you’re a product that is already very mature, then maybe you have other goals for those things. It’s a super interesting discussion to have, and that’s one of the discussions we’re having now with teams at Spotify and other people that are using our experimentation tooling. What should we do with this information? And what’s good and what’s bad? And I think it’s different for different parts, even of organizations within Spotify. What’s good, depending on how they’re looking at it. But for sure, we wouldn’t look at the learning rate only. So we would say we want the learning rate to be reasonable. But then we, of course, should probably aspire for having a high win rate. That’s nothing bad in itself. But at least if we have a high learning rate, we know that we’re not wasting our experimentation efforts. We know that experiments we’re running, we’re actually learning from. If we’re running a ton of experiments that are not powered and neutral, then we will never be able to say these things didn’t have an effect. We can’t separate between if these things didn’t have an effect or if we just didn’t run a good enough experiment to detect it. So on the one hand, you look at the learning rate and say like, hey, we want to utilize our experimentation bandwidth really well. So we want to have a high learning rate at all times. But then at each quarter, you can look at this metric and the distribution of these outcomes and say like, hey, you know what, we’re dodging a lot of bullets, but we’re almost never finding something good. Should we rethink our strategy or even more, if we’re finding a ton of neutral results and we see more and more neutral results in some part of the organization, maybe we’re hitting diminishing returns and we should try something different. Maybe we found some kind of local optima, maybe, or something like that. So I think it can be a quite strategic instrument if you have all of these, the distribution of all of these outcomes as part of the learning metric. 00:13:52.61 [Michael Helbling]: You know what’s worse than writing SQL? Probably writing that same SQL for the third time because you forgot where you saved it. 00:14:00.54 [Tim Wilson]: or explaining to an LLM for the 10th time that your GA4 medium field is a mess because three different interns had three different naming conventions. 00:14:10.59 [Michael Helbling]: Yeah, like organic, organic underscore social or, I mean, it’s like a crime scene of good intentions. 00:14:18.34 [Tim Wilson]: Which is why Askwise Skills feature really helps. 00:14:22.20 [Michael Helbling]: Record that data cleaning nightmare once as a skill, reuse it across different datasets, portable expertise, and their jam memory system remembers context, like the July data is doubled or use the product table, not staging. Exactly. 00:14:39.18 [Tim Wilson]: It’s context focused, not just code focused. Plus your data never touches the LLM. Semantic layer generates code that runs locally. 00:14:48.33 [Michael Helbling]: where your data presumably won’t judge you for that medium field situation. We can hope. We’re going to ask-y.ai. That’s ask-y.ai. Use code APH to jump the wait list and stop paying the context switching tax. 00:15:07.37 [Val Kroll]: That’s making me think as you were talking about that, that like even within an organization, like you were saying like companies who have everything to gain or you know, I think everything to Nothing to lose. I forget exactly. I never get that right. Well, apparently I can’t either. But even within Spotify, thinking about the different product teams that if it’s a group that’s working on the cancellation flow and thinking about retention, they’re probably having very different distribution of those outcomes as their goals or targets versus playlist creation which is like such an established user pattern is that like how you customize some of those conversations from like the center of excellence experience like perspective to kind of consult with those teams. 00:15:56.79 [Mårten Schultzberg]: Yeah, let’s say so, but I also add that there’s a lot of centers of excellence when it comes to experimentation at Spotify. Fortunately, we have many parts of the organization that have super strong experimentation organizations or champion groups or nerds. I like to think about it. I mean, look who’s talking. But anyway, no, but so I think, so that discussion happens locally in a lot of places and a lot of people are having those discussions. So it’s not like sometimes we get, you know, questions about how to think about things. And also, one interesting aspect of this metric is that sometimes you might find that You know, if you’re actually, we didn’t talk about, there’s one outcome here that we didn’t talk about, which was the, when you get an invalid experiment where something is wrong with the setup of the experiment. That’s the final sort of outcome in this learning framework. So you didn’t learn because something went wrong. For example, something went wrong with integration. Maybe you got imbalanced treatment and control group assignment for some reason, or you don’t get all of the data that you should get or something like that. And that’s of course an outcome that is the least fun one, so to speak. It’s just like, yeah, we couldn’t get this integration to work well enough. So we have used that one and worked really hard on getting that to as close to zero as possible. We want it to be possible for anyone to run a really high quality experiment. With Spotify running experiments on so many different devices and apps and combinations of those, it’s really tricky to always nail those things, but it’s obviously an important signal. So whenever that one is high, that’s something that teams come to us with and say like, hey, we don’t get our integration to work as well as we want to, how can we improve these things? And also when it comes to the neutral aspect, the quality of the sample size calculator starts mattering a lot. So whenever someone sets up an experiment and we try to predict what sample size they need, it’s a prediction, right? We’re looking at historical data saying like, yeah, well, given how historical data has moved, the variation in that data and the means and the treatment effects that you say you’re interested in finding, we think that you need to run your experiment for this long to reach this many users. And that’s a prediction that takes a lot of things into account, but it can always be improved probably. So that’s also a conversation that we sometimes have when people are like, in our use case, the sample size calculator is not good enough. 00:18:33.77 [Tim Wilson]: But that is a case where you, that’s one where you would come back. Like what is the scenario where you run it, they’ve got a MDE, they’ve got the estimates, you’ve got the sample size calculator, it says run this. If it comes back, I’m trying to understand the distinction between, actually, we probably just didn’t run this long enough versus, well, for what we ran and what parameters we put in, it’s a neutral result. Is there a distinction there? 00:19:13.12 [Mårten Schultzberg]: I can speak a little bit to it. In practice, when we do the sample size calculation, I don’t know how technical and nerdy I’m allowed to get here, but given the name of this podcast, I’m going to go deep. 00:19:26.24 [Tim Wilson]: We don’t want to hit if Matt Gershauf would have to think about it for a minute, that’s a little bit too technical. 00:19:34.37 [Mårten Schultzberg]: No chance, no chance. This is bread and butter for him, promise. No, so we never know the variance of the treatment group, right, before we run the experiment. We can always just think like maybe it will be a homogeneous treatment effect, or we could, I suppose, speculate about how the treatment will affect the variance, but it’s always gnarly, it’s difficult to do. So what we do always in practice, I think everyone essentially is saying like, let’s presume that the treatment effect is homogeneous. In practice, of course, when we start running the experiment, maybe the treatment effects only part of the treatment group, which will then disperse the distribution. If we have a beautiful distribution to start with, but some people get the large treatment effect, you will make the variation of that distribution larger. So the variance in that group will be larger. So the required sample size will go up. We do, in confidence in our experimentation tool, we do both. So we have the pre-experimentation sample size calculator, which uses historical data to make this prediction. And then during the experiment, we’re also collecting the data from the experiment and running the subsize calculation continuously. I actually wrote a paper about that. I think there is a blog post about that too. If someone wants to nerd in on that, that it’s actually valid to look at the power during the experiment. It’s a peaking that is non-problematic. You can look at that. Anyway, so you have those and you might have a big discrepancy then. So when you start the experiment, you might think that, hey, I can run this for two weeks. I will reach my whatever 10,000 users that I need. But then when you run it for a week, you realize that like, no chance. I will reach much less or I will need much more, maybe more likely. I thought I needed 10,000, maybe I need 40,000. And that’s just not possible given the traffic that I have on this page. And in that case, it might be done a conversation about like, hey, how can we make this better? And so one way that we do it in practice is that we say like, okay, maybe instead of us trying to predict it, you can point to a similar experiment. If you know you have a similar experiment, we’re changing the same kind of thing. But yeah, it’s a tricky thing. It’s a truly difficult problem to make good sample size estimations. 00:21:43.79 [Val Kroll]: And one thing that I found interesting, because there’s definitely like two different camps here, is that I hopefully I’m not putting, correct me if I’ve interpreted this incorrectly, that you do allow for multiple success metrics in this, which I know makes that a little bit more complicated. And I think it also talked about adequately powered guardrail metrics, deterioration metrics, quality metrics, which not a lot of organizations do or have the capability to do, but that was like, oh, well, definitely enough to talk a little bit about that. But how do you handle the multiple success metrics, especially if you’re looking at things further into the funnel that have a lower incidence? How do you think about that layer? 00:22:27.60 [Mårten Schultzberg]: Yeah. This is a rich topic. We have a framework for this that we have developed over the years. And it’s also a paper that is, I think, about to be published. It’s an archive, at least, where we go through exactly all of the details of how we’re handling the multi-metric that we call decision framework, statistically. But I can give the short version of it. So essentially, what we’re saying is that we have an explicit decision rule for the multi-metric setup. So we have success metrics and guardrail metrics. So success metrics are metrics that you want to improve, and guardrail metrics are metrics that you don’t want to harm. And so, for example, at Spotify, maybe we want to improve the music consumption, but we don’t want to harm the podcast consumption. We don’t want to do it at the expense of podcast, for example. So if you’re making a new music recommendation algorithm, you don’t want to harm any other consumption. And so the decision rule is essentially that at least one of the success metrics should have improved and none of the guardrail metrics should have been harmed. There are a lot of nuance here, because for the garter and metrics we’re using so-called non-inferiority tests, which makes everything much more complicated to talk about, but leaving that aside, it means that when we’re talking about power and false positive rate, we’re talking about the false positive rate and the power for that decision rule. So we’re saying we want that decision that we would make based on this rule. So at least one of the success metrics are significantly better, and none of the garter and metrics are worse. We want that to be the false positive rate of intent, and we want to have the power to detect given the sample size. So we have to make the adjustments for multiple testing corrections accordingly, and then we have to make the power and sample size calculations accordingly. things to fiddle with there. But in principle, since the guardrail metrics all have to be not harmed, they are not giving you additional chances of succeeding, so you don’t have to correct for them in the same sense. But at the same time, you have to power them simultaneously, because all of them has to show simultaneously that they weren’t harmed if you’re using non-inferiority tests. I’m deliberately avoiding going in too much to know if you were to test because it’s like such a tongue twister to talk about. But if you’re interested in… You still said it eight times. 00:24:51.78 [Tim Wilson]: Good. 00:24:52.60 [Mårten Schultzberg]: Yeah, no, but that was… No, but it’s tricky. So, yeah, so that’s how we do those things. So it’s a bit messy, but… 00:25:02.95 [Val Kroll]: So back to the culture side of this, how do you coach product teams to not just pick 50 success metrics? Because they are so excited about this new feature. It came from up high, and we really want this to, we want to find some success. And obviously, there’s a statistical part of it, like the correction, but culturally, how do you guide that conversation away from? No, it shouldn’t be like a pick list of up to 75 metrics to find something that went quote unquote up. 00:25:36.58 [Mårten Schultzberg]: Yeah, yeah, yeah. No, I mean, this is a conversation that we have. I think it’s Spotify. It has settled, but like this is a conversation that we have from time to time. And I think it’s a It’s a sort of healthy discussion to have because it’s not… I think this is more tricky than it might seem. I want to give the answer that, no, but of course you should just have a discussion and decide on the metrics. I’ll come back to that because that’s ultimately what we do a lot at Spotify, but there is more to it. There is also the fact that we’re making a lot of changes and we are truly interested in any kind of effect that it has. It’s a true statement that if actually this change that I made affected a metric that I didn’t think about, like some weird metric, weird from my perspective metric, if that was truly the case, I would want to know. So from one perspective, I can really understand this. I want to look at all of the metrics and just see which one that I affected. But then on the other hand, you get this obviously super hard problem of like, cursor dimensionality type issues here where you’re looking from too much, so you’re either just going to find noise, or you’re going to find noise, and then you have to control that, and then you’re going to have, instead, very low power to find things. But I think there is merit to the type of experiments where you’re just like, I just want to see what happens when I do this. And I don’t really care. Of course, I care what it is that happens, but I am ultimately interested in all things. But in practice, of course, this is hard. So again, at Spotify, it’s not like the central experimentation team, which I’m part of. building the tooling, we are not dictating these things. It’s rather the other way around that we are, I like to think about it as that we are sort of cultivating what the teams that are doing experimentation are thinking about this. So we have a lot of discussions with them. So the way it works as Spotify is that we don’t decide the defaults and how things should work in the platform. It’s rather that we talk to all of the product teams that are experimenting the 300 teams in various forms and then We collect what they’re saying, and we’re refining it, and then we’re putting that into the tool. So when it comes to this, how many metrics you should have, there’s not one answer at Spotify. It’s different in different parts of the organization. But in most of these parts, there have been very explicit conversations where people have talked about, like, hey, how should we trade off here? actually getting super high precision in the things that we know we’re interested in versus getting interesting insights and stuff that we could be interested in. And this is sort of traded off in various parts of the organization and in various projects, depending on how and what stage those projects are. If it’s like a very new product, then you probably see, or you often see experiments with much more metrics because you’re just interested in understanding what happens when we ship something like this, what kind of behavioral changes does this cause? Whereas when we’re optimizing something, then we’re like, okay, we know pretty well what we need to measure here to do this and to optimize this in a healthy way. 00:28:38.19 [Tim Wilson]: to Spotify, massive user base, a lot of the ability to design, to try to cover and still be sufficiently powered seems doable. I’m thinking of a client we had that was in that same boat. It still feels like the risk, the slippery slope, fishing expedition of let me tell myself a story that I just want to see if it impacted anything. And the understanding required that if you go on a fishing expedition, you are, I think, if I understand correctly, your false positive rate can go way up because you detect noise as a signal, which then when you detect it, you get really excited. Nobody can rationalize why this metric changed. It turns out it was noise. Now we’ve wound up doing negative. We’ve learned something incorrect potentially, unless you have the discipline to say, if we’re going to chase that, we need to come up with a theory, and we need to have the rigor to validate that theory before we accept it as fact. That just feels coming from an analytic side, similar sort of thing. If I just point the machine at all the data and it finds anomalies or finds patterns, there’s a very good chance that it’s detecting noise that just happened to hit at a point where it can show some statistical merit. Somehow, some part of me is just terrified. While I love getting comfortable with We looked for X, we did not find X. That is still a learning and let’s work with our business partners to acknowledge that’s a learning and not have them just chasing for everything. That also feels like a challenge, you know? 00:30:40.48 [Mårten Schultzberg]: Yeah, no, no, I mean, I agree with everything you say, but I also feel like, I mean, I have the same uncomfortable feeling in my body when I think about this, like, let’s look at all of the metrics from a statistics perspective. But I also just like, I really want to, I also think it’s a cop out, not projecting on you now, Tim, but for myself to say like, you know, to say like, you know, We can only look at the metrics that we decided before because we decided that we found nothing. Let’s move on because it’s also obviously true to me somehow, even though I can’t come up with this is how you should do it and this is how it won’t lead to these incorrect learnings that you mentioned. But it feels like it’s a hard argument to make when someone says, yeah, but I looked at some other metrics and I learned something. And then you’re like, maybe you did, maybe you didn’t. And I can think about ways that you could do this. You could do sample splitting and stuff. You could take one part of the sample and look for groups. And then you could validate those findings in another part of the sample and stuff like that to make it much more plausible. Again, you would have the issue then of having lower powers actually find things, or lower precision at least. I just don’t want to be too much of a… Curious? Yeah, or like a grumpy statistician kind of person. But I do, I mean, I agree. I have the same feeling and I haven’t seen anyone do it well. So what I’ve seen is that people have used the argument of saying like, yeah, we must be able to be able, you know, it must be possible to learn more and then just throw all of the metrics at it. And then I think they’re just as well. Like that’s just as bad as not doing it, I think. So I don’t have an answer to it, but Maybe someone smart listens and then they can call me. 00:32:22.39 [Val Kroll]: Yeah. Let us know in the comments. 00:32:24.08 [Tim Wilson]: I mean, I, I mean, I, and I don’t know that this is the answer, but I’m, it does feel like, well, if you, if you throw that at it and you find something figuring out how to have the, the step, which is probably a combination of a data scientist or a statistician with the product manager to say, we need to come up with a, plausible theory as to what’s causing that surprising thing. And we need to have somebody with their bullshit meter turned on. Cause I mean, I’ve certainly watched people find things and they come up with a bullshit theory. They’re like, well, this is clearly happening. Cause obviously like left-handed people, when they’re in the Southern hemisphere, it makes sense that they would prefer the color blue, you know, and something that’s, It’s a theory that fits the data, but it’s not a theory that holds up to human scrutiny. 00:33:25.16 [Mårten Schultzberg]: I think one thing that I’m excited about is replication. I think if you have a streamlined enough way to run experiments and you have your velocity throughput for experimentation high, then one true possibility here is to replicate, to just say like, okay, I looked broad and deep here and I found something. I believe in it. I think I’ve made my people in the sudden hemisphere argument, but I believe it. And then for anyone who would say, I believe in it to the extent that I will now launch a new experiment, take 10% other people or a new random sample and run it again with only that metric or only the new metrics that I care about. And if I can repeat it, then I will ship it. Then I would be like, yeah, go for it. 00:34:16.19 [Tim Wilson]: Or potentially, if the theory is, well, it was this kind of incidental thing that happened to be part of it, but it wasn’t the core focus when we run an experiment where I’ve doubled down on that to say this should now I should now really detect a strong signal because it’s backing that up. 00:34:39.73 [Mårten Schultzberg]: That sort of touches a little bit on the other blog post that you mentioned that has to do with what the intent with an experiment is. I haven’t really talked about it yet. 00:34:51.12 [Tim Wilson]: Let’s talk about that one. Boy, I got giddy on that one too. 00:34:56.65 [Mårten Schultzberg]: Should you want me to give the TLDR on that one too? 00:34:59.45 [Tim Wilson]: Yes, please do. 00:35:01.50 [Mårten Schultzberg]: Yeah, so the idea with that one is I often have like it has sort of come from a lot of the conversations that we’ve had with people running experiments, talking about the learning framework. And then people are like, hey, we have a lot of neutral experiments here. We run high quality experiments, but we don’t find things. And so one thing that I’ve sort of identified from working with teams that Spotify, but also externally other companies is that People are often sort of starting to optimize the idea in their head before they’ve tried that the idea is at all something that will affect the user users. And so what I mean by that is that people are, you know, when they identify something that they think is like, this is important for our users, like, let’s, let’s use a stupid example, like a button color or something, you know, like we think it’s important. And then immediately, instead of saying, OK, we should first answer the question, is it important or not? Do users care or not? Instead, they immediately start thinking about which color is the best. And so they jump from, we have no idea if people care about this to having the conversation about which color is the best. So sort of presuming that people care at all which color this has, besides having a high enough contrast so you can see it. And so this blog post was me just trying to formulate that, like the distinction between identifying if an aspect of your user experience is something that you can optimize if it has sort of an effect on users in any way, people care about it on the one hand, and optimizing that once you have identified that it’s something that people care about on the other hand. So sort of identifying something versus optimizing something. And so I think that this thing that we talked about now is a little bit about maybe if you run an experiment where you thought something, you thought that it was important with some aspect, or you tried to optimize it, and then you find something something new, some metric that you didn’t anticipate to move. That might cost the sort of idea in your head to be like, hey, maybe there is a mechanism here that people care about. Maybe people actually care about how many items we show on this screen. I was thinking about the ranking, but as a side effect of that, we showed more things. So we saw that, I don’t know, lower down that the list clicks increased or something like that. And maybe that’s an indication that this is a mechanism that people care about. I think this going in between the states of identifying something to optimize and optimize the thing you have identified and doing that explicitly and deliberately is something that a lot of product teams would benefit from. It’s easy to fall in the trap of trying to do both at once, I think. 00:37:48.86 [Tim Wilson]: Totally. Is a cousin to the optimizing I mean, the framing of say, which is kind of a, I think it might even be in the article, like the case for taking a bigger swing, take the big swing first, make sure that connects, even if it’s a, you know, in a while, it’s like, yes, there’s something here, now we can tune it. And that, I think of it from a, I mean, from a marketing analytics perspective, where companies will say, Let’s just try it out and see what happens. It’s kind of a death knell. It’s going to be an underinvestment in a new channel or a new tactic where logically, it’s going to be really hard to detect a signal because it winds up getting kind of tempered down to a pretty subtle change. The logic is, well, if this thing actually matters, then we can make a nominal investment and we’ll see this outsized lift as opposed to saying, does this matter at all? Double down on it for some period of time. Go hard. See if you actually see something and then say, okay, we definitely need to be in this channel or using this tactic or doing this to the user experience. Now we need to sort of figure out Did we actually spend twice as much as we needed to, we can get the same? Where are the diminishing returns? It does feel like culturally it’s a tough, human nature is risk averse. Saying, try something and know that you’ll find that it is okay to find that it didn’t work. A big swing with a neutral result feels like it has a lot more merit than a little small tap with a neutral result. That’s the fun in that. 00:39:39.89 [Mårten Schultzberg]: That’s precisely it. This actually what provoked me to write it was discussions about the neutral outcome in the learnings framework where people are like, people are like, yeah, but neutral is no fun. I don’t care if it was powered or not. I don’t want neutral. And that got me thinking, well, if you don’t like the neutral result, it means that the question you posed wasn’t interesting enough. Because I would be like, if I’m convinced as a product person that people care about this thing in our app, if I change this, people are going to care. And then I make a drastic change and nobody cares. I’ve run the experiment, I have high precision in my estimates and nobody cares. If that’s not the learning to be excited about, I don’t know what is, to be honest. That really shows that I’m 100% off with my understanding of what people care about, which is truly strong learning. But on the other hand, this change that I made was like, yeah, I really think our users care about this aspect and I made a minuscule change to it and I didn’t find anything. I might think for a long time about if this was the right change that I made, or if it was… You just get stuck in weird things. But one way that I have sort of sold that, because I agree that people are risk-averses to run both. If you run a neighbor test, people tend to want to be like… But I think I know what users like. I want to go for the identify and optimize at the same time version of this thing, where… I try to choose the right value for my customers or my users. But I also say, just also add then, if you haven’t actually identified that this is something that people care about or that matters for your business or where it might be. Add the more sort of provocative version. I call it maximum viable product, I think, because of course, this has to be reasonable. If you make some button larger than the screen, then of course, you’re going to see some change. So it has to be within the limits from what is still a usable function, but that is still extreme. So the maximum change that you think is like, but this is still, this is not 00:41:48.80 [Tim Wilson]: You’re saying doing that within kind of a multivariate, say we’ve got our control, we’ve got what the optimized and identified at the same time version, and then we have an identify only version. And it’s okay if that identify version detects like the biggest effect, you can say, yeah, that was kind of hedging to make sure that that we got something out of it. And if that one that was identified and optimized simultaneously didn’t, then we’re probably still on a good track. It just turns out we’re not so omniscient that we can come up with the perfect variant in one shot. 00:42:30.52 [Mårten Schultzberg]: I think it’s smart also from, I mean, a lot of companies at least, Spotify and other companies that I work with, they’re all struggling with having big enough sample size, right? both because they have limited traffic, but also because they’re interested in small effects, generally speaking. But the nice thing about making a very drastic change is that it should have a large effect. If you’re making this maximum viable change, then that should cause a large effect. So you should be able to say, yeah, but now I pull this lever as hard as it’s possible to pull. So this should cause maybe 5% change, like whether it’s good or bad. And so you can maybe run smaller experiments. If you’re in a situation where it’s hard for you to know what you should, like you have a hard time finding bandwidth essentially for optimizing things, then I think it’s a smart idea to do these more drastic changes to identify what you should then spend larger experiments on optimizing. Because the truth is that when you start optimizing, even if it’s a nice convex surface for this thing, button size or something, the closer you come to the optimum there, the larger samples you’re going to need to be able to identify those steps. 00:43:44.98 [Val Kroll]: It seems like the framing that I really liked in this article is the building the right thing versus building the thing right. And it feels like the stakes couldn’t be higher in everything you guys are just talking about in a product context because it’s not just about changing a button color. In a lot of cases, this isn’t about UX. It’s about adding additional features or different capabilities. and you’re hoping to impact things like customer lifetime value, not just did they get to the next screen, right? So it’s not just like checkout flows, right? I think I was thinking about this. I’ve actually spent more time than the average human should thinking about the changes that have been happening lately inside of my United app. So I’m United Loyal, I fly United and the app has been changing a ton lately. And we went from, there was one place where I could change my seat to every single screen within this app. I can change my, which I do appreciate. I’m definitely someone who loves feeling a lot of control over changing my seat. But I’m like, what were the conversations that happened internally that said, you know what? The user needs to be able to change their seat while they’re checking in their bag, while they’re checking to see what gate their flight is at. Anyway, just to bring this back to an actual question, building the thing right, and maybe the feature is great, the new functionality that you’re adding, but maybe you have gone about it the wrong way, which has impacted the ability for someone to understand What exactly this is capable of? Maybe it was a micro copy issue, or maybe it was in the wrong place in the flow, which feels more like optimization. Even though this framing and big swing versus small change, that sounds really objective. If you put them side by side, that’s clear. I’m especially interested because now you are in a product role to get a little meta about it. How do you think about what is, when would you ever recycle a concept in a different context? Because it does feel like the optimization killed your ability to understand if it was viable. 00:45:59.02 [Mårten Schultzberg]: The truth is here that this is difficult. I think especially starting with that building the thing right versus building the right thing. Some things you have to do quite a lot of building to even check if it’s the right thing. If you’re building a new feature, there might be a lot of things that you have to get in place to even see if it’s something that people cares about. once you’ve seen that they care about it, like maybe they don’t like it. And that’s because you haven’t built it right yet. So like, I mean, it’s this is a very stylized blog post, of course. But the truth is, is much more muddy. So yeah, I mean, in practice, I think that one of the things that have been discussed a lot that Spotify and other places is like, okay, but with experimentation, where is the room for the product intuition and making bets on things and stuff like that? And I’ve always liked to say that these are completely uh you know they’re they’re augmenting each other they’re helping each other like it’s no they you can make you can have this strong intuition still and you can make these bets what experimentation helps you with this actually validating that your bet was good and helping you change your direction if it wasn’t good and so What I’m trying to say is that, of course, sometimes, and maybe not even rarely, when we’re building experimentation tooling, we have to build for quite some time before we can answer either of these questions. And it’s hard to disentangle them even. So I’d say that we build a completely new feature for experimentation, then some new methodology or something. It’s hard to even have What’s the dimension along here I can test if this is a lever worth pulling? That’s maybe a question more for market research or user research, all those kinds of things. Yeah, so that’s the truth. I think it’s just a lot of, I think the teams that I’m writing this blog post for that I’m thinking about are the teams that sort of have a product already and they’ve been owning it for a while and they feel a bit stuck in terms of like they’re not getting the sort of return of investment rate that they would like from their expectation. They see that they have a lot of neutral results and they’re wondering if they should run much longer experiments or what they should do about it. But yeah, I don’t know. Felt like partly cop out from your question there. 00:48:30.79 [Val Kroll]: No, no, it’s good. It’s, I mean, there’s no clear question. 00:48:34.94 [Tim Wilson]: Come on, Ben. I mean, it’s, he basically said that it’s like, it’s like intuition with experimentation combines. It’s kind of like you need to combine like the facts and the feelings. 00:48:45.65 [Val Kroll]: I knew exactly where you were going with that when he said. Come together. 00:48:50.46 [Tim Wilson]: Cheesy. So. 00:48:51.58 [Val Kroll]: Okay, so. Before I lose the thread because I last question, by the way, because we’re, we’re don’t do that to me. No, no, no, no. I’ve got like three more, but I’ll go fast. I’ll go, we’ll go fire around. Okay. So you’re talking about, um, uh, no one really likes the neutral results talking about some intuition with product. I’m going to talk about those outcomes. So obviously if there’s a win positive outcome, it ships. If it hurt the experience, it doesn’t ship. If there was an issue with the test set up, you hit an SRM or whatever, it doesn’t ship neutral. I want to talk about that. Are there scenarios where the product intuition says, even though this was neutral, it makes sense for where the roadmap is going or some decisions we’re making from branding, like maybe we’re This is building towards a bigger bet in the larger ecosystem to make things easier to share, more social. How do you think about the ship or no ship kind of action as it relates to those neutral results? 00:49:51.28 [Mårten Schultzberg]: It’s a great question. My general recommendation there is that as long as you’ve decided before you run the experiment that you’re going to ship if it’s neutral, I’m all good with it. I think that there’s a ton of situations where it makes sense to ship something if it didn’t change anything, especially if you’re building infrastructural type changes or if you’re building towards something. We’re building a lot of Spotify, building out AI features, as everyone else I suppose, but there’s a lot of changes that we’re making to our infrastructure just to be able to support features that we’re planning to build. And when we’re making those changes, the idea is that we’re hoping that nothing will change. Maybe we’re doing stuff to make things faster or something like that, but that’s a bonus if it changes anything at any point. So there’s a lot of changes that we are expecting won’t make any difference. So what we do then is that we essentially run what we call rollouts where we only have guardrail metrics, actually. So we say, as long as As long as we can prove that we didn’t harm these metrics, we’re going to ship it. So then by using the rollout, you’re sort of declaring your attempt from the beginning that like, hey, we’re planning to ship this as long as it’s not bad, which can sort of be a quite nice way to just make it explicit. That’s completely fine. But then again, I think that I just want to add a small caveat here that they also, I’ve heard a lot of product people at Spotify and other places talk about that like, And this, even maybe if a metric doesn’t look great or if it’s neutral and stuff like that, there is this, I think, almost human fallacy to say, like, this is strategically imported, let’s ship it anyway. And so I think it’s, even though that’s true, and I think that’s why it’s sort of an easy fallacy to fall into, or like it’s an easy trap. That can be true, but I think everyone should think about how large proportions of the things we ship should be shipped from the argument this is strategically important. Pretty small proportion is my general sense. 00:51:58.64 [Val Kroll]: Everyone gets three here. Something like that. 00:52:00.88 [Mårten Schultzberg]: I would love if I could give people a budget for those kinds of things. I think it’s all about trying to avoid the pitfalls of changing the objective when you see the results. We do that all the time at Spotify. We’re shipping a ton of things that are neutral. A lot of them are shipped with rollouts where we just explicitly say, we are planning to ship this thing for some reason. It might be business statistically or we have to improve our back end to scale for more traffic or whatever it might be. We’re going to ship it. So we just want to know that we’re not harming things. 00:52:40.72 [Val Kroll]: I like that. Okay, so Tim, I’m sorry. I have to sneak in. So what you’re talking about here is a very nuanced It feels like a nuanced analytical discussion. Should this be a rollout or how should this be exactly validated? How do you think about the education? Because you’re not talking about an audience of 400 people who are deeply steeped in the analytics or the rationale for why you’d make some of those choices. How do you think about the education piece to these different product teams? 00:53:14.68 [Mårten Schultzberg]: Yeah, I mean, it’s super important. So I’ve spent, I wouldn’t say majority, but a very big portion of my time at Spotify building educational material and mechanisms for this. I think that we have, I think, two strategies for this. I think the first one is to keep the the tool as simple as we possibly can, so have as few options as possible. So we’re talking about a lot of nuanced stuff here, but we also have removed a lot of stuff from our platform and simplified a lot of stuff and removed a lot of options, so made it quite opinionated. to minimize the things that people actually have to understand and know. So that’s one side. On the other side is that we have very explicitly and deliberately built educational material and tooling for experimentation for many years. So with confidence, we have this whole boot camp of self-serve courses. We’ve also given a bunch of courses. We have something called Quick Starts, which is a very basic tutorial for like, this is how you run an experiment, this is how you run a rollout. and those kinds of things. I know it’s a super important thing, but I think it has to come from two sides here. You have to try to make the thing that people should learn as simple as possible because people don’t have time. People have a lot of other things that they need to be good at and learn and understand, and then you have to create the material so that they can learn those things that they have to learn. That’s our solution to that. I mean, we have thought a lot about that. There’s a lot of things that everyone that joins Spotify is onboarded to experimentation immediately, and they go through certain what’s called golden paths at Spotify, which is like onboarding to certain things. And so if you’re a mobile developer, then you learn how to work with our feature flags in mobile, and you run an AA test as part of your mobile engineer. onboarding, for example. So there’s like, we have infiltrated the whole organization with experimentation onboarding and materials. And that has helped. 00:55:22.91 [Tim Wilson]: Wow. Wow. And Val, I’m going to have to put some duct tape. I was like… And we’re going to have to move to wrap. But I just have seven more. Val and the role of Moee Kiss on this episode. 00:55:35.21 [Val Kroll]: Yeah, right? 00:55:36.49 [Mårten Schultzberg]: No. I have zero stress at least, so don’t worry about me. 00:55:42.56 [Tim Wilson]: Well, this great discussion, I love sort of the thinking about what are we doing, why are we doing it, and how can tooling and education and culture and framing all sort of come together. So thanks for coming on for this discussion. But before we leave, the last thing we do on the show is go around the horn and we share a last call, something that might be of interest to our users. And Mårten, you’re our guest. Do you have a last call you’d like to share? 00:56:20.13 [Mårten Schultzberg]: Yes. So one thing that I’m completely, like I have been for actually for many years, but now renewed is the YouTube channel Three Blue One Brown. If I’m not the first one, that just makes me happy because it’s the best. The thing that I’m particularly thinking about now is the videos on Transformers and LLMs. This YouTube channel is essentially a channel that visualizes a bunch of math. That sounds maybe not fun, but it is so insanely good. They have a long series on linear algebra that I think if I would have actually seen it when I was taking linear algebra, it would have helped me a lot. But they also have a bunch of super, super nice things on LLMs and Transformers, which I think is like If you are, like most people, like hearing that word many times and you have like, yeah, it’s some kind of neural net. Maybe I haven’t used a neural net once or twice, but like you have no idea really how it works. Those videos are so very, very good. So I recommend them highly. 00:57:33.50 [Tim Wilson]: We have reached out to have, we had an exchange trying to get him to come on the show. I think it might have been around to talk about neural networks. He was in the process of like moving. So he’s on our list to try to get him on. That’s a- Good reminder. That’s a good one, good reminder to go back because they are, they’re like, I’ve sampled some of those and I’m like, this is so clear. And how does a human being have the time to produce something like this? 00:57:59.15 [Mårten Schultzberg]: Yeah, I mean, Grant Sanderson who has that channel. I mean, he seems to be like one of the true geniuses alive. Like, I mean, just a side note here is like he’s doing this super nice like animations of math and you just built that library himself, the library he built. It’s just… 00:58:19.19 [Tim Wilson]: Come on. We’re going to use this call out when this comes out to reach out to him again and say, hey, come chat with us. 00:58:27.48 [Mårten Schultzberg]: I would listen 100%. 00:58:29.52 [Tim Wilson]: Awesome. Val, what about you? Do you have a last call? 00:58:33.83 [Val Kroll]: I do. And it’s actually related to today’s episode. So this is a medium article published on Unix Collective, article written by James Skinner. It’s called Escaping the AI Sludge Why MVPs Should Be Delightful. And there’s a lot in here, one of the cases he makes is that like using AI is just like regurgitating like we’re not going to get to that delight level if we’re just, you know, using AI to help, you know, develop those different. net new versions that are being tested within a product context. But he talks about the MLPs. I’m obsessed with MVP’s, Mårten, I should tell you, just understanding different people’s perspective. But the MLP is the minimum levelable product. And he also referenced one, the minimum viable whatever, because there’s so many acronyms related to this, with people trying to figure out exactly what that level of fidelity should be, what type of investment you should make before you experiment. He does talk about experimentation at the end, which I do love, but there’s a lot of really good examples. And I love reading from that design product perspective. So, but it’s a, it’s a good read, about 10 minute read. So it’s a good one. And Tim, how about you? Do you have a last call for today? 00:59:47.36 [Tim Wilson]: I’ve got a smidge of housekeeping and a last call. So we are now like into month number two of 2026, which means we’re heading into a conference season. Actually, I am, sitting in Budapest, Hungary as you were listening to this, if you’re listening to it when it came out. A couple of analytics power hour conference attendee appearances coming up in Nashville. If you’re in the States, there’s the Datatune conference that Val and I will both be attending on March 6th and 7th. Some critical mass of the Analytics Power Hour crew, we will be recording a show with a live audience at the Marketing Analytics Summit in Santa Barbara, California on April 28th and 29th. Those are PSAs more than last calls. My last call would be friend of the show, past guest, Katie Bauer, the wrong but useful sub-stack wrote a post called The Next Data Bottle Neck, which I thought it was a unique and really thought-provoking take on the whole drive towards conversational analytics and not the will it or won’t it or the technical challenges of it, but when looking at what people are asking for and why they actually seem to be mundane requests that they seem to be kind of just simple data fetching requests, not these super nuanced things. So she has a lot of musings that can be a little unsettling for the analyst, but then she actually kind of wraps by making the case that really it goes back to good analysts really thinking about the business deeply. So it’s a worthwhile read. So I was a threefer, but I’ve labeled two of them as being a housekeeping writer in the last class. 01:01:45.78 [Val Kroll]: Can I ask one more question then? 01:01:48.19 [Mårten Schultzberg]: That’s how you get airtime in this show, right? 01:01:51.33 [Tim Wilson]: I’m drunk on power. Is Michael as drunk on Tamiflu? Tamiflu? Tamiflu? Tamiflu? I don’t know what the flu medications are. Yeah. By the time this comes out, he will be back to good health and he will vow to never get sick again and cede the mic to me. So this was great. Thanks again, Mårten, for coming on. This was a really fun discussion. 01:02:17.28 [Mårten Schultzberg]: My pleasure. It’s really nice. Thank you so much for having me. 01:02:21.69 [Tim Wilson]: Awesome. Everybody get your Spotify subscription up to speed. This is what’s driving Spotify’s next round of growth is the confidence podcast appearance. 01:02:33.83 [Mårten Schultzberg]: Quarterly call coming, so like, please. 01:02:35.93 [Val Kroll]: There you go. 01:02:38.11 [Tim Wilson]: Perfect. If you are listening and you’ve enjoyed this show or other shows, we would always love a rating and review. I’ll do a little call on audible and read out this one from Apple Podcast that just T5272018 left. It was titled Smart and Funny. And it was love the insights and laughs I get from this podcast. You all have a high bar for analysts and the value they can add, which I so appreciate. And you share all of that perspective via hilarious and authentic banter. Keep it up. Wait, let me check. That is our podcast. Yeah, that is this one. So that was kind of nice. We’ll always love to get ratings and reviews. Theoretically, that is how we expand the reach of the show, that and recording video and putting them on YouTube. So we’ll just double down on the ratings and reviews. If you’re a fan of the show and would like to have a sticker for your laptop or water bottle or whatever, you can go to analyticshour.io and request a sticker. We’ll ship one over. If you have something to say, a thought for a topic, criticism, your own little witticism that you’d like to share, you can reach out to any of us or the show as a whole on LinkedIn. You can catch us on the measure slack or you can just send an email to contact at analyticshour.io. So, with that, for Val and for Michael in absentia from his sickbed, I’m Tim Wilson and no matter what your reason, whether you’re identifying or you’re optimizing or you’re being just aggressively neutral in your findings, you should always keep analyzing. 01:04:24.79 [Announcer]: Thanks for listening. Let’s keep the conversation going with your comments, suggestions, and questions on Twitter at @analyticshour on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst. Those smart guys wanted to fit in, so they made up a term called analytics. Analytics don’t work. 01:04:49.43 [Charles Barkley]: Do the analytics say go for it, no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 01:05:03.47 [Tim Wilson]: Yeah, we’ve sent, Australia is the one that’s the real Australia. 01:05:08.27 [Val Kroll]: Singapore. 01:05:08.99 [Tim Wilson]: We’ll take weeks. Singapore, one made it all the way to Singapore, came back to Ohio. Never came to me, turned around and went back to Singapore. So it was like eight weeks. 01:05:22.69 [Val Kroll]: The box was like smashed. The gift wasn’t ruined, but the box was in shambles. 01:05:29.07 [Tim Wilson]: There is now more packing material. I did change after seeing that. It’s a process update. 01:05:35.73 [Mårten Schultzberg]: I guess I should save all of my comments about it for the actual recording. 01:05:40.98 [Val Kroll]: Yeah, we’ll get into it for sure. I’m very excited. 01:05:43.92 [Mårten Schultzberg]: It wasn’t that terrible. The distortion wasn’t that terrible. 01:05:47.21 [Val Kroll]: So every time you do that while we actually record, because you’ll definitely be doing that multiple times, I’m just kidding. 01:05:52.61 [Mårten Schultzberg]: Yeah. 01:05:54.07 [Val Kroll]: Yeah, it looks like. Last for me. 01:05:55.93 [Mårten Schultzberg]: Part of your signal yelling at you. 01:05:58.98 [Val Kroll]: Your guests. 01:06:02.77 [Tim Wilson]: All right, let’s try it again. 01:06:12.37 [Val Kroll]: Rock flag and focus on those learnings. The post #290: Always Be Learning appeared first on The Analytics Power Hour: Data and Analytics Podcast.

Is this your show?

Claim this listing to keep it up to date, reach guests who want to pitch you, and manage bookings with Guestify.

Claim this listing

More Business podcasts