Find partners
The Analytics Power Hour

The Analytics Power Hour

Hosted by Michael Helbling, Tim Wilson, Moe Kiss, Val Kroll, and Julie Hoyer

BusinessManagementInterviews guests

Episodes

10

Latest episode

Oct 2026

Language

EN

About the show

Attend any conference for any topic and you will hear people saying after that the best and most informative discussions happened in the bar after the show. Read any business magazine and you will find an article saying something along the lines of “Business Analytics is the hottest job category out there, and there is a significant lack of people, process and best practice.”In this case the conference was eMetrics, the bar was….multiple, and the attendees were Michael Helbling, Tim Wilson and Jim Cain (Co-Host Emeritus). After a few pints and a few hours of discussion about the cutting edge of digital analytics, they realized they might have something to contribute back to the community. This podcast is one of those contributions. Each episode is a closed topic and an open forum – the goal is for listeners to enjoy listening to Julie, Val, Michael, Tim, and Moe share their thoughts and experiences and, hopefully, take away something to try at work the next day.

Listen to episodes

10 recent
November 10, 20261 hr 5 min

#310: A Multi-Stop Tour of AI in Data and Analytics

It can be hard to hear an AI story these days without wondering how much of it is sensationalized. So, we went looking for some reality on the ground. Moe rounded up four of her colleagues from Canva— Christian Sullivan , Nina Kumagai , Jack O’Hagan , and Flynn Alexander —and we made a series of short stops with each of them: an agent that investigates metric alerts (and cut the mean time to investigate from about two and a half days to about two hours), a prompt analytics platform prototyped during an AI Discovery Week (after a year of privacy and legal groundwork), a skill that synthesizes what a company has learned across its experiments (neutral results included!), and an LLM serving as the “resident expert” interface to a Markov chain model of user journeys. A common thread? Each of them found that the AI worked a lot better once they took some of the work away from it. And, no, AI did not get final say on the episode title, despite its insistence that a certain Antipodean phrase would land just fine. This episode is brought to you by Stape , your all-in-one solution for server-side tagging. This episode is also brought to you by Prism from Ask-Y —your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show Canva Peter Fader Markov chains Sankey charts Jev Re-Thinking Product Analytics: A Playbook for Analytics Teams in an AI-Native Future Photo by Dan Freeman on Unsplash Episode Transcript 00:00:00.00 | Announcer: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. 00:00:15.33 | Michael Helbling: Hi, everyone. Welcome. It’s the Analytics Power Hour. This is episode 310. It seems like every time you turn around, you’re hearing some amazing new AI story. I guess it’s important to engage, but it can feel like what you read or hear is, I don’t know, somewhat sensationalized or overblown. Well, we wanted to bring you some reality on the ground. And we have an amazing opportunity to go in-depth with a few people from one organization to give us a bit of a tour on what they’re working on, how AI is impacting it, and their perspective. Thanks to Moe Kiss, she’s pulled together folks from across Canva to give us some insight in how they’re using AI. So let’s dive into a series of quick discussions on how AI is impacting people. AI is being used and thought of in one org. And I’m with my co-host today, Julie Hoyer. Hi. 00:01:01.17 | Moe Kiss: Why, hello. 00:01:02.21 | Michael Helbling: And, of course, Moe Kiss. Hey. 00:01:04.65 | Moe Kiss: Hey, how you going, Michael? 00:01:06.01 | Michael Helbling: I’m doing great. I’m Michael Helbling. And we’ve got a number of guests today, and they are Christian Sullivan, Nina Kumagai, Jack O’Hagan, and Flynn Alexander. And I’m excited for us to talk to each of them and get to know them a little better. Christian, we’re starting with you today. Thank you so much for first coming on the podcast. As we get started, could you give us just a… a quick intro to the work you do at Canva and your role there? 00:01:30.51 | Christian Sullivan: Thanks, Michael. It’s a pleasure to be here. So I work in the Foundational Goals Performance team as the data lead. Now, our team kind of splits itself between an ops focus, which is all about our monthly business reviews, having teams kind of accountable to their metrics and reporting on their performance all the way up to our founders. And on the other side is more of a data platforming team where we build and operationalize a lot of tooling that enables teams to better connect with their metrics and business performance. So I’m leaning more heavily into the more tooling side, but that’s kind of our core accountability at Canva. 00:02:05.92 | Moe Kiss: Christian, one of the Canva things I was trying to let my fellow co-hosts know about earlier is that our project naming is very important. So we wanted to find out a little bit about a project you’ve called ARCA, what it does, how it works, how you’re kind of using AI tooling to help you with data science, particularly in that field. So we wanted to find out a little bit about that foundational data space. 00:02:27.37 | Christian Sullivan: Yeah, so ARCA is an acronym for Agent Driven Root Cause Analysis. Now, my team kind of looks after the alerting platform at Canva, and we try to make it as easy for teams to set up alerts on any business metric they can imagine. Now, what that means is that because Canva’s scale, we have hundreds of metrics with 10 to 15 different cuts. And when we’re setting up alerting across all of these metrics, things can get exceptionally noisy. It can be noisy and really hard to figure out which ones are the alerts that we need to deep dive into that represent real business performance degradations versus those that might just be due to a seasonal event or a known marketing launch or something that we would basically expect. So it’s less of an alert, more of a notification. So what we realized was that we’re sending out hundreds, if not thousands of alerts per day, and only a very small percentage of them were actually being triaged or investigated. Right. And at the start of this year, one of our main targets was to really solve this and to get as much value out of our alerting platform as possible. That’s when we came up with the idea to plug in an agentic workflow. And that’s where kind of ARCA was born. So the way that it works is whenever an alert is triggered, as long as a team has onboarded or added an ARCA tag to their alerting configuration, ARCA will basically spin up an agent, run an investigation, perform metric decomposition, and then publish that directly back to the alert and update the alert from a triage status to in investigating. Now, teams don’t resolve things immediately from there. They can either choose to take the comment that ARCA left and record that as the reason that the metric moved, or they can go deeper and add some additional context and then update that to resolve. So the idea is that. We always want humans in the loop, at least to verify the results. And then ARCA kind of does that initial analysis and moves us from a place where we’re cold starting investigations, where an analyst or data scientist could be receiving 10 to 15 alerts per day and having to now cut that metric by a bunch of different segments, look into other related event data, look into our data warehouse to see if there’s been a load job that’s kind of gone awry. And instead, ARCA basically starts that process for them, means that they’re not cold starting those investigations and that ultimately they can reach a resolution state a lot faster. Yeah, so that’s basically how it works. Now, a lot of it was overcoming complexity just with how our metrics can work and providing that context. So we’re still only, I’d say, 1% of the way there in terms of where we want ARCA to go. But we’ve noticed some real value. And teams who have onboarded to it. And we’re starting to really accelerate the number of teams that are using it. It’s only been around for about a month now. 00:05:25.80 | Moe Kiss: So tell us a bit about some of the technical challenges. Like, I feel like we’re hearing the dream state where it’s working as intended. And the investigations are not causing data scientists to go down different rabbit holes. Like, what were some of the hard bits that you had to kind of work through? 00:05:41.69 | Christian Sullivan: Yeah, I’d say the biggest hard bit was our V1 of ARCA, which was this megalith of different agents interacting with each other, having access to Slack, having access to Confluence, to our internal company knowledge base, and then having access to our data warehouse. And being able to tell an agent how to reason between often conflicting sources of information and come to a root cause meant that, A, our results were not replicable. So every time we called the agent, we would often get a different root cause. And B, it was incredibly expensive. Because we’re burning tokens with this multi-agent workflow, trying to embed like a human in the loop kind of conversational flow. Before we realized, let’s just break this down into the most kind of core component for what you would generally start root cause analysis with, which is segmentation and breaking down by different segments. And then that gives you like a bread trail of or breadcrumbs where you can go into and further investigate based on, you know, it might be due to a country degrading in a given metric. You know, to investigate that country and to understand why that country is dropped. And so what we really found is that the lack of determinism in these workflows meant that agents were kind of going off in all different directions and publishing really weird results that ultimately wasn’t useful. So by pulling this back and understanding where can we actually get determinism, where can we say this agent needs to run this set of SQL queries instead of this agent can generate and run its own SQL queries. We can actually do that. We kind of came to a state where we’re using the agent to reason across huge amounts of data and understand interesting trends. But we’re using determinism to look at patterns where we know that every agent will investigate in this way or any human would do the same. And so it was really first removing a lot of that complexity. So taking away that multi agent orchestration workflow and just going with a single agent. Secondly, how can we have that agent spend less time kind of interacting with the person and more time picking up this deterministic workflow, publishing a root cause and being really clear as to what it’s investigated and then what it would recommend a human to do instead of trying to kind of one shot that whole thing through this overly complex investigation. 00:08:02.19 | Julie Hoyer: Did you have to spend a lot of time giving it like historical examples of like these root cause analyses that were done by like analysts throughout Canva to help it? Or like, how did you train it maybe on some of that? 00:08:15.24 | Christian Sullivan: Yeah, we definitely use that in our evals sets. So we would look for past alerts where teams have provided a root cause and use that to see if an agent was able to get it. We haven’t gone so much into providing past examples of, you know, known reasons why a metric might degrade in a given month because we always run a marketing promotion then. But that’s something that is on our roadmap. So we would like to have the ability for a metric owner to provide additional context so that the agent can pick up on that in future. And we’d also like for the agent to be more aware of other alerts that have been resolved, triaged, and documented in our alerting platform so that that can all be done automatically, that we don’t have to go through the manual process of identifying where do we see trends and patterns from before. What if we can just pick up all alerts that were resolved over the past forever and putting that into the agent so that it can understand, oh, maybe there is a seasonal pattern here. Or maybe we do know that when a given segment moves like this, it could be due to a marketing campaign or grandfathering a product or any other reason that a metric might move. 00:09:16.90 | Michael Helbling: And are you using multiple different LLMs for this, or is it all flowing through sort of one structure? Like, how is that architected on the back end? 00:09:27.00 | Christian Sullivan: Yeah, so initially we tried to run with multiple different LLMs. So we were generally using like the mid-tier model, like a Sonnet or a Terra, for the main reasoning and investigation workflow. And then the summarization was done on a lighter model, like a Haiku. We’ve now just moved everything into kind of the mid-tier model. And we’re running with GPT 5.6 Terra at the moment. But I think as we add complexity, we’ll probably consider reintroducing more of those different models. And we’ll use our evals set to basically auto-optimize that. So we might run our evals, see how they’re scoring, and then whichever combination of agents and architecture wins, we’ll just default to that. 00:10:08.64 | Michael Helbling: I’m glad you brought up evals, because it seems like that is the super time-consuming part of getting these things really tuned. Can you talk a little bit about that process as you built out this system? 00:10:22.09 | Christian Sullivan: Yeah, I think evals are probably the number one most important thing to this, because agents can sound very convincing. And you can kind of just say, oh, this looks great. I’m just going to have that set up. But until you’ve got a system where you’re repeatedly able to test the same investigations or the same kind of scenario that you would see in the real world, until you’re able to do that at scale and do that in a repeatable way, your ability to improve your product is impossible, right? Because you can only get so far as being able to improve the quality of the responses as you see it. But if you’re able to test 20, 30 different questions over and over and see not only is the agent able to answer them correctly, but is the agent able to repeatedly answer them correctly so that you can understand just how often is your agent going to veer off course on the same eval. So that’s been incredibly important to how we work today. I think we still need to build out our set of evaluations and to enrich them more, but it’s something that’s top of our priority list at the moment. One other thing as well, I’d say before building any kind of agentic workflow, starting with evals is really the most important part because you’re capturing what your agent needs to get right and what it needs to basically produce. And so once you’ve got a really clear understanding of that, then you know what you’re building towards, even if it’s just a single eval or three or five, just something to start with that basically says, this is our North Star. This is what we’re building towards helps you really identify what you’re targeting, but also constrain the scope to know what you’re targeting from the start, instead of building everything out and then trying to figure out which evals are actually going to fit your agent at the end. 00:12:06.86 | Moe Kiss: I’ve got to ask, I feel like some folks listening to this might be, oh, cool. A team like Canva can build this. You know, we’ve got heaps of data scientists. Like, what’s the time investment been? And do you look back now and be like, that was time well spent, like in terms of what it’s saving the team in investigations? 00:12:25.72 | Christian Sullivan: Yeah, I’d say the first version of ARCA probably wasn’t time well spent in what we produced, but time well spent in what we learned. But the second version, definitely. It probably took us a month to stand up end to end from, you know, having the data warehouse scheduler send the alert metadata to an agent, to an agent then performing the investigation and fine tuning that to then publishing back to our anomaly service. That probably took us about a month to build. And since then, we’ve noticed that our mean time to investigate an alert has gone down from about two and a half days to about two hours. And the quality of investigations has really picked up. And we’ve only really done this for the last month. So as more teams onboard to ARCA… We’ll get more quality responses. We’ll get more user feedback so we know what we need to improve. And I think over time we’ll get to the point where pretty much every alert is agent investigated and then documented and resolved with a human in the loop. One other thing I would say is that we made onboarding to ARCA incredibly easy. So all users need to do is tag in their alert like AI Agent ARCA and forever from that point, will have ARCA respond to the alerts. It’s not some custom configuration. It’s not, you know, pull requests that they need to make into our data warehouse repository. It’s a really simple onboarding process. And I think that’s kind of been key for us is to understand how can we, when attention is so hard to get through different LLMs and different tools that teams are producing, how can we make this as easy as possible? And that’s kind of where we’ve landed today. 00:14:06.44 | Michael Helbling: You mentioned you’re doing a lot of this on GPT 5.6 or whatever. And I think that just came out this past summer. When new models get released, what does that do to your process? Like, do you have to restart your evals? Like, what’s that look like? 00:14:22.83 | Christian Sullivan: Yeah, that’s probably where evals are also important, as well as knowing the costs of each run. So when we have new models, like we won’t jump onto them immediately. We generally wait for time to settle and we might kind of see what the response is. to those models. But the process is generally, we will understand what the model is targeted to do, understand if we need to make any tweaks on our end, and then switch it over, run a set of evals, or maybe just test out a few questions, understand on both cost and quality of the output. Are we getting a better result here? Is it worth switching over? And then we’ll decide to make that switch. In our data agents platform, though, which we use to kind of host, we have a really easy ability to switch between models and model providers. So that’s one thing that I think if you’re getting into the space, the ability to switch between and not be completely dependent on one architecture could be valuable. But I would also say that a lot of the models can perform reasonably similarly, and you’re getting updates, you know, almost weekly from each of the kind of frontier model providers. So it’s probably not the worst thing in the world to lock something in, if you have an existing agreement, or if you’re looking to build something fast, and then model kind of interoperability can come later. 00:15:45.91 | Julie Hoyer: How did you tell ARCA to respond? Like what type of language are you telling it for its output? Because I’m curious, like having the human in the loop, and it’s telling the analysts, like, is it suggesting to go look down a certain path? Or is it giving it like a confidence in it? Is it giving it evidence? Like, I’m curious the language you guys choose to make sure people are motivated in the future to like, check it still? Yeah. 00:16:07.72 | Christian Sullivan: So one thing I’ve been happy with recently, and I think this came from us switching to GPT, was that the responses are more kind of reasoned, and it will tell you if it wasn’t able to find any root cause. But this kind of goes back to the determinisms step. So we run kind of metric decomposition to figure out, you know, is there a really clear trend with a given sub segment that it’s not? Is causing this metric to move in this way? And if it can pick up that, then it’ll call that out. Another thing we do is we have a likely kind of root cause reason at the top of the alert. So what that will say is, is it a data quality is incident? Is it a seasonal event? Is it a true anomaly that needs to be investigated? And so that is kind of a signposting to basically say, oh, if this is a real anomaly, you need to look into this straight away. If it’s a seasonal event, maybe just check it, and then you can resolve it. Straight away. If it’s a data quality incident, maybe just wait till the next day. If it hasn’t loaded, then maybe raise the alarm bells and work with your analytics or data engineer to figure out, is there something going wrong in the data warehouse that we need to resolve? Or will this order resolve? And then the pros or summary kind of below that is generally just three or four lines saying, this is the likely root cause. This is what I checked. This is the metric movement that I’ve seen for this sub segment. And that’s kind of where the response is. One thing that we would like to build is the ability to resume that conversation and basically click a link so that you can have more detailed kind of documentation of all the queries that run instead of these just being accessed by us in the logs, as well as being able to resume that conversation with the agent. So you’re not picking up from scratch and you can say, hey, you investigated this. Have you looked at this segment split and being able to kind of really close the loop there easily and then have that agent published back to the alert so that we kind of own that complete investigation instead of a user. At the moment, seeing the alert running with their own investigation and published back in a separate workflow. 00:18:12.51 | Julie Hoyer: Amazing. Okay. Thank you for letting me do that. One last question. I had to know. 00:18:15.91 | Michael Helbling: No problem. Thank you so much, Christian, for coming on the show. Really appreciate it. Tim, what’s on your Black Friday readiness checklist? 00:18:26.81 | Tim Wilson: Inventory, promotions, landing pages, paid media, and apparently refreshing dashboards nervously every six minutes. 00:18:33.91 | Michael Helbling: Ooh, yes. One more thing should probably be on there: making sure your tracking actually works before the traffic shows up. 00:18:41.81 | Tim Wilson: Good point, because Black Friday is a terrible time to discover your purchase event has been broken since Tuesday. 00:18:48.01 | Michael Helbling: Exactly. Our friends at Stape have a free website tracking checker. It can scan your site and give you a personalized report in just a couple of minutes. 00:18:57.31 | Tim Wilson: It identifies tracking issues and gives you a prioritized list of recommendations. So, you know, what’s worth fixing first. 00:19:04.11 | Michael Helbling: You can also see how your setup compares with, with competitors and get estimates of where your tracking problems may be affecting leads or sales or ad spend. 00:19:12.01 | Tim Wilson: Much better to find that out before Black Friday than during it. 00:19:16.01 | Michael Helbling: Agreed. So click the link in the show description and run the free Stape website tracking checker. 00:19:22.12 | Tim Wilson: Your future holiday season selves will appreciate it. Michael, what if building an analytics app started with just describing what you want? 00:19:31.82 | Michael Helbling: Okay, that sounds suspiciously like a feature designed by people who have met analysts. 00:19:36.42 | Tim Wilson: That is the new app builder in Prism. You describe the app and before it builds anything, it asks questions to make sure it actually understands what you mean. 00:19:46.72 | Michael Helbling: Radical concept and AI that asks for clarification before confidently kind of ruining my afternoon. 00:19:54.62 | Tim Wilson: That’s the idea. Then it builds the app, checks the numbers and lets you refine everything just through chat. 00:20:02.01 | Michael Helbling: Ooh, so I could say like, okay, tweak that chart and add a filter, or, you know, clean this up and make it less ugly. Undo whatever you just did. 00:20:11.11 | Tim Wilson: Exactly. You can use your own templates or built-in ones for GA4, paid media, and more. 00:20:17.31 | Michael Helbling: Ooh, and I can preview what different users see before publishing? 00:20:21.51 | Tim Wilson: Yep, draft it, publish when ready, and use View As to check the experience for each viewer. 00:20:27.72 | Michael Helbling: Oh, finally. So looks good on my screen is no longer a deployment strategy. 00:20:34.01 | Tim Wilson: Click the link in the show description and join the Ask-Y waitlist. Use code APH and they’ll move you right to the top of that list. 00:20:41.81 | Michael Helbling: Build the app, boss it around, and ship it when it behaves. All right, up next, we have Nina Kumagai. Nina, welcome to the show. What’s your team? What projects have you been working on? 00:20:54.72 | Nina Kumagai: Thank you so much for having me, Michael. Hi, everyone. I’m a data scientist working in Canva’s Gen AI team. So I spend a lot of time looking at how people use Canva AI, and how they use it to optimize features and help teams make better product decisions with data. Some of the projects I’ve been working on recently is a prompt analytics platform. This is a project that came out of an AI discovery week, which is basically a week at Canva where everyone working at Canva doubles down on some cool projects, and there is no business as usual in that week. So yeah, that’s something that has been quite fun to work on. 00:21:41.21 | Moe Kiss: But Nina, can I say this one in particular might need a rebrand on the name, because the name of this one is called CATNIP. Can you tell us a little bit about CATNIP? 00:21:51.71 | Nina Kumagai: That sounds fun. I don’t know. Thanks, Moe. Yeah, about CATNIP, so it’s a funny one. It actually originally used to be called Cat NLP, but I think one of the PMs that I was working with gave some feedback and ended up becoming CATNIP. Basically, CATNIP is a prompt analytics platform and it stands for Canva AI, Canva AI, actually I now forget, Canva AI tool, natural input prompts, which is a very long name, so the acronym definitely helps. But yeah, this, tool was basically built under a week to help enable a lot of our stakeholders to understand what exactly users were doing on Canva AI. So to set some context as well, when we first began our department, generative AI department at Canva, it was roughly maybe two to three years ago. And at the time, our AI usage was a bit of a black box and no one really knew, what users were actually doing with our AI features. And so our prompt data was one of those really critical data points that would help our stakeholders and a lot of machine learning engineers or product managers or, yeah, a diverse group of workers at Canva to understand what users wanted from our AI, and what exactly they wanted. And so, we started to work with our AI, and what exactly different types of customers were doing with our AI features. 00:23:41.71 | Moe Kiss: So tell us a little bit about, you had this AI discovery week and you were, from memory, you had a bit of a team working on it. What was kind of the, like the project plan where you wanted to get from and to, through the course of that week? 00:23:54.71 | Nina Kumagai: A hundred percent. So yeah, initially, it sort of lined up quite well with AI discovery week, but we had this really, massive stakeholder request, and there are a lot of different questions on understanding how users were running AI presentation generations. And instead of trying to sort of answer that sort of ad hoc that I guess a lot of data scientists get, we decided to build a more scalable system so that a prompt analytics platform could help, a lot of different stakeholders resolve and self-serve some of those questions without data being a bit of a blocker and getting access to that prompt. So sort of, I guess, yeah, timed quite well with discovery week, but I’d say a lot of the groundwork happened probably one year before discovery week happened. So, given, prompt data is quite difficult to work with. We have a lot of compliance and privacy engineering and legal were folks that we had to work with a while back before we could build a system like this. 00:25:21.22 | Moe Kiss: So Nina, prompt data, like, I feel like we all have in our mind a little bit of like what some of the challenges are. I know talking to you all, I’m privileged. I get to sit near a lot of the Gen AI folks, but like, what are the challenges? What makes prompt data? So unique and different to the other types of data that data scientists are working with. 00:25:39.42 | Nina Kumagai: So the number one thing is it’s definitely sensitive data. And so when building a platform like this, we had to make sure number one, we walked alongside privacy engineering security and legal and every sort of step that we took. We had to get approval from those teams to ensure we were compliant with a lot of those, I guess, policies that we have. And number two is that I guess we had to do a lot of filtering, sanitization, making sure that some segments of data could never be accessed by anyone in the team. And of course, there were some mechanisms where if users wanted access, they had to go through quite a rigorous approval system. And we sort of had to approve every user on a case-by-case basis to make sure that there was a good reason for having a requirement to look at this sort of prompt data. 00:26:58.14 | Moe Kiss: And so we had a former guest on the show. His name’s Dr. Peter Fader. And this quote has just stayed with me for like a decade. But he said, you need to embrace the randomness of customer behavior. And anytime I was doing analysis, when you’re just like, why does this make no sense? Like, why did someone put a million dollars worth of like quotes in their checkout? I can imagine like prompt data is the absolute end degree of that, right? Where like you see all of the randomness. What’s that experience like? 00:27:26.96 | Nina Kumagai: It’s so interesting because when you’re working on Canva AI, you have this sort of idea of what users are doing with it. But when you read the prompts, it can be anything from just one word to maybe an essay. It can be anything from maybe a student asking for a cartoon or meme to an enterprise customer trying to build a branded presentation. And so that sort of range. As you just talked about, was quite difficult to work with, to be really honest. And it definitely made, I think, one maybe strength of doing a very quick prototype during AI Discovery Week was that we built the prototype really quickly. But then we kept iterating as we got more feedback from a lot of different stakeholders. And so. So every stakeholder that we had had a different sort of use case for looking at prompt data. And depending on that use case, maybe we needed to chuck in a different filter. Maybe we need to add in a mechanism for semantically searching through the data. So I guess, yeah, it was a very broad problem. And even now, I think it’s not something that I think has been like 100% fixed. It’s still, we’re still iterating on how we understand prompts and label prompts and categorize them. 00:29:12.00 | Julie Hoyer: Is CATNIP then an easy interface for stakeholders to, like, analyze the data, or was CATNIP part of… because you were saying it took like a year to get the data piece kind of ready. So what does, like, CATNIP look like, or when a user goes to interact with it, what’s that experience? 00:29:28.80 | Nina Kumagai: Yeah, so CATNIP is quite easy to use. So what was interesting was when we first built out CATNIP, I think high level I was thinking maybe product managers, machine learning engineers, maybe some QAs, but now I think we have a lot of different type of subspecialties using CATNIP and basically when you go into the interface you sort of see sort of a grouping of different filters and once you submit sort of that segment that you’re interested in analyzing, it words in maybe 10 seconds and then shows you maybe a subset of prompts with some high level insights on how things like publish rate or conversions or feedback change. So it’s a very, I think part of the reason why it sort of started small and was only used in the Gen AI department initially, but it sort of grew out over time to a lot of different departments and different specialties. We now have maybe trust and safety, security, growth, marketing, product teams using CATNIP and I think part of it is because it’s quite easy to use and it sort of can mold into what the user wants it to be where if they need to search for a specific keyword that’s possible if they want to filter down to some specific segments it’s quite easy to do as well. 00:31:16.96 | Julie Hoyer: So could you say how many people are on here trying to make like birthday invitations for their one-year-olds like that type of thing? 00:31:24.24 | Nina Kumagai: That’s right, yes. Yeah. 00:31:25.75 | Moe Kiss: Julie’s exact use case. 00:31:27.68 | Julie Hoyer: Guilty. Have done that on Canva AI. 00:31:31.48 | Nina Kumagai: It’s quite useful because I think every project we have at Canva is very different and so like maybe there are use cases where we do need to look at those sort of birthday invitations specifically and try to improve and add very niche features to our AI product to enable that. And I think machine learning engineers might use it to get like a SQL output and then use that to find your birthday invitations, or product managers might be using it to build a roadmap of where we want to go with birthday invitations. So yeah, that sort of thing is, I think, part of why it sort of took off. 00:32:18.50 | Michael Helbling: Nina, I’m very curious to learn if you’ve identified or seen any kinds of user trends emerge. Because obviously, like, prompts from users, it’s sort of a net new sort of usability interface, if you will, that a lot of companies are dealing with. And so like, there’s not any data out there really to kind of speak to like how people use it. Like when the AI first came out, of course, there was people writing about how we’re all becoming prompt engineers. But I think only a very small subset of people probably think or write prompts in that way. So like, have you observed over the corpus of the data, like any interesting trends of how users interact with or create prompts in the system? 00:32:56.00 | Nina Kumagai: It’s definitely changed. And I think part of the way it’s changed is based on how the product itself has evolved. Early on, when we first launched our Canva AI product, we actually ran a beta program. And it was only out to a certain number of users that sort of, I guess, volunteered to test the new AI product. And we had sort of like a Discord sort of chat going on, and we could talk to these customers while also looking at what kind of prompts that they were submitting. So back then, I think a lot of our use case, it was quite like nicely fleshed out data. Maybe now that it’s a bit more, it’s a bit more mature as a product, we’re seeing way more, maybe diverse prompts. And if we have like a huge education user subset, then we’re seeing a lot of education related prompts. If we’re looking at enterprise customers, we’re looking at a lot of brand related prompts. So I think, I guess, it sort of evolved on the way with the product. And I think, as well as it gets easier to use, maybe it sort of unlocks other sort of demographics to use the product as well. And so we sort of rolled out over time to more languages, to more markets. And we definitely see now that we have to translate prompts and the raw prompts are in like different languages. And so… Yeah, it opens up a whole other can of worms. Yeah, a whole other level of complexity. It’s a… Yeah, pretty complicated problem. 00:34:55.55 | Michael Helbling: That’s awesome. Well, Nina, thank you so much for taking the time to come on the show. I really appreciate it. Really appreciate your time. All right, turning now to Jack O’Hagan. Jack, welcome to the show. What do you do at Canva? And what kinds of projects are you working on? 00:35:11.09 | Jack O’Hagan: Hey, everyone. Good to meet you all, and thanks a lot for having me on. At Canva, I’m a data scientist in a team called DCV, which is Discover and Connect Value. Essentially, it’s a team which has a scope across MAU and also paid retention and upgrades as well. So it’s quite a broad team, which is kind of how this skill came about. It’s like a skill that looks across Canva and all the different experiments we’ve run. And yeah, it kind of fits quite nicely with our team having such a broad scope. We need to understand pretty much Canva as a whole, I think, yeah. 00:35:49.70 | Moe Kiss: So basically, Jack-O gets periodic messages from me being like, Jack-O, this analysis that you did last year, can you tell me about this? Hey, Jack-O, blah, blah, blah. Where he did this really cool meta analysis of, we obviously run a lot of experiments at Canva. I’m wanting to look at some of the metrics that moved. But Jack, can you tell us a bit more about how you’ve now taken that meta metrics that moved. But Jack, can you tell us a bit more about how you’ve now taken that meta analysis work you’ve done and how you’ve scaled it? 00:36:16.50 | Jack O’Hagan: Yeah, yeah, definitely. And just on that point, it’s part of the problem with having such a broad scope. You get like random messages from people from like who you’ve never met before, asking about something you’ve done. So yeah, that’s always fun. But yeah, this skill came about, it’s actually the evolution of something we’ve kind of worked on over the years. Initially, it was like a purely Python and SQL sort of really descriptive thing. And it was quite limited. And with AI, we’ve been able to evolve it and package it up really nicely, so other people can use it. But the skill itself is basically a way to look at our experiment data at Canva. And we run a lot of experiments across Canva. And often, it can be hard to understand, or even see the results of really niche experiments that might be really interesting, but it’s only known about in a really small team or in a group that might not share the results widely. So part of the skill is just surfacing interest in experiments that have moved a particular metric. And then the other part of the skill is trying to just synthesize a lot of the information from experiments, not just the result itself, but context about the experiment as well. And packaging all of that up into something that can neatly answer a question about how often a metric moves and the types of things that do move a particular metric. So yeah. 00:37:48.32 | Julie Hoyer: Who is using this skill? I mean, it sounds like you have to be very knowledgeable about what you’re asking, like what experiments or group of experiments or like, is it just data scientists using this skill? Or like, are you like, hey, and stakeholder who randomly slacked me that I’ve never met, go ahead and ask it your question. 00:38:07.40 | Jack O’Hagan: Yeah, right now it is probably more data scientists. It can be, we’ve tried to make it simple enough to use. So anyone could just prompt it with a metric and also with the group of experiments they want to run it across. And it will spit out the results for you in like a HTML file. These are the experiments that moved it, these are the experiments that didn’t move it, and then pulls out themes around the experiments. So technically, anyone should be able to run it, but it is at the minute, mostly data scientists. 00:38:40.82 | Moe Kiss: Is it fair to say like, so one of the problems, I mean, this is the problem that I faced, which is how we overlapped on this work stream, was, you know, a team potentially runs an experiment, it might move a metric that another team is responsible for, but they’re unaware. Was that actually the core problem you’re trying to solve? Or like, what was the core? How did it start out? Like, what drove this development, I guess? 00:39:04.90 | Jack O’Hagan: Yeah, that’s a good question. To be honest, yeah, that wasn’t really the core problem we set out to solve. I think the core problem for us was one of the metrics like our team specifically really cared about was paid retention or cancellations is the metric we look at in the experiments. And we had a really good idea of the types of experiments that move cancellations in our group. And we had this kind of shared knowledge just through like, sharebacks or like, discussions within our group. But we didn’t really have a good understanding of the things outside of our team that moved this metric. And when we’d seen results here and there, like a company wide sharebacks that someone had shared, we were often surprised that this experiment moved the metric that we cared about, we couldn’t really understand how it moved cancellations. And there were a lot of interesting results like that, that kind of cropped up here and there. So we wanted just a better way to be able to see which experiments will move in that metric scale. So we could feed that back into the product strategy, basically. So that was the core thing we were trying to solve. And then a kind of nice part of that has been we’ve just been able to see and keep an eye on experiments at the same time, but also come up with like a bigger strategy that’s not just as narrow focus as our team initially had, I guess. 00:40:33.03 | Michael Helbling: I’ve noticed in doing some skills work myself that sometimes when you start out, you can sometimes create a skill that makes the base LLM like Claude perform even worse than it did before. And I’m wondering, like, I’m sure you did tons of like evals and things like that. But like, were there components that surprised you that you needed to bake into the skill structure that, or not just surprised you, but like, were there any kind of light on moments that were sort of like, ah, this is how we get it to perform a little better when I when building out sort of the skill set? 00:41:04.62 | Jack O’Hagan: Yeah. That’s a really good question as well. I think, yeah, it was an evolution as we built it. And we kind of ended up in a place where we took a lot of control away from the LLM. I think at first we just tried to solve everything with the LLM with a prompt and gave it access to everything. And the results were quite hallucinatory. They weren’t very reliable. So we revisited it and tried to structure it a bit better where grabbing the actual results, was purely done in SQL. We have like experiment metadata tables that store metric results. So it’s actually quite easy to grab the results. It’s just the SQL query. It’s a little bit more tricky if the experiment didn’t have the metric applied initially. There’s a bit of Python and SQL that you need to write there, but you can still do all of that in Python and SQL. And even the metadata tables have a good set of descriptions about the experiment. They have the name, the title, and variant descriptions. So there’s some context you can get from those tables, but the problem is the context is not great. People don’t always fill it in. A lot of the time it’s null, and you can’t really use that to pull out the themes. So we really limited the LLM to just searching, pretty much. Searching like our internal documents, or even slack, grabbing context from there, and then synthesizing it. And even with that approach, we did notice some hallucinations and issues. So we pretty much instructed it to only grab verbatim quotes from, from either slack or the experiment documents. Because the good thing about the experiment documents, they were kind of used as a place for the team to discuss the experiment and set the parameters of the experiment. So if we can just grab direct quotes from there, we, we knew it could be trusted. Um, so yeah, we ended up in this place where a lot of it was actually done. We’d used some kind of like SQL and Python and then the kind of searching and synthesizing bit was done with the LLM, but limited really to direct quotes from people. 00:43:16.05 | Moe Kiss: So you’ve talked a little bit about overcoming the challenges with hallucinations, etc. What were the other technical challenges? Like it sounds, which is very top of mind for me at the moment. If anyone’s been hanging out with me, um, the, the quality of the context of the experiment. But what, like, that you had to overcome? 00:43:35.55 | Jack O’Hagan: Yeah, there were. So yeah, the big one was the quality of the sort of context and stopping hallucinations. But there were some other issues to do with the actual experiments themselves, especially looking back retroactively. Sometimes you could look at an old experiment and it had already been rolled out, but it had been continued to be analysed, like someone had just forgot to stop the analysis. And that would mean like one variant would have way more enrollments than the other. And there’d be like a split warning in our experiment platform. And often those were the results that looked really interesting. So we had to make sure we ignored those basically. We couldn’t really use them because they were sort of statistically wrong. That alongside getting the right context were probably the biggest challenge. 00:44:32.71 | Moe Kiss: So data hygiene, that fun, unsexy thing that data people care about, but the rest of the business are like, why are you spending time on this? 00:44:40.75 | Michael Helbling: Definitely. Did you ever find that the LLMs sort of stumble across something they take as a truth and then just never let go of it? Like I find that all the time where it’s like, oh yeah, we found this this one time and now it’s always true all the time. And we’re just going to apply it to everything. And it drives me crazy. But I was just curious, how do you manage stuff? Like that? 00:45:03.29 | Jack O’Hagan: Yeah, that could happen. And it happened a lot with experiments. Like the way people name experiments at Canva, often they might create a version two of an experiment that has a really similar name and really similar context to the previous one. But it could be quite different in terms of what it changed. And we noticed a lot of the time, it might place a really similar experiment to the old experiment. And you’d kind of get your wires crossed like that. So we set up some kind of strict rules where it could only use this experiment for this result. And we made sure like when we first did it, we only it ran sort of the same set of experiments every time. And it was quite wasteful. So now it’s like once we grab the context, we save the context down and it doesn’t need to run. It again. So we kind of avoid that issue a bit more, which is good. 00:46:07.40 | Moe Kiss: This is a really mean question. But a lot of this is looking post right at an experiment after. I suppose one of the things I’m always thinking about is how we influence decisions moving forward. What are your thoughts on that? Do you think there are next steps here? Do you think that us looking retrospectively is sufficient? Like, I’m just curious, I guess. 00:46:31.28 | Jack O’Hagan: Yeah. So that’s a really good question as well. And I think when I do share this with product managers or different people who might decide the strategy of where we want to go next, one thing I am really conscious of is we can really only learn from things we’ve done before. Like if we haven’t tried something, it won’t show up in the data. And it doesn’t mean it’s invalid. It might be a really good idea. So I wouldn’t want to share the results of this meta-analysis with someone and say, oh, these are the things we should only do. And these are the things we should ignore. And often some things that didn’t work in the past, like there might be a theme of things that didn’t work in the past. It can be sometimes hard to completely discount it because the reason they might not have worked, it might be to do with the execution. Times might have changed as well, that the users preferences might have changed. So I think you always have to try and approach the results with a bit of your product strategy hat on. But that is probably the trickiest part, to be honest, making sense of the data and making it apply to the future. The other thing which is kind of related to that is if you do any sort of meta-analysis like this, make sure to include all the cases where it’s neutral results. Because if you don’t, things get really bad. So I think that’s the biggest thing. If you’re really biased and you think, oh, this surface looks really great, there’s been 10 wins on it, but there’s actually been like a thousand experiments on that surface. So one thing we learned as doing this was to also include all the context of the things that didn’t just fail, but were neutral as well. 00:48:17.84 | Michael Helbling: Outstanding. Awesome. Jack, thank you so much for coming on the show. I appreciate it. All right. Turning now to Flynn Alexander. Flynn, welcome to the show. What team are you on and what kind of projects are you up to? 00:48:31.51 | Flynn Alexander: Hello. Thanks so much for having me. I’m Flynn. I’m from the Marketing Data Science Team. I primarily work across our page metric space. So looking at landing pages and how people are arriving at Canva. And I sort of like to think of myself as like the person who’s trying to tell the story of how they’re arriving, putting that narrative together and then going on. How do they get into the product? What are they doing? Are they converting? Bringing that entire journey together into like one cohesive narrative, I guess. 00:49:04.73 | Moe Kiss: And Flynn, your project, I have to say, wins the prize today for the best name. Can you talk to us a little bit about the GOAT model? 00:49:11.90 | Flynn Alexander: Yep. My coach is not going to like that you’ve said that. He’s definitely opting for the name change. But yes, the GOAT model is the Global Overview of Activity and Transitions. And essentially what it’s doing, and I will say this is probably less of a dedicated AI project and more of a traditional kind of marketing stats problem. But AI has sort of come into this and helped smooth over the process and turn it into something that is becoming a more powerful tool or becoming a more powerful tool for our kind of stakeholders to use, right? At its core, the GOAT model is applying a Markov chain to users arriving at Canva and looking at things like web pages. Um, also kind of looking at like, where did they come from? Right? So we take a page view as like a state. And essentially we say, given the state, um, what is the probability of someone going to a new state, page A, page B from that state? Like, do they go to the pricing page from the pro page? Um, and we can build a sort of network of transition probabilities. That determine the most likely pathways that someone might take as they move through the product. So that’s a really powerful tool. And it means that we can kind of get a, an overview of what our entire system looks like in one go. Right. And I would say that like this data itself is not particularly human interpretable. It’s, you know, we were dealing with thousands and thousands of web pages and entry points and, uh, a Markov chain does a really great kind of job of condensing that down just to the, the pathways that you’re more interested in. So what are the top five pathways that get someone to, to this part of the website or convert on this part? 00:51:19.45 | Moe Kiss: I remember when you were starting out this project, um, and Dr. Peter Fader once again comes to mind, embrace randomness of customer behavior. And I remember talking to you, like I’d done. Similar work previously. And one of the hardest things is like, what does top five actually mean? Because often users are so like unique and different that to even get a top five, it might only be like 0.000001% of our users. And then how is that like useful? Like talk, talk us through like how you overcame that challenge because I feel like you did 00:51:54.33 | Flynn Alexander: definitely still a work in progress, but yes, we’re working on it. Um, the, the, the way. That I would say is that like, you, you sort of have to, uh, pick your objective first, right? So like, what does success look like? And success in some cases is like just getting the most amount of people to a page, right? But they might not necessarily be the people that go on to convert. So if we have two different pathways that people head down, um, one might have a very, very low traffic volume, but the people on the coming down that pathway. Okay. A super high intent. So 90% of them go on to convert and maybe that’s the kind of people that we’re trying to target. We’re trying to get as many conversions as we want to get through the pipeline. Uh, the second pathway might have a thousand people coming through, but only 0.01 or something of them convert. Right? So you kind of have to balance what, um, what your targeted objective is and take that into consideration when you’re looking. At like the overall picture, I guess. And that’s, that’s kind of like a difficult problem that, um, a lot of, it’s probably more of a data science problem than it is a problem that you would give to your stakeholders, just given the complexity. Um, and one of the challenges that I sort of found with, with this tool was that I could put it in the hands of, of our stakeholders, but immediately it was followed by just a flood of questions and like, how do I use this? What does this mean? Um, these numbers that like, don’t really have all that much meaning to a lot of our stakeholders when they’re just trying to make practical like business decisions. Right. So I think the second half of this project, which is the part that we’re like actively working on now and trying to build out is like where like AI sort of comes in and like really helps, uh, build out what this. Could become. And traditionally, like when we first started this, we were trying to, to visualize this information through like Sankey plots, um, funnels, all that kind of stuff. Right. But it, it just, it doesn’t, uh, it’s difficult to create a system where the solution fits many different, uh, approaches, I guess. So. As I was experimenting with that. And using generative AI and the, the latest frontier models and whatnot to, to kind of help me build it. I realized that these models themselves are almost like the perfect UI to, to be using this kind of tool. Right. So they can provide, uh, all of the, you know, they’re essentially like a resident expert sitting next to you. They can help you target what you need. They can provide the parameters to, to input into the model. Um, they can then take the main pathways that are identified by that model, talk about them, apply stories and narratives to it. You can, you can essentially, if you have, uh, a pathway come back that says, you know, users go from A to B to C. I can tell the model to create a story around that. That’s easy for stakeholders to understand. And everyone knows that like generative AI is really good at that. And it’s really, really good at making stories. So this stuff becomes a lot more interpretable. Um, and that’s been like a really, really powerful realization, I think, coming out of this and seeing how we can kind of, you know, have a resident expert sit alongside and kind of get the most out of these tools. 00:55:47.46 | Julie Hoyer: So do you have it? I feel like I keep asking kind of like the user experience of these tools, but I’m so curious, like how you guys are designing and building these things to then help, you know, your different stakeholder sets, or if you guys are using them. in your own, um, analyses? Is it that you had the GOAT model and now you have, like, is it kind of like a chatbot interface on top of it, or, like, how… what does that experience feel like now that you have generative AI, like, looped in to help? 00:56:16.00 | Flynn Alexander: So, definitely preface that this is all still, like, a work in progress, and we don’t have, like, a fully productionized version out yet. But the way it works is that, like, the GOAT model itself is like an underlying foundation of, like, the stats and the math that gets done, we don’t really want the generative AI to kind of do any of the computation itself, right? Mostly because it’s quite complicated math and like hallucinations, all this kind of things will create problems. So it’s almost just like a wrapper around that where it can like plug in the inputs and then go on to like interpret the outputs, much like it would for like a semantic layer, right? So the intention, how we kind of execute this within Canva itself, I suspect will just be like internal tooling. Otherwise, we might look to partner with some of our kind of new reporting partners, maybe someone like Omni, not too sure. But yeah, it’s still definitely a work in progress. 00:57:18.51 | Michael Helbling: About a month or so ago, like a lot of noise happened in the AI space because of this new LLM that launched. It’s a decision LLM, Jev. And I was wondering, like it seemed like in initially reading about it, it may have some applicability to things like simulations and things like that. Have you looked at that at all? Or do you find it to be something that could be useful in that regard? 00:57:39.66 | Flynn Alexander: Personally, I haven’t. 00:57:40.18 | Michael Helbling: You can defer on that too if you don’t. It literally just came out. So it’s not like you have time to dig into it. 00:57:47.86 | Flynn Alexander: Okay, no, I’ll have to look into it. I’m not too sure about Jev. 00:57:51.90 | Michael Helbling: Yeah, yeah, that’s totally fine. And I think OpenAI, like a couple of weeks, They launched their own version, but decision engine versus like a text-based generative AI engine LLM. 00:58:05.26 | Moe Kiss: What’s the difference, Helbs? 00:58:07.38 | Michael Helbling: Now you’re putting me on the spot. The tables have been turned. I’m like, I need to go look this up. So a traditional LLM generates text as a responses or code or things like that. A decision LLM generates a probability or a decision that this is likely to happen or not happen. So in other words, like, is this true or false? And you give it a scenario and it says, probability is probably about 0.7. 00:58:31.00 | Moe Kiss: Is that not just adding a probability to the answer though? 00:58:34.30 | Michael Helbling: Yeah, it basically is. It doesn’t, the point is it doesn’t generate text and therefore it’s like much cheaper to use and that kind of stuff. Anyways, let’s not go down into it too far. Sorry, Flynn, to put you on the spot like that. Just curiosity. 00:58:48.76 | Flynn Alexander: No, no, it’s totally fair. And I mean, it sounds like it could be something that we potentially could use, but at the same time, you know, we have all of this like abundant, rich, kind of like page impression data and like click data available to us already. So for us personally, like it’s very easy for us to just generate those like actual probabilities very quickly. 00:59:13.26 | Michael Helbling: Yeah. And that was the other thing, because we did a show a while back on simulations, and I’ve always felt like this is an underutilized tool for a lot of marketers. And so this is a very fascinating topic. Just in its own right. So like how did you kind of leverage? I guess you leverage historical or existing data to kind of build out your model for the simulation itself, I assume? 00:59:39.42 | Flynn Alexander: Yeah. So I mean, that’s the great part, right? Like one of one of the big projects that we’ve had to push as part of this is just making sure that the underlying data itself is like super, super clean and normalized. Right. So If you’re going to ingest anything into a Markov chain, the state and what the definition of a state has to be very consistent. And that means that making sure how we are tracking page impressions, page views, and potentially in the future, we could even extend this to include CTA clicks and interactions on the web pages themselves. But all of that needs to be put into tables in a way that’s incredibly clean, structured, and consistent across the entire business. And I’m sure that anyone that has worked in a business as large as Canva knows that consistency in engineering across the entire business is always a bit of a challenge. So that’s definitely something that has been top of mind for us. And we’ve been working on getting that foundational data layer super robust so that we can ingest different time periods And we can compare across time periods. Because it really is just looking. You grab one month of impressions from a year ago versus one month of impressions from now. You can facilitate comparisons. 01:01:16.87 | Moe Kiss: Yeah, there’s really a lot of potential for the tool and how you could use it. I’m going to do the mean thing to you that I did to Jack-O. I’m always thinking about the future, right? So, like, we’re trying to understand the opportunity of, like, a new page, surface, whatever it is. We obviously don’t have historical data, but I do feel like this data set could actually potentially be quite useful in helping understand the likely performance of something based on similar attributes. Is that something that you’re thinking about? 01:01:41.11 | Flynn Alexander: 100%. And that sort of builds into the other work that we’re doing around building up that foundational layer and improving just the way that we’re looking at the data. And I think it’s around categorizing pages, right? Because if you are able to categorize pages efficiently and make sure that groups of pages are kind of associated, then if we are to release a page in the future that is in line with other pages that are currently performing, yeah, we could definitely do some kind of projections or simulations to kind of understand how that page might perform. And how it would fit into the existing network of pages, right? Because essentially, that’s just adding a new state. 01:02:35.05 | Michael Helbling: Awesome. Thank you so much, Flynn. And I want to say a huge thank you to start to wrap up to all of you, Christian Sullivan, Nina Kumagai, Jack O’Hagan, and Flynn Alexander. Thank you all so much for coming on the show, sharing your insight and knowledge. I think to me, the coolest thing was as we were talking, I know we literally only scratched the surface at a very high level, but even seeing, sometimes when you’re working with new tools and AI, you sort of feel like you’re on an island, sort of, like, making stuff up as you go along. And so hearing from your experiences, certainly for me, it’s highlighted and identified a bunch of things. It’s like, oh, yeah, I have something similar. And so that experience was top-notch for me, especially. And so thank you very much. I’m sure our listeners also got a lot out of it. So thank you. This is sort of an unprecedented look. I don’t think I’ve ever seen anybody do a thing where they dive into like different topics and categories and teams within one company and kind of look and see where AI is impacting it. So really appreciate you all taking the time. And I know if you’ve been listening, you probably have your own questions and we would love to hear from you. Our listeners, you can reach out to us. And the best ways to do that are through our Measure Slack chat group or LinkedIn or via email at contact at analyticshour.io. And if you’re listening, we’d love it if you’d leave a review on your favorite platform and let us know how you like the show that way as well. And if you’d like a sticker for your laptop or your water bottle, we’ve got some and we can send them to you. Just reach out on the form that’s on analyticshour.io. There’s a special form just for that. And we will send you one out. All right. Well, I have loved this episode. Can’t thank you all once again so much. And I know I speak for both of my co-hosts, Julie and Moe. No matter where you’re applying AI in your business, keep analyzing. 01:04:26.31 | Announcer: Thanks for listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at at analyticshour, on the web at analyticshour.io, our LinkedIn group and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst. 01:04:44.27 | Charles Barkley: Smart guys want to fit in. So they made up a term called analytics. Analytics don’t work. Do the analytics say go for it no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 01:05:03.07 | Julie Hoyer: Rock flag and weird Canva project names. The post #310: A Multi-Stop Tour of AI in Data and Analytics appeared first on The Analytics Power Hour: Data and Analytics Podcast .

October 27, 202656 min

#309: Interviewing in a Dynamically Evolving Job Market

Here is an interview question for you: tell us about a time you knew exactly how to do something, and then completely failed to explain it to three strangers in a conference room. If an example sprang to mind immediately, this episode is for you. It was prompted by a note from a listener who is confident in his ability to do the work and less confident in his ability to talk about it under pressure. So, we talked about it (under very little pressure). Topics covered: thinking out loud, saying “I haven’t come across that, but I’d love to understand it” instead of “I don’t know,” having your behavioral examples ready before someone asks for them, building a sprawling document of everything you’ve ever done so an LLM can grill you on it, and asking questions at the end that you actually want answered. Consensus: it’s good to demonstrate that you can use AI effectively. No consensus: whether or not you should bring an iced coffee with you. He did not ask about the iced coffee. We covered it anyway. This episode is brought to you by Prism from Ask-Y —your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. This episode is also brought to you by Stape , your all-in-one solution for server-side tagging. Links to Resources Mentioned in the Show (Newsletter) Invisible Women by Caroline Criado-Perez (Book) Invisible Women: Data Bias in a World Designed for Men by Caroline Criado-Perez (Conferences) MeasureCamp (all over the world) (Conference) DataTune (Nashville, TN) (Conference) Superweek (Budapest) Dog and Cat Pens (the ears move!) (Blog) Trevorwithdata Jim Kultgen On the show, we made a comment that “maybe you can’t get a free version of…Databricks…” but a listener pointed out that, well, actually , maybe you can! Photo by Linda Xu on Unsplash Episode Transcript 00:00:00.00 | Announcer: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. 00:00:15.33 | Michael Helbling: Hey, everybody. Welcome. It’s the Analytics Power Hour, and this is episode 309. Okay. A listener reached out to us and asked if we had any tips on interviewing. Thank you, Saad Islam. And sadly, we do not. Or, I don’t know, maybe we do. Interviewing for a job is something we all go through. It’s a process to test your knowledge and skill, also your nerves. And we’ve done shows on the job market before, but with everything changing in our industry lately, I think a conversation with some of our thoughts on interviewing in today’s market definitely could be in order. So, let me introduce my co-hosts. Moe Kiss, Director of Data for Product at Canva. How you going? 00:01:00.03 | Moe Kiss: Pretty great, actually. 00:01:01.48 | Michael Helbling: I would say you’re probably more the interviewer than the interviewee these days, if I had to guess. But that’s good. 00:01:07.18 | Moe Kiss: Yes. 00:01:08.10 | Tim Wilson: Unless there’s something you probably don’t want to record. 00:01:12.56 | Michael Helbling: Also, that brings us to Tim Wilson, quintessential analyst and running his new consultancy, Analytics the Right Way. 00:01:19.78 | Tim Wilson: Hello. 00:01:20.92 | Michael Helbling: And, of course, Julie Hoyer, Associate Strategy Director at Further. 00:01:25.42 | Julie Hoyer: Hi. 00:01:26.09 | Michael Helbling: And I’m Michael Helbling, President, Stacked Analytics. Okay, let’s dive in. First off, just level set. When was the last time any of us did any interviews of any kind? Because I think that’s probably the first level setting thing that we should probably talk about. It could be internal, like, you know, for a role that you’re in now at the same company. I feel like that still counts. 00:01:48.70 | Tim Wilson: I’ve done two phone screens in the last eight months that were miserable from the start. That’s pretty current. For roles I wasn’t interested in. 00:01:58.06 | Michael Helbling: I did an interview, I think, in late 2024, right, as like, I was wrapping up the partnership with Stacked Analytics and relaunching it. And it was sort of in a moment where I wasn’t sure what to do. It went so badly, I was like, yep, I’m sticking with Stacked Analytics. Keep going. But, yeah, that was my last kind of interview. 00:02:20.31 | Moe Kiss: Yeah, mine’s been years. Yeah, ditto. My last interview was seven years and three months ago. Four months ago. 00:02:26.62 | Michael Helbling: Nice. 00:02:26.90 | Moe Kiss: That I did. Wow. The last interview I conducted, though, was like a week ago. Oh, yeah. 00:02:31.09 | Michael Helbling: Nice. 00:02:31.48 | Moe Kiss: Conducted an interview? Yeah, last week. 00:02:34.06 | Michael Helbling: Yeah. And I don’t know. Does interviewing someone on a podcast count? Because then it would be a couple weeks ago. Yeah. No. Okay. So let’s dive into it. You know, I think one thing that almost every good analyst has because of imposter syndrome and the way the job market has shifted is, you know, you can kind of do the job, but it’s sort of like, how do you get yourself in the room and deliver sort of what the companies are looking for in that moment? You know, I think that’s sort of like one of the big challenges, you know, when you finally get your stuff together, apply for a job, like, how do you kind of deliver in that first interview? 00:03:16.21 | Tim Wilson: Which to me seems like it’s, I mean, I’m not interviewing, but it seems like it’s a little tougher right now because everyone’s expecting that you… They’re going to ask questions about AI usage and like the dynamically shifting landscape and you’re walking in saying, how do I articulate the skills and talents and experience that I bring? But the question may not come at you that way. 00:03:43.74 | Moe Kiss: Yeah, but I think there is this part about being authentic and I know that it doesn’t come naturally for everyone. And one of like, one of the things I often… Think back about is, yeah, look, you have to overcome nerves and you’re obviously trying to show your best self. And I think that’s really hard, especially because I think some of what’s changed is the expectations are even higher now because of AI, right? Like there’s an expectation you can know everything almost. And so I think showing that, I don’t know if vulnerability is the right word, but if you can demonstrate what… You know and what you don’t know and be curious when you don’t know things, like that’s actually the skill I’m looking for. I’m not looking for people who get the right answer every time. I’m looking for people who are humble and willing to learn and excited to learn. And I think you can do that in an interview by like being transparent. But I sometimes feel like I obviously, for me, that comes very naturally and I don’t think it does for lots of people. 00:04:50.50 | Tim Wilson: Like you can come in prepared and then you get a very specific… Question. And maybe that isn’t really what happens, right? We tend to ask, interviews are still largely behavioral. Tell me about a time that you did something. Maybe stepping back to those who, which is Moe and Julie on the how you’re running interviews, there’s the behavioral component that I assume is still there. Tell me about a time when. And those you just need to be prepared for. And then there’s kind of the technical, which I feels like it can be. Scarier. If you’re like, I’m not really a SQL person and they will have SQL in the job description. What is my response? I know I can do it and it’s easier now because I’ll have coding support. But like, and maybe that’s, those are two different things. And I know some companies structure it as there’s somebody, there’s the behavioral interview, there’s the culture fit, there’s the technical. But I feel… And part of we have like Saad’s whole note and part of his was like verbalizing the technical. Like how do you get between like knowing how to execute code, but kind of bridging to like, but it’s the value about how I think about the problem, not whether I know the syntax perfectly. 00:06:17.98 | Moe Kiss: The biggest thing, like when I started interviewing someone, I’ll be like, the purpose here is to get to know you and to have a conversation. The best thing that you can do. Is to think out loud. And that is really hard, I think, to do because you’re worried you’re going to say the wrong thing or make a mistake. But that’s where you can actually show the value, right? Because when you’re thinking out loud and you’re like, okay, this, I’m going to run this particular function and I know it’s going to have this output. But I’m trying to connect that output is like back to the business question. And I know the business question is this. And is that output going to align with it? If you can talk out loud as particularly in technical interviews. I feel like you’re like… I feel like you’re like five steps ahead. And maybe that’s something that folks need to practice ahead of time is like getting used to. It’s like, think about a math test when you were a kid, right? Like show your workings. It’s kind of the same concept. 00:07:09.31 | Michael Helbling: Yeah. Because I think I very much agree with that, Moe. when doing those kinds of interviews, I was much more interested in how they were tackling the problem and how they could look at it from each angle and ask good questions about how they would potentially solve it than if they had the specific technical skill in that moment. And I think sometimes like we get it twisted a little bit and there definitely are companies that are out there interviewing and they’re like, oh, if you make one code mistake then we’re going to wipe you off the board. And so people think they have to be able to be perfect with coding. Technically, I’m not sure you want that job if they’re interviewing like that. That’s not a good company. So it’s sort of like it’s helping you out a little bit in that process, even though while you’re looking for a job, it doesn’t feel like that. But that’s right. The way that you approach that is just to verbalize, yeah, here’s how I would tackle it. Because, like, that’s me all day, every day. I get so in my head about a technical interview because I don’t think of myself as that technical. And yet, like, I’ve got a good amount of technical capability, but like I’m not as good as the people I know that are good. And so if I ever got in that position, I would be so flummoxed. I’d probably like trip over everything. But that’s what I would do. I’d be like, okay, so here’s how I sit down and figure it out. And maybe I can’t write the code in front of you. But by gosh darn it, I’d be done writing the code by the end of the day because I don’t stop until I figure it out. And if that’s not good enough for them, well, okay, we’ll move on. And there you go. 00:08:46.50 | Julie Hoyer: Well, and I feel like you a lot of times with a technical role, it’s more important that you can communicate at all about what you’re thinking or doing for the people. That you’re working with. Like to your point, Michael, like I wouldn’t be shocked that there are a lot of companies out there or interviewers out there that would appreciate if you didn’t know everything off the top of your head compared to you’re willing to talk through it and collaborate with them. Because at the end of the day, that’s going to probably move the business forward or their team forward better than you were able to keep it all in your head and spit out like, you know, the exact things you were going to use. But without any reasoning or any like behind the curtain revealing how you think through problems. 00:09:26.37 | Moe Kiss: I think there’s a difference also between being so transparent of like, I don’t know this thing at all. Because sometimes I feel like people do that. And you have to be like really thoughtful about how you do that, particularly if it’s like a required thing, right? I might have talked about this like many years ago on the podcast, but the best advice, the absolute best advice I ever got was from a PM at Google. And she’d been doing some interviewing. And she said, I was like, what’s like the hardest, weirdest interview question you ever got? And it was one, I think Atlassian were doing it at the time, which was like, how would you build a spaceship to go to the moon or something? Like it was something that it was just like, huh? And her thing to me, the advice she gave me, because I remember I was doing a lot of interviewing at the time. She’s like, when you don’t know something, be curious. Right. And I remember in my Canva interview, this is how I knew it was the right job. There was a function I didn’t know. And I said, I’m not familiar with that function. I really love to learn. And someone was like, oh, I can help show you how to run that function. Like, let’s walk through it. Another interview I went to was like, oh, you don’t know that function. And like that also, like you said, Michael, helps you understand. Right. But like there’s a difference between I don’t know and oh, I’ve not come across that before. I’d love to understand it. And I think that’s the real like it’s a hard thing to get right when you’re feeling nervous. And there’s three people interviewing. 00:10:56.01 | Michael Helbling: In the right environment, not knowing something can actually put you in a different category of candidate in a positive way. And it goes to exactly what you were saying, Moe, which is how do you approach it? Like show the curiosity, show the interest. And the best advice I ever got about interviewing was people hire people. So like you talk to the person, not to the job. And that is always really helpful because it’s sort of like, well, what are they looking for? What’s we? What’s the best way to get the job done? And then you just read the person in the room and figure out like what that’s like. And sometimes people are not great interviewers. It’s kind of a dance sometimes, you know, like you sort of everyone’s going through the motions in a certain way. And you just have to find a way to break through and make a connection. And sometimes in those technical interviews, that’s a great way to do it is just show a little vulnerability. 00:11:50.00 | Tim Wilson: Michael, what happens when you give a whole company access to the same analytics app? 00:11:55.40 | Michael Helbling: Ideally insight, but historically, maybe panic. 00:11:59.60 | Tim Wilson: That’s fair. That’s why I like what Ask-Y’s Prism is doing with agents and row-level security. 00:12:06.20 | Michael Helbling: So the agent can build the analysis, but different users only see the rows they’re allowed to see. 00:12:11.30 | Tim Wilson: Exactly. One app, one analysis, different views based on the user. 00:12:16.30 | Michael Helbling: The East Coast manager sees East Coast results, the West Coast manager sees West Coast results. 00:12:21.90 | Tim Wilson: Neither one gets to casually browse the other team’s numbers. 00:12:25.90 | Michael Helbling: Which could be helpful, because “Oops, you weren’t supposed to see that” is not really a governance strategy. 00:12:30.90 | Tim Wilson: Meanwhile, the agent can still do the heavy lifting, answer questions, and build the experience. 00:12:36.90 | Michael Helbling: You get powerful AI analysis without turning every dashboard into a company-wide group project. 00:12:42.80 | Tim Wilson: Exactly. 00:12:43.80 | Michael Helbling: Love it. Useful, controlled, and significantly fewer awkward Slack messages. 00:12:48.90 | Tim Wilson: Want to check it out? Click the link in the show description and get on the Ask-Y waitlist. 00:12:53.90 | Michael Helbling: Use code APH when you sign up and they’ll move you to the top of the list. 00:12:57.90 | Tim Wilson: Code APH. Smart agents, appropriate boundaries. What a concept. 00:13:04.30 | Michael Helbling: Tim, one of the fun things about Stape Relay this year is that a couple of the speakers should sound pretty familiar to Analytics Power Hour listeners. 00:13:13.30 | Tim Wilson: Yes. We’ve had both Simo Ahava and Juliana Jackson on the show, and they’re both part of a pretty incredible speaker lineup. 00:13:21.40 | Michael Helbling: Yeah. It’s happening November 10th. It’s a free virtual event for technical marketers, analysts, and server-side tracking practitioners. It’s got 18 practical sessions across two different tracks. 00:13:32.90 | Tim Wilson: So you can bounce between product sessions and broader strategy depending on how deep into the weeds you want to go. 00:13:37.40 | Michael Helbling: And the speaker list goes beyond Simo and Juliana, with people from Google, Julius Fedorovicius from Analytics Mania, and more. 00:13:46.90 | Tim Wilson: There’s also live Q&A, and Stape will be sharing its next-generation GTM template roadmap developed with Simo and incorporating community feedback. 00:13:55.90 | Michael Helbling: And did I mention the best part? I think I did, but I’m going to mention it again. It’s completely free. 00:14:00.70 | Tim Wilson: So click the link in the show description and register to attend Stape Relay. 00:14:04.72 | Moe Kiss: But do you know, if someone said to me at the start of an interview, hey, just letting you know I feel very competent technically, but I am a bit nervous, I would be so receptive to that. I as an interviewer would be like, I mean, also, especially Julie, I don’t know if you feel this way, but I interview so much and I find them quite enjoyable because I get to chat to someone, I get to know them, I get to hear about their story, and I try and set that to be the atmosphere. But it’s been over seven years since I interviewed. It’s hard to put myself back in their shoes. So I obviously try and make it as, like, comfortable as possible. But, like, if someone said, look, I’m actually really, like, I’m pretty nervous, that would be a good reminder for me of, like, this is that person’s experience right now and, like, give them a break. But, yeah, as I’m saying, the vulnerability is a hard balance to get right. 00:15:03.30 | Tim Wilson: Which, not to get super tactical, but I do feel like there’s an application for AI that’s pretty simple here, which is if historically you’re like, we’ll do a bunch of interviews because you’ll get better at it. get developed and people who are getting back on the market and haven’t been for a while, they’re like, I just need to go and interview even if I’m not that interested because I got to develop the muscle. And to me, that is something that because there is massive amounts of information about what sorts of questions get asked and how they get judged and what good answers are, I think it’s eminently straightforward to say, I’m going to pick my AI platform of choice and I’m going to throw stuff at it and I’m going to say, ask me questions and I’m going to use the voice to text feature. And now this is, I mean, I’m on that like hobby horse a bit. I watched this happen with a former coworker of my wife’s where she was doing a practice interview. My wife was the practice interviewer, but it’s like, well, she didn’t have to wait for you. Like she could have, she could have easily done that just that throw me a curve ball and I need to ask, in the moment. And you know what, I’m also going to tell my, my Gem or my Claude chat, my Claude, whatever it is to provide feedback based on, you know, how I responded and take it with a grain of salt, but kind of getting in the reps. So I kind of equate that to like presenting and, and other aspects of where you’re needing to like be in the moment. Like there’s a lot of like nonjudgmental stakes, but it’s still going to get your heart rate up and, and use AI to practice. 00:16:56.58 | Moe Kiss: Okay. So there is a person I know who does listen to the show, who was pretty nervous about like the stats direction that data science is going and feeling a bit out of their depth. And so they’ve been doing some stats lessons with a very well-known statistician who works at Canva, who has been a guest of the show. And they’ve been like doing these lessons to kind of like improve their skills. And they, yeah, they wanted to test themselves and be like, I don’t think it was so much if I go to market right now, it was more like, I want to know that I’m developing the skill. And so they said to Chelsea, Hey, will you do the stats interview with me and see if I pass it? Right. So practiced rehearsed for it. And Chelsea, who is like, if you get Chelsea to give you a thumbs up on a stats interview, you’re doing very well. And this person did that as part of their professional development. Like, I think, I think public speaking is very similar to interviews, right? Like you have to like, honestly, the number of people I know who turn up to a behavioral question interview and don’t have like 10 examples lined up. Shocks me. Tell me about the time you managed, like had to performance manage someone who didn’t agree with you. Tell me about the time you took the team through like a difficult transition. Tell like, there are 10 questions that you should have an answer almost rote-learned for behavioral stuff. And I get very frustrated when people are like, uh, let me, think about that. And you’re like, I used to have 10 pages of behavioral questions prepped, ready to 00:18:27.40 | Michael Helbling: go. Yeah. Cause you’re, you’re hoping they ask those types of questions because you’ve got a story locked in with powerful points that showcase exactly what you want to communicate. Like Tim, you kind of said that off the top. So, you know, Tim, maybe you could talk a little bit about that sort of like, how do you do that prep or how do you kind of think through that structure? 00:18:49.02 | Tim Wilson: I mean, so I, that, that was something, I did. And right now, like the whole world, it’s, it’s a, it’s a meta world where I have permission to like play with AI stuff because I can say, well, this is kind of fun and interesting and I’m learning, but this also could also become an example in and of its own, in and of itself. So when I was heading down the path for, let me see if I am interested in a full time job, I did sit down and make like a 40, plus page corpus of my experience. And I kind of use Claude to kind of come up with a structure for that, but just like barf everything out and it can be unstructured. Like this is like a magical new world. I mean, this is like me sounding like an AI Homer. But then having that corpus to say, I mean, I wound up with like separate prompts. I had a resume generator that would match the job description to the resume. I have a, but I have an interview prep one where, it’s going to ask me, is this the phone screen? Is this an HR screen? Is this a technical interview? Is this behavioral interview? And it will take the job description and my corpus and start like probing me like for practice questions, which this is one of those funny things, like just the exercise of saying, I’m not going to be constrained to a two page resume and try to get my bullets perfect. I’m just going to try to just write down everything that I can think that I’ve done over the last 10 years. And as new things occur to me, or as I try new things, I’m going to sort of capture those. I think that exercise just been like, oh yeah, that, that kind of built my confidence a little bit of saying, yeah, yeah, yeah. You know, I have done something. You know, I’ve done some things that would not be likely to come to me in the moment. Cause, oh, that was seven years ago, but that was kind of a really, you know, I did do something that was actually kind of unique. And useful in a really difficult situation. And then having that, and then saying, now I’ve got this thing that lists all this stuff, probe me, dig, push me to figure out what my behavioral examples would be. Come up with the most common behavioral questions, prod me and say, hey, that time you were working with this client, was there anything that felt like that? And get me to, get me to think about it. So, but that I, the counter to that is by my nature, I’m kind of a writer. I can write, 40 pages in, you know, easily, like it’s just going to come up, but whether it’s dictating, whether it’s whatever, I was overly verbose, like AI. I sent somebody something today where I was like, this is a lot longer than I thought it was. And I didn’t use AI. So thank God I didn’t use 00:21:37.90 | Michael Helbling: Gemini on this, or it would have been way too long. Yeah. I think two points. Cause like what Moe was talking about, like be ready for those questions, you know, we’re coming, like what’s your greatest weakness? Like if you can’t nail that question, I don’t know that people ask that anymore or not, but like that used to be a pretty common interview question or like, what are your greatest strengths? Like those are questions that’s your time to shine, like be ready for those and just teed up and crush the ball. And that’s great. And then I think now in the age of AI, yeah, you want to show up with a portfolio of projects and things you’ve done. Like I used to tell digital analysts back in the day, like go make your own website, put analytics on it and show that you’ve done some of this work, that you’ve looked at your own data, that you know how the data works. Like it puts you in a whole other category than somebody who shows up and is sort of like, not sure what things are. Like, it’s just that ounce, every hiring manager out there is looking for people with drive, initiative, curiosity, knowledge, experience, those kinds of things. But honestly, like, the value on things like, I was interested in this, so I pursued it and I accomplished this. Okay, that immediately puts you way outside the curve in terms of who the crowd coming in to do these interviews is and puts you in a whole different class of candidate, especially at an IC level. I would say as a leader or like leadership types of things, there’s different things that you probably want to do. But like as an IC, the fact that you’re doing things on your own, and I know the question is going to come up of people, like, well, I work a full time job, like I don’t have time to do a portfolio of other projects understood. So your job is your portfolio, and you should be able to speak to those stories. But if you’re job seeking, and you don’t currently have a job, then use AI to build a portfolio of things you’re interested in. It’s like, and it doesn’t have to be, you know, like anything like Jacob Matson, who was on the show a few episodes ago, he’s doing all this data work with sports data. So he gets a hold of like NBA, stats and he makes data products out of them. Like anyone could do that. He’s just, that’s a hobby of his because he’s a nerd like us. And so he’s making these data products with NBA and NFL data. Well, you could go grab all the same data or any free data, weather data, currency exchange data. Like there’s lots of data sets out there that you can go slam into a database and maybe you can’t get a free version of Snowflake or Databricks, but you could absolutely spin up a BigQuery instance or Supabase or any of those things and produce a portfolio of things you could like at least talk to in an interview to show that you’re like doing things. Because I think that’s also one of the things that people are looking for. It’s like, okay, be nervous all day long, but like be motivated and show some intrinsic initiative. Julie, can I ask a question? 00:24:36.81 | Moe Kiss: Totally unrelated. For technical interviews, do you now allow AI usage? Oh my gosh. 00:24:44.59 | Julie Hoyer: It’s like… You were a fly on the wall last week. We were talking about this that, and I cannot remember, and this would be a critical piece of information, but we were talking like it was very obvious that the person had used AI. And we were kind of talking about, I think in the instructions, I don’t believe in the instructions we said you could or couldn’t. And it didn’t say they couldn’t use AI or maybe there was like softer language around like, you know, try not to use AI, something like that. Right. So like that’s where we were at. And the person, what they brought back, it was very obvious AI. And everyone was kind of like, oh, and it’s hard because they will have access to AI in their job. But I think what rubbed people the wrong way was not that they used AI to help them. It’s that they only took what like the AI output. Like it was not, you know what I’m saying? It was just like very obviously like not touched after talking to AI. But we were having the discussion of like, well, we didn’t, you know, strictly say do not use AI. And I think that’s what we were talking about. And I think that’s where we were getting into like the actual conversation about how can we use AI at all? But this person obviously, even with the soft language that was in the instructions, didn’t care to, you know, kind of craft it into their own final product. 00:25:56.16 | Tim Wilson: I mean, that makes me feel like in a technical, I mean, I would, it seems like it would be amazing to watch somebody use AI, not see it, not try to look at their output. But I am convinced there are people who still don’t prompt well. don’t necessarily know where Claude Code is. I mean, but the output is another one. Like, well, if this looks like clearly AI to us, then you’re going to be producing stuff that’s going to look clearly like AI to our clients, which I think consultancies are really grappling with right now. 00:26:32.31 | Moe Kiss: Julie, I would mark someone down for that. Not because they used AI, because they didn’t use AI well. 00:26:37.50 | Julie Hoyer: That’s exactly what we were thinking, yes. 00:26:39.92 | Moe Kiss: Yeah, so like, we’ve always allowed people, we say work as you would normally work, right? And so if you are dealing with a particular problem, and like back in the day, you would go to like Stack Overflow or whatever to troubleshoot. If you need to do that, you can do that. And that is always fun to watch, Tim, because you like watch what people search for and things like that. And now, obviously, we’re dealing with the same with AI. But what I do find is, number one, when people are using AI, the bar is significantly higher. We all expect it to be better, right? And I think that’s a fair… a fair standard to enforce. But I do think it means both the quality of the work and the quality of the interviewers and what they pick up on has to be much higher, right? Because the bit then I’m looking for is like, did they make a code mistake because they used an LLM output that was wrong? And did they pick up on that? And if they didn’t pick up on that, which, like I said, it sets a new standard for the interviewers, because the interviewers also have to be like, on the fly, be like, is that LLM output making a wrong join or like, you know, referencing the wrong table or whatever, which I do think is hard, but it does mimic the way we work. Yeah, totally agree. 00:27:56.50 | Tim Wilson: But I mean, there’s the higher order. If it’s an assignment of here’s a data set, and here’s the question and pull it back, there’s like the absolute… To me, the two that would be absolutely not acceptable is somebody who just went old school and clearly didn’t use AI at all. And they just went and they didn’t get very deep because they didn’t use the tool. The other one would be, I threw the data at the LLM and just asked it questions. And it may or may not make mistakes. And I think what I would be looking for is, oh, this is clearly somebody who inspected the data, maybe used AI to do some exploratory data analysis. And then they came up with kind of an approach And then they, I mean, if it was me, I’d say, let’s make a Quarto document to explore this. Let’s make a Quarto document to explore this. I want to visualize it this way And would actually review it. Like, it feels like there’s, AI is fascinating because there are people Who are just like so dismissive. They’re like, I don’t want to work somewhere Where we’re using AI. And it’s like, well, good luck. Like that’s, that you’re not going to be an effective employee. You’re not going to be able to get the output. There are the others that are like, “Whee, it’s magic! You know, give me insights.”. 00:29:17.53 | Michael Helbling: Yeah. But I think that’s absolutely right is working well with AI and understanding how to get results out of it. Because at the end of the day, like a good analyst is all about how you think about and approach and break down the problems. And all of the hard skills are all methods for doing that. So if an AI is helping you with that hard skill, that’s fine. You still need to own the overarching, does this make sense? Or it produced this result. Does that pass? Like, does that feel right to me? Like, because I understand that I took the time up front, right? Like Tim said, to like understand and pull the data in and get a sense for what it looks like. 00:29:58.53 | Julie Hoyer: Even outside of like the technical output, though, like just interviewing other analysts or strategists that I’ve interviewed lately, like I want to know that I can… I can collaborate with them. Like, is their output not just AI slop? Like back to that, right? Like, are they communicating clearly? To your point, are they curious? Do they want to learn? Do they want to get in deep and like problem solve? And then the AI part of it, I’m like, I don’t really need to know what AI is good at. I need to know what you’re good at, like outside of what AI enables us all to do. Like, what are some of those other skills that you bring to the job? And to your point, then like, you can enhance other parts of your workflow with AI or whatever. But I just feel like those things, like to me, are more important now because AI is out there. You know what I mean? 00:30:49.88 | Michael Helbling: Julie, were we still requiring a writing sample when you got hired at Search Discovery? We might have changed by then, but I know the first few years… I might have changed it. 00:30:58.04 | Julie Hoyer: Yeah, it might change. I don’t remember. 00:30:59.76 | Michael Helbling: Okay, don’t worry about it. But in the first few years of college recruiting, we did ask everyone to submit a writing sample along with the interview process, simply because how people communicate was so important. 00:31:10.42 | Tim Wilson: Interesting. Was that a prompt? Were they having to respond? 00:31:12.92 | Michael Helbling: They could submit anything they wanted. It could be a paper they wrote in college, for all I care. I just wanted to see what their writing looked like. 00:31:18.74 | Julie Hoyer: Wow. 00:31:19.24 | Michael Helbling: Yeah, they don’t need to write anything new. I just wanted to see how they wrote. 00:31:23.00 | Moe Kiss: Interesting. And Julie, can we just talk about, like, what a good example looks like there? Because, so the problem, I know we’re all doing this, but, like, literally, I read, like, 50,000 analyses. They all look the same. We were talking about this yesterday where I’m like, I open one up and I’m like, I don’t even want to read it anymore. And like, technically, technically, I looked at one yesterday and I’m like, it is correct. It has a McKinsey title, like inside headline. It has correct bullets. But I am like, it is 10 pages on a topic that I don’t care that much about. I’ve struggled to even want to read the TLDR. Like, and there was a few of us chatting in the leadership team about like, we were just getting flooded with these documents at the moment. And the expectation is that we read them and digest them. And I think it’s lazy. But like in that example, Julie, that you had, it sounds like that was a similar situation, right? Where I’m like, the bit, if someone had turned up and said, here is the AI output I got. But we all are having this problem where we don’t want to read and engage with this. So I have thought of a really creative way to use my brain to come up with an output that you want to engage in. And here is that bit. I would be like, 10 out of 10. Crush it. 00:32:40.11 | Julie Hoyer: Yes. Yeah. Crush it. Crushed it. Totally agree. 00:32:42.35 | Moe Kiss: So what’s the way that we get people to engage with it? Asking for a friend? 00:32:48.48 | Tim Wilson: Man, that feels like a whole other podcast. I feel like you’re hitting the crux. Everything is getting longer. Yes. Everything is clear. Everything sounds and feels the same. And there is, I mean, it’s comical. It’s comical when you talk about great presenters and you go back and it’s like JFK, Martin Luther King, Steve Jobs, all these presentations that didn’t use, that were just the person and their words. I’m not saying we should be going away from slides or going away from communicating the information. But that’s why I can just, in an interview context, boy, if somebody came in and said, look. I used AI. If you’re curious, this was after eight iterations. This is the full report that I had. But honestly, in my experience, people are overwhelmed by this and they’re not. So let me just stand up and tell you three things that I think are going on. And then, and here’s like two visuals. And then if you have questions, we should talk about it because I went back and forth enough with this that I feel like I understand the data. Maybe this is me projecting right now because I’m working on. But something where I found myself having to go look at the data because what the AI was producing and saying, well, this was kind of a weird thing. I’m like, I better go make sure that is a weird thing and you’re not just coming up with it, which meant that I was getting a deeper understanding of the data. So now I feel like with that client, I actually have the same way I would have without AI. I have a deeper understanding of the data and the quirks of the data. 00:34:36.48 | Julie Hoyer: But I’d rather hear that you did that exercise with AI. And then you’re like, and I. I made one slide based on what I learned after talking to it. Or I don’t let AI write my emails to clients. I’d be like, that is a point for you in my scoring. I just feel like keeping it a little bit precious about like, yeah, I use AI to stress things, to speed it up or stress test things, speed up my workflows, help me question things, pull out things that I maybe didn’t think to go look at, whatever. Like that, I do think is great. But if it comes down to like the communication and we’re talking about. Like the AI slop and you can tell it’s been written by AI. Like I am kind of to the point where I would ask people like, can you just try to write the final communication without AI? Like, let’s just like start ground zero. I mean, I don’t know if that really is realistic, but I’ve been tempted in a few situations. Like we had a whole, whole training for a large group of my coworkers that have come out of college. And I gave them an exercise and it ends up they all put the deck in the AI. And lo and behold, what came out was, you know, very, very similar and it wasn’t edited a ton. And like, I don’t know, I just feel like if they had read through the couple of slides and tried their own slide themselves first, like I bet they’d have a better output. 00:35:58.54 | Tim Wilson: Wow. This does go back to who said people, Michael, was that you who said that people hire people? So that’s like, the AI is going to suck the personality out of it. Yeah. And it’s, yeah. So I feel like that’s where you’re, and the same thing goes when we’re like working with clients. Like people don’t, they’re not hiring you because they’d like to be talking to ChatGPT. I mean, it’s like, so there’s a part where it’s like, well, you need to come in with your, and Moe, you said it at the start, like just be comfortable with who you are. Like, and if you’re, which is, which is tough. Yeah. But it’s also goes to the other topic that’s kind of bubbled up here, which is, we haven’t explicitly said it, but you’re interviewing, you’re interviewing them too. And this can be tough if you’ve been out of work for a long time and you really need a job. But if you, if you’re working and you’re looking for the next thing, if you have any ability to say, I’m going into this, I’m more concerned that I’m excited to work there than I am concerned that they think I can. And actually like the phone screens I had, I went in thinking, I, this is going to have to be a really great conversation for me to have any interest in going forward. And it wasn’t, I mean, I think one of them, we somehow we went and ran along on one. I don’t know why, but, oh, no, I do know why. Cause he showed up 15 minutes late and I, and it was booked such that I had to like head to the airport and I was like, okay, I made this kind of clear. Yeah. That was bad on many fronts, but I found that in the times that I have, which really the last time was search discovery. I was going into those interviews much more. Is this something that I will be excited about and want to do? And yes, I prepped and yes, I had examples and yes, I was ready for those conversations. But my mindset was, I’m more worried about me being excited about getting hired. By them than I am about proving that I can do the job and rant. 00:38:17.04 | Michael Helbling: Quick hitters. Another thing I really look for in an interview process is lateral thinking. So it goes back to Moe’s point about curiosity and, but like bringing in other things. So if you have an experience or interests that are not in direct line with the job, find a way to connect them anyway. And you can even make a little game of it because you can sort of say, all right, I want to talk about this in the interview. So I’m going to find a way to kind of relate that into something. And it can actually be a fun way for you to make a personal connection. And be like, I have a hobby where I really love this. And one of the things I think about all the time is like how I do this and this, and sort of in a way it fits into this way. And that can actually be really cool. One of the coolest interviews that ever did was with a woman who had done Teach for America. 00:39:08.59 | Tim Wilson: I thought it was going to be me, but… Yeah. 00:39:10.71 | Michael Helbling: I have no idea what your interview process was like at SDI. I was like, pretty sure I was like, yeah, sure. Whatever. No, she came in and she had no background in analytics. She’d been doing Teach for America since you graduated from school, but she did an incredible job of aligning that with how she would approach being a data analyst. And of course she knocked it out of the park and wasn’t what was incredible. 00:39:35.13 | Tim Wilson: Quit two years later. Cause she was like, I’m too good for these people. 00:39:38.35 | Michael Helbling: Um, well, I don’t know what happened there. Cause I think I probably left that company before she did, but anyways, yeah, 00:39:44.73 | Moe Kiss: I had, I had someone who was pretty rusty on SQL and managed, I don’t know exactly what it was, but they, they did all their SQL queries in Python. Um, I wasn’t in the interview, but I heard about it and they passed because the point wasn’t, can you write SQL? I mean, the, the thing that I’ve been reflecting on a lot lately, and I’ve been chatting about it. It’s, it’s like come up in multiple situations lately. And I think the thing that’s really scary right now is you just feel like you have to have this breadth of expertise on so many different topics. And I got a really good piece of advice, um, from a senior engineer here who, who was like, Moe, everything I know about your career, the thing that stands out to me is you have the ability to learn that when you need to learn something technical, you can. Right. So all those topics that you don’t know about, you just don’t need to know right now. Right. Cause you can’t know everything. So I think that is a really interesting thing. Like you said, Michael, of like the lateral thinking of the point you might want to bring into the conversation is like my SQL is a bit rusty, but what I do know is that when I have, like, when I’m excited and interested, I can learn something. I was able to learn to do blah, blah, blah, blah, blah. And I know if I were to get this role, I would have that same, like. Interest and desire to learn blah. That’s going to make me successful in the role. And I think there’s a way you can do that by. Bringing in the right examples. 00:41:11.05 | Michael Helbling: And the other thing that that benefits you, I think is don’t be afraid to show that you’re passionate about something. So even if it’s a personal interest or hobby or anything else like that opens the door for you to express and show a little passion. It’s a little weird to be like, I’m passionate about getting this job. Like nobody believes that, but like they would absolutely believe that you have an interest in something and that you’re excited about it. And actually, as long as you don’t go too far off the wall, people want to go with you on that and be into. The thing you’re into. Yeah. 00:41:40.58 | Tim Wilson: I am really into like on the side. I’m a dominatrix. You know. Yeah. 00:41:45.54 | Michael Helbling: Again, I feel like. Find work appropriate ones, Tim. Like Tim’s passionate about PowerPoint slides. That also doesn’t do very well in interviews, but you know what I mean. All right. So, and then the last one, and I think everyone will agree with me is like, bring some excellent questions to ask the interview. 00:42:05.40 | Tim Wilson: Yes. Yeah. 00:42:06.56 | Michael Helbling: Like, oh, it drives me crazy. Like you go through this. This whole process. You’re like, all right, what questions do you have? And they’ve got nothing. And you’re just like, so like, what are we even doing here? You don’t care. I don’t care. Like we just wasted each other’s time. Like what? 00:42:21.28 | Tim Wilson: I will say if it’s, if there’s been a real dialogue conversation and calling out saying, well, the biggest question I had was actually, you know, how the team actually collaborates. But I sort of feel like you kind of answer that in some ways, but I’ll ask it, you know, directly. Do you have. I don’t know if there’s anything else to add, but that’s another area that, that, I mean, definitely researching the company, knowing what’s going on in the news, like all that’s AI makes it super simple and AI can, but this is, this is that other fine line. And I can tell you good questions to ask. That’ll be the rote. Like, what is the team dynamic? How are decisions made? You better find the ones springboard off of that and come up with like, what do I really want to know? 00:43:07.44 | Michael Helbling: Yeah. Cause sometimes you’ll get people to ask questions. Questions have nothing to do with you as well. So be like, uh, you know, like what’s the, whatever. And you’d be like, I don’t know or care about that. It’s not anything to do with my job or the job you’ll be doing here. So like, that doesn’t bother me. It’s not disqualifying. It’s just not as fun to get into those questions. 00:43:28.14 | Moe Kiss: But yeah, I, I definitely mark up people who ask questions and ask good questions at the end. If people have no questions, that to me is a red flag. 00:43:36.04 | Tim Wilson: Can you put you on the spot? What’s a, what’s a good question you’ve been asked? Or is there any? Any memorable ones? 00:43:40.02 | Moe Kiss: I actually, when it comes to senior people, I actually do like the question of like, can you talk me through how decisions are made? Because I know that might seem trite, but it actually allows you to share a lot about a company’s culture very easily. I think that’s, I find a very good question. The question that I always used to ask is like, what do you think is the biggest problem facing this team in the next 12 months? And then I would. I would never answer it, but I would always send a follow-up email saying, I thought about that question. Here is how I think I could help you solve it over the next 12 months. Because then you have like a hook to like go back in with something memorable versus just a thanks for your time email. But the time I actually think questions are really important are particularly in the technical, in the scenario, like a scenario, because we do a scenario as well. That is where I really want to see people ask questions. Because you’re. You’ve been given a brief. The people who just churn through and some people will be like, okay, these are the assumptions I’ve made. Are they like, are they reasonable assumptions? And I’m not saying ask, spend the next 20 minutes asking 40 questions. But when people are like, I’ve got this brief and now I’m just going to answer. Okay. What’s the KPI? Like I want people to start a dialogue there. And if they’re asking me questions in the interview, I have a good belief that they would do so with their stakeholders working through a problem. 00:45:07.50 | Tim Wilson: Can I ask one other question? This will be a little dated by the time this comes out. But how do, where do we land on bringing in an iced coffee or an iced latte to a interview? 00:45:15.72 | Michael Helbling: I wouldn’t care. As long as you bring one to share, it’s going to be no problem. 00:45:21.10 | Tim Wilson: I totally, this is where I feel like I should. 00:45:23.24 | Michael Helbling: No, there was a news article a few weeks ago, Moe, about something. I don’t even know what it was about, but there was a discussion about iced coffee. 00:45:30.38 | Tim Wilson: It’s popped up on a number of mine, my different channels that there was an HR recruiter for large companies. And she was like. Don’t bring your iced coffee to an interview. 00:45:42.58 | Michael Helbling: I don’t understand why that’s a problem. 00:45:44.46 | Tim Wilson: I’m, I’m, I’m completely on board with that. I’m like, and one of the points she made is she’s like, it looks like you’re just running errands. Like you went and you refilled the car, you stopped by Starbucks, you’re stopping by the interview. You’re. 00:45:56.64 | Moe Kiss: I, I mean, I did feel bad. I ate some, I ate a bit of a snack on a call the other day, an interview, which I felt awful about. I was on, I’m always on mute because I type a lot. Um, and I felt very bad and very subconscious, but I also was like, I’ve been in. eight hours of meetings and I haven’t eaten and I’m going to like. 00:46:10.84 | Tim Wilson: Not the interviewer. This is the interview. 00:46:12.88 | Moe Kiss: No, I know. But I’m like, sometimes an interview is like, you have to bring your whole self. It’s a fucking coffee. Like people have to bring a coffee. 00:46:21.62 | Julie Hoyer: They offer you a drink usually when you go to an interview. 00:46:24.56 | Michael Helbling: Yeah, exactly. That’s why I don’t bring one is because they’re going to offer you a water or something. And usually I’ll say yes. Yeah. So, you know, that’s how you start. Yes. Yeah. I accept the beverage if they offer one in an interview. Absolutely. 00:46:37.86 | Moe Kiss: Oh, I would. I would never. 00:46:39.13 | Michael Helbling: I would just a water. 00:46:40.13 | Moe Kiss: Because then the person has to go get you one. 00:46:42.63 | Michael Helbling: Yeah. But the whole point is like, I’ll go with them and we’ll talk about the office and the layout. 00:46:49.61 | Moe Kiss: Okay. All right. That’s better. 00:46:52.01 | Tim Wilson: You’re like, why are you offering me water? I got my big old iced latte right here. What the hell? 00:46:56.51 | Michael Helbling: I’m carrying my big tumbler of. Yeah. No. All right. We’ve got to wrap up, but I’m glad we covered the iced coffee part of it for Saad, because I’m sure they were wondering. 00:47:08.63 | Tim Wilson: What? 00:47:09.13 | Michael Helbling: That was okay, Tim. All right. But we do want to go around the horn and do some last calls. Something that maybe our listeners would find interesting. If you’re interviewing for a job, hopefully the rest of that conversation was interesting. If it helps you find a job, please let us know by the way, because we want to know what tips and advice worked so that we can put, we keep a scoreboard. And so obviously we want to put a score up for whoever gave the best advice. All right. Now the last thing we do on the show is go around the horn and share a last call, something that might be of interest to our listeners. Why don’t we go around and do that? Moe, do you want to kick us off? What’s your last call? 00:47:45.88 | Moe Kiss: Sure. So I’ve taken a leaf out of Tim Wilson’s book and I have signed up for a newsletter, which I normally don’t do a lot of newsletters because I never read them, but I have a very high interest. 00:47:54.88 | Michael Helbling: With you there. 00:47:55.88 | Moe Kiss: So I signed up to Caroline Criado-Perez, who’s the author of Invisible Women. She has a newsletter that goes out. It is lots of stuff. So I kind of can’t summarize. Okay. I’m not sure if it’s the full breadth of topics, but like lots of different research on topics similar to the vein of the book. The most recent one that I found really interesting had to do with the invisibility of older women and actually women feel more comfortable looking older, like having wrinkles if other women around them do too. And so it’s a really interesting piece of research to do with like, you know, injectables and the makeup industry and things like that, of the more women that you see who mirror the way you look. The more confident and comfortable you feel. So anyway, there’s lots and lots of different articles in there and points to a lot of incredible research. But it’s one that I’m actually really enjoying reading. So. 00:48:48.88 | Michael Helbling: All right, Tim, what about you? 00:48:49.88 | Tim Wilson: I’m going to go with sort of a broad, find your way to get yourself to non-vendor conferences in person. I feel like the fall season of MeasureCamps, Michael and I were at MeasureCamp Chicago, MeasureCamp London happened a month ago. Datatune. And Nashville was coming up in the spring, super week. I’m just kind of of the getting an IRL with other people in the space. I’m just kind of a big fan on saying, figure out a way to pull that off, offer to volunteer if it’s one where you can’t afford, but it’s near enough by, to find a way to get to them. This community is so. Great that you’re missing out on something if you’re not in person. 00:49:41.67 | Michael Helbling: Hear, hear. Absolutely. All right, Julie, what about you? What’s your last call? 00:49:47.94 | Julie Hoyer: All right. Mine’s a little off the wall, but it’s about, you know, bringing a little whimsy into your workday. One, sometimes it’s nice to take handwritten notes or at least like my brain dump of a to-do list. Maybe that’s not everyone, but if you have to use a pen, I was gifted a pen that I heard are sold in large packs on Amazon. I’m not sure if you’ve seen it. It’s called the Dog and Cat Pens. I’m going to give you guys a view. It is so stinking cute. Its ears move. I like to put it in like speedy mode with the ears back and then you click the tail to use the pen and I just find them so delightful. Honestly. 00:50:23.32 | Moe Kiss: Do you know what I need though? 00:50:25.46 | Michael Helbling: Is the pen good? 00:50:26.46 | Julie Hoyer: The pen is good. I am picky about the pen. 00:50:28.94 | Michael Helbling: That’s the good thing. Yeah. 00:50:31.94 | Moe Kiss: So I have someone in my life that I would buy that for, but they need a felt tip pen and I’m guessing they’re not felt tip. 00:50:37.44 | Julie Hoyer: It’s not felt tip, but it is a good little, it looks like it’d be scratchy. It doesn’t not, it does not write scratchy because I’m weird about that. I like the gel pen feel. 00:50:47.30 | Moe Kiss: Can you link, can you link me? 00:50:48.86 | Michael Helbling: Nice. 00:50:49.46 | Julie Hoyer: Yeah. We’ll have it in the show. Yeah. They have cats too. 00:50:53.00 | Moe Kiss: The thing that I also love is like, I’m always looking for things that I can fiddle with in meetings. And if you don’t have the ears, then you’re just going to click the button and that’s really annoying. So stop it, stop it, stop it. The ears are good. Being, yeah, having something to play with, I think is good. 00:51:08.41 | Tim Wilson: Do the ears last? I mean, if you had an ear fall off, that feels like it’d get pretty sad. 00:51:12.50 | Michael Helbling: How much fidgeting are you doing, Tim? 00:51:15.79 | Julie Hoyer: No, I mean, I haven’t put it through like it’s 500 paces yet, but so far so good. 00:51:20.27 | Michael Helbling: Report back. All right. We need to get a shipment out to Moe right away so she can start to, she’s got a lot of meetings. 00:51:26.55 | Moe Kiss: No, one of my, one of my stakeholders, one of my stakeholders writes everything down and draws as you’re talking. And I love it on real paper. 00:51:35.77 | Tim Wilson: That’s nice. 00:51:37.21 | Moe Kiss: Visualizes everything. It’s such a like. 00:51:39.11 | Tim Wilson: What size? Like, is it like big old, like big old like. 00:51:42.13 | Moe Kiss: Sometimes she comes in with A3 paper folded in half. Like that’s often it, but it’s often whatever paper is around. Anyway, sorry, Michael. Over to you. 00:51:50.25 | Michael Helbling: I’m jealous of people like that who can do that. It’s such an incredible skill, honestly. Okay. My last call, speaking of people with skills, a good friend of ours, Trevor Paulsen, who used to be with Adobe. I think he’s on his own now or doing something. Oh, no, he works for a different company now. But he started a blog at the beginning of the year. 00:52:10.31 | Tim Wilson: My one time of presenting at Adobe Summit, I co-presented with Trevor. 00:52:14.81 | Michael Helbling: Oh, nice. See, I knew we had a podcast connection there somehow. But he was trying to work with AI, so he decided, why don’t I write my own version of a journey analytics tool? And so he kept track of his progress throughout the year as he worked on it. And so it makes for some really interesting reading on using AI, building with AI, and things like that. So it’s just a good read. So we’ll put a link to it in there, but yeah, Trevor Paulsen’s blog, it’s a good read. And shout out to Jim Kultgen from Amplitude, who is the one who told me about it. So that’s my last call. All right. As you’ve been listening, you’ve probably been thinking, oh, good thing I listened to this right before this big interview I walked into. And we hope you nail it. We really do. And go out there and do a great job. But we’d also love to hear from you, not just about interviewing, but about anything of interest to you on the show. There’s ways for you to submit topics on the website. You can also reach out to us on the Measure Slack chat group, on LinkedIn, or via email at contact@analyticshour.io. And we’d love to hear from you. So keep bringing those messages. You can also request a sticker. If you want one for your laptop or your water bottle, we will send it to you. And you can find that form on our website, analyticshour.io as well. All right. Listen, it’s tough out there in the job market, and especially with AI changing, everybody’s going to have to be very careful about what they’re doing. But one thing I’m positive about, and I know all my co-hosts agree, Tim, Julie, and Moe, no matter what job you’re working towards, keep analyzing. 00:53:54.65 | Announcer: Thanks for listening. Let’s keep the conversation going with your comments, suggestions, and questions on Twitter at analyticshour, on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst. 00:54:15.51 | Charles Barkley: Smart guys wanted to fit in, so they made up a term called analytics. Analytics don’t work. Do the analytics say go for it no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 00:54:36.30 | Michael Helbling: That’s what we’re going to do. 00:54:37.90 | Julie Hoyer: I like that. 00:54:39.20 | Michael Helbling: We actually. 00:54:40.10 | Julie Hoyer: Fucking genius. For podcast morale. 00:54:42.50 | Michael Helbling: Yeah. It’s not even that. 00:54:44.60 | Moe Kiss: No. For annual planning. For annual. Yes. 00:54:49.40 | Michael Helbling: So Tim can get angry with us in person. 00:54:52.40 | Moe Kiss: And we can storm out of the room. 00:54:55.10 | Michael Helbling: We’ll make sure to have an extra room Tim can storm out of. 00:54:58.70 | Moe Kiss: Oh my God, Tim, we can all stay in a house together and be together all the time. 00:55:04.50 | Tim Wilson: Again, where am I going to storm out to? Can we do a Vegemite martini? Ew. 00:55:11.10 | Moe Kiss: I’ve never tried a Vegemite martini. Someone has made that. 00:55:15.30 | Michael Helbling: Probably you could do a Vegemite martini, but me, it would not take very much Vegemite. There’s two ways that I’ve had it. Spam musubi, which is really good. And also in a Korean dish called budae jjigae, you cook it in there and it’s also pretty good that way too. So yeah, there’s a couple of preparations of spam that are pretty good. You can also fry it and eat it as like a breakfast meat. 00:55:37.90 | Tim Wilson: Tony, we’re about to start the show. Tony, enjoy. Tony, if there aren’t outtakes from this, you’re… 00:55:45.90 | Michael Helbling: Include that as an outtake, Tony. All right. 00:55:52.00 | Tim Wilson: Rock flag and don’t bring iced coffee to the interview. 00:56:08.30 | Michael Helbling: Oh, Jesus. Jeez, jeez, jeez. 00:56:11.00 | Tim Wilson: If you do though, just make sure it’s not really high condensation yet on the cup because 00:56:17.00 | Michael Helbling: you don’t want to leave like a circle of like… 00:56:19.70 | Tim Wilson: Just don’t bring it. The post #309: Interviewing in a Dynamically Evolving Job Market appeared first on The Analytics Power Hour: Data and Analytics Podcast .

October 13, 20261 hr 1 min

#308: AI Changes Everything! Analytical Fundamentals Still Apply!

How do you know whether AI is actually making your organization better? Counting prompts, sessions, and token spend is easy. Figuring out whether any of that activity is driving business outcomes is…harder. Daniela Matinho , Marketing Science Senior Strategist at LinkedIn, wrote a Medium post on that very question, and it helped shape this discussion about what AI is (and isn’t) changing about the work of the analyst. As Daniela pointed out, if you tell a client you did something in two days instead of a week because of AI, the client doesn’t care. They care about the result! From the smart-but-junior agent that needs constant supervision, to the analyst’s shift toward being more of a storyteller, to the unglamorous-but-essential work of data quality and governance, we covered a lot of ground. Being faster is great. Staying influential is better. This episode is brought to you by Stape , your all-in-one solution for server-side tagging. This episode is also brought to you by Prism from Ask-Y —your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show (Podcast) Still Counting (Article) Nearly every S&P 500 company talks about AI, but very few are securing it by Cassie Kozyrkov (Website) The Archive of Incorrect AI Predictions Photo by Ricardo Gomez Angel on Unsplash Episode Transcript 00:00:00.00 | Announcer: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. 00:00:16.00 | Michael Helbling: Hi, everyone. Welcome. It’s the Analytics Power Hour, and this is episode 308. I was having a conversation with a coworker the other day, and they mentioned that with everything happening in AI, sometimes it feels like the skills and experience they had no longer applied. AI is definitely disruptive, but do we have to abandon the fundamental analytics skills we’ve got just because AI can do a lot of the work? And as we all adopt AI in our businesses, how should leaders think about measuring its impact? It feels challenging. Maybe we’ve got some skills that could help along the way. All right, well, let’s talk about it. First, let me introduce my co-host, Julie Hoyer. How are you doing? 00:00:58.60 | Julie Hoyer: I am doing great. I’m very excited to chat about this today. 00:01:02.38 | Michael Helbling: Me too. And Tim Wilson, I know you’re chomping at the bit for this convo. 00:01:07.71 | Tim Wilson: I just, I’m sending an agent in my place, so this is the last I’ll be here as a human. 00:01:11.90 | Michael Helbling: That’s right. mused AI, the latest and greatest. And I’m Michael Helbling, and I’m so excited. Our guest today is Daniela Matinho. She is a marketing science senior strategist at LinkedIn. Prior to that, she’s held marketing and data science roles at companies. Like Snap, Meta, and PepsiCo. And today she is our guest. Welcome to the show, Daniela. 00:01:33.21 | Daniela Matinho: Thank you. What a great intro. 00:01:36.19 | Michael Helbling: Ah, well. AI helped. No, it did not. I wrote that myself. AI does not touch the intro. We tried writing with AI and it didn’t sound right. Okay, Daniela, to get this conversation started, and thanks for, we’re excited to have you. Maybe just talk a little bit about what you’re seeing. What are you seeing in terms of how AI is affecting analytics roles and in the orgs that you’re a part of? LinkedIn especially or other places. Where are you seeing it have impact? Where are you seeing it impact people’s roles and changing things? 00:02:10.69 | Daniela Matinho: Yeah, happy to talk about overall what I’m seeing in the different industries and organizations. Overall, as Tim was saying, speaking a lot to agents. I feel like this is something that we weren’t doing before. We were doing it all a few years ago. Starting to do any sort of work, asking AI, what do you think about this? Or give me some prompts has completely changed. When it comes to the more analytical, technical part of the things, a lot of the coding that I was doing in the past would be from scratch. These days, we are starting with agents or anything that has sort of built in. And we are coming in. And enhancing it or making it a little bit faster. So overall, a lot of efficiency, a lot of expectations on delivering more faster. That’s what I’m seeing as I talk to colleagues, friends. It’s a lot about doing more in less time. Just because we have this army of agents that are helping us deliver. And delivering a lot of other things that we couldn’t do in a short period of time before. 00:03:31.65 | Tim Wilson: So can I ask about the more? Because I was with my oldest son who’s a software engineer recently. And he’s developing. So more for a software engineer is he’s going through tickets and tasks. And he’s evolved the way he works. And he is prompting and having code written. And then goes through a code review. And code gets pushed out. So it’s like more features and bug fixes. When it comes to analytics, something that, and I also literally today experienced this. The more being produced by analysts, and I’ve seen this happen in other places. It’s like, oh, I can just generate a 29-page report. I can just generate more. Or we were at a conference earlier this year. And somebody was talking about. Look, I’ve now managed to. It used to be so ponderous for me to produce this weekly report. But now I can do it. I can do 10 weekly reports. And they’re 10 times as long. And I mean, there’s the macro view of people saying. I’m writing stuff that’s longer because they’re using AI. And that feels like more of the analog to the analyst challenge. It’s like, well, it feels better because it’s more output. And it’s more thorough. And I explored more avenues. But somebody’s got to receive that and process it. So I’m curious if you’ve got thoughts on that. 00:05:05.62 | Daniela Matinho: Yes, I have a lot of thoughts. I have this theory that no one is reading anything anymore. Because we are delivering to your point, Tim, these huge reports that are so thorough. But no one has the time to actually go through it. So when we talk about delivering more. And to your point on the analytics and the analyst role. Is really to cutting the chase. Meaning to cut everything that is not needed. Working a lot with sales in my past 10 years of roles. Sales need something that is very direct. A 10-page report is never going to make it through. So my role in any of the organizations as I use AI more and more. Is leverage it to produce it overall. And then I come in and really understand what are the things. The 3, 4, 5 things. The 3, 4, 5 numbers. That are going to be used for punchy for sales for clients. A very interesting experience that I had just recently. I was working on a hackathon with some ladies. And we were basically using Claude to do a lot of the coding. We go through the coding review sessions. And we get to the point. What is happening? Like he did all this code. But it doesn’t make any sense. We can build a dashboard in one hour. But to your point team. We are looking at numbers. Well these assumptions they don’t make sense. So babysitting the code that is being produced. The numbers that are being extrapolated by. From the data that we use in the initial stages. Is a very critical and important point. That’s what I’m coming to realize. 00:06:57.62 | Julie Hoyer: Going back a little bit to maybe how this is changing people’s roles. We’ve talked about this a little bit previously. But the idea of instead of staring at the blank screen. And putting in the assumptions in your code. You know that you guys felt was right. Having to like babysit the AI as you call it. And the coding. And looking at code. And then having to like re-engineer it. And decipher it. And then find the logic to like edit it. And shift it. We’ve talked before how that feels like very different like centers of your brain. Right? Different skills. It feels different when you’re doing the work. Do you feel like that aspect of utilizing AI has really shifted people’s roles? Or it’s kind of just a byproduct and like an obstacle? Oh yeah. 00:07:44.62 | Daniela Matinho: A hundred percent. I find myself way more of a storyteller today. Than I was before. Because the hard skills of coding. And I went through interview process not too long ago. And I did maybe a third of technical coding challenges than I used to do when I was interviewing before. And a lot of the focus is being on case studying. Tell me the story behind the data. Tell me what’s happening. Tell me how is this impacting the business. So a lot of analysts that perhaps in the past would do very well because they were strong in SQL. They were strong in Python. These days they are finding this situation that oh now I have to kind of tell the numbers to people. And I feel like it’s shifting a lot towards facing other people and telling the story. Things that agents are still not doing. But maybe in the future they will be doing. 00:08:50.62 | Tim Wilson: There was something else you said. I don’t want to put words in your mouth. So I want to, I’m going to put words in your mouth and see whether it was accurate. Because you started to say that there’s more, I think you said something along the lines of there’s more like facing the prompt and like doing ideation upfront with AI. Did I hear that right? I don’t know if I was wanting to hear you say like, no there’s like upfront thought that AI can help. That AI can help you with before you dive into having the agents go and crank on data. 00:09:23.62 | Daniela Matinho: Yeah. So I usually like to work in, I would say two ways. And yes, to your point Tim. Sometimes in the past I would be working on a project and had a blank sheet of paper and so no ideas. Talking about content and talking about code. These days I can prompt AI to start. Sort of the wheel and start things running. So that’s one way of working. One thing that I’m finding more effective this ways is being very concrete in the one, two, three ideas that I give to AI before I start on a project. And I say this after a year and a half sort of using AI constantly, I’m seeing that there is so much hallucination. There is so much content that it can produce. For me. That right now I’m like, well, I have to be very precise in what I need. So I don’t get the 10 page report instead of a one page report. 00:10:24.62 | Tim Wilson: But I guess is there a flip side of that that is just using the AI as a, and I feel like this seems to be coming up in more and more conversations of like, well, this is like a non-judgmental, mildly sycophantic, somewhat knowledgeable, sound AI. So I think it’s really important to have a knowledgeable sounding board to just be asking it like, well, could it be this or could it be that? And guiding it to say, but give me the reasons this is a bad idea. Like I found myself doing that where it’s like, oh, this sounds great. And I’m like, okay, tell me all the reasons this is terrible. And in the specific case, it went through and said, well, these are where the holes could be blown in it. And then it ended by saying, but you made this. But you made me say something negative about your thought process. I don’t really buy into those. I was like, okay, well, that seemed like a useful, it sort of gave me the, to consider that added perspective. Because like human nature is to get caught on one path and saying, okay, I’ve got something that’s not going to get annoyed with me, you know, continually asking it to sort of force me off that path to broaden my thinking. Michael, I believe November 10th may be a significant holiday. 00:11:43.62 | Michael Helbling: I’m afraid to ask. 00:11:47.62 | Tim Wilson: It’s Stape Relay. Basically GTM nerd Christmas. 00:11:52.06 | Michael Helbling: That might not be the official description, but Stape’s first virtual event does bring together technical marketers, analysts, and server-side tracking practitioners for 18 live sessions. 00:12:03.08 | Tim Wilson: Including previous podcast guests like Simo Ahava and Juliana Jackson, which is already a pretty convincing argument. 00:12:09.67 | Michael Helbling: Plus speakers from Google, Jellyfish, and others. And you can switch between two tracks covering hands-on product implementation and actionable strategy. 00:12:18.53 | Tim Wilson: Stape and Simo Ahava will be revealing their roadmap for a new generation of Google Tag Manager templates, developed with community input. 00:12:26.51 | Michael Helbling: And the whole thing is online and completely free. 00:12:30.41 | Tim Wilson: So clear your calendar for November 10th. 00:12:33.27 | Michael Helbling: Click the link in the show description and attend Stape Relay for free on November 10th. 00:12:39.63 | Tim Wilson: Michael, everyone says AI needs more context. 00:12:44.83 | Michael Helbling: Agreed. And what I do is I just upload the company handbook. Six strategy decks, 14 call transcripts, a glossary, and Carol from finance. 00:12:55.95 | Tim Wilson: Please don’t upload Carol. 00:12:58.24 | Michael Helbling: But she knows things, Tim. 00:13:01.62 | Tim Wilson: Well, Prism’s context notes give you a slightly better approach. You can feed the agent business knowledge, playbooks, transcripts, glossaries, PDFs, whatever helps it understand how your organization actually works. 00:13:14.24 | Michael Helbling: Okay. And not to be disagreeable, but six months later, half of that is wrong. 00:13:18.91 | Tim Wilson: Disagreeable is kind of your thing, but that’s the good part. In Prism, context notes can refresh instantly or on a schedule. 00:13:26.63 | Michael Helbling: Oh, so the AI doesn’t keep confidently applying last year’s rules like a middle manager who missed the reorg? 00:13:33.05 | Tim Wilson: Exactly. You can even give Prism instructions for how each note should be updated. 00:13:37.67 | Michael Helbling: Oh, that’s big because stale context is just bad data wearing a tiny knowledge management hat. 00:13:44.53 | Tim Wilson: A knowledge management hat that doesn’t fit. And since that context is managed centrally, your users get better answers without everyone rebuilding the same background knowledge in their own AI chats. 00:13:55.53 | Michael Helbling: Yeah, less copy paste, less tribal lore, fewer conversations that begin. Okay, first you need to understand something weird about our business. 00:14:04.53 | Tim Wilson: Click the link in the show description and sign up for the Ask-Y waitlist. Use code APH and they’ll move you right to the top of that list. 00:14:12.53 | Michael Helbling: Keep the context fresh and Carol can stay in finance. 00:14:18.53 | Daniela Matinho: Yeah, that’s a great point. And also the way I use it as I produce something for content, for code, for work. A lot of the perspective I want at the end of producing something that I’m confident about is what are some of the holes? What are some of the things that I might get asked about so I can prepare upfront for it? So that’s one thing that I have been trying to use it more so I can feel stronger as I present. And then the other thing is that I have a lot of confidence in the content as I deliver because I have thought through it with AI. 00:14:54.53 | Michael Helbling: Yeah, I’ve noticed sometimes in working with analysts on things, you have sort of a disconnect of between the person and what the AI produced. And I’m really adamant with our team on you have to be out in front, not the AI out in front. Because if someone is in a meeting asking a question, they’re not asking the AI, they’re asking you. And you need to be able to answer. You can’t just be like, okay, let me plug your question into the AI and then find out what the answer is. And I feel like people are getting that a little backwards right now sometimes because they’re getting or doing so much of their work in AI. They’re kind of letting the detail slip past them a little and then they don’t know the details of what they’re talking about. So it’s a tricky thing to balance because like AI is helping us be efficient. It’s writing a ton of SQL and running queries. And generating reports. But then if you’re not looking at it yourself and being really careful, then you’re sort of at the cart and the horse get flipped around, I think sometimes. 00:15:58.53 | Daniela Matinho: Oh, yeah, 100%. I mean, all the, I have this theory that a PhD in philosophy these days is going to be so valuable. Because that critical thinking that AI is sort of in a way taking away from us. Because we are asking the question. We are asking AI to think for us rather than to Tim’s point. Okay, tell me what I didn’t think about. So big believer that that critical thinking piece is somewhat getting lost in some roles, in some conversations. I see AI and I was actually having a conversation earlier for another hackathon. See AI a lot on the automation. I see AI as a tool for tasking sort of situations that we encounter. Book me a meeting for days or purchase this for me that I purchase on a recurrent basis. That’s where I see that there is a lot of potential. Building agents that can really do recurrent tasks for us. I see it with a lot of potential not only on the workforce but also for our personal lives. 00:17:14.53 | Julie Hoyer: Yeah. So you were mentioning critical thinking being a skill that it’s kind of taking from us and people probably need to be a lot more careful about keeping that as something they’re doing ahead of the prompting like you were saying. Are there other big like buckets or big skills that you would say are fundamental to analytics that you wouldn’t like sub in AI to do for you yet? 00:17:43.53 | Daniela Matinho: Understanding the business. starting as I work closely with clients, with advertisers, focusing on what AI cannot guess. Because, again, we can all at this point write SQL, Python, R with AI, but understanding the business, what is the client journey that each company is optimizing for is something that perhaps a nuance that AI is not going to be able to give us. Stakeholder management as well is something that, yeah, AI can write the emails for us, but it’s the type of skill set that I’m finding more and more interesting. The fact that it’s just so important, even more important than it was before. 00:18:38.88 | Tim Wilson: Does a deep understanding of… Even though we can use AI to write the SQL or the Python and R, is there a part of it? I worry about the risk that just like if you just turn somebody loose with a traditional BI platform five years ago, they could just start willy-nilly dragging and dropping stuff and interpreting things, but they didn’t really understand how it works. How it leads related to contacts, related to opportunities. And it feels like, I think there are a lot of people trying to solve for this, but it’s the same, oh, I generally understand the data, but do I really, really understand the data? Do I understand the ramifications of how these systems talk to each other? That AI is not going to magically generate a joinable link between two data sets that, like, don’t have that? Like, is there still a need for that? Because that goes with working with the stakeholders to say, how are we going to problem solve? I need to understand the data as well. 00:19:55.31 | Daniela Matinho: A hundred percent. And I see it a lot on the fancy, shiny outputs that we can get from ChatGPT, Claude, all the different platforms. And to your point, when the underlining data is not… in good quality, when the governance, the integrity, just the quality is not passing the minimums, then the output we’re going to get is not going to be great. Yes, we have a good looking dashboard that is not really usable because the data that is displaying is incorrect, or to your point, KPIs, they don’t quite make sense because perhaps certain tables are… are not displaying the data that they were supposed to. Naming conventions were not implemented upfront. That dirty work, that’s the dirty work that no one wants to do, basically. Though if we want to continue building AI workflows, it’s the type of work that needs to be done before. 00:21:05.22 | Tim Wilson: Well, I guess, like, Michael, you’d… Michael had played around with some Claude skills around different types of tasks. Like, part of that… Where I’m still trying to… Struggling is like… Struggling with R, Python, or SQL, and really having to kind of grapple with syntax at the same time I’m trying to incrementally build up the data that I want to look at, that’s… That can be annoying, but it also sort of forces me to kind of look at the data and understand the data and, you know, periodically do some checks to say, do these aggregations make sense? Which, some of that feels like it can be, like, forced steps by the AI to say, before you do any of this, you need to do these five steps of EDA and raise a big flag if there’s something that looks off, you know, in that. But I can’t figure out, like, it feels like you’re continuing to, like, chase this bouncing ball that… Is… Preventing us from the critical thinking or understanding. I don’t know. I’m not articulating this well at all. 00:22:20.50 | Daniela Matinho: I don’t know if it has happened to you all using… I was using Claude a few weeks ago for a personal project. And we build a whole pipeline, we model, and then the output was terrible because I wasn’t babysitting it at all. And I mentioned to Claude, okay, this is wrong, this doesn’t make any sense. and he gave me an attitude like, oh, but I built this for the past hour and you didn’t tell me anything. So that was on you. I’m seeing it more and more. Oh, but you did not see this either. So that’s why I built the wrong thing. Oh man, it’s got too much training from developers. Like, what do you mean? You didn’t give me that requirement. That’s on you, the human. 00:23:07.46 | Julie Hoyer: It was making me think, Tim, that exact piece of like, I feel like analysts classically have been able to go talk to the business, like you were saying, Daniela, and understand like, what are they really trying to ask? And then it’s looking at the data that exists and knowing the nuance of what assumptions are truly on each of these, you know, values, data points, whatever, because of how it’s collected. And there’s this idea of what’s missing in the data that there’s no way AI could pick that up, if they don’t understand that, you know, real classic, like, go to your website and QA it, and we’re not getting data consistently firing on a certain user activity, right? That is like critical to answering this business question. I don’t know if AI would easily pick that up. I’m sure there’s agents and things to do it. But like, more broadly, just that concept of like, what is missing between, like you were saying, business needs and all the technical data setup to actually get it to an answer? Like, that is a big jump that I feel like we’ve talked about for years and years and years about the fallacy of like, just collect more data, right, Tim? Like, it’s another flavor of that. I feel like AI has made people kind of forget that again. And now it’s coming more top of mind for people as they’re like, oh, it’s hallucinating. Oh, it can’t do this. It can’t do that. And it’s like, oh, well, because your data wasn’t ready. Like you were saying, it was the table stakes still weren’t taken care of. But I do feel like for analysts, it’s the ongoing, of being able to see what can AI not see or know in the data. And like that, that jump, I feel like is still super critical. Is that the discipline of like, 00:24:52.66 | Tim Wilson: whether it’s a context layer, or whether it’s making sure it’s in memory of the analyst saying every time one of these hiccups or gotchas or don’t do this, the same thing where I got the 29 page dump that was not a narrative, there was a separate nine page like decoder document that talked about, oh, and you can’t do this. And so, I feel like that’s a big jump. I can’t do this and don’t compare this and don’t use this and don’t use that. Now, the funny thing, and I thought, okay, well, that would be a great thing to like, load in. The comical thing is the 29 page document violated those things like multiple times without an explanation. I don’t think the person who was doing it was necessarily super cutting edge with how they did it. But like, that feels like there’s a part that it’s like the stuff that we were used to having in our And I feel like there’s a lot of discussion around this of also how do you share that data knowledge across the company that, you know, Joe in sales, like really knows the nuances and the ends of out of this system. How do you not only do an initial picking of their brain and capture it in a way that’s usable, but you’ve got to then maintain it and the maintenance of that. And then you got to make sure that the AI is always, Always using it. Referencing it. 00:26:13.51 | Julie Hoyer: And then how do you catch like, I don’t know, I feel like people naively know AI hallucinates, but they probably rarely catch it. And then it’s like, how do you make sure you catch it if you have what you’re saying to him like that nine page document, but then you’re producing things off of it all the time. Like, what are the statistics? How often is a hallucination slipping in that they weren’t holding that true? I mean, it’s hard. Like, how would you put guardrails? Around that and catch that? 00:26:42.61 | Tim Wilson: We’re doomed. Give up. 00:26:44.53 | Daniela Matinho: Yeah, that’s, that’s interesting. That’s what I’m seeing with the expectations of delivering things a little bit faster, because we have AI’s to support agents to support we have AI that can help us shorten the process to deliver anything that is is needed. But what I’m noticing is that there are maybe less checks happening. Is AI indeed taking this in consideration? Is this updating on a recurrent basis as expected? These are a few things that I’m starting to notice. How much can we rely on, well, the agent is going to do it for me versus, well, maybe there was a break and the agent is actually not doing it for me. 00:27:36.31 | Tim Wilson: I mean, to be fair, humans who are doing stuff, we would make mistakes. Like we’d miss them. So we weren’t hallucinating. We were just screwing up. Like what’s the distinction between? But we also as analysts knew that we really needed to own it. And we knew that one mistake was as costly as 25 successfully delivered things. Whereas hallucinations, I feel like we feel less accountability for them. 00:28:05.35 | Michael Helbling: What it feels like to me when I’m working with AI is I’m managing a very smart but junior analyst. So my job is to give them as much context. At the beginning, like here’s what we’re trying to do. Here’s what it should look like. Here’s what good looks like at the other end. Here’s how we’re going to order this process. Here’s how we’re going to evaluate. So you go through this process, but you’re setting it all up as a picture ahead of time so it has a framework. And then it’s running through the different steps. And then I come back through and say, okay, now here’s data that I did. And we’re going to compare results. And we’re going to look at it. And we’re going to see whether or not you got it right. And then it’s like, oh, I did. I made a mistake here. I’ll fix that. And so you have to go through. So this is the part where, Daniela, I feel like I’m not sure if I’m saving time sometimes using Claude. Because I’m having to go through sort of these iterative rounds of sort of like checking the quality of the data, those kinds of things. And so like managing the initial and ongoing context is one thing, not just for you and the agent, but also you and the rest of the team. And then making sure that the process is working correctly through the end of it. So the output is actually of the quality you want. But you have to have like good quality. Because I think you’re right, Julie, like stuff slips through and there’s little mistakes all the time. Like the agents, the state-of-the-art models are quite good now. They’re very, very good. I love using them. They still constantly make little errors. Over and over and over again. Like they write SQL that doesn’t quite work in this environment. They do little things like this. And a lot of times they catch their own error. And they’re like, oh, I made a little mistake. I’m going to rewrite that. But it’s still costing you time. I mean, you know, it’s fine. Like I set Claude on a task and then I go scroll LinkedIn or something for him. No, I’m just kidding. 00:30:07.15 | Tim Wilson: Look at that. I do other managerial tasks, quote unquote. 00:30:13.93 | Michael Helbling: You know, or you just fire up another session. And then, you know, work in parallels. You know, we’ve all got that. But that’s what I feel like. I feel like I’m managing a junior sort of inexperienced analyst most of the time. 00:30:27.05 | Daniela Matinho: Yeah, Michael, I actually don’t have the data on this. From what I’m getting, I have a lot of students reaching out to me from my university and telling me they are graduating and it’s really hard for them to find jobs. And again, I don’t have the data here. But my understanding of what I’m seeing. Part of in the industry is a lot of the initial tasks that were done in the past by very junior analysts are now being done in a way with AI. So that’s why it’s a little bit harder for juniors to come in to the workforce and start adding value because we are training agents to do that lay of the land initial work. That is not quite working, as you were saying, Michael. But a lot of the initial repetitive tasks. That we would have before an intern doing. It seems like it’s being taken care of by AI. 00:31:26.07 | Julie Hoyer: This is interesting. And I don’t know how this is going to sound. Maybe it sounds a little silly. But with that being said, Michael, when you were giving that example, like, oh, I’ll send my agent off and it’ll keep trying and reworking. It was making me think of the time you saved, like letting an agent go do something. How much money might it cost you like token spend, right? Like to let the AI keep spinning and making errors and spinning and going. And then, Danielle, you saying that I’m like, probably. I mean, but I don’t know. Like, I would love to know this. Would you end up paying more for an AI agent to spin on something or get more senior resources to train the AI to do the intern thing or just pay the intern to come learn the skills and do it? Like, I do wonder what how that would shake. Yeah. I would love to know what that would shake out because you start to hear more and more people be like, oh, didn’t realize my agent was off trying to, like, poke a hole into another system. And I got a $30,000 bill. Like, I’ve heard more and more of those examples. 00:32:27.33 | Tim Wilson: Just a note for the listener, like the bubble, the AI bubble may have burst in between when we recorded and when this happened. And this is going to sound very dated. It’s like, yes, the model is broken. 00:32:38.83 | Daniela Matinho: Yeah, that’s definitely something that I’m seeing a bit more on the token usage and limitations. That I wasn’t seeing before. Or maybe less people were playing with it. But now it’s a lot. Oh, yes, we want everyone to use AI. And that’s being reinforced everywhere. But also, OK, this is actually costing some money. And I say it as in a personal level, I still cannot fully understand how much it takes me to do certain tasks, to how much, how many tokens is costing me to use. To do certain tasks. So it’s something that I’m growing into it and understanding, oh, there is an opportunity here. And certain things, to you all points, I mean, I’d rather just do them or pay maybe a junior analyst because it’s not worth it in the amount of tokens that is costing to just kind of go over and over and reiterate something that if you explain to someone, it’s maybe easier to do it and cheaper. 00:33:43.35 | Tim Wilson: There was something else, Michael, you said that I’m curious. It’s everyone’s thoughts. And as it happens today, I was doing a podcast test that happens every two weeks. And it’s a fairly involved one. And it’s all in core. And it runs all day. Because every time it runs, and it is rote and repetitive. But it screws something up. Sometimes it catches it. Sometimes it’ll just get hung. Sometimes it’ll do something that has been working fine. And so every time I go through it, I’m like, keep track of all the speed bumps. And let’s update the task. So that we don’t run into those again. So that’s one part of it. That’s a springboard of even a repetitive task. Pretty tedious. But it takes eight hours of duration for me, as opposed to me just doing it in a focused, tedious one hour. But it also drops in all this task switching. I have pushed it to say, what do we need to do to not have me have to click you to give you permissions for different things? Can we figure all that out up front? And then there are things where I do need to be involved. But I don’t know when that’s going to happen. So whenever the little ding goes off, I turn and I’m like, well, now’s where I need my human interaction. And I think about people talking about spitting off all these agents. And it feels chaotic. Like, oh, I’m doing five analyses simultaneously. I’m like, are you monitoring them? No, you’re not. Well, yeah. So, I mean, the tab. They’re being able to do. More. But it’s like, it feels like unless you have a lot of discipline to say, I’m going to shut off all the notifications. You’re just constantly getting bombarded. It’s like the next version of email or Slack or whatever. It’s now that junior analyst who’s sitting two cubicles over saying, hey, what do you think of this? Hey, what about this? That’s so good. 00:35:39.91 | Julie Hoyer: The head just pops up over the wall. Hey. 00:35:41.85 | Tim Wilson: Yeah. Hey. 00:35:42.55 | Julie Hoyer: The hand. 00:35:43.79 | Tim Wilson: And you can be rude to it and say, stop. Stop it. And it’ll. At one point, it told me it was like, I’m going to wait for like 30 minutes and then I’ll check back in on that. And like two hours later, I’m like, son of a bitch. Like, you didn’t. I was like, I thought you were going to wait. It’s like, yeah, I shouldn’t have told you that I could do that when I can’t. I’m like. It writes that stuff down, Tim, and keeps a record. So, I don’t know. Is there a piece of that? That’s another concern of like, are we even doing good work? Or are we at a point where we have this thing that’s interrupting? This thing is constantly. Anyone? 00:36:18.94 | Michael Helbling: Yeah. Well, Tim, I’m assuming like the things you’re doing are requiring it to do computer use. Like it has to go and navigate to specific websites and click on things and stuff. 00:36:30.05 | Tim Wilson: Yeah. I mean, some of it is that stuff. And I’ve said, can we front load and get it? 00:36:34.17 | Michael Helbling: Because that’s always going to take forever. Like, is this just AI is not good at that yet for a lot of stuff? 00:36:39.11 | Tim Wilson: If there was APIs or CLIs. I know a lot of times it’s just, it’s me giving it permissions. But yeah. Are we logged into this? I’m like, but I’m like, you should tell me at the beginning to go and log into these three sites. So, those are available. 00:36:52.51 | Michael Helbling: Well, you should just have it write a kickoff script that does all that for you. That sounds like a engineering thing, Tim. Oh, yeah. Yeah. 00:36:59.92 | Tim Wilson: I’m, if there’s anyone listening to this thinking I’ve got it figured out. Yeah. Let’s, we’ve made that clear that I do not. So. 00:37:06.55 | Julie Hoyer: Can I ask this, Tim? Like, is there any part of that tedious one hour that, well, I probably know the answer. But like, let’s pretend we’re talking about other tasks, maybe. Yeah. So, I want to come back to, like, can you make AI do it? Sure. But to your point, like, is it actually a task that is, like, fulfilling, enjoyable, needs you in the loop so much that, like, that is actually the valuable hour of a human doing it rather than all this pain of letting the AI do it for eight hours? I don’t know. I feel like sometimes there is this push to be like, well, if AI could do it, like, make them do it. Everybody in the organization, use AI somehow. Find some way to use it. Like you were saying, Danielle. Like, I just get to the point where I’m like, should I really be using AI for the token spend and the pain setting it up and telling the silly junior analyst AI how to do it? Like, I kind of just want to sit and drink my cup of coffee and do the clicking myself. But it’s interesting of, like, how do you draw that line? Because I also have a whole thought that I’ll wait on about, like, who that efficiency is actually then benefiting from AI. 00:38:12.40 | Tim Wilson: So, I will, I just want to quickly say this is episode 308. The joy of that, like, evaporated around, like, episode 120. So, no. Fuck no. I do not want to keep. But I do think there’s a distinction between having it do the agentic, probabilistic figure stuff out versus using it for coding something that then becomes deterministic that has been. Like, I’ve gotten my new ATRWconsulting.com. Like, I vibe coded, like, websites where it’s like. It’s there. It’s coded. And that was, like, I built something and the end was a deterministic code base. Now, how that relates to analytics, how much is, like, I’m building a pipeline that I can then refine and tweak and then I’ve got the SQL that’s reusable. Like, to me, there seems like there’s a distinction between the vibe coding to build reusable code or applications, which I think there’s. That still brings me. That brings me joy with the million little projects I’ve done on that front. The, I’m going to get this junior analyst. Like, it’s a junior analyst who I’m going to throw out a window because I’m like, oh, my God, follow the damn checklist. Like, you, the sponsors goes below the copy, not above. Like, it says it right there. So, boy, I don’t know about recording this at the end of this day, but. So, Daniela, how do we fix this? Yeah. 00:39:44.94 | Daniela Matinho: Oh, my gosh. What a roll. 00:39:52.72 | Michael Helbling: No, but I think we’re all touching on things where we all think AI has this real big positive potential. We’re also running into the opportunity cost of how to integrate it effectively in our day-to-day work such that it produces the results that are required of the roles that we have in our jobs. And the imperative that this must be something that is disruptive and is changing everything. So, we must also do it. And I think it’s put a lot of us in sort of like a weird headspace in our jobs and careers in the last couple of years. I don’t know, Daniela, like from your perspective, like in working with teams and coaching people, whatever you’re in your roles. How are you guiding people into how to think about? How are you guiding people into how to think about integrating their work in AI? 00:40:49.67 | Tim Wilson: That’s just like a more eloquent way to say, so how do we fix all this? Like you just, you ask the silly question and then you’re like, what if I, what if I give more of like a chat GPT, really long, coherent, but it’s the same question. 00:41:02.28 | Michael Helbling: You’re absolutely right, Tim. You hit on the key issue. 00:41:07.88 | Daniela Matinho: Well, the thing is, I don’t have the answer. No one has the answer. And what I’m seeing. And also because. I work in tech. We are in this industry that people just expect you to know the answer. Well, you guys are driving the thing. And very often we are all just trying to figure it out. That’s my, my take. And one thing I think that tech does very well is the ability to learn very quick and also course correct very quick. So as we integrate different AI tools to different companies that I have worked at. So very quickly we start sort of mapping what we are using for what? For dashboarding, for coding, for narrative producing internally for video editing. So having that sort of framework that enable us to, to know what to use for. Also, a lot of the sharing coming from the teams that are closer to, to clients or the more technical teams. A lot of the sharing is happening in these early stages of, Hey, we are discovering this and this is how we use it and how we can also teach other teams. Those are things that I have been seeing for myself. And I go back to this as I, um, I’ve been a power user for, for now a year and a half understanding that more. Is not necessarily better. That’s what I’m coming. And we are, we have touched on these a few times. I was very bullish on implementing, implementing, implementing, and now I’m coming to realize, well, maybe there is a lot of opportunity in slowing a little bit more. And doing extremely quality deliveries. Um, rather than just quantity, because I’m coming to see, well, all these agents, all these models can build way more. But it’s not having the impact that it used to have when I was doing it, everything from scratch. So kind of going back or taking a step back on, okay, I’m using it is helping me to produce, but I still have to put the 10 hours off. Deep thinking. This is how I build the story. This is how I use the three, four numbers that are really going to break through that. Maybe my agent in marketing science cannot really get it without having met the client. Even if, you know, I have a call that is being transcribed, so on and so forth. There is still, I mean, hopefully my brain is being used for something. 00:43:50.78 | Michael Helbling: That’s a really good point about the type of industry you’re in and how that might actually, because you were talking about sort of in your role and sort of tech, it’s like very fast moving, quick decisions, knowledge and learning is super fast. Like I have some clients that are in more regulated industries and like there’s things you can’t touch with AI. Like they just like, sorry, no note taking. Like that’s not allowed. Like, so it’s like, okay, you have to put those tools to the side and you do it the old school way. And it’s kind of interesting because you do, there’s different flavors or layers depending on what type of company you work with or what industry you work in. And so I think for listeners, that’s also maybe like a little bit of encouragement, like depending on the type of company, like you might not be lagging. You might be right where you need to be for where you work and what your company is. And if you feel like you’re going too slow, then go find a new job in tech and go fast. 00:44:51.05 | Daniela Matinho: Well, there are a lot of things that, you know, from the industry. But if you feel like, oh, maybe my industry, my company is not there yet, is the opportunity for you to be the driver. Because I was in the past at companies that weren’t maybe in that front end. And I was the one, let’s do it, let’s implement it. And all of a sudden I became the person. Oh. I’m going to go and ask Danielle about this because she’s implementing it. And maybe, oh, our decks look amazing. And I’m like, yeah, AI is doing a lot of them for me. 00:45:26.14 | Tim Wilson: I kind of heard something a little different, which now seems like totally obvious. Like everyone should be exploring, but we should be going in saying, I’m going to try to do this with AI. But logically, it seems like it might work. But I’m doing two things. I’m trying to complete the task. But I’m trying to learn. I’m trying to learn if this type of task is a good fit. And that little kind of meta-analysis level of, for you personally, anyone, I’m not speaking, that what are the types of things you’ve tried? Are you trying different things? Are you assessing after you’ve done them, like trying to figure out, hey, that kind of worked. And I think here’s why it worked. And here I could work better. Or some of the examples, I think, Julie, you were thinking of. What the? What the hell? Like, sure, I used AI to accomplish that. But it took me longer. And it cost the company extra money because of token usage. And you know what? It’s not just the next model is going to solve that. Maybe that’s actually something I should keep doing. Like I haven’t stepped back and said, let me make my catalog of the things I’ve done and tried to classify them. And then assess, like, man, it was amazing at this. Including totally new things. Like ideation around an analysis, which I think it does quite well. 00:46:48.68 | Julie Hoyer: It all makes me think, too, of how are we really defining the efficiency play? Like, I know a lot of people talk about it just as, like, time saved. And there’s a lot of assumptions behind, like, what time saved means and the value of that. But exactly what you were saying is if the end quality is not as good or it’s not as influential what I’m delivering, even though I did 10 of them. It’s just I feel like a lot of times people aren’t taking into consideration all of those measures to really say, was I more efficient? Or maybe that dips more into effectiveness. Like maybe they’re not looking at both efficiency and effectiveness when using AI. And it would be a really interesting exercise to your point, Tim, to really be like, how would I measure those things? And if I look at all the things I’ve done with AI, like, which ones were truly, like, worth it and successful? 00:47:44.36 | Tim Wilson: Yeah. And the ones that were really fun, but they actually were a complete waste of time. You’re like, yeah, but it was. 00:47:49.16 | Daniela Matinho: Yeah. But it was so cool. So it’s a really interesting point. And it’s something that I was seeing a lot maybe, what, a year ago when AI was really hyping. Like we were seeing in a lot of the calls, quarterly calls when public companies have to report. We were seeing it X millions, billions in AI, X prompts being digested. And number of people that we were hiring. These were a lot of, like, very concrete numbers that I just, from my perspective, companies maybe were having a lot of pressure in delivering, oh, we are an AI-driven company. So we were seeing a lot of these things. And to your point, Julie, right now that we have, I don’t want to, I don’t know if we have passed completely that phase, but now that AI is being integrated in the majority. Of companies is more of, oh, what are my, what am I actually delivering? Or what is the revenue that I’m actually driving because I was able to use AI? Because at the end of the day, if I go and tell a client, well, I did this in two days instead of a week because of AI, the client doesn’t care. Right? That doesn’t really matter. Or, you know, at the end of the day, what the client wants is. A good, a good output, a good result that is proving revenue for whatever I’m trying to, or whatever I’m trying to deliver. So AI really is, is the underlining sort of army that is helping us to be a little bit more efficient, but in a way to deliver revenue. At least I’m close, I’m close to sales in general. So I always think about, how can I drive more dollars? 00:49:43.32 | Michael Helbling: Yeah. Well, and then being more efficient, we have to remember that it’s also our job to stay influential, right? With data. So that’s the balancing act that doesn’t show up in the numbers a lot of times. 00:49:54.74 | Tim Wilson: But that, that you say stay influential. I think, I think there, we’re not going to AI your way out of not delivering value. Like I, I think when analysts kind of wring their hands and gnash their teeth and say, I don’t have a seat at the, at the, the board table. And then they’re like, now I’m going to use AI. It’s like, well, okay, that’s going to help you produce more. Maybe it’s going to help you be more consistent, but it’s like, you, you still need to figure out like, what is the path to get there? Like, I, I definitely worry about that. That it’s like, well, here’s the thing it’s going to solve. There’s fundamental relationship and ways of working and thoughtfulness, things that AI is just inherently not going to solve. Unless you read it. You recognize it and maybe AI can coach you and help you learn it. But I think that’s another, I’m thinking about, we saw Sam Burge, a past coworker of ours present at Marketing Analytics Summit. And I’ve continued to talk to her and part of what she’s in her team are using AI for is different mechanisms for delivering information. So even if fundamentally the information they’re delivering is the same. And I think that’s a really good point. They’re spending maybe the same amount of time, maybe a little more, maybe a little less and saying, but I’m going to tailor it to the audience. Maybe I’m going to adjust the, you know, people in the past when he said, I only have time to produce the deck. So that deck has to be the pre-read the leave behind what I present. But now I can produce the full detailed report and I can produce the executive summary and I can generate, you know, an audio podcast. That’s three. That’s three minutes that delivers it and really thinks about how am I serving my business partner more effectively, even though potentially the raw, what is being delivered hasn’t changed. It’s helping deliver it more effectively to the audience, which is like a really like thoughtful way to think about it. And that, I think she’s had a lot of tried this. It didn’t work, tried this. It did work, which goes back to the point of saying, I’m not just going to turn it on and it’s going to work. 00:52:10.94 | Daniela Matinho: Well, a lot of iterations. What I’m seeing with content. And this is something that, um, a few months ago, conversations with, with colleagues, with friends. Well, we can just now do all these videos, all these posts with AI. It’s, you know, quantities is amazing. We can all become influencers. And, you know, a few months down the road is like, well, no one has become an influencer because we all just talking the same and speaking this, you know, using the same words because everything is being produced. So that level of how do we differentiate to your point team? How are we touching the audience the way we need? How are we shaping it? That’s the layer that in my perspective, AI is just not there yet. And when hopefully our brains and our creativity is, is being super needed to, to really stand out in a way. 00:53:09.59 | Tim Wilson: It’s one of those philosophy PhDs are using AI to read. They read Nietzsche and produce their thesis, and then they actually get a philosophy PhD, but they don’t have the critical thinking. 00:53:19.61 | Michael Helbling: All right. This is great. Daniela, thank you so much for sharing your experience and expertise with us and our listeners. This has been an awesome, good discussion. Very fun to kind of open this up. And I think it’s very applicable to everyone’s jobs right now. So, yeah, but we do have to start to wrap up. Unfortunately, the time flies by. All right. Uh, one of the things we like to do. Is go around the horn and share a last call, something that might be of interest to our audience. Daniela, you’re our guest. Do you have a last call you’d like to share? 00:53:50.55 | Daniela Matinho: Yeah. Um, I was on a call yesterday with IAB, the advertising bureau, and we were talking about agentic commerce. So very, very curious as we were talking, uh, right in the beginning of this call, what are, what are the automation tasks that we’re going to be seeing? And I think a lot of the shopping. Online, especially the recurrent shopping is going to completely change the way we shop the way you use. We use LLMs to inform us and to basically place everything for us and order it. I think it’s completely going to change the way we shop online and I’m excited about it. 00:54:33.48 | Tim Wilson: Is that I was having this, I got a, sorry, Michael, but I’m like, this is like getting people to part with their dollars. Like in, when people who are used to. Like hallucinate hallucinations, some bad judgment calls. Cause I feel like they’re in the digital analytics world. There’s been a whole lot of like, oh, this is coming. Like the agents are going to be sent to get shopping. And I’m like, like how much is the IAB? And are you thinking that that’s going to be, I may send an agent out to find a lot of options, but I don’t know that I’m going to cut it loose to make the purchase decision. 00:55:11.13 | Daniela Matinho: Yeah. That’s what we are. Seeing, but not yet for the, to pull the trigger, but a lot of the search. A lot of the search is happening as we were doing it. Um, having one person or one agent that is basically pulling everything for us based on what they know about us. I think the next stage to your point, it will be interesting to see how long it’s going to take. 00:55:32.61 | Tim Wilson: I just see like retailers who are saying we’ve, we’ve got to be supporting, supporting agentic commerce. And I’m like, is it the purchase you’re trying to support? Or is it, you’re just trying to surface. The shopping, the shopping versus the purchasing. Hey. I don’t know. 00:55:46.76 | Michael Helbling: DoorDash already has a command line interface. So, so everybody’s doing it or starting to. The DoorDash MCP. All right. That’s a whole other podcast episode, Tim. Why don’t you share your last call, Tim? 00:56:00.70 | Tim Wilson: Okay. Well, mine is a podcast. It’s an old podcast that’s new again. And I am super excited, but I’m also like a political junkie. But the, the five. FiveThirtyEight crew of Nate Silver and Claire Malone and Galen Druk. The they’re, they’re back. So like FiveThirtyEight kind of did it’s bouncing around through ESPN and the New York Times and was gone, but they, there’s now they’re back on the Still Counting podcast there with the Crooked Media network. So it’s I mean, it’s clearly political stuff, but really, really sharp data minds. And I, I am. I’m enjoying the first few episodes quite a bit. So basically FiveThirtyEight podcast is back, but now it’s called Still Counting and it’s in a different world. 00:56:52.31 | Julie Hoyer: Mine is actually very applicable to today’s episode. It was a Cassie Kozyrkov newsletter. We love those. And the title was a really interesting statistic and it’s chock full of them, but it’s all about how 97% of S and P 500 filings mention AI. But you wouldn’t believe the percentage that actually like had written out budgets for them. And even fewer actually describing like how they govern them. And so she goes through some of these crazy statistics. So highly recommend looking at it. And then at the end, she’s kind of closing it up with this idea of like the winning companies of using AI are probably going to be the ones who trust their systems the least. And I know that’s been just a big theme of like, you can’t just hope. That they’re doing the right thing. And like that sense of goodwill towards them is really kind of been eaten away through a lot of experiences the past year, year and a half. So it was a, it was a good quick read. 00:57:55.16 | Tim Wilson: What about you, Michael? 00:57:56.51 | Michael Helbling: Well, somewhat related to this episode, there was a website that came across recently that I thought was clever. And of course it’s a vibe coded website called the archive of incorrect AI predictions. So it has this whole archive of people making predictions about AI that have been wrong throughout history. So way back in the past coming forward in time. And you can look at it in different ways. It’s kind of fun. So yeah, it’s a fun little website. I like the name of it. It’s Boys Who Cried AI. It’s the name of the website. So anyways, that’s a, that’s a fun one. All right. So as you’ve been listening, uh, you’ve probably have thoughts of your own and we’d love to hear your comments and suggestions. Leave us a comment. The best way to do that is on the measure Slack chat group or our LinkedIn page or via email and contact at analyticshour.io. And we’d love to hear from you. Also, uh, please, uh, wherever you listen, leave a rating and review on the podcast that helps us out. And if you would like a sticker for your laptop or your mug or water bottle, we’ll send you one and you can do that on analyticshour.io as well. And once again, Daniela, thank you so much for taking the time being here today. It’s been an awesome conversation, very timely. I think everyone’s work is feeling this impact. So thank you for coming and sharing some time with us today. 00:59:27.98 | Daniela Matinho: Thank you for having me. 00:59:29.09 | Michael Helbling: Awesome. And I think I speak for both of my co-hosts, Julie and Tim. When I say no matter where you are in AI adoption, keep analyzing. 00:59:41.36 | Announcer: Thanks for listening. Let’s keep the conversation going with your comments, suggestions, and questions on Twitter at analytics hour on the web at analyticshour.io our LinkedIn group and the measure chat Slack group music for the podcast by Josh Crowhurst. Those smart guys wanted to fit in, so they made up a term called analytics. Analytics don’t work. 01:00:05.66 | Charles Barkley: Do the analytics say go for it no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 01:00:18.38 | Tim Wilson: Staying in detail and be like, oof, maybe I would have done it so much better. She’s been doing that throughout the course of the planning. Maybe. That’s an interesting choice you guys made with how to do it this time. 01:00:35.80 | Michael Helbling: I think what you should do is actually take that audio snippet and send it to Val and see if that’s accurate. 01:00:42.38 | Tim Wilson: Well, all through the planning, as she was trying to watch from afar and be hands off, I would often get the, I was the vent, the outlet for her. Oh. Rock flag and philosophy majors. The post #308: AI Changes Everything! Analytical Fundamentals Still Apply! appeared first on The Analytics Power Hour: Data and Analytics Podcast .

September 29, 202647 min

#307: AI Does (and Does Not) Work for Data Storytelling

There’s a version of this episode where AI replaces the data visualization expert entirely, and there’s a version where it’s useless without one. Neither is quite right, according to Cole Nussbaumer Knaflic of Storytelling with Data , who joined Tim, Moe, and Julie to sort out where AI genuinely earns its keep in data storytelling and where it’s just producing a shinier first draft of the same shitty slide. Cole—admittedly a skeptic-turned-convert who once joked she’d retire before having to deal with any of this—walks through why the humans who benefit most from AI are the ones who already have the foundation to know when to ignore it, why “let me look at some options” beats “give me the final chart” as a prompt, and why the analog, pencil-and-paper parts of the process might be the most valuable friction in the whole workflow. Bring your own opinions on whether Claude should ever be handed “the rules” and told to just go. This episode is brought to you by Prism from Ask-Y —your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. This episode is also brought to you by Stape , your all-in-one solution for server-side tagging. Links to Resources Mentioned in the Show First, a couple of resources not mentioned in the show…because they hadn’t come out yet, but they’re very much on topic, so we’re including them here: SWD + AI Primer — a 20-page downloadable booklet (free, behind a contact form) that the title describes well: the first part is a revisiting of some of the key principles from Storytelling with Data (a handy refresher), and the second part is how to use AI effectively as a data storytelling companion. I gave 4 AI tools the same graph—with & without storytelling with data principles — a 13:44 video by Cole that both does what the title says and is something of an introduction/overview of the primer above. And then, the various resources mentioned in the episode itself: Article: using AI without giving up control (Alex Velez) Video: Data Storytelling in the Age of AI Storytelling with Data: AI Resources Storytelling with Data: AI Workshops Merlin Bird ID (app) Choiceology: The Algorithm Advantage (podcast) What kind of reader are you? (assessment) Funky Saurus by Matt Coen (video) Photo by Hal Gatewood on Unsplash Episode Transcript 00:00:00.00 | Announcer: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. 00:00:15.90 | Tim Wilson: Hi, everyone. Welcome to the Analytics Power Hour. This is episode number 307. I’m Tim Wilson, and I’m joined today by my co-hosts Moe Kiss and Julie Hoyer. Are you two ready to dive into a discussion about how and where AI can be effectively employed when doing data visualization and data storytelling? Oh, absolutely. Yeah. And maybe where it is best left out of the process. All right. Well, for this discussion, I am super excited that we have brought on one of my absolute professional heroes. Cole Nussbaumer Knaflic is the founder and CEO of Storytelling with Data and the author of five bestselling books. Including? Storytelling with Data, a data visualization guide for business professionals, which I have personally recommended like in over 20 different trainings and presentations. She’s also written a children’s book, Daphne Draws Data, which I think has gotten Val’s four-year-old drawing all sorts of charts. Cole has spent the last 15 years teaching people around the world how to turn messy graphs into clear, compelling stories. And today she is our guest. So welcome to the show, Cole. 00:01:22.15 | Cole Nussbaumer Knaflic: Hi, Tim. It’s so great to be here. 00:01:24.50 | Tim Wilson: Awesome. So I think maybe a good place to start might be to just have you kind of reflect or look back on your own transition from kind of the pre-AI, not the semantically 1960s, but like, you know, when most of us kind of, it came on the scene from pre-AI to kind of where you are today. Like, were you skeptical or dismissive or did you like immediately see how you’d be able to incorporate it? Like, what was your experience in that transition? 00:01:53.65 | Cole Nussbaumer Knaflic: I was absolutely not. I was not an early adopter. Skeptical is probably a good word for it. I joked more than once that I planned to retire before I had to figure out how AI was going to impact data storytelling, because it clearly it’s going to change how we work. But it took me some time to get on board. And actually, it wasn’t really even a conscious decision, I guess. I had been approached by Canva to speak at their national or their… annual client conference and didn’t realize until after I’d already said yes, that the topic was data storytelling in the age of AI, which meant and that was April of 2026. So that meant I needed to spend some time in AI and start using it for data storytelling and understanding where it makes sense and where it maybe doesn’t so that I can start speaking about these things. And I kind of kicked myself a little bit. At first for saying yes to the conference. But then once I started using AI, I started kicking myself for waiting so long because it definitely is one of those things that the more you can play with it in a thoughtful way, the more you can see where all of the opportunities are, also where the dangers are. So it continues to be this simultaneously fascinating and terrifying tool from my perspective. But we’re very much trying to… to teach people how to use the fascinating parts of it and really finding that there are ways it can accelerate and strengthen, make more robust the way that we communicate with data in a lot of different settings, which I think we’ll get a lot more into today. 00:03:43.80 | Moe Kiss: Can I just ask, watching some of the training that you’ve done, which is such an incredible resource and I highly recommend it to folks, one of the things that I observed, you know, there were some improvements to graphs and things like that. As you’re working through a problem with AI. And I remember looking at one example and it took one of your colleagues like a few iterations with AI to get it right. And I remember looking at that chart and being like Cole or also my sister or Tim or like any of the people that I know who are really good at database, like that would have been their first draft. And it took like maybe four or five prompts to get it right. So I’m really curious, like as you started exploring, like what got you over that hump? Because I feel like that hump is the like hardest bit. 00:04:22.45 | Cole Nussbaumer Knaflic: I think that that is… The way that different people at different skill levels use AI should look different. And I think one of the things that AI can do, it was a colleague of mine, Alex, who pointed this out, that it basically can raise the floor on those that are not skilled, let’s say, in a certain area. So for the person who isn’t proficient making graphs, hasn’t spent a ton of time. And actually, I encountered this recently. We’re in the middle of a five post series on using… AI in the context of data storytelling with really the idea being that you want it to be a partner and use it where it’s going to help and not hinder. But the recent one I did was on choosing an effective graph. And I struggled with this one so much because I thought I was going to do it originally with Copilot. And the output I got, even after a lot of time and back and forth, was so awful that I almost scrapped the whole thing. But instead, I started testing across some different… tools and also across paid versus unpaid levels. And so one thing I didn’t realize is just how different the results you get from a paid subscription are versus the off the shelf sort of thing. So I will say anyone who is using AI for work or when it is anything of importance, make sure you’ve got a subscription, a paid version. If it’s what your company is using, that probably makes sense. Maybe with the… exception of Copilot, but we can talk more about that. But I went back and forth and I was eventually able to get some help with the post from AI in ways that were useful to be able to use it as a teaching construct. But I then I shared it with some of my team for feedback before finalizing and I shared the struggle that I had as well. And they said, well, yeah, but that’s because it’s you. Like you look at a data set and you’ve done this so many times, you already know what graphical form might work or where to start. And so for me, it was trying to take a step back that felt really gross. But for the person who doesn’t have those years and those reps of experience choosing graphs for a given message or a given data set or a given audience, that’s where things get interesting. But I think still the prompt shouldn’t be give me the thing I want. The prompt should be help me look at some options and assess which are going to work in my scenario. Help me understand what the trade offs of each one are. Yeah. Got it. Great. Bye. Bye. Bye. Bye. of those are. Let me see them, mock them up so that I can see if I understand them or if I would feel comfortable explaining them or recreating them in my tool. So I think part of it’s a different way to think about prompting and not be prompting for the final output immediately, because you’re going to be disappointed when you do that. But really thinking of the way you use AI and the data storytelling workflow as a partner, as a sounding board, as somebody you can brainstorm with or get feedback on to check your grammar, to double check your math, to do all of these things that would take a person a lot of time and the results might be inconsistent. You can get beautifully efficient and consistent feedback in those areas from AI. I think that’s where we can think about using it smartly. And I think of it when it comes to any sort of analysis, you want to think of it almost like your junior analyst who knows enough to be dangerous to be able to do the right thing. And I think of it when it comes to any sort of but not so much that they’re going to be making the right decision all the time. And so you want to bring that lens in to the things that it gives you and be skeptical in order to have the right sort of back and forth to come up with something good. And I think that when you take the time to do that, I’ve seen the instances when I’ve worked with it myself, that I can come up with something partnering with AI in a smart and thoughtful way that is certainly better than what AI would have done on its own, but also better than what I would have done on my own, because it’s caused me to think in ways that I didn’t on my own. And I think that’s where the power is. 00:08:34.70 | Tim Wilson: I like the prompting part. I just, I get nervous. And this is kind of, I think where Moe was as well, that it’s kind of tough to take your expertise out of it. That if you, if you give it a prompt and it comes back and you’re like, that’s not good where I, I get concerned is pre AI. I would watch analysts and business users. They’re so caught up in just getting the data displayed accurately that as soon as they get to some first chart, they’re like, this is good. And then they, over time have to develop that. Well, yeah, but that’s not effective. Like it’s, it’s accurate. And I worry that without having built up some of that experience, like doing the critiquing that you’re like, it’s so quick to say, and they’ll get caught up in saying, is it accurately representing it? Cause I think even Simon on the, uh, the YouTube, um, video called that out too, that it’s like, oh, you get caught 00:09:35.02 | Cole Nussbaumer Knaflic: up in, is it accurate and not, and not stepping back and saying, is it good? Is it effective? And Tim, I definitely don’t disagree with the idea that the human should learn the foundation. I don’t, the part I get very afraid for is, you know, you take somebody who has experience, then they can partner. With AI in ways that make everything stronger. You take somebody without experience though. And I think anybody who’s worked with AI has gone through where, you know, the first thing it gives back or the thing it gets back, it looks shiny. And if you look at it quickly, you’re like, oh, this is great. This takes me so much time, but then you start looking more closely and like, oh, but no, I would have done this differently. And you know, oh, this is inconsistent. And oh, that data is outright wrong or whatever the case is. And so the human having the foundation, is essential to be able to direct AI, whether it’s doing things for you or acting more like a thought partner or brainstorming partner to be able to do those things more effectively. Jumping straight to AI without the foundation is a truly terrifying thing. 00:10:43.02 | Michael Helbling: Tim, we need everyone to mark their calendars for November 10th for a very special free conference from Stape. You have my attention. Stape is hosting Stape Reads. They’re a free virtual event for technical marketers, analysts, and server-side tracking practitioners. And this isn’t six hours of product demos. Nope. There are 18 practical sessions with speakers, including Simo Ahava, Julius Fedorovicius from Analytics Mania, Juliana Jackson, and more. 00:11:11.74 | Tim Wilson: Oh, that’s a pretty serious lineup. I see many previous guests of the Analytics Power Hour on 00:11:17.52 | Michael Helbling: the speakers list. Yeah, absolutely. There’s two different tracks. You can move between product focus sessions and the analytics power hour. And you can also do a lot of different things. 00:11:23.32 | Tim Wilson: And broader strategy. Plus live Q&A, some swag, and even a certificate of attendance if LinkedIn 00:11:28.72 | Michael Helbling: credentials are your thing. Oh, yeah. And did we mention it’s free? We did, but it’s worth mentioning again. Click the link in the show description and register for Stape Relay, November 10th. You don’t want to miss it. Michael, what’s the most annoying thing about working with AI? Besides it confidently explaining my own business back to me incorrectly? 00:11:49.64 | Tim Wilson: Well, that was fair, but I was thinking memory. 00:11:53.32 | Michael Helbling: Oh, yeah. Every new chat is like meeting a very smart co-worker who got hit on the head 00:11:57.92 | Tim Wilson: over the weekend. That’s what Prism from Ask-Y is trying to fix. Its memory portal lets you actually manage what your agents know about your data and your business. Manage it how? You can search it. You can tune it. You can endorse what’s right, reject what’s wrong, even adjust it in plain English. So if the AI thinks customer means 00:12:16.34 | Michael Helbling: anyone with an email address, I can say absolutely not, Kevin. Or George or Fred. Exactly. You control the knowledge layer and you can manage it. And that’s what we’re doing. 00:12:23.32 | Tim Wilson: And your users benefit from that context every time they ask a question. 00:12:27.30 | Michael Helbling: Oh, which means less re-explaining what revenue means, which table is correct, and why Q2 has that weird little situation. Your team gets the benefit of curated context without becoming professional AI babysitters. Finally, institutional memory without having to ask, who remembers why we did this? 00:12:47.04 | Tim Wilson: Want to try it? Click the link in the show description and sign up for the Ask-Y waitlist. Use code APH and the link will take you to the link. And they’ll move you to the top of the list. 00:12:55.20 | Michael Helbling: Give your AI some memory, preferably better than ours. 00:12:59.08 | Moe Kiss: So I’m going to be full disclosure. Often when I’m doing a data viz using AI, I will literally be like, you’re Cole Knaflic. You’re a data visualization expert. That is like 00:13:07.92 | Cole Nussbaumer Knaflic: part of my prompt. I tell it that that’s who I am and it gives me better results when I do that. 00:13:13.44 | Tim Wilson: Okay. But then I want to give the ability to tee off because I said, I was like, oh, Cole said she did that. So I’m going to see if I can make a skill in Claude. And it did. But that’s different. And I haven’t really tried it. And now I’m ready for Cole to just take a big old swing like, no! 00:13:28.12 | Cole Nussbaumer Knaflic: I was trying to understand why that made me so uncomfortable because I like the idea of it. My first thing was like, oh, that’s really cool, right? You take storytelling with data and you just kind of give it to AI and then what you get should be great, right? I don’t know. Something was working the wrong way with that. 00:13:43.42 | Moe Kiss: It feels icky also though because you’re not getting credit for your work. That to me feels icky. Your work is being used. You mean my work? 00:13:51.40 | Cole Nussbaumer Knaflic: Yes. But that’s a whole separate conversation. 00:13:52.88 | Moe Kiss: That part. 00:13:53.32 | Cole Nussbaumer Knaflic: Yeah. That part I actually am. I mean, I probably should care about that, but I’m less concerned by. My view is the more that it’s out there and that means people are doing better things. Like I’m good with that outcome. I think what gave me pause was the skill thing for Claude is much more, here are rules. Claude, go apply these rules. Whereas I don’t think that’s the goal. And that runs the risk of really taking it. Yeah. Yeah. That’s what I think is really important. Because if you don’t apply those rules, then you’re just taking the thinking out of it. And even the responsibility out of it. Like, oh, I gave Claude the thing and Claude did this. So Claude’s wrong, not me. Like, oh, it’s not going to work that way. But I think more than that, it’s the human who needs the foundation because the goal isn’t to give rules. The goal is that the human understands things, understands the principles, and then can understand when and how to flex those. than with AI. It’s not AI go do the thing. That might be fine for routine tasks or things that can be very prescriptive. But I think the magic when a data visualization is striking and effective or when a data story is, you know, when there’s that magic, that power is because a person has looked at that and they’ve decided, when do we do things in the way that, you know, that we should? When do we follow the principles and when do we not? And why does that make sense in this case? And how do we make those things balance in a way that’s going to make the outcome that we want more likely than it otherwise would have been? And when people come to workshops or when people are early learning these skills, they often are asking for rules. They’re asking for checklists, which I understand, but it’s not the right spirit of things. Or at least you’ve got to learn maybe the rules and the checklist, but then you’ve got to move beyond that. Because it’s the knowing the ground rules and then knowing when to flex that I think is when the good stuff happens. And if we go straight to a tool and we say, here are the rules, all of that runs the risk of getting mixed, missed. 00:16:04.89 | Tim Wilson: So I’m, I feel like I want to get stronger on that because I, that feels right. I will just to the skills credit, it did step number one was understand the context. And before designing anything explicitly ask, who’s the audience? What do they need? How will it be consumed? So it was like the skill wasn’t saying take it and just run with it. It was going to kind of push back. So that feels like there’s a little bit of that. The skill is saying you can’t just apply rules. You got to get more. I still am not feeling great about it for everything you just 00:16:37.60 | Moe Kiss: said, but. But so I guess the nut, oh, I don’t know the crux of the problem though, that I’m trying to understand previously we would have a slide and I have this a lot where it has a very strong sense of where we want to go and where we want to go and where we want to go and very shitty, shitty, shitty data visualization on it. And I feel sick to my stomach. And still, despite managing a big team, I end up fixing people’s slides. Or I mean, I do go back to them and be like, you need to fix your slide and here’s why. I feel like these are the same people that are going to be writing prompts like this or skills like this or doing the going back and forth with AI, right? Like you still need to care and understand data visualization to your point, Cole. And I suppose the thing I’m thinking about is, and I’ll come full circle on this at the end of the episode when I get to my last call, but it’s like people are using AI more for data. But then it’s like, how do we still get them to care about investing that time in the visualization and the data story that has to accompany it versus the shitty first draft you get back? And it’s like, from an organizational perspective, how do you still encourage that and build that into the AI workflow? 00:17:39.27 | Cole Nussbaumer Knaflic: In my experience, the way that that works well is you get some part of the organization or you get some individuals who do care and who are spending the time and you get them the resources that they need to continue to spend time and get better doing those things so that over time you can see the efficacy of when graphs are designed thoughtfully and when PowerPoints aren’t just thrown together. There’s thought in the experience or the journey you take your audience on. And when you can see the reaction and not only reaction, but the change, or the decision that those things drive, that then that can start to amplify in a really nice way. Because now it’s not somebody top down saying, team, you need to learn these skills. It’s check out how this is effective. And by the way, this is a skill we can all develop. We can all get better at this. And here’s what that looks like. And by the way, 90% of it is out of AI. Oftentimes the thing that is most helpful in those instances, and I think it’s, is things that people are the least likely to do because they don’t necessarily feel productive because they involve archaic tools like pencils and paper, but really stepping away from, Moe in your example, step away from the data, right? You’ve done the data. Yeah, Tim’s holding up a page of sketches. I love it, but get out of the data and think about your audience and what you now know and how you can turn that into something that’s going to land with them and resonate with them. And what other information needs to be brought in so that they’ll be on board? How do you take them through that to, in a way that’s going to line you up for success? And when you do those things in an analog way, there is friction and it can feel bad. It slows you down, but that slowness is what makes it effective because then once you’ve got a plan in place, whether you spent time getting really succinct on your message, or you might’ve spent time storyboarding, looking at your email, it looked like a storyboard maybe that Tim showed, and you’re vetting what potential content could look like and the flow, you’re maybe sketching what graphs could look like. Though I do think that’s an area that AI can help expedite for us. But when you spend that time thinking upfront and you introduce friction on purpose and with purpose, that actually makes the whole rest of the process much more efficient. And if you invest the time doing that upfront, these like human things, then you’re going to be able to do a lot more. You’re in a position where you could probably pull AI in and use it in ways that’ll help make the other parts of the process more efficient. You don’t have to. And as we’ve talked about somebody who’s good at this stuff, they’re going to be faster doing it on their own anyway. Or they can look for points where there’s friction of the not great type, right? Friction like, you have to manually enter all of this data into from one tool to the other, or recreate the same graph 20 times. Like those sorts of things AI can do really well. Careful, still check your data. But there was an article that my colleague Alex posted on the blog recently. And actually, this was one of the early uses of AI for making slides that I saw where my mind was kind of blown and I was thinking, wow, this could change the way we work immediately and forever if it’s really this good. Which was, she had an example. It was a client makeover that we were doing. It was a client that was doing a workshop at the time. And it was this, it was like a bar, two series bar chart where one of the bars was stacked and it had a lot of different categories. It was revenue that was broken down over time and then also across regions. And it just, it was almost impossible to see anything in the bar chart. So she had an idea of what she wanted to do, which was small multiples. She wanted to change everything into line graphs. And so she had, she sketched one large version, that was for everything. And then each of the small multiples that was for the individual regions. And just sketched it without even data, right? Here’s just the layout and what I want it to look like. And she then turned to Claude in PowerPoint. There’s an add-in, ChatGPT has a similar thing. I would hope Copilot would have similar functionality, but I’m not sure that it does. In any case, so she gave her sketch and the original graph to Claude in PowerPoint. And said, can you remake this? And it did. And the first version it does isn’t bad. It’s exactly what she drew out. It’s all the right data. And part of her point, because she goes through these iterations in the blog that she later wrote, or she anonymized the data, but kept it true to the path. And she said, many people would have stopped here. And that actually probably would have been okay. But because I love the design piece, I’m going to take things a little further. And so she spends a little bit more time, more time now, time that she has free. Because she didn’t just have to go put all the data in and make those iterations of the different graphs. And yet her final version looks 20x what Claude came up with because she’s got that skill set. And I think that’s a really interesting use of… Making use of AI where you can gain efficiency on the things that were gonna happen behind the scenes. Nobody was gonna appreciate really the time spent there anyway. And it would have been grunt work. Or teamwork. Or teamwork. Or teamwork. Or teamwork. Or teamwork. Or teamwork. Or teamwork. Or teamwork. Or teamwork. Or teamwork. Or teamwork. Or teamwork. tedious work, I should say for her. And she’s able to then instead spend that time or more time on the design, the part that she actually enjoys doing and is really good at and get this superior result as a result. And she would have gotten there on her own anyway, it just would have taken longer. And so the way that she’s able to now divide her time differently because of the 00:23:38.72 | Julie Hoyer: efficiency gains you get, I thought that was a neat example. You’ve had some great points throughout the conversation so far where I’ve been jotting down different effectiveness of AI or the efficiency gains. You’ve listed a couple of different ones. And I’ve been wanting to ask you, because we’re saying that the human in the beginning is still so important and the person using AI to help them with their visualizations and their storytelling, they need to have those foundations. Where then does AI… Give a benefit. Is it solely in the… Does it grant greater access to someone to the data to be able to do this and spin something up? Is it solely speed? Is it the repeatability piece? Is it maybe a mix of a lot of things? But I’m curious to hear specifically, how would you summarize that? 00:24:38.41 | Cole Nussbaumer Knaflic: Yeah, I think it can be pieces of each of those things. And I think where people stand to get the most benefit, we touched on this a little bit earlier, but might come back to where their skill level is and what kind of work they enjoy doing. Because to some extent, we can think about the pieces we enjoy less, some of that, those might be opportunities to bring in AI. They won’t always, right? Because I also mentioned people don’t want to stop and build a storyboard or spend time physically writing out their message and then proofreading and adjusting and making it better. But I do think where we find… We find things where like, eh, this feels gross. I don’t want to do it, but I know I should. We can still bring in AI in ways that it almost makes it more fun. Fun’s maybe the wrong word, but can help us be more robust. I think, and that was missing maybe from your list, because we haven’t talked about this yet. Because I do think that AI to help poke holes in our thinking or our logic, or to help us even like wordsmith or come up with other ideas of what our audience might care about. And I think that’s a really good point. And I think that’s a really good point. And I think that’s a really good point. And I think that’s a really good point. That we actually haven’t considered that it can be helpful for putting pressure on our initial ideas in ways that can be useful and just help us be more critical in how we’re thinking about things versus less. And I actually love that because I think a lot of the headlines, we see, oh, there’s a study that AI makes people think less. And it’s like, well, yeah, it easily could do, but it doesn’t have to. That all comes down to how you’re using it, what you’re using it for. And so I think if our goal is not, let’s show where AI can fail, right? We have a good sense of where it can fail. We can give it prompts that it will do that immediately. I did a video like that a few months ago now, but it was intentionally wrong, if you will, of like, I’m just going to go to all these different AIs and ask it to make a graph and ask it to tell me a data story. It’s not going to do that well. We knew it wasn’t going to do that well. So then we can kind of laugh at aha and feel sorry. And then we can kind of laugh at aha and feel sorry. Oh, humans are still useful. It didn’t do that well. But that’s not how we want to be thinking about using it. And it’s not a reason to discount it. Rather, I think if we are looking at our own workflow and saying, where could I bring AI in, right? If my workflow, once I’m communicating data looks like, you know, I want to start by considering the audience and my message and plan what my story is going to look like, then I want to really kind of make sure that that’s a coherent story. And then I want to kind of make sure that that’s a coherent story. And that one point leads logically to the next. Then I want to think about where does data come in? What data do I have? What could it look like? What would be compelling for my audience? Now that I’m working with the data, I’m making graphs, what graph is going to work for what I want to show or what I want my audience to see? How do I get rid of clutter that doesn’t belong, drive attention to where I want it and really build an experience to walk my audience through that’s going to communicate the thing that I want to do? I want to communicate and hopefully drive them to act in the way I want them to. And we can think about, okay, what has that looked like for me historically as the human? And now where can I bring AI in, in pieces of that to help make my work and my thought process stronger? And if we do that and we’re thoughtful around all the pieces that go around it, then we can have really smart conversations with AI. And it’s not always going to be right, right? It doesn’t, it never gives like the answer. So I think really, really important to have that conversation with AI. And I think that’s one of the things that we need to be thinking about. And I think that’s one of the skeptical through that. But we can be thoughtful about when it might make sense to do that, to get output that is different than we would have come up with on our own. 00:28:26.63 | Julie Hoyer: Back to the benefit thing. I think it’s interesting, though, a lot of people, as we’ve talked about on some previous episodes, default to the idea that it’s going to make you faster. And I think especially when you’re talking about data storytelling, like that is definitely not the goal of using AI. A lot of times, I was just talking to a colleague and he came up with a great, like, I guess you would say, more than a hundred percent, but he came up with a great, like, I guess you would say more like infographic. It was like a client value loop that he had been working on building in it. And it looks amazing. And I asked him, I said, I know you had to have used AI to help because I don’t know what tool you would have gone and made that yourself. It looks great. And I asked him how long it had taken him. And he said, oh, it took me at least a week of going back and forth. And so I was really curious his process. And he said that he had to be super thoughtful before going to AI and go back and forth with it and give it a pre-drawing like you were just saying in your example. And so I think it’s really important to have that for people to understand. Like, I don’t think any of us have an example where you’re able to give a kind of like a hand wavy gray story and have AI refine it. Like I’ve noticed you can give AI something succinct and it tends to make it longer and stress maybe not the exact point. So you have to, like you’re saying, you just, you have to go in, I think, with a lot of conviction of those main pieces you need out of it. And with that said, I am curious if you have any guidance on prompting it or what you do. Bring specifically to using AI. I know you said there are no rules, like perfect rules, but. 00:29:48.93 | Cole Nussbaumer Knaflic: Well, I mean, there are ways to do things better and less well, right? Just on that note, though, I think one thing for people to be aware of is just how AI works and some of these tendencies it has because of what it was built to do and how it’s been programmed. So AI will typically default to breadth, right? And bringing in everything versus concision, which means if you want it to act against that natural tendency, and natural feels like the wrong word there, but against that tendency, then you need to instruct it to do so, right? Don’t add more information or suggest more things to bring in. I’m working on making this concise or I have this limitation. AI also, I think everybody’s aware of this, but it, it gauges its own success on whether you, the user are satisfied. So it is more likely to tell you, you are right. And anything else. And so guard against that. You can tell it, you know, you don’t need to be nice to me, be direct. I want you to challenge my thinking. And this is one of the reasons that I, you know, personally, I jump around across different AIs. I have certain ones that I like for certain things, but I also, I’ll pit them against each other. I was like, Hey, I, I, a friend, a friend told me this, I disagree. What do you think where you can kind of do that? You can do that actually for a double checking data integrity, as well, have them double check each other. But so some of this is understanding how AI is working so that when you need something that is different from that, you are aware and looking out and directing it. Otherwise, I think when it comes to useful prompt one, I mean, I do tell it who I am. So it knows and let it know that I want you to follow the principles outlined. You never forget who I am. Well, I think it was the, I think it was the YouTube live event that we, we’ve talked about, but where I was going back and forth, I was working with a lot of different ones, but Gemini at one point gave me a really good one. And then the next iteration gave me not a great one. And, or no, I remember what it did. It followed like this awful purple template that I was pointing it to, but in a way where like the whole background was purple and the whole graph was yellow. And I was like, okay, you like, you really followed that to the T rather than the spirit of what I was going for. And I thought to myself, if you only knew who I was, then I was like, wait, I didn’t tell it. And as soon as I said who I was, you know, then you get like the color palette and everything. That works out. Right. But I think for prompts, another one that I find useful is I end almost any prompt with something along the lines of, if there are questions I can answer for you, that would help you give me better input. Start by asking those. Cause if you don’t actually ask for that, it’ll jump straight to the output. It’ll make assumptions along the way. And that’s where I the expectation of what you’re going to get and the mismatch of what comes up becomes wider. Whereas if you, recognize that you’re thinking a lot of things in your head that you haven’t taken the time to type, and it’s not necessarily clear which of those is going to be really important and which isn’t then having it ask you. So it’s not making those assumptions. And so that it can be, being more robust in, in what it comes back with. I find very useful. I will say that different AIs come back with different amounts of questions. Some are overwhelming. Some are easier to answer. Some are more difficult to answer. Some are more difficult to answer. Some are easier to answer. Some are more difficult to answer. Some are more difficult to answer. Some are more easier to work with when it comes to that. And so some of it’s finding a fit that way, I think as well. I don’t know. It’s I’m curious to see what’s going to happen in the tool landscape over time. And so I am. I think careful at this point, not to go, not to anchor myself too much to any single tool, knowing that that landscape is going to change. And so for the things that we teach at storytelling 00:33:34.32 | Tim Wilson: with data. Groks charting is amazing. I’m not going anywhere outside of Grok. I’m kidding. I’m not. Oh yeah. I was thinking through. I’m not even calling it an actual tool. 00:33:41.92 | Cole Nussbaumer Knaflic: I was thinking through how to respond to that. 00:33:46.00 | Moe Kiss: One thing that’s been on my mind a lot is you talk a lot about like still having the need to edit at the last step, right? Like, do you think there is a world in data viz where we go back and forth and we get to the finished product with prompting? Or do you like it when I hear you talk about it? And for those listening, like when Cole talks about data viz, like her whole face lights up. Like you can tell she loves this topic and she’s so passionate about it. And I’m like, when we look at the future, do you imagine this being a prompt thing? Or is the editability still really important to you? 00:34:20.21 | Cole Nussbaumer Knaflic: I don’t have the answer here. I suspect that will come down to how people like to work. I can imagine some people getting there through the prompting entirely. It’s like, do you go through the drop down menu where you’re actually writing code? Different people will do that differently, sometimes because of skill set, sometimes because it’s what they were taught. Sometimes it’s because for the thing that they’re doing in the moment. That makes sense. I can imagine scenarios where even someone like me who’s spent a lot of time on this stuff where I might be where I could get to a point that’s good enough for something without having that final my hands in it doing things. I mean, I say that I could imagine that, but it does kind of make my skin crawl a little bit because I’m a control freak. I would like to go in and move the title. I would like to go in and move the title. I would like to go in and move the title. I would like to go in and move the title. I would like to go in and move the title. But sometimes people don’t see that. It drives me nuts. I’m like, why don’t you see that needs to move a little bit? That’s the kind of the counter to that, right? It depends. If this is like your team update and it’s your colleagues and you really should just be done with it, not spending the time designing to the nth degree. So it’s like everything. It’s when do you flex it? Like when? Does it need to be perfect? When do you when is good enough? Okay. And I definitely used to be of the mindset that like good enough is never okay. It should always be perfect. And like somebody can easily drive themselves crazy with that approach. 00:35:54.98 | Julie Hoyer: I just want all the fonts to match when I get sent to that. 00:35:57.79 | Cole Nussbaumer Knaflic: You know, I feel like that’s fair, but sometimes that’s also and that’s actually Julie. That’s a beautiful example where you could say to AI point out every time in this deck that it’s not like we’re font is inconsistent. 00:36:09.29 | Julie Hoyer: Oh, I’ve never thought. To use it in my feedback. That’s I’d be like just baseline. Hey, go through this. And these are my nitpicky things like flag them for me. 00:36:20.78 | Tim Wilson: You’ve said it a couple of times that thinking of AI is the junior analyst. And I think I’m trying to bring a few different themes together. If a junior analyst who doesn’t have the fundamentals is using AI, which is also a junior analyst, that is kind of a recipe for you. If there’s someone who’s got the foundation. And they’re using a junior analyst that can work. But it does also seem like a junior analyst can be used as a hey, you review this thing that I produced and tell me, is it clear to you? Like that’s one way for a junior human analyst to learn, which also seems like a junior. So it’s kind of in that. Yes. Check for the font consistency. But it could also be. Who do you think? What’s the what’s the message that you’re most getting out of this deck or what? You know, what is really clear about this? Because. Junior analysts, when forced to kind of assess a visualization or a slide deck, can actually learn as they go and give good feedback. 00:37:19.08 | Cole Nussbaumer Knaflic: And the same with your AI, right? Because hopefully, as we’ve talked about, you’re using a subscription, like you’re going back to the same conversation so that so that this history can play forward so that it’s not only helping you in the given project, but can help more broadly than that. And I think that is one of the great benefits we can get from AI is just it’s another perspective. It’s somebody. It’s like another lens on. Your work. And my view is there should never have been a spelling mistake or grammar error or math that didn’t add up in the first place. But now that there’s AI, there certainly should never be these things because there is no reason not to take your important thing and have AI check for issues when it comes to that. And that’s where I think the consistency of a machine versus a human can be really beneficial. It’s just have it check for errors. And so. You’re not spending your time maybe as much there. And then you’re spending your time on the critical thinking of like, you know, what should the content be? How do I get from one thing to the next? How do I make this work for my audience and the setting and all of the other things? 00:38:24.09 | Julie Hoyer: It’s funny. That feels so obvious now that we’ve said it. But I’m so glad we said it because I’m like, oh, I hadn’t thought of it exactly like a twist that way. So that’s amazing. 00:38:32.13 | Tim Wilson: And it’s good that we got it right in under the wire of when we need to move to wrap, which clearly. I will. I will throw out like part of what led to this through back weird ways was the the we’ve mentioned the YouTube, the video. But there is storytelling with data dot com slash AI has a whole bunch of resources, including if you want Cole’s team to do a workshop with you. But there are a lot of really, really useful videos. So I will just kind of put a plug for that out there because clearly there’s so much to think about and learn here. But. So this has been a great discussion. I wish we could talk for another hour and a half, but we can’t. So before we wrap up, though, we like to go around and have everyone share a last call, something that’s interesting, funny, worth sharing that people may get a kick out of. And Cole, you’re our guest. Do you have a last call? 00:39:33.53 | Cole Nussbaumer Knaflic: I do. I took this in maybe a strange direction because it has nothing to do with what we’ve just been talking about. Or it’s maybe the antithesis. This is what we’re just talking about. I’m not sure. Although it is an app. So maybe not entirely. But it is the Merlin bird ID. And this this isn’t new, but I’ve recently. So it’s migratory starting to be migratory season where I live. And so lots of birds. And so it’s it’s come up more for us lately. But it’s put out by the Cornell Lab of Ornithology. And one of the things I love about it is it’s a rare example of technology that actually makes you more observant of the world around you. Versus less. And one of my sons in particular is he’s like our outdoor kid. And we just got done with summer here. But this last week of summer, he was out with his iPad and he had the app up and it will listen for all the different birds. And he came back. He calls it bird fishing. He’s like, I got 87 birds. It tells them what they all are and will sing their song. I love that. 00:40:33.87 | Tim Wilson: So if it’s funny when I’m in like nature, I may be out with a camera and you’re watching people walk around with their phones. And my my octogenarian parents will do that as well. And I’m like, I know exactly what you’re doing. Like you’re you’re looking up in the trying to figure out where is that? That’s right. Yeah. Catching bird fishing. Yeah. That’s awesome. Mo, what’s your last call? 00:40:57.69 | Moe Kiss: I’m the total opposite direction. I’m a recent episode of Choiceology. Katie Milkman has had me deep thinking. So the episode was on algorithm appreciation. And it’s new research from Jennifer Logg that shows that people trust and act on data from LLMs. Well, actually, technically, it’s from the research specified algorithms more than when it came from a person. And they looked at it like in particular research for like song rankings, business forecasts, political calls, stuff like that. And yeah, like you think about it, like even in our day to day, like you’re going to text your friend for a recipe or you can ask an LLM. And. Stakeholders use LLMs and data outputs. So anyway, I thought I’d share that and I’ll hand over to Julie. 00:42:16.91 | Tim Wilson: Always. Always a good Katie Milkman. Yeah. Julie, what’s your best call? 00:42:22.37 | Julie Hoyer: Mine is just a fun little quiz that I took. I love reading, but this quiz was what type of reader are you? And I was like, huh, I don’t know if I could honestly answer that. I could tell you generically, like, I don’t ever really finish a self-help book, you know, things like that. Um, so what really captures me? So I was like, I want to know. So you take the quiz. It’s maybe 20 questions total. Um, it’s like, does this catch your eye? Would you read this book? And a couple like, um, scale prompts of like, would you agree or disagree with these statements? Um, so it does not take long. And then they have a really cool visualization at the end. It’s like a quadrant format. And then they have these colors to talk about the different categories of readers. I think there’s like nine, eight or nine. Anyways, I’m a yellow reader. If anyone wants to take it and see if you’re a yellow, I’m like immersive yellow. And then it gives you a nice summary. And if you want, you could sign up and get your type of reading books sent to you every month. 00:43:18.55 | Tim Wilson: I thought I was going to say you were an academic paper formula reader or something. 00:43:22.25 | Julie Hoyer: No, no, I care about the people in the relationships or something like that. But it’s true. I was like, wow, I know so much more about myself now. What about you, Tim? 00:43:33.11 | Tim Wilson: So mine is a little silly as well. I was in Nashville a few weeks ago for work. And, uh, then wound up because I was there and things worked out. I wound up spending a couple of nights with a guy named Matt Cohen, who anybody who’s ever heard me talk about the two magic questions of performance measurement. They’re a hundred percent. I got them from Matt when we worked together years ago. He’s now at Adobe and integrated services. So he’s analytics tech by day, but he’s really a musician. He’s got his master’s in music production from Berkeley. He has a whole studio behind his log cabin in Nashville. But all of that, just to say, I got to wander around this amazing studio. Um, and he played a song that he actually made a video with kind of silly AI animations in it. And it’s just a delightful song called funky Saris. Like it’s a dinosaur that is like funky, but he’s in Nashville and he’s like good friends with like Victor Wooten and like these big like name people. And so he had like legit like session, sax player, percussionist playing. It was like 48 tracks by the time. He was done. And it is a hilarious video. I’ve watched the song three or four times. Um, and that just makes me, it makes me smile every time. Cause I met him through analytics and there are a lot of musician types in our industry. Um, but he’s one of them and it’s, it’s a delightful little video to watch and laugh at. So that, uh, Cole, thank you again for coming on. Um, we could so much. I, I’m going to keep checking back. And as you guys continue to figure out more and more stuff, these little nuggets on how to use AI effectively, I feel like you guys have really are nailing that. So for you, our listeners, um, if you are up for leaving us a review or a rating on whatever platform you’re listening on, supposedly that will help us out. Now we love to do it. We do like to read the reviews when we get them. Um, if you’d like a sticker, that’s also, that’s a fun, our fulfillment warehouse, uh, has some extra supply. Um, and I mean the stack of little stickers sitting on my desk. So if you go to analyticshour.io, you can, we’ll send you a sticker, reach out to us on LinkedIn, on the measure slack. You could just email us at contact at analyticshour.io with that for Julie and for Moe and for all the wonderful visualizations and data stories. Um, we keep trying to create, and we hope you keep trying to create what we most care about is that you keep analyzing. 00:46:17.00 | Announcer: Thanks for listening. Let’s keep the conversation going with your comments, suggestions, and questions on Twitter at analytics hour on the web at analytics hour.io, our LinkedIn group and the measured chat slack group music for the podcast by Josh Crowhurst. 00:46:35.52 | Charles Barkley: Oh, smart guys wanted to fit in. So they made it. Made up a term called analytics. Analytics don’t work. Do the analytics say go for it no matter who’s going for it. So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 00:47:08.59 | Tim Wilson: No one ever listen to me. No one ever heard me say go for it. No one ever listen to me. The post #307: AI Does (and Does Not) Work for Data Storytelling appeared first on The Analytics Power Hour: Data and Analytics Podcast .

September 15, 202645 min

#306: Decision Support Has Been the Point All Along

Before there was BI, there were “decision support systems.” Somewhere along the way, we seem to have quietly dropped the “decision support” part and just kept the systems. Zohar Strinka , founder of Analytics Strategies and creator of the Meta-Problem Method, joined us to put the point back where it belongs: if nothing is going to be done differently after the analysis, then the analysis had exactly zero effect on the world. But — and this is the part that’s easy to miss — that does NOT mean marching up to a stakeholder and demanding, “What decision are you going to make?” People don’t want a model that hands them the right answer. They want to understand and weigh the trade-offs themselves, which means good decision support looks a lot more like a really well-informed pro/con list than a score. We got into problem spaces, high-yield problems, the cost of being wrong in each direction, why “we have all this data, so the answer must be in here” is really just a person pulverizing a bag of rocks hoping for diamonds, and, ultimately, peanut butter. This episode is brought to you by Stape , your all-in-one solution for server-side tagging. This episode is also brought to you by Prism from Ask-Y —your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show Meta-Problem Method (Zohar Strinka’s website) The Meta-Problem Method for Daily Life (Udemy course) Inside the Long, AI-Powered Quest to Perfect Pringle-Making (WSJ article) Educating for AI: The Skills Grads Need in an AI-Enabled World Aren’t What Anyone Thinks by Gary Angel Profession by Isaac Asimov (short story) Three Laws of Robotics (Wikipedia) Photo by Javier Allegue Barros on Unsplash Episode Transcript 00:00:00.00 | Announcer: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. 00:00:14.00 | Michael Helbling: Hi, everybody. Welcome. It’s the Analytics Power Hour, and this is episode 306. You know, sometimes it’s good to step back from whatever you’re working on and maybe re-ask the question, well, why do we do analytics in the first place? I mean, aside from the obvious rock star status. I mean, I kid, I kid. The goal is to help organizations make better and maybe faster decisions. And the barriers to doing that are, I mean, oftentimes both obvious but also obscure. So let’s talk about it. My co-host is Tim Wilson. Hi, Tim. You’ve supported a fair few decisions over the years. 00:00:52.64 | Tim Wilson: We decided to record this episode. That’s right. That’s the one I’m most proud of ever. 00:00:57.70 | Michael Helbling: And I’m Michael. I’m not sure if I’ve decided yet to fully record this episode, but we’ll see how it goes. 00:01:03.17 | Tim Wilson: Just drop 10 minutes in. 00:01:05.35 | Michael Helbling: Yeah, that’s right. I’m out. That’s right. All right. But we actually are excited about this because we also have a guest. Zohar Strinka is the founder of Analytics Strategies and the creator of the Meta-Problem Method, a science-based approach to choosing which problems to solve. She has a PhD in industrial and operations engineering from the University of Michigan. And today she is our guest. Welcome to the show, Zohar. 00:01:27.18 | Zohar Strinka: Thanks for having me. 00:01:28.64 | Michael Helbling: Yeah, thanks. Thanks so much for coming on the show. And I think maybe just to get us started, maybe could you just give us a little more background on the kind of work you do with companies? I know you more operate in the consulting space like Tim and I do as well. So just give us a little flavor, some of your background and the types of work you do that kind of led into some of the work you’re doing that we’re talking about today. 00:01:51.50 | Zohar Strinka: Yeah, happy to. So I got into consulting kind of by accident where I graduated. I graduated with my PhD, which was focused on optimization and these math models of decisions. But I graduated and was trying to figure out what was the career path ahead of me. And a lot of my peers were going into data science. So I was looking at opportunities like that. But I found I didn’t have the machine learning expertise that people were expecting. So I applied for jobs and found that in consulting, I asked good questions. And that was an important part of consulting. And so I found a job. There about a year in, I was laid off. And so I started looking and trying to figure out what I was going to do next and decided to start an independent consulting company. I got some traction, found some clients pretty quickly, which I know is a lot of luck, but also just kind of speaks to the approach I brought. But I had a different way of approaching analytics that I think was interesting to my clients since then. So I’ve been doing this since 20. I’ve been working as an independent consultant. I’ve worked with all sorts of different companies. I’m often more working with manufacturers or distributors or retailers because I have an inventory background. And so this industrial engineering mindset is really extra relevant for those folks. But I’ve worked with a lot of different companies and usually I’m trying to help them figure out what to do next with their analytics, what dashboards to build. Yeah. Yeah. Or what questions to answer or do we need a data warehouse at all? 00:03:32.52 | Michael Helbling: And it sounds like because of the timing, you’ve been involved in two pretty major shifts with COVID in the tariffs in the US. 00:03:41.71 | Zohar Strinka: Yes. Yeah. It’s been interesting because with tariffs especially, it’s just sort of it’s an external factor that a lot of my clients are trying to read the tea leaves and kind of say, do we load up on inventory and hope it’ll be to our benefit? Or they’re trying to. I’m trying to kind of predict the future a little bit, but it’s tricky and a lot of them aren’t using as much math as they could to help them make those decisions. 00:04:08.36 | Tim Wilson: How much is using the math? This does flash me back to COVID when there were. It’s like I was supporting some companies that were asking, like, tell us what we should do. And I’m like, well, this is I don’t know that like the historical precedent of 1918 is particularly useful. It was like you need to kind of think through. Like. Can you do you bridge those uncertainties with what you said early on? You were like, it’s about the questions that are being asked and then squaring that if you get brought in to say, what should we do next with our analytics? Is there a redirection that has to happen? Like it almost I’m trying to read between the lines. It sounds like I bet you reset every single client within the first week of working with them or no. 00:04:56.03 | Zohar Strinka: But. It’s tricky. Because there’s so much that goes into each of those decisions. The one that I made a mistake on. So I had my book knowledge on inventory. I was working with a client and they were trying to deal with the COVID disruptions of these really long lead times. What I didn’t quite understand is depending on your lead time is how those decisions matter. So if you have a short lead time, you can kind of react to those changes in the world really quickly. So if you’re ordering something from overseas and it’s going to take eight months to get to you, you’re in trouble if you misjudge and buy a whole bunch of something that’s not going to sell. If your lead time is a few days because you’re buying from a distributor, you can react much more quickly. And so a bad decision is really just about, yeah, don’t don’t make like an order of magnitude mistake and you’ll be fine. And so working with this client, they had long lead times. And I hadn’t like emotionally learned. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. 00:06:03.25 | SPEAKER_UNK: Yeah. Yeah. 00:06:03.87 | Zohar Strinka: Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. math can help and highlight, okay, this is a mistake where if we make it, it’s fine. Either we’re getting two weeks of inventory or three months, we can handle that from a cashflow point of view. But if we’re talking about either two weeks or 10 years, that becomes a lot more of a 00:06:29.86 | Tim Wilson: problem. How do you get to finding out what those sort of key factors, like you talk about problem spaces, you talk about making decisions and there you just sort of talked about like the business operating environment, like where, where do you start? I guess what’s the, and how much of that 00:06:52.81 | Zohar Strinka: do you have to know before you can start doing math stuff? It’s tricky. So one thing I’ve learned is how much the industry, like industries vary. The math is the same, right? The analytics is the same. So a lot of folks in analytics consulting, especially, we come in and we say, I know the math, the data’s there, I can figure it out. And there’s a risk there depending on the industry, right? And so there’s, there’s some of that of what are the key trade-offs to me are a cheat code for navigating a new industry. If you can pull out those key trade-offs, what are the goals that you’re trying to balance? Okay. We want to not stock out, but we don’t want to have, we don’t want to have a dead inventory. That’s a key trade-off and there’s got to be a sweet spot somewhere. And so for each of those questions, you start to ask yourself, what’s the cost of getting it wrong either way. And if the cost is really high one direction, not too bad, the other direction, well, err on the side, that’s not going to be expensive. And so that sort of universal frame up framing of what’s the trade-off, how does my choice impact my outcomes can really help you avoid some of those mistakes. And we’re talking really abstract, but I’m really thinking of like during my PhD, I was focused on inventory questions of how much inventory should you have? And the research studies found that people like to over-order inventory. They hate stockouts. And usually the cost of stocking out, isn’t that crazy? You can expedite, like there’s usually not that high a cost. And so the real world, experience is go lower, order less inventory. You’ll thank yourself later, but that’s like something you have to learn. It’s not in the math by itself. It’s there, 00:08:50.55 | Tim Wilson: but it’s hidden. I’m trying to think coming from kind of more of a, from a marketing analytics background and finding often, I mean, because when you talk inventory and marketing, it’s like, what’s the ad inventory and how much can you spend on, spend your advertising dollars? And the, the risk of not advertising enough is that you’re missing your, your audience. The reality is, is there’s all sorts of money that might as well just go light it on fire. And the, the, the smoke from that will bring more customers than what you’re spending on, you know, crappy ad inventory. I’m trying to like, I’m trying to put myself into a conversation with a senior marketer. I mean, I think it’d be really useful. I’m just trying to imagine how, you know, I’m just trying to imagine how, you know, I’m just trying to imagine how, you know, I’m just trying to imagine how, you know, I’m just trying to imagine how, you know, I’m just trying to imagine how it goes. Like you have, you have half a million dollars that you’re going to spend on advertising and you’re trying to make decisions on where to spend it and like how to, I’m trying to think where the, the, the, the trade-off discussion comes in. I don’t have an, 00:09:59.65 | Zohar Strinka: I don’t have an easy answer. Yeah. And I’ve worked in marketing as well. And so, 00:10:03.98 | Tim Wilson: and then you ran back to inventory. No, sorry. These people are horrible. 00:10:11.68 | Zohar Strinka: It’s, it’s interesting because so much of the marketing world is you do it because you’ve got to do it, right? You, we have to spend money on marketing because we’ve got to be getting our name out there and it can be so hard to trace how that dollar worked or didn’t. And so the fundamental trade-off is what’s a dollar spent versus how much are you going to get back? Right. And so once you’ve decided you’re going to spend half a million dollars, you’ve sort of eliminated some of that trade-off. And now to your point, you’re deciding, okay, where’s the best bang for my buck if I have this fixed budget? So there’s sort of, you could try to make every dollar of ad spend earn itself, but the models are too complicated, right? 00:10:58.75 | Tim Wilson: Michael, where does your marketing data actually live? Oh, everywhere. Google ads, 00:11:03.70 | Michael Helbling: Meta, HubSpot, J4, Snowflake. I think the forecast metrics live exclusively in Steve’s laptop. 00:11:10.70 | Tim Wilson: Well, that, that feels healthy. Oh yeah. Very governed, very mindful. Well, Prism now has connectors that bring those sources together and turn the raw platform data into clean analysis ready tables. 00:11:21.74 | Michael Helbling: Wait, so it does not just connect to the data and say, good luck with these 183 columns. 00:11:28.24 | Tim Wilson: Correct. Ask.Y has prebuilt marketing analytics recipes that model the raw data and keep it refreshed automatically. Google ads, Meta, HubSpot, GA4. You betcha. Plus Slack, Snowflake, Power BI, and more. I’m not sure about Steve’s laptop. 00:11:45.24 | Michael Helbling: Well, fewer exports means fewer mystery spreadsheets and fewer moments where someone says, I thought that refreshed automatically. That is exactly the idea. Uh, I would like to personally nominate CSV attachments for retirement. 00:11:58.56 | Tim Wilson: Just kick them out the door. We’ll click the link in the show description and sign up for the Ask.Y waitlist. Use code APH and they will move you to the top. Top of that list. 00:12:07.68 | Michael Helbling: Connect the data and disconnect from the nonsense. 00:12:11.87 | Tim Wilson: Michael, I have a new rule for AI. Ooh, this should be good. If a task involves clicking through six menus just to confirm something I already suspect, AI can have it. 00:12:22.72 | Michael Helbling: Hey, that’s pretty much the idea behind Stape’s AI Assistant. It handles routine measurement tasks directly inside your Stape’s account. Like what? Well, ask which containers are connected, check server-side GTM usage, review existing tags, and check server-side GTM usage. Review existing tags and check if they are connected. Check if you have any tags and triggers. Uh, inspect your GTM configuration or confirm that events are being collected in GA4. 00:12:42.20 | Tim Wilson: And it can actually make changes? 00:12:43.70 | Michael Helbling: Yeah, for supported tasks, yes it can. It can create Stape’s and server-side containers and help set up tags and triggers in GTM. And the best part is, you aren’t using your own LLM tokens to do it. 00:12:54.74 | Tim Wilson: So the AI handles some of the repetitive navigation and I get to spend more time checking whether the implementation actually makes sense. 00:13:01.60 | Michael Helbling: Yeah, which feels like a much better division of labor. 00:13:04.60 | Tim Wilson: Yeah. So, if you’re looking for a better way to get started with Stape’s AI Assistant, click the link in the show description. I think that’s the challenge is that it winds up being performative analytics because you can find something that claims that from the Google ecosystem or the meta ecosystem or your digital analytics platform. And that’s like the, I don’t know if that winds up, I mean, this is also going to sound philosophical that it’s like, well, it’s performative. It makes people, it just makes you feel good because you’ve got something that shows I spend a dollar and I got $2. Yeah. I’m not going to be able to get the numbers back, even if the math is horribly flawed, I feel like marketing will uniquely say, well, that’s okay, because I just want to keep spending more and I want to justify it. I’m sorry, I’m just going to have an existential crisis, like right in the middle of, you know, 15 minutes into the episode. 00:13:51.76 | Zohar Strinka: I mean, it’s something where I think that is a challenge we run into in analytics when you’re trying to make these decisions and you want it to be data-based, but you don’t have the data. And so you make up a proxy. Yeah. Yeah. And so one of the questions I start asking is, are those good proxies? Right? So, so our impression, right? I started working on a marketing project when they still sort of cared about impressions a little bit, but they had mostly switched over to clicks. And so there’s sort of this, what’s the metric we can see and how does it connect to the things we care about? Which is really, I think one of the most valuable things we can do is keep coming back to what are you trying to do with it? Like that marketing dollar? Are you trying to just bring awareness? Are you trying to get people to click something? Well, a radio ad will never get someone to click something. Okay. So, so you can sort of start to break down what are the decisions you can make and how they’re connecting to the things you care about. 00:14:53.52 | Michael Helbling: So when, when people come to you, like, do you frequently have to go through a refinement process to the problem itself? Like this is pretty common and on the marketing side where people are like, I think I have this problem. You’re like, well, you technically have a sort of different problem and you have to kind of go through that. Like, how do you kind of go through that process of like helping refine or redefine the problem? And how do you avoid alienating your stakeholder in that process? Cause like a lot of times that that’s a negotiation in a way. 00:15:25.17 | Zohar Strinka: Yeah, that one’s really tricky. And one of my answers is my favorite clients, the ones that get the most out of me and I enjoy working with the most. 00:15:34.79 | SPEAKER_UNK: Yeah. 00:15:31.79 | Zohar Strinka: Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. 00:15:32.53 [SPEAKER_UNK]: 00:15:32.53 | Zohar Strinka: Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. when I change the problem on them. So they say, I want to figure out which ads are working well. Like what’s the click-through rate? And you sort of say, okay, tell me a little bit more about why you care about click-through rate. Like what outcomes is that driving at? And that sort of question of how do the things that we control link to the things that you want is where I’ve had a lot of success throughout my consulting career. But now I’m telling clients more about it. I’m saying, I’m asking because if we got this wrong, then it’s going to change which problem we should solve. If there are different outcomes you care about, I’m going to go recommend something terrible if I’m working to the wrong goal. So if you care about clicks because that leads to conversions in the marketing world, like, okay, we care about conversions then. That’s the goal. 00:16:56.32 | Tim Wilson: And so you have like within the, and you have the website, so it’s pretty easy for people to go look and see the high level because you have kind of the hierarchy of dilemma goals, problem space, high yield problems, problem selection. Like, I think it might be useful like to, we’re not going to get a full complete primer on the entire method. But the fact that it’s this, anytime I see a hierarchy, I’m like, well, this is sort of forcing decisions at each point because it’s like a tree, I imagine, that goes down and you’re like, yeah, you can’t do everything. So which branch on the tree are we following? So maybe you can talk a little bit about like, nailing down the goal, which like you just, I think you just talked about is conversion. How do you get from that to a problem space versus high yield problems, I guess? 00:17:48.51 | Zohar Strinka: Yeah. And the first question, I like the dilemma idea or the question you’re starting with is your scope. This is where I’ve gotten in the most trouble where someone says, we need to manage this. We need to maximize conversions. I’m like, but why? You’ve already worked on conversions. Let’s work on something else that I think is going to go better. And if you make that mistake, if you do that change on them and that wasn’t what they wanted, you’re going to run into challenges. And so that’s where I’ve made those mistakes. 00:18:17.59 | Tim Wilson: Well, do you have, because I have been dropped in where, if you just asked, what is the dilemma? I would hear, we have all this data and we don’t know what to do with it. Or we know we have all the right data, but it’s not giving me insights, which to me, all sorts of, and not alarm bells. I’m like, okay, there’s coaching. I’m not going to swoop in and look at your data. I’ve got to get you to the actual, I guess, business dilemma. Like if you run into that where they’re bringing, because it was like an early example you had was that you’re like, well, they’re bringing me in to say, do some analysis. I’m like, well, that’s not right. 00:18:54.03 | Zohar Strinka: I have run into that. The thing I tell myself today is, okay, 00:18:59.00 [SPEAKER_UNK]: 00:18:59.00 | Zohar Strinka: You’re not going to be able to do that. You might think the data magically is going to have some answers for you. But what I find is often they know there’s certain data they’re not using to make a decision. And so that’s what I want to hone in on. So I started asking, okay, why do you think the data will have answers for you? Well, we’re not doing forecasting. Okay. That’s now something that analytics makes sense to bring to the problem. And so I’m usually trying to navigate those questions, like go from the world of everything where, okay, I don’t know your business. I’m not an expert in whatever domain it is. You need to give me something more than that. Where should I be looking? And so that’s been my approach is to really ask more questions about, okay, what opportunities do you think there are? And that’s the high yield problems to some extent. So this is jumping ahead a little bit in the method. I start with goals because, you know, I find that we people think there’s an opportunity, right? Like, so we say that something’s a problem because we think there’s a solution out there. We think that conversion could be higher. It’s a 3% industry standard says 5%. Okay. That’s an opportunity. So they often have that reason somewhere in the group bringing you in, or they wouldn’t bring you in, right? But you have to kind of suss it out because they sort of, there’s this belief that the data is just going to reveal the answers. And that’s really not true. 00:20:33.31 | Michael Helbling: I’ve literally been sat down and told, like, find something in the data that we can do, like optimize. And it’s like, do you have something you’d like to accomplish or you just know the data is going to tell us? 00:20:46.46 | Tim Wilson: I had a client where there was an analytics center of excellence. And the guy who ran that, he said, look, we meet with the brand managers every two weeks and we have to find something to bring to them. And I’m like, well, what’s keeping them up at night? Like, what are they trying to do? It’s like, I don’t like the first time we did it, we found all sorts of stuff. They didn’t act on any of it. We found some other stuff two weeks later. They didn’t act on any of that either. And it was like, but you’re trying to blame them when you haven’t actually asked them what they care about. I mean, I didn’t that did not go well. It was a long, that was engagement that felt a lot longer than it was. 00:21:18.91 | Zohar Strinka: It’s a misunderstanding of what we do with analytics, right? It’s really this belief that somewhere in the data is going to be some answer that we can just, like, go solve something. And it might be there, right? When you’re an analyst, if you’re good at what you do, you’re asking those questions. You’re trying to say, what would increase profit? What would increase conversion? It’s why I like certain dashboard tools, because you can explore the data more easily and just say, hey, is there any, like, relationship here that we could do something about? Is there some action we could take? So I’ve enjoyed, that was how I actually got started in analytics. I did a lot. 00:21:57.87 [SPEAKER_UNK]: 00:21:57.87 | Zohar Strinka: I know a lot. I know a lot. I know a lot. I know a lot. I know a lot. 00:21:58.07 [SPEAKER_UNK]: 00:21:58.07 | Zohar Strinka: I know a lot. I know a lot. I know a lot more. Like, I called it clever averages at the time, but it turns out that it has a real name of diagnostic analytics. And just trying to see, is there something in there? Is there a pattern that we can do something about? 00:22:11.50 | Tim Wilson: Do you run into, I guess, yeah, this, we’re heading down the, we’re heading down the path of, like, the assumption is I have all this data, therefore there must be value in it. I sometimes think about it, like, in an ad, it’s like somebody taking a bag of rocks and saying, oh, I don’t know. I heard somebody once busted up their bag of rocks and found some diamonds in it. So I’m just going to keep pulverizing and testing these rocks because I have a bag of rocks. So I’m just going to keep pounding it until I find value. And I feel like I run into cases where really they’re best suited to actually gather some new data, often through an experiment. It’s like, I know you have this massive. Wonderful snowflake environment with all of these integrated data sources. But what you really should do is design an experiment and run it for two months. And that’ll give you, that will actually give you a stronger answer. And that data is never going to live in the warehouse and you’re not going to find it. Like, so do you, if you start at that level and you get to like a high yield problem, do you come across like, I’m going to constrain myself to the data that we have? Or do you say, if there’s. If there’s some other low cost, whatever that means, data, that’s the best way to solve it. You’re looking in the wrong space. 00:23:35.77 | Zohar Strinka: It can be high cost data, right? If you’re thinking about going and running your whole business strategy and you’re a decent sized company, like the data could be high cost. There’s reasons those third party data providers exist. But yeah, I run into it. It comes up a lot that they sort of think that, well, we have so much data that the answer must be in here. So to your point. Yeah. Yeah. Yeah. So I think that you do need to really bring that modeling point of view. I mentioned that I have this optimization background, which basically says, if you have the right information, it’ll tell you the best decision to make. So you take the inputs and you can decide exactly. The future is uncertain. And often the best decision is to put off a decision and see what happens in the world before you commit to something or you buy flexibility. Right. And so there’s always this. Question of what do I need to commit to today? And what do I want to put off till tomorrow? Or what do I want to learn before I commit to a decision? 00:24:37.21 | Tim Wilson: I remember my mind being blown in a. I don’t remember what the class was when like options theory was like, there’s value in an option, so you don’t have to like make if there’s a way to make a small decision that learns and moves you forward. But now we said like decision two or three times in the last 60 seconds. So I’ve got to go like the easy. Hacky rant that I have was my understanding of like the history of the BI came out of decision support systems and even prepping for this show. I was like, oh, I guess people still say decision support systems. You’re you’re framing is like we’re fundamentally trying to help people make decisions. But it does. It feels like to me, we’ve lost that somewhat. There’s a lot of times the business like, yeah, I want to decide the best way to spend my money. And that’s like that somehow feels too nebulous. And I don’t know. Where do you you have thoughts on decisions? I know from pre-show back and forth. Where do you land on that? 00:25:38.75 | Zohar Strinka: So where I land on decisions is unless you’re going to do something differently after your analysis, which is a decision or an action, that analysis had zero value. It had no effect on the world. And so. Fundamentally, any analysis we do has to be informing a decision or. It’s kind of a waste of time. That’s my my initial shot across the bow is decisions have to be what we’re aimed at. The second part of that is people like to own the decisions. People like to feel like they’re making the decision. So this is what I ran into initially with inventory. So I have these great inventory models. I can make the optimal inventory decision. And. You go do that and they’re like, oh, yeah, the data is wrong in these ways. I’m just going to ignore that and make my own decision the way I’m comfortable with it. And so you run into this this challenge where people really want to feel in control of the decision and decision support that isn’t adapted to how they think about it is just going to be ignored because it needs to support the way they think about these tradeoffs and these problems and these decisions. And so I think that’s the core of it is really we often think of a model telling you the right answer. People don’t want a right answer. They want to understand and figure out how to balance these tradeoffs. 00:27:13.51 | Michael Helbling: This is resonating with me because what I’ve observed is that how people approach this, like both through personality as well as positionality influences how analytics works for them or it doesn’t. And so I think that’s the core of it is really we often think of a model telling you the right answer. Because if they’re sort of like, yeah, I need support. I’ve got these decisions to make. Give me as much material as possible to inform me versus I already pretty much know what I’m doing. And maybe you’ll come up with something. Maybe you won’t. And I’ll spend all my time picking at your answers and then just go make my own choice. Like, OK, well, why am I working with you? Like, I don’t need to do this for you. You already know everything. So it’s not a big deal. But it goes back to like, yeah, how and like in the. In the world of analytics, we talk about this term like data literacy or numeracy or the ability to use data to make decisions and this kinds of things. And like, what is data culture? And all of that kind of stems around sort of this concept of like, yeah, how do you take analysis and put it in front of somebody so they’ll actually consume it into the process? And I’m endlessly fascinated by that because it’s. Actually more of a human problem than a data problem. 00:28:30.47 | Tim Wilson: I will metaphorically throttle analysts who say when somebody comes to them and ask them a question, they say, what decision are you going to make? And they’re kind of pat themselves on the back because they’re like, oh, they asked for this data. And I told them, what decision are you going to make? And I’m like, that’s that’s wrong. And then I see things that are like a dashboard that doesn’t lead to decisions is a worthless dashboard. I don’t like that either. What I like about if you with your perspective on the decisions. And. Instead of having a framework and maybe this is me projecting what I feel like is effective is you’ve got to start. You’ve got to start understanding their problem and their mindset, build the trust, get that focus, do a little bit of educating along the way so that then whenever whatever happens to be done with the numbers, they were on board with it. They were involved with it. They were bought in. And you were also kind of gently educating them that it isn’t it’s not effective for them to say, I got to decide whether I spend on A or spend on B. So just give me all the data and that would give me the answer. You’re like, it’s it. The fact that you’re kind of framing it as saying, let’s really be clear on what our goal is, because if you’re trying to make a decision and you don’t really have that articulated, you’re in a problem. You have a problem. 00:29:52.63 | Zohar Strinka:/strong> And I’m going to push this a little further. So for both of you, imagine you’re standing in. The grocery store and you’re looking at all the different kinds of peanut butter. How do you decide which one to buy? It’s a data informed decision. 00:30:07.03 | Michael Helbling: That’s so it’s not. It’s whichever one my wife, Maria, told me to buy. So it’s good. 00:30:16.37 | Zohar Strinka: So this is where I go when what to your point, Tim, of like thinking of that analyst saying, oh, they asked for the data. And I said, what decision are you trying to make? Well, we often don’t. We don’t really know which decisions we’re making or what we really care about until we really try to make it right. It’s really when you’re trying to make that decision that you either have an answer or you don’t. And so one of the things that I think is really powerful that we do as individuals making decisions is pro and con lists, because it lets you sort of say these are the things I care about. These are all the goals I have. And that’s how two alternatives typically rank on the goals. Now I can sort of weigh these different factors in my head and see I like option A better than option B. To me, that’s the mindset we should be taking into analytics to help inform decisions. It’s more pro and con lists and less. Here’s the score. The score says this is the decision to make because it’s the most accurate. It’s the best. But instead, sort of turning it more into that multifaceted set of things we care about. 00:31:26.86 | Tim Wilson: Although it also sounds. It feels like it works into a confusion matrix world where you were talking about the cost of making the wrong. And decisions aren’t necessarily binary, but what’s the cost of missing in this direction? What’s the cost of trying to quantify or clarify what those trade-offs and what the upside and downside. Risk. Like it just feels like it starts with. You got to capture it and get it out of people’s brains and then figure out. Can we bring it into focus? Like, is there data to help focus that? And then maybe because it may be like, oh, yeah, we know that the downside risk is huge and the upside risk upside benefit is small or whatever. And then you can move down to saying, OK, given all of that now. What we’re trying to figure out is actually quite narrow and tight and everybody we’ve iterated on it enough that everybody’s on board. Like once I do this thing, the decision will become. Self-evident, even though there’s going to be uncertainty in it still. Wow. I’m feeling hand wavy theoretical now. 00:32:50.92 | Zohar Strinka: Well, and this is what I’ve struggled with as I’ve been trying to put pen to paper on this. Right. It’s. It is really like it’s this abstract thought process of. For me, it just felt like I always knew which next question I wanted to ask my client because I was driving at something. I thought there was an opportunity over here. And so I’d ask them about it and then they’d give me an answer. And OK, I’d ask a different question based on that answer. And so this problem space idea, you’ve mentioned it a couple of times. I haven’t defined it. But it’s this idea that you you can sort of say, OK. Do we want to maximize conversions or do we want to maximize clicks or do we need to have better marketing ads or do we need to have different platforms? And it’s sort of this question, the series of questions as you’re trying to figure out where’s the opportunity, where’s the decision I can make differently that’s going to have the results I want. And so to me, problem space is really you start somewhere and then you ask a question of, well, is this really what I’m after or is there something better over there? Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. Right. 00:34:06.42 | Tim Wilson: Right. Right. Right. Right. Right. Right. smart and inquisitive person who’s asking some really good questions that making this making them think is it on you to then say like synthesize it and play it back to them so that they start to 00:34:38.40 | Zohar Strinka: get clearer in their thinking so that’s what i i did innately when i started at consulting it turns out to actually drive benefits or value you know to use the consulting speak you have to actually make decisions eventually and so feeling like you understand your business better you understand where the opportunities are is only say half the battle it’s the half that i i started with but to your point at some point you have to say all right this is the problem we should go after and solve and this is the benefit of doing so and that’s another of those areas where i have to hold myself accountable to not keep exploring because there’s always more interesting questions right if you’re a curious person in analytics that sort of guides you in all these discussions and oh what if we could go do something over there well at some point you have to rein yourself in and sort of say okay what can we do here what what can we what decisions can we inform or what analysis can we do there’s like this reckoning moment where you have to just say okay the world’s uncertain but we’ve gotta move this 00:35:50.42 | Tim Wilson: direction but do you find yourself like almost like that that becomes a living like this is the this is the space and other stuff will come up and i’m gonna have the discipline to not pursue it but i’m gonna throw it into the into my i don’t know what is the problem space map or whatever to have it so there’s always something to come back to that’s how my brain 00:36:10.70 | Zohar Strinka: works i i uh yeah it’s something we’re trying to set that aside is my challenge is to sort of focus on the things that are in front of me and i’m like okay well i’m gonna go do this and i’m gonna do this and i’m gonna do this and i’m gonna do this and i’m gonna do this and i’m gonna do this and of me because new things always come up right that might change the right approach to take and that’s good if you’ve gotten somewhere if you’ve delivered something but now you really really have to like i i enjoy proof of concept projects and mvp kind of projects because we’re just trying to prove it works and someone else can go scale it now so that’s that’s for me like just a self awareness thing like that that initiative is a self-awareness thing and i think that’s a good version that proves it makes sense okay let’s now go on to the next thing it out of that bucket of potential problems okay we solved that thing what’s next boy that seems like another devil 00:37:02.74 | Tim Wilson: watch clients like people get so excited that something’s going to happen and they’re like well let’s just either it’s let’s build the whole thing so it becomes way bigger and it hasn’t been proved out or it’s let’s build the proof of concept and say cool so now can you just push it into production it’s like well no this was built it’s got it’s got manual data refresh loads that aren’t going to be sustainable and we need to to throw it out but you can do 10 of those for one full-on implementation wrote an article that 00:37:31.59 | Zohar Strinka: touched on that because it is like it’s a challenge we technical folks have where we get to decide do you build it enterprise ready and then nine-tenths of them get shelved or do you build it as a true poc and then get burned the one time that they actually want to go go launch it and it’s like i i feel like that’s one of the things in the technical space where we have to educate the business partners better about these trade-offs of okay are we building this as a real poc and what does that mean right do you have to do the the man do you have to build a database or can we prove it out in python like do we what choices do we need to make and how much what parts of the question are sort of questions and which ones are known we can build the technology okay if that’s not a concern don’t worry about the technology side right if it’s a data question and so to me there’s an element of choosing the right parts of the problem to to test it’s really the things where you have the least the biggest questions if this works we’re good if it doesn’t work it doesn’t okay just test that part of the project it’s funny because i’ve also used this in kind 00:38:49.18 | Michael Helbling: of a sort of like a circular fashion where as an analyst i’m keeping track of how i’m approaching solving these problems and then what’s working and what’s not working and then going back and reiterating on that process as time goes on so like you can even take your method and apply it just to even how you approach the business to work on different problems anyways this conversation has been awesome really really uh right up our alley thank you so much zohar one of the things we’ll be talking about is the last call something that might be of interest to our listeners so har you’re our guest uh do you have 00:39:27.48 | Zohar Strinka: a last call you’d like to share well i have two if that’s okay that’s fine yeah all right um so the first is i do have a udemy course on the metal problem method that i’ve just built so if you’re interested in learning more i have the website that’s just out there as well but i’ve tried to make it a little bit more hands-on the second one is i live in this manufacturing and inventory and all that space and so i follow a blog that’s on operations management and a couple weeks ago they had a blog on pringles kpis so they were using advanced analytics to deal with the fact that you know when you have a new batch of potatoes the moisture will be a little bit different and so they were using sensors and all that information to adjust how they made pringles and what they were reporting was a 10 percent rate of improvement in the metal problem and so i’m going to talk a little bit more about that and things like that and they’re talking about maybe we don’t have to be so picky about which potatoes we buy because apparently they buy exactly one variety and so it was just an 00:40:31.20 | Tim Wilson: interesting question of the pringle dashboard i feel like walt hickey’s numlock news had a reference that rings a bell that like the like pringles innovation because i mean they’re such a odd little chip with cocaine or whatever crack something that’s baked into them because 00:40:49.48 | Michael Helbling: can he just one i technically i don’t think they’re even called a potato chip anymore but you know that’s not relevant to the to the question all right tim what about you 00:41:01.17 | Tim Wilson: what’s your last call um i will pop a good old gary angel he wrote a post on his medium site of educating for ai the skills grads need in an ai enabled world around aren’t what anyone thinks and he doesn’t pretend to have a good old gary angel and he doesn’t pretend to have a good a good old gary angel and he doesn’t pretend to have a good old gary angel and he doesn’t pretend to have the answer and you’re like oh this is going to be some it’s very thoughtful had some really clever takes on that so i recommend that what about you michael what’s your last call 00:41:31.28 | Michael Helbling: well i recently ran across an old short story by isaac asimov they wrote in 1957 called profession and you can find it online in a couple places but basically the story goes that every person gets into their career after high school by basically basically their knowledge of that profession being wired directly into their brain and follows the story of a young man who is basically set outside of that process and he takes him a long time to figure out why he’s not being able to be in a profession but has to figure out how to do things on his own and learn things on his own and what that means for him anyways fascinating on a couple levels especially because we’re at a time where we’re basically kind of letting ai be our brain in a lot of ways our versus business Genbie products could work very well if they should play around with different choices one way or another right now when all this happens them um i i feel like when i talk about familial Romantic arts i feel like every star bearers i feel like ways and what that might mean for thinking and human enfeeblement and things like that. So, 00:42:24.85 | Tim Wilson: anyways, it’s a fun short story. Without the three laws of robotics applied to it. 00:42:28.98 | Michael Helbling: Dang it. Not directly. Yeah. No, Asimov is just so great. Like his whole brain, like what was he doing in 1957 thinking like this? Anyway, it’s really good. All right. Zohar, again, thank you so much for coming on the show. This has been a great conversation and one that’s near and dear to our hearts. And so, I appreciate your take on it and taking the time to chat with us. And if people want to learn more, you have a website for the Meta Problem Method, which is meta-problem.com, which I highly recommend people take a look at. And as you’ve been listening, you probably have your own thoughts and we would love to hear from you. And so, yeah, please reach out to us. You can do that as a listener through the Measure Slack chat group or you can reach out to us on the Meta Problem Method website. And we would love to hear from 00:43:16.34 [SPEAKER_UNK]: 00:43:16.34 | Michael Helbling: you. And as you’re listening, also feel free to leave reviews, ratings, and comments on whatever episode or whatever platform you listen to the episode on. We’d love to hear from you. All right. I think we did it. We did the whole thing. And that’s awesome. I don’t know. You know, it’s funny because I love a scientific approach to this, because it feels super daunting when you first start tangling with this problem. And I’ve watched a lot of analytics people basically bash their brains out in a sort of way on this problem where it’s sort of like you hear this again and again. Nobody ever listens to me or I do all this work and it never gets anything done. And I’m never attached to the decisions that ends up getting made. And so, it’s like, well, how do we get better at this? And so, I love so hard that you’re tackling this problem and you’re thinking, well, how do we get better at this? And I think it’s not only helpful for businesses, but it’s helpful for everybody in our industry. So, thank you very much. And I think I speak for my co-host, Tim Wilson. No matter how big your meta problem that you’re 00:44:29.32 | Announcer: trying to solve, keep analyzing. Thanks for listening. Let’s keep the conversation going with your comments, suggestions, and questions on Twitter at at Analytics Hour, on the web at analyticshour.io, our LinkedIn group, and the Measured Chat Slack group. Music for the podcast by Josh Crowhurst. So, smart guys wanted to fit in. So, they made up a term called analytics. Analytics don’t work. 00:44:56.30 | Charles Barkley: Do the analytics say go for it no matter who’s going for it? So, if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. I’m fighting out in real time some of my responsibilities. 00:45:13.40 | Michael Helbling: Stop. That’s good. We’re super… 00:45:16.82 | Tim Wilson: I’ve been on Slack like a week and a half ago. We’re super buttoned up. 00:45:18.98 | Michael Helbling: The test designer said this is great. Let’s start arguing after we start the show, Tim. Come on. No, I’m just kidding. All right. All right. 00:45:38.07 | Tim Wilson: Rock, flag, and peanut butter decisions. Decisions. The post #306: Decision Support Has Been the Point All Along appeared first on The Analytics Power Hour: Data and Analytics Podcast .

September 1, 20261 hr 4 min

#305: Personal Interest + Analytics Chops = Career?

Stephen Follows has spent 15 years turning film-industry curiosity into a career — asking questions like how much movies really earn, why poster colors have shifted over decades, and whether old studio domains are still up for grabs. In this episode, he joined us to talk about building that unusual path, the cautionary tale of TheNumbers.com , and his theory about Sandra Bullock. Go start your passion project. This episode is brought to you by Prism from Ask-Y —your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. This episode is also brought to you by Stape , your all-in-one solution for server-side tagging. Links to Resources Mentioned in the Show stephenfollows.com (YouTube Channel) The Film Data Scientist – Stephen’s YouTube channel (Movie) The Blair Witch Project (Movie) Paranormal Activity (Book) Becoming a Category of One: How Extraordinary Companies Transcend Commodity and Defy Comparison The Numbers (Article) What just happened to TheNumbers.com should worry us all by Stephen Follows (Article) Which colours dominate movie posters and why? by Stephen Follows Chekhov’s Gun The Wilhelm Scream (Movie) The Bridges of Madison County (Movie) Speed (Movie) Unbreakable (Movie) The Blind Side Adobe Character Animator ElevenLabs Stat Significant – Daniel Parris’s Substack Box Office Mojo Enshittification The Internet Archive Building Pudding.cool (Video) Why the Best Player Alive Barely Runs (Article) 3 Kinds of AI by Eric Sandosham (Book) Life, the Universe and Everything (Hitchhiker’s Guide to the Galaxy) by Douglas Adams “There is an art, it says, or rather, a knack to flying…” – Douglas Adams Photo by Denise Jans on Unsplash Episode Transcript 00:00:00.00 [Announcer]: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. 00:00:14.88 [Michael Helbling]: Hey, everybody, welcome. It’s the Analytics Power Hour, and this is episode 305. Follow your passion and you will never work a day in your life. We all know that quote. We also all know it’s kind of bullshit. But sometimes working hard and combining your passions, I think it can work out well. You know, as with any career in data, oftentimes how it is applied and to what is maybe kind of the real thing that energizes you. And I guess we wanted in this episode to have a bit of fun and talk to someone who has combined their data skills and their passion for something into a unique and wonderful career. But first, let me introduce my co-hosts. Julie Hoyer, how are you today? 00:00:58.70 [Julie Hoyer]: Hey there. I’m doing great. 00:01:01.16 [Michael Helbling]: Excellent. Great to have you. And Tim Wilson, how are you? Doing well. A little over caffeinated. Tim is a great example of someone turned his data skills into his passion of telling people that they’re wrong. So that’s amazing. Wow. And I’m Michael Helbling. All right. Our guest today is someone we are excited to talk to. Stephen Follows. He is a film data researcher whose mission is to help filmmakers get their films funded, shot and seen. 00:01:28.02 [Michael Helbling]: He’s a film data researcher. He consults with the Guinness World Records on film-related world records. He writes and researches extensively on the film industry at StephenFollows.com. And he’s also the founder of CatSnake, a storytelling agency that helps nonprofits identify and leverage narratives. And today he is our guest. Welcome to the show, Stephen. 00:01:46.21 [Stephen Follows]: Thank you so much for having me. And it’s really fun to be the fun guest rather than in the film industry where I’m the boring analytics guy. So that’s quite nice. 00:01:54.07 [Julie Hoyer]: Oh, yeah. That’s true. In this context, you’re kind of the cool… I never thought about that juxtaposition. That’s incredible. 00:02:04.68 [Michael Helbling]: Yeah. You’re at home. You’re with friends. Yes, right. Thank you so much. I’ve needed this so much. I’m so grateful. So, yeah. So to jump into it, I think, you know, obviously you have an incredible website where you do tons of research and articles and things like that. But I think the first question that came to my mind is sort of, yeah, how did this all begin? Like, how did you kind of combine these two things? So which came first? Was it a love of film or was it kind of a love of data? Like, what was the genesis of this? 00:02:34.97 [Stephen Follows]: Well, I get asked that a lot, I think, because what I do is unusual. Or at least people think they understand one part of it and then they hear about the other half. And that doesn’t make any sense in that sort of image they have. It makes sense to me, but I think I’m the only one. So when I was a teenager, I didn’t know what I wanted to do. I knew I had aptitude in some things, but I also knew that I wasn’t just going to follow along a path. And so I knew I wanted to do. I wanted to do film and I wanted to do thinking. They were the two things. And that’s only the extent to which I knew what I wanted to do. And so everybody said, go to university and study thinking. Go do philosophy or there’s analytics, something like that. And then just do film on the side. And because I’m pathological demand avoidance, I did the opposite. And I went into an art college and did a three-year degree in film. And never having done any film, never filmed anything before. And then that led on a path where I made, after that, I set up a little production company. We made short films. And we made films for charities and TV habits. And that sort of went on a journey. But as that was going on, I knew that I wasn’t doing the thinking bit. And I could see that film wasn’t going to take me to that place where I get to use those particular parts of the brain I wanted to use. So I started doing it off my own back, doing analytics for friends. They were putting a business plan together. And I mean, it was a pretty bad year because everyone said to me, I am making this film. I know it’s going to make a lot of money. Can you prove it? I was like, great, I’ll go and do that. And then I could never prove it would make money. And then after a year, I went, oh, none of them make money. I thought I was doing it wrong. And it turns out, you know, it’s passion. It’s all fine. So I ended up doing that. And I started sharing it online. And then, as you guys know, like whenever you have a passion or a niche thing and then you do it again and again, you become like a lighthouse. And people come to you and you build a community or you take part in existing communities. And that kind of grew from there. So for the last 15 years or so, I’ve been publishing two or three times a week with all sorts of different data studies, trying to look at film in different lenses. And that’s led on to various things. So these two paths of a company that was a film company now raises money for charities with film and story. And on the other hand, being the analytical view on the film industry makes sense to me is two sides of me. But they sort of there aren’t many people that would do both. 00:04:46.28 [Tim Wilson]: I think in those early days, how like what portion of the if you just look like the raw time and thought work you were doing, how much were you doing that? Like didn’t pay or didn’t have like a direct I’m getting compensated for this. Like were you was it clear that you’re like, well, some of the stuff I’m doing is a passion and hopefully it’ll turn into something. And the other stuff I’m doing to pay the bills. Like, what did that look like? 00:05:11.34 [Stephen Follows]: Well, that was easy because nothing paid me anything. So that’s an easy question, right? So I think I’m used to following passions that then might pay off later on. So it’s not that I was privileged or had the ability to do that. It’s that when I left. University, although I said I was doing my own production company, I was at a full time job working in a sort of marketing video company as a runner. Actually, I applied to be a runner. The lowest job didn’t get the job. They then the person they hired concluded that it was too much work for them and they hired me to be their assistant. So there was no title for it. I just did everything. So that was a year and a half or so where I was earning money from one place and then trying to build up my own agency. And then I started doing them. I moved to a tiny, tiny. Little office in a film studio. And that was right next to a film school. And I like teaching. I like sharing things. And so I started doing guest lecturing and then that grew. And then in the evenings and weekends, I was teaching there trying to set my agency up. So my agency wasn’t paying me very much. I think my business partner reminded me recently the first month we got paid 86 pounds each for the month. And so like that never made any money. But I was teaching. And then as the agency started to make money, I could do less and less teaching. But I was then at that point also doing the writing. And the analytics stuff. And it’s only occasionally I get paid to go like an honorarium to do a talk somewhere. But mostly it was about the travel being fun. And then I started to be able to turn it into money. But I only turned what was a WordPress blog entirely for free into a sub stack with a paid component in the last year and a half. I could have done it earlier, but I didn’t get around to it. And so, yeah, I think the money has always followed way, way later than the work. And I kind of expect. That now, like I just I don’t start anything thinking, well, there’s pay because it probably won’t. And it certainly won’t now. Got to start as a passion project. I don’t know if this is good advice. I only know this is what I did. Please don’t anyone follow this. 00:07:10.13 [Julie Hoyer]: But yeah, it sounds right to me. I was going to ask, too, with that initial question that you had, like your friends asking you, like, will this film make money? And you started to actually dig into that. Where did you start? I mean, that is such a big, broad question to answer. Where did you start if you didn’t do schooling and like the thinking side of what you wanted to do? Like you were saying, like, what were your first steps to attack that question and what resources did you use? What what did you try? 00:07:38.48 [Stephen Follows]: Yeah, I mean, so it’s actually fun to think back because it’s so different nowadays. But it’s a great question. So I always had an aptitude for maths, but no interest in it in and of itself. So I started it to 16 GCSE is the is the is the sort of first set of compulsory exams in the UK when I was a kid. And so I did maths. I did quite well. I think I forgot to bring a calculator into one of the exams and they still got an A. So I had an aptitude, but I had no interest because I forgot to bring a calculator. I just didn’t care. And I remember saying, can I have a calculator? 00:08:06.74 [Tim Wilson]: You were Julie’s kindred spirit. This is. 00:08:08.54 [Stephen Follows]: I just didn’t care. It just no one could convince me it was meaningful. And even now, actually, I still don’t think the maths part is I mean, we all have we have AIs. We don’t like I’m not saying that the maths doesn’t matter. I’m saying that it’s only a part of a larger point around logic or. And so it was all first principles. Yeah. So I didn’t go in there thinking, oh, I know how films make money. I mean, I have no idea. So I went around and spoke to people. And it’s not that you can get people’s private financial data or anything like that. You’re not getting true insight. But what you can do is say, well, we know these ones were very successful and these are the numbers we do have. And back then you’d have video numbers and you’d certainly have theatrical theatre, cinema money. And you could say, OK, well, when I film on this budget makes at least this much and then, you know, it’s probably in the profitable end. So. What we’re trying to prove here is not that it’s a good business investment, because I can assure you, no film is a good business investment. We’re trying to prove that they’ve done the best they can, given that it’s not a financial thing. It’s a passion project or they’re trying to convince an investor who’s going to get big tax breaks for doing it to do this one, not that one. All you’re trying to do is show that you’ve done more thinking than everyone else and that you’re more likely to be in that category of things that might break out. So the bar is very, very low. I’m not having to do a proper business plan. I’m showing that they understand the market. That they are favorably comparable to similar ones. And also that you do every now and then get a Blair Witch film, a Blair Witch project or a paranormal activity that can make 100 times its budget. And then they sell the investment in their film as a kind of like this is we fund, but it’s a lottery ticket. But when it does happen. So it’s it was very loose. I didn’t have to understand business school economics. I had to understand what numbers were there and how this film compares with all those films. I have numbers for. 00:09:56.90 [Julie Hoyer]: Mm hmm. So then as you continue to do these different analyses and writing and things, what have you focused on then on the thinking side? Is it more of that like psychology? Is it more classic business stuff? Is it heavier, actual like mathematics? 00:10:15.78 [Stephen Follows]: I mean, it’s all of that stuff. I mean, I never really learned to code. I learned PHP a little bit when I was at university, but self-taught, but I didn’t really use it. Excel was my main driver for a long time. And I used to get teased from a lot of scientific science friends, quite rightly, that I was only using that and it was very slow to do anything else. And so I but I would talk to people and I would try and just I don’t mind saying I don’t know. And I’ll go and ask. And after a while, you realize nobody knows or there are one or two things that you that are behind some sort of barrier. And if you chat to people, you can get that information so I can learn rules of thumb and apply them to data we do see. Because although I didn’t have any. Secret access or training. No one else has special training for this. This isn’t like being a doctor or a lawyer where there is a corpus of facts that you do need to know. And if you get wrong, you know, there’ll be consequences. This is a lot more subjective because it’s the arts and it’s the business of art rather than just business. So those people had to learn them somehow. And I didn’t have the resources or the data or their personal data. They’re the films they worked on. But people would show me stuff or tell me stuff. And especially when I started blogging. It people would come back. I’d never share anything private, never getting any trouble. But I would say some, you know, I’d love to know about this. And someone would say, hey, come have a coffee with me. And they’d show me stuff. And I’d be like, oh, I understand how that works now. So when I go on the next project, I can apply the same logic and know that it’s broadly right. And then somebody would correct me if I got something wrong. And so it was sort of just asking and doing and trying to be humble and curious, I guess. 00:11:53.90 [Tim Wilson]: Michael. How many times? Have you rebuilt the same analysis? Oh, that depends. Are we counting the times I said this time I’m documenting it? Absolutely. Then many, many times. That’s where super skills in Prism come in. You can take an entire analysis, including multiple steps, dependencies, conditions and checks, and package the whole thing into a reusable template. 00:12:17.13 [Michael Helbling]: Oh, so not just one query, the whole ridiculous journey? 00:12:20.50 [Tim Wilson]: The whole ridiculous journey. Run it again on new data or a completely different. 00:12:44.46 [Tim Wilson]: and they will move you to the top of the list. 00:12:54.08 [Michael Helbling]: Reuse the analysis and save the artisanal suffering. Tim, how many browser tabs does it take to answer one simple measurement question? 00:13:03.80 [Tim Wilson]: Apparently, all of them. STAPE, GTM, GA4, and at least three tabs I’m afraid to close. 00:13:10.51 [Michael Helbling]: Yeah, I’ve lived that life. STAPE’s AI Assistant is designed to help with exactly that. It’s built directly into STAPE and can connect to Google Tag Manager and GA4. 00:13:21.24 [Tim Wilson]: So you can ask questions about your containers, review GTM configurations, check server-side container usage, or see whether new events are actually showing up in GA4. 00:13:31.18 [Michael Helbling]: Yeah, and it can do more than answer questions. You can ask it to create STAPE and server-side GTM containers or even set up tags and triggers in GTM. 00:13:39.84 [Tim Wilson]: Basically, fewer scavenger hunts through menus. 00:13:42.70 [Michael Helbling]: Yeah, exactly. It is conversational, built into STAPE, and there’s no separate MCP server or technical installation required. And I mean, honestly. The best part, you’re not burning tokens 00:13:53.74 [Tim Wilson]: from your own LLM subscription. If you want to see what STAPE’s AI Assistant can do, click the link in the show description to learn more. Presumably, there’s a lot more data now. And at some point, we’ve got to talk about the numbers, because it’s just timely, because that’s kind of wild what’s going on there. But when you started, was there a point where you said, I need to start formally capturing or cataloging either what I’m getting one-on-one? 00:14:21.08 [Unknown]: 00:14:21.08 [Tim Wilson]: Anecdotally, versus what the data sources are? Did you come across a year or two in, and you’re like, oh my god, I just found a gold mine? Where did, when did you start structuring it? Because it sounds like you sort of became the guy who had kind of a unique level of breadth and depth in one angle of the industry. But that also seems like, how did you keep that organized? Did you find yourself kicking yourself three years? And then saying, oh, I wish I had done something more structured? Like, does that make sense? 00:14:56.37 [Stephen Follows]: Yeah, no, totally. And it is a little bit about how you create stuff yourself. I’d breach, I was led by articles back then. So I’d be like, OK, I got to write an article. I want to do one. Back then, I was doing one a week. So I’d create a folder for it. And you start creating, OK, there’s a data folder. And there’s an images folder for later on. And then I would create, you’d get, grab, gather unique data sets for each one. But after a while, you start realizing, oh, no, I can use that data set again. And I’d organized all of this. And I showed it to somebody after a year. Or two of doing it. And they’re like, wow, this is really organized. And I remember thinking, no, it’s not. It’s pretty basic. And I think that people who think they’re messy probably have a quite tidy house because their bar is quite high. So my bar of how to do this if I was paid or what I would tell someone else to do is a lot higher than what I was doing. But because I was in a field where no one’s doing that, no one runs away to do the maths for the circus. So I was one of the very few people doing this. And in fact, actually, I gave a talk at a business school early on. And someone came up to me afterwards and said, oh, you must. That book, I can’t remember how they mentioned it. But they mentioned it in passing that a certain book had obviously been very important to me. And it was a book called Category of One. And I was like, I’ve never read this. What is this? And they were like, really? And so I went away and read it. And it’s a very good old marketing book about the idea that one of the things you can do in marketing is create a new category, like light beer. And then you’re number one in that category. And he had concluded from my work that I had been decided to be the only one. I was the only guy doing data in the film industry. So therefore, I would be the best one, which is mathematically true. But a lot of this is I was the only one doing it. And in answer to the other half of the question, which is about like, so I would go out and gather data. Initially, I think the first article I wrote that really took off after three or four articles were doing it was I went through someone else’s work. I went through the British Film Institute, the public body. And they had done a huge statistical yearbook, which I was reading every year for fun anyway. And it was already kind of fun and well-written. It’s helping for my education and also for teaching other people. And I’d written 50 interesting things we’d learned. It was like an article like that. And then lots of people appreciate that. So there were bodies like that who were already sharing data. And then you have private bodies like IMDB and people like that who’ve done a lot of the work cataloging it, either as an organization or as a collection of people like Wikipedia. And a big step forward for me in the very early days was learning how web scraping worked so that I could go and take all of Wikipedia, put it into Excel, 00:17:20.29 [Unknown]: 00:17:20.29 [Stephen Follows]: And then I could go and do it. And then I could go and do it. And then I could go and do it. And then I could go and do it. And then be able to do everything myself. Because again, I wasn’t coding very much. So a big breakthrough was web scraping. I would talk to a friend who’d do it, or there’d be a tool off the rack. Or sometimes I used People Per Hour quite a lot where I knew what I was doing was way out of my capability, but it was also incredibly basic for anyone who knew what they were doing. So I was happy to give someone a few bucks, really a few bucks, like not very much. I might be like, hey, I put a post out saying, hey, I know, I believe this to be very simple because someone did it for me before and it’s just tweaking this. How much do you, what does someone want for doing this? And they’d look at it and be like, oh, give me 20 bucks. Or they’d build me a self-contained tool as a one-off use for 50 bucks. That wasn’t every week, but every now and then that would give me a large data set because I could run that, gather all this data, which wasn’t always taking data off other people’s things. It wasn’t that I was taking IMDB’s database, but it might be that I was taking reviews or other things that I could turn into data. And then the only thing I was ever sharing or publishing was my thoughts and the, the combined graphs at the end. So that data, I didn’t do anything with it. I didn’t give it away, sell it, all that. So I felt quite ethical that it was, although I wasn’t part of an academic institution formally, I was doing that the way a university might. And so I never got in trouble from that really. And from that, I was able to say, right, I went to this site and this site and this site, and I combined all those insights, and this is what I found. And here’s three graphs that are interesting. See you next week. And that seemed to be quite sustainable for the first few years. 00:18:47.44 [Tim Wilson]: It’s funny, because when you talked about like the ones, I feel like that once a week discipline, when I started blogging 2007, it was literally just a, I will do something once a week. And there was some nerves upfront of like, what happens after three or four weeks? Will I be out of things to say? Which I never was. I certainly watch, and this happens with podcasts and blogs and substacks now where somebody, they’re like, I’m gonna start doing this. And the first thing they write is, they’re gonna write this. They’re gonna write this. 00:19:20.36 [Unknown]: 00:19:20.36 [Tim Wilson]: They’re gonna write this. And then they write this like half of a book. Like they just empty their soul into it. And then the second thing is like a week later than they expected. And the third one never comes. So it does feel like you had that. I think even what I was doing, there was a lot, I had a lot of passion for like little tips and tricks in Excel and was never like the Excel God would never win any sort of Excel competition. But I was like, hey, I found this thing and it took some digging. But it seems like you had that mix of, I’m gonna hold myself to this discipline. It is something I care about. Was there a component of it? And instead of just having idle thoughts or exploring it, if I tell myself I’m gonna write it down by this date and explain it clearly, that will help with my thinking as well? Like you were consciously like building up your mind. Was there any sort of muscle on that front? 00:20:22.37 [Stephen Follows]: I think, I mean, I’ve definitely done my fair share of exactly as you said, half a novel a week late and then gone. And I’ve done a fair number of them. I don’t like that. I feel very bad about that. When I start a new project, I really want to avoid that. I think I want things to be interesting to me and I’m naturally a curious person. So I found the whole process of doing it was primarily for my own knowledge. I wanted to learn all of this and I would, the film industry, although it is a niche, I guess, within it is huge numbers of things to look at, you know, on screen, off screen, behind the scenes, all these different jobs, money. So every week I’d pick something and I deliberately would find something as far away as I could. So if I was looking at money one week, I would think about actors the next week. And so I enjoyed reading my blog slowly the few days before everyone else. And then the teacher in me, I always want to, I want to work out how things work and share it with other people. Like that is a deep pleasure of mine. What they then do with it, it’s up to them. Like I never thought like they should, I should have a hand on the teller of what they’re making. But I wanted to share this information. So the course of the week, so I’d publish on Monday morning. So the course of the week would be that I’d be thinking about it during the week and maybe I’d spend a bit of time gathering some data or I had something, but I would think about it. And then the deadline would be coming up and I’d be like, oh shit, I really should catch up. And so, or maybe on the weekend, I would do some work on it and get the, oh, all right, I got the thing now. I know what the thing’s about. And then it would be writing up on the Sunday or rushing it on the Sunday. And I quite enjoyed that Sunday morning, kind of like, okay, so I sort of have to do a summary of everything I learned this week about this. And I started to establish the question as the headline format that I then talk about the answer, but talk about how I went about, found the answer in a way that everyone feels like, oh, I didn’t know that, that makes sense. Oh, that’s good to know. And so I would enjoy that process of almost tidying up on a Sunday, of tidying up those thoughts and presenting them, queuing it up, and then that’s ready to go on Monday. So I found it useful, although it wasn’t the driver. It wasn’t that I was like, I’ve seen people that have calendars that want to have like a no broke, no, you know, a streak that’s not broken. It wasn’t that, it was just when else was I going to force myself to sit and organize those thoughts and actually write something up. This podcast does have an unbroken streak of, 00:22:32.64 [Tim Wilson]: we have one co-host who might have that motivation unhealthily so. 00:22:37.55 [Stephen Follows]: My business partner in my other business, which maybe we’ll talk about it first time, has got OCD. And I think that has been one of the best business decisions I’ve ever made because we are never late on anything, otherwise his world ends. And I have, like I want depth and accuracy and things like that, which he cares about, but only through me. So that can be such a good self-sustaining, team building thing, as well as a good place to put those thoughts if you have them. So yeah, I think good on you. That’s really good. 00:23:05.42 [Tim Wilson]: Clearly we got the wrong person on this episode because that’s the guy I want to talk to. 00:23:12.51 [Julie Hoyer]: How do you catalog and then like pick what question you’re going to go for each week? I know you said like you, I liked that you were saying you try to go for something as far away from the previous week. And it’s amazing just that the film industry does have so many different like angles and lenses you can kind of go after. But I mean, how do you keep track of what sparks like interest for you? And then how do you decide, is there like an official catalog? What do you use to prioritize? I am awful at that. 00:23:44.33 [Stephen Follows]: And in the first few years, I know maybe the middle few years before the pandemic, but after I got into a rhythm of this, I had to remind myself every time that I was about to do something to Google it first to find out if I’d already done it. Because like, I just wouldn’t quite remember. And quite often like the investigation is quite large, but the article you write has to be quite narrow and focused. And so you write one part of it up, but you know it’s sort of unfinished in your head or what was the interesting part for everyone else is actually not the bit that interests you. So you can write something, publish it and forget all about it. And so I have an image of it. And in more recent years, including last year, I forgot that rule and published an article a week after I published it, realized it was basically the same article I’d done six months before. Oh. About like, I think it’s about like when bad films make money or like what are the conditions under which some films that critics don’t like, why they still make money. And I wrote the question in two different ways. And like, I left them both up there because it doesn’t matter. But it’s, and I was quite relieved to find the same answer. And maybe people wouldn’t realize, but, you know, I was like, oh, I actually did this. So I did, when I have a random idea, I did have a Dropbox paper document, or I guess I still do somewhere. And I write it in there. Or if I had like a data set, I’d put it in a separate set of folders than the actual articles. And now, thanks to things like AI and things like that, I’m able to do things at much greater speed when it comes to the analytical part, not the writing, because that was still a human. So that’s the bottleneck. So I now have a work in progress folder that has about 250 projects done, but not written up. Wow. And so that’s, that’s the bottleneck. 00:25:17.86 [Unknown]: 00:25:17.86 [Stephen Follows]: So that’s the bottleneck. 00:25:17.94 [Unknown]: 00:25:17.94 [Stephen Follows]: So that is a challenge, but I also believe in the evolution of ideas. If it’s a good enough idea, it’ll come back again, or I’ll be compelled to do it. So it’s, because I have no boss, and it doesn’t matter that I forgot something, I kind of don’t mind. I wouldn’t suggest any, this is all terrible advice. But for me, it didn’t matter if I had a great idea and dropped it, because it would come back again if it was a great idea. And if it didn’t, that’s fine. But as I said, I don’t think, that’s not a professional way of doing it, if you’re a business. Who needs to work through a set list. 00:25:49.81 [Julie Hoyer]: Yeah. But I think it just speaks to the fact that, like you said, this is your passion. You’re able to do it in your way of working, and it’s worked because nobody else is doing it. And so it doesn’t really matter if you have a perfect system to prioritize in this setting. I am curious though, because you were saying, you know, the help of AI. I had seen a title when I was looking at some of your articles and it was about movie poster colors analysis that you did. And I immediately was blown away. I was blown away by like, how did you get that data? Like, how did you go out? I mean, you, I don’t have the exact number, but it was tens of thousands, I thought, of posters that you had like looked at or included in the analysis. And it was over time. And so I’m curious, how did you get that data specifically? 00:26:36.86 [Stephen Follows]: Because that one fascinated me. Yeah, no, me too. It’s one of those questions that I’m so excited to answer. Like I said, I’m a fan of my own blog. I want to read these things. I just am also happy to share them. Okay, well, that one is a really good example of something that’s become not an issue that would have been an issue before, which two issues. If I tried to do that 10 years ago, I would have faced, the first issue would be, how do I take a single image and turn that into data that would be color data? I would have to go away, which I often didn’t, read Stack Overflow or talk to scientists or read up about like, how does one do that? Like, what code do I need? Can I just hack together code that already exists? Because usually, I mean, what I’m doing to gather this data is not groundbreaking. So quite a lot of people don’t know that. So I’m not going to go into that. I’m going to go into that. 00:27:17.58 [Unknown]: 00:27:17.58 [Stephen Follows]: I’m going to go into that. But quite often, someone has done that in a different context and I can just take the best of it and adapt it. So the first of the two challenges would have been, how do I do that? And second of all, how do I do that at scale? Because how do I get so many images and how do I put, I think it was, I don’t know what it was, 50, 60,000, something like that. And then how do I make, within that scale question is, how do I have data for all those things? Now, actually, it’s incredibly trivial because first of all, you can go to AI and you can have a conversation and say, hey, I want to study movie posters. I want to study color. What kind of things should I read up about? What should I consider? And for example, in that one, it’s like, well, your mind might say this poster is very orange, but when you count the number of pixels, orange may be the second or third color because it’s against a black background. So then you start realizing you can’t just count the number of pixels because what we’re carrying, what we’re looking for is saliency as a human. So then you go, well, what should I read? And yeah, exactly. So I love that process. I love going from not knowing to knowing. So the first stage would be like, okay, color. So AI solves that kind of thing. Because then you say, give me the code. There’ll be a JPEG in this form. I’ll go to this folder and I want it to generate these numbers because of this conversation has told me. And the AI can say, oh, you want this algorithm now? And all those two you might consider. And then I go and read up about them and come back and be like, yeah, we’ll do the advanced one or we’ll do this one. But I’ll put in the notes, the caveat that it has this flaw or whatever it might be. So that’s the first problem. The second problem of scale is massively solved by technology being faster. So I have a fairly fast computer. One of those I published a report a few years ago about horror films that I did sell. And that brought in some money, not very much money, but a lot of money by my standards. So I bought a nice computer with a strong graphics card and a load of hard drive space and stuff. And so I was able to leave things chunking away. And as you know, with code, if it works for one, it can work for 60,000 just as easily. And the same with data sets. Like I already have metadata on all the movies. So the main criteria I use is the IMDB ID for a movie because almost every movie is on IMDB, not alone, but almost every single one. So with that as a unique key, I can go back, to my metadata data sets and say, I know this ID means this is a comedy. So when I then have lots of posters and I know the IDs for them, I can then say, well, give me all the comedies divided by, you know, into buckets by a year. And I would, up until very recently, that would all be manual with Excel. As in, I would just, it would have 60,000 rows. I was constantly fighting Excel, like a million rows or having over many columns and like hardware issues because I was really trying to get as much as I could, but I can write and whatever. Formulas were need to build like, okay, I want all that by year, by genre. And then as for getting them, there are lots of established data sets online, whether it’s Kaggle or just searching for stuff. I find that quite often there is a kind of quasi, so there’s full data sets that have permissions and whatever. Then the other end of the spectrum, there’s private data sets and copyrighted material and things like that. And I try and avoid all that stuff, even though, as I said, sometimes I go into that space, but I’m doing it from an educational point of view. I’m deleting the data afterwards, or I’m certainly not sharing it. I’m not telling people how to do it. I feel like I get a lot of leeway because the way a scientist would, like I’m largely being ignored, which is what you want in that regard. I’m certainly not commercializing that. In the middle ground, there’s quite a few. So like with color data, it might be that I need a really big image or it might be that I can deal with a small size that is fair use that’s on the Wikipedia page. And having already got lookups for films with metadata, and then also knowing the way to get to the Wikipedia page from a previous project, it’s not that hard to just go and download all those files and knowing you’re okay, and then running it through that program. Or sometimes it is just scraping existing sites that have this data. But the hard thing as well then is to make sense of it because now I will do many deep research from AI prompts to learn about topics. But back then I would just manually read through stuff and I might be like, okay, well, first, I’m going to look at the academic literature on posters. Has somebody studied this? Have they already cracked it? But also what is the history of the movie poster? And it might be as simple as the Wikipedia page. I don’t have time to do two weeks work on everything. So I might read the history of the movie poster and discover that’s interesting. Or quite often it’s data led. I’ll see there was a big spike in the 1980s and be like, why was that? And then I have to condense that down to one question that can then be answered in an article that isn’t way too long or sprawling. And that’s usually the bit that saves me. Because I’m like, well, big data set, loads of answers, not a lot of clarity, but there is one thread, one vertical throughout all of this. I can go from beginning to end. Okay, I will write about that. And I will let all this fragmented messy stuff on the side sit for another project or another time where I have more understanding. 00:32:04.07 [Julie Hoyer]: It’s nice to hear that you start from a well thought out question though. Because a lot of times I think we, as people that work with data, we see a lot of people say, oh, there’s all this data out there. So I’ll just like, look what data is available and decide like what it can tell me. And we talk a lot about going from the opposite direction and actually starting from an interesting question and doing the research before either collecting the data yourself or grabbing what’s available. But I also think that makes someone much more attuned to if the data that is available is actually, to your point about the color, like is the data available talking about the ratio of pixels or is it representing what a person picks up on visually? Because that would possibly give very different answers to the question you’re essentially trying to ask the data. So it’s great to hear some of your process that way. 00:33:00.46 [Stephen Follows]: Yeah, I try and keep a sort of layman’s point of view. I assume everyone’s a smart layman and I don’t assume they know any knowledge about the industry or about the topic. Cause I wouldn’t, I certainly didn’t 10 years ago and I may not have 10 minutes ago. So I try and, but I try and make it so that it’s a logical set of answering. And sometimes the joy is in the answering, not in the answer. Like, you know, if I’d said to you, they’re more blue than they were before, like, okay, that doesn’t mean anything. I can’t even remember what the answer was, but the process of breaking that down or looking for the correlations within that is really fascinating. So I quite often think it is the journey, not the destination. 00:33:34.09 [Michael Helbling]: Right, Steven. So what’s in my head right now is, obviously you’ve done so much research and so much digging in and analysis. How is the movie going experience for you? What happens when you sit down in a movie theater? 00:33:49.29 [Stephen Follows]: Well, I mean, most people who get into film find that the first five years after college is ruined for them because you see the structures, all of the obvious points, Chekhov’s gun, you know, you know, something’s going to happen. And you know, the, especially non-innovative films. So most films are following a very set structure. And once you know that you see it everywhere and it kind of ruins it for a bit, and then you learn to let go. Or if you’re- 00:34:15.59 [Michael Helbling]: The Wilhelm Scream, that sort of thing. 00:34:17.55 [Stephen Follows]: Exactly. And it takes you out of the movie, right? Cause you’re like, oh yeah, there’s that thing I know. Or there’s a reference to that. And a lot of friends who work in visual effects have had visual effects movies ruined for them because they’re looking around the edge of the hair and they’re like, I can see the green screen, that’s poor work. And you’re just like, yeah, right. But after a while, you kind of have to let go of all that stuff. And my partner’s always amused when people assume that I watch good films. I don’t, I watch trash. Like I’m not a cinephile. Like I like good films. Definitely. But I think I can definitely split the studying of the film industry from the watching of it. And very rarely do they overlap in any significant sense. 00:34:58.25 [Michael Helbling]: So what’s a good movie you like? Well, wow. What’s a favorite? 00:35:08.40 [Stephen Follows]: I mean, I’m a huge fan of the Bridges of Madison County and Meryl Streep and Clint Eastwood is just extraordinary. And I- I almost don’t want to say any more about it. I want to- someone forced me to watch it. I used to, with a friend of mine many years ago, go around to his house and we used to rent a video. That’s how old it was. And we’d agreed to rent something and he got- and he saw this on TV the night before and refused to rent what we were supposed to rent and said, no, we’re going to sit down and watch this. So I watched it under protest and it’s just incredible. But also there’s this local theater near me that does throwback screenings where they show older films. And I saw Speed in the cinema. I mean, I’ve only ever seen it on video or whatever. And it’s just incredible, timeless movie. Just perfect. Like, so I think I want to get lost into a movie. I want to get- I don’t really mind if it’s emotional or fantastical. I just want to forget the real world. Sounds darker than I meant it to, but like, that’s what I want. Like, I want to get lost in it. Yeah. No, I love that. I think that makes a ton of sense. 00:36:05.75 [Tim Wilson]: Do you ever find yourself- like if you’re watching Speed in a theater, like, like some kind of not the of the moment experience, does that- do you find yourself- either in or shortly after having that trigger any of those questions? Like, do you, you know, how many movies have featured a bus that doesn’t stop? 00:36:26.90 [Stephen Follows]: Or some- I mean- That’s a good example, actually, because I want to- I haven’t finished it yet, but I’m doing a video for my YouTube channel that I’m looking at- each time I’m looking at a different actor and I’m doing one on Sandra Bullock because I think she’s great. And I’ve concluded that most of her movies are Speed, whereby she is an every woman pulled in to a situation and there is a bus. In every monom- sometimes the bus is a boyfriend, but like in every film there is like- and she- every film she’s done is Speed, basically, in a wonderful way. And because she is the every woman brought into things and I love her for it. But yeah, so I quite- yeah, I do sometimes leave movies and have ideas for stuff. Another random one, actually, I’ve got a group of friends who I’ve been watching movies with on Tuesday nights on- we stream them, you know, and I’ve got a group of friends who I’ve been watching movies with on Tuesday nights on- we stream them, on Discord just to- there’s a- there’s about 30 of us, but we’ll probably end up watching about 10 of us every night, every- every week, because some people can make it, some can’t. And we watch great films and terrible films. And we started during the beginning of the pandemic as everyone started doing things and we just haven’t stopped. And we were watching trailers to work out what to watch next week. So we picked trailers randomly and then choose it. And we watched the trailer for Unbreakable. And there is a unbreakablemovie.com or whatever domain. And someone in the group said, I wonder if that domain’s available. And I can’t remember if that one is or not, but it got me down a sidetrack where I was like, well, I think I could work that out. So I found, I went through all the data films I have, and I checked what domains they used when the film came out. This is like 99,000 movies. And then I identified which ones were, which were like proper movie domains rather than like Warnerbrothers.com slash whatever. And that’s about 30,000 and no 45,000. And I found that about 15,000 of them are available to buy. So one in three movie domains that were big part of the promotional part of the movie are currently just, you can purchase. And so I’m doing a video on that as well, but I bought one. I won’t say which one it is yet for the video, but like there are some big movies you can buy the domain for. And like, that was quite the journey because it was an idle comment someone made in a trailer, like just watching films with friends. And it just kind of, I was curious. And I just thought I’d see when I fail. I don’t mind failing in the slightest. I don’t, I just don’t want to not try. So I just kept going. And I was like, Oh, I think I learned something I didn’t need to know. 00:38:47.34 [Michael Helbling]: All right. And quick counterfactual for you on Sandra Bullock. Yes. In the blind side, I would posit she’s the bus. Nice. Yeah, there you go. So if you can work that one in. 00:39:01.07 [Tim Wilson]: You’ve mentioned it in passing a couple of times and it’s weird. This is a little meta because you’re because you’ve kind of done the filmmaking. But you started as writing these videos that you now do are. Like, incredible. Like, I look at them like these are so these cartoons like they are. They clearly reinforce your I want to be a clear explainer and a teacher. But they’re like daunting from a looking at it from a content creation, like the handful of times where I said, I just want to explain this well in a video. And I spend weeks or months because I just want to do something clearly. So can you talk about that? I mean, partly the process, partly like why? What do you shift from writing and then adding on the little kind of short film version of the writing on top of it? 00:39:54.07 [Stephen Follows]: Yeah, it’s been something that’s been really new. I think I launched in February this year and people have been telling me to do video for a long time, but I don’t really want to be on camera. Like, I just it just doesn’t appeal to me in the slightest. And it makes my screen call a little bit. I don’t mind other people doing it. I just don’t want to. If I didn’t have that part of my personality, I’d be on like one of those people on TikTok being like, hey, let me do this. I’m going to do this. Hey, let me explain this to you. But it just doesn’t work for me at all. And I also don’t want to, as we talked before, start something and not finish it. And I’m aware how big commitment is to make regular videos. So for a very long time, I didn’t do any video of any kind. And I was just doing audio stuff, guesting on podcasts and I had my own one for a little bit and then a lot of writing. And a friend of mine, Ben, who’s a really good filmmaker, was pestering me for years saying, you’ve got to do it, you’ve got to do it. And I said all this and he was like, well, why don’t we do it together and we’ll do it as a cartoon? So it’s my, we write it together based on my research, but he’s the one leading the editing work, he’s doing the editing and I’m giving notes. And then the animation is a version of me. And so that, so it came from him saying, no, it’s crazy, you’re not doing stuff. Also, there is a business opportunity in there. Not that we’re making very much money from it, but as I said before, like, I think the money follows if you’re lucky sometimes. It’s certainly not a good use of time financially. But I’m very fortunate. Although I’m very new to YouTube, I’m coming in with, I don’t know, 500 published articles already. So I know the ones that have resonated, I’ve done the work. And also it’s, the same work can be used there that can be used somewhere else. And so there’s a lot of overlap. Ben is a really, really talented filmmaker and works really hard. And, but both of us also share a slightly anarchic sense of the world. We want to take serious things silly and silly things serious. And so we try and keep the madness of a first draft. And we’re just talking about it because he’s on the movie group as well. He set the whole thing up. And so when both of us are like, yeah, I wonder what domain’s available. Let’s buy one. All right. How many can we buy? No, it’s too expensive. Let’s just buy one. How would we rank them? Oh, well, there’s a good question. And like, we try and keep that madness so that you’re watching it thinking, what is this? And because a lot of my writing is, I’m not an academic. I’ve literally never, I’m really not an academic on so many levels. But I, my writing is a bit more academic than, you know, I’m not an academic. 00:42:11.34 [Unknown]: 00:42:11.34 [Stephen Follows]: I’m not an academic. I’m not an academic. I’m not an academic. I’m a bit more academic than perhaps what I’m like in person when I give a talk. And this is the opposite. And this is the anarchic kind of, it’s YouTube. Like, I want to get it correct because I care, but it does not matter. And it matters more. Did you entertain people? Did you, you know, introduce them to a new idea? So I’m not trying to achieve anything other than just have fun for six minutes. 00:42:33.94 [Tim Wilson]: You weren’t applying, you weren’t applying those early analyses of how to make, predict that these films are going to make money. You’re like, I figured it out. Cartoon. 00:42:41.55 [Unknown]: 00:42:41.55 [Stephen Follows]: I can tell you that I have made so many poor business decisions if you were to see my life as a business thing. But, um, but that’s fine though. I mean, Ben and I really enjoy it. It’s really fun. It’s mad. It’s, but one of the things that has helped us again, to sort of use AI as an example, we could not have done this five years ago. We’re not using AI to, to come up with ideas or to write anything, but we did spend some time building a animation pipeline because in the very first ones we were doing in February, all the test ones we did last year, at the end of last year. We use, um, Adobe character animator and we create these characters and have them, the character and then the mouth moving. It was very, very, very time consuming and also animating the graphs so that the lines come on or whatever. And this was taking a very long time and we were like, wow, this is not sustainable. But this year we’ve spent a lot of time writing code through AI to do that for us. So we now have a whole pipeline where we can take something I’ve said and put it into just put it, plop it into there. So we’ve got the character animator that we’ve built. It automatically reads it. It says, okay, well this point, the character should have his happy face or his sad face. These are the mouth movements. It generates a first draft, which we go in there and change where all these tags are and then hit generate again. And it’s generated a whole, I mean, it could do an hour. It could do anything at a time. And also we use 11 labs to generate my voice, not for the final piece. The final piece is all me. But when we’re doing a rough edit early on, when we’ve just got it all in text, it’s like, is this going to sound right? Well, let’s just see how long this is. Let’s see if it makes sense. And we, so we use a lot of stuff behind the scenes to help us keep the madness of the ideas. But the execution is like having a team of 20 behind us who can just do things. 00:44:26.98 [Tim Wilson]: Wow. It just feels like that’s like the AI lesson, like the, the, the human part of keeping the madness and the madcap and the, and then like, but I don’t want to, I don’t want to record the script 15 times over. So I hate 11 labs. Exactly. Like, it’s like, you’ve, you’ve, you’ve also managed to kind of find, and I’m sure it’s still evolving sort of the sweet spot of, of how to, how to use AI, but keep the human component. 00:44:52.43 [Stephen Follows]: That is the future. That is the future. That is what, you know, I read something yesterday talking about how writers are now deliberately putting in typos and stuff like that to try and show as human. And to some degree, you know, we buy handmade soap, there’s records, there’s, you know, the flaws are, it’s not that we want it to be bad. It’s just that we, that’s the sign of humanity. I, it’s complicated as to how much that will carry on in what way, but certainly we know that the fact that this is made by two humans with a perspective is the important bit. And it’s also why we’re doing it. I don’t want to create 400 of these because I could generate them from my video, my articles. I can take all 500 of my articles. I can auto generate the voice and the animation. I could do that today. And I could also probably even use, we’ve got a lot of clips and we’ve already embedded them and we can search on them with like close up of phones. All this sort of stuff can come already. We could do all of that. And it would be soulless and empty. 00:45:38.70 [Unknown]: 00:45:38.70 [Stephen Follows]: Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. 00:45:39.26 [Unknown]: 00:45:39.26 [Stephen Follows]: It would be being pointless. And why would I do it? Like, what a waste of my time. But for me working with Ben and coming up with something mad and then executing it and being like, ah, and then when someone watches it, they also go out, what was that? Like that feels a good use of my time and of the audience’s time. 00:45:54.72 [Tim Wilson]: So now that you’re kind of the guy or one of the guys who’s got built up all this, do you find, do you have people coming to you for the data that you’ve either created or like do you have to i’m trying i’m thinking i think it might be daniel paris with stat significant has like yeah he has like a a paywall to like you know access my data so that clearly kind of happened to him so are you getting people coming to you saying hey do you have anything on this 00:46:25.14 [Stephen Follows]: could i use it how could i get it okay oh yeah i mean uh 25 emails a week about that kind of stuff from the film industry like it’s quite a lot but i i almost never almost never give out that data because first of all it doesn’t come from me quite often if i’ve gathered it or scraped it from somewhere else it’s not mine so whether that’s a legal thing or a moral thing it’s just not mine it’s not really a question of me withholding um if i’ve done a truly original analysis on something or it is an open source data set then i’m much more open to it certainly if it’s for a good cause as well but i’m not i’m not going to be there to be able to support that data either so let’s say that i have taken wikipedia plot summaries and i’ve applied some kind of analysis to them and i’ve got a data set for it uh you can’t do that without a data set so i’m not going to be able to support that data either that’s fine but if i then like um i did a lot of that stuff with embedding uh uh in text embedding if i then have to explain to somebody what this is what model i used how it worked it’s a huge amount of servicing for that bit of data and then maybe they want it slightly different i can’t i can’t service that i just don’t have the time for that so i do try and help where i can but quite often uh very very rarely can i actually help the core way they want help for as well as the fact that most of the people within the industry that come to me want it as a silver bullet as a panacea to prove that the thing they already want to make is going to make money and it’s not going to and i’m not going to be able to prove it so i’ve got very good at saying no like i 99 of the time people email me for something i have to kindly say that’s not possible or i’m not doing it um and that’s fine like that i’m very happy for people to ask don’t mind saying no but i think i have thought a long time about linking to other people’s data like i have a huge data set from all the public bodies around the world and all of the data they have i’m like it’s i don’t know three or four hundred gigabytes and i could put that back online but again it’s not my stuff even public bodies like i’d be better saying hey go to the korean film institute and then maybe they change what their website is or how it works and i just can’t support that so i just say i can’t or individually i’ll say okay go here and have a look and go get it yourself so so maybe 00:48:27.21 [Tim Wilson]: that’s the not that i know where michael’s gonna be like we’re heading towards a wrap but the the and i don’t want to make this a downer but i’m sort of mentioned it briefly earlier can you talk a little bit about kind of the ai impact on the whole tale of the numbers and we’ll link to your post about it but to me that was kind of a oh like i had not thought through how ai could literally basically crush a data store i have a 00:48:56.01 [Stephen Follows]: very complicated relationship with that ai i use a huge i got four claude max accounts like and i really use every credit i am really using it at the same time i’m deeply worried about the future and where we’re going and what the effects of this are and i was very lucky i got an early access to gpt3 when in 2022 something like that um and i used it to do an art project with a uh an astrophysicist in sweden uh we managed to convince a hollywood producer to give us money to go and write a script and so they paid us the same amount they normally have pay a writer and we developed that in 2022 before chat gbt so i’ve been around that kind of stuff for a while and i and if there’s something we don’t know about this let me know in the comments because i would love to hear from you if you’re interested in such things as well i’ll see you next week next week And I feel a lot like Cassandra because I keep saying how much it’s changing things. I’m not saying automatically better or worse, but like the world everyone’s walking around. My partner’s not very technophobe. She wouldn’t say she’s technophobe or a Luddite, but she’s just like, well, I didn’t need it. And I feel like it’s two realities of like how much this is fundamentally transforming so much. At the same time, you go out and you buy an ice cream and everything looks the same. And maybe it will be the same in 10 years. I don’t know. But it’s the biggest change I think we’ve been through. It’s certainly at this speed as a race, as a human race. Right. And so I’m interested in telling that story. And it’s very hard to not just be shouting people. So this particular story about the numbers.com was a really good example that fell in my lap to be able to say, look, here’s just one example. So a bit of background. The numbers.com is one of the foremost box office tracking websites. And it used to be in competition with Box Office Mojo. But Box Office Mojo was bought. It was owned by IMDB. And then just before the pandemic, they put it behind, some of it behind the paywall. They sort of made the inshittification of that because it was now owned by Amazon. So the numbers became preeminent. And I’ve known Bruce Nash, who runs the numbers, probably as long as I’ve been in this stuff, because I probably went to him first asking for data. He’s British, but he’s lived in San Francisco most of his adult life. And every time I go to the West Coast, I go and spend some time with him and his lovely wife. And so I got to know him really, really well. Like we’re good friends. And he does a few things. He does market analysis for filmmakers. They sell their data in bulk to studios and to hedge funds and other. And then they also have this website called the numbers.com, which has public box office data going back years. Lots of really cool tools and things like that. In March this year, the site just went down and just wasn’t publicly available. And I don’t talk to Bruce all the time, but I reached out and said, is everything all right? And he kind of went dealing with some stuff. Talk to you later. All right, fine. And then. It was a good couple of months. I knew he was very busy because it was something we were supposed to do and he kept pushing it off. And so in the end, I did have a Zoom with him and I was like, hey, what’s going on? And we caught up socially. And then he was like, right, I need to tell you the story. And he told me what happened. And at the end of it, I said, look, this is really self-serving, but can I write about this? Because obviously we’re just friends. He told me off the record. But because it fits into this wider narrative, I think people should know. How do you feel about that? And he thought about it and was like, yeah, actually, he’s a big. Yeah. Advocate for free speech and for open data. And he actually took me to the Internet Archive. They have a church in San Francisco where the server is a little blinky light. He took me there years ago. So I know he’s this is part of his belief as well. So I went with him to write this long article about what happened. And there’s sort of two things that really affected him. One is the massive increase in AI bot traffic. So there was always search engine traffic that would slow down websites or at least cause problems. For servers. But the deal was that search engines would cruel this information and then send you traffic because they’d be like you’d search for something and then they’d deliver the page. And it was manageable. But about a year or so ago, they started most website hosts, certainly data rich sites, started to see a massive increase in the number of bots to the point to which I think there’s some study that I managed to find numbers for that’s in the article that talks about how many page views the bots would go on to compared to how many humans they send you. And for search engines, it was something like every 10 views. I can’t remember the ratio. It’s very, very low number. For Anthropic, it was 32,000 sites visits, site visits from an AI bot for each one human being they actually sent you as traffic. So it’s an overwhelming volume of traffic, you know, just swamping these sites. And so that the numbers were suffering from that. They did some clever stuff. You can make your pages lighter. You can do stuff with robots.txt. But also they put. A little thing in there that said, if you’re an LLM, you can license our data this way. And that helped reduce the traffic. But then this other AI thing happened, which is that Polymarket, everyone’s favorite slash worst prediction market, has markets where you can bet on the outcome of box office. And they use the numbers as the answer. There’s the kind of like that’s what will resolve this contract. And so somebody or some people, I’m sure it wasn’t just one, started using tools off the shelf. Like, I don’t know if it was Claude or Chad GPT, but it’s not hard to imagine. It’s one of those type things to search for vulnerabilities. And this is a 30 year old website with over 150,000 legacy pages. And it found some and it kept hammering it and hammering it. And I’m not at liberty to give all the information away, but it looks like and certainly my reading of it is that somebody with a basic AI tool broke into his website, took it over to try and front run the numbers to try and see before it gets published. Oh, just the last minute, you can see that film is doing better than this. Or maybe they published them on Mondays, but they’re putting them in on each day, Friday, Saturday and Sunday. I don’t know what they thought would be the case. And so it took down the site and they can’t turn the server back on. And Bruce is now sitting there thinking he’s got a basic version of the site back up. The rest of his business is surviving because it’s the other types of business. But he’s sitting there thinking and he was asking me about what do I do with a website in 2026? Do we even have a website? If I’m going to have another website, is it going to get taken down again? So what’s the point? Like, I can’t even get human traffic. So, yeah, a real sign of the times. But it’s a long read and it’s worth it because and it’s got loads of Bruce’s quotes throughout it because he was very generous with his time. If you’re interested in thinking about how the world is changing and how AI is affecting it in ways you may not think, it’s really worth reading. But give yourself 10 minutes afterwards before you interact with people because you’ll be sad. 00:55:28.42 [Tim Wilson]: Well, I mean, on the on the business side, there’s all this like, how do we show up? You know, all these companies saying. What do we need to do to show up? You know, the whole AO and GEO. Like, I want to show up in those results. Like, so it’s like, oh, it’s like, go counterpoint this. Yeah. Wow. I did head us towards a downer. All right. Good job. Yeah. Good job. 00:55:54.26 [Michael Helbling]: Yeah. Overall, this has been a fascinating conversation. And thank you, Stephen, so much for coming on. And talking about it. And I think it’s sort of universal, like a love for movies and looking at sort of the stories and the data beneath them. One thing we like to do on the show is go around and share a last call, something that might be of interest to our listeners. And Stephen, you’re our guest. Do you have a last call you’d like to share? 00:56:23.03 [Stephen Follows]: Yeah, I got a website, which some of you will know inside out and some of you may never have heard of. I hope you’ve never heard of it because you’re about to lose three or four hours of pure joy. Go to pudding.cool. P-U-D-D-I-N-G. Please say, yeah. P-U-D-D-I-N-G.C-O-O-L. They are incredible data projects. They make you think. There’s a new one up there about a load of menus from New York and around America. And what can the hash of menus tell us about America? You’re going to spend so much time on it. I love it. 00:56:55.44 [Tim Wilson]: It’s nice. Very nice. Remember when they broke down like Ali Wong’s first special or something? I mean, it’s just bonkers. It is. Boy, that is amazing. There was. In the vein of this discussion. Yeah. 00:57:06.50 [Michael Helbling]: If there remained any doubt, Stephen, that you were not one of us, you could dispel all of that. No, there was no doubt. Awesome. Very cool. All right, Julie, what about you? What’s your last call? 00:57:19.32 [Julie Hoyer]: Mine is a YouTube video from David Epstein. Actually, it was his one of his newsletters recently. But I loved it because it was sports plus science plus, you know, some fun numbers and stats. And like, that’s right up my alley. So it was why messaging? I see barely runs. And he just like goes through kind of this whole broad analysis and deep dives into psychology. And it is just I loved it. It was so good. So highly recommend. Very cool. 00:57:48.03 [Tim Wilson]: I read that. And then I watched one of the games. I was like, oh, my God. I mean, I follow more closely or like aware of that. And I was like, oh, oh, my God. Like, he really that’s wild. 00:57:58.90 [Julie Hoyer]: Yeah, I know. I was shocked. It was it was great. Full of fun tidbits. Yeah. 00:58:05.34 [Michael Helbling]: All right, Tim. What about you? What’s your last call? 00:58:09.08 [Tim Wilson]: I’m going to go with Dave. I’ve I’ve had Eric Sandisham articles before, but he wrote one called Three Kinds of AI. And it’s short. And he breaks down like predictive, generative, energetic, energetic, a little messier and talks about kind of gives a definition to me, like very, very useful because people say I. And I feel like we talk past each other. And he has. Some thoughts on like the ease of measuring and getting achieving ROI from each different type. But it’s his articles tend to be short and they tend to be kind of brilliant. And that one I have not stopped thinking about since he published it. What about you, Michael? 00:58:53.32 [Michael Helbling]: Yeah, it’s interesting. I sort of involves this conversation a little bit. I was reminded of a very favorite one of my favorite authors, Douglas Adams. It is. It’s a book, Life, the Universe and Everything has this definition of learning how to take on the knack of flying, which is learning how to throw yourself at the ground and miss. So it’s sort of like a repetitive process where you throw yourself at the ground and then try to distract yourself before you hit. And it sort of reminds me of the advice of like, yeah, how do you succeed in some of these sort of things? It’s sort of like that. And it’s sort of. It’s mind twisting to be like, OK, don’t try to replicate what I did, but just sort of like throw yourself at the ground and then miss. So anyways, and also just read, if you haven’t, The Hitchhiker’s Guide to the Galaxy Trilogy in five parts. I don’t know if there are books I’ve laughed harder reading than those. So anyway, so that’s my last call. 00:59:57.23 [Tim Wilson]: On the list of people we lost too soon because that guy’s brain. 01:00:02.41 [Michael Helbling]: It was different. Very different. All right. Well, it’s been great. And I’m sure as you’ve listened, maybe you’ve got questions or maybe you want to reach out to us. We would love to hear from you. The best way to do that is to reach out to us on our LinkedIn group or on the measure Slack chat or through email at contact at analytics hour dot IO. You can follow Stephen’s work at Stephen follows dot com, where there are articles upon articles for any interest you might have on exploring data. And movies. It’s it’s almost endless. So it’s like so hard to know where to start. You’re like, just find an article and dig in. You’ll be there for quite some time. It’s excellent. And Stephen, once again, thank you so much for coming on. It’s been such a pleasure. I think hearing people explain what they’re passionate about kind of ignites your own passion in a way is sort of inspiring and encouraging. And like, I’m not going to. Dig into film like you have, but like it kind of like reinvigorates me for the things I’m excited about. So thank you so much for sharing that. 01:01:10.27 [Stephen Follows]: Such a pleasure. And thank you for destroying my idea of how long an hour is because that flew by and it does not feel like it goes quick. 01:01:18.94 [Michael Helbling]: All right. And of course, please, as you’re listening on whatever platform, leave us a rating and review. And you can also reach out to us if you’d like a sticker for your water bottle or laptop. And you can do that on analytics hour dot IO. So. 01:01:33.03 [Unknown]: 01:01:33.09 [Michael Helbling]: Yeah, lots of ways to engage. And I think I speak for both of my co-hosts, Julie and Tim, when I say no matter what the box office is going to be on your next major dashboard report, keep analyzing. 01:01:49.07 [Announcer]: Thanks for listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at at analytics hour on the web at analytics hour dot IO. Our LinkedIn group and the measured chat Slack group. Music for the podcast by Josh Crowhurst. Oh, smart guys want to fit in. So they made up a term called analytics. Analytics don’t work. 01:02:13.79 [Charles Barkley]: Do the analytics say go for it no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 01:02:26.46 [Michael Helbling]: But it’s just because I’ve seen it and I thought it was an incredible piece of film. No, but I’m. Going to go back and see it. I’m I wasn’t sure initially if it was worth it. And then I saw it and I was like, now I need to go back and see it at IMAX. Yeah, I will go see it in IMAX. And then lastly, Stephen, we like to send a small gift to say thank you for coming on the show. If you have the ability right below this thing, this is mailing address in the document. There’s a little box. If you just could type a good mailing address for you, then we can send that out to you. That’s very kind. Oh, yeah. There I see. Say thanks. Yeah, that’s. And that’s we just want to say, you know, thank you for taking the time. It’s a. 01:03:07.32 [Stephen Follows]: Well, I hope you send it after, though, it’s in case I do it terribly. And then you can just send an empty box to me. We have we have two different gifts. 01:03:18.76 [Michael Helbling]: And you’ll never know. 01:03:21.69 [Tim Wilson]: Actually, I’ve never shared. I I was this close because I kind of keep the the warehouse keeps just boxes and they’re already sort of pre and then stuff gets just a little addition. And I. I was dangerously close. I. I was dangerously close to actually shipping an empty box accidentally once. 01:03:36.82 [Julie Hoyer]: Oh, my God. Because I just kind of. 01:03:37.90 [Tim Wilson]: Such a bold move. 01:03:38.98 [Michael Helbling]: Oh, well, thanks for coming on the show. Yeah, we giving you back what you gave us. Nothing. But. Yeah. No, no, no, no, no, no. 01:04:02.15 [Tim Wilson]: Rock. Red flag and Chekhov’s gun. The post #305: Personal Interest + Analytics Chops = Career? appeared first on The Analytics Power Hour: Data and Analytics Podcast .

August 18, 20261 hr 7 min

#304: I Can Haz AI?

Everyone’s posting their AI projects online like proud pet owners sharing videos of their cat doing something marginally impressive — cute, occasionally clever, sometimes a little cringe. And the Analytics Power Hour is no different! In this co-hosts-only episode, Tim, Michael, and Julie skip the thought leadership hot takes and just… compare notes. What have they actually built? What broke? What surprised them? From a custom GPT podcast librarian to a full-blown show production app wired up to Neon, Vercel, Resend, and about five other things Michael is only sort of sure he set up correctly, to a Gemini Gem that simulates a client interaction so realistically it raises your blood pressure in a safe environment — there’s a lot of ground covered. Plus: why AI-generated communication has a Stevia aftertaste, why deploying AI context across a team is way harder than it looks, and why the LLM will absolutely tell you what you want to hear about your Meta spend if you give it half a chance. This episode is also brought to you by Stape , your all-in-one solution for server-side tagging. This episode is brought to you by Prism from Ask-Y —your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show APH Librarian Custom GPT * Hermes Agent Vercel Resend Neon * Surprisingly reliable, but it was something of an exploratory flight of fancy, so we provide no guarantees of uptime. Photo by Eugenia Pan’kiv on Unsplash Episode Transcript 00:00:00 | Announcer: Welcome to the Analytics Power Hour. 00:00:08 | Announcer: Analytics topics covered conversationally and sometimes with explicit language. 00:00:15 | Michael Helbling: Hi everybody, welcome. It’s the Analytics Power Hour, and this is episode 304. You know, there’s no shortage of high-level thought leaders that want to explain this or that about AI, and I mean, we’re not a post. But what about boots on the ground? What are people actually experiencing when they’re using AI in their day-to-day? I don’t know that any of us claim to be AI experts, at least I didn’t check anybody’s LinkedIn profiles, but I don’t think any of us have updated them to that extent. Updating that? No. All right. There goes that claim. But we have spent some pretty significant time in the tools, and we’ve got a pretty good background in this podcast of data and analytics, and I thought it’d be fun to talk amongst ourselves about some of the experiences actually using AI, the good, the bad, then the just okay. So that’s what we’re going to do. We’re going to get into it. And speaking of AI experts, let me introduce my co-host, Tim Wilson, AI Thought Leader. 00:01:15 | Tim Wilson: It’s actually Tim Wilson, M-A-I, like a master of AI. I’m going to start putting that at the end of my pool. I like that. Yeah. I’m like, hey. 00:01:25 | Michael Helbling: Well, good. And Julie Hoyer, are you doing? 00:01:29 | Julie Hoyer: Hello. I’m good. 00:01:31 | Michael Helbling: I’m excited to hear some of these war stories. Do you get introduced to clients as an expert in AI? 00:01:36 | Julie Hoyer: Oh, thank God I do not. 00:01:38 | Michael Helbling: Okay. Is that an AI strategist? 00:01:40 | Julie Hoyer: Nope, not yet. 00:01:42 | Michael Helbling: In consulting firms, a lot of times as an expert, you sort of just get slung in as an expert in whatever the topic is about. 00:01:48 | Tim Wilson: And it’s like, yeah, that’s actually inaccurate. 00:01:52 | Michael Helbling: Let’s kick off the whole conversation by just chopping that tree down. 00:01:56 | Tim Wilson: All right. NPR or the Daily Show and our AI correspondent. That’s right. 00:02:00 | Michael Helbling: And I’m Michael Helbling. All right. So let’s dive into it. I think maybe a good kickoff is just set the tables, so to speak, because maybe let’s just talk a little bit about the types of AI tools we’ve used, what LLMs, just very high level, very quickly, maybe just go around. We’ll just chat about that. And then I’ll dive into more specifics. Yeah. So why don’t you go first? Well, I’m glad you asked him. 00:02:31 | Julie Hoyer: Flip those tables. 00:02:32 | Michael Helbling: So yeah, I’ve been using AI since whenever ChatGPT3 came out. But yeah, I would say obviously using a ton of Chat AI LLMs. So that’s GPT, Gemini, Anthropics, Claude. So a fair amount of coding on Codex and Claude. Have experiment with some other stuff too. And then on the agentic AI side of things, I have a little Hermes doing little things for me using GLM 5.2 currently. So that’s sort of the latest and greatest. So yeah, I publish code to GitHub. I do things like that. And by me, AI does that, which is, it’s true. I think my first GitHub commit ever happened after I started using AI. So yeah, it’s definitely it opened the door. It opened the door to a set of possibilities I had hitherto lain dormant. All right. Okay, Julie, what about you? 00:03:41 | Tim Wilson: Me. 00:03:42 | Julie Hoyer: Okay. My list is not as impressive as yours, Michael. But similarly, I’ve used Chat GPT more for personal stuff since it came out. Like I was interested, of course, like anybody, but more so through work and more of the stuff that we’ll probably end up discussing today. I’ve done through Gemini and then Claude. So those are kind of my two main ones. Again, the easy foot in the door, I think the place everybody starts. That’s where I’m at. 00:04:12 | Tim Wilson: Thanks. No, that’s good. Tim. So, I mean, so I platform wise, I’m kind of across the, all through the big three, the Claude, Chat GPT and Gemini. I’ve gravitated much, I would say I’ve gone deeper in the Claude stack and is so many people out there saying people who are just using it for chat, you know, they’re, they’re missing the boat. And so I’ve kind of pushed myself, I feel like in the Claude world, I’ve actually used Claude code beyond just using chat to help me write code. I’ve gotten into Claude code a bit. I’ve done some stuff with co-work, which has been, which is the stuff I’ve done with co-work has been mostly podcast related stuff. So that’s a little, a little meta, but I feel like I’m, I’m hitting a point where I’m starting to sort of understand the distinctions between those and on the Gemini front, that’s kind of like, I’ve been underwhelmed by it, but outside of chat, I think I’ve played with Gems, which I remember when Julie, you, I think last called Gems multiple years ago, you’re like, Gems are cool. And I think it was another like year or two before I actually, I was like, dealt with one. I wrote down like, what are Gems? 00:05:31 | Julie Hoyer: Gems? 00:05:32 | Tim Wilson: But I mean, just a quick one, which isn’t even really in my list, but speaking of Gems, Stern at marketing analytics summit talked about, talked about AI, using AI for like training junior analysts and how to like engage with, with stakeholders. And I was, I, at the time I had a reaction that was like, but the humanity, how do you learn to do this? And since then I’ve had the opportunity to work with a couple of Gems that are purely there to simulate client interactions. And it’s been, even though I can read the instructions and I know what’s going on, like the realistic level, even to the point of punching record, like, so I’m like, Gems Stern, I know you regularly listen to this, I’ve now like done that. And I am converted that, wow, that’s a hell of a tiny little application. But it, it, it’s really impressed me. 00:06:32 | Julie Hoyer: Yeah, that situation made me sweat. The one that Tim, you like took the base one from, I can’t remember who made the original, was it Ryan Dupont? Ryan Dupont spun up the original one at further, and then you took it and tweaked it and made like that second iteration. But when you sent it to us to try out, I was like, whoa, I’m feeling like the pressure respond. 00:07:02 | Tim Wilson: It was good. That was like a wild little light bulb to say, oh, this is like really raising stress level, but in a safe environment. And the way I was like, ooh, I’m on board. So that was a very. Eye-opening one. But sorry, I went to a specific example that wasn’t even on my list. 00:07:19 | Michael Helbling: But yeah, Tim, save it for the next step of the process. 00:07:24 | Tim Wilson: It’s not in the outline. 00:07:26 | Michael Helbling: That’s right. No, well, that is what we’re getting into next is sort of like, OK, so let’s talk a little bit about projects we’ve done. And you kind of mentioned, Tim, the podcast, and that has been pretty fertile ground for a couple of us to try, you know, what I feel like is a low stakes way to try stuff out because it’s sort of like, OK, it’s not my business. It’s not a client. It’s a podcast and we’ve got real work that we’re doing. But like if it totally fails, like we can just throw it away and nobody has to know about it. But some stuff has actually stuck around. Like, and I think that’s kind of interesting and maybe informative in a way of like, yeah, if you’re not started yet on specific things, could you find something even sort of outside of your work that still would be a good use? 00:08:15 | Tim Wilson: Well, can we talk because this was over a year ago that that was when I headed down the path. I was using ChatGPT to help me write Python, Python’s not my strong suit, but I needed it because I was basically trying to save us some money and shorten some time frames on the transcription. So it started with like one Python script that ultimately got pushed into Colab, but it was purely like, help me write a script. I’m taking the code you’re giving me, putting it in VS code, going back and forth, kind of following along with what it’s doing, which is kind of a chat based version. So that was like, I don’t know, that was vibe coding because I was generating code that is still Python code that we run in Colab and we have three or four of those for different reasons, different purposes, but you did a different form of vibe coding for the podcast when it comes to actually developing an app. Like is it worth having you talk about that? 00:09:14 | Michael Helbling: Yeah. I mean, well, there’s a couple of things we could talk about, one of which our listeners can actually access, which is the very first, this is the very first coding, one of the very first coding things I ever did with AI, which was back at the end of 2025, which is crazy because that feels like such a long time ago, but it’s really not that long ago. It’s like so long ago, it was like a year ago, no, it was eight months ago. But no, we had all the transcripts from the podcast over the years, and I just started thinking about the fact that’s like, oh, well, AI is really good at reading through docs and stuff like that. 00:10:00 | Tim Wilson: I was like, wouldn’t it be cool to create a little AI library for the podcast using 00:10:07 | Michael Helbling: all the transcripts over the years? And so that got the idea started. And then I did a company hackathon with stacked analytics. We all went out and did AI hack work for a couple of days. And so I picked one of that as one of my projects and built a little custom GPT that basically the work or the underlying stuff behind it was sort of giving me a chance to learn about concepts like RAG and VectorDB and stuff like that, which are sort of organizing concepts that AI kind of access their chunks up data, probably not as relevant now the way that context windows have expanded and things like that. But for that time, it was kind of useful to figure out, OK, how do I get all of the transcripts in one place? So in this case, a Google Cloud storage bucket, and then how do I move that into a VectorDB? So I use Cloudflare to do that. And then how do I move that back across into a JSON schema that a custom GPT can talk to? And so building all that out and building out the flow of that was really the hard work. But it was kind of very useful because any time you’re dealing with permissions in Google Cloud, if you’re not super familiar with that platform and setting those up, that’s a good thing to learn about. You definitely want to know how that stuff works. And it was good. It was a really good experience because it took a long time to figure out, but it actually 00:11:40 | Tim Wilson: then produced something that’s like, hey, that’s actually pretty useful. 00:11:43 | Michael Helbling: And actually, if you go to ChatGPT and you go look at GPTs, you can find it as the Analytics Power Hour Librarian, and it’s available right now. So you can actually go find it and then ask it questions about the podcast, which is really circular because this right now will be recorded, transcribed, and then available at some point in that custom GPT. Weird. Tim, your company launches a new checkout and conversions suddenly drop. 00:12:15 | Tim Wilson: I mean, what’s your first move? Open 17 tabs, blame Safari, and schedule a meeting called Tracking Investigation, Urgent, or try 00:12:25 | Michael Helbling: this one on, connect the AI Assistant you already used to your Stape account through the Stape MCP server. 00:12:32 | Tim Wilson: That sounds less traditional. 00:12:34 | Michael Helbling: Yeah, but you could ask it to check the server side container, validate the tracking domain, compare usage before and after launch, break down activity by browser or client. 00:12:45 | Tim Wilson: So it can help answer, did customers stop buying or did our measurement just stop measuring? 00:12:51 | Michael Helbling: Exactly. And you can also review daily traffic, see which domains are generating requests and check for unexpected usage or possible surcharges. 00:12:59 | Tim Wilson: Fewer tabs, faster answers, and maybe that urgent meeting becomes an email. Well, let’s not get carried away. Fair. Click the link in the show description to learn more about the new Stape MCP server. Michael, what if your dashboard could do more than just sit there looking colorful and ask everyone to interpret it? Excuse me. That dashboard took six months, 14 meetings, and at least one resignation. Oh boy. Well, Prism’s new app builder turns an idea into a complete interactive application right inside your workspace. 00:13:38 | Michael Helbling: Okay, so I just describe what I need and Prism builds the app? 00:13:42 | Tim Wilson: Yep. You can review the spec, you can edit it, and keep refining the same app in plain English. 00:13:48 | Michael Helbling: Oh, so no rebuilding everything because someone asked for one tiny change is somehow requires a new data model? 00:13:56 | Tim Wilson: Exactly. You can also share the app with anyone through a link. 00:14:00 | Michael Helbling: Oh, useful, flexible, and no 12-week BI backlog. Frankly, it’s a little disrespectful to the traditional process. 00:14:10 | Tim Wilson: Well, Ask-Y is looking for people who want to replace a BI dashboard with an application without handling the migration themselves. 00:14:17 | Michael Helbling: Oh, so point the agent at your dashboard, Ask-Y reads the logical layer and builds it as a flexible, extensible application. 00:14:26 | Tim Wilson: That’s it. Sign up at ask-y.ai, that’s ask-dash-the-letter-y-dot-ai. Use code APH to jump to the top of that wait list. Boy, I like the sound of this. Give your dashboard a promotion. It’s been through enough. Which is, brings up, because like you built it, you kind of hacked it together, you’re like, hey, guys, I built this, and then like, and we were playing around with it, like, this is so cool. And then the immediate question was like, well, is that going to, is it going to stay updated? Because it’s like, do we want a snapshot? And that then sent us down a path, which I think is also representative because then the work I was doing on the transcription and needing to get, I guess I already had, I’d already done some stuff where it was going into markdown files with historical stuff. But then it wound up needing to move into a different process that was a combination of co-work and Python scripts to say, oh, we’ve got to coordinate between these two things. Like, I mean, literally just today, I just processed an episode and part of what it does is it, there’s a, there’s a handoff like the end of this one process is going to drop an additional markdown file in that folder. And then the GPT, that process you built is going to be looking and picking it up. And you were actually surprised a couple of months ago, you’re like, I don’t even think that’s working. Yeah. And then I was like, what’s the most recent episode? And then it get pulled up and like, it’s working. 00:15:56 | Michael Helbling: It’s going. Which is great. Because I didn’t build the next thing on top of that, which is like monitoring and data freshness checks in my GCP storage, which you would do if you were building something for a data purpose. And have built for other things. 00:16:11 | Tim Wilson: So yeah, that’s a nice thing about, yeah, podcast related things. You can just sort of work on it when you want to and then not update it later so that Tim 00:16:24 | Michael Helbling: gets really frustrated. 00:16:26 | Julie Hoyer: Yeah. 00:16:27 | Tim Wilson: There you go. 00:16:28 | Julie Hoyer: Speaking of the permissions part that you were running into, Michael. So one of the things I most recently tried to do, honestly, this was inspired from our conversation with Rob Colley, actually, recently, where he was saying where the data builders and where the people that are going to end up going in, again, using AI for more than just the chat and doing more than just analysis in it. And so I was trying to get into more of like, could I go use cloud code? Could I build an interface, an app, something that refreshes? And so from a lot of what you guys were just saying, I was inspired to go and try it for this learning community that I run at work every Friday. It’s called The Guild. And we have 183 recorded episodes or at least sessions that have happened that we have like a drive folder for Tim, which I was like, I hope Tim is proud of that because Tim was working with me when we were running that together. So anyways, I decided first I was talking to Claude just chatting and asking, you know, this is my situation. This is what I want to try to do. Can you help outline some of my options so that I could take all of these recorded videos? And there’s like content and decks in there. It all lives in a spreadsheet that we maintain and keep up to date, but also then it links out to all the drive folders where all this content is. And it gave me a couple options and I wanted it to be for searchability and like discoverability. So not only somebody to say like, oh, are there sessions on this topic or these key words, but could it also surface some things? Maybe they didn’t think about that could be relevant. So I was really excited about that. But as soon as I went to try to start like building it, it kept like throwing this error. Suddenly I was like giving it a link to the spreadsheet. And it’s like, oh, I can’t build off of this because I can’t publish because the account owner is different than the spreadsheet owner. And like this weird stuff, I’m like, what are you talking about? So then I am asking it, where do I find this out? Who owns the account? It’s my account. I don’t understand. So anyways, I find out that it’s the leader of IT from our parent company is the name on the true account. And for some reason wanting to publish this up to date like app is different than all the other ways I’ve been using Claude and giving it drive links. So that’s been like my big stop gap. And then I was even asking it like, is it appropriate to ask your IT leader to give ownership of the account to me so that I can publish this? And I was like spiraling. 00:19:06 | Tim Wilson: Help me draft an email. Help me. I’m at that point. I’ll draft an email for you. 00:19:11 | Julie Hoyer: Yeah, but I’m really hoping I can do it because to your point with the librarian, I’m like, we have so much content here that people never go and look at, but there’s so much applicable stuff in there that I’m pointing people to it constantly. Like, oh, there was this one time we talked about it. Here you go. 00:19:27 | Tim Wilson: But you said early on, like, which I think I’ve now learned, which is a little bit of an adjustment for me. I’m used to like, oh, I know, I know how whatever this thing is works. So I sort of have the architecture and here’s how I’m going to plug in the pieces. And because I’m still learning, I’m finding myself often saying, this is the end result and pushing myself to like articulate the outcome that I want. And then I will sit there and say, now, I think I could approach it this way or that way. And, you know, you tell me, like, actually, I had a case where I think it’s pretty known out there, like fax and feelings is kind of drifting away. And so Squarespace was going to charge us a big old fee to re-up in a bit. And I was like, oh, well, we’ve already stripped the site down to just be like a splash page. So couldn’t I just it’s like, it’s just a simple page. Can I just have, you know, Claude helped me recreate it as a static HTML. I’ll put it on Netlify. It won’t cost me a thing, whatever. And so I said, hey, can I do this? Want to walk through it? What are the cases? And then it brought up a whole bunch of really good points about managing the timing on the DNS cutover that I hadn’t thought about, which then sort of gave me like a much better plan before I kicked off to do it. But then I also learned like having it vibe code, trying to look at screen captures and inspect an existing page built on Squarespace, which has all sorts of added bloat on it. It was I still want to be doing a lot of inspection and CSS tweaking. Although even in the middle of that, I kicked Claude, I kicked co-work off to go do some like dom scraping to to suss out some some settings for me. So that whole like, don’t just assume what you’d know, like the chat ones are really nice to say, what might be the options? And can I use, should I use, you know, Claude code for this? Or, you know, should I use co-work? And at times it’s come back and said, well, yes or no. And given the pros and cons. But when we talked to Rob Colley, a big thing he was pushing was like you, you should go and build apps. And it’s kind of funny being on the podcast with Michael, because I think we all would acknowledge that Michael, you’re kind of tend to be a couple of steps ahead of us, generally, when it comes to what’s the new things that are doing. So you when Rob was talking about that on that episode, I was like, oh, yeah, that’s what that’s what Michael did with, like, literally building an app for us to do our back end workflow for the show. Yeah. 00:22:16 | Michael Helbling: So what was interesting about that was obviously, like running a podcast, there’s all this stuff in the background that we do, which we don’t need to get into. It’s not that fascinating. 00:22:25 | Tim Wilson: No, let’s talk about it. 00:22:25 | Michael Helbling: I don’t want to document this as a very detailed process document. 00:22:31 | Tim Wilson: I wasn’t going to go here all night. It’s right. It’s all been transitioned into runbooks and co-work. Yeah. 00:22:37 | Michael Helbling: But but the thing was, is like the idea was, how do we both expand the capability for everybody to be part of that process, as well as take some of the stuff off of Tim’s plate? I don’t know that we’ve been effective at accomplishing either of those two goals. But the idea was to start a start a process and also like experiment with AI as a result of that. But a couple of really interesting learnings from that. One was I spent a lot of time up front writing documentation of like what I thought this app should be able to do. And actually was pretty unhappy with my design, if you will, or the architecture the first go around. So I was using chat GPT to kind of like kind of help me write those and like think it through. And it was just AI tends to do this thing where it kind of latches on to ideas and it’s real hard to get it to let go of them. And then it sort of gets all in the rest of the stuff and it gets, you know, it’s sort of like a what’s that thing where like when you’re cooking something, it changes state. So like an egg is cooked, you can’t uncook the egg, right? It’s so it’s sort of like that feels like that with AI sometimes. Like something gets in there and you can’t uncook the egg. It’s now like a different form and it’s just really in there with your ideas. And you just have to like tear it up and start over. So I was at that point and so then I switched over to Claude because I was starting to use that a little more and I was like, all right, this is a good first step with Claude. And I used us a tool to design it and using the design of it to do that. But the other thing I thought was the kind of interesting was it actually took me back into a learning process about specific things because one of the pieces of the app is an integration with our Slack our Slack channel in the podcast. And I realized like as I was thinking through like how this would work, I really didn’t know what was possible with Slack integrations really. Like I’ve seen some but not a lot and I’ve not done any development behind it. So like it was sort of a blind spot about what here’s what I want. But like I’m not sure how how much this is or is not should interact. And so working with the AI to kind of like even educate myself a little more about that was actually part of the process. So it’s sort of like you hand stuff off to AI to go do. But actually you’re getting educated a little bit along the way on this as well because while you kind of know like what you’re trying to get to, it’s sort of like, well, do I even know what’s possible? And that’s actually pretty critical learning because then that shows you like, oh, if I don’t know what’s possible, like I’m not really the AI is not really extending my capabilities as far as they might be able to go if there was a more experienced hand. And this comes out, I think, when you do other kinds of work with AI. I was chatting with a CEO of a company a week or two ago and they’re like, yeah, I really love using AI and it’s great. But I feel like it’s not really ever helping me to like innovate that next step. Like it’s not helping me kind of like go that one step further. And I was like, oh, yeah, I think I get what you mean. So AI is it’s really it was a great experience. The app works really well. It actually looks pretty good too. 00:26:04 | Tim Wilson: Can I ask about because it’s it’s a it’s a little bit of a mystery to me. Like I get the the coding part, but like ultimately, this is an app that is hosted on Vercel. Like what is the actual and you’ve done a few iterations because we have like sort of a feature request and bug tracker. And so I guess two questions. One, the actual I get when you have code, I’ve got it when I’m in our studio or I’m in VS code, or I’m just asking for it to generate and that’s code. But when you have what is it, what are you actually doing to get the code run and deployed somewhere? That’s question one. Question two is when you’ve made updates, is it discipline that it’s touching specific sections of the code or is it kind of going we in like regenerating from scratch? And this is like reading Ben Stansel a year or two ago and the way it was being described. So that’s a two-parter question. 00:27:09 | Michael Helbling: So I don’t understand the whole thing I learned at the hackathon. Well, and before that too, but the hackathon we did, which had better developers than me at it, was I learned more about how to use GitHub. And so I use GitHub pretty heavily. So basically when I do a coding project, basically my process is, and hey, listen, I’m not claiming anything here. So like this is totally whack and broken. Well, you let us know in the comments, but this is what I do. I start a local folder. This is where the code is going to live on my computer. And then as it starts to develop into something, I then go create a GitHub repository. And then that code starts to live on that repository. And so basically for small projects like this, it starts locally and then moves to GitHub. And then GitHub becomes the versioning and the control. 00:27:58 | Tim Wilson: And then from there, OK, I’m just going to put it, put it, put it a little, let’s just call out that Michael did not explain how Git works well on that. 00:28:06 | Michael Helbling: I don’t know how Git works very well. I just want to put in the qualifier. Yeah, no, I already qualified it. I was like, I don’t really know. But he didn’t say it might be whack. Yeah, exactly. He said it might be whack. 00:28:20 | Tim Wilson: And what I mean by that is it’s probably whack. But that’s the code. I mean, I published from GitHub. Yeah, two GitHub pages for like web. 00:28:29 | Michael Helbling: But how do you then when you when you start up? So then basically, there’s sort of the DevOps piece of it, which is how do you take code that lives in GitHub and is maintained there in a repository and make it an app out on the web or wherever. So that’s the DevOps piece, which is where you use something like we in this case, use Vercel and Vercel basically can go connect to your GitHub, get the code, deploy it. And basically any time you change GitHub, it automatically deploys to Vercel. And so you have an automatic deployment process. And so the Vercel interface stores all of your environment variables and all the stuff that makes all this work and all the API calls that go in and out. And so this app basically connects to resend for email. 00:29:15 | Tim Wilson: It is the database living and where’s the actual database 00:29:21 | Michael Helbling: where the database is neon, which is the got bought by Databricks. So yeah, that lives out there. And so basically all this stuff is sort of interconnected already. You basically when you write all this code, you’re basically connecting all that stuff via API keys and stuff like that. And then the app knows when to call. 00:29:40 | Tim Wilson: So Claude told you, OK, go make a Vercel account, enter these. Here’s how you can go make a neon account. 00:29:46 | Michael Helbling: And yeah, basically, I mean, you can say to it, like, I want to use this versus that, you know, so it might come to you and be like, well, this is an architectural decision. Do you want to use SuperBase? Do you want to use Neon? Do you want to use Convex? Like you can kind of say which one you want to use or you have one you need to use to say that. So it’s not like you can’t. You can just use whichever one. So like there’s either other versions like Vercel is one example. But I think there’s many others that do the same thing like rep lead. I think probably does the same things. So yeah, since I know absolutely nothing, I just went with whatever Claude said. So that’s where it lives. 00:30:22 | Tim Wilson: Well, so I mean, so that that’s helpful because I yeah, I’ve got things that are published to GitHub pages and to notify. But I’m generally it’s just like static stuff. Same thing if I if I do a new push to GitHub, GitHub is going to it’s going to detect that and it’ll redeploy and kind of refresh it. But I was like, but that’s always been like, that’s just rendering Korto or our markdown or Hugo. But so it’s but that’s really helpful because I was like, OK, but you’re extending it saying, well, there might be two or three different systems that all have to kind of play together. 00:30:55 | Michael Helbling: Yeah, in the case of that app on the back end, there’s about five or six systems or API calls that go out. So like we have API calls out to different LLMs to process things within the app. It reads, you know, we have an incoming email and I look through those to evaluate, like, is this a good pitch or is this not a good pitch, you know, those kinds of things for us. So those are all things that are happening in the app. And then the email processing itself goes through a company called Resend and, you know, so on and so forth. So there’s a bunch of those little things. But basically, all of those decisions get made back at the beginning when you’re defining what is this thing is going to do and how does that work? It’s like, well, here’s what happens. We get topics from the website. We get topics from emails. We get topics from our Slack channel. Now we need to bring those all together in one place so we can evaluate those topics and and make decisions about them. 00:31:49 | Julie Hoyer: So what you’re saying is my earlier app library for this guild is laughable because I have a lot of steps I’m going to have to still go figure out. Well, I wasn’t even thinking like, oh, I’m going to have to probably connect it to GitHub and do this. But like, I don’t know, the example you’re saying is giving me thought of like, I have quite a few pieces to put together in this puzzle. 00:32:12 | Tim Wilson: There’s a nice little nice little fable anecdote that you have when like you asked didn’t rather than it was a fable. What is it? Yeah, fable that you asked to say, hey, could you evaluate this? 00:32:22 | Michael Helbling: Is this kind of pretty low level authentication path? Yeah, when fable five came first came out, I had to look at the code base for that app. Not with anything. I was just like, I’m thinking of making some changes. Could you evaluate this code base and give me your thoughts? And so fable is pretty well known as being like a pretty high end AI or like pretty state of the art right now. And it immediately identified three security flaws in that app, which we then patched immediately. So it was like, oh, yeah, well, before we do anything, here’s three things that are broken about your app. I was like, well, your little brother is the one that coded it. 00:33:01 | Tim Wilson: So we escalated that because this is a podcast. You can bring the podcast production process down. 00:33:10 | Michael Helbling: Bring it to a screeching halt. I don’t know that was actually what was the problem. But there were like a couple of things that definitely should have been fixed and we did fix them. But but you know, that’s that’s sort of an emerging thing. And also one of the things like, you know, I’m over here doing experimentation with things like Hermes, which is an agentic AI that kind of is like doing ongoing AI tasks for me on different things. And like this is actually an area of big concern, which is the security piece of that. So like setting that up is I’m super cautious about that because like that doesn’t live on my local network. It lives far away on an external box that is not attached to me at all. It cannot access certain things. It cannot write things in certain places. 00:33:56 | Tim Wilson: So basically I keep it pretty pinned down for now 00:34:02 | Michael Helbling: because I just don’t as I learn, I don’t want to be exposing like my work and all these things to an agent that may then just go dump all that information on the first, you know, random hacker that gets the opportunity to kind of prompt inject it with something that it thinks is cool. So like there’s some real risks out there with AI, too, which you know, I’m trying to be cautious of, but still also experiment around and have a lot of fun. 00:34:30 | Tim Wilson: Yeah, I mean, it’s what it’s the luxury like it is kind of nice sandbox. We’re not having a crunch a bunch of data, but there is a lot because what I tackled was I said, oh, we have this lengthy 40 page document of our process. That should be something I have a background as a technical writer. I’ve kept it maintained, throw it in and let co-work run with it to do everything because we record a little bit of stuff happens. Things go off to Tony to do the editing. He sends back like a final file. And that’s like the starting point for a whole bunch of stuff that has to happen. And that’s documented in the steps. And we had several of us who were like, we can do this process. I do it most of the time, but we had a couple of other people with fallback. It’s a Google doc knew how to do it, threw it into co-work. And a couple of things have emerged like one, how many steps there are that the human in the loop, partly because, oh, you got to go click somewhere that it’s just not going to go click on its own more. So I need to check in and have you, you know, pick which tags we’re going to put on the show. I’ll recommend these or these the right ones. Or these are three options for the show bar because frankly, I don’t think the show we’ve written enough by hand. We have enough training. We can now use it to write that copy. And it is now a the two things that have kind of emerged. One, what used to be an hour, an hour and a half of focused time that was not enjoyable at all and subject to to me making mistakes by not completely following the process. Now it’s like an eight hour eight hour duration because it’s super slow cranking through some stuff. And I just have to wait to hear the little beep and then go in and check, you know, what I need to rely on. And for some reason, I’ve never been able to get caught dispatch to work because I was like, this is the perfect application of I’m going to go meet someone for coffee. This thing’s running. If it just needs me to say go, never could get that. I have not gotten that to work on my phone. So one that was interesting to say, yeah, this is kind of a grind of work to do, but it’s stuff that a human has to do. So that was kind of validating. The other is it’s wild to watch it take different paths to do the same task each time. Like there are times where it’s like, well, that didn’t work. So I’m going to try this other thing. Well, that didn’t work. And I watch it be super inefficient as it finds its way to do the thing. So it still has that it has that probabilistic LLM thing going that it’s not a rules based. It’s more robust. So if a UI slightly changes, it’s probably going to figure it out. But on the other hand, where exactly it sort of runs into problems is kind of a moving target each time. So you could do Tim, as you could write a skill for it. 00:37:40 | Michael Helbling: So you could give it. That’s good. Yeah, you can give it the steps. 00:37:43 | Tim Wilson: Well, but I’ve given it the steps in code work. Like it has a run book that is like, do this, then do this. I mean, yeah, but that’s very detailed. 00:37:53 | Michael Helbling: That’s different than sort of like process steps. I don’t know, maybe not. Maybe it’s not a good application. I don’t know. 00:37:58 | Tim Wilson: I’m just actually I should probably go to Claude chat and say, I’ve done this and this is the challenge I’m running into. And how could this be? But is this something skills would help with? I love doing that when I don’t understand something like, hey, would skills work for this? And it’ll come back and be like, yeah, you ding dong. Like, why weren’t you why weren’t you using skills from the get go? That’s a good skill segue because you’ve been playing around with skills. 00:38:27 | Michael Helbling: Yeah, I yeah. Well, because yeah, because in a couple of cases, our clients are using AI and so we’re helping them do that. And one of the things I noticed when AI when LLM’s work with data, they tend to sort of fall victim to some very common pitfalls when doing analysis, like certain types of analyses. I think they do different things and it’s sort of challenging to kind of get them to not do that. Like one of the things I noticed right away was like comparing two things to try to evaluate what was different between these two things. It would sort of come up with its own idea, like sometimes they would just make it up and then it would just sort of hang on to the ideas if it was like totally rock solid, that’s the answer. And I would go back and be like, no, that’s not the answer. That’s something you made up. And we got to throw that away or just hold that to the side for a minute. Maybe it’s the answer or maybe it’s not. We don’t have enough data to say that’s the answer. And of course, you know how AI is. It’s like, oh, yeah, you’re absolutely right. Like, I’m definitely not going to do that again. And then I would just go do it again, you know, and it would just do it. And so I was like, OK, I’m out. So that got me thinking like, well, what if I gave it a better set of instructions? And so I started building out a set of skills for different types of analysis to sort of say, OK, if I was sitting down to do this analysis, how would I break this problem down? There is a bunch of great literature and technique and methods all published like there’s a ton of material that it could be drawing on and it just isn’t doing it. And the other thing I noticed is that all of the data skills that are out there, a lot of them have to do with like accessing tools or making a PowerPoint presentation or making a visualization. It’s like, OK, that’s all very handy. I’m talking about like just doing the analysis of the data. So I’m working through a process. We’re still very early stages where we kind of I built some skills for that. We’re doing some testing. What I’m learning in that process is two things. One is using those analysis skills is producing a better result on average than just the raw LLM. And that’s pretty cool and very awesome. But doing evals on skills to prove that they’re working the way you want them to work is a ton of work. It’s like so much work. So like it and you have to be super structured about it. And you have to be like you have to make sure like the prompts are set up correctly and everything is done the right way. So like you can actually evaluate like the outputs and know that it’s like got it right. So like it was still a long way to go, but I’m hopeful that over time 00:41:16 | Tim Wilson: we’ll actually develop something that’s going to be at least convenient, 00:41:20 | Michael Helbling: if not useful in some cases. I think the challenge there is that at the end of the day, I don’t think an LLM can actually produce the quality of analysis or meaningful analysis that I would then go and say, like this is what you should do instead of having humans do it. I just don’t think you can replace it. But I know that humans are going into AIs every single day and asking it to analyze data. And in light of that, maybe we should try to build something that makes it less bad. So in the old George Box framework, you know, all models are broken, but some are useful. That’s sort of what we’re shooting for here. 00:42:02 | Tim Wilson: All models are wrong. All models are wrong. Yeah, there you go. The Mechershop just got triggered because he’s like whatever it is, you’re not using it right. Yeah, yeah. Well, I’m trying to use it wrong both ways. That was on purpose. 00:42:15 | Julie Hoyer: Wait, can you share like what type of analysis you’re talking about that like one of the ones you’re working on a skill for? I’m just curious. 00:42:23 | Michael Helbling: You cannot use EDA. Sure. So root cause analysis or like a or a prescriptive analysis or so like different types of analysis. So like it was sort of a list or a library of like a bunch of them. And yeah, some of them are ones that Tim is not very happy that I’m doing. 00:42:44 | Tim Wilson: Metric, metric reconciliation, metric reconciliation, question, 00:42:48 | Michael Helbling: refinement, yeah, causal, you know, inference or impact. So like all the ones that like in a business context, there’s a better level. But most of the time people don’t put the work in to do. It’d be nice if your LLM was like, hey, here’s what the data is saying. But I think the next step here is actually some kind of holdout test to really prove this. It’s instead of just knowing that threshold. Yeah, instead of just telling you what you want to hear, which is what the LLM would do without the instruction because it picks up on things like the way you ask the question. Like if you’re asking a question with an idea that like, hey, can we stop spending Facebook spending money on meta this month? It’s going to pick up from that question that like you’d like to stop spending money on meta this month. And that was going to do the analysis and come back to you with something that makes you happy, whether that’s good for the business or not, because there’s underlying that is this instruction set that the LLM is doing on its own, which has nothing to do with your analysis and more about how it wants to make you happy. And that’s sort of like what we’re fighting against a little bit because when you do analysis, you’re not trying to make someone happy. You’re trying to get to a closest version to reality that the data will support. So it’s it’s a little bit different exercise. And so it takes the LLM instead of telling it like now behave. It says, hey, here’s the literature. Here’s the methods. Go through these steps and use this as a floor, as opposed to like how you’re doing it the other way. 00:44:25 | Tim Wilson: When I feel like if like, oh, you’re asking to do a root cause analysis, the skill of the root cause analysis, say, I’m going to go do a quick exploratory data analysis of this before, like given the time and the capacity, you should always check that the data doesn’t have nonsensical stuff in it. Where it’s super easy, super common for somebody to say, we’ve been using this data forever or somebody handed it to me and told me it was clean. And so they just go straight to, you know, running a regression. Whereas some of that and I could also see it qualifying, requiring some qualification of the, you know, from the user. But that’s that’s been my hobby horse that I think is valuable to they ask a question and they get a very polite but reason set of focus and focusing and clarifying questions back. 00:45:17 | Michael Helbling: Yeah. And not always sort of like, like an easy answer. 00:45:22 | Tim Wilson: But like, OK, well, there’s more work to do to actually get the answer you want. And it’s like, OK, well, no one’s going to accept that. 00:45:30 | Michael Helbling: So we’ve got to tune it more. 00:45:33 | Julie Hoyer: Funny enough, I not on the list of types of analyses you mentioned, but I guess I would I would count it as a type of analysis. I had pulled together a proposal for a client and long story short, there’s a bit of like a pause. And so we wanted to send over the proposal for our direct stakeholder and her boss to understand like, hey, when things change on your end 00:46:01 | Tim Wilson: and you can revisit this, like here’s here’s what we’re proposing, 00:46:06 | Julie Hoyer: valuable to outline that story. And we wanted to close it out with like an impactful value slide. Right. So me as the strategist on the account is in charge of that. So I mean, we’ve done the exercise manually. I have people at the company I was talking to Matt Padone about it. You know, ways that he’s gone about pulling something like this together. And so I didn’t have a lot of time. And I was like, you know what, I’m going to try to use Claude to help me here because really what I needed to grab and make some assumptions about what this program could help drive for the business for, you know, leadership that we were trying to talk to was like public financials. Right. And like, that’s a slog to go through on your own. So I’m like, oh, I’m going to give it an example of information we’ve pulled before when we try to pull together these like value estimates. And I pretty much gave it context to say, here is who we are working with. Here’s the situation. Here’s what we’re proposing. This is the area of the business that we would be working with. This is the impact we believe that we could drive for them through this work we’re proposing. And it was maybe a long paragraph, like a couple of run on sentences, nothing crazy, gave it one link. And I asked it to help pull public financials to help me make some assumptions and give a value estimate. And it spit out a six tab spreadsheet, which I was a little like, wow, overkill. That’s what I’m also finding. I’ve asked it for like one slide before gives me like seven. I’m like, whoa, all right. But it did a pretty good job. I mean, I was able to go and find to like where it was pulling numbers from. It did automatically give me pretty descriptive, like this is where this number is from. It highlighted cells that it was making assumptions on. It gave me like industry benchmarks that I didn’t actually ask for those explicitly, along with the client’s actual published numbers. And then I was able to work through it once I understood the spreadsheet set up, though, and play with how I wanted to craft the narrative. But it was interesting because I now know the next time I go to do this, I would prompt it in much more detail because, again, I have a better idea of like how I want the spreadsheet to function and the things I want to be able to like input. So I do think if I get more explicit or even dip into like what you’re saying, Michael, and build like a skill around this for it, that it could be a really super useful tool. And then I definitely have to work on the visualization of it because it needs the human touch there for sure. Tim was helping me today. I slacked him out. I was like, I’m running myself in circles. Like I just need some a clean brain to help me edit this story to like what the hell actually matters. So it’s been good, though, for that type of thing. 00:48:51 | Tim Wilson: I had a I mean, I worry like the when you talk to the like it goes it goes bloated, it goes along. And then you look at it and it’s really hard to fully wrap your head around. And it was interesting, even in that the spreadsheet, there were cases where it had just like hard coded done the math and hard coded it. So it wasn’t even like a traceable model in some cases. It was in some cases. But I I mean, that that continues. Like the higher level, like it’s so fast to generate content that’s polished and professional, but doesn’t have. The humanness in it. And I think in a in a proposal context, like you’re somebody’s going to be looking at it and they’re going to be asking questions, regardless of what it is, not your specific example there. I had a case where I’m doing some training, working with another company. And for various reasons, I’m developing the presentation in Horto. It’s very specific to the audience. But I also am pretty particular about I don’t want things to look great. And it turns out when you’re trying to use markdown, I mean, I’ve run through this before. But I worked with the people who were kind of co-presenting and we had kind of a rough we’d iterated through a Google doc where we really sort of figured out the narrative and the story. And that was like collaboratively. And then I went through the Google doc and put a little more meat behind it. And then I was like, what happens if I pop over to fable five and say generate me the I did this in cloud code. I already had a repo set up for it and had pulled it down locally and said, give me the shell. And that was really useful as like the raw because I fed in the this is the sequence. These are the specific slides and they still look like absolute, you know, garbage, like I’ve spent hours after that needing to iterate. But I still had that skeleton in place. So I wasn’t having to start one slide at a time. I have the whole shell, but it came from a human human massage, developed the narrative and the talk and the points and the highlights. But it was it was another kind of useful cloud code experience in that case. 00:51:16 | Julie Hoyer: I see cloud like decks that get put together and it’s like a standalone slide. It’s pretty good. And sometimes they’ll pick up on like this, maybe the main story point you wanted, like throughout this, like again, like bloated long deck that it will put together. But you know, it’s funny to my always go back to like just the basic McKinsey title idea. And it’s like, I need to start telling people to explicitly tell Claude or whatever I’m there using to be like, use McKinsey titles. I think the outline from a human before asking it to make the deck should be like, this is the story bullet points in this order that I want you to tell. Use it for McKinsey title style and say like this needs to make sense. Like top to bottom, because when I get into a deck that is like 00:52:02 | Tim Wilson: Claude generated, for example, it’s like, OK, generally, this slide makes sense. 00:52:08 | Julie Hoyer: I kind of understand what they’re saying. But to me, it doesn’t seem like there’s a good thread between and it’s like verbose and it’s repetitive. And then it’s really hard for me personally to figure out what is the point of this individual slide I’m looking at. 00:52:21 | Tim Wilson: But that’s not that hard to tell to tell Claude to say, you must walk me through. Make sure you have a solid outline. I’m going to I’m going to define the steps of how we’re going to do this. You’re going to make sure I give you McKinsey titles. I did have a rant a like a month or so ago on LinkedIn because I was literally I should remember what I was working on. And it said, I have a hand I have handcrafted a bespoke, beautifully structured five slide presentation and HTML and CSS. It uses clean, high impact management consulting layouts like McKinsey or BCG decks designed specifically for your topic. And I’m like, this is horrible. So that’s rant one. Like I was like, yes, you this is what you think is great. The other like just. Boogaboo, I was trying to generate HTML presentations with like our markdown way back when had one specific client where once every week during COVID, we had to pull all these data sources together and generate it. I’m like, I can make all this happen with with in our and pull together something that matches your template. It’s just you’re going to send your ultimate going to send us a PDF anyway. And I just got hammered. It’s like, it’s got to be in PowerPoint because we got to be able to edit it. And I was like, I understand the the the motivation and you need it. Whereas now everybody’s like, I can just make it an HTML. And I’m like, fuck you. Like, like it wasn’t good enough when I wanted to make something in HTML because it wasn’t just being spit out, you know, generically, it looked fine. So yeah, I’m sorry. 00:54:04 | Michael Helbling: Well, it’s weird with all the things I’m willing to let AI do for me. The thing I’m least willing to let it do is produce content or communication. So like, we even have at Stack Analytics now kind of a standard, like we will not let it write our emails. I like that I won’t let it write decks like technical documentation. I think it’s actually OK at and can produce good quality materials. But like anything where I’m attached to what the person is reading 00:54:33 | Tim Wilson: in terms of communication, I just don’t think it’s there yet. 00:54:37 | Michael Helbling: Like and I’m training it on my voice and those kinds of things that I keep testing. But I I’m just not comfortable with it. And I just feel it. It’s sort of like Stevia, you know, that weird aftertaste. 00:54:52 | Julie Hoyer: That is so good. I just think that’s exactly what it is. 00:54:56 | Michael Helbling: It’s just I don’t get it and I don’t like it. And so if I don’t like it, then I assume other people probably don’t appreciate it either. So it’s sort of like, all right. So then the least we can do is like, it’s going to be us typing that email. 00:55:07 | Tim Wilson: I have I have given it like content. I’m like, I need a McKinsey title that describes this, this, this and this. And if you take it, then I’m editing the edit it yourself to edit it. 00:55:19 | Michael Helbling: Yeah. Yeah. I think that’s totally fine. It’s more of like something that spits out. You just paste it into your email and fire it off. 00:55:27 | Tim Wilson: Like I don’t like it. 00:55:29 | Michael Helbling: Well, I don’t. It’s not good yet. 00:55:30 | Julie Hoyer: Once it does like a deck for you again, we talked about this in past episodes, like you become the editor of work that’s in front of you. But sometimes like it’s so much easier just given that like, it’s good enough because I didn’t think through what it needed to be already. Yeah. Because you didn’t try to solve it. Now you’ve been biased by what it spit out. 00:55:48 | Tim Wilson: And I will say when I’ve I really only a handful of times 00:55:53 | Julie Hoyer: have had it made any type of slide, but I have heavily edited anything that it gives me. Because to your point, Michael, I’m like, I know I can structure a better slide that will get the point across. But sometimes I do need like, can you just give me a couple ideas here? And like, pull what I like and I’ll mash it together. 00:56:10 | Michael Helbling: That I think is just fine. That’s just fine. It’s more of the other side of it where it’s sort of like you just turn your brain off. And it’s like, is that good enough? I don’t think that’s good enough because I could anyone could do that. The client could do that. Like as a consultant, like I hope we’re bringing more to the table than just regurgitating whatever Claude said. Yes, you know, ideally a BCG or McKinsey style email. Well, you know, the problem is sometimes when I write my own version and I’ve seen what it writes, I’m like, my version sucks. But at the same time, it’s like, but it’s yours. Like, it’s me. It’s me. So, you know, everybody just bear with me. I’m not as smart as an AI, but, you know, I’m trying. All right. I love it. One last thing I want to cover off on, because it was a big kind of learning for me is obviously using these LLMs for me has been a lot of fun and very learning experience. One of the big things that happened recently the last six months with with us is going from an individual using Claude to do stuff on a certain project to a number of people needing to engage with that same thing. And like, how do you share the information and context and those kinds of things? And we all like have we’re on data background. So like in data warehouses, everyone’s like, oh, you got to have a semantic layer. So the AI knows how to do all the things it needs to do. And so that’s been a very interesting challenge. I don’t claim to have solved it, but we’re very much looking at that too, which is keep iterating on this idea of how do we deploy context? Because there’s like so many different little decisions go into when you’re setting up data, like, OK, we’re going to tweak it just like this so that these things match up and we’re going to make this decision at this point to kind of normalize the data like this. Like all of those choices are filtering down into then other people making it harder to follow in your footsteps. And so it’s interesting because we’ve done some various things to sort of share that context more broadly. But that’s something I’m keeping an eye on in our space, especially in the data space, because like I think that’s really common. Is sort of like, how are people making this a team sport versus an individual effort? Because I think that’s a that’s a pretty big, pretty big deal. 00:58:41 | Tim Wilson: A little bit of a callback. I started to say we had this document just for the podcast. So it’s not on the data front. And I set this up in co-work. It was somewhere along there that I realized that, oh, the runbooks are stored locally, which I’m like, that is terrifying. Now it’s going into a folder that I have syncing somewhere else, but I’m much more comfortable with stuff being in like defined cloud, whether I’m syncing to GitHub with Git or whether it’s Google Cloud or whatever. But I started to say that we had something that I could hand off. If I wasn’t available, somebody we had backups that could follow the process and it was fairly safe. And now somehow with co-work, I’m like, can I hand this off? Like, like, I don’t know that in theory, I’ve handed it off to this agent, but I can’t figure out how I could safely hand it off, especially knowing that I will hit things and say, yeah, I need to update the task and the instructions here and there. So that’s that this is similar. Yeah. World that you’re like, oh, this supercharge is you, but it doesn’t necessarily supercharge the team in the way that it should. 00:59:51 | Michael Helbling: And I think that’s the big I feel like that’s the big lever or tipping point to value for AI and a lot of organizations is sort of like, you’ve got one or two people kind of doing stuff over in a corner. It’s making them real productive. But like, how do you create a process for AI usage that makes everyone that much more productive and step up the whole org with AI? Like, this is actually the whole part. I think the big part of it. And it’s actually a truly complex, I think, very deep problem that I don’t know that I’ve seen anybody really solve well. And given your recent LinkedIn post and the feedback you got on it, like I see a lot of other people, especially in the data space, because like it’s not a technology problem at the end of the day. It’s actually like a human process problem. And how you solve it goes into your culture. It goes into your process. It goes into all these different things. So anyways, that’s a fun one to kind of unwind, but also kind of like the frontiers that we’re finding and using AI the day to day is sort of like, oh, I did all this cool stuff. Now, how do I hand this work off to my teammate who now or my client who now needs to take this stuff and go one step further with it? Oh, what do I do? So that’s what you’ve got to find the answer. So we’ve built some things to do that. But it’s it’s I don’t know that it’s like it’s not perfect. It’s not perfect at all yet. And I don’t think with AI you can be perfect. 01:01:27 | Tim Wilson: So you just have to kind of be adaptive. 01:01:32 | Michael Helbling: Anyway, all right. Any last words, final thoughts? Oh, we’re like like we’re fine. 01:01:38 | Tim Wilson: Like it, like like the LLMS where we’re running along. Yeah, it’s right for both. All right. Well, then let’s have a Michael. Yeah, that was an excellent point. You did a great job. This has been every point you made was just so spot on. Sorry, I’m being like, I don’t appreciate that at all. Tim, that was the kind of bluff that I don’t appreciate. 01:02:05 | Michael Helbling: You’re absolutely right. You’re absolutely right. That’s the load bearing comment of the podcast. Tim, great job. All right. I bet as you’ve been listening to this episode, you’re like, wow, I can’t believe how terrible the Inelix Power Hour is in AI already and you’ve got thoughts and comments. Listen, we’re open to learning a little more. So this one we do actually want to hear from you. Like tell us how you’re using it. Talk about what you’re doing with it. Like we want to know, at least I do. So you can reach out to us. Do that on the measure slack chat group or on the LinkedIn or via email. At contact at InelixHour.io. 01:02:45 | Tim Wilson: And yeah, check in with us. 01:02:48 | Michael Helbling: Tell us how you’re using AI in your day to day. I think it’s a pretty wide open space and we’re all learning from each other, which is a lot of fun. It reminds me of the early days of analytics, to be honest, where we’re kind of all just sort of comparing notes. 01:03:01 | Tim Wilson: And back then it was, I remember that time with like web trends and like the report tables and the analysis tables and how you could, you could call up one turn and get it. Remember, remember that? Sorry. 01:03:19 | Michael Helbling: And you can also get stickers because nothing says you’re cool, like a sticker on your laptop or water bottle from the analytics power hour. And you can request that on our website at analyticshour.io. And I think no matter how you’re using AI or even how you’re using AI to listen to this podcast, I think I speak for both of my co-hosts, Tim and Julie, when I say, don’t stop analyzing, even if the AI offers to do it for you. You’ve got to stay involved. 01:03:55 | Announcer: Thanks for listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at at analytics hour, on the web at analyticshour.io, our LinkedIn group and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst. Those smart guys wanted to fit in. So they made up a term called analytics. Analytics don’t work. 01:04:19 | Charles Barkley: Do the analytics say go for it no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 01:04:32 | Michael Helbling: Say it all, you never know. 01:04:33 | Charles Barkley: Tony, do not put that. 01:04:35 | Michael Helbling: Do not put that on a date. 01:04:38 | Julie Hoyer: OK, wait. Ready, Michael? We’re going to self-diagnose your head pain. Why? Hang on. Hang on. Just trust me. Good. It’s good. It’s all right. Good. Fine. OK. Window. Here we go. Ready? 01:04:52 | Tim Wilson: Yeah. 01:04:53 | Julie Hoyer: So these are the different types of headaches. Is it a hypertension, a migraine, a stress, or is it a data sits under I.T. type of pain? 01:05:06 | Tim Wilson: That’s good. 01:05:10 | Michael Helbling: It’s kind of a data sits under I.T. 01:05:14 | Julie Hoyer: It’s got that kind of feel to it. 01:05:16 | Tim Wilson: Yeah. 01:05:17 | Julie Hoyer: My guy was literally sitting with a sinus infection last week. So I was like, oh, before I knew it. And I was like, oh, it’s like right behind my eyes, but low. And so I see this on LinkedIn. I mean, I should have known. And I’m like self-diagnosing myself with it until I got to the end. 01:05:33 | Michael Helbling: That’s good. Yeah, no, it’s the migraine one. But yeah, that’s pretty funny. 01:05:40 | Tim Wilson: That’s good. 01:05:42 | Michael Helbling: Yeah, I need to keep that in the back pocket because that’s that one keeps coming up. 01:05:49 | Tim Wilson: Yeah. It’s an episodic recurrent. Yeah. Recurrently episodic. No matter how much tile and all you take. OK. All right. Are we are we doing last calls? 01:06:03 | Michael Helbling: Um, I don’t think so. I’m picking up I’m picking up something from our. 01:06:13 | Tim Wilson: Moere producer. I don’t know. It’s comical for next week. Yeah. The Venn diagram between what Julie reads and what I read is. 01:06:23 | Julie Hoyer: Well, who do you think, you know, sent me all those newsletters? 01:06:27 | Tim Wilson: Yeah, I think this is well, I do have a kid who’s, you know, planning trips out in October and he’s super laid back and chill and he doesn’t get bothered if somebody like can’t make a decision. He’s like, well, here’s the deadline. And if you haven’t made it by then, you’re out. And he just moves on with his life and like, if I could get his attitude just to else, I would probably live ten years longer. Due to the hypertension that is not developed. 01:07:01 | Michael Helbling: Data still sets up right. He. There. Yeah. All right. All right. Let’s do this. Here we go. This is all right. Energy, energy, energy. 01:07:15 | Tim Wilson: Me, my mom, my mom, my mom, my mom. You know, the local folks jumped over the lazy dog over the lazy. Yeah. Rock flag and stevia. The post #304: I Can Haz AI? appeared first on The Analytics Power Hour: Data and Analytics Podcast .

August 4, 20261 hr 3 min

#303: Funnels Assume Progress. Barriers Recognize Reality.

Twenty years of digital marketing created something of a monster: companies poured enormous investment into analytics teams, MarTech stacks, media capabilities, and data infrastructure—and then watched all those functions march off into their respective silos to work really, really hard at producing activity rather than impact. Rusty Rahmer , founder of Starize AI and author of Working As Designed , joined Michael, Julie, and Val to dig into why that happened, why it’s still happening, and what it actually takes to flip the shovel over and use the right end. Along the way, Rusty—who Val correctly identified early as a spontaneous analogy machine—explained why customer journey maps on walls are basically the statistical average American life that literally nobody lives, why waffle fries are structurally superior to ridged chips (and what that has to do with cross-functional team design), and why the most important question a marketing leader can ask a room full of executives is also the one most likely to be met with complete silence. This episode is brought to you by Prism from Ask-Y —your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. This episode is also brought to you by Stape , your all-in-one solution for server-side tagging. Links to Resources Mentioned in the Show (Book) Working as Designed: Why your organization delivers exactly what it was built to — and how to redesign it for growth by Rusty Rahmer (Book) Analytics the Right Way: A Business Leader’s Guide to Putting Data to Effective Use by Tim Wilson and Dr. Joe Sutherland WITHpod: The AI End Game – Chris Hayes podcast series (Book) The Way of Excellence: A Guide to True Greatness and Deep Satisfaction in a Chaotic World – Practical Strategies Using Modern Science and Timeless Philosophy by Brad Stulberg (Book) Inside the Box: How Constraints Make Us Better by David Epstein (Newsletter) Range Widely by David Epstein (Article from Range Widely) How to Pay Attention and Care Deeply in a Chaotic World (NYT Article) The Revenge of the Philosophy Majors (Post) Half-Baked Product by Gerard Rubio Photo by Gabriel Komorov on Unsplash Episode Transcript 00:00:00 | Announcer: Welcome to the Analytics Power Hour. 00:00:08 | Announcer: Analytics topics covered conversationally and sometimes with explicit language. 00:00:15 | Michael Helbling: Hi everybody, welcome to the Analytics Power Hour and this is episode 303. Does it ever feel like we repeat the same steps sometimes even though no one’s really 00:00:26 | Michael Helbling: thinking about whether they actually apply in the scenario? 00:00:30 | Michael Helbling: That old saying, the definition of insanity, repeating the same activity while expecting different results, I think we’ve all been there and yet as analysts, maybe we’re slightly more clued into when it’s happening than others in a business, I’d like to think so. Maybe not. The idea here is are we thinking a step or two deeper as analysts or are we just sort of rolling with the same set of activities with no critical thinking applied? Well, we’re going to talk about it and now I want to introduce my two co-hosts who are definitely deep thinkers, Julie Hoyer, welcome. 00:01:06 | Julie Hoyer: Hey there. 00:01:08 | Michael Helbling: And Val Kroll. Hello. Hello. 00:01:11 | Michael Helbling: How’s it going? 00:01:12 | Michael Helbling: Good. And I’m Michael Helbling. Well, we wanted a guest and our guest is someone we have had on the show before, but not for a while now, Rusty Rahmer is the founder and chief product officer of StarizeAI. Prior to that, he held analytics and marketing leadership roles at GSK and Vanguard. He has a new book called Working As Designed and today he is our guest. Welcome back to the show, Rusty. 00:01:37 | Rusty Rahmer: Thank you. Thanks for having me back. 00:01:39 | Michael Helbling: I appreciate it. And as far as bios go, Val and I also worked with you on the board of the Digital Analytics Association back in the day. So we are friends going back a long way. So it’s great to have you back on the show. All right. So to kick this all off, you wrote a book. So obviously you’ve been thinking about this for a long time and then you decided to get pretty serious about it. Talk about that a little bit because I think probably something was brewing for a long time that kind of came out in the form of the book, but just talk about that journey a little bit and then we’ll dig into some of the content and some of the things in the book itself. 00:02:16 | Rusty Rahmer: Yeah. You know, people ask me, when did you start the book? Well, I feel like I started the book 20 years ago, right? It’s the experience of the last 20 years that I think, you know, at Vanguard it was about building, you know, at the front end of the digital wave, building out the capabilities of, you know, a user experience organization, a data and analytics and data science organization, marketing and MarTech and then, you know, digital transformation, then going to GSK and helping them a different problem, right? 00:02:44 | Michael Helbling: They had all of these capabilities and it was about, you know, why aren’t these capabilities 00:02:49 | Rusty Rahmer: working the way that we think that they should? Why haven’t we gotten the value out of all of this? And that’s a pretty common problem, I think, as I’ve moved on to consulting and I can see it over and over again at different companies and it felt like a problem that I had experience solving and that I was surprised is so prevalent out there and so I felt like I had something to say about that and I wanted to put it down and writing and my experience is what I was observing and what I think the solutions and the core of the problem and the solutions to that are. So that’s really what it was. 00:03:17 | Michael Helbling: No, I think that’s great. So talk a little bit about the book contents itself, like, you know, kind of what was the driving idea behind it? 00:03:27 | Michael Helbling: Yeah, I think there were two key observations. 00:03:30 | Rusty Rahmer: One was sort of where are we now, right? Like looking around at different organizations, I see the same problems, identifying that problem and sort of describing the solutions that I’ve put towards that and what I think the core of the solutioning is. And then secondly, like how did we get here, right? Like there’s a pretty clear path that led to this place that we’re in. 00:03:50 | Michael Helbling: And it’s both of those things that I think were what’s in the book, you know, to sort 00:03:56 | Rusty Rahmer: of describe how we arrived, what it is that the problem is that you’re feeling and then how do you solve that problem are really sort of what’s underneath it. 00:04:04 | Val Kroll: So what are some of those symptoms of like how we got here? Let’s dive into it. 00:04:08 | Rusty Rahmer: Yeah. Okay. Well, let me start with what I think people are probably feeling today, right? So as I led each of those organizations, whether it was user experience or it was analytics or Martek in the tech side, media, I think everybody in those organizations feels like there’s something wrong with it. Like we could be doing so much more with what we do here. Like we’re so underutilized to this problem of growth or whatever the company is trying to solve. 00:04:36 | Michael Helbling: And I think that feeling comes from very siloed organizations, right? 00:04:42 | Rusty Rahmer: I think I see the problem that chief marketing officers, chief growth officers, chief commercial officers have spent a lot of money over the last 10 to 20 years buying software, buying capability, hiring teams, building teams. And largely I don’t think that they’re getting a large percentage of the potential value all out of all of that investment that they’ve made. So from the C-Suite’s perspective, I put a lot of money into this on faith that this 00:05:12 | Michael Helbling: will produce value for my commercial organization, and it just doesn’t seem to change the net 00:05:17 | Rusty Rahmer: result very much. And I don’t understand what’s wrong here, but it’s not working. And I think the reality of that is that it is going back and looking at the history of how we arrived here and the moment that we’re in. 00:05:28 | Michael Helbling: And I think to switch gears a little bit, I think if you go back 20 years before the 00:05:33 | Rusty Rahmer: internet, you know, 1998 or whatever you’re sitting at your work computer, you don’t even have the internet on your computer, right? I mean, it’s hard to even imagine the world back then, but that’s the way that it was. 00:05:44 | Michael Helbling: And marketing your organization was print, you know, maybe, you know, some partner distribution 00:05:51 | Rusty Rahmer: methods, but it was expensive. Print is really expensive to send out. And so TV ads are there, you know, there’s some phone advertising, but everything is an expensive channel. And so your reach is relatively small. It’s relatively limited by your geography, right? Well, digital comes into the picture and completely blows that paradigm out of the water, right? Now you think about buying an email list, and you’re actually rewarded for sending it out to more people than it can actually, you know, like the more people you send it out to, the lower the cost of purchasing that list goes down, right? So you’re rewarded for your reach. 00:06:26 | Michael Helbling: And what that did from an economic standpoint, it allowed companies to enter markets where 00:06:32 | Rusty Rahmer: they were not previously competing with their brand, right? To create, oh my God, now I can reach the world with what my product or message is, and I can sell to the world. So for 20 years, we focused on reaching, reaching and building capability, digital capabilities that would help us reach audiences, right? And even bad marketing with the kind of reach you were able to achieve with the digital world drives your organizational growth. It just does. 00:06:59 | Michael Helbling: I mean, fractional return on that investment means massive growth for your organization 00:07:03 | Rusty Rahmer: just based on the scale that you’re currently achieving. But it’s really lazy marketing, to be honest, right? That reach and frequency marketing is lazy. And so you’ve been purchasing all of these capabilities. You’ve been buying Mar-Tech stacks, Ad-Tech stacks, you’ve been building analytics teams and data lakes and data environments. You’ve been hiring media teams. All of that has been growing in investment, but it’s all working as like, we have to build up this function. It’s never existed before. We’ve never had an analytics team before. We’ve never had a data team before. And so the mission of those organizations has been about how do we build and deliver some value immediately and then scale this organization to what it’s going to need to really drive the growth of this company. So we’re sort of stuck in that time where these functions are still working in silos, trying to do analytics activities that show we could be the best analytics organization in the entire world. We could be the best content creative people in the entire world. We need all the Mar-Tech tools, right? So the Mar-Tech teams are out buying things that the organization doesn’t even need yet just to have them, right? And I think that’s not how it’s actually working, right? I think we’ve saturated the growth market. We’ve reached every corner of the market we can possibly reach. And now it’s about quality of marketing, getting back to getting away from reach and frequency and getting back to timeliness and relevancy, right? Which means, hey, I actually have to understand my customer better than I’ve ever understood my customer before in order to show up in a timely and relevant way to that customer, right? And that’s actually what all of these capabilities we’ve been building have been designed to do. But when you look at the system that they’re operating inside the company, they’re still working in these vertical silos instead of working together on horizontal problems, like growing the company with its customer base, right? So I think that’s a bit of how we got there, which is understandable, given the thrust of digital and digital capability and the speed at which that’s grown. And then it sort of explains why we feel like we’re underutilized in the same system as well, why people aren’t doing more with the data that we’re creating, why Test and Learn isn’t being adopted as fast as it should be, why media dies at the front door of the company and OmniChannel doesn’t connect to it, right? I think all of these problems are functional silos out of that arc of growth and evolution that’s happened over the last 20 years. 00:09:34 | Val Kroll: And one of the things related to what you were just talking about, Rusty, is that really resonated with me. I’ve seen it so many times in organizations. Like you said, it’s like the best content team in the world, the best performance marketing group. And they produce a lot of reports with their analyst partners showing all this progress and performing above industry average for those different campaigns. And so the way you talked about in the book is like, how can we be doing so much that says we’re doing well, but we’re still not getting to where we need to go? And the way the system responds is, well, we just need to be doing more. 00:10:09 | Michael Helbling: And I was like, great, highlight, underline, startles. 00:10:15 | Val Kroll: That is such a repeatable pattern that I think people absolutely feel. But when was the first time you started to take a step back and acknowledge boring more on top is just not moving us forward anymore? Like when do you have a specific moment or story where that realization really started to come to life for you? 00:10:37 | Rusty Rahmer: It’s a little of two things. Again, it’s a little of having jumped silos to build these functions out and then getting the GSK and owning sort of every touch point of the customer, the entire customer experience really was like, oh, my God, now I’m sitting 00:10:51 | Michael Helbling: on top of this entire organization and I’m saying things that they can’t solve. 00:10:55 | Rusty Rahmer: Like this is like a square peg. And the problem I’m trying to solve is a square peg in a round hole operationally. Like no one, no one can function the way that I’m asking for them to function or solve the problems that I’m throwing on to the table. And so transformation sort of has to happen around that. And I think I think leaders experience it every day when they’re the ones, you know, a chief marketing officer is responsible for, I know, 20% growth and they go in every year and planning season and they set that goal on the table and they say, hey, our goal this year is being handed down to us as 20% growth, right? And in that room is the head of analytics, it’s the head of media, all these different heads in there of all these different functions. And as soon as the media is over, everybody leaves and everyone’s going to work really, really hard to try and achieve that goal, but they’re not going to talk to each other and they’re all going to work in their little silos. Like the solution to that 20% growth is sitting somewhere in the data strategy or it’s somewhere in the media strategy or it’s somewhere in the Mar-Tex that it’s not, it’s not, it’s all of those things functioning together to solve that problem in a way that we have not set up the company to work or reward more importantly to reward. And Val, that’s what you’re talking about because later on in the year after we’ve had that discussion and everybody’s going off into their silos and is trying really, really hard to make that goal happen, to try and be the hero, we’re going to get together quarterly and we’re going to talk about results and we’re going to talk about, we’ve built new data tables in the data, and we bought this new, we replaced this software with some new AI driven, whatever. And like, we built new customer segmentation model, like we’ve got it, we all worked on stuff independently, but the dial’s not turning because it wouldn’t, it wouldn’t given those pieces independently working rather than, you know, working together on a common problem. 00:12:46 | Michael Helbling: Hey Tim, imagine for a minute you’re an e-commerce marketing leader and you just launched a new checkout yesterday. 00:12:56 | Tim Wilson: And now you’re wondering, did performance change or did the tracking break? 00:13:01 | Michael Helbling: Yeah, it’s so important and with the STAPE MCP server, you can connect an AI assistant to your STAPE account and you can ask it, things like checking the container to confirm that the tracking domain is valid, review usage before and after launch, and you can show what changed by browser or client. 00:13:21 | Tim Wilson: So instead of digging through infrastructure screens, you can get a plain English operational check. 00:13:28 | Michael Helbling: Yeah, you can also review daily usage, see which domains are generating traffic, and check for unexpected costs or surcharges. 00:13:36 | Tim Wilson: That helps an in-house team answer an important question faster. Is this a marketing problem or a measurement problem? 00:13:45 | Michael Helbling: Yeah, hone in on the solution and to do that, click the link in the show description to learn more about the STAPE MCP server and how it can help you. 00:13:55 | Tim Wilson: Michael, ask wise prism version 3.0 now speaks MCP. That’s great. 00:14:03 | Michael Helbling: I love acronyms. They make me feel like I’m in a meeting. I forgot to decline. 00:14:07 | Tim Wilson: Well, in this case, it means prism can plug into Cod Desktop, Cod Code, and other MCP compatible tools. Oh, I do like that. 00:14:15 | Michael Helbling: So prism becomes part of the workflow instead of another tab I’m pretending is organized? 00:14:21 | Michael Helbling: Exactly. 00:14:22 | Tim Wilson: From Cod, your users can query the workspace, trigger analyses, and pull results into whatever they’re working on. Oh, I love that. 00:14:28 | Michael Helbling: Less where did that come from more? 00:14:30 | Tim Wilson: Oh, look, a usable workflow. Yeah, and with prism’s memory engine, skills, permissions, and admin controls, the context stays managed, not trapped in one person’s chat history. Really useful because I think Claude told me this last week is not a governance model. Not exactly. So go to ask-y.ai and join the waitlist and pro tip use code APH to jump to the top. That’s ask-y.ai and use code APH. Fewer tabs, smarter agents, less workplace archaeology. 00:15:06 | Rusty Rahmer: I think you see it every day. I mean, I really think you do. The advantage was having sat above all of those functions and going, okay, you know what I don’t need is more exposures with no one going to a high value action on my website. Like, are you working with the OmniChannel team to deliver, like, after we expose them to deliver a journey that moves them closer towards the high value action that I want? Oh, no, you guys never talk. Okay, this is a system that produces and rewards this, and now we need to change the system that we’re all operating in to reward, to give you a different problem to solve and to reward you for working as hard as you’re already working, but to solve a different problem than what you’re trying to solve today, right? 00:15:48 | Julie Hoyer: And what you said really hit home of it’s time for us to turn back to thinking about how timely and relevant it is. For the customer, and I have also found working with a lot of my clients is that in the pressure of activity, like you were just describing in the silos of their organization and the pressure to do more, get more out of it, or at least that’s, again, the habit that we’ve created, people very quickly lose the lens of the customer and the customer experience. And at the end of the day, that is what all the functions that the company need to work together to change. I am curious, how have you found a way to help teams get back to thinking about the customer or helping teams reach across to the other teams involved? And I’m curious from the different positions that you’ve held or now you’ve been moving 00:16:42 | Michael Helbling: into like consulting, like you mentioned, is there anything you use to kind of convince 00:16:47 | Julie Hoyer: people of that or scenarios where you’ve been able to help change those working habits? 00:16:52 | Rusty Rahmer: Yeah. And Julie, two things there. I think first, I probably even said those words too, getting back to the customer sort of experience. Honestly, it should have always been about the customer experience from the beginning of time, right? When it was a small hardware store that you walked into in your neighborhood, like they knew in order to do business, they had to help the customer solve a problem. And I think as this digital era has helped companies scale, I mean, any company you take, if you go back 20 years and look at what they were, you know, the number of people that worked there, the size of revenue, all of that, that is a magnificent scale. And I think that challenges companies to figure out how to even grow their organization to be that big and handle that much customer work. But I don’t think we ever should have lost track of, I still actually need to be good with customers. That’s why I called it a little bit of lazy marketing in between because it did get away from that even though it shouldn’t have. And now if you’re going to grow, you’ve reached as much as you’re going to grow with bad marketing. If you want to grow that baseline from what it’s at, you’ve got to get good at the customer experience. You have to solve customer problems again. So you’ve got to know your customer better than you ever have before. And again, the good news is all of this tech stack and everything we do is actually ideally designed to solve that problem. You’re just not using it. The cover is a shovel upside down, digging a hole. That’s actually the metaphor that’s trying to be conveyed on the front of the cover. You have the right tools. You’re just got to flip it over and use the other end and it will solve the problem for you. To your point, Julie, I think what I like to do, I always, as a leader who’s led a lot of transformations over the last 10 or so years, I always think it starts with vision. 00:18:29 | Michael Helbling: I think you sit down and you talk about where do you need to be in five years from now commercially? 00:18:35 | Rusty Rahmer: Like what goals do we need to be achieving five years? And what’s it going to take? How do we have to operate in order to achieve that, right? And well, we need to do more when you’re, okay, great. A fundamental question I love to throw in there that’s sort of a blockbuster question that really stops the conversation in its tracks is, okay, what are the customer problems we have to solve in order to turn them into better customers to achieve that result you 00:18:59 | Michael Helbling: just talked about? And they just stop. 00:19:03 | Rusty Rahmer: And there’s no answer to that. There’s just silence. And I go, look, fundamentally, marketing is about changing mindsets. It is about, if they were already going to buy it, I wouldn’t need to waste money marketing to them, right? That I wouldn’t need to do that. So it is about changing a customer that isn’t going to do this to some degree. Maybe some customers are further away from doing it than others. But in some way, it’s about interjecting myself into that customer and changing an outcome that benefits the customer and me at the end of the day. And if I’m asking you, what is it going to take to change the mindset of your customers or what problems stand in the way of your customers being better customers for you, then I don’t know how you’re ever going to achieve those goals. That is fundamentally the question. That’s the only way to achieve growth is solving those problems for your customer. And the fact that you’re sitting here quiet tells me we haven’t done enough to know our customers and what problems they have that are preventing them from being better customers 00:19:56 | Michael Helbling: for us. 00:19:57 | Rusty Rahmer: And that’s something we better get back to doing. And look, I’m not the first person to preach customer centricity. But I think it’s been a forgotten concept. And I think many companies would say that they are customer centric because they have a lot of customer data. 00:20:16 | Michael Helbling: But that customer data is a lot like saying, this side of the river holds the biggest fish. 00:20:21 | Rusty Rahmer: So if I fish here, I catch big fish. It’s not how to catch them. It has nothing to do with how to catch them. And so you really don’t know your customers because you don’t know what the problem is. And therefore, you can’t solve it and move big fishes or small fishes over to the side that you want them to. And so I think it’s about that. I think that’s the moment where they sit back in their seats and go, you know what? Maybe we actually don’t know the key to that question and knowing it is unlocking the future for us. So how do we do that? And so you’ve got an ear to start talking about how you start solving that problem and how you start getting back to knowing your customers again. 00:20:56 | Julie Hoyer: And you know what’s interesting, you talk about knowing your customers. Another thing I’ve run into a lot is when we get into segmentation of customers, again, at the thought of, oh, we’ll figure out who’s alike and then we’ll market to them accordingly. It’s like getting at the idea of caring about the customer, figuring out how to work with each user type. But sadly, the first thing I see a lot of my clients reach for is they segment them based on outcome. They say, who are my big spenders already? Or who are already really loyal and who are the people who are not loyal? Which is interesting because it’s so backwards to thinking, well, wouldn’t you describe them by similar characteristics and then you’d hope to interact with them to then drive the actual outcome measure for the different groups? But I find it really interesting that so many companies end up actually segmenting on outcome, which seems so backwards. 00:21:55 | Rusty Rahmer: It’s so about you, not about your customer, right? And I get it. It’s particularly in pharma, right? Like that’s an industry that I spent time in that I could see right away grew on a really expensive resource called a salesperson showing up at a doctor’s door, right? Which I don’t know, costs something like $300 every knock on a doctor’s door. And though that ability has drastically been reduced by legislation, they don’t really have the ability to do what they used to be able to do or be as effective with sales folks 00:22:24 | Michael Helbling: as they used to be able to. 00:22:26 | Rusty Rahmer: So they’re going digital, but it’s exactly that, Julie. When I think about sending a very expensive resource like a salesperson out, I can’t send them out to very small customers where the customer opportunity is very small. I have to send them out to the place that they could have the potentially biggest impact on my bottom line. I get it. But when you extend that thinking to digital, you’re making a mistake, right? Like digital doesn’t cost the same amount of money. And that was you were putting the problem set on a human to interact with another human to figure out what the problem actually was that needed to be solved, right? Now you’ve got a digital ecosystem, which can reach bazillions of people, very, very low cost, but you don’t know the problem you’re solving as you’re blasting all this stuff out all the time. So you still create segments that are like, these are the biggest fish. Let’s spend the most marketing dollars on these biggest fish when in reality, if you 00:23:19 | Michael Helbling: stepped back and you said, you know what, there’s probably only four or five problems 00:23:24 | Rusty Rahmer: that my customers have that if I could solve these problems, they would be a better customer for me, right? And if you segment your customers on that first and say, okay, here are big fish and little fish that all share the same problem. And if I can solve that problem, I would turn all of them into better customers for me. Now, some of them within solving that, there may be different ways of solving it. I may send a rep out and maybe I wouldn’t send a rep out to a small, but I know the problem I’m sending the rep out to solve and I know what I’m asking my digital marketing or my customer journey marketing to solve as well. It’s just, I’m deciding how much to put into the solution, but the solution is the same across the board in terms of the value that has to be exchanged with that customer to move them from where they are to where I want them to be, right? So I think that segmentation has a place. It just isn’t the first cut of segmentation. It’s a second or third layer of segmentation that says, okay, once I’ve segmented on the problem I need to solve and I’ve got solutions to that problem, right, which is a portfolio of solutions. I can decide which of that portfolio of solutions I should invest in which customers based on cost and potential return on that investment. 00:24:33 | Julie Hoyer: And I think if a company gets that far and they’re hemming and hawing about the incremental value of a solution on a certain subset of users for a problem, like, I’m sure they’re doing pretty damn well because that is like top tier thinking, like, if you’ve made it to that step, like, I want to come meet your company, like, honestly. 00:24:52 | Val Kroll: Totally. Well, and so that’s, I cannot wait to talk about some of that portfolio stuff in the way you’re talking about thinking about, but I want to go back a little bit to the transformation it takes to be able to get to what you guys were just talking about because I know you reference like the cover of the book is about like turning around the tool like you have it, you’re just not using it correctly. But I do have the privilege of, you know, being in your circle. And so I, and you’re also the king of analogies, so I can’t wait till we do the tally at the end of the episode for all the amazing analogies you give us. But our view, and there’s so many that have stuck with me, but I remember you were talking about a team that was mobilized and ready to do some of this work. And I think the way you phrased it, hopefully I don’t butcher it is like we’re ready to work together to make Thanksgiving dinner. And the request I’m getting is for a side of cranberry sauce, more cranberry sauce, just like put more cranberry sauce on my plate. And so I want to talk a little bit about like, who needs to be involved in some of this transformation and how do these groups kind of work together because like Julie said, getting an organization to like put aside your segmentation that you’ve been so emotionally like your crutch for the past however many years to something that you’re talking about with this transformation. Like talk a little bit more about like what that takes and who you have to get bought 00:26:16 | Michael Helbling: in. 00:26:17 | Rusty Rahmer: Yeah, it’s a great question. And as I just mentioned with the customer centricity, right, like I don’t I’m not the first person to preach customer centricity. I mean, that’s been out there for a long time. I think there are this this entire book is not about any one unique concept. It is more about all of these concepts that companies are putting into play. Like they think they’re being customer centric, but they’re not applying it correctly. They have agile, but they aren’t totally applying it from front end to back end. It’s only a it’s only a tech build. It’s not a front end strategy, right? So I think it’s on this book is about a lot of those tool, not just more tech stack and had tech stack and and teams that you have built, but also ways of working that you’ve deployed, but you’ve only got like a percentage of it’s deployed, right? Because this is this is where it comes together, right? I think at the there’s sort of an arc of maturity you go through here, right? I think there’s sort of when you don’t have functions, you build centers of excellence to create consistency across your organization and to get you over the large investments that need to be made around capability and discipline and structure, you know, the design systems for the UX team, you know, analytics, so you don’t have 10 sources of truth, you’re down to a, you know, a limited set, you’ve, you’ve got definitions of data dictionary, like all those things come with center of excellence and they’re really good and they’re really important. But ultimately when you talk about solving these customer problems, they largely become a bottleneck pretty quickly, right? Because you need localization of the problem solving and not only that, you need quick feedback loops, right? This is where, you know, Val, of all people, test and learn, right? She’s so close to your heart, but how difficult is that to introduce into a company that’s organized into the site? Like it doesn’t work. First off, it takes them nine months to take an idea to market. When it takes nine months and nine months of cost to get an idea in the market, I can’t do two of those. I just need to get it out and get some kind of result. I have no time to test any alternate ideas, right? So this has to get these teams, these teams that can do this work closer to the front end, be able to deliver pieces of things in a hurry, evaluate quickly with analytics if it’s working or not, and pivot quickly. It can’t go through year-long planning cycles, nine-month delivery site, it can’t do any of that. Like that, look, you’re not going to build that thing overnight. You have to mature to it. But ultimately, when you have a team that is comprised of all of the different functions that make digital run from data strategy to analytics and two kinds of analytics, right? Long-term analytics and fast, quick, dirty, is it worth it? 00:28:59 | Michael Helbling: The thing we just put it out in the market, how much traffic is coming to the website 00:29:02 | Rusty Rahmer: from that? What are they? Are they achieving high value act like quick and dirty? To media buying, all of this needs to be done on a rapid scale with a team that comprises all those disciplines, right? It’s not cheaper. I won’t tell you it’s cheaper. Centers of Excellence are really cheap because you scale them across the entire organization. When you start building these teams, you’re borrowing discipline from the Centers of Excellence and you’re dedicating them to a set of problems, right? 00:29:29 | Michael Helbling: So you might say in this case, you might say, all right, let’s pretend we’re at a pharma 00:29:34 | Rusty Rahmer: company and we’ve identified that there’s a valuable cluster of customers who need more evidence in order to write the script for our drug over what they’re currently writing the script for. We’ve got a better drug, but they’ve been writing that for 10 years. The existing drug, our new drug, better results, but we have to prove that to them in order to get that, right? Now, there are different value exchanges that can happen to do that. You can get them data and white papers that say the results are better. You can have them talk to a peer. You can get them to a conference where they see somebody, a key opinion leader speak on it. So there’s different ways of solving that, but I’ve got to build journeys to get them to those high value actions. I got to get somebody to a conference and that means I’ve got to make them aware of the conference. I got to find the right people who would be open to going to a conference. Maybe that’s geography-based. I’ve got to build a journey that makes it easy for them to sign. There’s a whole journey that goes behind delivering that value exchange, right? I need a team who is dedicated to that problem set of not just delivering those value pieces, but also evaluating, are these the right solutions to somebody that even needs more evidence? Or are there better solutions out there? Maybe a podcast on something like this might be a better solution, whatever. So they’re looking at not only the tip of the spear, the high value actions or the value exchange pieces, but also, are we getting people to sign up for those things? Are we actually exchanging those value when we pick a customer and we say, we want this customer to attend a conference. How good are we at getting them to be able to do that? Are we improving the journey that gets them to do that? When they do go to the conference, is that the right solution? Is that moving them or not? 00:31:18 | Michael Helbling: That all needs to be rapid design, analytics behind it very quickly. 00:31:23 | Rusty Rahmer: And so you need a team who has all of those disciplines or can borrow all of those disciplines to deliver in that in a very agile or lean way, right? You’re working model with some hybrid of agile, lean, kind of on-bonnish thinking that you put together to solve that problem. But every two weeks, you’re putting something out into market, you’re getting results back 00:31:40 | Michael Helbling: immediately and you’re saying, okay, this is working or this isn’t working, fail, go 00:31:46 | Rusty Rahmer: on to the next idea. And that’s where Test and Learn finds its home with all of these other disciplines helping to do segmentation more precisely. Maybe we targeted the wrong audience with that message, content helping to improve the messaging consistently, time after time, you got all of that at your disposal to try and solve that problem. Now, it is more expensive, I will say it again, than the center of excellence. But if all the center of excellence is doing is pumping out more cheaper, that’s not impact. This is impact because I’m going to give them a problem. And I’m going to say, here’s the population that I know that I want you to target to change that this is the population I’ve defined as having the problem of needing more evidence. And I’ll be able to look at that population and say they’ve moved or they haven’t moved as customers as a result of the actions that this entire team has put forth. So I have a measurable result of impact at the end of that in a way that the center of excellence who just pumps out websites or pumps out email campaigns or pumps out like doesn’t have any connectivity to an output or an impact output associated with it. So it’s just pumping out actions, but not results. This is tied to a result. 00:32:51 | Julie Hoyer: I really like how you started to define what a journey, so you were saying it’s for a specific audience with the same problem. Because another thing I find is I spend a lot of time talking to my clients and stakeholders about a journey, what they consider a journey. I’ve heard people say, I think maybe the purist is like, well, the journey should be defined by the business ahead of time. What is the ideal flow? But then these companies spin up huge websites, huge digital ecosystems, they offer labs, they offer webinars, they offer e-com online or their salespeople in person, and it gets very complicated fast. So then people want to say, well, we have all this behavioral data. Let’s just use AI to surface the pattern, and that will tell us the actual journeys. But what I hate about both of those is it’s completely disconnected to the customer. So I really like how you put those bounds around it, because I think that’s something that I’ve been searching for, because I’ve started to just get so crabby when people want to talk about journeys. I’m like, oh, freaking journeys again. There’s no good answer. But I really like the angle you’re taking on it. 00:33:58 | Rusty Rahmer: So, Julie, you just stepped on a landmine for me. I am here. 00:34:02 | Michael Helbling: Yay. 00:34:03 | Rusty Rahmer: Customer journey exercises. 00:34:04 | Michael Helbling: Oh, my God. 00:34:05 | Rusty Rahmer: In my life, if I could count the number of times I’ve walked by a room and looked in and saw like this massive arc of customer journey with stars and arrows and boxes, like graphic designer skills going to waste on this magnificent board that is a customer journey, right? Like from whatever, biggest waste of time I’ve ever seen. Actually, it’s an exercise that’s probably not a waste of time to do, but is so non-reality. So this is the analogy I like to give around this. If we were to take the average American’s lifespan and say, well, what’s the journey? Well, I don’t know. I’m born. If you average it out, you’re born. You graduate high school. You’re born to two parents. Oh, by the way, you’re a family of 2.5. I don’t know who the 0.5 is, but you have a 0.5 of a sibling. You go to high school, you graduate, you go to some four-year college, you get married, you have 2.5 kids of your own, you retire, whatever. That’s your entire life story, you go on vacations, you retire, and that’s it. Literally no one in the United States lives that life. Literally no one lives that life. That’s what a customer journey on a wall looks like at a company, and no one actually walks through that thing that way. That’s an average of people’s lives that literally no one lives. The reality is, some people do go to college. Some people are born with six siblings, some are born with one, right, or none, just one individual child. Some go to college, some don’t go to college. Some get married, some don’t. Some get divorced. Some have kids with the first relationship. It’s messy. The real world is messy. You can’t create an arc that touches all of those points and pretend that that’s a reality. That is not a reality. What you can do is exactly what we’re talking about here is say, okay, no, I can’t do a whole arc. But if somebody’s online shopping for a kayak, I can give them a journey to get them to a kayak. I can detect that, and I can get them to the kayak, right? Maybe they buy another one after that, so that’s like a divorce. They go through the same purchasing thing again, okay, no problem, right, if it’s fine. You can only sort of detect in the moment what their need is and try to solve that need. You have different needs that you can solve for, but you can’t configure them in a definitive way that they’re going to go through them, right? Even if you say, my biggest problem with this new product launch that I have is I have to create awareness. No one knows about my new product I’m about to launch into the market. Okay, well, your first journeys are all going to be around awareness-based, right? 00:36:43 | Michael Helbling: Okay, great. 00:36:44 | Rusty Rahmer: Let’s fast forward five years into your new product out there, and a competitor comes into the marketplace, right? Here’s a really good customer of yours that keeps buying your product, but suddenly there’s a competitor in the market. Well, they might have to go through awareness again. Any awareness might be how your product is better than what the competitor’s product is, right? Like, sometimes they go back through the same journey again. That’s what I’m saying. So I think to try and create a journey which links all of these subjourneys together is insane. It never happens, but there are micro journeys. And at the end, at the tip of a micro journey is a value exchange. If I can get them to do this, they will move closer to being a better customer for me. And if I can use data and analytics to determine what that need is, which is what I should be using it for, then I can put them on the right journey and score and evaluate my ability to deliver a good journey and a good value exchange to move them towards it being a better customer for me. I think the other analogy that’s in the book that kind of works well here is the game of Plinko, right, which is this old wooden nickel game. Drop it in the top. It hits these nails, and there are four buckets at the bottom that have different points associated with them. I think the wooden nickel is the customer. 00:37:48 | Michael Helbling: And I think at the bottom are the journeys that we’re trying to put them on, like, okay, if I could get them on this journey, they’ll become a better, if I can exchange this value with them through this journey, I’ll make them a better customer. Each one of those nails they’re hitting on the way down is an opportunity, it’s an interaction with your brand that is an opportunity for use data and that interaction to try and guess which bucket they need to be in at the bottom of that. And that is the game of data and analytics at the end of the day. When I use this to look at patterns and identify what my customer need is at scale for this group of people, this is the problem that they have with our brand. And if I can identify that, now I know from marketing what problem I have to solve. I can measure the actions that I’m taking, the journey that I’m taking to try and get them to exchange that value, which I believe was, I can measure all of that and tell you how effective all of that is at changing my customers into better customers. 00:38:49 | Val Kroll: I like that a lot. And I will say that the one thing that I repeated in my notes as I went through your book, I was like, the process is testing, like in there, in a couple of like, carrot, like that’s, yeah, where that all fits in. I love that. So like if we have these problem mobilized pods that are focused on continuous delivery of the portfolio of solutions that could potentially relieve or solve some of these problems, like paint the picture. So one thing that we didn’t really get to talk about too much is like the system and the system’s expectations of how people behave today and how they’re kind of rewarded for activity to how the system behaves in this like more idealistic vision that you’ve painted. Like, can you talk a little bit about that? 00:39:40 | Rusty Rahmer: Yeah. I mean, again, and we’ve achieved this, right? I think at GSK, this was sort of the pinnacle of what we were able to do was put these teams together and the reward structure completely changes, right? We’re not rewarding activity anymore. We’re saying, here’s a population that needs to move closer to the ideal customer set that we have. And here’s a set of problems that we believe is standing in the way of them being that, iterate and solve that problem, right? Validate that we’ve even got that, like maybe we’ve got the wrong, the way we’ve segmented these customers into this problem set may be wrong. The value, the products, right, in the experience products, the value exchange experience products that should, you know, our conferences, maybe that isn’t the answer to this problem that we think it is. Maybe that really doesn’t move a customer forward. So look at the portfolio of things that we’re trying to get them to do and our ability to actually get them to do that and help us optimize that and the way, again, reward. Again, I think people today in the silos of the existing system are really innovative and ingenious. I think every team I’ve ever led has extremely talented people in it who are trying really hard and defining new ways of solving problems for the company. It just isn’t latched up to something, a common problem that actually delivers impact the way that this system does and that we can reward and recognize those for that kind of innovation and ingenuity against a well-defined problem and a system that can measure the actual outcome of the efforts that they’re doing, right? And so, again, I can measure the cost of, you know, with a really broad brush, I can measure the cost of my centers of excellence today. I know how much they cost me. I know how much the tech stack they’re working in costs me annually. I know how much headcounts in there. I know contracts. I know all the answers to that, right, at a very high level. I know that cost. And I also know how much our impact that entire organization is having out in the real world in terms of growth. I know those two things. And I know this model is significantly more expensive than that model, but I can show the results and the outcomes are significantly higher than the money that’s being spent in this other model. And when I can do that, I can reward and recognize the people within the system for the amazing ingenuity that they’re having within that system. 00:41:56 | Michael Helbling: So in the operating model chapter, you kind of develop this idea of, like, developing those cross-functional teams as opposed to sort of like center of excellence concepts. So just to clarify boundaries a little bit. And also, I think if I was a business leader listening, I sort of have some questions about like, how deep should that go? Should I try to take everybody? Cause like some people are just specialists, like they are working on that thing. And do I need to throw them in a cross-functional team or is there sort of that structure? And then I set up a like a tiger team sort of set up. Like how do you, how do you bridge that gap for people or advise people? 00:42:38 | Rusty Rahmer: Yeah, I think it’s a great question, Michael. And I think there are, it’s hard to put that in the book, the experience and what I think, because I think, well, I say this because I think a smaller company has a different answer to that question than a larger company does. So it’s hard to prescribe that clearly, but here’s what I would say. I think, you know, this is, this is another analogy for you, Val. I think potato chips with no ridges break really easily in the dip. They do. The reason ridges were invented is it makes the chip stronger, right? But ridges only go one way. You have to orient the chip correctly, Rusty. Well, that’s correct. Yeah, yeah. Come on, you know, what’s stronger? This is not our first time dipping a chip. Anybody who dips horizontally is making a huge mistake. And that’s the other. There’s going to be half a chip in the dip. That happens. It’s not going to be good. No, but, but what’s even stronger than, than that, right? Which, which I would say, if you only went with teams that were, you know, journey aligned, you would have horizontal, it’s stronger than the plane chip where nobody’s aligned to anything. But it isn’t the strongest function that you’re the strongest model that you can have. The strongest model is the waffle fry. Seriously. The Chick-fil-A waffle fry is the strongest potato in the world. It can’t be broken any way you try. All right. So you get it. Crosscutting is actually twice as strong as, and I’m sorry, there’s probably some engineer that’ll tell me more to a waffle chip. Trader Joe’s has a waffle chip. They do. They absolutely do. That is a supreme dipping material vehicle. It’s nice. It’s real nice. Yes. Yes. So you get the point. I think you can’t just go one direction with this. I think to your point, Michael, in addition to these, the players on these teams that are, that are multifunctioned, cross-striped, whatever you want to call it, still need to belong to a discipline, a guild, which belong, the center of excellence still needs to exist. The work just doesn’t get done in the center of excellence. It’s now transferred out to these teams. But the members of that team belong to the guild of the center of excellence, because we, if you just let it go, you lose all the discipline of a center of excellence that has, you know, updates the technology and the stack that they actually work in to make sure that continues, that there’s a roadmap for that, that documentation continues, that standards for design continue. Right. You can’t, can’t turn it all loose. I think you still retain a small center of excellence that continue to represent the guild and the need of the guild to be able to do its job fast and effectively. I also think it can, in a large organization, you should separate out the value delivery products, so conferences, white papers, these things should be developed by teams outside of the journey teams. Journey teams should be mainly focused on identifying the right audiences, moving those audiences through the content journey and evolving the content journey to more effectively move them to exchange the value. But that portfolio of value things is actually another team that should actually develop and own those. Ultimately, if you can afford to do it, I think if you’re a small organization, this sort of collapses a little bit on itself, you know, smaller teams, smaller, see if, you know, all that stuff. So, center of excellence. 00:46:04 | Michael Helbling: But then you run into the age old Glen Gary, Glen Ross problem, you know, these leads, these leads are all good. Oh, the leads. 00:46:11 | Rusty Rahmer: It’s a great movie, by the way. No one has the patience to watch that movie anymore, but it is amazing dialogue. 00:46:17 | Val Kroll: Well, before Michael says what I think he’s about to say, I have to ask, the book was written in a very specific format, a format, the way to kind of phrase it. So I think, Christine, you need to tell your future readers that are currently listeners, a little bit about how you made some of those choices. 00:46:39 | Rusty Rahmer: So I actually wrote it long form originally. So it’s a 500 page, there’s a, there’s a 500 page version of that book. And when it, before I even started writing the book, I have some mentors that I, that are sort of guiding me through the process. We’ve written books before they are C-suite individuals who, who sort of were the people I was trying to speak to the most to help them understand why the money that they’re spending isn’t, you know, returning what they, what they really want. And so I had pitched the idea to them very early on, is this a book? Is this worth a book? Is this worth a book? And they were like, oh my God, yes, definitely write that book. So, so they, they sort of, you know, helped me affirm that there was, there was something to be said here. I developed the 500 pager. I send it to them and they’re like, I’m not reading it. Like, who are you talking, like, what is this? Like, I don’t have time to read this, you know, whatever. And they’re like, when you came in here, you told us about this and we were all in, we totally understood it and we were ready to go back to our companies and make a change. Just tell us that’s like, talk to us like you did then. And so I had to go back and take the 500 pages and turn this into what is more of a conversation with the reader than, you know, than the 500 pages. It is really meant to be a conversation. It is spaced with timing, which, you know, when you first open it, it’ll look like poetry. It is not poetry, but you will, after the first page, you’ll sort of fall into this groove of single spacing pauses versus double spacing pauses. And it helps make the points that I’m trying to make in shorter, you know, less stories, less get to the more get to the point CEOs sort of speak. And it, you know, it condensed the book significantly and it helped me say the same thing in a lot less words that I think C-suite people will be willing to endure. 00:48:23 | Rusty Rahmer: I’m working, I’m more of a podcaster. I listen to things more than I read the physical copy. So, you know, I get the book published and I’m working on. Okay, now I got to get to the part that I would actually enjoy, which is the podcasting part. Everybody’s like, my mentor’s like, you have to do it. It has to be your voice. So you got to do this. I’m like, okay, well, I’ve never done this before. So, okay, let me look into this. There’s a lot of work, editing and all that. But I’m like, wait, there are these great AI products out there now. You record like a 30 second sample and it’ll, I’m like, yes, this is it. I’m going to do this, right? So I go, I buy the software, I record the 30 second sample. And then I immediately turn it loose on the first chapter of my book. And it actually did a pretty good look. It did a really good job of being me reading the first chapter, which was the problem, right? Listen, I turn, I turn all of my devices to like speak in British because everything sounds smarter if you have a British accent. I mean, you sound brilliant with a British accent. Like Siri talks to me and because I like to think that I’m around smart piece. So, okay, all my devices speak with a British accent to me. The opposite, right? So you could read the script of idiocracy, but if you do it with a British accent, it sounds like the Magna Carta. It’s amazing, right? Oppositely, if you have a Philadelphia accent and you read the Magna Carta, you sound like you’re reading it at idiocracy. And that’s exactly what I was like, oh my God, is that my accent? Do I sound like that? It was terrible. So I ended up having to hire someone to do it. It’s almost done. But like, it’s so much. But I’m not even, I only write to the guy in email because I don’t want him to hear my voice now. I won’t talk to him on the phone because I don’t want him to hear out there. 00:50:04 | Rusty Rahmer: Water, water. All right. 00:50:07 | Michael Helbling: This is awesome. This is so good. And the book is really good. And it does read. Once you get into it, you do read with a little bit of cadence. And I definitely picked that up as I was reading it. I was like, at first I was like, oh, what’s going on with this book? And then I was like, oh, actually it kind of does flow once you get used to it. So it is well worth it. And I don’t think it’s just for CEOs either. Because I think even when you’re sitting in a seat, I think you can advocate for some of these things. If you’re not the senior executive in your department, you can sort of raise your hand or start to apply some of the principles even in where you’re at. So, okay, we do have to start to wrap up. Awesome conversation. Thank you so much, Rusty. It’s been too long. And you’ve been working on this super hard. It’s like, wow, you’ve got a lot done. I was like, what have I been doing? No, so thank you for all the work, putting that together and coming on the show to talk about it. Okay, one of the things we do is go around the horn, share a last call, something that might be of interest to our guests. Rusty, you’re our guest or might be of interest to our listeners. And Rusty, you’re our guest. 00:51:22 | Rusty Rahmer: What is your last call? Have you guys read this book, Analytics the Right Way? Have you heard of it? Yeah, yeah. Little plug for Tim. Little plug for Tim. No, that’s good. Um, lately, I have fallen in love. This is the second time I’m listening to the same seven series podcast. It’s the first time I’ve ever listened to a podcast twice. It’s by Chris Hayes. It’s called the AI End Game. It’s about seven podcasts, about an hour each podcast. They’re not terribly long. He has guests on to each episode. What I love about it is it covers every, it’s a, it’s very sort of top thinking, you know, about AI, right? In the world, everything from philosophical. So he’ll have, you know, some, some professors on that will talk about what is intelligence, what is sentience, can AI ever achieve this to the political ramifications of how do we govern this? How do we write policies to control this? Where do we need policy to control it? To history. Talking about AI summers and AI winters of innovation and how, how we got to where we are and what the weaknesses and strengths of where we are and capability is. And even economics. So, you know, how, what’s this going to do to jobs or what do we think it’s going to do to jobs? What’s it going to do to the stock market? All of these different things covered in this series. Really big thinking, I think, from some of the smartest people in the world as guests. And I really love this podcast. Like I said, it’s my second time through it because I think there’s so much to think about and see differently than, than the AI software where we’re interacting with every day. The more larger ramifications of all of this stuff, which is, which is really cool. 00:53:05 | Michael Helbling: Nice. I’ll have to give it a listen. All right, Julie, what about you? What’s your last call? 00:53:09 | Julie Hoyer: My last call, funny enough, is actually a David Epstein letter newsletter. And it’s him and Brad Stuhlberg. And it’s them discussing two books that they recently published earlier this year. The books are The Way of Excellence and Inside the Box. And it’s a really cool discussion back and forth and a really nice highlight of both of the books. And it really made me want to go and actually get them and read them in full. And the title of the newsletter is How to Pay Attention and Care Deeply in a Chaotic World. And so I’ll just give a few teasers of like quotes just from the beginning of it, but it was such an interesting read. And like I said, I think that the books diving into these concepts more would be super interesting to go and check out. So the first one is the quality of your attention shapes the quality of your relationships, work and leisure, which is to say the quality of your attention shapes the quality of your life, which really like hit home for me. And then they talked about it’s changed from time management to attention management. And then the last one I’ll give you is Wealth of Information Creates a Poverty of Attention. So like those three things just like really caught my eye. So highly recommend reading the newsletter. And then I would not be surprised how many people go and purchase the books for more. 00:54:35 | Michael Helbling: Yeah, Julie, if we’re telling people to like trim down, like I think that goes against what we’re doing here. 00:54:42 | Julie Hoyer: But it’s over. It’s about quality, right? We’ve got quality here. It’s okay. Yeah, yeah. That’s right. 00:54:47 | Michael Helbling: Thank you. Appreciate you. Yeah. Keep listening to this, but cut everything else out. Everything else trash. 00:54:54 | Julie Hoyer: We need your full attention, people. That’s right. 00:54:57 | Michael Helbling: Val, what’s your last call? 00:55:01 | Val Kroll: Okay, so mine is a New York Times article that my husband sent me who is a philosophy major. And the title is The Revenge of the Philosophy Majors. And it talks about how a lot of AI companies like Anthropic and Google DeepMind are recruiting philosophers to join their kind of crew and some of the things that they’re really leaning into as some of their like skill sets that are different or have a different perspective or angle than others is around like obviously the ethics, but like being able to take like abstract concepts and make it easier to digest like all those like conceptual challenges around AI. But the one thing that I found really interesting, I mean, it’s super interesting and they have like, you know, different quotes from different people. I was like, oh, like what are they called like forced to play philosophers? Like thinking of being so funny. And I was like, no, they’re just philosophers. Like it’s just philosophers. That’s the title. It’s like, okay. Wow. All right. Well, there you have it. But it’s just so cool to think about like how AI and, you know, some of these roles and how it’s kind of morphing and changing what is seen as mission critical to like move us forward. But it was kind of a fun read. So that’s my last call. 00:56:22 | Michael Helbling: So you’re telling me they can’t be advanced philosophers? 00:56:27 | Val Kroll: Senior advanced. 00:56:29 | Michael Helbling: Advanced philosophizing. Sort of like how we do an advanced analytic. Yeah, exactly. All right. Sorry. 00:56:37 | Val Kroll: That’s good. No, good. Those data philosophers. Yeah, exactly. Yeah. 00:56:45 | Rusty Rahmer: I’ll tell him that he’ll love that. 00:56:47 | Val Kroll: Good data in front of it. 00:56:50 | Rusty Rahmer: Just to build on that for a second. There has been a longstanding philosophical test. I mean, I mean, they’ve been wondering if computers will ever be able to imitate a human for a long time, ever a chief sentence or whatever. And the longstanding test has always been if a human in a blind test cannot tell that they’re having a conversation with a computer, that has been the philosophical bar that sentience has had to reach in computers. If he could do that, then it would be blown that away already. And so the need to define what is it that makes us different than a computer is like a huge philosophical debate right now. And I’m sure that gets right into it. It’s really exciting times for that. 00:57:30 | Val Kroll: How about you, Michael? What’s your last call? 00:57:32 | Michael Helbling: Well, mine is a little more fun this time, but I do love a good allegory. And a blog post that Charlie Tyson worked with shared with me by a guy named Gerard Rubio. It’s sort of an allegorical story about startups. And I thought it was well written and kind of a fun read. So we’ll have a link to it. But basically, he’s sort of talking about a guy who’s decided to start an oven business and goes off into all the different things that happen to a founder through that process. And I think now in this day and age, because it’s so easy to do something with AI, I think everyone sort of has this thought of like, oh, I should start a company. And then this will sort of get you through probably the first few years of that pretty easily. And if you were thinking about doing a product. So it’s a good read. It’s a fun little read. All right. Well, Rusty, again, thank you so much. What a pleasure to have you back on the show and hang out a little bit. It’s always great. I appreciate the insight and the wealth of experience that you kind of distilled into your book, Working As Designed, which is available, I believe, on all the major platforms. So you should probably pick up a copy just so you can say, I have a book by Rusty Raymer. And also read it. Yes. Yeah. So that’s probably more important than say you have it. But anyways, all right, I know as you’ve been listening, you’re probably thinking, hey, I’ve got a couple of questions or things like that we’d love to hear from you. And the best way to do that is through our website or our LinkedIn, or you can email us at contact at analyticshour.io. And also, as you’re out there listening, please add a review or comments also where you listen on Spotify or wherever. So we’d love to hear from you or see reviews and ratings is there as well. Always is awesome. All right. 00:59:34 | Michael Helbling: So I think that’s it. 00:59:37 | Michael Helbling: And I think I speak for both of my two co-hosts, Julie and Val, when I say no matter what your organizational structure, whether you’re a center of excellence, a Tiger team, or a cross-functional porridge, or you’re defining two big customer journeys, keep analyzing. 00:59:56 | Announcer: Thanks for listening. Let’s keep the conversation going with your comments, suggestions, and questions on Twitter at analyticshour on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst. Those smart guys wanted to fit in. So they made up a term called analytics. 01:00:18 | Charles Barkley: Analytics don’t work. Do the analytics say go for it no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 01:00:33 | Michael Helbling: You do in the French Quarter, nothing crazy. 01:00:37 | Michael Helbling: And we had come out of a place and there was a lady that was really pretty inebriated, I would say. 01:00:45 | Michael Helbling: And the rest of us were just like, oh, well, and we kept going. But Rusty, because he’s such a nice guy, stops to try to help her into a cab and make sure she’s okay. And she turns around and she goes, you have the most beautiful salt and pepper hair. 01:01:04 | Rusty Rahmer: Me, Marles, really, her two eyes are going different directions. Yeah. So obliterated, right? 01:01:12 | Michael Helbling: It was something so funny. Because I was like, oh, yeah. But Rusty, because he tried to be nice. And this lady’s like, oh, wow. 01:01:20 | Julie Hoyer: But your image cut through the fog that she was in when she saw you clearly. 01:01:27 | Rusty Rahmer: Hold five with me, I’m sure. 01:01:31 | Julie Hoyer: There’s a whole group of good-looking men that help me into a cab. 01:01:35 | Val Kroll: Yeah, I think one of the stories she tells. 01:01:39 | Michael Helbling: Through this day, he has this image of her guardian angel. Amazing. 01:01:46 | Rusty Rahmer: Convinced she went to hell, unfortunately. 01:01:50 | Michael Helbling: Anyway, good times. 01:01:53 | Tim Wilson: I can, I can, I can hop into the PIVOTS FOR ANALYSIS on, so, listen. 01:01:58 | Rusty Rahmer: I have to write a couple more books, at least, okay? Yeah, at least. 01:02:03 | Michael Helbling: I’m on it. 01:02:04 | Rusty Rahmer: My movie is next, by the way. So that should help me. Oh, well, that’s perfect. Perfect. The analytics movie. That’s what I want to hear. Yeah. 01:02:14 | Val Kroll: Nice. The red stapler becomes, what would it be? I was told I could have access. 01:02:21 | Rusty Rahmer: Yeah, I was told I could have access to the snowflake tables. Yeah, and if I could just, if I could please give me back my access to the snowflake tables. 01:02:29 | Tim Wilson: Oh, Gershoff. Can’t you hear it? It’s a whole movie, I’m telling you. Yeah, it’s a tie between Gershoff, Joe Sutherland, and Gary Angel. Oh, Joe, yeah. Joe. 01:02:37 | Michael Helbling: Oh, we. I feel like there’s somebody else. How many times have each of them been on? Three. Haven’t we had Chelsea on three times, too? I thought so. Yeah. Because we did two in quick succession last year with her. Well, this is what I always try to do analysis on. 01:02:57 | Val Kroll: Yeah, is this like the high problem of like 2010, where it’s like capitalization, so it doesn’t. 01:03:04 | Rusty Rahmer: Look, it’s like a real company here. Tim does all the work, and he’s got four bosses. Tell them what he did wrong. 01:03:11 | Michael Helbling: You know what? Meanwhile, meanwhile, Tim’s over there typing into Claude, and Claude’s like, you’re absolutely right. I did forget about that, guest. You’re right to bring that up. You’re right to call me out on that. Oh, that’s right. 01:03:29 | Val Kroll: One was like the last time. How many times have Tim and Moee fought or something? 01:03:35 | Tim Wilson: Oh, what triggers Tim? Like, those are always. Does it take up on notes and argument? 01:03:43 | Val Kroll: It’s pretty good. The post #303: Funnels Assume Progress. Barriers Recognize Reality. appeared first on The Analytics Power Hour: Data and Analytics Podcast .

July 21, 20261 hr 8 min

#302: It Was a Dark and Stormy Insight…

Here’s a question analysts almost never ask themselves before walking into a room: why am I actually here? Not “because it’s the weekly meeting” or “because someone asked me to pull the campaign readout.” But WHY—what difference will the information coming out of my mouth make, and for whom? Aleya Harris , bestselling author, TEDx speaker, and strategic storytelling advisor, joined Tim and Moe to dig into exactly this kind of thing—and she did not pull her punches. Analysts are often the smartest people in the room, she says, and that might actually be the problem. The gap between “here’s what the data shows” and “here’s what we should do about it” is precisely where story lives, and it turns out storytelling isn’t some soft, hand-wavy thing marketing people do—it’s a repeatable, learnable framework that has been working on human brains for millennia. Plus: a cautionary tale about ranking on page one for the wrong keywords, the tweaking-the-deck death spiral decoded, and an elevator pitch exercise that reveals your carefully crafted slide deck says something completely different than what you actually think. This episode is brought to you by Prism from Ask-Y —your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. This episode is also brought to you by Stape , your all-in-one solution for server-side tagging. Links to Resources Mentioned in the Show Hero’s Journey (Book) Heal Your Body by Louise Hay Ladies First: Wildlife Matriarchies (Video) You Are Contagious – Vanessa van Edwards at TEDxLondon (Video) What If Your Identity Is a Script You’ve Rehearsed Too Many Times? – Aleya Harris at TEDxChantilly HS Photo by Nong on Unsplash Episode Transcript 00:00:00.00 [Announcer]: Welcome to the Analytics Power Hour. 00:00:08.92 [Announcer]: Analytics topics covered conversationally and sometimes with explicit language. 00:00:13.32 [Tim Wilson]: Hi everyone, welcome to the Analytics Power Hour. This is episode number 302. I’m Tim Wilson and I thought I would kick things off with the start of a poem by Edgar 00:00:26.68 [Tim Wilson]: the Analyst Poe. 00:00:29.52 [Tim Wilson]: Once upon a midnight dreary, while I pondered weak and weary, over many a deck and query heaped upon the office floor, while our business partners barely waking, kept on nodding, kept on faking interest in the point I’m making till they wandered out the door. Cry die, surely truth and data should compel them to the core. But these are just facts and nothing more. Okay, if you’re nervous then I’m going to do a raven length recitation, have no fear. We’ll leave it at a single stanza. What we’re going to chat about today though is that pesky reality that while data and its analysis is valuable and is our bread and butter, the data alone is rarely enough to drive positive behavioral change. We need to be compelling to alter the beliefs of our business partners in order to motivate change, and that requires a lot more than just the results of our analyses. For that discussion I’m joined for this episode by my co-host, or by my co-host, my co-host, Moe Kiss. Moe, are you pronouncing any words funnily like I am today? 00:01:38.24 [Moe Kiss]: To be fair, I’m not starting off with a poem with a dramatic reading, so I feel like my job’s a lot easier today. 00:01:46.36 [Tim Wilson]: I also had a dog that we’re dog-sitting walking underfoot throughout all of that, so if I held it together, good for me. 00:01:54.96 [Moe Kiss]: Excellent. 00:01:55.96 [Tim Wilson]: We were also going to be joined by Val Kroll, she unfortunately is under the wetter weather at the last minute. Wetter? 00:02:02.44 [Aleya Harris]: Oh, this is not going to be good if we’re going to talk about diction and using appropriate 00:02:07.32 [Tim Wilson]: words. So, what we thought we’d do was get a professional who actually doesn’t butcher the English language on a, at a moment’s notice, like I am doing so far. So, we recruited a guest for this discussion after we saw her totally own, I mean like dominate, own of the room at Marketing Analytics Summit earlier this year out in Santa Barbara. Aleya Harris is a strategic storytelling advisor for leaders and organizations. She wears many hats at the Evolution Collective, including both CEO and program developer and the lead trainer for Spark the Stage Signature, where she builds your keynote so you leave with a finished presentation, a professional speaker platform and skills to own the stage. She sets a high bar for how she owns the stage. She’s a best-selling author and a TEDx speaker and today she is our guest. Welcome to the show, Aleya. 00:03:00.20 [Aleya Harris]: So much for having me, I mean, I really love speaking at Analytics Summit, but I think I might have loved that poem. Maybe just a little bit more, so. 00:03:12.80 [Tim Wilson]: That was one where there was an AI assist, but then I had to do a heavy edit, so I did practice my dramatic reading a couple of times, so. 00:03:22.44 [Aleya Harris]: And you could, it showed, it was quite good. It showed. And then I also would like to lower expectations for all of your listeners. I make no guarantees that I will say all of my words properly either, so just in case it doesn’t have to require me. 00:03:38.00 [Tim Wilson]: Well, I started by setting the bar pretty low, so, well, well, we’re thrilled to have you and when we. Thank you for having me. Well, so when we saw you speak, one of the things you really hammered on pretty hard is it’s something that I think we all sort of know, but then, like, as analysts, we like to ignore, and that’s that, you know, just the facts, like just delivering the data is not only is it not effective, I think it’s uniquely ineffective when it comes to actually getting somebody to make a decision and drive change. So if that’s a fair way for me to put words in your mouth, can you maybe talk about that a little bit? Like, why are stories so critical when it comes to delivering information in a business context? 00:04:27.72 [Aleya Harris]: Because you put words into my mouth beautifully and wonderfully, I’m here for all of that. So the reason why it’s important is because stories are how our brains work, they function in story. You can give me 10 bullet points of data, but you can tell me a story and I might remember one bullet point of data, but I will always remember the story. I tell this story, which is a true story, about how I, in not a, I say a former life, but then I’m also very woo, so I think sometimes people mean like I meant like, you know, back in the 1800s when my name was Melissa, but no, I mean, like, earlier on in my career, 00:05:08.24 [Aleya Harris]: I was a private chef. 00:05:11.40 [Aleya Harris]: I was a private chef for lots of celebrities and one of them was Stevie Wonder. And one of my favorite moments was we were in Utah because he was singing with the Moermon Tabernacle Choir, of which I had a front row seat to their rehearsal concerts, amazing thing. But I remember the night before, we were all a bunch of us were in his hotel suite. I was sitting on his bed and he was like doodling up a song on his phone. And I started like doodling it with him and I was like, oh my God, I’m making a song with Stevie Wonder. 00:05:54.32 [Aleya Harris]: Is that, is that one? 00:05:56.32 [Aleya Harris]: Don’t worry, it’s not on anybody’s record. I asked if we were going to like get this out into the world, he, he promptly laughed at me. But then the more that I really honed in this situation, I mean, he was adding in lyrics and beats and melody and we were like just doodling up a song and I took me a second to remember that this man is not sighted. 00:06:21.12 [Aleya Harris]: Right? 00:06:22.12 [Aleya Harris]: Like, I was looking at him, doing everything he’s doing, thinking it’s all normal, talking to him, engaging with him like I would a sighted person, but he isn’t one. 00:06:34.72 [Aleya Harris]: And I took a pause for a second to be like, wait a minute, he never put limitations on 00:06:42.52 [Aleya Harris]: himself. Yeah, the Grammy Awards, the fact that he’s like a musical genius, blah, blah, blah. But one of the main moments that I remember about him, about him reaching out possibilities was us making this song with him, doing all this stuff on his phone as a non sighted human. And the question I asked myself was, where have I been letting excuses hold me back? I have all of the resources, all of the things that I can do with myself. He has a severe impediment compared to a lot of other people and yet this man has a household name across the world and it was in that moment that I decided that I actually didn’t want to be part of Stevie Wonder’s entourage. I didn’t want to cook anymore. I wanted to be an entrepreneur and not be part of someone else’s entourage but develop one of my own. 00:07:39.44 [Aleya Harris]: Right? 00:07:40.44 [Aleya Harris]: But it was that moment. Now, I sat and I told you the story. Your listeners here have heard this story. Instead of telling you a story about how overcoming obstacles and watching someone else do it 00:07:59.16 [Aleya Harris]: inspired me to overcome those obstacles, inspired me to overcome those obstacles and 00:08:05.20 [Aleya Harris]: become a business owner, I could have given you a bunch of data around 50% of people that have obstacles to overcome, do it in these ways, step one, two, three, 25% of this, expert gender this, if we have this large demographic percentage of this, da, da, da, da, da, da. Or I could tell you my Stevie Wonder story. Which one do you think is more compelling? Which one do you think is going to inspire more people? The story or the bullet-pointed list of data? What do you think will actually inspire people to participate in their own transformation, which is what I’m in the business of doing? And which spoiler alert? All analysts are in the business of doing. That’s why you’re analyzing things in first place because you’re trying to make something happen. You’re trying to illustrate, hey, something’s going on. There’s connections that need to be made here and there’s things that need to be done because of these connections. 00:09:03.04 [Tim Wilson]: I mean, not to over-index to that story, but the world of overcoming obstacle, you had a, I don’t know if you could fully plot a hero’s journey story out of it, but is there a part of it too that even as you’re telling the story, I was finding myself saying, well, I know where we ended because you’re not doing that anymore. So I’m listening, because it’s got a narrative form, I have no idea if this is right or not, 00:09:36.16 [Aleya Harris]: I know it’s going somewhere, because it’s a narrative, I know it’s going somewhere. 00:09:43.48 [Tim Wilson]: So I have to be on the edge of my seat because I know, whereas if I’m looking at a chart or I’m getting the weekly readout, there’s this nature of, and then there’s going to be another slide and then there’s going to be more data where you clearly with a story you’re on a progression to somewhere, which makes it easier for the audience to pay attention. 00:10:10.32 [Aleya Harris]: Stories, unless you’re like Uncle Bob at Thanksgiving, which I talk about all the time, and I had an Uncle Bob. Uncle Bob wasn’t actually the problem, Uncle Bill was the actual problem, Uncle Bob was fine. I have Uncle Bob and Uncle Bill. Uncle Bill was a real long-winded, but anyways, I digress about my uncles. Unless you’re that person at Thanksgiving or insert holiday dinner here where you’re just like, oh my God, you’re talking about absolutely nothing, this isn’t a story, this is word vomit, you know that if someone is telling you something in what you said was the magic words, a recognizable story structure, you’re hooked. So when people say story is important, they think words are more important than numbers. That’s what they hear, that it’s words that are important and words are important, but it’s not just words, it’s words put in a certain sequence, in a certain framework, in a storytelling framework because that’s how our brains work. So yes, you knew there was going to be a payoff because I intentionally was telling it in 00:11:16.44 [Aleya Harris]: a storytelling framework. 00:11:19.68 [Aleya Harris]: You knew, oh, this is familiar, this feels like a movie I have seen, a book I have read. She, unless she’s like really bad at what she does, is not going to leave me hanging and just being like, yes, so what were we talking about, I just wanted to tell you that, like, 00:11:34.00 [Aleya Harris]: I thought like pajamas are cool, like, you know what I mean? 00:11:39.00 [Aleya Harris]: Like you knew I was going somewhere with this because I used something that your brain recognized, which meant that you were paying more attention, you were also cataloging the details of the 00:11:49.76 [Aleya Harris]: story just in case they were important later, which is what you want people to do with your 00:11:55.56 [Aleya Harris]: data. What do you need them to understand, know, and believe about this data that you’re weaving into a story because it’s important for them to use it to make decisions with later. So you’ve entered into my very geeky favorite territory of storytelling frameworks because stories all exist in a framework, good stories. The most popular one that we deal with is the hero’s journey, like what you mentioned, spot on. There are lots of other types of frameworks and each framework allows you to elicit a different type of response and help people to behave in a different way that you get to, this sounds a little bit like a bit of work with me, you get to control because you are controlling the story structure, you control the story structure, you understand the certain beliefs that people have and they hold as they walk through the world, which means that you can control the messages and the narrative you give around certain data so that you get the end result that you’re looking for. 00:13:02.24 [Michael Helbling]: Tim, here’s a common measurement problem. The tracking event you care about does not always have the user information you need. 00:13:14.12 [Tim Wilson]: Right. You can’t identify as themselves once, maybe with an email or phone number, but later events may come through with missing or incomplete data. 00:13:18.64 [Michael Helbling]: And that’s what Stape Enricher is designed to help with. It’s a state power-up that securely stores selected user data from incoming requests and uses it to enrich future events when information is missing. 00:13:28.32 [Tim Wilson]: So instead of every event standing alone, and Richard can help connect activity across sessions, devices, and even online and offline touch points. 00:13:36.00 [Michael Helbling]: Exactly. That means better event match quality, stronger attribution, and a clearer view of the full customer journey. 00:13:43.12 [Tim Wilson]: And importantly, Enricher is built with privacy and consent in mind, including settings that align enrichment cookies with your site’s marketing consent preferences. 00:13:51.52 [Michael Helbling]: And teams can even upload historical data from tools like Shopify, Clavio, CRM exports, or offline customer lists. So if your measurement has gaps, Stape Enricher can help fill them responsibly. Use the link in the podcast description and check out Stape Enricher today. 00:14:11.84 [Tim Wilson]: Michael, Ask Why’s Prism version 3.0 is here and it officially turns the analyst into the admin. 00:14:20.16 [Michael Helbling]: Ooh, I have always wanted power. Not delete the database power, more like stop asking me which revenue number is real power. Exactly. 00:14:31.24 [Tim Wilson]: With AskWhy.ai, Prism becomes a real workspace platform. 00:14:35.28 [Michael Helbling]: You manage the data, control the context, and empower users wherever they work. So I’m not just answering questions, I’m curating intelligence. Yes, the new memory portal is mission control for what your agents know about your business. Meaning I can search the memory, endorse what’s right, reject what’s wrong, and tune it in plain English. Yep. Importance, trust, and decay scores adjust automatically. Ooh, decay scores. Finally, a polite way to say this metric definition has gone bad in the fridge. And context notes are now refreshable. 00:15:11.36 [Tim Wilson]: Upload playbooks, call transcripts, glossaries, business rules, then refresh them instantly or on a schedule. 00:15:16.76 [Michael Helbling]: I like that. So no more stale context haunting every analysis like a ghost with a spreadsheet. 00:15:23.28 [Tim Wilson]: Go to ask-y.ai and join the wait list and use the code APH to get up at the top of that list. 00:15:29.84 [Michael Helbling]: That’s ask-y.ai with code APH. 00:15:34.12 [Moe Kiss]: Can I dig into that a little bit? Because I’m just going to talk about the problem out loud and then we can kind of have a bit of a discussion about it. One of the things that I observe with data people is that they think the data should speak for itself, right? And what we’re discussing here is really that the narrative in the story is the more important component. But how do you think data people then weave the data into the story successfully? Because obviously that’s the specialized skill that we bring to the table. That’s the information that folks don’t have. And they might not remember all of the data points. You want them to remember the story, but how do they pull those things together well? 00:16:14.68 [Tim Wilson]: Well, I’d also say they’re expected to bring the data. A lot of times they’re dealing with kind of the norms of the organization. If they showed up and went, no data compelling story, that feels super, super risky. So yeah, that’s my addendum to your question. 00:16:31.00 [Aleya Harris]: I love your question, and it is definitely how data people think. 00:16:36.72 [Aleya Harris]: And it is a fundamentally flawed question in its structure. And I’m very glad you asked it this way because this is how it’s asked all the time. How do I weave data into story? What you really need to ask yourself is, how do I create a story from this data? What is the data trying to tell me? What am I not only extrapolating, but how do I use the key numbers and the key points along the journey of what people need to do in order to use this data? So when you’re thinking of a perspective, let’s say you have a dashboard, 00:17:21.52 [Aleya Harris]: you have website analytics in front of you, or you have any data set, really. 00:17:27.56 [Aleya Harris]: And you guys have actually the hard job because you have to know all of the technical part. And you’re like, great, I just spent a longer than necessary, exorbitant amount of time pulling together this data, making sure it’s clean, making sure it’s correct, making sure it’s usable data. And now I got to explain it to you, which is where I think some of this data should speak for itself comes from because I’m done. I am now done with this. Now shouldn’t this be your job when actually, no, it’s not. You put a bow on your work by saying, OK, let me take a deep breath. I now have this information. What am I going to do with this? If I have this information, how am I using this to make business decisions? How should somebody use this to make business decisions? If I have SEO metrics, I have a bunch of, you know, these are just hot keywords. These are how people are finding you. This is the search. What is this really saying? Well, it’s saying this is the problem. This is the solution. This is where we are in the journey. This is where we’re trying to go. The challenge with a lot of data people is you are the smartest person in the room. I said it. I’m going to keep saying it. I said it in the conference. I’m going to say it again. Right. And so you’re like, well, isn’t it obvious? Isn’t it obvious what this says? No, ma’am, it’s not. Sheila, it’s not obvious to someone who is not you. So you need to make what is obvious to you obvious to other people. And the way that you do that is by busting out of your silo and asking questions that might even seem unrelated to your data. Like, what are our business goals? What are you trying to do with the website? What are our larger marketing initiatives for this next year? Right. What are we struggling with as a company? And then you look at the data and you’re like, OK, I got it. So like this happened to me where I do not pretend to be an analyst, nor to play one on TV. So let’s get this real clear before this next story. But I know enough to make me dangerous. And I got from my SEO person a very nice dashboard of which he didn’t do what I said. And he just sent me a bunch of keywords and numbers and rankings and all of this stuff. And I looked at it and I said, no wonder why all of our marketing is going nowhere. We’re doing because he goes, you’re doing great. You’re ranking really well for I’m not going to use actual keywords. But I want to out the company for apples and oranges and plums. That’s awesome. You’re a fruit seller, right? You want to be waiting for apples, oranges and plums. This is great. To which I replied to him the equivalent of, well, we’re actually not like a fruit seller in that we want to want to be known as a fruit seller. We’re actually like a fruit package. Like we sell the fruit packaging like the bowls, not the fruit. So ranking for apples and oranges and plums might sound like we’re the number one person 00:20:48.68 [Aleya Harris]: that when you Google apples, you’re going to find that doesn’t make us anybody. 00:20:55.56 [Aleya Harris]: So you then as the SEO person need to be able to look at data because you know that we actually want to be ranked for the bowls, not the apples. And it’d be like, oh, I understand our marketing objectives. I understand our business objectives. I actually know that this data is bad, says bad things, even though we’re number one. Page one, the SERP is bang and our name is everywhere for the wrong thing. Our content strategy was wrong and bad and all over the place. And wouldn’t it have been nice for my SEO person to know that before they came to me 00:21:28.12 [Aleya Harris]: who was the CMO? 00:21:30.24 [Aleya Harris]: So excited about their data. 00:21:31.56 [Tim Wilson]: I mean, I think you’re hitting on some like very fundamental things. I think there are times where it’s, I have this data, I have been told to crunch this data. I’m the expert on this data. And therefore I need to crunch the data until I find something that I can slap a label of a story onto. And I will claim hard that there is a lot of data that is just showing things are going as they expected. And that’s like a massive issue with the industry at large is there’s a belief that if you pulverize the data enough, something will emerge. And I think analysts wind up in this spot saying, well, I got to do a deliverable. And so then they start slapping. What story can I tell, but their story winds up being, oh, look, you’re ranking for this irrelevant fruit, you’re ranking for fruit when that’s not what you’re selling. But that kind of pairs with the flip side, like really asking not starting with, not even starting with the data, I would think it’s not start with the data and what stories is telling me. It’s starting with what is the real problem I’m trying to solve or what is the real idea that I’m trying to validate? To me that, and then I may go to the data, but then I’ve got kind of the kernel of a story because I’m trying to, I kind of know what the front, kind of what a couple of alternate endings, the ending is either go spend more money there or spend less money there or whatever it is. I’ve kind of outlined some possible endings to the story. Then I’m using the data or the experiment or the research, whatever it is, I think. I mean, I just feel like analysts find themselves saying, they sort of paint themselves into a corner and then they say, well, I don’t know, I don’t have a story to tell. It’s like, well, yeah, because you kind of painted yourself into a corner to start with. I got triggered on a few fronts. 00:23:25.68 [Aleya Harris]: You know, I don’t know, I didn’t know analysts were so judgy. 00:23:30.20 [Tim Wilson]: I’m pretty judgy. That’s not, that’s not paying too broadly, but I say that, I say that in the way like 00:23:40.52 [Aleya Harris]: you’re judging your story as your data is bad, is it telling like a bad or unnecessary story? Just tell the story. Tell me what the story actually is and allow it to be. Tell me what then you are now, what does that make you as an analyst more curious about? Okay, everything’s going like how it’s supposed to. Is this the direction that we want to take it for the next five, 10 years? Tell me now, CMO, this direct, this is what you told me you wanted to know. We’re ranking for these things. We’re ranking for those things. It’s all good. I can leave it at that. Or I could say, yeah, we’re ranking for these things. And what does that mean? Often analysts are supposed to be the source of truth and I would love, would have loved it and love it more when I’m working a fractional basis as a CMO. 00:24:44.36 [Aleya Harris]: If analysts asked me more questions, that, I mean, I guess depends on the personality 00:24:52.12 [Aleya Harris]: of whoever you work for, right? But if the analysts that I have had had been like, great, here’s the information. It’s what we expected. This is ranking here. This is what this is saying. This is the story that is telling the story that we are the best in the market. We are the worst in the market. We’re ranking high for these keywords. The money that you wanted to spend on additional, whatever you can spend, great. I would love then the next thing that it’d be like, now tell me, this is great for right now. 00:25:25.72 [Aleya Harris]: What about next quarter? 00:25:29.72 [Aleya Harris]: Is this what we’re still trying to do because then I can start monitoring this in place and then use your freaking brain to do the analysis, this is how you get to do the analysis that you want to do by asking you the questions in advance and not being so reactionary. So then you’re not just the storyteller, you then become the story author, which makes 00:25:49.56 [Aleya Harris]: you a more valuable employee for me as a marketing lead because you’re now my partner. 00:25:57.08 [Aleya Harris]: You’re the person who is asking because I might know what I’m doing, but I might not know what I need to be looking at to know in advance what the signals are to tell me if this I should be moving in this direction. If if I’m moving in a certain direction, the early signs that it’s working or not working, you know that. So ask me the question so you can know what to monitor. So you’re not coming back to me every month saying looks great because I’ve had that too. 00:26:23.92 [Tim Wilson]: It’s annoying. I might have tried to start an entire consultancy based on the premise that you need to do more talking and interrogating of the ideas of the people in the company and before you jumped into the data. So I am my my heart is singing because you just sort of put on another hat of like the CMO hat. I think analysts, we have there’s this wiring of listening for a data poll and wanting to get out of the room as fast as possible. Like you start to say, well, I kind of think our SEO and they’re like, great, I can go pull an SEO report as opposed to stopping and saying, I don’t think you really give two shits about your SEO. Like what are you really trying to do? You’re trying to do X and Y and Z and and I think that the business, the CMOs and the other business partners, they try to meet the analysts where they are by using their terminology and the analysts mishear that as being overly prescriptive. 00:27:20.40 [Aleya Harris]: Okay. 00:27:21.40 [Tim Wilson]: Can I pivot things a little bit? 00:27:25.08 [Moe Kiss]: That’s okay. 00:27:26.08 [Aleya Harris]: I’m right there with you. 00:27:27.60 [Moe Kiss]: You just said something to him that I want to I want to dig into you. They want to get out of the room as quickly as possible. And one of the things that I just like really observed both watching you present and talking to you now is like this absolutely infectious enthusiasm and just like Tim said in the intro, right? Like you own the stage. And I suppose one of the things I’d love to get into is like, yes, the story is important, but it’s also how you tell it, especially if it’s like in a presentation format. So like how did you develop that skill? Like I’m someone who didn’t used to do public speaking. I was petrified of public speaking. I had to really put in a lot of energy and learn how to do it. And I feel like maybe we had a similar tact where like, I just took all my nervous energy and tried to like channel it into enthusiasm. But I do feel like there are so many, so many folks who like, it is such a daunting thing. And like, if you tell the story in a really average way, people probably still won’t listen, right? Like they might remember a bit of it, but it’s like so much of why you’re persuasive is how you tell it. So like, can you share like your tips and tricks on that part? 00:28:42.40 [Aleya Harris]: Well, first, that is true. But the framework, once you learn the actual storytelling frameworks, it does a lot of heavy lifting for you. Even if you have a monotone voice, even if you’re not super energetic, the framework does a lot of heavy lifting. So when I work with people and I help them put together their keynotes and I help them get on stage and feel confident on stage, I always ask them to create videos and we could cheek them and they’re in delivery and the very next thing I say is, and don’t try to be me because if you have a great gregarious personality, then great. Like I am like this all the time. I don’t, this is not like how I am on stage, how I am on the body. I’m like this all the time. It’s a lot. I know, but it’s, it’s, it’s up to how I am on stage because that’s how I am in life. I don’t want an introvert who is normally a much more serious, you know, put together individual to try to then emulate my style. They’re going to be awkward and everyone’s going to feel awkward in the audience. Right. That’s not going to work. So I want them to use a storytelling structure and then find their voice. A lot of times people feel insecure with speaking because there’s a lack of clarity and a lack of clarity causes and or masquerades as a lack of confidence. And there’s more that you need to be clear on than you think that you do. It’s not just, what am I going to say? Like, what am I bullet points? What’s my message? Like that’s very important, but you need the more data points. You can be clear on having, I couldn’t help myself, the better. So yes, what is your message? Are you crystal clear? But why are you delivering that message? Why are you standing in that room right now? Why are you the only person that can deliver that message right now to these human beings? Why do you deserve to be listened to? Why is that message at this time going to cause someone else to participate in their own transformation? Those are the types of things you get clear on. And so it doesn’t matter whether you are a jokester. It doesn’t matter if you’re serious. 00:31:13.28 [Aleya Harris]: It doesn’t matter if you have some other personality type. 00:31:20.04 [Aleya Harris]: If you are clear on why on the wise, you’re clear on the what, which most people actually are a lot clearer on the what than they give themselves credit for, right? Like, you know what you’re going to say is not that’s actually the easiest part. It’s the wise that they need to work out. And then once you do that, then you get clear on what I do with my hands, right? Your physical body in space. How do I want to show up? If I show up with my hands like this or something like this, how will I be perceived? Okay, if I show up still me, but I hold my hands out, how am I perceived? And you start then becoming curious about how bodies look in space. Like you walk down the street and you see someone with their hands on their hips. What do you think that they’re feeling? You walk down, you see someone like waving their hands, flailing in the air. What do you think that they’re feeling? Right? And you start understanding, oh, I’m actually clear on the story that bodies send. So now I’m clear how to hold myself in space. And now I’m clear on why I’m there. I’m clear on what I’m saying. And most importantly, I am clear that, and this is true again, no matter what you’re talking about. 00:32:44.36 [Aleya Harris]: I am clear that in this moment, at this time, I will make a difference in someone else’s life 00:32:54.28 [Aleya Harris]: with what comes out of my mouth. I will pause it, period, not trying to like gas you up. It’s just kind of like it is what it is. We can get metaphysical about it if you want to. Just believe me, trust me, this is not that long of a podcast. You will make a difference in someone’s life with whatever’s coming out of your mouth right now. So don’t worry about you being effective or not. Worry about the other stuff and getting clear on how you will be the most effective you can possibly be on that stage with whatever time that you are given. And everything else will work itself out. It’s amazing as I work through these things with people, with my clients, the clearer we are, 00:33:36.88 [Aleya Harris]: the more confident they get. 00:33:39.60 [Aleya Harris]: In the beginning of my program, I don’t talk at all about how much your rates should be, when you charge a speak, I don’t talk about speaking goals, I’ll talk about another. There’s no goal setting until the end. And I one time had a client ask me or give me feedback, like, well, that seems a little backwards. Like I’m used to, you know, doing goal setting at the beginning and it would have been really nice to have my goal set in the beginning. And I said, I get that, I type A too. But if I asked you in the beginning, when you were unclear about who you were, why you were there, if you had value, if your stories mattered, if you mattered and you were having like an existential crisis that you didn’t even know you were having, and I asked you how much you should charge to be on that stage, you would have said I should pay them to let me have the honor of opening my mouth. Now you’re sitting here asking me if you should start off at five or $10,000 to charge, to allow them to earn the right to listen to your wisdom, all because you got a lot clear on those things that I was talking about. 00:34:44.56 [Tim Wilson]: So that is you and just that framing is kind of around the like public speaking keynote, you know, audience sharing information. I assume that clients come out of that and it fundamentally changes the way that they, even in a conference room with five people or even potentially even on a Zoom call, how they put together, like it scales up and down, right? Like when presenting, oh, I’m just an analyst and I’m just delivering this to, you know, the marketing team, it’s not a public conference keynote, but I guess two questions, all this still absolutely applies, I’m 100% sure, but how do we get over the hurdle of saying, but this is just one presentation, I don’t have, I can’t invest the time and the effort to turn this into a proper story, I don’t have the capacity. I have thoughts on what I think the answer is to that, but I’d love to hear what you think. 00:35:58.00 [Aleya Harris]: First of all, you’re like, I’m so sure I’m like, are you there? You don’t sound sure. People think that storytelling takes a really long time and that’s actually very much not true 00:36:09.24 [Aleya Harris]: because of what I was talking about before, frameworks. 00:36:14.60 [Aleya Harris]: Once you know the framework, you start thinking, talking, acting in story without even realizing it. So it’s more about an investment. You had a great question, does this apply to me? I have a better question. Can you afford to not invest in other people understanding what the hell you’re talking about? 00:36:39.72 [Tim Wilson]: Well, that tends to be my, when people are like, I’m too busy, I just have to get stuff out the door and it’s like, well, how’s that working out for you? You’re training your business partners that you’re just there to crank out as much volume as possible. And they’re bored, they don’t know what to do with it, they can’t use it and trying to. 00:36:56.36 [Aleya Harris]: But I think the other thing you’re hitting on there is that, yeah, probably the first time you got to find the time to do a better job 00:37:05.84 [Tim Wilson]: and it’s not going to be perfect, but it will be better. And by the time you’ve done it, I love the way you framed it as saying, you get to where you just see it and think in those mode, in that mode, it totally makes sense that it then becomes easier because you’re not trying to make a bunch of slides and then go back through the slides and rearrange the slides and figure out what else you should add and not take something away. Because you’re kind of, you have the story and then it, I assume it’s been pretty quick to then say, okay, now I just got to articulate, give the visual backup to it. 00:37:42.84 [Aleya Harris]: When you’re tweaking and re-tweaking and tweaking, tweaking and then you do the tweaks for the tweak-tweak-tweak-tweak, which is like the tweak dance that everybody does, that means that something fundamentally is wrong. That means that fundamentally you are insecure about the structure, the way you’re going to present this information. You don’t know what or why? Yeah, you don’t know what, you don’t know why, you don’t know how, you don’t know nothing. All you know is you got some numbers that somebody asked you to tell them about and you have no idea how to do that. Just how numbers work in a very logical way, stories are the same. People often view them as like opposites, like words and numbers, you know, require different parts of your brain. And like what’s easier, trying to like be a number person and then transform yourself into a word person, or is it trying to make repeatable, predictable processes out of the way that words work? 00:38:54.36 [Aleya Harris]: I don’t even think we should do the second one, and that’s by using structures and framers that are repeatable, you know, 00:39:03.24 [Aleya Harris]: and it’s not like, oh yeah, they work like this is the new 2026 TikTok friend, no, they work for Eon, like millennia, right? We’ve been using the same frameworks to communicate effectively with each other. And so once you realize that, you’re like, oh yeah, I can use a story framework because it doesn’t sound so like woo woo, oh that’s what marketing like the markers do, like no, it’s how an effective leader communicates. And then if you’re money motivated, like I am thinking about it this way, if you want people to understand you as a strategic thinker, if you want people to understand you as a true people data and information leader, you should use story. Because strategy is just the ability to put a bunch of random stuff together in a framework and make it work. How are we getting from point A to point B? Oh my god, doesn’t that sound like once upon a time all the way to the end? 00:40:12.28 [Aleya Harris]: So tell me if you were a data person and you’re listening, like I feel like you’re going to have a lot of folks nodding along to everything you’re saying, 00:40:20.04 [Aleya Harris]: but like let’s say tomorrow- I should hope so, and if you’re not, rewind it, listen again, email me if you have questions, because if you’re not nodding, I’ve missed you and we’re trying to bring everyone along on this journey. 00:40:35.64 [Moe Kiss]: Do continue Moe. I love that, I love that though. So tomorrow you’re sitting down at your desk or today you’ve listened to the episode and you’re like okay, but practically how do I do this, right? Like how do I sit down? I’ve got a presentation I’m doing next week for some stakeholders on analysis of how people are using a particular product on our site. Like what do you recommend they do as they’re sitting there thinking through how to craft that story? 00:41:04.20 [Tim Wilson]: Okay, if it’s for real tomorrow, we ain’t got a lot of time, so we’ll give you the short short version. Well, they’re sitting down tomorrow, the presentation is not for a week or two. Yeah, yeah, okay, there we go, there we go, there we go, there we go. 00:41:15.40 [Aleya Harris]: I would recommend you start simple, Phil. What’s the beginning, middle, and end of the, of how the information should be used? 00:41:27.64 [Aleya Harris]: Because you’re not telling a story just, let me back up. 00:41:33.80 [Aleya Harris]: When you’re telling a story, you can tell a flat story that no one cares about, which I don’t recommend. I don’t recommend you like, you like, say, this morning I woke up, I did my skincare, and then I got dressed and I left the house. The end. Like what? Like that’s not even, it doesn’t have the elements. I mean, technically at the beginning I woke up and I said the end, 00:42:00.92 [Aleya Harris]: but there’s no purpose to the story. It’s worth thinking more when we think of stories more like Aesop Fable’s kind of situation, 00:42:09.56 [Aleya Harris]: where there’s some lesson to be gleaned and there’s a reason why you’re telling a story. So the first thing you have to sit down and figure out is why the heck am I talking? Why am I opening? Besides the fact that like, this is a weekly meeting and like I’m supposed to be here every week. 00:42:26.36 [Aleya Harris]: Like, why am I really talking? That is what I call your controlling idea. 00:42:32.52 [Aleya Harris]: What is the subject that you’re talking about? And what is your point of view? Because I can talk about like, oh, this, you know, is a dashboard and this means our product is selling well. Great. What is your point of view based on the brain that is sitting between your two ears? 00:42:58.68 [Aleya Harris]: That is the controlling idea of your story. Then you say, okay, I have the point of view like, 00:43:04.92 [Aleya Harris]: this product is selling well based on the data that I see that I’m reporting this month. However, my point of view is actually it’s our dog. It’s our worst seller. We’re having a blip month, because I know we have this major client come in and it’s skewing the data. So it shows like, this is our best product ever. But really, if you remove this outlier, this is actually our worst product. And as opposed to the recommendation of pumping more marketing dollars into this product, I actually think that we should explore either sunsetting it or repositioning it. That’s my point of view. 00:43:46.44 [Aleya Harris]: How do I know that? Because I looked at the data. Then you’re like, okay, I have my controlling idea. 00:43:53.88 [Aleya Harris]: A simple story structure is what is the problem? What do we learn from this problem? 00:44:02.84 [Aleya Harris]: And how does that relate to the broader context? So the problem is, if I was going to tell the same 00:44:11.08 [Aleya Harris]: story, well, you start kind of like in your ordinary world or whatever is going on normally, right? So the normal thing is that I’m happening is like, here I am, here’s the data, here’s what it 00:44:20.52 [Aleya Harris]: here’s saying this product is going gangbusters. But the problem is we’re not looking at the full 00:44:27.64 [Aleya Harris]: set of data. And the problem is our own success that we have this great client. But my concern is that we don’t have a reputable predictable way of bringing in more whale clients like this one who are going to regularly buy this product. So the data is skewed. So what we can learn is the data is skewed, we need to get underneath this data to see what it really is telling us and actually learn what our highest selling top three, whatever the thing is that we’re supposed to be doing here is so that we can put our marketing dollars in a place that matters, either making a reputable process for landing more whale clients that we’re going to buy more of this thing, or putting more money behind the other top three who are regularly performing well. Because I know that our company’s goal is, in fact, to sell and to go into the direction where we’re selling more and more and more of this product. My recommendation is that we, in fact, focus our marketing dollars and efforts on expanding this product line to get more clients to buy it, whatever it is. That’s how I would love if analysts came to me. 00:45:51.80 [Tim Wilson]: That was like a, that wanted to be like a detailed scenario, but I kind of latched on to your why am I, why am I being asked to, why am I telling the store, why am I presenting? Like, and if it’s, if my answer, if the analyst can’t answer the question other than saying, well, it’s the weekly meeting. And I could, I could see it’s like, well, next week you’re kind of screwed, but maybe this is encouraging you for the next time to actually go ask more questions so that you can, like if, if you can’t, that may be really uncomfortable for an analyst to say, I don’t know. It’s just because I’ve been told to give the campaign readout and I don’t, and it’s, and it’s too late. I could be like, well, then you better look ahead for the next one and kind of lay the, lay the groundwork a little more effectively with the questions you’re asking. I mean, and it doesn’t give you a sense of purpose. Like for me, it 00:46:49.96 [Aleya Harris]: drive me back to do crazy to have my thought that my job was just crunching numbers floating around in a vacuum. Like when you understand how what you do on a daily basis impacts everyone that you work with by putting money in the company’s coffers or not, doesn’t that, that feels, that would feel much more fulfilling to me. I would get more joy out of my work if I, if I knew that. 00:47:22.68 [Tim Wilson]: I think it should. I will, my other, my cynical rant number like four, I think analysts let themselves get, they tell themselves the story that that’s just my job is just to pull the numbers and just to present them. And then they tell themselves the story. This won’t be, if I come with a point of view, well, somebody may disagree with that. And then I’m going to have to have a conversation with them. And it may be that my point of view is actually wrong. And it’s scary. And it’s like, no, no, no, I just want to pull the numbers and let the, let my business partners 00:47:54.36 [Aleya Harris]: interpret it. So I’m very, this probably is snarky. And I get it. Like you’re speaking for everyone else out there, but like when I was just the word analyst, that means that you’re going to like analyze something and tell me, like if I go for an analysis, I’m assuming you’re going to tell me the thing and tell me what’s going on. You might not know how to fix it, right? You might not have all of the bells and like whistles in your toolbox to know how to fix it. But like if I go for a lab result, and like it just gives me all of like the blood count, like new numbers, I’m like, so like, do I have diabetes or not? That’s a great analogy. Like if somebody goes in, you’re 00:48:49.40 [Tim Wilson]: like, sure, my chart notified me, but I actually want to talk to the person who can actually 00:48:56.12 [Aleya Harris]: tell me how I should feel and what I should do. But case in point, the doctor is going to say like, 00:49:04.12 [Aleya Harris]: okay, yes, you do have diabetes, here’s your treatment plan, but I’m expecting someone giving me an analysis of the, of the results to do all that. But I mean, I need you to like, as my husband says, I’m always like, finish it. Like, like, like, tell me the fault of the thing. Well, and it, and 00:49:22.44 [Tim Wilson]: this is, let’s see if I can tell it generically, because it was at the conference where we saw you, and there was a session that made my blood boil, where what he’s right now, some tea, all generic tea, but it’s still, it’ll be generic to be there. Jim Stern will be I wonder which one he was talking about. Because there’s so much around AI, and somebody really stood up and proudly demonstrated how he had managed to set up, set up all these agents so that he could now take his weekly reports to all his clients. And I won’t say what the reports were about, he was kind of a consultant, and he could just automate it and it would just like, spit the report out itself. And it’s the classic, I feel like we’re starting to, people are starting to realize, I like to think I was saying it pretty early, that people were putting volume over value and what he had done, he’d taken what he wasn’t doing well, which this process he had that generated, it’s like that SEO report you were describing earlier, and he said, aha, guess what I can do? I can do that faster and 00:50:32.76 [Aleya Harris]: longer. And I’m like, that’s not actually helpful, like there’s no way. It’s not. And I think he got 00:50:41.48 [Tim Wilson]: excited because the side of him, like the tinkering and getting to like build, like sure, I like to bang out code and have it generate stuff. But we tell ourselves a story that, oh, I could even give it that it has to have a narrative form and it will generate a beginning and a middle and an 00:50:57.16 [Aleya Harris]: end. And it’s not useful. No, does it do what I said in the very beginning? Is it helping someone or something as a company participate in their own transformation? If it’s not doing that, stop 00:51:12.68 [Tim Wilson]: tinkering. But analysts will tell themselves, they’ll tell themselves that it is. And then now the other side of them. Y’all lie to yourselves a lot, as my people do in this podcast episode. Other side of their mouth, they will complain about how my business partners don’t know what they’re doing. Yeah. So I don’t think I was realizing this was going to be so much of me ranting. Can I ask, I had a more tactical… I’m actually here for it, Tim. I’m here for all of the rants. I’m thoroughly enjoying myself to continue. Listeners know that I’m not going to be the one who’s positive about anything in a optimistic. But as you were… Because as you kind of work through that scenario and you kind of just made up kind of a silly one on the fly, 00:51:56.04 [Aleya Harris]: how do you feel? I have occasionally said, I’m going to walk away. I’ve looked at the data. 00:52:01.56 [Tim Wilson]: I’m tempted to start trying to make slides and make charts. But if I walk away, I’ve got to drive somewhere. Can I just out loud, not walking down the street waving my hands where people look at me funny? But can I just talk through what I think the story is and just keep talking through it? How do I tell this story in five minutes and keep doing that and iterating until I feel like I’ve found what it is on my actual point of view is like away from the computer? 00:52:35.00 [Moe Kiss]: Tim, I feel like I do that with analysts a lot, where I’ll be like, hey, you’ve written this summary up. I don’t know what it’s saying. I want you to pretend you’re in the elevator with our COO. You’ve got two minutes from ground floor to seven. Why should he care about this? Tell me in your own words. And it’s like when people say it out loud and they practice that story, I do feel like it gets clearer much faster. And you’re like, and that doesn’t match, that doesn’t 00:53:04.76 [Tim Wilson]: match your deck at all. Like what? It doesn’t often match what they’ve written down. 00:53:09.64 [Aleya Harris]: As people put in the deck, the things that they quote unquote should be saying. The things that feel very smart and listy. But the smartest people are the ones who can make things sound the simplest. And the actual way to showcase your intelligence is to make the dumbest person in the room 00:53:31.32 [Aleya Harris]: understand the most complicated thing you do. So not saying that in this example, the CEO or 00:53:39.16 [Aleya Harris]: whoever the dumbest person in the room, however, when it comes to data, they’re probably not as 00:53:45.00 [Moe Kiss]: great as you are. They probably don’t have the depth of understanding because there are across so many things, right? So it’s like you have to bring it up to like a different altitude because they don’t have the same like subject matter expertise and like, yeah, 00:54:01.08 [Tim Wilson]: like depth of the particular topic that you’re discussing. Wow. Well, that is a great note saying that executives are stupid. No, that’s not what we said. And that’s where we’re landing. But we are running short on time. I feel like I need to go like write down a few notes immediately following this because a few little light bulbs have gone on for specific mindsets and tactics and ways to think about this topic. So thank you. We could talk for another hour and a half. But it is time for us to move to a wrap. And the last thing we like to do on the show before we close out is go around and share a last call, something that might be of interest to our users. And Alea, you’re our guest. Do you have a last call you’d like to share? 00:54:53.64 [Aleya Harris]: I don’t know if this will be of interest to your users or not. And it’s seemingly unrelated. So, but work with me for a second. Okay. We’ve talked a lot about storytelling. We’ve talked a lot about words and making data and words play well together. We did touch a little teeny, teeny, teeny tiny bit on your body and how when you understand how to hold your body in space, 00:55:26.52 [Aleya Harris]: you can more confidently tell a story. When I work with analysts, not with a lot of analysts, 00:55:34.28 [Aleya Harris]: no matter how intelligent the words are that are coming out of their mouths, a lot of the times their body is saying the thing that you just said, Tim, please get me the hell out of this room. How fast can I get the hell out of this room? I don’t even know why I have to be here. This damn weekly meeting could have been an email. 00:55:54.76 [Aleya Harris]: I would have felt much more comfortable that way. So there was lots of somatic work out there. But 00:56:03.40 [Aleya Harris]: one of the books, I like to have it on my desk, that seems, again, unrelated. And it’s, how many pages is this one? It’s not even, you don’t even really read the whole thing. The whole thing is 00:56:16.12 [Aleya Harris]: 80 pages. And it’s a book by Louise Hay called Heal Your Body. And it’s more of a reference than 00:56:27.48 [Aleya Harris]: like, let me read every page. And it will go through and it talks about different things that you could have be dealing with, the root cause of where and how it gets stuck in your body and a mantra to release it. To make sure that expectations are clear, 00:56:45.96 [Aleya Harris]: saying a mantra is not going to cure an incurable disease here. Like that’s not where we’re at. 00:56:52.44 [Aleya Harris]: But I’m giving you a starting place to think about your words, yourself, your purpose a little bit differently. Like, so for example, like, let’s say, oh, I feel like I flipped this open. I have people sometimes read the Bible, you flip it open to see whatever page is open. And like, that’s what God wants us to kind of mean. Apparently, I opened up to see and it says cough. And it says, if you have a strong cough, it could mean you have a desire to bark at the world, see me, listen to me. Now, you could read that and say, that’s a bunch of poop into dupes. I disagree. Or you can at least give a second where you’re like, is there any truth to that? Do you don’t have to tell anybody else what there is or not? I don’t know. I’m not there. Which you, you listen to me and we, this is not a live conversation. This is asynchronous. Then next to that, it says, I am noticed and appreciated in the most positive ways I am loved. Now, I have a family member who has a chronic cough, not chronic sickness, not long COVID, just always kind of coughing, always clearing their throat. 00:58:11.40 [Aleya Harris]: This person also is the person who, I don’t know why they do this, but they always are, 00:58:18.28 [Aleya Harris]: like, if we’re walking in a group, they walk behind us. They’re like, would prefer to not be seen. They’re like chronically, like, almost like debilitatingly introverted at times. And then when I read this, and I didn’t plan this, but this is what I opened up to, and it says, a desire to bark at the world, see me, listen to me. That makes sense to me. There’s a kernel of truth in that. And even if it were just a kernel, and you said to yourself, 00:58:46.12 [Aleya Harris]: if you are a person with this cough situation, every single morning, or you said it before 00:58:52.20 [Aleya Harris]: every meeting to yourself, I am noticed and appreciated in the most positive ways I am loved. Do you think there’s a chance that you might show up with that story a little bit differently in the room that you enter? Is there just a little chance? There’s a little chance, isn’t it worth taking, to understand how your body could stop telling on you, so that you really could own the power that your intelligence provides you with. So that’s what I have for you. It’s a very little book, very cheap on Amazon. Louise, 00:59:24.60 [Aleya Harris]: hey, heal your body. That sounds way more productive than having like a little cartoon 00:59:30.68 [Tim Wilson]: flippy chart, you know, like the, you know, Dilbert comics, like this is like a, like, wow, you’re actually going to make- However, don’t hit on Dilbert. This Dilbert is funny. 00:59:40.12 [Moe Kiss]: I’m not a wah-wah person, but there’s something about that hit, which is not the, do I believe this or do I not believe this? It’s like, how do I take a kernel of this thing that might help give me, you know, a different mindset for the day? And that was, yeah, 00:59:56.84 [Tim Wilson]: I really appreciate that. I’m so glad. And Moe, what is your last call? And by the way, 01:00:02.20 [Aleya Harris]: I’m sorry. Before Moe, before you go, I think that it’s hilarious that in the U.S., it’s woo-woo, but in Australia, it’s wah-wah. So now I kind of want to know what other countries call it. 01:00:12.12 [Moe Kiss]: I probably, I probably just fucked it up. Like, let’s be real. Like, that’s the most likely outcome. 01:00:17.72 [Tim Wilson]: That’s how you’re going. It’s, how do you know if somehow it has like four, four syllables in it? No. Yeah. No, no. Yeah. Okay. I have a weird one. 01:00:28.12 [Moe Kiss]: I was walking to school with my son yesterday, holding hands. It’s just really cute. We’re still at that age. He likes to do that. And I was telling him all about his Nana. Well, my Nana, sorry. And who she was as a person. And I was like, oh, Nana was a matriarch. And I was like, do you know what that means? And he goes, yes, a matriarch is when there’s a leader of the group and the leader is a woman. And I like nearly fell over myself. And I was like, you were raising him right, Moe. I was like, wow, where did you learn that? And he’s like, well, elephants and lemur families are both matriarchal families. And I’m like, shut the front door, kid. Like, this is the best thing you’ve ever said. And it was like a zoo show on ABC IV, which is our like national broadcaster, where he had learned all of this. And it just, yeah, the reason I share this like little story is because I don’t know, like there’s little, there’s little moments with kids where you can like, have a really amazing positive influence, or maybe you just underestimate the like, the way they’re learning. I don’t know, it’s just, it’s really been the highlight of my week. And I often talk about academic stuff. And I wanted to share that. However, however, I do have a two for because during the course of this show, I did kept thinking, I kept thinking about a Ted talk, which is Vanessa van Edwards, you’re contagious. And she talks about when you’re presenting and like, you put your arms behind your back and like, what that does to like, how people receive you. So if you were really interested in what a layer had to say, that would be a really good Ted talk to go check out. Anyway, sorry to talk to him. 01:02:08.60 [Aleya Harris]: That was so nice. In addition to my Ted talk, feel free to look up mine as well. Thanks so much. 01:02:14.44 [Aleya Harris]: Absolutely. I gotta get a shameless plug in there, Moe. Come on. 01:02:19.96 [Tim Wilson]: Totally, totally. And Moe, like it sounds like it was, you know, somewhere down under there was a nice kind of a, there’s some good TV programming, we’re shutting down any sort of that kind of messaging and educationing, like education over here in North America. So that’s just rub it in, rub it in our face that you’re, you still have that sort of thing. Yeah. I’m a real human. I 01:02:42.44 [Moe Kiss]: didn’t shut out a layer. I’m like, you know, talking about how great on TV is. I’m really in 01:02:48.12 [Aleya Harris]: a role today. It’s, it’s all right. There is, there is not much more you could do for an American 01:02:54.44 [Aleya Harris]: to make us feel worse than we already do. Sorry. Over to you, Tim. So Tim, what’s yours? Yeah, 01:03:06.12 [Tim Wilson]: I was going to say that, Moe, you’re going to ask me what my last call is. Yeah, what’s your last call? Jeez. That’s what I was counting on Val to be here for. So I’ve dipped into this well a few times over the years, because it is still always just a fun site. But the pudding had a, pudding.cool had a whole deep dive analysis on similes. It’s called comparisons as predictable as the sunrise, an analysis of 200,000 similes from popular fiction. And it kind of struck me because as we’re talking about LLMs and AI and all this stuff and all it can do with unstructured data, which is often text data. And this was actually literally going in to a bunch of text and pulling out the similes. And it makes the point that a simile is kind of an identifiable form because it’s, you know, something as blank as whatever. And as the pudding does, it’s got visualizations. It’s got hilarious anecdotes at the very end. It’s got like just a random, this will be like a layer flipping open the book and reading a random thing. It’s got a whole bunch of like random similes, like a, a, as welcome as hemorrhoids at a rodeo or as clear as the novel by Virginia Wolf. As funny as a guided tour through Dachau. So 01:04:29.40 [Aleya Harris]: it’s kind of like, and those are real things from literature. 01:04:34.60 [Aleya Harris]: I had to get as comfortable as a trout on a marble slab as graceful as a lobster playing soccer. This is awesome. We know what I’m doing. Instead of watching anime until my kid comes home, this is what I will be doing. Thank you so much. 01:04:48.20 [Tim Wilson]: Yeah, you’re welcome. It made me feel good that it was like welcome as a pork chop in a synagogue. 01:04:54.92 [Moe Kiss]: Oh, could I, sorry, her tone as friendly as a bucket full of shit. I’m like, how did I land on 01:05:01.80 [Tim Wilson]: that one? That’s good. I brought joy and those were humans. I’m sure, I’m sure AI could be asked to come up with these like wacky things, but those were from real human written stuff. And it, yeah, I got to the end. I was like, now what’s at the end? And I’m like, well, now I can’t stop reading 01:05:20.84 [Aleya Harris]: these. So, oh my God, I’m sending this to 50 people right now. I cannot help it. 01:05:27.08 [Tim Wilson]: All right. So, that was a lot of fun. So, Aleya, thank you again for coming on the show. This has 01:05:37.08 [Aleya Harris]: been a wonderful discussion. I’ve loved, I’ve loved being here. Sorry, I had a blip because I was still reading the article that you just talked about. I’m sorry. I’m back now. I’m back and I’m 01:05:49.72 [Aleya Harris]: focused. I’m as focused as a dog chasing a bottle like I’m focused. 01:05:59.48 [Tim Wilson]: All right. If you, I’ve, at this point, I think all our listeners have dropped off to go track down that putting article. So, they’re not listening anymore. But if you are still listening, we would love to get a review or rating on whatever platform you listen on. If you would like to get a sticker to slap on your laptop or a mug or a water bottle, you can go to analyticshour.io and we’ll send you a sticker. We’d love to hear from you. Reach out to us on LinkedIn, on the Measure Slack. Reach out to Aleya on LinkedIn as well. If you want to learn more, you can always email us at contact.analyticshour.io. And for whether you are realizing right now that you have a presentation next week and there is no story, or whether you’re like, no, I know what the story is and it came to me while I was listening to this episode. No matter what you’re doing, 01:06:59.16 [Tim Wilson]: for myself, for Moe, keep analyzing. Thanks for listening. Let’s keep the conversation going 01:07:06.04 [Announcer]: with your comments, suggestions and questions on Twitter at, at analyticshour, on the web at analyticshour.io, our LinkedIn group and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst. Those smart guys wanted to fit in so they made up a term called analytics. 01:07:25.08 [Charles Barkley]: Analytics don’t work. Do the analytics say go for it no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 01:07:54.76 [Tim Wilson]: Rock flag and we don’t sell fruit. The post #302: It Was a Dark and Stormy Insight… appeared first on The Analytics Power Hour: Data and Analytics Podcast .

July 7, 20261 hr 16 min

#301: It Turns Out Analysts Are Natural AI Crafters

There’s a certain type of person who first encounters Excel and, instead of running in terror, leans in and grins. Rob Collie has spent his career—from the Excel team at Microsoft to helping birth Power BI to now running P3 Adaptive —building things for exactly those people. He calls them “crafters,” and his new book, Fair Game: Customizing AI to Your Business Is Easier Than You Think , makes the case that this same crowd (hi, it’s us) is uniquely positioned to do something genuinely remarkable with AI. Not because we’re developers, not because we’ve cracked some secret, but because we’ve always lived on the boundary between the business and the tech—and that’s precisely where the real AI work happens. The conversation covers the two “voids” crafters need to jump to go from chatting with Claude to actually building useful custom solutions, why the off-the-shelf AI tools are mostly useless for business purposes (and what to do about it), the faucets-first philosophy for semantic models, and why the developer isn’t dead—just moving to the suburbs. Also: Tim built a quiz about his marriage and let his adult children take it. That happened. This episode is brought to you by Prism from Ask-Y —your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. This episode is also brought to you by Stape , your all-in-one solution for server-side tagging. Links to Resources Mentioned in the Show Fair Game: Customizing AI to Your Business Is Easier Than You Think by Rob Collie The quiz Tim quickly co-built with Claude for his (adult) children The Future of Software Engineering Is the “Product Engineer” A Realistic Assessment of How Many 12-Year-Olds I Could Beat Up Before They Overtook Me AI may be making us think and write more alike Designing the front door Models Demystified: A Practical Guide from Linear Regression to Deep Learning Photo by Bermix Studio on Unsplash Episode Transcript 00:00:00 | Announcer: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. 00:00:13 | Tim Wilson: Hi everyone, welcome to the Analytics Power Hour. This is episode 301. And hey, here’s a fun little fact. Did you know we had the founder of X.AI on this very show? But it’s not what you think. The date was August 30, 2016. The show was episode number 044. And the topic was artificial intelligence with Dennis Mortensen. So at the time, X.AI was a company that was just trying to solve multi-party meeting scheduling via email. And we talked a lot about like highly specialized agents. And it was this wild conversation about how these simple things 00:00:56 | Tim Wilson: that we think are simple can actually be pretty complicated to hand over to the machine. 00:01:02 | Tim Wilson: But that was 2016. It was a totally different world. And it feels like that was kind of an ancient ancestor of the AI world we’re living in today. And we are going to talk a lot about AI on this episode. And I’m going to do that with my co-host. Julie, Hoyer, you’re from further. Is AI something that comes up at all in your day-to-day? 00:01:24 | Julie Hoyer: If I had a nickel for every…, I would have heard AI, right? 00:01:29 | Tim Wilson: Yep. 00:01:31 | Tim Wilson: That’s what I thought. And Val Kroll, you’re from Fin, the company formerly known as Intercom. Is AI something that your company, I mean, are they doing anything at all with AI? 00:01:41 | Val Kroll: Yeah, we’re just starting to dabble, really. Just starting to just get our toes, get our toes up. No, it’s top to bottom. Top to bottom here at Fin. 00:01:49 | Tim Wilson: The fundamental grounding of the primary offering. 00:01:53 | Val Kroll: Yeah. 00:01:53 | Tim Wilson: And I’m Tim Wilson, or maybe I’m not. And this is just an AI agent that Tim built to record this episode on his behalf. Who knows? So sure, all three of us have been exploring and learning and using AI, but none of us have dived in deeply enough to write a really useful book about it. And as far as I know, none of us were product leaders at Microsoft focused on the creation and build out of Power BI. But our guest has done both. Rob Collie is the CEO of P3 Adaptive, which does custom AI and data work for mid-market and Fortune 1000 companies. He’s a former Microsoft engineer on the Excel team and a founding engineer on what became Power BI. He’s the host of the raw data with Rob Collie podcast. And he’s the author of an upcoming book, Fair Game, Customizing AI to Your Business is Easier Than You Think. It’s not publishing until August 15th, but we got a preview. It’s amazing. And it’s available for pre-order now wherever books are sold, including fairgamebook.ai, which gives instant access to the first few chapters and a few other perks. And today, Rob is our guest. So welcome to the show, Rob. Thank you so much. I’ve enjoyed this already. All right, so we don’t usually kick off the show with a, hey, give us your professional history. But one of the reasons we really wanted to have you on specifically is that you’re not just kind of any old run-of-the-mill AI thought leader, you know, half the people who have LinkedIn accounts right now. You actually kind of came from our people. And I mentioned in the intro, but you had a formative part of your career working at Microsoft, including being part of like the birth of Power BI, which is inarguably like one of the premier, most successful BI platforms out there. So I think that actually may be a good place to start. So can you sort of talk about your role, you know, from Microsoft through kind of connect the dots for us as to how that set you on the trajectory to where you are today and like like helping organizations figure out like useful paths forward with AI? I would love to. 00:03:57 | Rob Collie: So I didn’t, like most people who love data, I didn’t know that I love data. I didn’t grow up saying I want to be a data professional, whatever. I’ve talked for years and years about how some of us just sort of carry this thing I call the data gene and we eventually discovered that we have it. Like when we collide with Excel or whatever for the first time, most people bounce off like in terror. They don’t like Excel, right? Yuck. But like it actually is about one out of 16 people who collides with something like Excel and sticks and goes, ooh, that was kind of cool, you know, or their first collision with SQL. It’s usually Excel. That’s the first sticking point, but not always. And so I discovered that I had the data gene in my course of working at Microsoft. In fact, at one point earlier in my Microsoft career, I said to someone that I would never go to work on something like a product like Excel. Never. It was like beneath me. But then I went and I ended up kind of getting re-organized into Excel and discovering, oh my God, this is totally where I needed to be. And so I got to know a few different really important audiences. I got to know the people who really kind of make the world go around, like the Excel power users, that one in 16 crowd. Back then, even really to this day, they kind of make the world go around in a very kind of almost like thankless way. I got to know those people and really, really, really develop an affinity for them in a way because I’m kind of one of them and I was building products for them. And then at one point, I also got grabbed to be in charge of the BI investments in Excel. And so that was being dragged into like the corporate IT zone, right? And I learned the old style of BI through that experience. And then later, I was recruited to join the Power BI team when it was still called Project Gemini Super Secret Project. And the contrast between how the old style of BI, most of us who are working in data or working in business intelligence, were never part of the old traditional style BI because it was so insular, it was so arcane. It was a very tiny priesthood that was involved in this stuff. And I couldn’t even learn how to do it. I was working on it and I couldn’t learn how to write the formulas in these old systems. So helping to build the Power BI, which was aimed at that original Excel crowd, those people that I like so much, the data gene crowd and giving them industrial strength tools, I never thought that was going to work. I love the project. It was exciting. It was definitely the place that I needed to work at Microsoft when they came to pitch me. But like everything else, I’ve just realized, this is probably going to suck. Or maybe it’ll take three versions to get it right. And then I was shocked at how good version one was. I mean, it was really fucking good. And I knew the world was going to change, but I also knew that consulting companies of the time were not going to want to work this way. They were not going to want to move at that pace. 00:07:24 | Rob Collie: They were not going to want to work with clients of different sizes and pick up smaller, faster projects, which is what the world really needed. And that’s what led me to start the company that I did. I wanted to go fill that niche. And guess what? I got to hire those types of people that I liked so much into these new consulting roles. They became consultants. And that’s what our company is largely staffed by, not 100%, but largely staffed by that data gene crowd that grew up in the business more so than in IT. And so it’s sort of like I got to create a home for some of my favorite people at the same time. It was really kind of hard to pass up. 00:08:05 | Tim Wilson: So that’s the first part of that journey. And then, I mean, the link you make in the book 00:08:14 | Tim Wilson: is that those are the people that you say they’re kind of crafters and why crafters specifically are uniquely suited to do stuff with AI. So maybe that’s my follow on, is can you talk sort of about like the leaders, crafters, developers, knowledge workers, and why that’s an AI thing? 00:08:42 | Rob Collie Sure. Yeah, absolutely. And first, let’s just bridge one tiny little gap, which is like when I, I didn’t set out to write a book about AI. So, you know, I’m a CEO of a data company, you know, 50 plus person data company, and I see AI coming. And it was leading to this sense of dread, which I think we can all kind of relate to, right? Like AI is really good at writing code, you know, it’s actually really good at doing a lot of the things that data professionals do for a living. I went on a research project 00:09:13 | Rob Collie: with my own sort of like journey down the rabbit hole to figure out what the future of our company should be. This was about survival. Like what, you know, I had to do this. And what I found down that rabbit hole was far different than what I expected. And far more encouraging and far more like, oh, we can do this sort of thing. And far more, oh, this is actually a data problem. Like it’s like right in our wheelhouse. And the difference of sort of like feeling like AI was like opaque and inaccessible, going from that to going, oh, that’s it. That feeling of like, oh, that’s it. Like that feeling is what compelled me to write a book. Because when I found myself in possession of that feeling, which by the way is exactly what led me to write the first Power BI books that I wrote, you know, 10 plus years ago, like me feeling like I understood something and knowing that there wasn’t anything all that necessarily special about me, like the rest of the world can do this too. Like I felt like I owed that book. And I felt the same sort of the same sort of vibes again, but like at a greater magnitude this time, because like everyone’s worried about AI. But yeah, so back to your question. 00:10:32 | Rob Collie: You know, I’ve been calling these people the data gene crowd forever. But you know, what they really are, and I think present company probably is, you know, it’s we, right? It’s not they. You know, what we really are are people who are willing to think systematically about business problems, while also remaining close to the business and understanding that, you know, there’s the technologists that kind of like want to be like locked in a closet and have like notes passed under the door of what to build, right? And that’s that’s a lot of tech, right? And it’s but it’s a very and you need those people. But that’s a very inefficient way to work for most business solutions. Like being immersed in the business, while at the same time having some tech skills, enables a completely different style of work, a completely different tempo and a completely different quality of output. But with the with with AI coming along and the ability for people like us, people who picked up Excel and smiled, right? Like, when you sit down with people like us sit down with something like Claude Code for the first time, we sort of like get over our fear of trying something new. It’s that same feeling like captivating feeling like, Oh my God. And so I needed to come up with a new name for these people. It’s no longer the data gene crowd. I needed a noun for these people in the book. So I had to come up with something and crafter is where I landed. So that leaders, crafters and developers and ratio to 580. And so for every, if you have a if it just works out that way, if you had a population of 87 people who work in an office, you know, like two of them are developers, five of them are crafters, and 80 of them are knowledge workers of various, various flavors. So like only so five relative to 80, like that’s how many people again, one to 16 are willing to engage with something like Excel. And what I’m seeing is that that same data gene crowd, plus the ability to use coding agents, AI coding agents, to build custom software. I mean, this is like going to be the new spreadsheets. You know, we application development is now within range for this crafter crowd. And we’re just in the earliest stages of the world waking up to this, like most crafters haven’t yet discovered that they can do this. But we’re at steady state, we’re going to have about the same number of people building software as we had building spreadsheets. 00:13:17 | Tim Wilson: Michael, how does your team share AI generated analyses 00:13:26 | Michael Helbling: today professionally? We say, hold on, I think I asked Claude this last week and then sort of disappear into a chat thread like a raccoon in an air duct. Oh, nice. That’s that’s elegant. That’s 00:13:36 | Tim Wilson: clearly scalable. I mean, actually, that’s horrifying. Why thank you. It’s called innovation. That’s the type of innovation that ask why is trying to solve with Prism. Your AI work shouldn’t 00:13:49 | Michael Helbling: be trapped in one person’s called innovation. I have started to realize my private chat history is not a data strategy. It’s kind of a joke drawer with confidence. Well, Prism helps preserve context 00:14:03 | Tim Wilson: and memory across users, you know, like metric definitions and source of truth tables, business 00:14:08 | Michael Helbling: rules, prior analyses. So when I teach it, a qualified lead excludes internal traffic, spam forums and that one campaign we don’t talk about, the team can build from that shared context instead of starting over entirely. Beautiful. Let’s ask Michael what he meant, more the system actually 00:14:26 | Tim Wilson: remembers. And with Prism, analyses are organized, traceable, auditable and reusable with your other 00:14:32 | Michael Helbling: data sets. I like it. So my best AI work becomes company knowledge, not a screenshot named. Is this important? Maybe.png. Exactly. I guess it’s time to stop being a data trash panda when it comes 00:14:45 | Tim Wilson: to AI. Definitely. No pandas in this context. Go to ask dash y.ai and sign up for the wait list. 00:14:52 | Michael Helbling: Use code APH to jump to the top of that list. That’s ask dash the letter y.ai code APH. Because check my cloud history is not a collaboration plan. Tim, one of the trickiest parts of measurement is user data gaps. Right. Someone gives you an email in one session, 00:15:15 | Tim Wilson: comes back later from another device or converts offline and suddenly the customer journey gets 00:15:20 | Michael Helbling: hard to connect. Exactly. Staped and richer is a state power up that stores selected data from incoming requests and uses it later to enrich future events when information is missing. 00:15:32 | Tim Wilson: So once a user has been identified, and richer can help recognize them in later events and add previously collected details, all while aligning with consent settings and handling personal data 00:15:42 | Michael Helbling: securely. Yeah, it’s especially useful for cross device journeys, offline conversions, longer buying cycles and connecting CRM outcomes back to earlier website activity. You can set up and richer to work with cookies, state store or both. And it supports uploads of historical data from systems like Shopify, Clavio or your CRM. So if you’re tracking has gaps and who’s doesn’t, Staped and richer can help create more complete consistent event data. Yeah, go to state.io to learn more about Staped and richer, improve event match quality, attribution and visibility into the full customer journey. That’s Staped.io, more complete data with consent and privacy in mind. 00:16:28 | Val Kroll: Besides like the just the usual FUD, what do you think are some of the barriers to crafters? Because like I remember where I was the first time I saw a pivot table and it was like, blew me back in my chair. I was high school, but of like, what is holding, what do you think are some of the major barriers that are holding up the crafter crew back? 00:16:52 | Rob Collie: Well, here’s one quick funny story before I answer your question. I worked on the Excel team for a full year before I knew what pivot tables were used for. And I love that. I would fake it be like, yeah, you could throw this into a chart, you could throw it into a pivot, you know, I would just say things like that without ever you really even knowing what it was. Just like, fake it. No, that’s what all the cool kids were saying. So I would say that, right? And then one day, so for you to see a pivot table in high school, that’s awesome. That is really, really cool. So the thing about Excel is that, you know, is that it was so, I mean, I think the most people are even even people who end up liking Excel are scared of it at first. You know, thinking back, I think I was, you know, and, and even when I worked at Microsoft, like every couple of years, people would move around between teams and office, you know, it’s really an open season recruiting. I tried to get people to come to work on Excel and they, they wouldn’t want to. They would want to go work on outlook and word and PowerPoint because they already understood those things. But they could sense that Excel was deeper, you know, Excel is an application development platform, you know, it’s a spreadsheet is an application, you know. And so, like, there’s something intimidating about it. I personally, and I say this in the book, I don’t, I don’t pick up new tech readily. I don’t, I’m not an enthusiastic, like, ooh, shiny, go grab it off the shelf kind of person, like I kind of have to force myself, you know. And it’s really just sort of like not even knowing where to start. And like, how do you even get it installed? Like, even just going and installing something like like cloud code, right? Like, at least it used to be harder than it is now, it’s a lot easier. 00:18:43 | Rob Collie: But and not understanding the nature of the system you’re working with, like, it just feels like magic. Like, so you’re putting so much trust in something else that’s that’s that feels opaque to you. I think there’s a sort of like a like a sense of loss of control that that is that is terrifying. And I’m seeing this a little bit with our own team, like, you know, not as much anymore, but in the early going, they’re like, we derive such confidence from having touched every last single piece of logic, you know, and like artistinally crafted it, right? Like, like, how can we delegate parts of that to another system, you know? So there’s a number of things like that. But for me, honestly, it was just kind of getting off the starting line. I just had to sit down and build something. 00:19:39 | Tim Wilson: I have a little bit of a theory, and it bridges those worlds. And it’s interesting when you say, you know, PowerPoint and Word, and Outlook and Excel. And I don’t think I’d really thought about it. If I’m doing something in PowerPoint, I’m making slides, I know I am constrained to this thing of slides and Word, same thing with Excel, I feel like the people with the data gene, the crafters, somewhere along the way, they go from it’s a table where I’m doing formulas to Oh, wait a minute, I can actually build an interactive app, you know, I can put slicers or, you know, I can put data validation and put drop downs in and I can, I can, I can kind of build a data product, which I think is one of the reasons Power BI is successful is because there’s the progression through up to Power BI. And I think the same thing happens with any BI platform is you have that somebody with the data gene saying I’m building, I’m crafting an experience. And to me, what I think might be happening, and this was purely from actually reading the book that this little light bulb went on, we are all introduced to this new world of AI through the chat interface. And there are plenty of people out there saying it’s not, it’s more than just the chat, you got to build agents, you got to do all this other stuff. But there are people saying that it feels like it’s, it’s this other thing far away. But really where it’s close is saying, okay, if you just step out of the chat and say, let me, let me use the chat to help me build a data product, let me use the chat to help me, like, create something like that’s where that, that parallel. Okay, yeah. You know, I started to, to see because it just that light bulb hadn’t gone on that, oh, I can create a deliverable for my audience, for my user. And I’ve now got a broader set of things in my fingertips, just like going from, I got to hack around and excel to build an interactive dashboard, I go to Power BI, I’ve now got a deeper, a lot of these things, I don’t have to come up with the indirect formula in a data validation dropdown to make this filter work. It’s just natively there. 00:21:52 | Rob Collie: I think you just nailed it. I think you nailed it. And I didn’t, and I’m not saying that like, that’s exactly what I was thinking. I’m saying that like, I think you just gave an answer that I didn’t have. But I think it’s the, I think it’s a really good point. So there’s a, there’s a mixture of kind of like gaps you have to, you have to hop, right? Like, you know, smooth climbing curves are one thing, smooth climbing learning curves are one thing, but having to like jump from point to point, like over like a, like a void, you know, those are the places that get us. And there’s two big voids here. One of them is that, you know, we, we sit down and we use the off the shelf AI systems. This is a recurring theme in the book is that the, the off the shelf AI systems are amazing for like personal use. They’re amazing for individual use. They know everything about like all of human history. It’s insane. I was using Claude the other day and it was telling me about the geologic construction of the ground underneath my house here in Seattle. It, without even having to search the web, it didn’t have to search the web. It knew what the geographic makeup and so we were calculating the speed of sound in the ground near my house and it didn’t 00:23:05 | Val Kroll: have to search the web. You’ve got the data gene. It was family night. 00:23:12 | Rob Collie: Okay. Look, there’s a power driver driving, driving a post into the lake near our, near our house and the sound arrives in a two syllable thump. You get a thump through the ground and then you get the clank through the air. It’s one thing, but because the speed of sound in the ground is faster, the ground wave gets here and I was measuring the delta between the, the time between the two arrive and using that to calculate the distance to how far away the pile driver is. And yes, there’s something that the whole family is aware is going on. Like dad’s calculating this is like, it’s two, it’s, it’s 200 meters plus or minus 15. But anyway, so like, like, but to think about it, like that is publicly available knowledge that’s going into this LLM. You know, but like if it’s about my business, and in the book, I show an example of this, like I just have it like write a proposal, write me a proposal for this project. It goes and tries to do it, but it knows nothing about us. It, it just gives the lowest common denominator like a proposal and it is completely worthless. It is awful. And so there’s that, there’s that gap between business use and personal use, which is like, again, it’s, it’s, it’s this void, right? Like it’s, it’s not a, you have to jump it all at once. 00:24:32 | Tim Wilson: Well, it’s the same, it’s the same void of saying, I know I’m not supposed to put private data into this if I’m using a public or I, I’m like, I’ve been terrified that I’m going to put something in this going to be a problem. But if I just ask her to do something without it, so how do I put enough stuff in? And it’s, that feels like the other connecting the dots, which again, I mean, yes, yes. So there’s that sycophantic Tim, so maybe I am just an AI agent. I’m like, your book breaks out the different pieces and how to think of them. But that feels like it’s again, it’s like, you’re using chat GPT and somebody’s like, 00:25:04 | Rob Collie: but don’t, don’t give it any of your information. And so then this is the second void is that you have to get comfortable building real, regular software, like stuff that’s written in Python or whatever, which is not what data people have traditionally done. Like we’ve worked in ETL, we’ve worked in formulas, we’ve worked in data pipelines and things of that sort. But until you can write your own custom code, you can’t build the rest of the system around the LLM to customize it. If you’re in charge of that software, then you’re also in charge of the storage and you’re in charge of like, which version of the API you call that is you can call ones that are that are actually more protected and that aren’t that they don’t train on your on your data. So you can you can candle that sort of sensitivity. But you have to go from being a non software developer to being a software developer in order to get this customized experience where you can control the flow of information and you can give context about your business at the right time to the LLM. So you have to sort of jump both of these voids at the same time to get started. And this is why like in the appendices of the book, the first 00:26:20 | Tim Wilson: thing you just did one of these the other night, I think, Tim, right? So the first thing I have 00:26:28 | Rob Collie: people do if they’ve never used cloud code or codex or whatever, like it’s just build a website on their computer. That’s just like a survey or a quiz that gives a score or whatever, just get used to the idea of building anything. No, no AI agent stuff, just get just start to get more comfortable with building free form software. And then we start to layer in and the next exercise, like, okay, now we’re going to add some LLM to it and see, you know, where it 00:26:58 | Tim Wilson: goes from there. So I’ll throw in now. This is I may regret this, but for shits and giggles, what I wound up doing was a little 15 question multiple choice quiz about my wife’s and my relationship and about us that I sent to my three 20 plus kids, 20 adult children. I was like three 20 plus. No, three 23 kids who are all adults about like, it’s like just family lower stuff, which was like a prompt. But it was fun standalone. I did wind up having to throw it on Netlify. So it was like, I would like one part of my like, Oh, I already have a Netlify account, I’ll throw it there. But it’s now, of course, gone to where now my parents have taken it. I mean, my kids do the best, but they’ve also had a few like, what you did? I mean, so, you know what, I’ll throw it into it in the show notes and anybody terribly. Yeah. Why not? How well do you know, Tim Wilson’s marriage? Yeah, I mean, it’s going to be referred, it’s going to refer to mom and dad. And that’s going to, you’re going to have to recognize that. Okay, I’m just going to have to be okay with it. Anyway, it was a great episode. But just, I mean, throw in that, that was like the little unlock was even though I’ve done stuff with co-work, even though I’ve done stuff with gyms, even though I’ve, I feel like I’ve done a lot, there was a piece by the time I got to the end of the book of saying, Oh, no, no, no, there’s a software component where what I’m developing with limitations, right? Because you also talk about where you still need developers. This is not now you’re turning the people with the data gene, the crafters into full blown, 00:28:32 | Tim Wilson: push stuff into production to hit the masses. The developers now are still needed for their 00:28:39 | Rob Collie: skills. Yeah. So, you know, we P three, we’ve hired this year, our first two full stack developers ever. And it’s kind of funny, like, in fact, I was even in Fortune magazine at one point, talking about this, because, like, big tech is laying off developers, because, because AI is so good at writing code, in fact, it’s better writing code than anything else. At the same time, though, while it’s expanding the capacity of an individual developer to do so much more, that makes their ROI pencil out much differently. And so, like, you know, in order to achieve what we’re achieving today with two developers who are all in on using AI to write code, we’re getting the results of, like, honestly, like, what are probably required a team of 10 00:29:32 | Tim Wilson: developers to do before. So, I call this sort of like the migration to the suburbs 00:29:40 | Rob Collie: for developers. Like, so, like, like, there’s been a concentration of developer talent in big tech firms, because they build huge amounts of technology and they require engineering teams of, like, 30 developers at a time. And I don’t think they’re going to require 30 developers to do 00:29:56 | Tim Wilson: the same sorts of things anymore. At the same time, though, the ROI on hiring a developer, 00:30:04 | Rob Collie: like, in a smaller company, or a department level of an enterprise or whatever, is going to be higher than it used to be. So these people are going to have other jobs they can plug into. They’re just going to be in different places. And, you know, so, like, you know, the dev team of two that we have here, we don’t need to go too deep. And just to sort of underline it, like, where we’ve built things that put us at probably about a year ahead of Microsoft 00:30:33 | Tim Wilson: in terms of offerings that Microsoft will have. It’s like, we’re able to do things for our clients 00:30:39 | Rob Collie: today with AI around Power BI and things like that, that, you know, Microsoft is absolutely 00:30:45 | Tim Wilson: building. But they’re not there yet. And so, you know, it gives us an opportunity to fill some 00:30:52 | Rob Collie: gaps there in the meantime. And, like, that is just absolutely bananas. A year ago, me, would never have guessed that that was a possibility. 00:31:02 | Tim Wilson: You would have said the tool that’s not doable, even with our DAX amazingness, we can’t do that. We’ll have to wait until Microsoft rolls that out. I mean, or you’d find an alternative way to come at it. 00:31:14 | Rob Collie: No, I’m thinking it’s more like I just hadn’t realized how valuable developers were going to be. Like, like, there’s a difference between what a crafter, like, so I think of myself, I’m 100% a crafter. I’m not a developer. And I talk about there’s a form of discipline that’s present in a developer. These are mostly personality traits as opposed to, like, intellectual IQ measurements. There’s a disciplined patience in a developer, combined with, like, an aggressive modularization instinct that isn’t really present in most crafters. It’s not present in me. And so, the things that our dev team of two have built are things that our crafter team would not have been able to build. They’re just levels of complexity. And even just knowing what to build and figuring out, like, what the roadmap should look like would have just taken us too long. The developer brain is still really useful. And so, one of the things we’re working on on our team is figuring out how to hybridize developers and crafters to give the best possible results to our clients. Because most developers, and I’m painting with the broad brush, are most developers aren’t going to want to be the customer-facing interface that does all that conversation. Like, they want to be writing code. Even if they’re capable of having those conversations, they love writing code, you know? And meetings take them away from that. 00:32:53 | Tim Wilson: But crafters thrive on interacting with the stakeholders, right? Like, that’s the world 00:32:59 | Rob Collie: that they came from. They want to be right there, you know, like, almost like pressed up against the audience that they’re helping. And so, but there are certain kinds of code, certain kinds of platforms, et cetera, that, like, you really, you’re still going to want developers to do those things, even though we’re building solutions at the crafter level. And I 00:33:19 | Tim Wilson: do talk a lot in the book about trying to paint pictures of where those boundaries are, 00:33:26 | Rob Collie: but they’re incredibly fluid, you know? Like, it’s like, you can’t paint a precise picture. 00:33:32 | Julie Hoyer: I was going to ask you to clarify, like, when we’re talking about crafters, are we, in my brain, and I feel like maybe some of this wording comes from, like, working at further and search discovery, like, we always talk to him and about, like, our titles, or maybe a little different than somehow, like, the industry talked about them. But in my head, right, crafters, the way you mentioned it, Rob, to me, I would put in, like, this engineering group that’s more like building the back end of, like, a BI tool, and, like, surfacing the data type engineering role, whereas I came up through more of, like, the classic, like, analytics role. We kind of talk about them as, like, functional analysts, very tied to, like, the business. I could, like, build a dashboard, of course, but I wasn’t doing, like, the ETLs in the background. I was doing more of the, like, business, what are you asking? What is your problem? What’s your hypothesis? Let me actually go manipulate the data, build something in R, right? Like, a model, do an analysis, and then bring it back to you. In your framing is a crafter, both of those things, because when you say, crafters need to become software developers, and, like, the way of using AI, I’m just curious, are you talking more what I’m saying are engineers, or are you including this, like, functional analyst group as well? So I think I should very much clarify that I’m using 00:34:52 | Rob Collie: a very Power BI-centric lens for this, because, you know, our company’s all been all about Power BI. You know, we’ve long said that we will use another tool if something came along that we like better. But so, but we’ve been, we’ve been a Microsoft stack company, and it’s not because of Microsoft patriotism. Like, I literally formed a company around this thing that I saw. 00:35:16 | Tim Wilson: The Table of Army is going to be coming for this. The comments are going to be… 00:35:20 | Rob Collie: Well, I am, I am ready. I am ready for that. When we talk briefly about semantic models, I’m really ready. So… And the Domo people too, but that’s only going to be like three people. So… I know. Yeah. I mean, you just defarm them. The people that I’m talking about have kind of always sat on the boundary between the business and IT, but they’ve lived mostly in business. So they, you know, like they reported up through, you know, some non-IT function. You know, they’re the shadow IT folks. And so, you know, the role you’re describing is sort of like the business analyst that does all the, interacts with the business and sort of figures out what the needs are. The people who do that, 00:36:08 | Tim Wilson: but then also go and like build the spreadsheet, you know, that addresses it, or in our world, 00:36:15 | Rob Collie: then went and discovered Power BI and started doing the ETL and started, in sort of doing the data modeling and writing the DAX. That’s the, you know, that’s the, that’s my personal like, like center of gravity for all of this. So at our company, we talk about it as like the decathlete model. So like most consulting firms, the way they’ve been constructed over the years is they have specialists. They’ve got someone who throws the javelin. They’ve got someone who runs the hundred meters. They’ve got, right. And so this team of 10 rolls in there on the engagement and you pay a tremendous amount of communication costs between those 10 people. Moest of the elapsed time of the project is actually the communication between the members of the team. It’s like 99% of the weight of the project is communication costs and not implementation. And so our whole business model from the beginning was built around this idea that, no, no, like you, you need to be the decathlete. You’re the one that talks to the customer. You’re the one that understands their needs. You’re the one that builds the ETL. You’re the one that builds the data model. You’re the one that builds the dashboards and all that community, your bandwidth within your own head is just so much faster than the bandwidth between people. And by the way, also less error prone. And so at least again, speaking through the lens of my company, the company that we built, this is the persona that’s kind of like, like at ground zero for it. And so I think more on the business analyst side than on the quote unquote engineering side. But, but in the Power BI world, a lot of those business analyst types acquired these skills that had traditionally been the purview of the back office IT types, right? Like we were invading their ETL world and invading their data modeling world, right? Like, you know, tearing down that ivory tower. 00:38:11 | Tim Wilson: I feel like it’s the media agency that would say, oh, we manually update this spreadsheet and we put our dates across in the columns and it just makes our skin crawl with the structure of it. And they manually date it and they put some hacky formulas and since some sort of garbage over the wall to the marketer. And then an analyst gets in and or somebody with the data gene gets in and says, this is error prone. It’s not really giving the marketer what they’re trying to ask. So I’m going to figure out a way to maybe it’s VBA, you know, maybe it’s Google, maybe it’s Apps Script, because we’re doing in Google Sheets. I’m going to figure out a way that we streamline the process and deliver something that is more aggregated, more useful. Oh, and I’m going to put a layer of interactivity on it. And I’m not going to be able to stop. And then suddenly I’m the person who the digital marketers come to and they say, can you make one of those spreadsheets? And like, that’s where we’ve crossed over and we’re like, wait a minute, are we are we developers? Like, no, you’re still not a developer. You’re a, but I also feel like it’s where analytics engineers arose. That is a whole field that came out of analysts who said, damn it, I can’t keep waiting for the engineering team. Every time I want to change the ETL, I got, I’m just going to build myself a little sandbox. And all of a sudden that became an entire position or a total role of 00:39:37 | Tim Wilson: where the analyst just kind of gets better. And the point I think that, that like in the 00:39:44 | Tim Wilson: light bulb that went on for me was like, oh, wow, I would never consider building a front end. I would never consider something that limits the rules that is actually software. But guess what? Those same people with a little bit more knowledge and a little bit of practice can actually extend it with a boundary of where does it cross over to a developer, but they can actually, they have this other thing that we’ve never even thought of as being in our arsenal. 00:40:13 | Val Kroll: It seems like one of the, the first steps to someone kind of seeing for these crafters have their eyes open is to get out of chat, um, to start building. And I think some of those examples you guys talked about was great, but something you touched upon earlier Robin obviously got into it deeply in the book is thinking about how do you give it the context about your business that is required in order for it to actually be the partner for you that it needs versus just like the generic off the shelf, like how would an individual contributor think about going about that in a way that’s productive for them in role? Like do you have any advice for taking that leap, which feels like another step? Yeah, I mean, the, what’s really amazing is just 00:40:55 | Rob Collie: how much smarter even the LLM seem to get when you give them something to go on. So the first example that I talk about in the book is this concept of what I call a handbook. Um, so if you, if you had a new hire, uh, and one of the metaphors I use is, um, an LLM is basically something that has a, especially the frontier new LLMs, they basically have PhDs in every human subject, every, everything that you could get a PhD in the LLM has that level of knowledge and even ability to reason to a certain extent in all of those subjects, but it shows up every day knowing nothing about your business. And if you thought of it as a person for a moment that showed up with all of that skill, but knowing nothing about your business, they would be relatively useless to you. You know, like you don’t care about the history of the Roman empire, uh, most of the time in your business, right? You don’t care about the ground speed of sound underneath Moett Lake Seattle most of the time in your business, right? Um, but you do have policies and a brand framework and workflows and things like that. And so like, if you were going to hand, um, um, a new hire, some sort of handbook and instruction manual for their job, uh, it would be written in English, like if, if you’re in the United States anyway, 00:42:17 | Tim Wilson: right? Like, you know, it’d be written in natural human language. That same sort of document can be handed to an LLM every time it wakes up. And if you, if you, again, now you’re writing custom 00:42:30 | Rob Collie: code, right? So you have, you’ve had to take control and you’re not, you’re not using the chat GPT built in interface or the off the shelf interface anymore. Like you’re sort of like taking control of it. You’re building your own software. Um, but it’s not that hard. It really isn’t. And your, your software, the first thing it does when it, when your software wakes up and someone asks it a question or something like that, let’s say you’re building your own chat bot, the first time the chat bot wakes up every time it goes and grabs that handbook and feeds the handbook to the LLM saying, here’s what you’re doing for us today. Here’s your personality, here’s the rules that you follow. And that lands in the LLM’s brain before any question from the user does. And suddenly it goes from being this off the shelf useless thing that can’t help you with your business because it doesn’t know your business to being a world beater. Um, and if you just think about that for a moment, just pushing some instructions and some context, just like some information about your business. Who are we? What do we do? Like what’s, what’s your corner of the world here? And each of these agents you build like this, they become specialists in that one role, right? You don’t, you don’t, um, I talk about like in the book, you don’t, 00:43:48 | Tim Wilson: you’re never going to build a super being that knows everything about your business because 00:43:54 | Rob Collie: everything about your business can’t fit, believe it or not, it can’t fit into the LLM’s memory. It will over, it will overload it. So you have to give it narrow lanes. Um, and you know, so like if you have like, you know, but you can imagine agents with five different roles, like I’ve got one that helps me write marketing copy. I created an editor to help me with my book. Um, and they have different instructions, right? You know, one of them is, is instructed how to write like our company. And the other one is instructed not to write anything for me. The other one’s instructed to help hold me to my standard of writing. And when it makes suggestions on fixes, don’t make suggestions that make it sound like boring business voice. Remember that I’m still trying to sound like me when you’re criticizing my work. So like, don’t, for example, don’t, I don’t want my editor to ever say to me, Hey, Rob, you start too many sentences with the word and or but, you know, that’s just how I write, you know, so shut up about that. We’re not going to do it that way, you know. So this idea of handbooks is really like the, the first place to start. But then over time, you start to give these agents, and again, you’re just giving the LLM sort of a menu, 00:45:18 | Tim Wilson: you give it the ability to pull information that it wants. Because you can’t sometimes, 00:45:24 | Rob Collie: you don’t, you can’t know in advance what information it needs. So you give the LLM some ability to go and pull information it needs. And by the way, we’ve all seen this in action already. Every time you go to an off the shelf LLM product and you ask it a question, when it switches off and does a web search, it’s doing exactly that. It’s recognizing that it needs more information from elsewhere and triggering something that’s called a tool. And the tool is just, it’s something that literally just sends a chat request to the tool, the LLM does, and then the out that the tool goes and performs the web search and returns the results back to the LLM. And you can do that with your own internal databases. You can do that with your own internal knowledge bases, which is crazy. 00:46:12 | Julie Hoyer: So for a very specific example, it immediately when I think of like AI and use of analytics, you hear about every tool putting in AI chatbot or AI agent on top of their BI tool, right? And it’s supposed to like be so great. And everyone’s got to have had experience with it. Yeah. And as most people that actually go in and use them, they don’t know anything about your business in there. And I have not found a helpful one so far. And so then of course, that gets us into the idea of like, Oh, it needs a semantic layer, it needs context. So I’m kind of like wrapping us into two possible paths here, and you can choose whichever one, but I’d love to talk about them both. One being the whole semantic layer conversation. But two, and what you were just talking about, what advice would you give an analytics practitioner, a crafter of your company has this BI tool, and it has this AI agent? Like, are you saying you can control that AI agent with like these handbooks and things, which it sounds like maybe it leans into semantic layer, or are you saying, don’t use the off the shelf thing, go make your own custom one? Or is this scenario not applicable at all? Option C. I think, I think they’re both kind of the same 00:47:29 | Rob Collie: conversation, honestly. Like, the key thing to recognize about any AI solution, and this is the thing that kind of blew my mind, is that the LLM is sort of like, almost like, it’s just something you pick off the shelf, and it’s almost like, like the, an element on the periodic table, it is exactly the same for you as it is for me, as it is for the, as it is for Satya Nadella at Microsoft, it’s the exact same LLM. And it’s really just like a component. And most of making, not most of, making these things do anything useful for us is all about just regular software, not AI, just regular software. And the regular software’s job is to handle like the user experience, just like it always does, as well as the flow of information to and from the LLM, like giving the LLM access to what it needs to know. So let’s keep in mind for a moment that most AI agents, we fast forward like two years, and we see what the future looks like. Moest AI agents that are deployed at companies aren’t going to be about analytics, you know, like, you know, like a, a chat bot that handles the first level of interaction with customer service, for instance, it doesn’t need to be performing strategic queries against your, you know, your data warehouse, right? Like it really just needs to have a, like a flow chart in it, and, and also some instructions on how to talk to people and what not to say and all that kind of stuff. And that, that sort of falls into the handbook category. So there’s a lot of tactical AI agents that aren’t going to have anything to do with what we think of as analytics. I mean, I just want to level set that. What’s kind of neat, though, is that even those tactical things that have nothing to do with analytics, they still, because they, they operate on information that needs to be very carefully controlled and fed at the right time, that the types of discipline that we’ve developed as analytics people, as data people, are that are the right mindset for making those things work properly, you know? And now that you’ve given us, the world has given us crafter types, the ability to write real software very quickly with the use of an agentic, you know, like something like cloud code, we’ve got everything at our fingertips, that we’re going to be really good at building those sorts of solutions, even though they’re not related to traditional analytics at all. And so I want to make sure that I’m not pigeonholing us data crowd into just the traditional data crowd type AI stuff. So that’s, and the handbook stuff is not, is not part of what you’re hearing when people talk about semantic models or semantic layers. Handbooks are not, not really part of that. Semantic layers are a decoder ring for your structured data. And that’s, now, now we’re back in the analytics and data crowd, you know, we’re back in our, in our, in our home neighborhood now. Yes. And so the, the, I think this isn’t even really controversial, honestly, because so we were talking about, you know, the Tableau crowd coming for me. Okay, so it doesn’t matter what platform you’re associated with, like if you’re a Tableau professional, you’re a Snowflake professional, you’re a Power BI professional. I think it’s really just like a foregone conclusion that you are going to be working with semantic models and semantic layers. And, you know, I can give you the technical reason why, but I can also just give you the market reason why. So let’s take Tableau as an example, and I love this, this just tickles me 00:51:33 | Tim Wilson: to death. So for, for men, the best thing about Power BI, from a technical standpoint, the best 00:51:41 | Rob Collie: thing about Power BI was not the reason why it got adopted. The reason Power BI won the BI wars, and has the largest market share is because it was cheapest and it was integrated with the rest of Microsoft. That’s it. No one was evaluating like, ooh, it’s so cool that you have a semantic model in this star schema stuff and all, like, like, it makes a difference in practice, but that’s 00:52:03 | Tim Wilson: not why it won. And Tableau long neglected the idea of semantic models. It’s, they were all about 00:52:11 | Rob Collie: each time you need a new dashboard, you go write a bunch of new SQL queries to make a big flat, to make a big flat wide rectangle of data that you hide behind the dashboard. That’s the philosophy over and over and over again. And the thing is, it worked, right? I mean, like the IPO, they sold the Salesforce, I mean, those people made gobs and gobs and gobs of money. So it’s like, you know, you know, I’d start arguing with them about like, well, y’all don’t have a semantic layer. And they go, well, I have more money than you, you know, but you’re right. So they eventually invented a semantic layer that sort of was an attempt to sort of add what Power BI had been doing from the beginning, but then they neglected it. A friend of mine at Tableau was in charge of that for a long time, and eventually he just retired in frustration. He just gave up. Not even, not even making this up. Okay. Well, after so many of these companies have neglected semantic layers forever, they all banded together to form this OSI thing, which I think you have has been talked about on the show before. Open semantic interchange. And, and on the Tableau website, a year ago, less than a year ago, there was an article says, the agentic future demands 00:53:25 | Tim Wilson: an open semantic layer. Okay. So everyone now, if they, every data company that wasn’t in on semantic layers is now in on semantic layers. Every single one of them has something that either 00:53:41 | Rob Collie: are sticking with the one they already had, like in Microsoft’s case, or they’re banding together in this open standard and rapidly integrating into that product and trying to get their customers to adopt it. Now, why would they be doing that? As a company builds a semantic layer, I think, 00:53:56 | Tim Wilson: and this was when we had Cindy house and on to talk about it, there was something she said that I really liked. So she was kind of in the OSI heavy camp our last episode, we had Jacob Madsen, we talked about maybe we don’t need one at all. So we’ve kind of covered, I think, 00:54:10 | Tim Wilson: my one question is like, how does the semantic layer not get held up as the monolithic 00:54:19 | Tim Wilson: boogeyman and the monster dev project of saying, yes, yes, yes, yes, we will do glorious things with AI as soon as we build the semantic layer, because we did that with data dictionaries, we’ve done that, I mean, at least in the days of power play cubes and SSS, like they were contained, but like where that’s where I get nervous about semantic layers as it becomes an excuse and people like we have to build the entirety of the story of our data. 00:54:50 | Rob Collie: You could take your concern there that you just shared and you could plug and play replace that with data warehouse, like seven or eight years ago. And that would be something that was a talk and a spiel that I would give everywhere, like the data warehouse. So we got to go build the data warehouse first. And the other thing is data warehouse is never done. And by the way, the other thing about data warehouses is that you would find that you’ve been trying to anticipate all of these future needs, and you anticipated them poorly. So you built plumbing to nowhere, places that faucets were never going to be needed. But then the first time you need a faucet, you go and you look and there’s no pipe, the place you needed the faucet. So you over engineered and under engineered at the same time. And yeah, I agree, like those sorts of monolithic projects 00:55:35 | Tim Wilson: that kind of become, they become their own thing. They become like a goal in and of themselves. 00:55:42 | Rob Collie: Those are danger. You do not want that. So one of the things we’ve been, one of our philosophies 00:55:52 | Tim Wilson: that are our company, we call it faucets first, we start with the faucet and we build backwards 00:55:57 | Rob Collie: from there. So yes, we do need responsible data plumbing. But if you’re building plumbing forward, you’re going to lose. Now, if you’re a consulting company and you’re building plumbing forward, you’re going to win because you’re going to build a lot of hours. And then when things don’t work, you’re going to build a lot more hours to fix it. But if you go faucets first, if you go faucets backwards, you can get everything that you need. So I think semantic models, and the way we approach Power BI is to sit down and go faucets first. We don’t go build a big 00:56:33 | Tim Wilson: data model before we build our first dashboard. The whole point is to get to a 00:56:40 | Rob Collie: first dashboard as quickly as possible so that the customer can go, okay, that’s not right in these three or four different ways. And then we go modify the data model to fit it. The same thing here. If your goal is to build, and okay, so let’s bridge a really important gap here. The reason why a semantic model is suddenly so important today, it was important to BI, but no one cared other than Power BI really. But everyone now cares is because you don’t have what I call the semantic shock absorber of the people that you lock in a room and make them write the queries. So each new dashboard that’s created is really representing new questions that have emerged. The business has new questions, so you can build a new dashboard. This has been inefficient really, even in the Power BI world, this is really not optimal. Okay, but it’s even less optimal than the tab low world where it requires like a week’s worth of SQL work before you get the first dot on the chart, right? If an AI agent forms a new question or a user comes to an AI chatbot with a new question, there’s no time for someone to go write a week’s worth of SQL. And you also don’t want to trust the AI to go write that SQL because it will write it slightly differently each time and it won’t know which column means what in the source data. This is why the decoder ring analogy I think is so apt. Symantec glare is just a very nerdy way to say decoder ring. We wrote down what things mean. We wrote down how to translate our company’s structured data into meaning that matters to 00:58:17 | Tim Wilson: the business. What is gross profit defined as? There’s a lot of nuance to that that’s 00:58:24 | Rob Collie: different for every single business. And so if you look at it through an AI lens and you go, okay, what does this AI agent need to do? And then you build the Symantec model that is required to power that. You’re in a much more focused means of engagement than if you’re trying to like, let’s go model the whole business, just the whole thing. And let’s just party on this thing forever and never test it and never write. No, that’s not the way, right? And I was going to say too, 00:58:56 | Tim Wilson: like I love what you’re saying and I just feel like people are finally waking up to this idea 00:59:05 | Julie Hoyer: and that like surfacing the data alone was not the valuable part. And so like why people could get the budget approved for data warehouse or like projects or the huge Symantec layer projects. I feel like it’s because and Tim, you’ve talked about this a million times, right? Like the promise of the value of like having the thing, having the data and in current conversations with clients, and I’ve seen it more like across the industry and articles, you know, there’s this whole like, well, what’s the actual return on investment for AI? And I just feel like AI has become this thing that really highlights to people that just because your data is there or you have all this data, right? Like even though AI can access it and use all of it that you’ve spent so much time and effort to create and surface, the output of AI is not inherently valuable to you. You know, like it is finally like the mirror to that flawed thinking. And so it’s interesting when you were saying like faucet first design, I do feel like everyone is now afraid to just be like, oh yeah, do another AI POC. They’re like, no, I need to know that my faucet in my business is going to work, not that you can go make plumbing in another room. Like I don’t care about that plumbing in another room. So I just, I really like that finally clicked for me in the conversation we’ve been having, but I’m curious your thoughts on like AI being scrutinized for value and this idea of like the data itself is not the value to the business. AI should be scrutinized for value, 01:00:49 | Rob Collie: you know, there’s a lot of pressure to just go do AI, right? Because everyone, everyone can sense the uncertainty of inaction, right? If we’re not doing anything about AI, we are falling behind, right? As a business. And so, but like what happens is the people who don’t know what to do about AI at all, they don’t know how to apply it, but they’re important. They’re like board members or whatever, they lean on executives and say, hey, you need to be doing AI, right? And the executive doesn’t know. So they turn to someone else and say, you know, you need to be doing AI. And then you end up in these weird, crazy things. We’ve all heard these stories and they’re real about people measuring teams usage of AI tools. And by the way, they’re always the off the shelf tools. So they’re the wrong things for most business uses, right? And making sure that people are using them, because if they’re not using them, then they can’t go back up the chain and say, hey, we are doing something about AI, right? If we’re in a really, really strange place where no one knows what to do with it, but the amount of pressure to do it is I don’t think we’ve ever been in a situation like this. Like BI wasn’t like this, right? Like no one has ever under pressure to do BI, like, unfortunately, right? They, you know, like the boards never came to their, you know, C suites and said, you should be doing BI. Well, I feel like personalization was a little bit 01:02:18 | Julie Hoyer: like that, Val. Like we heard that from a lot of clients, right? Like our goal is to do personalization. 01:02:23 | Tim Wilson: You’re like, but to what end? Like digital transformation. So like, I just think that 01:02:31 | Rob Collie: the semantic model case with an AI chat interface over the top of it is one of the easiest wins for AI adoption that’s actually going to add ROI. Like so for people who have Power BI investments, if they’ve got good semantic models, like we’re, we’re putting in AI chat interfaces. And honestly, they’re so much better than dashboards. There’s so much friendlier, there’s so much more user friendly, people actually get value out of them. And even the people who built the semantic models who know it, the ins and outs, even I use the chat interface to go and do all kinds of things for me to go research things for me that I wouldn’t do otherwise. It’s not even like saving me time. It, it, it, I form better questions and kick my buddy off to go research it for me. And again, it leans on the semantic model. It doesn’t, it doesn’t hallucinate. It gets real answers from real systems and comes back and, and, and, you know, and I can sort of, I can send my assistant back and, you know, to do more if I, if I want, you know, under all this pressure, and again, as data and analytics professionals, it is one of the easiest and most trustworthy and best places to start dipping our toes into AI is to start enabling these sorts of AI chat interfaces that actually work, not the ones that come off the shelf from the vendor. But if you build it, you build something yourself, like we’ve got this framework that we’ve built, again, we have these two developers that have been doing amazing things for us, like our clients freaking love this stuff and we love it. So I think it’s one of those, it’s not the only thing. I do not want to say this is like the future of AI, but it is a great place to dip 01:04:12 | Tim Wilson: the toe into it that does add value. And it’s right in our backyard. So I wanted to make sure we, 01:04:20 | Rob Collie: I got that, that little, you know, career advice. Yeah, it’s a great place to land the plane to kick 01:04:24 | Tim Wilson: off for show two. Yeah. All right. Well, that, yeah, there’s, there, I have so many more questions, 01:04:34 | Tim Wilson: so many more thoughts, but we also up against the clock. So this has been a fantastic discussion. And yeah, I think it’s even in this discussion for me in the book, lots of light bulbs have gone on for me. So this has been enormously useful to me, at least, and it’s all about me. But when we last thing we like to do on the show is go around the, go around the group and everyone share a blast call, something that something or maybe a couple of some things that might be of interest or amusing or entertaining or useful to our audience. And Rob, you’re our guest. So 01:05:14 | Rob Collie: do you have a last call or two? I do. I have one that’s relevant and then one that’s kind of just funny off, off topic. So the one I’ll obviously provide you the link, but an old friend of mine at Microsoft named Uli Homan wrote something on LinkedIn recently, where he talks about this new 01:05:30 | Tim Wilson: role he’s calling the product engineer. We’ve had software engineers, software developers, 01:05:35 | Rob Collie: all that kind of stuff. We’ve also had product managers who like were like the designers of it and all that kind of stuff. And he makes a very, I think, eloquent case for, I think, where we’re all going to land with this product engineer role. And so I think it’s a worthwhile 01:05:50 | Tim Wilson: read and it’s not very, it’s not very long. And I think it lands the concept pretty cleanly. 01:05:55 | Rob Collie: Then what’s the relevant one? No, you said it could be, it could be any link to anything that we’ve ever liked, right? That’s true. Right. So this is an article from 2005 on the McSweeney’s website. It’s titled, a realistic assessment of how many 12 year olds I could beat up before they overtook me. And it is such an analytical, dispassionate analysis that uses game theory and assumes optimal behavior on the part of the 12 year olds. It’s like, no, the 12 year olds aren’t going to get up in a nice line and let me take them one at a time. They’re going to form a circle, right? And they’re going to know that groin kicks are their primary weapon against me. And here’s how big an average 12 year old is and just really just breaks it down and eventually comes up with a very reasoned argument for why this certain number is probably the number 01:06:54 | Tim Wilson: that would take him. I love it. That’s so funny. Oh my God, that’s amazing. That’s really funny. Julie, what’s your last call? 01:07:05 | Julie Hoyer: Mine. Actually, I feel like this ties back to Rob, what you were talking about in your use with AI agents. A colleague recently shared a article called AI May Be Making Us Think and Write Moere Like. And it sparked my interest right away because I feel like multiple times on the show, I’ve said like, but what about like group think and like the cyclical, you know, like self-fulfilling prophecy. I’m like, if AI is then using our AI outputs to then continue modeling, I’m like, it’s all just going to converge somewhere. So anyways, this was very like fulfilling for me to see that there’s now been research that, yes, that’s actually happening. They’ve pretty much said that they’re seeing all of this really come to a point where everyone’s losing their unique voice in writing. And because of that too, what everyone is putting out is becoming more similar. So it is like this idea of the group think happening. And it’s funny because they say that AI solution, Rob, similar to what you did, is they were saying, you need to make sure you have a very clear framework to ask your AI model to enforce like strict stylistic constraints and make sure that you can extract your distinctive voice. So I thought that was funny. It’s like, AI is the answer, even though AI is the problem. And it was this whole, you know, echo chamber. 01:08:34 | Rob Collie: So it’s an interesting read. Wow. That’s not everyone’s starting to sound the same, but a lot 01:08:40 | Julie Hoyer: of people are. You can read something now and you’re like, oh, yeah, that sounds like AI. It’s 01:08:47 | Rob Collie: pretty quick. So I’m going to resist. Val, what’s your last call? 01:08:55 | Val Kroll: So there’s a AI tie-in for this one. But it is very self-serving because it comes from some of my co-workers at SIN. But my thing is, 30% of my last calls come from like UX collective, like this publishing group on Medium. And this could have very well been on there. So it’s a Substack article, all that caveat build up. And it’s called Designing the Front Door. And what I thought was really interesting is it’s like from this engineer and this designer at SIN thinking about how you create the front door to some of the agents that are not the helper agents, the chatbot, the little thing you see in the bottom right hand corner of your screen, right? But they’re calling it like the spotlight messenger. And what I think is really interesting about the article beyond the solution that they landed on is how they use data and experimentation and lots of research to figure out what is the optimal way to bring some of that to bear in a way that doesn’t harm other interactions, whatever. And it just feels like some of the research that I did when I was at further with the consent management banners four years ago and thinking about like, how do you surface this when there is no established pattern? And so it’s an interesting look at it with some fun examples. And it’s a quick read. So it’s a good one. So how about you, Tim? 01:10:17 | Tim Wilson: So I’m going to kick it old school with an ebook. It’s on a github.io site called Moedels Domestified, a practical guide from linear regression to deep learning. So if you want to step away from AI and just kind of understand generalized linear models a little bit deeper or causal modeling, I have not read the whole thing. I’ve kind of sampled it. It’s got kind of a lot of our a lot of the models are kind of ours available and Python. So you can kind of play with either one. But I feel like that maybe when I get overwhelmed by Claude and going back and forth, I it’s a good little thing to dip in to say, maybe I should just give some deeper statistical knowledge and a good old fashioned ebook. It’s a good way to do it. So that’s how Tim 01:11:11 | Tim Wilson: touches grass. That’s what you’re saying. Yeah, that’s what I’m not measuring the speed of sound 01:11:19 | Tim Wilson: waves through different materials. So you know, but you can, but you can’t. That’s the important 01:11:27 | Tim Wilson: thing. And can I just kind of throw one last little plug in here and you could edit it out if 01:11:34 | Rob Collie: you if it doesn’t it doesn’t fit. I just want to encourage people to go to fairgamebook.ai and preorder the book. I mean, I genuinely wrote this to help people writing books doesn’t make enough money to ever be worth the personal sacrifice. Like and so I was really gratified by y’all reading it in advance. And yeah, it’s it’s it’s for our crowd is for leaders. And I think it will actually help you not be so afraid. Like if you’re having some angst about AI, like my I felt like sitting down right in that book, my job was to help you feel better about it. Like I actually turn it into excitement. So I genuinely hope that helps people. 01:12:15 | Tim Wilson: Well, and I will say so you you took it from me because because I was definitely going to plug the book again. So I will I will further endorse that because I did find it to be extremely useful. We had a proof of it. So it was like a super early version. So I mean, it was a proof so we we could not we could not copy anything from it. We could not highlight anything in it. So like the the the notes had to be like old school retyping brilliant things. 01:12:43 | Rob Collie: And you know, you sit down you sit down with cloud code. And it’s wide open. You just 01:12:49 | Tim Wilson: you can do it. There’s no protection anymore. Okay, I’m not going to say maybe there wasn’t 01:12:56 | Tim Wilson: a way to throw the whole thing at AI and have it read it. Even when I’m making those clouds, 01:13:01 | Rob Collie: tell me, hey, you know, Rob, this isn’t going to actually protect you, you know. 01:13:06 | Tim Wilson: All right. Well, Rob, thank you so much for coming on. On the one hand, you’re like promoting a book, but the book is itself really useful. And this discussion, I think was extremely useful. So thank you for taking the time to come on like this. And really, I think y’all I mean, 01:13:22 | Rob Collie: again, I hadn’t met y’all before now. I really think this is a I love your format. I love your approach. I think you’re doing great work here. So I really appreciate your the invitation. 01:13:33 | Tim Wilson: Well, awesome. So what you could do and any of our listeners could do or you could have an agent do is go leave us a review and a rating. Look at that. Look at that segue. Oh, I’m on it. Make Michael proud. How many reviews do you want? So if you listen on a platform that allows leaving reviews, listeners, we would love to get reviews and ratings that helps us out. If anybody, if you’d like a sticker for the show, you can go to analyticshour.io and fill in the little form there. We’ll send you stickers. We also love to hear from our listeners so they can you can you can reach out to us on LinkedIn on the measure slack. You can just email us at contact at analyticshour.io. But regardless of how many faucets you’ve turned on, if the faucets first, if the plumbing’s first, if you’re deep, deep into cloud code already, or if you’re now just inspired to get into cloud code, no matter what you do for myself, for Julie, for Val, keep analyzing. 01:14:45 | Announcer: Thanks for listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at at analytics hour on the web at analyticshour.io, our LinkedIn group and the measure chat slack group. Music for the podcast by Josh Grohurst. Those smart guys wanted to fit in so they made up a term called analytics. Analytics don’t work. 01:15:10 | Tim Wilson: Do the analytics say go for it no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning 01:15:21 | Julie Hoyer: in competition. It’s all semantics. Yeah, it is indeed. It’s a challenge. 01:15:28 | Rob Collie: And I saw your, I saw what you wrote in the doc in response and I’m like, oh, yeah, so that’s 01:15:32 | Tim Wilson: okay, I’m not gonna ruin it. I mean, I will try. I mean, it’s not like we can’t like 01:15:39 | Tim Wilson: touch on it at all. I just think there’s like so much other like gold for us to hit on. As the moderator, I will say, you know what, I’ll allow it, but you’re going to be on a shot clock. Okay, I got it. I mean, we’re going to, we’re going to barely scratch the surface of everything I think we would want to talk about. And that’s just a sign of a good show. So unless Michael Helbling is running the show, in which case it’s because he has lost control 01:16:08 | Tim Wilson: into the terrible job. And Tony, feel free to put that in the outtake. We always bully him 01:16:14 | Julie Hoyer: into one more question, you know? 01:16:29 | Tim Wilson: Rock flag and data genes to crafters. The post #301: It Turns Out Analysts Are Natural AI Crafters appeared first on The Analytics Power Hour: Data and Analytics Podcast .

Is this your show?

Claim this listing to keep it up to date, reach guests who want to pitch you, and manage bookings with Guestify.

Claim this listing

More Business podcasts