#308: AI Changes Everything! Analytical Fundamentals Still Apply!
How do you know whether AI is actually making your organization better? Counting prompts, sessions, and token spend is easy. Figuring out whether any of that activity is driving business outcomes is…harder. Daniela Matinho , Marketing Science Senior Strategist at LinkedIn, wrote a Medium post on that very question, and it helped shape this discussion about what AI is (and isn’t) changing about the work of the analyst. As Daniela pointed out, if you tell a client you did something in two days instead of a week because of AI, the client doesn’t care. They care about the result! From the smart-but-junior agent that needs constant supervision, to the analyst’s shift toward being more of a storyteller, to the unglamorous-but-essential work of data quality and governance, we covered a lot of ground. Being faster is great. Staying influential is better. This episode is brought to you by Stape , your all-in-one solution for server-side tagging. This episode is also brought to you by Prism from Ask-Y —your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show (Podcast) Still Counting (Article) Nearly every S&P 500 company talks about AI, but very few are securing it by Cassie Kozyrkov (Website) The Archive of Incorrect AI Predictions Photo by Ricardo Gomez Angel on Unsplash Episode Transcript 00:00:00.00 | Announcer: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. 00:00:16.00 | Michael Helbling: Hi, everyone. Welcome. It’s the Analytics Power Hour, and this is episode 308. I was having a conversation with a coworker the other day, and they mentioned that with everything happening in AI, sometimes it feels like the skills and experience they had no longer applied. AI is definitely disruptive, but do we have to abandon the fundamental analytics skills we’ve got just because AI can do a lot of the work? And as we all adopt AI in our businesses, how should leaders think about measuring its impact? It feels challenging. Maybe we’ve got some skills that could help along the way. All right, well, let’s talk about it. First, let me introduce my co-host, Julie Hoyer. How are you doing? 00:00:58.60 | Julie Hoyer: I am doing great. I’m very excited to chat about this today. 00:01:02.38 | Michael Helbling: Me too. And Tim Wilson, I know you’re chomping at the bit for this convo. 00:01:07.71 | Tim Wilson: I just, I’m sending an agent in my place, so this is the last I’ll be here as a human. 00:01:11.90 | Michael Helbling: That’s right. mused AI, the latest and greatest. And I’m Michael Helbling, and I’m so excited. Our guest today is Daniela Matinho. She is a marketing science senior strategist at LinkedIn. Prior to that, she’s held marketing and data science roles at companies. Like Snap, Meta, and PepsiCo. And today she is our guest. Welcome to the show, Daniela. 00:01:33.21 | Daniela Matinho: Thank you. What a great intro. 00:01:36.19 | Michael Helbling: Ah, well. AI helped. No, it did not. I wrote that myself. AI does not touch the intro. We tried writing with AI and it didn’t sound right. Okay, Daniela, to get this conversation started, and thanks for, we’re excited to have you. Maybe just talk a little bit about what you’re seeing. What are you seeing in terms of how AI is affecting analytics roles and in the orgs that you’re a part of? LinkedIn especially or other places. Where are you seeing it have impact? Where are you seeing it impact people’s roles and changing things? 00:02:10.69 | Daniela Matinho: Yeah, happy to talk about overall what I’m seeing in the different industries and organizations. Overall, as Tim was saying, speaking a lot to agents. I feel like this is something that we weren’t doing before. We were doing it all a few years ago. Starting to do any sort of work, asking AI, what do you think about this? Or give me some prompts has completely changed. When it comes to the more analytical, technical part of the things, a lot of the coding that I was doing in the past would be from scratch. These days, we are starting with agents or anything that has sort of built in. And we are coming in. And enhancing it or making it a little bit faster. So overall, a lot of efficiency, a lot of expectations on delivering more faster. That’s what I’m seeing as I talk to colleagues, friends. It’s a lot about doing more in less time. Just because we have this army of agents that are helping us deliver. And delivering a lot of other things that we couldn’t do in a short period of time before. 00:03:31.65 | Tim Wilson: So can I ask about the more? Because I was with my oldest son who’s a software engineer recently. And he’s developing. So more for a software engineer is he’s going through tickets and tasks. And he’s evolved the way he works. And he is prompting and having code written. And then goes through a code review. And code gets pushed out. So it’s like more features and bug fixes. When it comes to analytics, something that, and I also literally today experienced this. The more being produced by analysts, and I’ve seen this happen in other places. It’s like, oh, I can just generate a 29-page report. I can just generate more. Or we were at a conference earlier this year. And somebody was talking about. Look, I’ve now managed to. It used to be so ponderous for me to produce this weekly report. But now I can do it. I can do 10 weekly reports. And they’re 10 times as long. And I mean, there’s the macro view of people saying. I’m writing stuff that’s longer because they’re using AI. And that feels like more of the analog to the analyst challenge. It’s like, well, it feels better because it’s more output. And it’s more thorough. And I explored more avenues. But somebody’s got to receive that and process it. So I’m curious if you’ve got thoughts on that. 00:05:05.62 | Daniela Matinho: Yes, I have a lot of thoughts. I have this theory that no one is reading anything anymore. Because we are delivering to your point, Tim, these huge reports that are so thorough. But no one has the time to actually go through it. So when we talk about delivering more. And to your point on the analytics and the analyst role. Is really to cutting the chase. Meaning to cut everything that is not needed. Working a lot with sales in my past 10 years of roles. Sales need something that is very direct. A 10-page report is never going to make it through. So my role in any of the organizations as I use AI more and more. Is leverage it to produce it overall. And then I come in and really understand what are the things. The 3, 4, 5 things. The 3, 4, 5 numbers. That are going to be used for punchy for sales for clients. A very interesting experience that I had just recently. I was working on a hackathon with some ladies. And we were basically using Claude to do a lot of the coding. We go through the coding review sessions. And we get to the point. What is happening? Like he did all this code. But it doesn’t make any sense. We can build a dashboard in one hour. But to your point team. We are looking at numbers. Well these assumptions they don’t make sense. So babysitting the code that is being produced. The numbers that are being extrapolated by. From the data that we use in the initial stages. Is a very critical and important point. That’s what I’m coming to realize. 00:06:57.62 | Julie Hoyer: Going back a little bit to maybe how this is changing people’s roles. We’ve talked about this a little bit previously. But the idea of instead of staring at the blank screen. And putting in the assumptions in your code. You know that you guys felt was right. Having to like babysit the AI as you call it. And the coding. And looking at code. And then having to like re-engineer it. And decipher it. And then find the logic to like edit it. And shift it. We’ve talked before how that feels like very different like centers of your brain. Right? Different skills. It feels different when you’re doing the work. Do you feel like that aspect of utilizing AI has really shifted people’s roles? Or it’s kind of just a byproduct and like an obstacle? Oh yeah. 00:07:44.62 | Daniela Matinho: A hundred percent. I find myself way more of a storyteller today. Than I was before. Because the hard skills of coding. And I went through interview process not too long ago. And I did maybe a third of technical coding challenges than I used to do when I was interviewing before. And a lot of the focus is being on case studying. Tell me the story behind the data. Tell me what’s happening. Tell me how is this impacting the business. So a lot of analysts that perhaps in the past would do very well because they were strong in SQL. They were strong in Python. These days they are finding this situation that oh now I have to kind of tell the numbers to people. And I feel like it’s shifting a lot towards facing other people and telling the story. Things that agents are still not doing. But maybe in the future they will be doing. 00:08:50.62 | Tim Wilson: There was something else you said. I don’t want to put words in your mouth. So I want to, I’m going to put words in your mouth and see whether it was accurate. Because you started to say that there’s more, I think you said something along the lines of there’s more like facing the prompt and like doing ideation upfront with AI. Did I hear that right? I don’t know if I was wanting to hear you say like, no there’s like upfront thought that AI can help. That AI can help you with before you dive into having the agents go and crank on data. 00:09:23.62 | Daniela Matinho: Yeah. So I usually like to work in, I would say two ways. And yes, to your point Tim. Sometimes in the past I would be working on a project and had a blank sheet of paper and so no ideas. Talking about content and talking about code. These days I can prompt AI to start. Sort of the wheel and start things running. So that’s one way of working. One thing that I’m finding more effective this ways is being very concrete in the one, two, three ideas that I give to AI before I start on a project. And I say this after a year and a half sort of using AI constantly, I’m seeing that there is so much hallucination. There is so much content that it can produce. For me. That right now I’m like, well, I have to be very precise in what I need. So I don’t get the 10 page report instead of a one page report. 00:10:24.62 | Tim Wilson: But I guess is there a flip side of that that is just using the AI as a, and I feel like this seems to be coming up in more and more conversations of like, well, this is like a non-judgmental, mildly sycophantic, somewhat knowledgeable, sound AI. So I think it’s really important to have a knowledgeable sounding board to just be asking it like, well, could it be this or could it be that? And guiding it to say, but give me the reasons this is a bad idea. Like I found myself doing that where it’s like, oh, this sounds great. And I’m like, okay, tell me all the reasons this is terrible. And in the specific case, it went through and said, well, these are where the holes could be blown in it. And then it ended by saying, but you made this. But you made me say something negative about your thought process. I don’t really buy into those. I was like, okay, well, that seemed like a useful, it sort of gave me the, to consider that added perspective. Because like human nature is to get caught on one path and saying, okay, I’ve got something that’s not going to get annoyed with me, you know, continually asking it to sort of force me off that path to broaden my thinking. Michael, I believe November 10th may be a significant holiday. 00:11:43.62 | Michael Helbling: I’m afraid to ask. 00:11:47.62 | Tim Wilson: It’s Stape Relay. Basically GTM nerd Christmas. 00:11:52.06 | Michael Helbling: That might not be the official description, but Stape’s first virtual event does bring together technical marketers, analysts, and server-side tracking practitioners for 18 live sessions. 00:12:03.08 | Tim Wilson: Including previous podcast guests like Simo Ahava and Juliana Jackson, which is already a pretty convincing argument. 00:12:09.67 | Michael Helbling: Plus speakers from Google, Jellyfish, and others. And you can switch between two tracks covering hands-on product implementation and actionable strategy. 00:12:18.53 | Tim Wilson: Stape and Simo Ahava will be revealing their roadmap for a new generation of Google Tag Manager templates, developed with community input. 00:12:26.51 | Michael Helbling: And the whole thing is online and completely free. 00:12:30.41 | Tim Wilson: So clear your calendar for November 10th. 00:12:33.27 | Michael Helbling: Click the link in the show description and attend Stape Relay for free on November 10th. 00:12:39.63 | Tim Wilson: Michael, everyone says AI needs more context. 00:12:44.83 | Michael Helbling: Agreed. And what I do is I just upload the company handbook. Six strategy decks, 14 call transcripts, a glossary, and Carol from finance. 00:12:55.95 | Tim Wilson: Please don’t upload Carol. 00:12:58.24 | Michael Helbling: But she knows things, Tim. 00:13:01.62 | Tim Wilson: Well, Prism’s context notes give you a slightly better approach. You can feed the agent business knowledge, playbooks, transcripts, glossaries, PDFs, whatever helps it understand how your organization actually works. 00:13:14.24 | Michael Helbling: Okay. And not to be disagreeable, but six months later, half of that is wrong. 00:13:18.91 | Tim Wilson: Disagreeable is kind of your thing, but that’s the good part. In Prism, context notes can refresh instantly or on a schedule. 00:13:26.63 | Michael Helbling: Oh, so the AI doesn’t keep confidently applying last year’s rules like a middle manager who missed the reorg? 00:13:33.05 | Tim Wilson: Exactly. You can even give Prism instructions for how each note should be updated. 00:13:37.67 | Michael Helbling: Oh, that’s big because stale context is just bad data wearing a tiny knowledge management hat. 00:13:44.53 | Tim Wilson: A knowledge management hat that doesn’t fit. And since that context is managed centrally, your users get better answers without everyone rebuilding the same background knowledge in their own AI chats. 00:13:55.53 | Michael Helbling: Yeah, less copy paste, less tribal lore, fewer conversations that begin. Okay, first you need to understand something weird about our business. 00:14:04.53 | Tim Wilson: Click the link in the show description and sign up for the Ask-Y waitlist. Use code APH and they’ll move you right to the top of that list. 00:14:12.53 | Michael Helbling: Keep the context fresh and Carol can stay in finance. 00:14:18.53 | Daniela Matinho: Yeah, that’s a great point. And also the way I use it as I produce something for content, for code, for work. A lot of the perspective I want at the end of producing something that I’m confident about is what are some of the holes? What are some of the things that I might get asked about so I can prepare upfront for it? So that’s one thing that I have been trying to use it more so I can feel stronger as I present. And then the other thing is that I have a lot of confidence in the content as I deliver because I have thought through it with AI. 00:14:54.53 | Michael Helbling: Yeah, I’ve noticed sometimes in working with analysts on things, you have sort of a disconnect of between the person and what the AI produced. And I’m really adamant with our team on you have to be out in front, not the AI out in front. Because if someone is in a meeting asking a question, they’re not asking the AI, they’re asking you. And you need to be able to answer. You can’t just be like, okay, let me plug your question into the AI and then find out what the answer is. And I feel like people are getting that a little backwards right now sometimes because they’re getting or doing so much of their work in AI. They’re kind of letting the detail slip past them a little and then they don’t know the details of what they’re talking about. So it’s a tricky thing to balance because like AI is helping us be efficient. It’s writing a ton of SQL and running queries. And generating reports. But then if you’re not looking at it yourself and being really careful, then you’re sort of at the cart and the horse get flipped around, I think sometimes. 00:15:58.53 | Daniela Matinho: Oh, yeah, 100%. I mean, all the, I have this theory that a PhD in philosophy these days is going to be so valuable. Because that critical thinking that AI is sort of in a way taking away from us. Because we are asking the question. We are asking AI to think for us rather than to Tim’s point. Okay, tell me what I didn’t think about. So big believer that that critical thinking piece is somewhat getting lost in some roles, in some conversations. I see AI and I was actually having a conversation earlier for another hackathon. See AI a lot on the automation. I see AI as a tool for tasking sort of situations that we encounter. Book me a meeting for days or purchase this for me that I purchase on a recurrent basis. That’s where I see that there is a lot of potential. Building agents that can really do recurrent tasks for us. I see it with a lot of potential not only on the workforce but also for our personal lives. 00:17:14.53 | Julie Hoyer: Yeah. So you were mentioning critical thinking being a skill that it’s kind of taking from us and people probably need to be a lot more careful about keeping that as something they’re doing ahead of the prompting like you were saying. Are there other big like buckets or big skills that you would say are fundamental to analytics that you wouldn’t like sub in AI to do for you yet? 00:17:43.53 | Daniela Matinho: Understanding the business. starting as I work closely with clients, with advertisers, focusing on what AI cannot guess. Because, again, we can all at this point write SQL, Python, R with AI, but understanding the business, what is the client journey that each company is optimizing for is something that perhaps a nuance that AI is not going to be able to give us. Stakeholder management as well is something that, yeah, AI can write the emails for us, but it’s the type of skill set that I’m finding more and more interesting. The fact that it’s just so important, even more important than it was before. 00:18:38.88 | Tim Wilson: Does a deep understanding of… Even though we can use AI to write the SQL or the Python and R, is there a part of it? I worry about the risk that just like if you just turn somebody loose with a traditional BI platform five years ago, they could just start willy-nilly dragging and dropping stuff and interpreting things, but they didn’t really understand how it works. How it leads related to contacts, related to opportunities. And it feels like, I think there are a lot of people trying to solve for this, but it’s the same, oh, I generally understand the data, but do I really, really understand the data? Do I understand the ramifications of how these systems talk to each other? That AI is not going to magically generate a joinable link between two data sets that, like, don’t have that? Like, is there still a need for that? Because that goes with working with the stakeholders to say, how are we going to problem solve? I need to understand the data as well. 00:19:55.31 | Daniela Matinho: A hundred percent. And I see it a lot on the fancy, shiny outputs that we can get from ChatGPT, Claude, all the different platforms. And to your point, when the underlining data is not… in good quality, when the governance, the integrity, just the quality is not passing the minimums, then the output we’re going to get is not going to be great. Yes, we have a good looking dashboard that is not really usable because the data that is displaying is incorrect, or to your point, KPIs, they don’t quite make sense because perhaps certain tables are… are not displaying the data that they were supposed to. Naming conventions were not implemented upfront. That dirty work, that’s the dirty work that no one wants to do, basically. Though if we want to continue building AI workflows, it’s the type of work that needs to be done before. 00:21:05.22 | Tim Wilson: Well, I guess, like, Michael, you’d… Michael had played around with some Claude skills around different types of tasks. Like, part of that… Where I’m still trying to… Struggling is like… Struggling with R, Python, or SQL, and really having to kind of grapple with syntax at the same time I’m trying to incrementally build up the data that I want to look at, that’s… That can be annoying, but it also sort of forces me to kind of look at the data and understand the data and, you know, periodically do some checks to say, do these aggregations make sense? Which, some of that feels like it can be, like, forced steps by the AI to say, before you do any of this, you need to do these five steps of EDA and raise a big flag if there’s something that looks off, you know, in that. But I can’t figure out, like, it feels like you’re continuing to, like, chase this bouncing ball that… Is… Preventing us from the critical thinking or understanding. I don’t know. I’m not articulating this well at all. 00:22:20.50 | Daniela Matinho: I don’t know if it has happened to you all using… I was using Claude a few weeks ago for a personal project. And we build a whole pipeline, we model, and then the output was terrible because I wasn’t babysitting it at all. And I mentioned to Claude, okay, this is wrong, this doesn’t make any sense. and he gave me an attitude like, oh, but I built this for the past hour and you didn’t tell me anything. So that was on you. I’m seeing it more and more. Oh, but you did not see this either. So that’s why I built the wrong thing. Oh man, it’s got too much training from developers. Like, what do you mean? You didn’t give me that requirement. That’s on you, the human. 00:23:07.46 | Julie Hoyer: It was making me think, Tim, that exact piece of like, I feel like analysts classically have been able to go talk to the business, like you were saying, Daniela, and understand like, what are they really trying to ask? And then it’s looking at the data that exists and knowing the nuance of what assumptions are truly on each of these, you know, values, data points, whatever, because of how it’s collected. And there’s this idea of what’s missing in the data that there’s no way AI could pick that up, if they don’t understand that, you know, real classic, like, go to your website and QA it, and we’re not getting data consistently firing on a certain user activity, right? That is like critical to answering this business question. I don’t know if AI would easily pick that up. I’m sure there’s agents and things to do it. But like, more broadly, just that concept of like, what is missing between, like you were saying, business needs and all the technical data setup to actually get it to an answer? Like, that is a big jump that I feel like we’ve talked about for years and years and years about the fallacy of like, just collect more data, right, Tim? Like, it’s another flavor of that. I feel like AI has made people kind of forget that again. And now it’s coming more top of mind for people as they’re like, oh, it’s hallucinating. Oh, it can’t do this. It can’t do that. And it’s like, oh, well, because your data wasn’t ready. Like you were saying, it was the table stakes still weren’t taken care of. But I do feel like for analysts, it’s the ongoing, of being able to see what can AI not see or know in the data. And like that, that jump, I feel like is still super critical. Is that the discipline of like, 00:24:52.66 | Tim Wilson: whether it’s a context layer, or whether it’s making sure it’s in memory of the analyst saying every time one of these hiccups or gotchas or don’t do this, the same thing where I got the 29 page dump that was not a narrative, there was a separate nine page like decoder document that talked about, oh, and you can’t do this. And so, I feel like that’s a big jump. I can’t do this and don’t compare this and don’t use this and don’t use that. Now, the funny thing, and I thought, okay, well, that would be a great thing to like, load in. The comical thing is the 29 page document violated those things like multiple times without an explanation. I don’t think the person who was doing it was necessarily super cutting edge with how they did it. But like, that feels like there’s a part that it’s like the stuff that we were used to having in our And I feel like there’s a lot of discussion around this of also how do you share that data knowledge across the company that, you know, Joe in sales, like really knows the nuances and the ends of out of this system. How do you not only do an initial picking of their brain and capture it in a way that’s usable, but you’ve got to then maintain it and the maintenance of that. And then you got to make sure that the AI is always, Always using it. Referencing it. 00:26:13.51 | Julie Hoyer: And then how do you catch like, I don’t know, I feel like people naively know AI hallucinates, but they probably rarely catch it. And then it’s like, how do you make sure you catch it if you have what you’re saying to him like that nine page document, but then you’re producing things off of it all the time. Like, what are the statistics? How often is a hallucination slipping in that they weren’t holding that true? I mean, it’s hard. Like, how would you put guardrails? Around that and catch that? 00:26:42.61 | Tim Wilson: We’re doomed. Give up. 00:26:44.53 | Daniela Matinho: Yeah, that’s, that’s interesting. That’s what I’m seeing with the expectations of delivering things a little bit faster, because we have AI’s to support agents to support we have AI that can help us shorten the process to deliver anything that is is needed. But what I’m noticing is that there are maybe less checks happening. Is AI indeed taking this in consideration? Is this updating on a recurrent basis as expected? These are a few things that I’m starting to notice. How much can we rely on, well, the agent is going to do it for me versus, well, maybe there was a break and the agent is actually not doing it for me. 00:27:36.31 | Tim Wilson: I mean, to be fair, humans who are doing stuff, we would make mistakes. Like we’d miss them. So we weren’t hallucinating. We were just screwing up. Like what’s the distinction between? But we also as analysts knew that we really needed to own it. And we knew that one mistake was as costly as 25 successfully delivered things. Whereas hallucinations, I feel like we feel less accountability for them. 00:28:05.35 | Michael Helbling: What it feels like to me when I’m working with AI is I’m managing a very smart but junior analyst. So my job is to give them as much context. At the beginning, like here’s what we’re trying to do. Here’s what it should look like. Here’s what good looks like at the other end. Here’s how we’re going to order this process. Here’s how we’re going to evaluate. So you go through this process, but you’re setting it all up as a picture ahead of time so it has a framework. And then it’s running through the different steps. And then I come back through and say, okay, now here’s data that I did. And we’re going to compare results. And we’re going to look at it. And we’re going to see whether or not you got it right. And then it’s like, oh, I did. I made a mistake here. I’ll fix that. And so you have to go through. So this is the part where, Daniela, I feel like I’m not sure if I’m saving time sometimes using Claude. Because I’m having to go through sort of these iterative rounds of sort of like checking the quality of the data, those kinds of things. And so like managing the initial and ongoing context is one thing, not just for you and the agent, but also you and the rest of the team. And then making sure that the process is working correctly through the end of it. So the output is actually of the quality you want. But you have to have like good quality. Because I think you’re right, Julie, like stuff slips through and there’s little mistakes all the time. Like the agents, the state-of-the-art models are quite good now. They’re very, very good. I love using them. They still constantly make little errors. Over and over and over again. Like they write SQL that doesn’t quite work in this environment. They do little things like this. And a lot of times they catch their own error. And they’re like, oh, I made a little mistake. I’m going to rewrite that. But it’s still costing you time. I mean, you know, it’s fine. Like I set Claude on a task and then I go scroll LinkedIn or something for him. No, I’m just kidding. 00:30:07.15 | Tim Wilson: Look at that. I do other managerial tasks, quote unquote. 00:30:13.93 | Michael Helbling: You know, or you just fire up another session. And then, you know, work in parallels. You know, we’ve all got that. But that’s what I feel like. I feel like I’m managing a junior sort of inexperienced analyst most of the time. 00:30:27.05 | Daniela Matinho: Yeah, Michael, I actually don’t have the data on this. From what I’m getting, I have a lot of students reaching out to me from my university and telling me they are graduating and it’s really hard for them to find jobs. And again, I don’t have the data here. But my understanding of what I’m seeing. Part of in the industry is a lot of the initial tasks that were done in the past by very junior analysts are now being done in a way with AI. So that’s why it’s a little bit harder for juniors to come in to the workforce and start adding value because we are training agents to do that lay of the land initial work. That is not quite working, as you were saying, Michael. But a lot of the initial repetitive tasks. That we would have before an intern doing. It seems like it’s being taken care of by AI. 00:31:26.07 | Julie Hoyer: This is interesting. And I don’t know how this is going to sound. Maybe it sounds a little silly. But with that being said, Michael, when you were giving that example, like, oh, I’ll send my agent off and it’ll keep trying and reworking. It was making me think of the time you saved, like letting an agent go do something. How much money might it cost you like token spend, right? Like to let the AI keep spinning and making errors and spinning and going. And then, Danielle, you saying that I’m like, probably. I mean, but I don’t know. Like, I would love to know this. Would you end up paying more for an AI agent to spin on something or get more senior resources to train the AI to do the intern thing or just pay the intern to come learn the skills and do it? Like, I do wonder what how that would shake. Yeah. I would love to know what that would shake out because you start to hear more and more people be like, oh, didn’t realize my agent was off trying to, like, poke a hole into another system. And I got a $30,000 bill. Like, I’ve heard more and more of those examples. 00:32:27.33 | Tim Wilson: Just a note for the listener, like the bubble, the AI bubble may have burst in between when we recorded and when this happened. And this is going to sound very dated. It’s like, yes, the model is broken. 00:32:38.83 | Daniela Matinho: Yeah, that’s definitely something that I’m seeing a bit more on the token usage and limitations. That I wasn’t seeing before. Or maybe less people were playing with it. But now it’s a lot. Oh, yes, we want everyone to use AI. And that’s being reinforced everywhere. But also, OK, this is actually costing some money. And I say it as in a personal level, I still cannot fully understand how much it takes me to do certain tasks, to how much, how many tokens is costing me to use. To do certain tasks. So it’s something that I’m growing into it and understanding, oh, there is an opportunity here. And certain things, to you all points, I mean, I’d rather just do them or pay maybe a junior analyst because it’s not worth it in the amount of tokens that is costing to just kind of go over and over and reiterate something that if you explain to someone, it’s maybe easier to do it and cheaper. 00:33:43.35 | Tim Wilson: There was something else, Michael, you said that I’m curious. It’s everyone’s thoughts. And as it happens today, I was doing a podcast test that happens every two weeks. And it’s a fairly involved one. And it’s all in core. And it runs all day. Because every time it runs, and it is rote and repetitive. But it screws something up. Sometimes it catches it. Sometimes it’ll just get hung. Sometimes it’ll do something that has been working fine. And so every time I go through it, I’m like, keep track of all the speed bumps. And let’s update the task. So that we don’t run into those again. So that’s one part of it. That’s a springboard of even a repetitive task. Pretty tedious. But it takes eight hours of duration for me, as opposed to me just doing it in a focused, tedious one hour. But it also drops in all this task switching. I have pushed it to say, what do we need to do to not have me have to click you to give you permissions for different things? Can we figure all that out up front? And then there are things where I do need to be involved. But I don’t know when that’s going to happen. So whenever the little ding goes off, I turn and I’m like, well, now’s where I need my human interaction. And I think about people talking about spitting off all these agents. And it feels chaotic. Like, oh, I’m doing five analyses simultaneously. I’m like, are you monitoring them? No, you’re not. Well, yeah. So, I mean, the tab. They’re being able to do. More. But it’s like, it feels like unless you have a lot of discipline to say, I’m going to shut off all the notifications. You’re just constantly getting bombarded. It’s like the next version of email or Slack or whatever. It’s now that junior analyst who’s sitting two cubicles over saying, hey, what do you think of this? Hey, what about this? That’s so good. 00:35:39.91 | Julie Hoyer: The head just pops up over the wall. Hey. 00:35:41.85 | Tim Wilson: Yeah. Hey. 00:35:42.55 | Julie Hoyer: The hand. 00:35:43.79 | Tim Wilson: And you can be rude to it and say, stop. Stop it. And it’ll. At one point, it told me it was like, I’m going to wait for like 30 minutes and then I’ll check back in on that. And like two hours later, I’m like, son of a bitch. Like, you didn’t. I was like, I thought you were going to wait. It’s like, yeah, I shouldn’t have told you that I could do that when I can’t. I’m like. It writes that stuff down, Tim, and keeps a record. So, I don’t know. Is there a piece of that? That’s another concern of like, are we even doing good work? Or are we at a point where we have this thing that’s interrupting? This thing is constantly. Anyone? 00:36:18.94 | Michael Helbling: Yeah. Well, Tim, I’m assuming like the things you’re doing are requiring it to do computer use. Like it has to go and navigate to specific websites and click on things and stuff. 00:36:30.05 | Tim Wilson: Yeah. I mean, some of it is that stuff. And I’ve said, can we front load and get it? 00:36:34.17 | Michael Helbling: Because that’s always going to take forever. Like, is this just AI is not good at that yet for a lot of stuff? 00:36:39.11 | Tim Wilson: If there was APIs or CLIs. I know a lot of times it’s just, it’s me giving it permissions. But yeah. Are we logged into this? I’m like, but I’m like, you should tell me at the beginning to go and log into these three sites. So, those are available. 00:36:52.51 | Michael Helbling: Well, you should just have it write a kickoff script that does all that for you. That sounds like a engineering thing, Tim. Oh, yeah. Yeah. 00:36:59.92 | Tim Wilson: I’m, if there’s anyone listening to this thinking I’ve got it figured out. Yeah. Let’s, we’ve made that clear that I do not. So. 00:37:06.55 | Julie Hoyer: Can I ask this, Tim? Like, is there any part of that tedious one hour that, well, I probably know the answer. But like, let’s pretend we’re talking about other tasks, maybe. Yeah. So, I want to come back to, like, can you make AI do it? Sure. But to your point, like, is it actually a task that is, like, fulfilling, enjoyable, needs you in the loop so much that, like, that is actually the valuable hour of a human doing it rather than all this pain of letting the AI do it for eight hours? I don’t know. I feel like sometimes there is this push to be like, well, if AI could do it, like, make them do it. Everybody in the organization, use AI somehow. Find some way to use it. Like you were saying, Danielle. Like, I just get to the point where I’m like, should I really be using AI for the token spend and the pain setting it up and telling the silly junior analyst AI how to do it? Like, I kind of just want to sit and drink my cup of coffee and do the clicking myself. But it’s interesting of, like, how do you draw that line? Because I also have a whole thought that I’ll wait on about, like, who that efficiency is actually then benefiting from AI. 00:38:12.40 | Tim Wilson: So, I will, I just want to quickly say this is episode 308. The joy of that, like, evaporated around, like, episode 120. So, no. Fuck no. I do not want to keep. But I do think there’s a distinction between having it do the agentic, probabilistic figure stuff out versus using it for coding something that then becomes deterministic that has been. Like, I’ve gotten my new ATRWconsulting.com. Like, I vibe coded, like, websites where it’s like. It’s there. It’s coded. And that was, like, I built something and the end was a deterministic code base. Now, how that relates to analytics, how much is, like, I’m building a pipeline that I can then refine and tweak and then I’ve got the SQL that’s reusable. Like, to me, there seems like there’s a distinction between the vibe coding to build reusable code or applications, which I think there’s. That still brings me. That brings me joy with the million little projects I’ve done on that front. The, I’m going to get this junior analyst. Like, it’s a junior analyst who I’m going to throw out a window because I’m like, oh, my God, follow the damn checklist. Like, you, the sponsors goes below the copy, not above. Like, it says it right there. So, boy, I don’t know about recording this at the end of this day, but. So, Daniela, how do we fix this? Yeah. 00:39:44.94 | Daniela Matinho: Oh, my gosh. What a roll. 00:39:52.72 | Michael Helbling: No, but I think we’re all touching on things where we all think AI has this real big positive potential. We’re also running into the opportunity cost of how to integrate it effectively in our day-to-day work such that it produces the results that are required of the roles that we have in our jobs. And the imperative that this must be something that is disruptive and is changing everything. So, we must also do it. And I think it’s put a lot of us in sort of like a weird headspace in our jobs and careers in the last couple of years. I don’t know, Daniela, like from your perspective, like in working with teams and coaching people, whatever you’re in your roles. How are you guiding people into how to think about? How are you guiding people into how to think about integrating their work in AI? 00:40:49.67 | Tim Wilson: That’s just like a more eloquent way to say, so how do we fix all this? Like you just, you ask the silly question and then you’re like, what if I, what if I give more of like a chat GPT, really long, coherent, but it’s the same question. 00:41:02.28 | Michael Helbling: You’re absolutely right, Tim. You hit on the key issue. 00:41:07.88 | Daniela Matinho: Well, the thing is, I don’t have the answer. No one has the answer. And what I’m seeing. And also because. I work in tech. We are in this industry that people just expect you to know the answer. Well, you guys are driving the thing. And very often we are all just trying to figure it out. That’s my, my take. And one thing I think that tech does very well is the ability to learn very quick and also course correct very quick. So as we integrate different AI tools to different companies that I have worked at. So very quickly we start sort of mapping what we are using for what? For dashboarding, for coding, for narrative producing internally for video editing. So having that sort of framework that enable us to, to know what to use for. Also, a lot of the sharing coming from the teams that are closer to, to clients or the more technical teams. A lot of the sharing is happening in these early stages of, Hey, we are discovering this and this is how we use it and how we can also teach other teams. Those are things that I have been seeing for myself. And I go back to this as I, um, I’ve been a power user for, for now a year and a half understanding that more. Is not necessarily better. That’s what I’m coming. And we are, we have touched on these a few times. I was very bullish on implementing, implementing, implementing, and now I’m coming to realize, well, maybe there is a lot of opportunity in slowing a little bit more. And doing extremely quality deliveries. Um, rather than just quantity, because I’m coming to see, well, all these agents, all these models can build way more. But it’s not having the impact that it used to have when I was doing it, everything from scratch. So kind of going back or taking a step back on, okay, I’m using it is helping me to produce, but I still have to put the 10 hours off. Deep thinking. This is how I build the story. This is how I use the three, four numbers that are really going to break through that. Maybe my agent in marketing science cannot really get it without having met the client. Even if, you know, I have a call that is being transcribed, so on and so forth. There is still, I mean, hopefully my brain is being used for something. 00:43:50.78 | Michael Helbling: That’s a really good point about the type of industry you’re in and how that might actually, because you were talking about sort of in your role and sort of tech, it’s like very fast moving, quick decisions, knowledge and learning is super fast. Like I have some clients that are in more regulated industries and like there’s things you can’t touch with AI. Like they just like, sorry, no note taking. Like that’s not allowed. Like, so it’s like, okay, you have to put those tools to the side and you do it the old school way. And it’s kind of interesting because you do, there’s different flavors or layers depending on what type of company you work with or what industry you work in. And so I think for listeners, that’s also maybe like a little bit of encouragement, like depending on the type of company, like you might not be lagging. You might be right where you need to be for where you work and what your company is. And if you feel like you’re going too slow, then go find a new job in tech and go fast. 00:44:51.05 | Daniela Matinho: Well, there are a lot of things that, you know, from the industry. But if you feel like, oh, maybe my industry, my company is not there yet, is the opportunity for you to be the driver. Because I was in the past at companies that weren’t maybe in that front end. And I was the one, let’s do it, let’s implement it. And all of a sudden I became the person. Oh. I’m going to go and ask Danielle about this because she’s implementing it. And maybe, oh, our decks look amazing. And I’m like, yeah, AI is doing a lot of them for me. 00:45:26.14 | Tim Wilson: I kind of heard something a little different, which now seems like totally obvious. Like everyone should be exploring, but we should be going in saying, I’m going to try to do this with AI. But logically, it seems like it might work. But I’m doing two things. I’m trying to complete the task. But I’m trying to learn. I’m trying to learn if this type of task is a good fit. And that little kind of meta-analysis level of, for you personally, anyone, I’m not speaking, that what are the types of things you’ve tried? Are you trying different things? Are you assessing after you’ve done them, like trying to figure out, hey, that kind of worked. And I think here’s why it worked. And here I could work better. Or some of the examples, I think, Julie, you were thinking of. What the? What the hell? Like, sure, I used AI to accomplish that. But it took me longer. And it cost the company extra money because of token usage. And you know what? It’s not just the next model is going to solve that. Maybe that’s actually something I should keep doing. Like I haven’t stepped back and said, let me make my catalog of the things I’ve done and tried to classify them. And then assess, like, man, it was amazing at this. Including totally new things. Like ideation around an analysis, which I think it does quite well. 00:46:48.68 | Julie Hoyer: It all makes me think, too, of how are we really defining the efficiency play? Like, I know a lot of people talk about it just as, like, time saved. And there’s a lot of assumptions behind, like, what time saved means and the value of that. But exactly what you were saying is if the end quality is not as good or it’s not as influential what I’m delivering, even though I did 10 of them. It’s just I feel like a lot of times people aren’t taking into consideration all of those measures to really say, was I more efficient? Or maybe that dips more into effectiveness. Like maybe they’re not looking at both efficiency and effectiveness when using AI. And it would be a really interesting exercise to your point, Tim, to really be like, how would I measure those things? And if I look at all the things I’ve done with AI, like, which ones were truly, like, worth it and successful? 00:47:44.36 | Tim Wilson: Yeah. And the ones that were really fun, but they actually were a complete waste of time. You’re like, yeah, but it was. 00:47:49.16 | Daniela Matinho: Yeah. But it was so cool. So it’s a really interesting point. And it’s something that I was seeing a lot maybe, what, a year ago when AI was really hyping. Like we were seeing in a lot of the calls, quarterly calls when public companies have to report. We were seeing it X millions, billions in AI, X prompts being digested. And number of people that we were hiring. These were a lot of, like, very concrete numbers that I just, from my perspective, companies maybe were having a lot of pressure in delivering, oh, we are an AI-driven company. So we were seeing a lot of these things. And to your point, Julie, right now that we have, I don’t want to, I don’t know if we have passed completely that phase, but now that AI is being integrated in the majority. Of companies is more of, oh, what are my, what am I actually delivering? Or what is the revenue that I’m actually driving because I was able to use AI? Because at the end of the day, if I go and tell a client, well, I did this in two days instead of a week because of AI, the client doesn’t care. Right? That doesn’t really matter. Or, you know, at the end of the day, what the client wants is. A good, a good output, a good result that is proving revenue for whatever I’m trying to, or whatever I’m trying to deliver. So AI really is, is the underlining sort of army that is helping us to be a little bit more efficient, but in a way to deliver revenue. At least I’m close, I’m close to sales in general. So I always think about, how can I drive more dollars? 00:49:43.32 | Michael Helbling: Yeah. Well, and then being more efficient, we have to remember that it’s also our job to stay influential, right? With data. So that’s the balancing act that doesn’t show up in the numbers a lot of times. 00:49:54.74 | Tim Wilson: But that, that you say stay influential. I think, I think there, we’re not going to AI your way out of not delivering value. Like I, I think when analysts kind of wring their hands and gnash their teeth and say, I don’t have a seat at the, at the, the board table. And then they’re like, now I’m going to use AI. It’s like, well, okay, that’s going to help you produce more. Maybe it’s going to help you be more consistent, but it’s like, you, you still need to figure out like, what is the path to get there? Like, I, I definitely worry about that. That it’s like, well, here’s the thing it’s going to solve. There’s fundamental relationship and ways of working and thoughtfulness, things that AI is just inherently not going to solve. Unless you read it. You recognize it and maybe AI can coach you and help you learn it. But I think that’s another, I’m thinking about, we saw Sam Burge, a past coworker of ours present at Marketing Analytics Summit. And I’ve continued to talk to her and part of what she’s in her team are using AI for is different mechanisms for delivering information. So even if fundamentally the information they’re delivering is the same. And I think that’s a really good point. They’re spending maybe the same amount of time, maybe a little more, maybe a little less and saying, but I’m going to tailor it to the audience. Maybe I’m going to adjust the, you know, people in the past when he said, I only have time to produce the deck. So that deck has to be the pre-read the leave behind what I present. But now I can produce the full detailed report and I can produce the executive summary and I can generate, you know, an audio podcast. That’s three. That’s three minutes that delivers it and really thinks about how am I serving my business partner more effectively, even though potentially the raw, what is being delivered hasn’t changed. It’s helping deliver it more effectively to the audience, which is like a really like thoughtful way to think about it. And that, I think she’s had a lot of tried this. It didn’t work, tried this. It did work, which goes back to the point of saying, I’m not just going to turn it on and it’s going to work. 00:52:10.94 | Daniela Matinho: Well, a lot of iterations. What I’m seeing with content. And this is something that, um, a few months ago, conversations with, with colleagues, with friends. Well, we can just now do all these videos, all these posts with AI. It’s, you know, quantities is amazing. We can all become influencers. And, you know, a few months down the road is like, well, no one has become an influencer because we all just talking the same and speaking this, you know, using the same words because everything is being produced. So that level of how do we differentiate to your point team? How are we touching the audience the way we need? How are we shaping it? That’s the layer that in my perspective, AI is just not there yet. And when hopefully our brains and our creativity is, is being super needed to, to really stand out in a way. 00:53:09.59 | Tim Wilson: It’s one of those philosophy PhDs are using AI to read. They read Nietzsche and produce their thesis, and then they actually get a philosophy PhD, but they don’t have the critical thinking. 00:53:19.61 | Michael Helbling: All right. This is great. Daniela, thank you so much for sharing your experience and expertise with us and our listeners. This has been an awesome, good discussion. Very fun to kind of open this up. And I think it’s very applicable to everyone’s jobs right now. So, yeah, but we do have to start to wrap up. Unfortunately, the time flies by. All right. Uh, one of the things we like to do. Is go around the horn and share a last call, something that might be of interest to our audience. Daniela, you’re our guest. Do you have a last call you’d like to share? 00:53:50.55 | Daniela Matinho: Yeah. Um, I was on a call yesterday with IAB, the advertising bureau, and we were talking about agentic commerce. So very, very curious as we were talking, uh, right in the beginning of this call, what are, what are the automation tasks that we’re going to be seeing? And I think a lot of the shopping. Online, especially the recurrent shopping is going to completely change the way we shop the way you use. We use LLMs to inform us and to basically place everything for us and order it. I think it’s completely going to change the way we shop online and I’m excited about it. 00:54:33.48 | Tim Wilson: Is that I was having this, I got a, sorry, Michael, but I’m like, this is like getting people to part with their dollars. Like in, when people who are used to. Like hallucinate hallucinations, some bad judgment calls. Cause I feel like they’re in the digital analytics world. There’s been a whole lot of like, oh, this is coming. Like the agents are going to be sent to get shopping. And I’m like, like how much is the IAB? And are you thinking that that’s going to be, I may send an agent out to find a lot of options, but I don’t know that I’m going to cut it loose to make the purchase decision. 00:55:11.13 | Daniela Matinho: Yeah. That’s what we are. Seeing, but not yet for the, to pull the trigger, but a lot of the search. A lot of the search is happening as we were doing it. Um, having one person or one agent that is basically pulling everything for us based on what they know about us. I think the next stage to your point, it will be interesting to see how long it’s going to take. 00:55:32.61 | Tim Wilson: I just see like retailers who are saying we’ve, we’ve got to be supporting, supporting agentic commerce. And I’m like, is it the purchase you’re trying to support? Or is it, you’re just trying to surface. The shopping, the shopping versus the purchasing. Hey. I don’t know. 00:55:46.76 | Michael Helbling: DoorDash already has a command line interface. So, so everybody’s doing it or starting to. The DoorDash MCP. All right. That’s a whole other podcast episode, Tim. Why don’t you share your last call, Tim? 00:56:00.70 | Tim Wilson: Okay. Well, mine is a podcast. It’s an old podcast that’s new again. And I am super excited, but I’m also like a political junkie. But the, the five. FiveThirtyEight crew of Nate Silver and Claire Malone and Galen Druk. The they’re, they’re back. So like FiveThirtyEight kind of did it’s bouncing around through ESPN and the New York Times and was gone, but they, there’s now they’re back on the Still Counting podcast there with the Crooked Media network. So it’s I mean, it’s clearly political stuff, but really, really sharp data minds. And I, I am. I’m enjoying the first few episodes quite a bit. So basically FiveThirtyEight podcast is back, but now it’s called Still Counting and it’s in a different world. 00:56:52.31 | Julie Hoyer: Mine is actually very applicable to today’s episode. It was a Cassie Kozyrkov newsletter. We love those. And the title was a really interesting statistic and it’s chock full of them, but it’s all about how 97% of S and P 500 filings mention AI. But you wouldn’t believe the percentage that actually like had written out budgets for them. And even fewer actually describing like how they govern them. And so she goes through some of these crazy statistics. So highly recommend looking at it. And then at the end, she’s kind of closing it up with this idea of like the winning companies of using AI are probably going to be the ones who trust their systems the least. And I know that’s been just a big theme of like, you can’t just hope. That they’re doing the right thing. And like that sense of goodwill towards them is really kind of been eaten away through a lot of experiences the past year, year and a half. So it was a, it was a good quick read. 00:57:55.16 | Tim Wilson: What about you, Michael? 00:57:56.51 | Michael Helbling: Well, somewhat related to this episode, there was a website that came across recently that I thought was clever. And of course it’s a vibe coded website called the archive of incorrect AI predictions. So it has this whole archive of people making predictions about AI that have been wrong throughout history. So way back in the past coming forward in time. And you can look at it in different ways. It’s kind of fun. So yeah, it’s a fun little website. I like the name of it. It’s Boys Who Cried AI. It’s the name of the website. So anyways, that’s a, that’s a fun one. All right. So as you’ve been listening, uh, you’ve probably have thoughts of your own and we’d love to hear your comments and suggestions. Leave us a comment. The best way to do that is on the measure Slack chat group or our LinkedIn page or via email and contact at analyticshour.io. And we’d love to hear from you. Also, uh, please, uh, wherever you listen, leave a rating and review on the podcast that helps us out. And if you would like a sticker for your laptop or your mug or water bottle, we’ll send you one and you can do that on analyticshour.io as well. And once again, Daniela, thank you so much for taking the time being here today. It’s been an awesome conversation, very timely. I think everyone’s work is feeling this impact. So thank you for coming and sharing some time with us today. 00:59:27.98 | Daniela Matinho: Thank you for having me. 00:59:29.09 | Michael Helbling: Awesome. And I think I speak for both of my co-hosts, Julie and Tim. When I say no matter where you are in AI adoption, keep analyzing. 00:59:41.36 | Announcer: Thanks for listening. Let’s keep the conversation going with your comments, suggestions, and questions on Twitter at analytics hour on the web at analyticshour.io our LinkedIn group and the measure chat Slack group music for the podcast by Josh Crowhurst. Those smart guys wanted to fit in, so they made up a term called analytics. Analytics don’t work. 01:00:05.66 | Charles Barkley: Do the analytics say go for it no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 01:00:18.38 | Tim Wilson: Staying in detail and be like, oof, maybe I would have done it so much better. She’s been doing that throughout the course of the planning. Maybe. That’s an interesting choice you guys made with how to do it this time. 01:00:35.80 | Michael Helbling: I think what you should do is actually take that audio snippet and send it to Val and see if that’s accurate. 01:00:42.38 | Tim Wilson: Well, all through the planning, as she was trying to watch from afar and be hands off, I would often get the, I was the vent, the outlet for her. Oh. Rock flag and philosophy majors. The post #308: AI Changes Everything! Analytical Fundamentals Still Apply! appeared first on The Analytics Power Hour: Data and Analytics Podcast .
