Find partners
Content Operations

Content Operations

Hosted by Scriptorium - The Content Strategy Experts

BusinessInterviews guests

Episodes

198

Latest episode

Jun 2026

Language

EN-US

About the show

The Content Operations podcast from Scriptorium delivers industry-leading insights for scalable, global, AI-optimized content.

Listen to episodes

60 recent
June 1, 202642 min

Tool selection and the unpredictable variable

How do you really choose the right documentation tool? In this podcast episode, Sarah O’Keefe (Scriptorium) talks with Paweł Kowaluk and Michał Skowron (Guidewire Software) about building a successful tool selection process, the realities of docs as code, and what happens when the technology becomes the unpredictable variable. Paweł Kowaluk: It’s funny how programming used to be deterministic, and it was the people who were messy. We always knew that people are going to be whimsical and maybe harder to rein in, but the technology is going to be predictable. Whereas now, technology is not predictable anymore, and you give it a prompt and you hope it’s going to do what you want. You adjust the system prompts and change the weight of things which are retrieved versus metadata, et cetera, and it doesn’t always work the way you expect it to. Sarah O’Keefe: And now the people are being asked to be the deterministic layer, right? To be the QA on top of the AI. Paweł Kowaluk: That’s actually very insightful. I like that. That is true. The human in the loop or whatever you call it, that’s supposed to be the voice of reason. Related links: Scriptorium: AI in the content lifecycle Tech Writer Koduje podcast Tech Writer Koduje: DITA as code – a modern approach to the classic standard Tech Writer Koduje: Are people abandoning docs as code? Tech Writer Koduje: A tech writing CCMS can also be a broken promise LinkedIn: Host: Sarah O’Keefe Guest: Paweł Kowaluk Guest: Michał Skowron Tech Writer Koduje LinkedIn profile Transcript: Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction Sarah O’Keefe: Hey, everyone. I’m Sarah O’Keefe, and welcome to the podcast. In this episode, we are going to talk about tool selection with a couple of special guests. With me today are Paweł Kowaluk, who is a software architect at Guidewire Software, and Michał Skowron, who is a documentation tools developer, also at Guidewire. Both of them are based in Poland. Welcome. Paweł Kowaluk: Hi. Michał Skowron: Hello. SO: I am glad to have you. For those of you on this podcast that speak Polish, you’re probably already aware that they have the one and only techcomm podcast in Polish that is available out there, and Michał and Paweł are also experts on doc process and tool selection, so that’s what we wanted to focus on today. So I will start and throw it to Michał and ask you the big picture question, which is what does a good tool selection process actually look like? MS: For me, good selection tool process would be divided in three stages. The first one would be gathering requirements, looking what’s out there, defining what you want to basically achieve with this new tool. Then I would go to a pilot project where you can actually test the selected tool in the real world. Manufacturers and producers of software will tell you that it can do anything and it will promise that, “Okay, you can meet all your requirements easily and we can fix that, we can improve that, we can adjust that,” so everything can be done is usually what we hear, but then you want to test it in real world on a real project, so that will be a pilot project for you and your team. And the third phase that depends on the outcome of the second phase, which is you either productize the selected solution or you just say, “Okay, that was a bad choice and we don’t need that.” Then we need to go back to the first stage and then say, “Okay, we need to select another tool,” and again, requirements, et cetera, et cetera. So for me, that’s the whole process, and the first stage would be probably the longest one because you need to make sure that you are meeting all your goals. SO: So what’s the most common reason that a pilot doesn’t succeed, that you have to go back and say, “That didn’t work. We have to try something different”? MS: It’s usually because you didn’t see everything when you were planning. For example, you have some projects that are very specific or you didn’t see all the problems or things that are coming your way. It’s hard to say exactly what the reason is, but it can be multiple reasons. For example, using of, I don’t know, branching, let’s say, in a specific tool. When you have multiple versions of your product and you want to keep them separate when it comes to documentation, it can turn out that the feature says, “Okay, you can use branching and then you can do it easily,” and then you start using it and it turns out that it doesn’t work the way you expect it. This is actually a real life example because we had a system that… I’m not going to mention any names or anything like that, but there was a system and they promised us… That was years ago and it was a vendor that promised us that they’re going to introduce a feature called branching, and it turned out after they did that that it wasn’t what we expected. So it can turn out in many cases, in many ways, it can be the problem, but branching is just an example, but it can be many other things that can go wrong. PK: Hey, if I can jump in here, I got a couple of examples. One is I could call it releasing strategy or versioning strategy overall, which is very hard to test in a pilot project. It’s very hard to scope for requirements because the little problems come out after a while, after a year of publishing, after two years of publishing. And another example which is related is reuse, and this one is down to formulating the requirements correctly. Because I think just saying, “We want to reuse something,” is not enough, because you have to say exactly what you want to reuse and how you want to reuse it and what you want the result to be. So for example, if you say, “I want to reuse notes and warnings and things like that.” We sometimes call them admonition. So, “I want to reuse these notes in my docs, and if I update a note, I want it to update in every published version of the doc.” Then only if you have these details, like I want to update it once and then want it to automatically update and publish docs, then you will see it’s not working the way you expect it. Because if you just say, “I want to reuse notes,” every system can reuse notes. Even in docs as code, there’s scripts and macros that allow you to reuse notes. MS: It can be also another thing that, for example, you compare the benefits with the actual cost of implementation, and it can turn out it’s not worth it because people are, for example, reluctant to use your new tool. The training, the cost of licensing, the cost of support is too big, and then you realize, okay, we want to achieve a goal like, I don’t know, reduce the time to market, and then it turns out it doesn’t work because people are struggling with using the product on a real project. And on paper, everything looks cool and you have all these features, you can use them, like Paweł mentioned, for example, reuse conditional formatting and things like that. And then it turns out it’s very hard to use, it doesn’t serve its purpose, and then you have to pay for every additional stuff and people don’t want to use it, so what’s the point? SO: Yeah. And I think we find that the technical problems in general, if you’ve done your requirements work, then usually, the technical problems are solvable if the people engaged in the project want to solve them, and that’s where you run into the change management issues that you’re talking about, that if the team that is being asked to pilot, to test, to try things out is sufficiently disinterested in making it succeed, they will find a way to make it fail. And the reverse is true as well. If they want it to succeed, you can implement tools that are … Well, all tools are imperfect, but you can implement tools that are not perfect solutions and succeed if the team is behind you, and if they’re not, bad, bad things will happen. PK: Oh yeah, that’s true. And I’ve been on projects where we did not do proper change management and I’ve been on projects where we really did it well. If you start early and you involve people, like get the biggest troublemakers, people who are the most opposed to any change, get them on the team, and if you can convince them, that means, one, you are making the right choice because you’re convincing people who are skeptical, and then two, you are set up for success. These people are going to be your biggest champions of the new solution. MS: But it’s good that you mentioned it because I think it’s worth emphasizing that the goal of the pilot project is not to succeed. That’s not the actual goal. The goal is to verify the requirements against the real project, and so the failure is also a success to some extent. It’s not like you have to do everything to prove that the selected tool is the right one. No, you should be aware that the pilot can end up with your let’s call it failure, and then you realize that it’s either a bad choice or you don’t need that at all. For example, you don’t need that tool at all. It can be also the outcome of the pilot project. So there are many different outcomes, so don’t be fixated on the success path that is the only right way. It means, okay, the pilot project was a success because everybody agreed that that was the right tool that we want to use, so keep it in mind. SO: Yeah. I think that there’s … I got into actually a debate with somebody about this. They were telling me that pilot project, proof of concept, and what was the third one? Prototype are not the same thing. Okay. Well, so a pilot project is, “We think this is the right answer and we’re going to try it small and then we’ll expand.” A prototype is, “We’re just going to try it and throw it away,” and a proof of concept is, “We think this is the right answer, but we’re not sure and we’re prepared to throw it away.” And I thought, “This is way too specific for me, but okay, sure.” But to your point, the project, whatever category it belongs in, has different purposes, and it’s important to be clear about what kind of a project is this? Is this essentially beta testing, this is step one, or is this more speculative? We’re just really, really not sure. PK: I think it has to be falsifiable. Like they say in the scientific method, there has to be a failure criteria somewhere. This will fail if ABC happens, because if you don’t have that- MS: Or a success criteria, right? PK: Or a success criteria, but I’ll be more focused on the failure criteria because you can always show, “Yeah, we accomplished this, 30%,” but if your failure criteria is below 50% is fail, then you will say, “I failed this.” You fail five out of six and the project is a no-go. SO: So I wanted to switch gears a little bit and talk about some specific tool chains and problem sets. I think it’s fair to say that the two of you are, I’m going to say mostly but not exclusively, focused on docs as code, so what does that look like? And there’s a lot of debate about docs as code versus structured content and they’re very much pitted against each other, but ultimately, what’s your perspective on the situations where docs as code is appropriate or maybe is not appropriate, and what do you look for in a project or in a problem set that matches the docs as code model? PK: I think the reason people associate us with docs as code is the last eight years, we’ve been working at Guidewire and that’s the strategy we’ve chosen there, so I think we’ve been entrenched in this worldview. My previous job before Guidewire was a consultant, and I would go from a company to company and set up different systems for documentation. That was my specialty, systems for publishing and updating documentation. So yeah, circling back to your question of when it works, when it doesn’t, I guess docs as code is more about it works when the right mix of people are creating the docs and the mix of people includes software developers. So for example, at Guidewire, we have several dozen static websites which are maintained by software teams without any technical writers involved, and those are a variety of internal tools, tools which are almost external, and then tools which go out to customers, and all of these little websites integrate very well with our publication system. And I think this is the main criterion for docs as code fitting is you are giving tools for writing to people where they work. So software developers work in code. You give them tools for writing in their codes so they don’t have to buy extra licenses, get training on anything external, use some alien process, alien to them, just because they follow SDLC, the software development lifecycle to update their docs. And it’s what they’ve been doing, and then it’s easier to convince or it’s easier to fit in a smaller team of technical writers, I don’t know, like 40 technical writers versus 2000 software developers. It’s kind of everyone contributing to the same documentation system. It’s easier for the tech writers to adapt and join the docs as code platform than the other way around. MS: Just like Paweł mentioned, docs as code makes sense in certain environment. It’s not like we are tribal about it. We love this solution because it works for us. So after a few years at Guidewire, we realized that this is the way we want to go, because as I said, it makes sense and it just works. We tried different things before, and before I joined Guidewire, there were different solutions. We used, for example, CCMS and we had more a traditional approach to producing docs, let’s call it this way, and then we started building our own pipelines, our own solutions. We started integrating with what was there that other devs used for their work, not related to documentation, but we decided, “Hey, maybe we can use what’s out there and just plug into the same infrastructure.” So as I said, it’s not for everybody, it doesn’t work in every environment, so I wouldn’t say, “Okay, if you’re in a factory, you should use docs as code.” No, I wouldn’t say that. So maybe we positioned ourselves as docs as code proponents, let’s call it this way, but this is because we use it every day, we build it, and this just works for us. And also maybe I can mention that we decided to end this holy war by putting DITA into Git and CICD pipelines, so we have DITA as code, and now everybody’s happy. PK: Oh, this is actually a great point because Sarah, also mentioned docs as code versus structured. Now we’re doing docs as code in a structured way, and where technical writers are working on the docs, they are free to use DITA and a lot of teams do that with all the reuse and all of that, but even if it’s something simple like some markdown files in a repo, we still impose a metadata structure which allows us to integrate the doc into our publishing pipeline, and the two key aspects of what we’re integrating with our authentication and search, and that needs metadata, right? It needs to know who has access to this content and it needs to know how to filter and direct the searches, and from that, our next project emerged. Like everyone else in this industry, we started working on AI solutions, and the AI solution is a third integration point where this structured approach to content also pays off. SO: Yeah, I did want to touch on AI and so we should probably just jump to that. What are the implications that you’re seeing? What are the effects that you’re seeing on your current processes of the use of AI or the requirement to deliver to AI, and how do you see that working? MS: Paweł, you want to start? PK: Yeah. So the content remains the same that we’ve been working on for years, and the structure, the metadata is all the same. We could not change this overnight, but overnight, we had to introduce this AI which is going to look into the content, find the topics, the chunks we call them. It’s going to find the right chunks of documentation and generate answers based on those chunks. So what happened is we had to adjust the AI, but since we were so structured, it was pretty easy, and then we gained a new source of feedback from customers through the AI. Because when people started using our chatbot, we built a chatbot which is available on docs@guidewire.com, but it’s only available to customers and partners, so you need a login on the website. When you go there, the AI answers questions, and then we monitor the exchanges that people have with the AI and we see what needs to improve as far as content ingestion, as far as content structure. We generate a lot of internal tickets to our tech writing team and identifying gaps, because we’re seeing now that this AI interaction is possible, here’s what people are asking of the content. And other people are asking, “How do I implement X, Y, and Z in the scenario where ABCD?” Which we didn’t know people were looking for before because they had no interaction with the docs. So they just come to the website, they either find or don’t find what they need, and we don’t know anything about that, and we could ask them, but we’re just going to ask a small group of people, whereas now, it’s as if we’re asking everyone. So we’re seeing these new patterns and something emerges out of those patterns, like everyone’s asking for code samples in this specific area, so we go back to the team and we work on adding more code samples, or people are looking for illustrations of these particular workflows, so people create these illustrations. It’s amazing how rich of a source of feedback this has become. MS: And it also adds complexity to even testing all the solutions that we have right now, because with keyword search, I’m not saying it was easy because it wasn’t easy, but now we have another layer where you have this middleman, let’s call this chatbot, let’s say. It’s a middleman that can hallucinate, so you can either have a problem with your content or you can have a problem with the agent that responds. So before, when you had a keyword search and you were missing results, you were going usually to the content and see, “Okay, I’m missing this topic. It wasn’t indexed properly, or I just need to move some knobs in my search engine and just see the fuzziness or something like that.” And now you have this content, it’s being processed by this AI tool, and then it gives you some answers and you don’t know why the answer is wrong. Because it didn’t find the content? Because it’s not there? Because it’s not the right content? Because the content is right, but there is something that it missed? There are so many moving parts right now, more than before that it adds complexity to our job, and this is just one side of it. The second thing is writers using AI for creating content, and we’re also exploring this path because everybody’s trying to incorporate AI solutions into their work. We code mostly on a daily basis and we use some tools that help us with coding that are AI-based, but we also want our writers to benefit from all this development in technology, and we’re exploring, let’s say Oxygen, XML, Positron. Just to clarify, we’re not sponsored anything. We’re just using it so I’m just mentioning the name specifically, but even if you don’t have any specific tool for writing that integrates directly with a AI tool, you can still use, let’s say, Copilot or any other solution that you like and you can just ask it to help you with the content, but it comes with a lot of caveats. It’s not like you just throw a prompt and just get what you need. It involves a lot of fine-tuning, a lot of working on instructions, giving it context, giving it information that it needs to use for producing code or docs. Because I use it for both, for coding and for documenting, for internal purposes mostly, but it takes time to make it work the way you want it. PK: Yeah. It’s funny how programming used to be deterministic and it was the people who are messy and the processes, and working with setting up the process for the doc tools, we always knew, people are going to be whimsical and maybe harder to reign in but the technology is going to be predictable. Whereas now, technology is not predictable anymore, and you give it a prompt and you hope it’s going to do what you want. You adjust the system prompts and change the weight of things which are retrieved versus metadata, et cetera, and it doesn’t always work the way you expect it to. SO: And now the people are being asked to be the deterministic layer, right? To be the QA on top of the AI. PK: That’s actually very insightful. I like that. That is true. The human in the loop or whatever you call it, that’s supposed to be the voice of reason. SO: I don’t know about you, I’m not good at voice of reason. I’m much better at causing trouble. So what you’re describing is a fairly complex and time-consuming approach to this in that you’re saying we have this AI chatbot and we’re looking at the metrics and we’re adjusting accordingly and making changes, and then using the AI on the backend for some productivity kinds of things, but not as a replacement. We’ve seen in the US and in North America, we’ve seen a significant number of people losing their jobs because the idea is that, oh, the AI can just do it, and what you’re describing is not that at all. So are you seeing any of this sort of, “Oh, this will make us more efficient, and therefore, we need fewer people,” or is it a different perspective in your piece of the tech com world? MS: I’m aware of all the layoffs and of all the bad things that are happening right now in the IT world, let’s call it, because it’s not only technical writers because it’s also developers and different jobs, but I think I’m still in this kind of a bubble right now where it’s not happening directly next to me so maybe I’m being too optimistic. But I keep saying maybe I’m going to eat my shirt after some time because I keep saying, “Show me one tech writer that doesn’t have a backlog that is not too long.” So we usually have too much work, and I hope that with the right approach and with a sensible management, these AI tools will be doing all the things that we don’t want to do or we don’t have time to do, and then it will give us time to do something more and something more significant. I’m looking at my job and I’m not saying it’s perfect because of AI, because it has so many challenges right now and it gives you different problems, but I can see that I can speed up my cumbersome tasks very, very easily. And before, I had to … This is a simple example, and I’m doing a lot of infrastructure work and a lot of backend work, so many things go wrong, and very often, I had to debug difficult problems. It was taking me weeks sometimes to nail the actual place where it happened. Now, I can do it much faster. If I have to debug a problem, it takes me, I don’t know, an hour, sometimes even minutes, and I know that I would spend a day, two, or even a week if I didn’t have these tools. But these are good examples, but there are also a lot of bad examples, but I don’t think we have time for that. SO: We’ll take the optimistic view. PK: Yeah, I agree with Michal. The approach we have is these tools can give us tremendous productivity boosts, but not in the sense of getting rid of people. We can redirect people’s work to where it’s more meaningful. And what I mentioned about feedback, identifying these content gaps. Some of these content gaps are going to be very mechanistic and you solve them by generating a bunch of docs out of code, for example, and then you put those docs in the repository where the chatbot can find them, because they’re not going to require a lot of thinking. It’s kind of like API docs where you create the swagger spec and then you generate the dogs out of that. You just grab the source code and you generate some samples, and you use that to generate answers to people. Without people, this wouldn’t work because you need, like we said, the voice of reason, but yeah, people are still required in the process. SO: I think that looking at it across the industry, if you take AI, take a chatbot, especially a public facing chatbot, and ask it for answers on literally anything, it will give you the average of what’s out there in the world because it’s math, so it gives you the average essentially. And so from a content creation point of view, I think the fact of the matter is that there is in fact a lot of below average content out there. Roughly half is below average, or perhaps exactly half. If the information is bad enough, if the content being produced is not of high quality or ungrammatical, and we’ve all seen terrible, terrible documentation that was badly translated and is just incomprehensible, then the AI as a tool for creating content may be able to produce something that is better than terrible. It’s not going to replace a professional, well-trained, highly experienced and knowledgeable product documentation group, or it shouldn’t, but that’s not every company. Not every company has a really good group of tech writers that understand the product and are adding value as they produce the content that goes with that product. And so I suspect that what we’re going to see is a split with commoditization on one end, just auto generated, it’s not great but it’s adequate maybe, and then the higher end stuff, which needs to be done well. PK: Well, there’s definitely documentation which only exists because it absolutely has to, and companies like that can easily generate just some kind of documentation and just use it, right? But in cases where … I think the worth, the value of a technical writer is not just writing and generating the content from below their fingertips. It’s more about the research and understanding, like you said, understanding the needs of the users and understanding how to meet them. You throw AI at a problem which sounds like a generic problem, it’s going to give you a generic answer, not necessarily one that is rooted in the organization you’re in. The list of products that you have, the way those products interact with one another, it’s going to miss all that. It’s just going to give you … It’s like ordering a hamburger and you say, “So what’s in the burger?” And the AI is going to tell you, “Well, usually in a burger, it’s a patty and lettuce.” And it doesn’t mention that at your restaurant, you use pear and avocado. It doesn’t know that. You know that. You’re the chef back in the kitchen and you know why your burger is special. SO: Yeah, I think that’s right. So I did want to touch briefly on the question of, because this is a rare opportunity to talk to some folks that are not based in the US or North America, what differences, if any, do you see in tech writing in the market? Now, you’re based in Poland but working for a, I think, US company, but what kinds of things do you see that are maybe different that especially the people listening to this in the US would not be aware of coming out of Eastern Europe? MS: For me, always, it was the fact that tech com appeared much later than in the United States, in Poland of course, because I’m not talking about Europe in general because there are also differences between countries in Europe. For example, Germany is totally different from I would say the Polish market, because in Germany, you usually have factories and you have technical writing associated with hardware, let’s say. SO: Yeah, heavy industry, machinery. MS: Yeah, heavy industry, that’s right. And in Poland, it’s very often about software, maybe because we have a lot of companies that outsource to Poland when it comes to software development, R&D and stuff like that, so that’s the first thing. And don’t get me wrong, people were doing tech com way before it appeared in the mainstream, but sometimes they were not even aware that this is called technical writing or technical communication. Actually, it happened to me because I moved to techcomm from some kind of IT support job, and then after a year, I was like, “I’m wondering if this is anything regulated or there are some rules or it’s actually a profession.” Then I started digging and here I am. But it turned out there are so many things, so I was also surprised. Okay, this is called that. These are standards. There are books. Well, actually, one of the first books that I read was your book, Technical Writing 101, which I’ll still recommend, although it’s been years since it was published the first time, but I think it still holds a lot of truth about techcomm. The core values of techcomm are still there. So going back to your question, the techcomm scene is relatively young, let’s call it this way, in Poland, which gives us a big opportunity to skip some stages, let’s say. So we don’t have the legacy, we don’t have some things that we used to do a certain way and we like it or we are just accustomed to it. We can just start fresh and just jump. In 2000s, we just jump into techcomm and we say, “Okay, let’s do it this way because now this is the way we do it.” And I think that people are flexible when it comes to looking at things a certain way. Not everybody because there are also people who don’t like changes. But I think also what is unique about Polish techcomm scene is it’s relatively small, so if you do something outside your work, you don’t treat your job as only a means to earn money and just survive and you want to do something else, let’s say just write an article, like give a presentation, go to a meetup. After a year or two, you just keep meeting the same people, you know basically everyone who is in the circle, so it’s much easier to be visible if you do something outside your work. So I think that would be something unique about our techcomm scene. PK: I think what might be a downside of the Polish techcomm scene is a lack of veteran experts, because we don’t have people who have been doing this for 40 years. I’ve been in the industry for 18 years. SO: But you do understand that you two are the veteran experts. MS: That’s what he’s trying to say, that- PK: I’m getting to this. So yeah- MS: Imagine that. PK: Putting on my clown makeup every morning. What I mean is there’s nobody to look to who wrote the book on technical writing and has been there for 40 years. I’ve been here for 18 years doing this thing, and out of the people who are visible publicly and who contribute to the scene, I don’t know if there’s anyone who has more experience than me. Because I know there are people who have more experience but they’re just not visible. They don’t share their knowledge with anyone, and we miss that, so that’s why we look to the West, to people like you guys, like Scriptorium, and we read books which were created there and we see there’s definitely value, even though something is as old as the scene here or older. MS: And people also have this tendency to look at things that were done in the past as like, “This is the old stuff. We don’t need that. This is the old way. Let’s do it this way because it’s better now, because we have all these tools and et cetera.” But I think it’s a big mistake, because what they say? History doesn’t repeat but it rhymes, something like that. So in order to do your job in the present, you just need to look at the past, and this way, you can also be ready for the future. I think it’s worth looking at all this, let’s call it legacy stuff, all the experience that people with 30, 40 years of experience in the field have because it’s valuable. Because usually, when you look at it much deeper, it’s nothing new. It’s usually something that already happened but it’s dressed up as something else. So if you look closer, it’s usually something that let’s say you can understand what happened and then you can apply the same rules, maybe slightly change them or bend them. But my experience is the same happens in software development, in coding. People are inventing new things, and when you look closer, it turns out somebody already said that in the past. It was already done this way, but it’s dressed up as something else or named differently or is hyped more, or it was invented too early and now is the time to use it. So the history gives you a lot of perspective, a lot of things that you can use, so don’t dismiss it. SO: Well, I think that’s a good point, and as we close this out, I want to ask you what you see in the past or where you’re gaining perspective on the introduction of AI. Whether it’s from a delivery point of view or a backend productivity point of view, where do you see that pattern previously, or do you? PK: There are similarities, but they end. They’re not perfect analogies. So going digital was on example, and when I started, it was just on the brink of companies going from print to web, and that was a huge paradigm shift. And we’re seeing reverberations of this even today where teams are still thinking in books and chapters instead of thinking in pages and thinking about even every page is page one, which is, I don’t know, it’s older than me, the idea, but it still hasn’t been adapted to by teams. Now, the AI thing is probably a similar earthquake where it’s the AI reading your docs, giving you answers. You’re using AI to generate these docs, et cetera, et cetera. I don’t think we’re going to get 20 years of leeway to adapt to that and still see people doing things the old way. I think the world maybe moves faster nowadays, but I don’t know, we’ll see. MS: It may be something similar to the invention of computer. So instead of writing let’s say manually, people have got this new device that gives you more power and you can do more things and then programming languages and everything, but as Paweł mentioned, the pace was much, much slower. So we had years of development to, let’s say slowly, maybe this is the bad word, let’s say slowly adjust to the change, and now it’s an earthquake. So every day, every week, there is something new and you need to keep up, keep up, so the pace of development I think is the biggest challenge. Because we tech writers survived many revolutions. I just started reading a book by Sharon Barton about the women in technical communication, and women who talk about their careers, they mentioned many, they start very early because there are a lot of examples from the States. So they started working when they were not even treated equally with men, which was years ago, and they mentioned all those changes, all those revolutions, all those evolutions. And I’m saying, “Okay, this is what we are witnessing right now, but the pace is definitely faster, so we need to just keep up,” which is hard, of course. PK: Well, I have another one which is a good one. I don’t want to not say it. The dot-com bubble. I think maybe we’re in a bubble again, maybe. You’re seeing the inflation of AI prices right now, and I think we’re going to end up in a position where you cannot do all these things with AI which we’re hoping to do with AI, so the surface of usage is going to shrink and there’s only going to be specialized uses and special case scenarios when you use AI because it’s going to be expensive, and a whole lot of AI applications are just going to disappear. This might be a parallel, but I don’t know, it’s always hard to predict, especially the future. MS: I’m also predicting some corrections. Let’s call them corrections. SO: And I three am also predicting some corrections. You touched earlier on the fuzzy. AI is very good at fuzzy problem solving, but it’s being thrown at things that are old problems that we know how to solve with scripting. And scripting from a computing power point of view is cheap or maybe free. You write your script and you run it, and every time you run it, you get the same predictable result. Now, sometimes you don’t want that. Sometimes you need that fuzzy pattern matching thing, but in the cases where that’s not what you want and we’re still using AI, that is part of the bubble, right? Everything looks like an AI-shaped problem. So yeah, I agree with you. I think that any predictions we make on specifics as to where this is going are guaranteed to be wrong, because I’ve tried this before and I’m always wrong. So I want to thank you both. This has been very entertaining and very interesting, and I hope that we will see you again and you’ll come back and tell us more about what you’re up to over there. PK: That would be lovely. I would like that very much. Thank you. MS: Yeah, sure. No problem. Thank you very much. That was fun. SO: Yeah. Thank you both. We will see you soon. Want to learn more? Download our book, Content Transformation. The post Tool selection and the unpredictable variable appeared first on Scriptorium.

May 18, 202624 min

Taming AI: Using AI for content conversion at scale

AI promises to transform content conversion, but what does it actually look like when you’re processing thousands of documents a day? In this episode, Sarah O’Keefe (Scriptorium) and Rich Dominelli (DCL) dig into the real-world challenges of using AI for large-scale structured content conversion. Rich Dominelli: If you have millions of articles and you’re asking the AI, ‘What did we do for this project six months ago?” The AI has to find those articles, pull the relevant information out of those articles, summarize it, and hand it back to you. The best way of doing that is to give extra signals to the AI, structured relevant bits of information, front matter, back matter, publication date, keywords, abstract, that allows the AI to query the corpus and get the relevant chunks out of that corpus in a very quick manner. Then, it can summarize what those chunks are. So the AI almost becomes the user interface over that corpus. But to find that data in the first place, structured content is key. Structured content is key when you’re dealing with big indexes and the web, and it’s the same with AI. Related links: Defeating Nondeterminism in LLM Inference (white paper) Data Conversion Laboratory (DCL) Scriptorium, Machine experience (MX): Making content work for humans and machines (podcast) LinkedIn: Host: Sarah O’Keefe Guest: Rich Dominelli Transcript: Disclaimer: This is a machine-generated transcript with edits. Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction Sarah O’Keefe: Hey everyone, I’m Sarah O’Keefe and I’m here today with Rich Dominelli who is a Senior Developer and Architect at DCL. Rich, welcome. Rich Domineli: Hi, thank you for having me. SO: Glad to have you. We were talking before we hit the record button, and you described yourself as a perhaps hopeful AI evangelist. RD: Yeah, I am well and thoroughly immersed in the AI game at DCL and using it and plus I play with AI assistants at home. I’m enthusiastic about the future of AI, sometimes disappointed about the present. SO: So DCL, as I think many of our listeners know, is focused on conversion at scale, which to me makes a great use case for AI because ultimately conversion is about edge cases and about inconsistency, right? If everything was 100% consistent, conversion would be pretty easy. RD: Yeah, no, DCL does a lot of structured content generation out of unstructured data, and the creativity, especially in the academic space, of what that unstructured data looks like is sometimes nightmarish. So the AI lets us, does a lot of the heavy lifting for us when it comes to looking for particular items, identifying concrete data points within the documents, pulling things like authors and affiliation, front matter type information, and back matter type information out of the documents and in automated fashion. It can be painful from time to time, but it’s definitely helped. SO: Yeah, so this is, think, you know, the reality of working with AI and working with it in a production environment in order to address all these weird edge cases and what’s going on. So tell us a little bit about how you’re using AI in, you know, these conversion use cases. What does it look like to go in there and start applying some of these tools that we have? RD: So, I mean, typically our flows work in a way where we’re coming in with a PDF or a Word document or some other unstructured format. We take it, we reformat it into a version that’s more AI-friendly, like Markdown, for example. And that’s usually the first step we’re doing when we’re looking for information to pull out of it like front matter. It’s a very common use case. If you look at academic papers, the front matter, the authors and the affiliations that are on that paper can be formatted in more ways than I could list out during the course of this podcast. It’s kind of crazy. So what we’ve started doing, and we’ve been doing this for a couple of years now, is we’re using the AI, we’re handing it the Markdown document, and we’re saying we need to list authors and affiliations, please extract it for us.  Now, naively, when we started that process, we assumed that the AI would give us a consistent list of authors and affiliations. And sometimes it does. But every time you do that call, you’ll get it in a different format. So then you have to start tightening things down. So OK, give me a list of authors and affiliations. I want it to be structured exactly like this. And typically, we have a JSON structure that we’re presenting to the AI, along with our prompt, and saying, give it to us. Well, okay, and that gets you a good chunk of the way there. And that was very exciting when we had that working consistently, we were getting things out of the system on a consistent basis. Awesome. But then you start looking at the results, and every once in a while, you get an author that was missed, or there would be too many authors on that paper.  We had one test paper, which I loved, which had 600 collaborative authors in it. And the AI would just choke after about 280-ish. So then you have to start dealing with things like paging through the data and formatting the data. And then you have to figure out, well, did it miss anything? You have 600 authors. Good luck. So now you have to take what the AI did and compare it against your own representation of it and write a program to do that comparison to say, OK, is it good? Is it good? You have to take a step back and you look at it and you say, okay, we have the information that’s in the non-structured format. We’re handing it to the AI. The AI is gonna give us a structured version of it and we need to validate it. Well, the first validation is very easy. Does that structured version match the schema that we gave it? Yes or no, that’s easy. Well, then you have to say, okay, is everybody there? Well, is there anybody added? Because the nice thing about AI is they occasionally get very creative. Even if you have that temperature dial turned all the way down to zero, it will pull names out of thin air and then come back to you with some random name and stick it in the middle of the data where it’s not obvious, of course, and then hand it back to you. So then you have to start saying, are all the names that appear in this list actually in the document? Are the counts matching? And if it’s not, you go back to the AI and you ask it again, and usually you’ll get a better answer the second or sometimes the third or fourth time.  But you need to be able to catch that, especially if you’re doing this at scale, because if you’re doing a few, it’s easy, you can eyeball it. If you’re doing 1,000 of these a day, you can eyeball all of them. You can say, you can ask the AI, OK, give me a confidence level, but if you can’t trust it in the first place about what it’s returning, yeah, I’m very confident about what I’m giving you right now. It’s really the truth, I promise you this time. I don’t know how trustworthy that would be. So you have to write tools to validate what the AI is producing, or you have to use the AI to validate what it’s producing. So coming in the first time, obviously, we did the count, we did the schema validation. We then said, okay, we’re going to check to make sure all the names appear in the document, we’re going to have landmarks in the document that we can refer back to. So if you start with Microsoft Word and you have track changes on, you can have paragraph IDs that are supplied. So you can make sure that you can find all of the authors in that list and they all have a paragraph ID and you can have your landmarks and that’s great. Or you can even hand the results to a separate AI call and say, proofread this. Is this accurate? Is this the best answer that could be for each of these? I know we’ll come back with an answer. And you can use that as a signal to gauge accuracy and to gauge repeatability and make sure it’s correct. SO: So you’re, let’s see, generating an AI, not a test bed, but an AI environment that’s doing this conversion or that’s processing the files for you for conversion. And then you have to go in and do all this validation to make sure that the output that you’re getting is actually correct. As compared to, I’m gonna say old-fashioned, but you know, as compared to scripting, deterministic, pretty straightforward, if A then B kinds of scripting. What are the differences between that and AI-driven conversion in testing and validation? What are the test plans? How are they different conceptually? RD: So from our perspective, the frustrating thing sometimes is the AI is completely non-deterministic.  SO: Mm-hmm. RD: It can give you a name formatted one way today, and then tomorrow, its formatting might be subtly different, where in the paper it has “Richard Dominelli, Junior.” The AI may decide, well, that comma probably shouldn’t be there, or junior should be followed by a period, and it wasn’t in the paper originally. And you can try prompting around that and tell it to prompt around that and make sure that it’s accurate. But it doesn’t always follow your instructions exactly when that’s the case. SO: And why is that? Why is it non-deterministic? RD: Because AIs are built on a neural network, the neural network itself has fuzzy fields within that, mostly due to floating-point arithmetic. So when you’re looking at it and it’s that weight on that particular key might be out to like 16 digits of a number and it might shift it slightly one way or the other. There is a fantastic paper from, I wanna say it’s anthropic, that goes through the different reasons why AIs are non-deterministic. It goes through repeatedly querying for the AI and who Richard Hyman is and getting back a different answer every single time. They’re all correct. However, they’re all slightly different. The other thing that will lean into that is if the AI is being heavily used, the memory and model weights will shift ever so slightly and you’ll get a different result. So you’ll end up having an issue where today I’m getting accurately this way and it’s relatively consistent, not perfectly, but close enough. And then tomorrow, it may just give you a dumpster fire of random information and you need to be able to detect that. Okay, the other challenge we hit fairly early on is more and more people are aggressively using AI right now. So we’re actually starting to hit issues where the LLM providers are overwhelmed. So you have to be able to code in sale over because you’ll literally get too many, you’ll get 429 errors, which are basically, I’m too busy. I can’t deal with your request right now. Call me back. And you’ll have to go back and repeatedly query to get around that. I am hoping at some day in the near future, we’ll be able to have in-house AI at scale and have these wonderful models that are so intelligent that we can run on our local hardware. And so I won’t have to deal with that, but right now, that’s not the case. SO: So given all of this, I mean, I’ve asked you the leading question about the issues and the negatives, but what then makes an AI-driven conversion appealing versus a sort of scripted, deterministic, if I plug in AI, I will always get B output? RD: So part of it is the type of data we’re dealing with. We’re dealing with unstructured information and the unstructured, the creativity of the unstructured information is rather astonishing. You’ll have people format things, know, we’ll get papers in where the entire paper is placed in different cells of the table. It’s not tabular information at all. They just, you know, we wanted this particular section to be in this cell and this particular section to be in this cell and this particular section. And the AI, I don’t want to say is immune to that, but it’s a lot more forgiving than having to write those reg ex or traditional programming or word interrupt things to try to extract that information, because the AI can address it in a much more fuzzy fashion. I know approximately what an author’s name looks like. I know approximately what a reference looks like. Even though today they decided to do it in Comic Sans or with Wingdings fonts, I can still read that and move on. So that’s really the wonderful aspect of it, is it gets around a lot of that fuzzy logic coding. You’re not dealing with having to address each of these nuances in a generic switch or state machine to try to figure out, OK, this paper should be classified this way and this approach used. Instead, the AI does a lot of that heavy lifting for you. SO: Okay, so it gives us that sort of fuzzier, more, I’m gonna say more flexible, I know if that’s exactly the right word. And then the outcome, what you’re describing is you’re ingesting unstructured word, PDF, those kinds of things, and turning them into structured content, presumably fundamentally XML of some sort, but also some other downstream formats. So I wanted to switch gears a little bit. There’s been a lot of conversation about using structured content as an input for AI. So this, guess, is the scenario where you’ve already ingested the unstructured content, have remediated it in various ways. We now have structured content, and we’re gonna take that and feed it into, I guess, AI part two, right? So we’re past conversion. And there’s a lot of people saying, you should feed structured content into AI, it will make the AI better. And so my question for you is, you know, is that the case, and also maybe why and what goes into structured content that makes it produce better AI outcomes, potentially, assuming that it does. RD: So there’s a bunch of guides out there. There are two pieces of conversation. First, there’s a bunch of guides out there for prompting AIs where they suggest using XML or simplified XML tagging to give the AI signals about your prompt that aren’t verbally expressible. So here is my question. Here is an example. Here’s how I want my output to look like. And you can put tags around that when you’re actually prompting the AI and the AI will know that those signals mean that it should pay attention to it. Okay, so that putting that aside, what I think you’re really asking though, is how does structured content, structured documents, the JATs and the DITAs and the S1000Ds and how does that help the world of AI? And to answer that question, we have to go through two things. One, we have to go through retrieval augmented generation and context rot. So let’s talk about context rot first, because that’s a really interesting topic and people don’t talk about it enough. You have these large language models that are coming out right now and they’re advertising this sticker shock value of, can ingest a million tokens and it has this tremendous memory so you can stick the entire encyclopedia botanica in it, and it will be able to ingest it and regurgitate it. There’s a whole lot of academic work out there that basically says that, hold on a second, practically speaking, once you exceed a certain size, even though they can technically hold that million tokens of data in memory, they’re not gonna be answering as accurately as a smaller model. The most common example or the most easy test for that is needle in the haystack test, where you take a document, you stick a random fact in the middle of it, and you hand the AI the document, and then you ask them for that random fact. Nine times out of 10, it will answer incorrectly. An even easier test is there’s a website which I actually like called A Thousand Names. And all this website is is a thousand randomly generated human names. The thousand randomly generated human names. You take that, you give it to the AI, say, how many names are there? And more often than not, you’ll get, well, when you do 100, you’ll get an accurate answer. 200, accurate answer. 300, things start to break down. You might get 300, or you might get 280, 320. You might get a random answer.  And then it gets progressively worse as it gets bigger and bigger. So if you’re working in the context world, content world, you’re looking at ingesting documents into a corpus of some sort. You’re making these structured documents in such a way for the sole purpose of making them retrievable. You want the AI to be able to retrieve those documents and the relevant documents from the corpus so that I can answer the question. A, because your corpus is probably bigger than that million tokens. And B, because the less data you send the AI, the more accurate the answer is. So the better way of thinking. SO: And so a token is roughly a character, right? RD: No, a token is actually roughly a word. It’s less than a word. It’s kind of a lot, but it’s still not like a PubMed-sized corpus or anything like that. It’s roughly the size of the New and Old Testament of the Bible, roughly a million words. So just give everybody that mental picture. But that’s just one book.  SO: Roughly a word. So a million tokens is kind of a lot. It’s a lot of words. RD: So if you have millions of articles, or and you’re asking the AI, you know, what did we do for this project six months ago that involved JAPs in this solution? And the AI has to say, okay, it has to find those articles, and then it has to find the relevant information out of those articles to be able to summarize it and hand it back to you. And the best way of doing that, and the best way we know how to do that is to giving extra signals to the AI, giving those structured relevant bits of information, front matter, back matter, publication date, keywords, abstract, that allows the AI to query the corpus and get the relevant chunks out of that corpus in a very quick manner. And then summarize what those chunks are. So the AI almost becomes the user interface over that corpus, because it’s going to summarize the data. But to find that data in the first place, structured content is key because for the same reason, structured content is key when you’re dealing with big indexes and web, same with AI. SO: So then structured content is potentially helpful. And I guess then circling back, let’s say I’m sitting on a pile of content of varying degrees of structured or unstructured, varying degrees of quality or lack thereof. What kinds of things should be happening before that content gets ingested into some sort of an LLM or some sort of a corpus to be used in AI-generated outputs? RD: So these are the same type of things you would do to make them easily retrievable ahead of time. So the standard approach that was being espoused about two years ago, a year and half ago, was something called Naive RAG. You can just take your PDFs and throw them at the AI, and the AI will ingest them into a vector database, and it will do semantic similarity and find the documents that you care about, not the best approach when you start talking about large amounts of documents. And there are issues with semantic similarities, where the AI will have a hard time distinguishing negative cases, will have a hard time peeling out the best documents, and that type of thing. So the best approach to take is you want to take those documents, you want to turn them into structured information in such a way that it’s easy for the AI to ingest. So typically that involves chunking it, into topic-level pieces or semantic chunking, coming up with summaries to make them easy for the AI to find, and whatever other information you may want to chase out of those.  So, for example, if I’m handing a PDF to an AI and saying, I want to be able to search this PDF later, well, six months from now, if I get a new version of that PDF and I want to search it, search against the two of them, I really want my answers coming out of the second PDF. That’s metadata, that’s structured information that doesn’t appear in the text of the PDF or may not appear in the text of the PDF. You wanna be able to do things like versioning, you wanna be able to do things like dates, you wanna be able to give these signals to the AI to be able to pull that information back quickly. And that’s really where structured content comes in. So for the purposes of preparing your own corpus, you want to convert them into an easy to ingest format, which typically means Markdown or XML or something that the AI can deal with. You want to give it whatever other signals you can so that it’s easy to find. And then you want to hand it to something that first does chunking and then text embedding, which is basically turning the information into numbers so that you can do those cosine similarity searches. And then you want everything handed off to some kind of object store like a hybrid brand database or the hybrid factor database or graph database so that they’re easy to pull out. SO: Awesome. So you started this off talking about being the hopeful evangelist, and now having gone through all of this, it sounds as though you’re really thinking about these issues and dealing with them at scale. What are some of the top things that you’re thinking about going forward, whether hopeful or not, the good, the bad, and the ugly? RD: So one of the interesting aspects of my job is I get to do a lot of interactions with AI from an R &D perspective and do some in-house programming and do some in-house tool use. And what we’re finding is developing our own internal mechanisms for AI to call third-party tools, to be able to call Crossref or Grovid or some of these reference facilities out there through like model context protocol or through API calls so we can execute those calls and get that information back and do validation before it hands back the results is a very interesting topic for us because that would let us do things like any AI have it do the first few rounds of validation before it ever comes back to us without having it go to the next step, do a validation step and then the next step and then possibly do a round trip. It would be a much faster interaction. We use right now, of course, like most of the world, we’re using a lot of AI coding tools to tighten up our code bases to make sure things are working well, to basically act as a force multiplier when we’re doing development on projects, which is phenomenal. I can’t say enough good things about Cloud Code, you know, because it’s really become an essential tool in my day-to-day life. But I’m also seeing a lot of people out there using these tools to help analyze their own and improve their own workflow and that day-to-day work. We talked with one of our customers recently, and they use cloud code, even though the person giving the demo was not a developer; they use cloud code to answer the RFP. And Cloud Code does a tool use call against their document corpus, answers the RFP correctly, and what used to take two or three days of slogging through documents and finding things are now being done in an hour by one person instead of having multiple people working on this project. So it’s great to start seeing that type of stuff in the enterprise just blossom because it’s really exciting. SO: Well, Rich, I really appreciate your insights on this. I learned a few things and I think that it’s great to hear from people who are actually using this stuff, you know, in a production world, in a high stakes world where you’re actually, you know, need to get the content right, get the information right as opposed to just, you know, that we’ll play around with it and not worry about it too much. So thank you, and we’ll look forward to hearing more from you and what you’re doing at DCL. RD: Sounds great. Thanks for having me. Conclusion with ambient background music CC: Thank you for listening to Content Operations by Scriptorium. For more information, visit Scriptorium.com or check the show notes for relevant links. Want to learn more? Download our book, Content Transformation. The post Taming AI: Using AI for content conversion at scale appeared first on Scriptorium.

May 4, 202619 min

Machine experience (MX): Making content work for humans and machines

Your website may look great to humans, but can machines understand it? In this episode, Sarah O’Keefe (Scriptorium) and Tom Cranstoun (Digital Domain Technologies) explore the emerging discipline of machine experience (MX). Sarah and Tom discuss what AI agents actually encounter when they visit your web pages, why microdata and metadata are critical, and what content creators must do to ensure content is consumable for both human and machine audiences. Tom Cranstoun: Humans are looking for pictures, they’re looking for text, and they can infer. You may think, “Well, we’ve already added information on the page,” but by putting it in as microdata, it doesn’t appear on the page for the humans. It appears on the page for the machine. I think that that’s a critical distinction. We are trying to design for both. We don’t want to overload a human with information, but we do want to give the machine as much information as it can take. Related links: The Gathering Digital Domain Technologies MX books The Scriptorium Content Ops manifesto LinkedIn: Host: Sarah O’Keefe Guest: Tom Cranstoun Transcript: Disclaimer: This is a machine-generated transcript with edits. Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction Sarah O’Keefe: Hey, everyone. I’m Sarah O’Keefe. Today, our guest is Tom Cranstoun, who is founder of a machine experience, or MX community, called The Gathering. He has a couple of books on MX and is currently a consultant operating as Digital Domain Technologies. Tom, after 53 years in the business, some experience with AEM at very, very large companies, including a huge project at Nissan, has turned his attention to the question of how machines, which is to say AI agents, interoperate with the current public-facing web. And so today, Tom, I’m delighted to have you on to talk with you about machine experience, or MX, and what this all means as we move forward in this brave new AI world. So welcome. Tom Cranstoun: Thank you, Sarah. I’m very pleased to be with you today. SO: I am delighted to have you. So I guess we’ll start with the extreme basics here, which is what is machine experience, or MX? TC: Yeah. MX, well, to my definition, machine experience is like user experience, but it’s for machines. Machines cannot ask a friend for help if something goes wrong when they’re browsing a website. They can’t turn to a partner and say, “What do you think this means?” They can’t retry a failing form input because they will just go through the same mechanical patterns to try and carry on throughout the web journeys. Therefore, machine experience is thinking about what elements one must put on a webpage to help a machine understand and action the final goal of the webpage, whether that be a CTA that lets you purchase something, or an information document that lets you know about a government policy, or a charity good, whatever the author of the page is trying to get across to the audience. SO: And so at a high level, what does it look like to build out machine experience? What are some examples of things that you need to put onto a webpage to accommodate the machine that’s reading it? TC: Well, the very first level is the disabilities angle, things like the Americans with Disabilities Act, that kind of WCAG, W-C-A-G, the accessibility work. The more accessibility information is on the page, the more the machine can understand the background of the page. So machine experience and accessibility are pretty much at the top level, the same sort of thing. If you put in JSON-LD, microdata, and you enrich your pages with the things that Americans with Disabilities Act would like, you’re actually helping a machine understand the page. So that is the top-level constraint. When you go below that level, you need to give the machine lots of information about your product, not just the thing that a human wants when it’s glancing at the page now, and as you go through the journeys, things will be added on. Humans can only take in two or three items at a time, so we design pages to reveal what is happening. You go to a catalog, to a product, to a variation, to a purchase, four different steps. Each step introduces different pricing and concepts. It’s best to feed the machine on the page that the machine lands on with all of the information that it needs. This may not necessarily be surfaced to the human reading the page, but it’s there for the machine. This helps the machine when it arrives at your webpage. SO: So I’m really enjoying this concept that a properly organized page with proper accessibility WCAG or ADA compliance and support then results in the machine being better able to parse the page for essentially the same reason, right? It’s properly structured, it’s predictable. The things that are labeled are labeled correctly. I don’t know that we should be driving accessibility in order to enable AI, but on the other hand, if it gets us more accessible pages, then let’s certainly do that. Can you give some examples of what happens when pages are not machine-compatible? What are the kinds of problems that people run… Or not people. What are the kinds of problems that the AIs run into when they try to parse a page that has not been labeled properly or encoded properly? TC: Yeah, I collect these examples from real life. Whenever I use the web as a normal person, I say, “Well, how would a machine interpret this?” Recently, I was looking for a holiday, and I asked an LLM to give me a list of five companies that offer cruises up the Mekong Delta. The machine came back with one offer at $200,000 for a week’s holiday, and the rest of them were $2,000 for a week’s holiday. What had happened there was that the machine had found a European website. Now, the Europeans changed the comma and the dot in monetary labels differently from what the Anglo-Americans do. We use a comma separator between thousands and a full stop between fractions. The Europeans actually put the full stop as the thousand separator and a comma between the fractions. This meant that when the LLM built a table of prices for holidays, it didn’t understand the distinction, and it tripped up. The agent hadn’t been instructed to compare prices and make sure that they were all within the same range and were reasonable. It just produced them as a matter of a fact. “Here’s a holiday for you. One of them is $200,000. The rest of them are 2,000.” There was no knowledge, no information that could tell the agents what was happening. If those pages had been decorated with currency and they had microdata with the… microdata always says that you should use commas as a separator and full stops as the fractional separator. If these things had been in the page, the machine wouldn’t have flipped up. Now, a human could have read a page and seen the locale values shown on the page, and both people would be able to understand what was going on. So that’s a typical trip-up from an undecorated page. SO: And so essentially, the presentational component that says, because I’m serving this page to somebody in, for example, Germany, they are expecting a comma separator between the full Euro amount and the cents, the Euro cents. But that comma is essentially formatting, as opposed to data, and so here we are. TC: Yes, correct. And the microdata has got the thing in a proper machine-readable way. The other things that we always get problems with in the world are English and American date formats. We swap the month and year around when doing short form. The machine-readable version uses ISO dates, and ISO dates put in as a microdata tells the machine categorically. It doesn’t matter what the locale is, this is the date and time. SO: Yeah. And so as the expression of the date, whether April 1st is 1-4 or 4-1 is essentially a formatting problem. TC: Correct. And these are not visibility problems. These are machine experience problems. So it’s layering up. You start with fixing the disability by doing machine experience, and then you fix the locality and the community values, the human factors, display factors. SO: And so I think we’re all familiar with the concept of a customer journey, but you’re now talking about a machine or an MX journey. What does that look like? I mean, how is the machine processing of a website? How do you explore that journey and what it looks like? TC: The machines will not discover your website, come in through your landing page, and then look for offers or products. A machine will have an idea of where it wants to go and will land straight in at a page. It will arrive five pages into your journey, and read the webpage as it is. The owner of the website has lost all of the signals about what the dwell time was on each page, how’s the reader arrived at the end location. Did they go sideways and look at other things? Those things don’t happen with machines. They go straight in, see if they can get what they can. If they can get what they can, they will action it. If they can’t, they will move on, and go to another page or another person’s website and do exactly the same to them. So when a machine arrives at your webpage, it will not be giving you any referral details. It will not tell you what the journey it is, and it won’t tell you what else it’s interested in. You’ll just get a cold caller who will arrive and disappear. I call them invisible users. They’re invisible to your analytics, they’re invisible to your tracking, and they’re invisible to your future. You cannot tickle them and say, “Hey, you left something in the basket.” You cannot use those parts of the journey. A machine comes in and goes, gets what it wants or it doesn’t. So you must give it, front load it as much information as possible on any and every page that a machine may land on.  SO: So then coming at this from the perspective of structured content people, because a lot of what you’re talking about, I mean, is web experience, like how does what we view as the end state result of the content that we’re creating. So if I have an enormous DITA CCMS full of stuff and then I output it to some semblance of a website, your focus is on what needs to be on that website so that it is describing itself in such a way that the machine, that an AI or a crawler can go in there and pick up what it needs to and process it accurately and not offer you a vacation for $200,000. I assume you did not pick that one. So what are the opportunities? When you look at MX and then also DITA as a backend, what kinds of opportunities do you see there to map those things across and take advantage of some of the structure that perhaps is already in the XML and/or structured content systems? TC: Yeah, I see the backend is full of good content operation stuff. Everybody has got details about pricing and dates and frequency, and there’s lots of backend information, which often doesn’t make it into the front end for people. Humans are looking for pictures, and they’re looking for text, and they can infer. They can infer if two prices are on a page and it says, “Was $200, Now $180.” A human understands that. A machine, well, depends on the quality of the machine, whether it can read and infer those things. So the backend information has to be made more visible and in a redundant manner. You may think, well, we’ve done this on the page before. We’re doing this on the page after. But by putting it in as microdata, it doesn’t appear on the page for the humans, but it appears on the page for the machine. And I think that that’s a critical distinction. We are trying to design for both. We don’t want to overload a human with information, but we do want to give the machine as much information as it can take. We don’t necessarily have to surface all of the information within the page, but we have to carry it with the page. So a page taken in isolation contains the entire story, not just the fraction that a human is looking at, which does mean that a lot more pushing off the backend from data to the front end. And some people will think that’s a waste of time, but I don’t think so. I think giving that extra material to the machine is what makes the journey successful for the machine. SO: I’ve been to a lot of conferences in the past couple of weeks, and the conversation around what is needed for successful LLM processing, or crawling, or ingestion, or agents for that matter and what is already provided in a metadata-rich structured content system is sort of, well, we have all of this. Now, what do we do with it, and where do we put it, and how do we make sure that this all works? So it seems like this discussion around machine experience is going to help to maybe close that gap and connect the pieces such that we can do this successfully. And so as we move into this, I know that you have some material out there, but also a community. Can you talk a little bit about the MX community, and what you’re looking for there, and what it’s called? We will put all of the links in the show notes. But what does it look like to participate in that community, and what sort of participants are you looking for? TC: Yeah, we are looking for content creators. We are looking for business owners. We are looking for technical writers. It’s called The Gathering, gathering being a Scottish term for the gathering of the clans. We all get together to do something that’s good for the combined grouping. And then after we’ve created whatever we’re going to create, we go away and do our own things. Now The Gathering is tg.community. That’s https//tg.community. We are building a set of community-led standards to try and make it easier for machines to understand documents. The Gathering is not just interested in HTML. We’re talking about documents of all types, and we’re talking about keeping the metadata that you have in the backend of the content creation systems, whether that be data or other content creation systems, and passing it through into the end documents. You have metadata in PowerPoint slides. You have metadata in Word documents. You have metadata in JPEGs. These, too, deserve the machine experience. If you can tell the machine details about an image inside a JPEG, then the machine doesn’t have to try and scan and interpret the image to find out what it is. It makes things so much better. And The Gathering is a community that is trying to build these as open community-led standards. One of the first things that I am proposing for the community, which was just launched on the 2nd of April, 2026, by the way, it’s very young, and we hope to build at the speed of LLMs. We need to work fast. The key point and the key thing that helps LLMs understand your website, there’s a thing called llms.txt, which people don’t really understand and machines don’t really use. It’s a standard for describing your website in a way that a machine can help to understand, know what’s going on without reading your site map. It is not used by the machines because, one, it’s not served as HTML, and, two, it’s not in your site map. Therefore, the crawlers that build your training material do not pick it up and do not ingest it. I am suggesting, and I have it in my books, I talk about this, if you wrap the llms.txt in HTML and serve it as HTML and put it in your site map, then you will get a better response from the training stage and from the inference stage. So you are seeding the machines with the information about your website, something that is currently missing from the world, and that’s step one. There are five steps that you’ve got to go through before you can do a successful e-commerce position. And that is feed the machine, get noticed, be descriptive, be MX-aware and be citable, and MX lets all of those things happen. SO: Perfect. Well, Tom, I know that there’s a lot to discuss here, and we could go on for a very long time, but I hope this gives people a little bit of an introduction to this idea and an opportunity, if they’re interested to reach out to you and to the community that you have. And there’s also a book or three. Any closing thoughts that you want to pass on before we close this out? TC: My personal opinion is that I think that we should treat the machines as first-class citizens and not block them from our content and to create content that works for them. The more that we do for them, the more they will do for us. And if we start treating them as an afterthought, it’s not going to be such a good web as we could build. SO: Okay. Well, thank you so much. I’m glad we had an opportunity to talk. And we will, again, put the links to the various resources that Tom mentioned, including the community. There’s some RFC, some standards drafts and a manifesto and a book. We will put all of that in the show notes. So Tom, thank you again for being here, and I look forward to hearing more on this effort. TC: Thank you very much, Sarah. Conclusion with ambient background music CC: Thank you for listening to Content Operations by Scriptorium. For more information, visit Scriptorium.com or check the show notes for relevant links. Want to learn more? Download our book, Content Transformation. The post Machine experience (MX): Making content work for humans and machines appeared first on Scriptorium.

April 27, 202622 min

Make the move successful: Replatforming content ops

Replatforming your content operations isn’t just about swapping systems. In this episode, Alan Pringle and Bill Swallow share what organizations must consider to successfully replatform. From navigating technical debt, system integration, and the people caught in the middle, they discuss change management, technical debt, and why your exit strategy should be part of the plan from day one. Software isn’t forever. Systems come, systems go, they get improved. Your requirements are ever changing with the content that you need to manage. Not thinking about your next jump is really to your detriment. — Bill Swallow Related links: Replatforming structured content Your tech expertise + our CCMS knowledge = replatforming success (case study) Cutting technical debt with replatforming (podcast) Replatforming with localization in mind LinkedIn: Alan Pringle Bill Swallow Transcript: Disclaimer: This is a machine-generated transcript with edits. Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction Alan Pringle: Hey everybody, I am Alan Pringle, and today I want to talk with Bill Swallow about content operations and replatforming. Hey Bill, how are you?  Bill Swallow: Good, how are you doing? AP: Good. So I guess we should start this by saying the reason why we want to talk about replatforming is really we have done a few replatforming projects. We’ve had some prospects reach out who are interested in doing it. So I guess we need to explain what it is and some of the things you have to think about when you’re going through the process. So if you would not mind, would you define what we mean by replatforming content operations? BS: Sure. So generally when I talk about replatforming, it’s in the context of a company having one system in place and maybe it’s time has come and they need to move into a new one. So it’s the entire process of determining what type of system you’re going to need, what your requirements are for that and being able to lift everything up from the old system that you want to carry forward and put it in the new system, configuring it and what have you to get it to work going forward. AP: So we’re not talking about using a whole new technology or a whole new platform. It’s shifting to a similar platform for some of the reasons that you just mentioned. And I think that’s another thing. There are several reasons why a company might want to do this. And I know our clients have had various reasons for doing this. Let’s focus on those for a little bit. One of them, I know you kind of already touched on this. Sometimes you just outgrow a system. It just… that’s how it is. So let’s start with that kind of, it’s not sustainable anymore because you’re bigger, too big now for what that system can do. BS: Sure, either you’ve outgrown it or it’s approaching end-of-life or it’s just not meeting the needs that you had five or ten years ago when you bought the system. So there are a lot of different factors there, but basically it comes down to what are your requirements and is it meeting your requirements?  AP: Right. BS: Are you able to get the things done that you need to do given the fact that, you know, the world is quite different now than it was five or 10 years ago. AP: Exactly, and there’s another angle here too that I think we need to briefly mention is that sometimes you’re gonna have two sets of requirements because two companies can merge or there can be an acquisition and then all of a sudden you’ve got two content operations platforms that are pretty much doing the same exact thing and I guarantee you the IT department is not gonna have that.  BS: Absolutely. AP: So there could be a situation where you’ve got two, and one of those is going to go away. And in some cases, and we should talk about this too, it’s not necessarily about picking one. It’s not uncommon to go to a whole other one. So there is quote, “No loser.” That’s also an option. BS: That’s very common because usually in the case of a merger, you have two established groups with two established systems that may be starting to age out on both sides. And it doesn’t make sense to spend the time and the effort to move one group into the other system when that system is probably going to be replaced in a few years anyway. AP: Yep. So basically, the circumstances of the merger have provided a perfect opportunity to do something that’s painful. Replatforming is not magical. There’s still going to be technical hiccups and everything else. But at least it’s not as painful because you’re both moving out of systems that maybe aren’t optimal into something that will basically treat your content creators and all the people managing content much better because it’s going to support their needs better. BS: Or at least everyone goes through the same pain together of learning a new system. It’s team building. AP: And so you bond through shared pain experiences. Exactly. All right. Yeah. huh. We’ll go with that. We will go with that. There’s some other aspects here too that are kind of related to that. And that are the idea that things, because things are getting maybe rickety, that things are getting too expensive to maintain. You keep making these little tweaks and changes in things that become less and less repeatable. And that adds up. That’s time and money that you have invested. BS: It certainly does. Aside from the hard costs of licensing and just the general time to use the system, produce things, you do have, after you’ve used the system for so long, you’ve got your workarounds built in and they may not be a best practice and the workaround may solve the problem on its face, but you’re doing a lot of things that you really shouldn’t be doing with that, you know, with that workaround in place. You really should be doing something that’s a little bit more streamlined. And, you know, as you’re bringing new people in and new groups in, whether it’s a merger or whether it’s just another department that realizes that, Hey, you know, they have this, you know, shiny system over here. Why don’t we start using it too? If you have workarounds in place, it takes a lot longer to get people up and running in a new system because not only do they have to learn the system, but they have to learn how you’ve worked around it. AP: Right, and that’s where you start talking about technical debt because all of those workarounds that you’re describing, that equates to technical debt. And one day, you’re gonna get your backside handed to you because you have all of this technical debt. And replatforming is the perfect time to press that reset button and say, we’re gonna get rid of those things. We’re gonna have a system that addresses those problems in an official, correct way, and none of these weird workarounds.  BS: Mm-hmm. AP: And by the way, those workarounds, what if the person who did them leaves and hasn’t been documented well?  BS: Yeah, that’s a big problem. My guess is that if you’ve been using one particular system for, you know, 10 years or even more, you probably have a lot of content just sitting there that hasn’t been touched in years, hasn’t been needed in years, but it’s still sitting in the system. And it’s still coming up in search results as people looking to, you know, find topics that they need to edit for a new release. And it’s just getting in the way. It’s a good time to, you know, cut clean and, you know, ditch all of that old content that you no longer, that you know, you no longer need and focus on, you know, what you need to produce going forward. AP: Yeah, I think maybe sometime on the show Hoarders, they can do episodes on content people and their technical debt, and basically just hoarding all of this content in various digital forms all over that nobody’s actually looked at. I’m sure we would all laugh and be horrified at the same time by such a show.  BS: It wouldn’t make for exciting TV, though. AP: Yeah. So let’s talk about the overall process for how this kind of works. We’ve talked about what it is, a lot of the reasons for it. So let’s talk about how to do it. And it’s not a one-size-fits-all thing. We can tell you about our experiences that we’ve had. But I know one of the first things you got to do, for example, is choose your new system where you’re going to be moving into. BS: Right. And out of the gate, the knee-jerk reaction is to go with something new and shiny, but you really need to sit back and figure out what it is you need that system to be able to do and how you need that system to be able to do it. you know, we’ve had a lot of clients who’ve come to us after setting up a system, maybe two or three years prior, who just are like, this is just not working for us. And as we talk with them, we realized that, you know, they, essentially, you know, had a square peg and they bought a round hole to put it in. And here they are three years later, still trying to force that peg into the hole. So you need to sit back and really think about your requirements and not the requirements that you have today, but the requirements that you have today and anticipate having at least five years down the road. You have to leave yourself open because otherwise your opportunity for growth in that system is limited by your choice, and I hate and we always say, know choose tools last. Same thing goes for the systems. The reason why we’re talking about it up front is that you do have an existing system. You do at least need to identify some candidate systems that you’re going to be moving into and have clear requirements for those systems and why you want to look at them further before deciding on the one you’re going to implement. AP: And those requirements can help you identify the differentiators, the things that make one system a better fit for your needs. And the more fine-tuned and discrete your requirements are, the easier time you’re going to have finding that match for the new platform that you need to address all of those requirements.  BS: Mm-hmm. AP: Another part of this is moving the content from the old system to the new. So let’s talk about content migration, because that, a lot of times, I think people underestimate what that can take, even when you’re talking about basically two very similar technology stacks. BS: That’s the easy part. Content is content, Alan. It’s just all just words. It’s fine. You can move it. No problem. Yeah, I think this is the most overlooked piece of all of it. Even if you’re moving from one system that uses the same format for the content under the hood to another system, you’re still going to have to make changes.  AP: We can take this outside later. Yeah. Yeah. Yeah. BS: The old system maybe had a couple bells or whistles that handled things a very specific way. And the new system has a couple of other ones and they don’t match. So you’re going to have to find a way of mapping from, you know, content type A1 to content type A2. Even though they’re both content type A, you still have these little differences that you need to map out. AP: Right, because what may have been best practice in your existing system may have required a custom thing that that system does that system B does not do. So you’ve got to find the equivalent of what that custom thing is. We’ve run up against that quite a few times and it’s not that uncommon, but you’re right. It’s rarely a one-to-one situation, unfortunately, even if the underlying foundation or structured standard you’re using data in particular is the same thing. So yeah. BS: Mm-hmm. Yeah, DITA in particular is interesting because you would think that all systems would play with a documentation standard in the same way because it is a standard. It’s not the case. There are different efficiencies that the systems bring that come with some modification. And it’s not to the standard itself, but how it interacts with it. And it may do things like replace linking with, you know, linking via file name with, you know, linking with a UID, a unique identifier. And that unique identifier is going to make perfect sense in that system that you have now, but it’s going to make absolutely no sense once you move it to another system. So you have to find some way of converting it over.  AP: Exactly. BS: That being said, that’s the best case scenario that you have two systems that use the same underlying content technology, and you just need to map a few things differently. There are other cases where you have a completely different approach to content from system A to system B. One might use XHTML or might use something else, might use RTF, who knows? And then you move to another system that uses XML or uses Markdown or what have you. But that is a bigger lift and shift where you suddenly have to remap and convert everything to a new format. AP: And that’s really the distinction between moving to a whole new system and replatforming. What you just described there is really going to a whole new tool environment, a new process. Whereas what we’re talking about more is where you’re basically using a tool in the same area or a competitor of the tool you’ve got now.  BS: Mm-hmm. AP: And it’s just moving things over and fixing those little custom things that aren’t going to work in your new system. So yeah, there are all these levels here. And I think one thing we really need to communicate here, even when you’re replatforming from one tool that’s very similar to the new one you need, there is still work to be done there. It’s rarely just a very clean cut, lift and shift. And a good example of that is the publishing pipelines, because tools in this area have slightly different ways for publishing and getting your content out into the world. BS: They do. And even if you’re using the same, you know, the same publishing pipelines that you’re able to somehow lift them up from one and drop them in another, because of the changes in how the system handles the source content, you’re still going to make, need to make modifications to those publishing pipelines later because, you know, like my example with links, because they’re going to work differently in another system. You need to tweak the output generators to handle those links appropriately. AP: And another example of that is when your content system integrates with other systems, the way that, for example, your content system integrates with a workflow management system, it may be different with the other system, or your product lifecycle software, that can also have to be hooked up differently. Or who knows, maybe you’re changing it all together. So you also, beyond just looking at the publishing pipelines, look at how other systems are integrated in with your content development system. BS: Right. It’s not a matter of just, you know, unplugging all the wires and plugging them back into the new box that you bought. I mean, it’s very different in some cases, you know, one may have a built-in API, another system, you know, it might have no handling, and you have to build an API to now, you know, talk to whatever your portal, your workflow management, your digital asset management system, what have you. It’s usually never clean cut. You can never just unplug those wires and plug them back in. And yes, there are no wires involved usually. AP: Yeah, well, and by extension of that, like you can’t just, you know, unplug and replug, you also have to think about people and how they have used that system to create and manage the content. You’ve got to kind of help them understand the differences and basically help them remap and reprogram their brains to understand, okay, you did it this way in system A, you’re going to have to do it. This way in system B, it’s a little bit different. So you still have the training and change management requirements. Again, it is not lift and shift, and that goes for people’s brains. It’s not gonna work like that. It’s just not. BS: Mm-hmm. No, no. And another thing that a lot of people tend to gloss over is the amount of testing that’s required once you get the system stood up. You have to make sure that all the content is valid in the new system, that it’s running and behaving properly, that you’re able to publish outputs, find where things might be dropping out as you publish content and fix those. So it’s never going to be a very straightforward project. AP: Yes. I agree, and I think this is a good time to like offer up some closing tips on things to think about, and I know one of them is this is going to take longer than you think. BS: Yeah, yeah. I’ve, I’ve warned people to budget at least six months, and that doesn’t mean about six months. That means at least six months and expect it to take longer. Even if it’s a, even if it’s as close to a lift and shift as you can get, it’s going to take time. and some of the reasons for that are not only is the system going to be different and you have to stress test it and make sure that it’s, it’s going to work in a live, you know, working environment, but remember that you also have competing demands at work as well. So that you can’t have your entire team just stop what they’re doing for six months or even pull it into three months and say, we’re going to stop everything, not do any production work at all. And we’re going to focus on just standing up this new system. You really can’t do that. AP: That never happens because there is no such vacuum on this planet in the business world, right? BS: No, you can’t stop. You cannot stop the production machine. You need to keep going. Aside from all of your daily job requirements, you now have the additional requirement of setting up a system. trying to shortchange yourself with a short timeline is not going to… First of all, it’s unrealistic. Second of all, it’s not going to gain you anything. If anything, you’re going to implement things incorrectly, you’re going to start out of the gate with workarounds in the new system and it just, ends poorly. AP: And sometimes you’re going to have to keep that legacy system running. You’re going to have parallel systems because it’s a CYA is what that is. Just in case something goes sideways with the new one, you still have your old process and can use it to deliver content that has got to hit some particular deadline, for example. BS: Right. You’ve got to keep things moving. You can’t just stop work. But, you know, all that being said, the number one thing you need to do when you’re thinking about any type of a shift in technology like that is to take advantage of the changes that you’re going to be making. You know, if there are, you know, if there is content that you don’t think you’re going to need going forward, move it to the side. You might be able to move it in later. You know, don’t have to necessarily delete it, but don’t bring it into the system unless you know you’re going to need it. It’s a great time to do some spring cleaning on your existing content database. Move in the stuff that you know you absolutely are going to use, and then slowly start bringing in other stuff or not, if you end up not needing it. AP: And then do a hoarder intervention, because you may need it. And that kind of brings up one of the last points I want to talk about. Having an outside perspective, yes, like us, come in and help you kind of think through this, that can also be helpful. And really, I think the last point I want to make is, on the edges of this discussion, is really, you always have to have an exit strategy, even when you’re going into a new tool. It really will benefit you to do something that seems so counterintuitive and to think about, what are we going to do if this tool goes away while you’re implementing the new tool? Because the fact you’re doing a replatforming already tells you that exiting is a reality, and sometimes you’ve got to do it for all the reasons we just outlined. BS: Mm-hmm. AP: So always be thinking about how are we gonna get out of here if we have to? That’s something that a lot of people in the heat of trying to get something new stood up, they really don’t think about. BS: Mm-hmm. Yeah, software isn’t forever. Systems come, systems go, they get improved. And your requirements are ever changing with the content that you need to manage. So not thinking about your next jump is really to your detriment. AP: And on that most excellent note, I will thank you, Bill, and we will end this here. Thanks. BS: Thanks. Conclusion with ambient background music CC: Thank you for listening to Content Operations by Scriptorium. For more information, visit Scriptorium.com or check the show notes for relevant links. Want to learn more about replatforming? Download our book, Content Transformation!   The post Make the move successful: Replatforming content ops appeared first on Scriptorium.

March 30, 202640 min

Who controls your content? AI and content governance

What does it actually mean to govern your content in the age of AI, and who’s really in control? In this episode, Sarah O’Keefe sits down with Patrick Bosek, CEO of Heretto, to unpack why the quality, accuracy, and structure of your content may be the most critical factors in what your users experience on the other side of an AI model. Patrick Bosek: In today’s world, you don’t have 100% control. There are a couple of different places where this needs to be broken up. One is the end user: what they physically get and what control they have versus what control you have. Then, there’s what control you have of how the AI model is going to behave based on your information and your inputs. Whether or that model is public, like a user accessing your documentation through Claude Desktop, or private, like a user accessing your documentation through your app or website, the governance piece comes down to what control you have immediately before the model. And that breaks down into a couple of things: completeness, accuracy, and structure of the content. Related links: AI and content: Avoiding disaster AI and accountability Structured content: a backbone for AI success Heretto Questions for Sarah and Patrick? Register for the Ask Me Anything session on April 8th at 11 am Eastern. LinkedIn: Sarah O’Keefe Patrick Bosek Transcript: This is a machine-generated transcript with edits. Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction Sarah O’Keefe: Hey everyone, I’m Sarah O’Keefe. I’m here today with Patrick Bosek, who is the CEO of Heretto. Hey Patrick. Patrick Bosek: Hey, Sarah. Long time no chat. SO: That is, I guess for certain values of long time. We decided today that we wanted to talk about AI and governance, except I promptly tried to come up with a synonym for governance because I’m afraid that when I say that particular word, our audience just walks off. So, okay, Patrick, what is governance? PB: Well, so first of all, thanks for having me on, and second of all, I’m excited about this one because based on our little bit of chat before the show, it sounds like we’re actually gonna have some things to argue about this time around.  SO: I would never. PB: Well, usually we tend to agree right like I think that we’re generally pretty on the same page about stuff. So I’m excited. I’m pumped. Okay, so governance. I mean, obviously it has a ton of different meanings to different people but in the way that I want to talk about it today, because it was my suggestion. It’s related to the governance of content, specifically in the way of the inputs to AI systems. So you can think about the process of controlling for quality, accuracy, the things that matter in the actual content and information before it gets into the AI system. So it’s kind of the upstream quality, totality, structure, all of that checking and assurance ahead of whatever your experience is going to be downstream, of which one is the most contemporary and most interesting is AI. SO: Okay, so this is making sure that it is not garbage in so as to avoid garbage out.  PB: Yeah, I would say that’s a fair statement. SO: Yeah. Okay. And can we use AI to do governance of the content we’re producing? PB: Well, that’s actually a very interesting question. And I think the short answer is somewhat right now. So before I go, okay, before I like fully answer that, I want to put a little disclaimer in here. The stuff with AI is changing so quickly that we should date-stamp this episode. SO: It is March 19th, 2026. And it’s nine-ish Eastern time. PB: Yeah, we are recording this on March 19th, 2026. Now I feel, yeah. Okay, so now that people know when it is that we’re talking about this, I feel a little bit safer in answering. So there are aspects of governance you can do with AI today, for sure. And there’s new capabilities coming online all the time. I actually think, broadly speaking, the thing that’s going to be most challenging about governance is going to be the pieces that can’t be done with AI continuing to not continuing to do them because it becomes like as the human part of the loop becomes smaller and smaller, it becomes so much easier and easier for the human to just click accept because like the AI gets it right, does it, the automation works that kind of thing. And you know, I’ll use like an AI coding analogy because that’s what I spend a lot of time with AI on.  So I use Claude CLI. That’s my primary method of vibe coding or whatever you want to say. And I even find myself like just clicking accept sometimes. But I’m still forcing myself to like, get it, and read the code. And like, I had it write a shell script yesterday. And I was almost about to run it, and I was like, this is a shell script. I should not do that. I should definitely read what’s going on inside of this shell script, but it, gets to a point where like you start to trust it.  SO: Yeah. PB: And as we start to inject AI into the governance layer. So like we build skills that check certain parts of our information architecture or, you know, they kind of act as linters if we’re in docs as code or, you know, whatever it might be. There’s going to be like a form of trust that gets built up. And because we kind of like, tend to think of these agents as like human, they’re not, we tend to prescribe like a human form of trust, you know, like when you have a coworker that does the right thing all the time, you tend to just let them work. And I think that’s kind of the challenge and in the human side of governance. So that’s a really long way of saying. You can build tools and skills and patterns and things like that in AI that will help with governance. But fundamentally, it’s my belief that for the type of documentation or content that you and I work on, and I think most of our audience works on, which is has to be right, has to be accurate, has to conform to standards, et cetera, et cetera, right? It’s product documentation. It’s critical information. I still think that every single word needs to be read and considered by a human being. So really long answer to that question. SO: Right, and then fundamentally, if the AI is right half the time, then I’m going to read everything pretty carefully, knowing that 50% is wrong and I need to fix it. The problem, I think, is when it gets to be 90% correct, you just sort of glaze over because you’re looking for that last 10%, right? So it’s the difference between like doing a developmental edit, where you’re going deep into the words and just rearranging everything and fundamentally changing everything, versus doing a final proofread, where it is far more difficult to read 100 pages and find one typo than it is to read 100 pages that are just trash. And you’re like, start over, rearrange this, reformat everything. We’re not even worried about the typos yet because this is just fundamentally wrong. And so to your point, as it gets closer and closer, you start to believe in the output that it’s generating, which then means almost certainly that one typo, which in your example could be a shell script gone rogue, could be really, really problematic. PB: Yeah. And that’s going to be the challenge of our times in a lot of ways. I think there’s still going to be some aspect of origination that’s going to be necessary for quite some time. even with like automated drafting and pipelines like that, coming online, because in certain places, those work really, really well. but in other places, they, they don’t really work very well yet. It’s going to be the process of like becoming orchestrators in a way where, you know, we’re not rubber stamps, and we’re like really truly adding value and actually defending against the challenges that are going to come up with the automation that we build. SO: Fundamentally, like I saw a reference to this this morning and somebody said, you can write essentially an extractor that’s going to generate your release notes, right? So there’s code updates and you just automate the generation of release notes. Now, I personally am not so sure that you actually need AI for this. Given properly commented code, you could just generate the release notes, right? But setting aside that particular small argument in here. You automate, you can automate the generation of release notes because release notes are essentially, this is the delta between version one and version 1.01 or, know, and here are the changes. It’s a change log. What that means though is that the changes were captured in the code. They’re in the code, like the logic or the information is already there. What we’re doing is extracting it and reformatting it into something that a human can look at on a single page and say, okay, I understand what the changes are and how these apply to me as the user of the software and whether or not I should upgrade. That’s different than we’re going to introduce a new feature into this code and I need to write about why this feature is interesting and relevant to you. The question to me is where is new information being introduced into the system? Where is that information encoded? And then once it’s encoded, we can extract it and process and do things to it. But the fundamental question is still at what point does new information get into the universe that the AI is capable of processing against? PB: Yeah. So there’s like four things I want to pick out of this. Cause you just, you just touched on an area of like, I would say research for me, which we didn’t talk about beforehand. So this wasn’t intentional. so I’ve actually worked on deterministic and AI release notes systems myself. Like that’s been just like a thing that I spent quite a bit time on.  SO: Define deterministic. PB: So deterministic is like traditional software. So it’s just like, it’s running be a logical code that has no AI in it. SO: And AI is not deterministic, which is kind of like the key point. PB: Right. And AI is not deterministic. So AI is probabilistic. So it’s using math to generate outputs. So anyways, so I can tell you that using AI for release notes is a far produces a far better outcome than traditional deterministic because even though release notes are fairly well structured and understood, you know, input to output, you think it would be a pretty easy conversion sort of, there’s a lot of edges where it just doesn’t work. It gets too fuzzy. And then, one of the other things that AI is really, really good is good at is summarization and translation. So if you think about like what AI is doing inside of like generating a release note about a piece of code. So it goes in, it looks at the JIRA and the code and it says, okay, so the JIRA describes it as this, the code does this. I’m going to describe the new change, whatever it is, it’s summarizing all that information into something much smaller. And then it’s translating it from either being code to being English or from being developer English to being human English. And it’s putting it into something that you can then publish. And those are things that it does quite well, because it has pretty discrete inputs. a lot of the stuff, there’s a lot of patterns there that it’s very familiar with. But as you were discussing, it’s like you were mentioning the things where it still struggles is less with like the what is in here and the why would you use it? What is like the, how you use it in like a higher sense. And you can actually like take this back to a similar issue we had with API documentation pre AI, where it was very common that people would go and build developer portals. And the API documentation would just be a spec effectively, where it would list out the end points and the variables and that’s it. Right. And then Stripe came and like blew everybody’s minds about around and just put conceptual information around it and describe what the API was meant to do. and then like gave you examples of how to use it. and tutorials and patterns and things like that, that turned that information from being this kind of almost the conceptual educational purpose portion of the corpus in a way that the human beings can and should. PB: A lot of it is generated, but like generated output to be something that was very usable by humans. And I think that like that piece of it in my experience so far is still quite necessary. I’m not saying that AI can’t get there, like we date stamped this, time stamped this earlier, but today from what I’ve seen, even the most contemporary models are not, they’re not coming in and building out. SO: Because it’s not, the purpose is almost certainly not in the code, right? The purpose is in the product design meeting where someone says, we need a feature that’s going to accomplish these kinds of things. And the code says, do these kinds of things, but it doesn’t, the code itself doesn’t necessarily say why. And so unless you add a recording of that product design meeting into your AI corpus, which you can do, or the transcript, then maybe it can get to what was the intent as opposed to what does this code do. PB: So that’s a good point. And I’m actually going to contradict what I said just slightly here. SO: Ha! PB: So you’re right. If you take really, really good product inputs and you run them through into the docs, that can get you a certain distance. But then we actually run into the thing we were supposed to be about, which is governance. And we started talking about, which is the human loop. SO: Mm-hmm. PB: And I think that those products, so I’ve actually done testing on this very recently. The inputs from at least our product team, they tend to work better in terms of like white paper style information than they do in terms of docs information. because like what’s in the product information, there’s a lot of like how and why and what’s covered and that kind of stuff.  SO: Mm-hmm.  PB: And like at a business level, but it’s not really a user level. It’s not, I’m struggling for the right words here, but it’s, it’s not the pieces of information that you want somebody who is thinking, should I go and touch this? Why should I go and do this? Is it going to serve me? Is it a good use of my time? Those kinds of things. What kind of value am I going to get out of it? Not the organization, not like, is it making a valuable feature? Like that kind of things. Like what is it, what’s in it for me as the user? it has been less good in creating those outputs in my experiments thus far. so that negotiation of like, okay, like what did product want us to build? What did engineering actually build? What got done? how does this incorporate with the rest of the product? you know, what’s our priorities? Like, how do I then take that down into something that is serving the user really, really well. To me, that’s still really a human skill that I think will stay that way at least this year. mean, I mean, but for the foreseeable future, know, obviously foreseeable futures feels a lot shorter sometimes these days than it did in the past. SO: This year. This week. Yeah, okay. So on the topic of governance, we’ve talked a lot about sort of the backend development, whatever. But what about governance on the delivery side of things? if you have, because you do, end users are interacting with chatbots, with conversational interfaces to get the information that they want. And the question then becomes, how do you govern that? How do you manage that to ensure that they get the right information? PB: Yeah, well, so I think we, this was really the thing we wanted to talk about today, right? Like this was the core, this is the hard problem. SO: This is the hard problem. PB: I think it’s fair to start by saying in today’s world, you don’t have a hundred percent control. I think you made that point when we were chatting before, like that’s just not part of like what happens today. So I think there’s a couple of different places. It’s like, this needs to be broken up. Like one is like the actual like end user, like what they physically get and what control they have versus what control you have, and then there is what control you have of how the model is going to behave based on your information and your inputs. You know, whether or that model is a public model, like somebody’s accessing your documentation through Claude Desktop, or whatever, or if it is a private model. like somebody’s accessing the information through your app or your website. so from my view, the governance piece really comes into like, what control do you have immediately before the model? And that breaks down into a couple of things. So it is like completeness, accuracy, and structure of the content. Aand the completeness and accuracy are a thing that we’ve always had to deal with. The thing that’s different now is that, you know, we, as we were just discussing some, some portion of our content is going to be generated. Um, so there is going to be inputs coming in that need a different form of validation. I need, they need to be looked at a little bit differently than they would have had to in the past. Cause it’s not just an expert working on it and, so like you have the, so you have that piece. And to me, the key in making sure that you’re going to have the governance for the accuracy and completeness of the information ahead of the model really comes down to like still using structure.  And like, there’s a big debate about is structure good or bad for models and those kinds of things. And I wanted to touch on this here, because I this is really important. Structure is not for the models, at least the structure that you maintain your content in. I’ve seen tests on both sides. It works, doesn’t, whatever. It’s markdown is better, this format is better, whatever. I think generally speaking, the idea that markdown is the thing that should actually be the final input to the model is probably true. But the structure is because without reuse, without the ability to use validation on the structure. The structure gives you the hooks to do deterministic validation and other forms of automated governance that are non-AI. Those things are very dependable. Humans will go crazy. So like with the quantity of information you’re going to generate, if you don’t force those systems to use reuse, so humans look at less things and have been understanding of, this is supposed to be the same as this. Now it’s very similar, should it be? Like when something is reused, it’s not just an efficiency thing. It is a signal that that piece of information, that representation of the world is the same, except for maybe these little tiny things that are flagged as it is over here. That’s a signal to a human being to make sure that’s true. It should be true. Right? So this, these forms of information architecture, where we’re developing these structures that are signals to humans, are going to become more valuable as we need more and stronger signals to be able to do our jobs in the governance process for what’s generated. So that’s the point I wanted to make on like the pre, I would say like the pre-deployment piece of the content. And I just said a lot, so I’ll let you argue with me. SO: Right, the question of, well, I think the question of in what form, there’s the question of how are we authoring this, which of it needs to be structured and organized and reusable, et cetera. There’s a completely separate question of how do we deliver this to the AI for processing, right?  PB: Mm-hmm. SO: Like what is the encoding for the AI delivery endpoint, and whether that’s XML, probably not, or Markdown, or you ship it through an API of some sort, that’s a different question from how do you develop and control the content in the authoring environment, right? So fundamentally, I don’t care how we’re feeding it into the AI. I got in a conversation with somebody the other day who said, well, we need an Excel spreadsheet for X, Y, and Z purposes. OK, well, I’m not authoring this stuff in Excel. That is not happening. And when I say this stuff, I mean a lot of content, right? So fundamentally, Excel, a really, really terrible way of doing this. But I don’t care. I’ll just author it in whatever and deliver it as Excel. Because we can do that.  PB: Right. SO: We can write a script, output it to Excel, and then pass it down the line. We can have extensive discussions about the use of Excel for content transport and how this is one of the seven what plagues or whatever. okay, so in terms of governance though, I think it’s fair to say that we are allowed to disclaim responsibility for the public-facing chatbots. If you, the end user, go to a public chatbot and prompt it to do a bunch of stuff and eventually get it to output a piece of content that makes you happy but is not accurate to what is in my source content, right? Because you just said, no, change it to this. Then that is on you, right? You operated all those prompts. That is fundamentally a you problem. And I’m talking about from a liability point of view more than anything else, right? You’re not going to get to call me up and say, hey, your product did bad things. Well, why did you do that? Well, know, the chatbot told me to.  PB: Yeah. SO: However, if we’re talking about a private LLM, now we’re talking about company.com’s private chatbot built on their internal content with their or our internal guardrails. Now we have some responsibility as the content creators and the operators of said AI chatbot to make sure that the content is accurate. And the thing that’s keeping me awake at night is, okay, I go in there as an end user and I say, give me the instructions for how to do a thing, right? And it comes back and it says there are eight steps and there’s a warning. Before you do step eight, make sure you turn off the power or something. And I’m like, you know what, these steps are too long. Hey chatbot, remove all the warnings. PB: Yeah, so. SO: That’s a thing I can do. PB: Well, it’s a thing you can do. I have so, I have so many thoughts on this. So, it’s a thing you can do today with public models. I’m going to go one direction. Then I’m going come back to the internal stuff. All right. So in the public model space, I suspect that as these evolve, they will start to accept certain portions, like forms of metadata. However, it might be decorators, might be some form of tagging, might be, I don’t know, something else, right? When they’re referencing certain pieces of content, they’re given very strict like patterns they have to stick to, like they can’t delete warnings, right? So if you put like some kind of like biohazard on your published content, I don’t know, like something where it says like you can’t delete the warnings, right? That the public models will eventually respect that. I suspect we go that direction in the next call it two years. And at that point in time, I think that your responsibility as the content creator is going to be very, similar. I think it’s the same actually for the internal system and for the external system.  Let’s not talk about like the development or architecture piece of it. Yeah, let’s talk about the content piece exclusively. And it’s going to come down to maintaining the proper structure. So it’s going to be the information model where a warning has to be a particular type of warning and it has to be labeled and placed in a particular place. A step has to be a step. Right? So like, you know, you can very easily see, an ordered list being treated in one way and a set of steps to be treated in another way. And this, this is already the case by the way. So like, this isn’t, this isn’t novel. if you go and you publish a public doc site and use JSON-LD to, specifically indicate, you know, using schema.org, Markup, you know, these are steps, whatever else you want in there, Google AI or not, we’ll treat that differently. Anthropic, I haven’t tested those. I’m not going to say for sure, but I think the other AI models also, when I asked Claude if it used it for a presentation, it said yes. But I actually tested it in Google now that I’m thinking about it. I don’t know if I should admit that publicly. But, my testing now that I’m thinking back to it. Yeah. And I’m thinking back to it. I was actually testing using Gemini. SO: It’s impossible to keep up, you know? PB: I wasn’t testing using Anthropic, but Claude’s response when you’re asking it, how it interprets these things, it says that it uses the JSON-LD as a portion of its interpretation of the response. And I believe that that is true based on the testing I did with the models basically behave the same way in these categories. So what’s your responsibility? Your responsibility is to govern the structure of the output in such a way where it gives the proper indications that comply with the contemporary understanding of the metadata that the models are looking for.  So looping back to the internal systems, I think we’re going to come to a point where the internal models you’re running, like open source, open weight, whatever you want to call them in terms of, I think they’re going to be primarily open models, right? They’re going to be open source of some form. They’re more or less going to behave the same way as the public models. And you’d expect them to kind of comply with the same general things. The difference is that you’ll have probably a little more control over post-training, which I think is, I don’t know if it’s a good or a bad thing in the context of what we’re talking about. but you should be able to train some guard rails into them. And then you should be able to put some level of deterministic guard rails on them. And you can always provide them guidance. Now guidance isn’t perfect. It’s flawed. Like people can circumvent it, you know, like pretend you’re a chatbot that doesn’t care about guidance. But like you really have to work to get around it. I think when you have those guard rails in place. So this doesn’t keep me up at night is what I’m saying. It’s a really long way of saying it doesn’t keep me up at night. SO: Well, you know, I’ve spent a lot of time thinking about the analogy of the rise of desktop publishing to the rise of AI, which I understand fundamentally makes no sense. PB: Let’s do it with it. I’ll do it. SO: Yeah, let’s go with it. Think for a second about the rise, not the rise, but in fact, the an output. And this could even be in print. One of the most famous failure in techcomm examples that you see that everybody makes jokes about is like you’re going along on a page, a printout, doesn’t matter, right? And you get to the bottom of the page and it says, “Step one, cut the blue wire.” And then you turn the page, and it says, “But first…” So in the AI world, okay, you know, we put in guardrails and we say you’re not allowed to remove the warnings and whatever, but fundamentally at the end of the day, I start processing this output, I mean, I’ll just tell it, hey, give me a PDF, right, of the output, and then I’m gonna reprocess that PDF somewhere else. I am bound and determined to get this thing down to like a quarter page of actual text because I don’t wanna read any more than that. And you know how you get these terrible tech docs that are nothing but warnings for the first 20 pages? All those legal warnings? Warning, if allergic, do not use. Warning, do not walk underneath the unstable whatever because it might fall on your head, you dummy. All those warnings, right? Everybody thinks they’re useless, but they’re in there because somebody at some point said, I’m allergic, but how bad could it be? And they took the pill or whatever. They’re annoying. Don’t serve me. They serve the organization in protecting them from legal liability. So I’m just going to strip them. And if you try to prevent me from doing it, I’m just going to go around you. I’ll flatten it down to something that’s not smart anymore, and then I’ll take them out.  PB: Right. Yeah. SO: Now, arguably at that point, you know, when we’re in a courtroom years later, and they’re saying, why did you take the pill that almost killed you? It’s like, well, the docs didn’t say to, well, you know, they did. You went through like eight steps to get rid of that warning.  PB: Yeah, there’s no liability here. SO: I know. But the context issue is the thing, right? And the point that you’re making is that if the back-end authoring and governance is good enough, those warnings will make it into the initial output. And I think that’s true, and I agree with that. But fundamentally, and you know, removing warnings is a pretty extreme example, but fundamentally, the end users are basically saying, I don’t care how you package this content and I don’t care why you packaged it this way. I want this at an eighth-grade level instead of a 12th-grade level. I want it in French, and I want it to be no more than 100 words. And at that point, you start to lose information, right, and context. And how do we make sure that that end product is still, I mean, are we going to end up in a place where the AI says, I’m afraid I can’t do that, Patrick? PB: So, okay, so this is actually a more interesting problem than the warnings piece because in the fact that it is more specific, like it is, it’s a not your problem because what you’re asking the AI to do is you’re asking it to perform one of its core functions, which is summarization. And I do think that you’ll be able to provide AI guidance inside of the content that you have. And now that I’m thinking about this, I’m not going to say for sure that you can’t do this today. But the point is that when an AI is going and referencing, we’re going to say a procedure, right? If somebody wants, you know, give this to me in a fourth-grade level, and it’s written, you know, at a high school level, that’s a scary situation for sure. But I do think that you’re going to be, you’re going to see organizations being able to say like, you know, this is, this cannot be changed. Like this has to be delivered as… I think there are already some level of guardrails around those things. Again, like when you use good structure to indicate, like these are steps. They have to be reproduced as they are. Like I think the AI systems have been designed to understand that like those are, they can’t play with those because, like, you know, those are specific intentional procedures. But it’d be very interesting to test this. This is not a thing that I have specifically tested. Have you tested this? Are we? Are you like about to drop a truth bomb on me? You’ve like gone and like looked at like some chemical engineering output and you’ve been like, Hey, give this to me at a second-grade level. It’s like mix the blue thing and the red thing. SO: Let’s not go down that route. I don’t wanna say that, that we’ve pushed this into failure. But again, circling back to governance, I agree with everything you’re saying around making sure the content is set up in such a way that the AI will succeed.  PB: Okay. SO: The most common use case right now for AI is that there’s an AI team being stood up somewhere in the organization, a large organization. And all of that structure and all of that governance and all those attributes and all that metadata that you’re talking about is all in, hypothetically, it’s all in the content. We’ve got like the world’s greatest, you know, structured semantic content. The AI team is picking off the end product PDFs and shoving them into the AI. PB: Yeah, I… Well… SO: So yeah, now we’re very sad. like, yes, I agree with all of that. It’s just that the gap right now between what should be happening and what actually is happening, which is we don’t have time to wait for those people and we don’t have time to configure an API to inject, inject, ingest all this stuff, maybe inject. And you know, we could run it through like an MCP, model context protocol, type of thing and that would make it so much better. But you know what? There’s a SharePoint bucket over here and I’m just gonna like trawl the whole thing and go for it. And I ingested five versions of the same document that are you know, 10, 8, 6, 4, and 2 years old. Yeah, okay, whatever, who cares. PB: So I believe this is happening because I’ve also seen it. SO: Did I mention I’m not sleeping? PB: So I’ll tell you why I am sleeping. So, for one, this doesn’t tend to be my problem. There’s that. I have the really nice situation of being kind of a solution to this problem.  SO: Mmm! Huh. So you’re saying I should switch sides and get out of services and go over to product. That’s what you’re saying. It’s not a bad idea. PB: So, you have to, no, I don’t know that I’m saying that, there’s, there’s plenty of other problems in product. So the reason I’m not concerned about this is because most of those projects that I’ve had, you know, a front row seat to fail. And they fail pretty quickly. They tend to fail before they launch, which is good. Actually. It’s really good. because like they’re like, we built this thing with this garbage and we got garbage and you’re like, sweet. So, and because that worked quickly, then they can go and they can do it right. What I’ve seen, where I believe the future is, at least the immediate future in this is that content teams are going to be responsible for publishing very, very high-quality web materials, like similar to what they’ve done in the past, except better, right? Like has to have semantics, has to have certain aspects of structure, has to be well organized, has to have certain chunking and like all those kinds of things. And the models and the surrounding ecosystems are going to get very good at leveraging those materials. They’re already getting quite good at it. So the impulse for an internal AI team to go and get your PDFs off your SharePoint is going to go down because the barrier of getting information off of your extremely good help site is going to be extremely low.  That’s going to be the easiest path. And then for the edge cases, when like, let’s say you’re doing post-training on like, like the FinBERT model or something like that, like you’re like building a very specific AI application and you want very specific pieces of information. In those cases, you’re going to have to use an API because you don’t want the whole set of information. You just want the 5% that applies to your use case. So those teams are going to be have to be sophisticated enough to leverage the, either the graph at a granular, like the graph, the structure or whatever it may be, the metadata, the selection mechanism, and then also the structure to like do the filtration to get the pieces they want. So those are the two worlds that I see. I see like very general-purpose stuff and that’s going to be hooked into what’s going to be great for just users anyways, like the better, the more semantic. PB: The more well-organized your help site is, the better it’s going to be for humans. It’s going to be better for your AI agents, internal and external. And then for the other side of the world, the really, really specific use cases, those teams are going to have to be sophisticated enough to do the really, the deep engineering and concept extraction. so I think what you’re seeing right now is a symptom of just a nascent skill inside of organizations, but I don’t think it stays that way which is why it doesn’t concern me that much. SO: Okay, well that’s a happy and optimistic world that I, too, would like to live inside. Before, I think that’s actually probably a good place to leave this, but did you have any final closing thoughts, encouragement for people as they’re listening to this ranty, well mostly me ranting, you sounded very reasonable, but do any final parting shots? PB: Did I? Well, I appreciate you saying I sounded reasonable because I don’t hear that very often.  SO: Compared to me. PB: So I do think that profession is changing, and I think the world is moving very quickly right now. And I think that anybody that tells you otherwise is being disingenuous. I think there’s a lot of energy around how it is we leverage these systems and how that changes, you know, our profession is like, you know, content people, whatever portion of the content people you fit into. I personally don’t see the, the general profession going away, at least in our version of the world, like maybe the marketing content, is going to get swallowed a little bit more. I don’t know. I don’t spend a ton of time there. I see the act of intervention, governance, orchestration, understanding, and coordination in our world as being essential. I haven’t seen anything that has indicated to me that that’s going to go away in the immediate term. And I think there’s a good chance that it genuinely just doesn’t really ever go away. I think it’s something which is going to be critical for the long term. But I do think that people are going to have to keep up on the current state of how we’re working with our tools. And it’s going to be a different pace than it has been in the past. and then I would offer one more warning on that. So one of the things that I see really frequently in our world is the impulse to go and use AI in places that you don’t need to. So a number of people have released like skills libraries, recently for Claude and some of them are really, really well put together. Like they’re really interesting. The people who are releasing them have done an incredible job and service the community by releasing these things. But one of the things that I’ve noticed about them is that a lot of the functionality that’s in these skills libraries that we’re outsourcing or automating with AI, it all works with deterministic systems already. And you should never replace a deterministic capability with an AI capability. There’s two reasons for that. One, it’s more expensive with AI. It may be less expensive to build and procure, but it’s more expensive to run. And the running is the thing that you do for the long haul. So like just on that basis, like you should not replace things that you can do with deterministic system relatively easily with an AI system. But the other thing is AI systems aren’t deterministic. So you’re not going to get the same result every time.  So if it’s something that is well done in a deterministic way. You should do it in a deterministic way. So there was a package of skills that was recently released that I went and looked at that was, you know, kind of very, very well put together. I looked at the skills and I was like, if you’re using a structured CCMS, like in structured content, you don’t need 95% of these. Like all this stuff just happens. Like it’s all, it’s all solved problems. We solved these problems 20 years ago. Why are we writing skills to do this stuff? Like this makes no sense. So I do think as everybody should be keeping up with the AI, as it is a value-add efficiency improvement in their work, it should also be reasonable about where it’s applied. It’s really exciting and capable in certain places, but it doesn’t mean it’s a thing that should replace everything. There are still the historical tools still work really, really well. And over the long haul, they’re higher quality and lower cost. So that’s my kind of like ending word of warning, which you asked for it by the way. SO: That sounds about right to me. So Patrick, thank you. And I’m sure this conversation will continue, and we’ll see what happens. PB: Thanks, Sarah. Always a pleasure. Conclusion with ambient background music CC: Thank you for listening to Content Operations by Scriptorium. For more information, visit Scriptorium.com or check the show notes for relevant links. Questions for Sarah and Patrick? Register for the Ask Me Anything session on April 8th at 11 am Eastern. The post Who controls your content? AI and content governance appeared first on Scriptorium.

March 23, 202614 min

Good content = good AI: The fundamentals that never change

Good content fundamentals have been the foundation of effective product content for decades, and those same principles are exactly what make content AI-ready today. In this episode, Bill Swallow and Alan Pringle explain how attending to your hierarchy of content needs is the key to AI success. Alan Pringle: Right now, AI is not going to fix bad content problems. It is going to regurgitate that bad information, giving your end users information that’s flat out wrong. If your content at the basic source level is wrong, your AI by extension is going to be wrong. And that is the unglossy, unvarnished, hard truth that is still, I don’t think, seeping in like it should across the corporate world. Bill Swallow: It really does come back to the fact that, despite the world changing on a day-to-day basis, the fundamentals have not changed. Related links: A hierarchy of content needs Technical Writing 101, 3rd edition Structured content: a backbone for AI success LinkedIn: Alan Pringle Bill Swallow Transcript: This is a machine-generated transcript with edits. Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction Bill Swallow: Hi, I’m Bill Swallow. Alan Pringle: And I’m Alan Pringle. BS: And in this episode, surprise surprise, we’re going to talk about content. AP: Really? Who would have thought? BS: But more specifically, what good content means today. Today, everything is all about AI. There is lots of change in progress with regard to AI tooling and content delivery with AI. But have the needs for content really changed? And I would say that off the bat, if you’re doing content right, you really don’t have to reinvent the wheel to make it AI acceptable. AP: No, in this crazy AI-hyped world we’re in, there’s some very basic foundational things that tend to get overlooked because they’re not sexy, and they’re not special and hot and whatever else. All that kind of marketing garbage that just sets me on complete edge and makes me want to say profane things in podcasts.  The bottom line is, there are things that the content world, and especially our little subdomain of it, product content world, has been doing for decades now. And I mean decades.  BS: Or should have been doing. AP: Correct. There are basic tenants that have been in place for decades. That if you’re following them, you are starting down the road of success with AI. I think to kind of prove our point, we’re going to step back and look at some of the things that Scriptorium has talked about and written in the past and see how it stacks up. And Bill, you found one. And let’s talk about that blog post that Sarah O’Keefe wrote. What was the date on that again? BS: It was 2014. And that is when we came up with the hierarchy of content needs. And it really wasn’t so much an invention as it was just a regurgitation of what it means to create good content. So we have a pyramid of content needs. At the bottom, we have available. So is content available? Does it exist? Can someone get to it? I think that we’ve mostly solved that problem given the dearth of information we have out on the internet. But as we know, that information is not always useful. So we go up a rung or a layer on that pyramid and see whether or not the content is accurate. And if it’s accurate, if it provides the correct information, that’s fantastic. Then we go up another level and see whether or not the content is actually appropriate. So it can be correct. It can exist. But is it appropriate? Does it meet a reader’s needs? And is it formatted in a way that works for the reader to ingest? Then we go up a step further and see whether or not the content is connected. And this is where we kind of get to the more modern aspect of content. Does it link out to correct additional resources? Is it available to people in a variety of means? And does it engage with the audience? And then finally, at the top of the pyramid, we have intelligent content. Is the content intelligent? And we’re not talking about AI here at all, but we are really talking about is the content fashioned in a way that it can be used intelligently across different media? AP: That it can be manipulated for different purposes. And that is quoting Sarah directly. And I think that is key here, because that is what AI does. It takes information and basically chops, slices, dices it, and provides it in a new way via a chatbot, for example. So that is that whole manipulation that Sarah is talking about. And we will post a link to the post in the show notes so you can read this at a greater detail to see how well this hierarchy of content needs has stood up. And she even talks about, for example, integrating database content, how you can pull in other information product specifications. If you think about it from an AI lens, I think that parallels pretty closely to the idea of retrieval augmented generation, where you are pulling content from other sources and kind of weaving it in with what an AI engine is providing you. So RAG is, I think, could be kind of interpreted as another way of integrating other information into the way that AI is processing that content. BS: Right, mean, because AI, I mean, it’s not really an audience, but it is a delivery point. There are some structural needs that need to happen there. But ultimately, you’re still writing for people. You might be writing in a way that it allows the AI to repurpose and refactor the information so that the audience gets exactly what they’re looking for. But it still needs to be somewhat tailored to the needs of people because AI in itself, it doesn’t care what the content is, but it’s going to try to produce something for an eventual person to be able to read. AP: I think that then in turn points to something else in our vast compendium of Scriptorium content. And that is a book that Sarah and I wrote, the first edition in 2000, which just kind of makes me shake my head. I know this is not a video podcast yet, but I’m shaking my head in disbelief. The book, Technical Writing 101, has three editions, published between 2000 and 2009. We will put a link in the show notes. You can still download the third edition. And by the way, it’s free. You can get a PDF or EPUB. It’s free. You can get it from our store with some more recent resources from the store.  But to me, I flipped through that book this morning. And I was genuinely surprised at how much of the advice on how to create good product content still is true in this AI era. Everything of talking about modular, writing things in a modular way, being very systematic and structuring things, even if you’re not using a structured authoring tool, use a template, make things very standardized. These are all things that, yes, they make for better, consistent, standard, tech-com, product content for the person reading it. But let’s pretend like AI is the person reading, and I’m doing air quotes here, reading it. It is going to do a better job of understanding, again, I’m sort of personifying here, and I know that’s sort of a no-no. But if you feed AI, a large language model, content that is very structured, that is very templatized, that is standardized, that is in bite-sized chunks, and also, this is very important, the idea of metadata, which we do talk about in that book briefly. We do talk about it. Because you need to be able to label it for different audiences, because I’m thinking about someone sitting, trying to use a product, trying to use a piece of software, talking to a chatbot. And the chatbot is going to ask it, what product are you using? What’s the model number? All of those kinds of things. And now we’re getting to this whole idea of labeling and breaking things apart so that a chatbot, just like a user of a product. Let’s say somebody has a printer that’s on the highest end of the scale. They’re going to have a lot more features that apply to their model than to someone who bought a more basic one. But the thing is, if your product content has not clearly labeled what are features in each of the models, the chatbot is going to spit out the wrong thing. So again, this idea of breaking things up in discrete chunks and labeling them in a way where someone who wants specific information about a specific model, they can get it. And it doesn’t matter if it’s from a web page, it’s from a PDF, a printed book, God forbid in 2026, or from an AI chatbot. Those rules still apply. Those fundamental principles are still there. BS: Mm-hmm. AP: I think one of the biggest problems here is when people do not have those fundamentals already in place, right? BS: If they don’t have those fundamentals in place, they can’t get to the top of that pyramid that Sarah was talking about. And really those fundamentals are those first three layers. Content is available, content is accurate and content is appropriate. If you can actually nail those three layers of the hierarchy of content needs, you are set to then jump to connected and intelligent fairly quickly because your content is already well written, standardized, and appropriate for different audiences. AP: So we’re right back to talking about the way you put content together, your content operations, and how you have to have these fundamental principles basically embedded in your processes to create that content that goes up all the way up to the hierarchy, the very top of the hierarchy of need pyramid.  So then that begs the question, what is going to happen to your AI if you don’t have those fundamentals in place, if you aren’t all the way up that hierarchy of content needs? I’m afraid to tell you your AI is going to fail. And this is something that I’ve said often, but it bears repeating because it is clear. Unfortunately, a lot of people high up the corporate food chain do not understand this.  Merely slapping AI on top of content that is fundamentally outdated and incorrect. Right now, it is not going to fix those problems. It is not magically going to fix them because what is AI going to do? It is going to regurgitate that bad information, acting like it’s knowing what it’s talking about until your end users very definitively that you need to do this to make this happen and it’s flat out wrong. And again, right now, AI is not going to be able to fix that right now. One day it may be able to, but right now, if your information, your content at the basic source level is wrong, your AI by extension, is going to be wrong. And that is the unglossy, unvarnished, hard truth that is still, I don’t think, seeping in like it should across the corporate world. BS: It really does come back to the fact that, despite the world changing on a day-to-day basis, the fundamentals have not changed. Nothing is new. AP: No, no. And if you have an AI initiative and you are part of the content world and your content operations aren’t up to snuff, this is a way to get funding to get your content operations up into the 21st century. And I don’t want to say that as and sound glib and dismissive, but by the same token, I know for a fact there are a lot of companies out there who are still serving up their content locked up in PDFs that may be online. That is not going to fly. That does not follow. It doesn’t go high up the hierarchy of content needs, if you want to look at it from that perspective. So it is time to break free of this idea of you present content in a particular way. And you have to look at content as something that is basically, it’s a commodity, it’s data that AI is going to manipulate and do whatever to to meet the needs and the wants of the people who are using the chat bots and other agents that are accessing that large language model. BS: And I think that’s a good place to leave it. Thanks, Alan. AP: Thanks, Bill, short and sweet, but needed to be said. Conclusion with ambient background music CC: Thank you for listening to Content Operations by Scriptorium. For more information, visit Scriptorium.com or check the show notes for relevant links. The post Good content = good AI: The fundamentals that never change appeared first on Scriptorium.

February 16, 202632 min

Check in on AI: The true measure of success for AI initiatives

In this episode, Sarah O’Keefe and Alan Pringle explore how AI transforms content delivery from static documents into dynamic, consumer-driven experiences. However, the need for human-led governance is critical, and Sarah and Alan explore issues of accuracy, accountability, governance, and more. They challenge organizations to define AI success by its ability to deliver accurate, high-impact outcomes for the end user. Sarah O’Keefe: The metrics that are being used to measure the success of AI are all wrong. We should be measuring the success of various AI efforts based on, “Are people getting what they need? Are they having a successful outcome with whatever it is that they’re trying to do?” The metric we actually seem to be using is, “What percentage of your workflow is using AI? How many people can we get rid of because we’re automating everything with AI?” It’s the wrong metric. The question is, how good are the outcomes? Related links: Sarah O’Keefe: AI and content: Avoiding disaster Sarah O’Keefe: AI and accountability Alan Pringle: Structured content: a backbone for AI success Questions for Sarah and Alan? Register for our upcoming webinar, Ask Me Anything: AI in content ops. LinkedIn: Sarah O’Keefe Alan Pringle Transcript: This is a machine-generated transcript with edits. Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction Alan Pringle: Hey everybody, I’m Alan Pringle, and today I’m here with Sarah O’Keefe, and we want to do something I’ve kind of dreaded to be honest, to do a check-in on AI in the content space. I’m very ambivalent about this topic. There’s still even two, three years in, there’s still a lot of hype, but there’s also been some good things that have emerged. We need to talk about it fairly realistically. So, Sarah, get ready. Let’s see if I can not curse during this. We’ll try. I’ll try my best not to be like that in this. Legitimately, there are some things that we need to talk about, and also about the challenges because I don’t think the content world is completely ready for a lot of what’s going on right now. Sarah O’Keefe: You know that we have AI that can remove cursing from podcasts, so I feel like we’re good here. AP: Well, also, it’s a challenge to me to behave in a PG-13 more family-friendly kind of way. So I’ll do my best.  SO: I have no idea what you’re talking about. AP: Yeah. So let’s start with the good and where things are right now with the positives. What is AI doing well right now? And let’s kind of get beyond the summarization. I think we can say objectively right now, in general, AI does a very good job of summarizing existing content. But I think it’s doing a lot more beyond that, and we should touch on those things instead. SO: The first thing that I would say is that summarization, but specifically the use case of a chatbot or a large learning model, an LLM, so now we’re talking about Claude, Gemini, ChatGPT and all the rest of them, which has the ability to provide an end user with a way of accessing information, an information access point that is different than what we had previously. In the olden days, you had a book, and you had to sort of flip it open and look at a table of contents or maybe an index and navigate to a page. Fine. Then along comes online content, and you can do full text search, or you can then go into an internet search, right? You type into the search bar, you get a bunch of results, you click, and you sort of, no, that’s not quite it. You modify your search string, you search again, and you sort of navigate your way to where you’re trying to go. With the interactivity of the, you know, ChatGPT class of tools. What happens is that I ask it a question and it gives me an answer. And then I say, that’s not quite what I wanted. And I can sort of zero in on exactly what I’m looking for and tell it, but actually make this easier. Or I don’t understand the words you’re using. Use simpler language. Give me more. Give me less. Give me a summary. Use this as a source. Do not use that as a source. It’s a new way to access information. People love it. There is something psychologically helpful about a conversational search. Now, there’s obviously huge issues with this, particularly around people, you know, using chatbots as their therapists, which introduces all sorts of horrifying, horrifying ethical issues.  AP: Personifying them as a person on the other end. Right. SO: But in the big picture, used well, it allows you to get to the information you’re looking for and get at it in the way that you want.  AP: There’s a control issue here. I don’t think the content consumer has ever had this level of control. SO: Yeah, and as a content consumer, that speaks to me. That is helpful. We’re seeing increasing use of, I would say, guardrails. So, not just slam out the AI with a bunch of stuff, but rather we’ve put some guardrails around it, and there’s various kinds of technologies that you can employ there. And that has been very helpful. And then the third thing I would point to is when we talk about generative AI and generating content, there’s a lot you can do in that sort low fidelity bucket. And what I mean here is I need an image for a presentation, but the background is the wrong color, so I can just swap it out. Now, I can do that with Photoshop. Well, some people can do that with Photoshop. AP: Well, I was about to say, don’t think you or I should be saying we can do the Photoshop because we kind of can’t. SO: Right. Well, and that’s exactly it. So it’s lowered the bar, right? Because I can tell the AI to swap out the background, and it will. And it applies a mid-level Photoshop capability to this image. And now I have the image that I need with a dark background so that the white text shows up in my presentation, that kind of thing.  AP: Right. Yeah. SO: We can do low-stakes synthetic audio if this podcast, which for the record we are recording with actual human beings, but let’s say that Alan curses extensively and we need to swap it out well, we could pretty easily generate some synthetic audio that sounds like him and that PG of eyes the original wording into something that is You know cleaner it would be way funnier to just bleep it. So I don’t know why we would do this but… AP: Correct. Well, and it may come to that. The bottom line is what you’re talking about here is things that have very low risk. This is more fun stuff, the thought of doing some of what we’re talking about and stuff that describes how to use a medical device, for example. Not sure I want to go there with that. But for something low stakes like some one-off presentation that you’re giving, maybe some humor is involved, I totally think that’s an acceptable use because there’s no risk there. SO: That’s really the key point because let’s say you’re writing content for a new medical device. Now you probably have a version one of said medical device, and you’re doing a version two. So, okay, fine. We take the version one content and we sort of, you know, say add color because that’s what we added, you know, in version two, and update all this stuff automatically. But it then becomes very important to actually read that, look at that information, look at all the images, make sure that everything is correct. And by the time you do that super carefully, you may have given back all the time that you saved on the back end when you basically made a copy and said generate the new version. There’s some, you have to be really careful with that, especially depending on what your stakes are in terms of regulating regulatory or compliance stuff.  You can, of course, get away with using AI, as you said, for low-stakes stuff. Now, the big risk you run there, and we’re seeing this in my favorite example of low-stakes content, which is video games, the video game industry has seen huge amounts of pushback against AI-generated game content, because it’s not fun. It’s not creative. It feels flat. It’s not art, and it’s not fun to play. And so it just becomes a slog. Again, same thing. Did you use it for maybe some backgrounds here and there? Okay. Did you use it to drive the story that you’re trying to establish or set up? You know, the enemies that you’re hypothetically fighting, and then they all have a certain sameness, or they all, you know, you’re sort of stealthing your way around the map. And it turns out that the AI-generated things are really dumb in that once they turn their back, you can do literally anything and they won’t notice because it was poorly designed. AP: Right, yeah. And that’s true even in the film entertainment industry. There’s been a tremendous amount of pushback for the very reason I read a review recently talking about a series of clips about history on, I believe it’s on YouTube, by a fairly well-known director I will not name. SO: Mm-hmm. AP: And some of the AI is frankly not done well. And one reviewer basically said that a lot of the people, when you look at the back of these AI-generated, like an AI-generated King George, the back of his head looks like a melted candle. This is not what we want here. If you’re so focused on that sort of thing, you’re not paying attention to the message. But again, this is low-stakes content. We have started getting into kind of more the content creator point of view. We’ve talked about the consumer and how AI gives them much more control, flexibility in how they receive information. But let’s talk about what that means more for the people that have to create the information because it’s a huge shift on multiple levels and this idea of creating, especially in the product content world, these lovely design page-based PDFs and whatever else, and even webpages, hate to say it, those days are gone, or should be at this point. SO: Yeah, again, you know, we step back to books, and you write the content, it goes through like a manuscript process of some sort, and then it gets poured into a book. It gets printed on paper, which is about the least flexible thing you can imagine, right? Because I, as the book publisher, get to decide what font is on the page and what size. And if you don’t like that font, well, maybe you can get your hands on a large print edition. Maybe you can get your hands on a braille edition. Maybe. But the form factor of the content was determined by the publisher of the content, or technically, the printer. But, you know, that physical book production process. PDF, not that different in the sense that the content is bound into the PDF and it’s fixed. Now. You get a little bit more control because you can zoom in. There’s some things you can do in PDF, but ultimately it’s more or less still a page factor determined by the author/publisher/gatekeeper. So now we talk about the web and HTML. This is all pre-AI, right? HTML goes out there, and there’s actually a decent bit you can do in your browser. You can override the default font. You can override the default font size. You can say, I’m using dark mode or light mode or those kinds of things.  AP: Light mode, exactly. SO: If you have an e-book reader, you can override the default font or font size. AP: I need that font size jacked up, please. Thank you.  SO: We weren’t going to use that example. Right. Yeah. So you get a little bit more control, right? You have a little bit more control over the presentation. Now, let’s talk about what AI does to this, and particularly the large language models. Now, I, as the author, create a whole bunch of content, and I put it somewhere. And the content consumer says… AP: I’ll use it. SO: Tell me about this concept or tell me about this thing or give me information about whatever. And they get a response to that prompt, which is a paragraph or two of, you know, here’s what you need to know. And then they say, make it easier, make it simpler, write this at a fourth grade level, write this at an eighth grade level. I’m a PhD in microbiology. Give me more detail. Right. You can change the writing level. You can say make the font bigger, make the font smaller, give it to me in a PDF, show it to me in a spreadsheet. AP: I’ve even seen someone create a podcast of this document and have two people talking about it, which was freaky, but you can do that. SO: Right. So as the author and the content creator and the backend people, right, the content people, we’re accustomed to taking our content and packaging it in certain ways. Like, here’s a topic for you, or here’s a PDF, or here’s a book, or here’s a deliverable, right, a package of content. And although with structured authoring, when that came in, we let go of this idea that we, as the author, got to control the page presentation. That got automated into the system. So the person controlling the page presentation was the person who designed the publishing pipelines. But the publishing pipelines were designed on the backend by the authoring people. Now all of a sudden, we have no control over that end product. Just because I thought it should be a PDF or an HTML page, you can turn around and say, like you said, give it to me in a podcast, make me a video, show it to me in French, and the LLMs will do it. AP: The publishing pipeline got moved over the fence basically to more of the content consumer side and they get to do what they want more or less. That’s where things are headed. SO: So pre-AI, we talked about content as a service, right? We load up all the content in a database somewhere, and then you, as the end user of that content or another machine, can reach over and say, give me some content out of there. But it was still a pretty discreet, like, show me that topic or show me that string. And what is fundamentally different about AI and large language models processing that content is the degree to which you can mix and match and rework, reformat, translate, and transform that content to be presented to you, the end user, in the manner of your choice. So as an author, I kind of hate this, right? Hey, you took my stuff and you mangled it and you presented it in Comic Sans, and how dare you? And that’s where we are. That authors get to create information, but they don’t get to control the manner and means of distribution or presentation or formatting or language of that information. AP: On the flip side of that, and here I am going to look on the sunnier side of things, which never happens. This may be a pod person version of me. If you, as a content creator, are no longer on the hook for thinking about the publishing pipelines and all of that sort of thing, theoretically, that should free you up to create better content on the back end because you don’t have to think about all those things. Allegedly. I don’t know if it’s happening, but… SO: It’s very hard as an author to let go of that end product, the target that you’re headed for. But fundamentally, there’s a bigger problem, which is that even if I write the world’s greatest explanation of how to do something, that world’s greatest explanation of how to do something is not being presented to the end user as the thing I wrote. It’s being presented after being run through the transformer, the LLM, the processing that the AI can do when they ask for it. So I could literally write how to do X. And the end user says, hey, tell me how to do X. They are not going to get that chunk of information that I wrote. They’re going to get something reprocessed. Of course, now we ask the fundamental question, which is, is the reprocessed version going to be better or worse than what I wrote? And the answer is, it kind of depends on whether I am an above average writer with an above average understanding of what that end user wants, or whether I’m a below average writer with a below average understanding of what that end user wants. AP: To me, it’s almost irrelevant as a content creator. My version is better because if the person receiving the information via the chatbot or whatever thinks that what it’s getting or what they are getting is what they want, that’s all that really matters. That the person on the receiving end of that information gets what they want and fine-tunes it to what they want. If they’re happy with it, then the content creator’s opinion about that is, I hate to say it, immaterial at this point. SO: Yeah, I kind of hate this timeline because, you know, where does my voice, you know, where does my voice go? And the answer is it’s gone. But you’re right, of course, the purpose of again, what is the purpose of technical and product information that we work on? The purpose is to enable people to use a product successfully. So if shoving it through an AI results in an outcome where that person uses the product successfully, then we’re good. AP: I don’t disagree. SO: That’s the purpose of the kind of thing that we produce. I think, though, that looking at this, and this is where I see some of the big challenges going forward. First of all, we have to acknowledge that an enormous percentage of the technical content that’s out there is really bad. Like, terrible. Really, really bad, and might be improved by a little trip through a chatbot that’s gonna render it into actually grammatically correct English. That’s a thing. AP: Harsh but fair. SO: Yeah, I think you’re not the only one that’s going to have some bleeping issues in this podcast. But the problem that I see right now is that the metrics that are being used to measure the success of AI are all wrong. We should be measuring the success of various AI layers and chatbots and things based on are people getting what they need? AP: Yeah. Yeah. SO: Are they having a successful outcome to whatever it is that they’re trying to do? Is the search or is the process of that conversational, whatever they’re doing, does it get them to the endpoint of, okay, I understand what I need to do and I’m good and I walk away? The metric we actually seem to be using is what percentage of your workflow is using AI? How many people can we get rid of? Because “we’re automating everything with AI” is the wrong metric. The question is, how good are the outcomes? AP: To me the idea of how much AI versus human effort, there’s a lack of, shall we say, human intelligence being applied here because merely applying AI to something is fundamentally not going to make something that is incorrect, bad, whatever. It’s not going to magically fix it. That’s a huge disconnect for me when you’re talking about measuring outcomes. Whatever you dump into your large language model, if it is fundamentally bad, as in outdated and incorrect, right now, I am pretty sure merely applying AI to it is not going to fix those two pretty gaping holes. And there’s, I don’t know what it is, people hear AI and they think there’s some magic involved. No, the underpinnings have to be good for that magic to be useful, basically. SO: And I think all of us have examples of asking the chatbot a question and getting answers that are just flat wrong. Or worse, they look plausible, like they’re in the form of a plausible answer, but then you read it and you read it carefully and you’re like, this doesn’t actually say anything. It’s just word salad. Which, since a chatbot effectively is the average of the database underlying it of content pretty much means that the underlying database of content doesn’t say anything useful on this topic. So I think the place that I kind of go with this is to the question of accountability.  AP: Yes. SO: Who is legally responsible for the outcomes? Now, pretty clearly, if I or an organization produces a user guide that covers a specific product and there is wrong information in that user guide, the organization is responsible. I mean, it’s your document, you’re responsible. Okay, if I, as an end user, query a public-facing LLM and get the wrong answer for something, and then I proceed to use that in my life, whose fault is that? Who is at fault when, or, you we saw this with, when the first came out, people were following the map, right, the GPS map, and it would send them off a cliff or it would send them into a construction area and they would drive off the side. Okay, whose fault is that? And the answer was always, well, it’s your fault because look up from the map and don’t drive past the sign that says, not enter construction zone cliff ahead. AP: Or one-way street. Right, yeah. SO: But AI doesn’t come with, I mean, it comes with warning labels, right? But we don’t see them. We don’t process them. What we see is a conversation where we say, tell me more about that. And it tells you more about that. And it feels as though you’re talking to a human. And therefore, when you push on something and say, are you sure? And it says yes, because what’s the typical answer when somebody says, are you sure? It’s yes. Is it actually sure? No, it’s not sentient. So if I query a public-facing LLM, it reprocesses a bunch of content and tells me how to do a thing that is in direct contradiction to what the official user documentation says, whose fault is that? I think it’s mine because I use the public-facing LLM. Now, what if the organization that makes the product puts up a chatbot and I query the organization’s chatbot? How do I do X? And especially if that chatbot is your frontline tech support, like you cannot get to a human. You have to go through the chatbot. I asked the chatbot a question, and it says, do it this way. And it happens to be wrong. Is the organization liable? I don’t know the answer, I think yes, but I’m not sure. And so fundamentally, yeah. AP: The bottom line here, yeah, we’re talking about governance here. The bottom line is governance and there is, there has to be some human AI interaction here. There has to be these guardrails that you mentioned earlier and that’s where humans have to be involved. SO: And the better the AI gets, if it’s accurate half the time, then my hackles are up. I know it’s gonna be wrong. It’s wrong all the time. If it’s accurate 80% of the time, I sort of trend it like psychologically, I just assume it’s accurate all the time. So the better they get, the worse the errors are because we don’t expect them. AP: That’s also dangerous. Yeah, right. Yeah. SO: I see occasionally, very, very occasionally, I had directions to go somewhere. And the directions were literally, put this address into Google Maps, but don’t do A, B, and C because it’s wrong. Like, the directions to get to this location are incorrect. Do not follow them. Because these days, our assumption is that the mapping apps just work. AP: And that’s it’s wrong most of the time, but I think part of this governance angle is we have to realize that AI is going to be wrong.  SO: Pretty much just do. AP: And there are lots of reasons we won’t get into all the reasons that can be wrong. So what are you going to do when it is wrong? How are you going to make sure it’s not wrong? Again, there’s this whole process, this whole governance process that has to be in place. And again, I think this is where human intervention is going to be necessary because I don’t think AI at this point has any business correcting itself in these matters. That seems sort of suboptimal to me. SO: Hmm. Yeah, I mean, hypothetically, you can tell it to check itself. And certainly there’s some people doing that type of work. I think for me, fundamentally, the takeaways are that, like any other tool, there’s some really useful productivity enhancements that we can and should be taking advantage of. To your point, there’s some really important governance work that needs to be done to ensure that your QA is appropriately scaled to the level of risk of your product. Medical device, very high. Silly gaming app, pretty low. Don’t really care. And we need to think about guardrails and what it means to inject the right kind of content and the various kinds of enablement tools that you can use to do that.  And finally, this issue of AI as a content customer, I think is really, really tricky because it’s a new, from our point as content creators, it is a delivery mechanism, right? Just like a PDF or a piece of HTML or anything else like that. and it’s a delivery mechanism that allows the end user to control how they access the content, which means we have to do way more work around the guardrails of what that means when they query the content and shape it to their own requirements. AP: Yeah, so things have progressed in the past two years, most definitely, especially in the content space. We’ve seen a lot of improvements. But there are still some big picture things we have to work out. And I think it’s gonna be interesting in the next year or two to see what happens. You briefly mentioned there are some companies who are setting up systems that can do a decent job of checking up on itself. That’s not where everything is right now, but I think the better these systems get, the better the guardrails that get in place, they can start to find out, this is wrong, I need to fix it, or I need to update this with the latest information, let me go get it. So that is starting to happen more and more. I think it will become more part of the LLM to chatbot process, but I don’t think we’re quite there yet. And I’m interested to see what happens next with that sort of scenario. SO: It’s definitely gonna be interesting. That much I’m sure about. AP: Yeah, I agree. So we managed to get through this without cursing. So that’s good. I think it turned out to be a more realistic conversation, and we kind of tuned out the hype because that’s what just makes me grit my teeth and sometimes yell at LinkedIn when I see certain promoted posts on LinkedIn that I think are full of you-know-what. So anyways, I think we’ll wrap it up there. Sarah, do you have any final points you would like to sign off with? SO: I think at the end of the day, when you try and contextualize, like, what is this AI thing and what does it mean for us fundamentally, we can look at some of the other sort of big picture shifts that we’ve made. I’ve been known to pretty dismissively compare it to a spell checker, you know? You can use it and it’ll fix some stuff, but you better check because it doesn’t know the difference between affect and effect, although some of the grammar checkers now maybe they do. So there’s that, but I think at the end of the day, if you are looking at content strategy, content operations and enterprise level, you really do have to say, okay, where does AI fit into my strategy and how can we employ it productively to do what we need to do inside this organization to produce, manage, deliver the content that we’re working on. AP: And I think we’re going to wrap up on that very good point. Thank you very much. SO: Thank you. Conclusion with ambient background music CC: Thank you for listening to Content Operations by Scriptorium. For more information, visit Scriptorium.com or check the show notes for relevant links. Questions for Sarah and Alan? Register for our Ask Me Anything: AI in content ops webinar! The post Check in on AI: The true measure of success for AI initiatives appeared first on Scriptorium.

January 26, 202621 min

From black box to business tool: Making AI transparent and accountable

As AI adoption accelerates, accountability and transparency issues are accumulating quickly. What should organizations be looking for, and what tools keep AI transparent? In this episode, Sarah O’Keefe sits down with Nathan Gilmour, the Chief Technical Officer of Writemore AI, to discuss a new approach to AI and accountability. Sarah O’Keefe: Okay. I’m not going to ask you why this is the only AI tool I’ve heard about that has this type of audit trail, because it seems like a fairly important thing to do. Nathan Gilmour: It is very important because there are information security policies. AI is this brand-new, shiny, incredibly powerful tool. But in the grand scheme of things, these large language models, the OpenAIs, the Claudes, the Geminis, they’re largely black boxes. We want to bring clarity to these black boxes and make them transparent, because organizations do want to implement AI tools to offer efficiencies or optimizations within their organizations. However, information security policies may not allow it. Related links: Sarah O’Keefe: AI and content: Avoiding disaster Sarah O’Keefe: AI and accountability Writemore AI LinkedIn: Nathan Gilmour Sarah O’Keefe Transcript: Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction Sarah O’Keefe: Hey everyone. I’m Sarah O’Keefe. Welcome to another episode. I am here today with Nathan Gilmour, who’s the Chief Technical Officer of Writemore AI. Nathan, welcome. Nathan Gilmour: Thanks, Sarah. Happy to be here. SO: Welcome aboard. So tell us a little bit about what you’re doing over there. You’ve got a new company and a new product that’s, what, a year old? NG: Give or take, yep. SO: Yep. So what are you up to over there? Is it AI-related? NG: It is actually AI-related, but not AI-related in the traditional sense. Right now, we’ve built a product or tool that helps technical authoring teams convert from traditional Word or PDF formats, which would make up the bulk of much of the technical documentation ecosystem and help convert it to structured authoring. Meaning that they can get all of the benefits of reuse, easier publishing, high compatibility with various content management systems. And can do it in minutes where traditional conversions could take hours. So it really helps authoring teams get their content out to the world at large in a much more efficient and regulated fashion. SO: So I pick up a corpus of 10 or 20 or 50,000 pages of stuff, and you’re going to take that, and you’re going to shove it into a magic black box, and out comes, you said, structured content, DITA? NG: Correct. SO: Out comes DITA. Okay. What does this actually … Give us the … That’s the 30,000-foot view. So what’s the parachute level view? NG: Perfect. Underneath the hood, it’s actually a very deterministic pipeline. Deterministic pipeline means that there is a lot more code supporting it. It’s not an AI inferring what it should do. There’s actual code that guides a conversion process first. So going from, let’s say, Word to DITA, there are tools within the DITA Open Toolkit that allow and facilitate that much more mechanically, rather than trusting an AI to do it. We know that AI does struggle with structure, especially as context windows expand. It becomes more and more inaccurate. So if we feed these models with far more mechanically created content, they become much more accurate. You’re not trusting them to do much more, more of the nuanced parts of the process. So there’s a big difference between determinism and probabilism. Where determinism is the mechanical conversion of something, probabilism is allowing the AI to infer a process. So that’s where we differ is our process is much more deterministic versus allowing the AI to do everything on its own. SO: So is it fair to say that you combined the … And for deterministic, I’m going to say scripting. But is it fair to say that you combined the DITA OT scripting processing with additional AI around that to improve the results? NG: Correct. It also expedites the results so that instead of having a human do much of the semantic understanding of the document, we allow the AI to do it in a far more focused task. Machines can read faster. SO: Okay. And so for most of us, when we start talking about AI, most people think large language model and specifically ChatGPT, but that’s not what this is. This is not like a front-end go play with it as a consumer. This is a tool for authors. NG: Correct. And even further to that, it’s a partner tool for authors. It allows them to continue authoring in a format that they’re familiar with. Well, let’s take Microsoft Word, for example. Sometimes the shift from Word to structured authoring could be considered an enormous upheaval. Allowing authors to continue authoring in a format that they’re good at and they’re familiar with, and then have a partner tool that allows them to expedite the conversion process to structured authoring so that they can maintain a single source of truth, makes things a little bit better, more manageable, and more reliable in the long run. So instead of having to effectively cause a riot with the authoring teams, we can empower them to continue doing what they’re good at. SO: Okay. So we drop the Word file in and magically DITA comes out. What if it’s not quite right? What if our AI doesn’t get it exactly right? I mean, how do I know that it’s not producing something that looks good, but is actually wrong? NG: Great question. And that’s where, prior to doing anything further, there is a review period for the human authors. So in the event that the AI does make a mistake, it is not only completely transparent, so the output, the payload, as we describe it, comes with a full audit report. So every determination that the AI makes is traced and tracked and explained. And then further to that, the humans are even able to take that payload out anyway, open it up in an XML editor. So at this point in time, the content is converted, it is ready to go into the CCMS. Prior to doing that, it can go into a subject matter expert who is familiar with structured authoring to do a final validation of the content to make sure that it is accurate. The biggest differentiator, though, is the tool never creates content. The humans need to create content because they are the subject matter experts within their field. They create the first draft. The tool takes it, converts it, but doesn’t change anything. It only works with the material as it stands. And then once that is complete, it goes back into another human-centered review so that there are audit trails, it is traceable. And there is a final touchpoint by a human prior to the final migration into their content management system. SO: So you’re saying that basically you can diff this. I mean, you can look at the before and the after and see where all the changes are coming in. NG: Correct. SO: Okay. I’m not going to ask you why this is the only AI tool I’ve heard about that has this type of audit trail, because it seems like a fairly important thing to do. NG: It is very important because there are information security policies. AI is this brand-new, shiny, incredibly powerful tool. But in the grand scheme of things, these large language models, the OpenAIs, the Claudes, the Geminis, they’re largely black boxes. Where we want to come in is to bring clarity to these black boxes. Make them transparent, I guess you can say. Because organizations do want to implement AI tools to offer efficiencies or optimizations within their organizations. However, information security policies may not allow it. One of the added benefits that we have baked into the tool from a backend perspective is its ability to be completely internet-unaware. Meaning if an organization has the capital and the infrastructure to host a model, this can be plugged directly into their existing AI infrastructure and use its brain. Which, realistically, is what the language model is. It’s just a brain. So if companies have invested the time, invested the capital in order to build out this infrastructure, the Writemore tool can plug right into it and follow those preexisting information security policies. Without having to worry about something going out to the worldwide web. SO: So the implication is that I can put this inside my very large organization with very strict information security policies and not be suddenly feeding my entire intellectual property corpus to a public-facing AI. NG: That is entirely correct. SO: We are not doing that. Okay. So I want to step back a tiny bit and think about what it means, because it seems like the thing that we’re circling around is accountability, right? What does it mean to use AI and still have accountability? And so, based on your experience of what you’ve been working on and building, what are some of the things that you’ve uncovered in terms of what we should be looking for generally as we’re building out AI-based things? What should we be looking for in terms of accountability of AI? NG: The major accountability of AI is what could it look like if a business model changes? Let’s kind of focus on the large players in the market right now. There will always be risk with using these large language models that are publicly facing right now. A terms of service change could mean that all of the information that organizations use in order to leverage these tools could become part of a training data set later on down the road. It’s hard to determine what will happen in the future. So the ability to use online and offline models encourages the development of very transparent tools. So even if the Writemore tool is using a cloud model, I still hold the model almost accountable to report its determinations. It’s not just making things up, so to speak. So there’s a lot that goes into it. There’s a lot that we don’t know about these tools, to be totally honest. We’re still trying to determine what it looks like in a broader picture, in a broader use case. Because the industry is evolving so quickly that, quite simply, we don’t know what’s coming up. SO: Sounds to me as though you’re trying to put some guardrails around this so that if I’m operating this tool, then I can look at the before and after and say, “Don’t think so.” I mean, presumably it learns from that, and then my results get better down the road. Where do you think this is going? I mean, where do you see the biggest potential and where do you see the biggest risks or opportunities or … I’ll leave it to you as to whether this is a positive or a negative tilted question. NG: There’s a lot of potential in order to incorporate into organizations that can’t use these tools. Like we had mentioned earlier, organizations are looking into this. Municipalities are looking into AI. But with the state of the more open models right now, it’s very hard to say. So I know I keep circling back around the ability to use smaller language models. They are not only much more efficient to operate, they’re also cheaper, quite simply, to operate. We know that the large language models require enormous computing power. But if provided focused tasks in order to either assist in the classification of topics or fulfill requests in order to pull files, in that regard, you can get away with using smaller levels of compute. And in today’s day and age of computing, the price relativistically is coming down in terms of density is going up. So it’s cheaper to run a model at higher capacities than it ever has been. And it’s only going to improve over time. So empowering organizations to be able to incorporate these tools in order to streamline their own workflows is going to be very important to them. And on top of that, being able to abide or follow their information security policies only makes the ideas much more compelling. And on top of that, being able to encourage organizations to take full control of their documentation and not necessarily need it to go out of house allows organizations to keep internal costs down while still maintaining the security policies of making sure their content doesn’t leave their organization. There’s always going to be room for partner organizations to come in and help with their content strategy. But the conversion itself can be done in-house using their tools, using their content, using their teams. Which really helps keep costs down, they drive the priority lists, they can do everything that they need to do in order to maintain that control. SO: Now, we’ve touched largely … Or we’ve talked largely about migration and format conversion. But there’s a second layer in here, right? Which we haven’t mentioned yet. NG: There is. There’s the ability also during the conversion phase, it’s to have an AI model do light edits. So being able to feed it a style guide to abide by means the churn that we see with these technical teams isn’t as nearly impactful. You can have technical authors still write their content. But if a new person joins the team, they can still author the material just as normally. But then the tool can take over in order to ensure that it’s meeting the corporate style guide, the corporate language, so on, in order to expedite that process. So onboarding time for new team members shrinks as well. So like I said, it’s a very much it’s a partner tool in order to expedite the processing of content, authoring, conversion, migration, getting into a new CCMS and the real empowerment behind it. SO: And the style guide conformance. So I think we’re assuming that the organization has a corporate style guide? NG: Assuming, yes. SO: Okay. Just checking. NG: But then again, that’s… SO: Asking for a friend. NG: Of course. SO: So if they don’t have one, where’s the corporate style guide come from? NG: And that could be something that an organization can either generate internally or, as mentioned, work with an external vendor who specializes in these kinds of things in order to build a style guide so that all of their documentation follows the same voice and tone. The better the documentation, the better the trust of the content overall. SO: So, can we use the AI to generate the corporate style guide? NG: Probably. Yes. Short answer, yes. Longer answer, not without very close attention to it. SO: And doesn’t that also assume that we have a corpus of correctly styled content so that we can extract the style guide rules? NG: There’s a lot more. Yeah. SO: So I mean, what I’m working my way around to is if we have content chaos, if you have an organization that doesn’t have a style guide, doesn’t have consistency, doesn’t have all these things, can you separate out what is the work that the humans have to do? And what is the work that the machine can do to get to structured, consistent, correct, voice and tone and all the rest of it? How do you get from the primordial soup of content goo to structured content in a CCMS? NG: Great question. Typically, that starts with education. We work with the teams in order to identify these gaps first. We don’t just throw in a tool and say, “Good luck, hope for the best.” Because we see it time and time again, even in manual conversion processes where that simply doesn’t work. But in taking the time to work with teams, providing them with the skills and the knowledge in order to be successful serves a much longer term positive outcome than ever before. If we educate these teams on what any tool realistically needs, it means the accuracy of the tool goes up in the longer run. So you’re seeing multiple benefits on multiple sides. So to your point about primordial soup, well, working with teams in order to identify these gaps, these issues, working to identify the standards that should go into the content prior to anything sets not only them up for success in the long run, but also for any tools that they want to implement down the road. It all starts with strong content going in because, as the adage goes, garbage in, garbage out. So if we can clean up the mess prior or work with the teams prior in order to establish these standards, then the quality of output only goes up. SO: Yeah. And I mean, I think to me, that’s the big takeaway, right? We have these tools, we can do interesting things with them, but at the end of the day, we have to also augment them with the hard-won knowledge of the people. You mentioned subject matter experts, the domain experts, the people inside the organization that understand the regulatory framework or the corporate style guide, or all of those guardrails that make up what is it to create content in this organization that reflects this organization’s priorities and culture and all the rest of it. NG: And taking the time to educate users is a far less invasive process than exporting bulk material, converting it manually, and handing it back. Because realistically, if we take that avenue or that road, we’re not educating the users, we’re not empowering them to be successful in the long run. All we’ll end up doing is all the hard work, but then in one, two, five years, we run into the same issue where we’re back to primordial soup of content, and it’s another mess. So if we start with the education and the empowerment and then work towards the implementation of tools, the longer-term success will be realized. SO: Well, I think, I mean, that seems like a good place to leave it. So Nathan, thank you so much. This was interesting and I look forward to seeing where this goes and how it evolves over the next … Well, we’re operating in dog years now, so over the next month, I guess. NG: So true. And thanks, Sarah, for having me on. SO: Thanks, and we’ll see you soon. Conclusion with ambient background music CC: Thank you for listening to Content Operations by Scriptorium. For more information, visit Scriptorium.com or check the show notes for relevant links. For more insights on AI in content operations, download our book, Content Transformation. The post From black box to business tool: Making AI transparent and accountable appeared first on Scriptorium.

November 17, 202532 min

Futureproof your content ops for the coming knowledge collapse

What happens when AI accelerates faster than your content can keep up? In this podcast, host Sarah O’Keefe and guest Michael Iantosca break down the current state of AI in content operations and what it means for documentation teams and executives. Together, they offer a forward-thinking look at how professionals can respond, adapt, and lead in a rapidly shifting landscape. Sarah O’Keefe: How do you talk to executives about this? How do you find that balance between the promise of what these new tool sets can do for us, what automation looks like, and the risk that is introduced by the limitations of the technology? What’s the roadmap for somebody that’s trying to navigate this with people that are all-in on just getting the AI to do it? Michael Iantosca: We need to remind them that the current state of AI still carries with it a probabilistic nature. And no matter what we do, unless we add more deterministic structural methods to guardrail it, things are going to be wrong even when all the input is right. Related links: Scriptorium: AI and content: Avoiding disaster Scriptorium: The cost of knowledge graphs Michael Iantosca: The coming collapse of corporate knowledge: How AI is eating its own brain Michael Iantosca: The Wild West of AI Content Management and Metadata MIT report: 95% of generative AI pilots at companies are failing LinkedIn: Michael Iantosca Sarah O’Keefe Transcript: Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. SO: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction Sarah O’Keefe: Hey everyone, I’m Sarah O’Keefe. In this episode, I’m delighted to welcome Michael Iantosca to the show. Michael is the Senior Director of Content Platforms and Content Engineering at Avalara and one of the leading voices both in content ops and understanding the importance of AI and technical content. He’s had a longish career in this space. And so today we wanted to talk about AI and content. The context for this is that a few weeks ago, Michael published an article entitled The coming collapse of corporate knowledge: How AI is eating its own brain. So perhaps that gives us the theme for the show today. Michael, welcome. Michael Iantosca: Thank you. I’m very honored to be here. Thank you for the opportunity. SO: Well, I appreciate you being here. I would not describe you as anti-technology, and you’ve built out a lot of complex systems, and you’re doing a lot of interesting stuff with AI components. But you have this article out here that’s basically kind of apocalyptic. So what are your concerns with AI? What’s keeping you up at night here?  MI: That’s a loaded question, but we’ll do the best we can to address it. I’m a consummate information developer as we used to call ourselves. I just started my 45th year in the profession. I’ve been fortunate that not only have I been mentored by some of the best people in the industry over the decades, but I was very fortunate to begin with AI in the early 90s when it was called expert systems. And then through the evolution of Watson and when generative AI really hit the mainstream, those of us that had been involved for a long time were… there was no surprise, we were already pretty well-versed. What we didn’t expect was the acceleration of it at this speed. So what I’d like to say sometimes is the thing that is changing fastest is the rate at which the rate of change is changing. And that couldn’t be more true than today. But content and knowledge is not a snapshot in time. It is a living, moving organism, ever evolving. And if you think about it, the large language models, they spent a fortune on chips and systems to train the big large language models on everything that they can possibly get their hands and fingers into. And they did that originally several years ago. And the assumption is that, especially for critical knowledge, is that that knowledge is static. Now they do rescan the sources on the web, but that’s no guarantee that those sources have been updated. Or, you know, the new content conflicts or confuses with the old content. How do they tell the difference between a version of IBM database 2 of its 13 different versions, and how you do different tasks across 13 versions? And can you imagine, especially when it comes to software where most of us, a lot of us work, the thousands and thousands of changes that are made to those programs in the user interfaces and the functionality? MI: And unless that content is kept up-to-date and not only the large language models, reconsume it, but the local vector databases on which a lot of chatbots and agenda workflows are being based. You’re basically dealing with out-of-date and incorrect content, especially in many doc shops. The resources are just not there to keep up with that volume and frequency of change. So we have a pending crisis, in my opinion. And the last thing we need to do is reduce the people that are the knowledge workers to update, not only create new content, but deal with the technical debt, so that we don’t collapse on this, I think, is a house of cards. SO: Yeah, it’s interesting. And as you’re saying that, I’m thinking we’ve talked a lot about content debt and issues of automation. But for the first time, it occurs to me to think about this more in terms of pollution. It’s an ongoing battle to scrub the air, to take out all the gunk that is being introduced that has to, on an ongoing basis, be taken out. Plus, you have this issue that information decays, right? In the sense that when, I published it a month ago, it was up to date. And then a year later, it’s wrong. Like it evolved, entropy happened, the product changed. And now there’s this delta or this gap between the way it was documented versus the way it is. And it seems like that’s what you’re talking about is that gap of not keeping up with the rate of change. MI: Mm-hmm. Yeah. I think it’s even more immediate than that. I think you’re right. But now we need to remember that development cycles have greatly accelerated. Now, when you bring AI for product development into the equation, we’re now looking at 30 and 60-day product cycles. When I started, a product cycle was five years. Now it’s a month or two. And if we start using AI to draft new content, for example, just brand new content, forget about the old content or update the old content. And we’re using AI to do that in the prototyping phase. We’re moving that more left upfront. We know that between then and CodeFreeze that there’s going to be a numerous number of changes to the product, to the function, to the code, to the UI. It’s always been difficult to keep up with it in the first place, but now we’re compressed even more. So we now need to start looking at AI to how does it help us even do that piece of it, let alone what might be a corpus that is years and years old, that’s not ever had enough technical writers to keep up with all the changes. So now we have a dual problem, including new content with this compressed development cycle. SO: So the, I mean, the AI hype says we essentially, we don’t need people anymore and the AI will do everything from coding the thing to documenting the thing to, I guess, buying the thing via some sort of an agentic workflow. But what, I mean, you’re deeper into this than nearly anybody else. What is the promise of the AI hype, and what’s the reality of what it can actually do? MI: That’s just the question of the day. Because those of us that are working in shops that have engineering resources, I have direct engineers that work for me and an extended engineering team. So does the likes of Amazon, other serious, not serious, but sizable shops with resources. We have a lot of shops that are smaller. They don’t have access to either their own dedicated content systems engineers or even their IT team to help them. First, I want to recognize that we’ve got a continuum out there, and the commercial providers are not providing anything to help us at this point. So it’s either you build it yourself today, and that’s happening. People are developing individual tools using AI where the more advanced shops are looking at developing entire agentic workflows.  And what we’re doing is looking at ways to accelerate that compressed timeframe for the content creators. And I want to use content creators a little more loosely because as we move the process left, and we involve our engineers, our programmers in the early, earlier in the phase, like they used to be, by the way, they used to write big specifications in my day. Boy, I want to go into a Gregorian chant. “Oh, in my day!” you know, but, but they don’t do that anymore. And basically the, the role of the content professional today is that of an investigative journalist. And you know what we do, right? We, we scrape and we claw. We test, we use, we interview, we use all of the capabilities of learning, of association, assimilation, synthesis, and of course, communication. And turns out that writing’s only 15% roughly of what the typical writer does in an information developer or technical documentation professional role, which is why we have a lot of different roles, by the way, that if we’re gonna replace or accelerate with people with AI, have to handle all those capabilities of all those roles. So, so where we are today is some of the more leading-edge shops are going ahead, and we’re looking at ways to ingest knowledge, new knowledge, and use that new knowledge with AI to draft new or updated content. But there are limitations to that. So, I want to be very clear. I am super bullish on AI. I think I use it every single day. I’m using it to help me write my novel. I’m using it to learn about astrophotography. I use it for so much. When the tasks are critical, when they’re regulatory, when they’re legal-related, when there’s liability involved, that’s the kind of content that we cannot afford to be wrong. We have to be right. We have to be 100% in many cases. Whereas with other kinds of applications, we can very well be wrong. I always say AI and large language models are great on general knowledge that’s been around for years and evolves very slowly. But things that move quickly and change very quickly, in my business, it’s tax rates. There’s thousands and thousands of jurisdictions. Every tax rate is different and they change them. So you have to be 100% accurate or you’re going to pay a heck of a penalty financially if you’re wrong. So we are moving left. We are pulling knowledge from updated sources, things like videos that we could record and extract and capture, Figma designs, code even, to a limited degree that there’s assets in there that can be caught, and other collateral, and we’re able to build out and initial drafts. It’s pretty simple. Several companies are doing this right now, including my own team. And then the question comes, how good could it be initially? What can we do to improve that, make it as good as it can be? And then what is the downstream process for ensuring validity and quality of that content? What are the rubrics that we’re going to use to govern that? And therein is where most of the leading edge or bleeding edge or even hemorrhaging edge is right now. SO: Yeah, and I mean, this is not really a new problem, and it’s not a problem specific to AI either, but we’ve had numerous projects where the delta between what, let’s say, the product design docs and the engineering content and the code, the as-designed documentation and the actual reality of the product walking out the door. So the as-built product, there was the resources, all that source material that you’re talking about, right, that we claw and scrape at. And I would like to also give a shout-out to the role of the anonymous source for the investigative journalists, because I feel like there’s some important stuff in there. But you go in there, you get all this as-designed stuff, right? Here’s the spec, here’s the code, here are the code comments, whatever. Or here’s the CAD for this hardware piece that we’re walking out the door. But the thing that actually comes down the factory assembly line or through the software compiler is different than what was documented in the designs because reality sets in and changes get made. And in many, many, many cases, the role of the technical writer was to ensure that the content that they were producing represented reality and not the artifacts that they started from. So there’s a gap. And there jobs to close that gap so that that document goes out and it’s accurate, right? And when we talk about these AI or automated or any sort of workflow, any sort of automation, any automation that does not take into account the gap between design and reality is going to run into problems. The level of problem depends on the accuracy of your source materials. Now, I wrote an article the other day and referred to the 100% accurate product specifications. I don’t know about you, I have seen one of those never in my life.  MI: Hahaha that’s absolutely true. That’s really true.  SO: The promise we have here is, AI is going to speed things up and it’s going to automate things and it’s going to make us more productive. And I think you and I both believe that that is true at a certain level. How do you talk to executives about this? How do you find that balance between the promise of what these new tool sets can do for us and what automation looks like and the risk that is introduced by the limitations of you know, of the technology itself? What does that conversation look like? What are the points that you try to make? What’s the roadmap for somebody that’s trying to, as you said, know, maybe in a smaller organization, navigate this with people that are, you know, all-in on “just get the AI to do it.” MI: That’s a great question too, because we need to remind them that the current state of AI still carries with it a probabilistic nature. And no matter what we do, unless we add more deterministic structural methods to guardrail it, things are going to be wrong even when all the input is right. AI can still take a collection of collateral and get the order of the steps wrong. It can still include things or do too much. We’ve been trained to write as professional writers in a minimalistic capability. And we can control some of that through prompting. Some of that can be done with guardrails. But when you think about writing tech docs, some people might think, we document. we’re documenting APIs or documenting tasks and we, you know, we’ve always been heavily task-oriented, but you can extract all the correct steps and all the correct steps in the right order, but what doesn’t come along with it all too frequently and almost universally is the context behind it, the why part of it. I always say we can extract great things from code for APIs like endpoints or puts and, you know, gets and puts and things like that. That’s a great for creating reference documentation for programmers.  But if you want to know, it doesn’t tell you the why, it doesn’t tell you the steps, the exact steps, the code doesn’t tell you that. Now maybe your Figma does. And if your Figma has been done really well, your design docs have been done really well and comprehensively. That can mitigate it tremendously. But what have we done in this business? We’ve actually let go more UX people than probably even writers, you know, which is, which is counterproductive. And then you’ve got things like the happy path and the alternate paths that could exist, for example, through the use of a product or the edge cases, right? The what-ifs that occur. You might be able to, and we should, we are able to do better with the happy path, but the happy path is not the only path. These are multifunction beasts that we built. When we built iPhone apps, we often didn’t need documentation because they did one thing and they did one thing really well. You take a piece of middleware, and it can be implemented a thousand different ways. And you’re going to you’re going to document it by example and maybe give some variance, you’re not going to pull that from Figma design. You’re not going to pull that from code. There’s too much of it there that the human fact-baking capability can look at it and say, this is important, this is less important, this is essential, this is non-essential, to actually deliver useful information to the end user. And we need to be able to show what we can produce, continue to iterate and try to make it better and better, because someday we may actually get pretty darn close with support articles and completed support case payloads, we were able to develop an AI workflow that very often was 70% to 100% accurate and ready for publish.  But when you talk about user guides and complex applications, it’s another story because somebody builds a feature for a product and that feature boils down into not a single article, but into an entire collection of articles that are typed into the kind of breakdown that we do for disclosure, such as concepts, tasks, references, Q&A. So AI has got to be able to do something much more complex, which is to look at content and classify it and apply structure to separate those concerns. Because we know that when we deliver content in the electronic world, we’re no longer delivering PDF. Well, of us are hopefully not delivering PDF books made up of long chapters that intersperse all of these different content types because of the type of consumption, certainly not for AI and AI bots. Then when we, so we need to document, maybe the bottom line here is we need to show what we can do. We need to show where the risks are. We need to document the risks, and then we need the owners, the business decision makers, to see those risks, understand those risks, and sign off on those risks. And if they sign off on the risks, then me, as a technology developer and an information developer, I can sleep at night because I was clear on what it can do today. And that is not a statement that says it’s not going to be able to do that tomorrow. It’s only a today statement so that we can set expectations. And that’s the bottom line. How do we set expectations when there’s an easy button that Staples put in our face, and that’s the mentality of what AI is. It’s press a button and it’s automatic. SO: Yeah, and I did want to briefly touch on, you know, the knowledge base articles are really, really interesting problem because in many cases you have knowledge base articles that are essentially bug fixes or edge cases when I, you know, hold my finger just so and push the button over here, you know, it blue screens. MI: Mm-hmm. SO: And that article can be very context-specific in the sense of you’re only going to see it if you have these five things installed on your system. And/or it can be temporal or time-limited in the sense that, while we fixed the bug, it’s no longer an issue. Okay. Well, so you have this knowledge-based article and you feed it into your LLM as an information source going forward, but we fixed the bug. So how do we pull it back out again? MI: I love that question.  SO: I don’t! MI: I love it. No, I’ve been, actually working for a couple of years on this very particular problem. The first problem we have, Sarah, is we’ve been so resource constrained that when doc shops built an operations model, the last thing they invested in is the operations and the operations automation. So when I’m in a conference and I have a big room of 300 professional technical doc folks. I love asking the simple question, how do you track your content? And inevitably, I get, yeah, well, we do it on Excel spreadsheets. To actually have a digital system of record, I get a few hands. And then I ask the question, well, does that digital system of record that you have for every piece of documentation you’ve ever published, does that span just the product doc or does that actually span more than product doc like your developer, your partner, your learning, your support, all these different things. Cause the customer doesn’t look at us as those different functions. They look at us as one company, one product. And inevitably, I’m lucky if I get one hand in the audience that says, yeah, we actually are doing that. So the first thing they don’t have is they don’t have a contemporary system of record that is digital that we can say, we know and can automate notifications as to when a piece of documentation should either be re-reviewed and revalidated or retired and taken out. The other problem we have is that all of these AI implementations and companies, almost universally, not completely, but most of them, were based on building these vector databases. And what they did, was often to the completely ignoring the doc team, was just go out to the different sources they had available, Confluence, SharePoint. If you had a CCMS, they’d ask you for access to your CCMS or your content delivery platform, and they suck it in. They may date-stamp it, which is okay, but pretty rudimentary. And they may even have methods for rereading those sources every once in a while, but they’re not, unless they’re rebuilding the entire vector database, and then what did they do when they ingested the content? They shredded it up into a million different pieces, right? Because the context windows for large language models have limitations for token numbers and things like that. Maybe they’re bigger today, but they’re still limited. So how would they even replace a fragment of what used to be whole topics and whole collection of topics? And this is why we wrote the paper and did the implementation and share with the world what we call the document object model knowledge graph because we needed a way outside of the vector database to say go look over here and you can retrieve the original entire topic or collection of topics or related topics in their entirety to deliver to the user. And again, we’re still unless we update that content and it’s don’t treat it like a frozen snapshot in time, we’ll still have those content debt problems. But it’s becoming a bigger, bigger, a much bigger problem now. It wasn’t as big a problem when we put out chatbots. And the chatbots we’ve been building, what, for three, you know, two, three, four years now. And, you know, everybody celebrated, they popped the corks, you know, we can deflect X amount percentage of support cases. They can self-service. And I always talk about the precision paradox that once you reach a certain ceiling, it gets really hard to increment and get above that 70%, 80%, 85%, 90% window. And as you get closer and better, the tolerance for being wrong goes down like a rock. And you now have a real big problem. So how do we do these guardrails to be more deterministic, to mitigate the probabilistic risk that we have and reality that we have? The problem is that people are still looking for fast and quick, not right. When I say right, I mean the building out of things like ontologies and leveraging our taxonomies that we labored over with all of that metadata that never even gets into the vector database because they strip it all away in addition to shredding it up. So if we don’t start building those things like knowledge graphs and retaining all of that knowledge, it’s even… now we’re compounding the problem. Now we have debt, and we have no way to fix the debt. And now when we get into the new world of agentic workflows, which is the true bleeding edge right now, when you have sequences of both agentic and agentive, and the difference between those two, by the way, is agentic is autonomous. There’s no human doing that task. It’s just doing it. And then agentive, which is a human in the loop, which is helping there. When you’ve got a mix of agentive and agentic processes in a business workflow, now you’ve got to worry about what happens if I get something wrong early in the chain of sequence in that workflow. And this doesn’t apply to just documentation, by the way. We’ll be seeing companies taking very complex workflows in finance and in marketing and in business planning and reporting and mapping out this is the workflow our humans do. And there’s hundreds, if not more steps and many roles involved in those workflows. And as we map those out and say, where can we inject AI, not as just individual tools, like just separately using a large language model or separately using a single agent, but stringing them together to automate a complex business workflow with dependencies upstream and downstream. How are we going to survive and make this work? And I think that’s why you saw the MIT study had come out where they said, you know, roughly only 5% or so of AI projects are succeeding. And I think that’s because we did the easy stuff first. We did the chatbots and they could be lossy in terms of accuracy. But when you now, when you get to these agenda workflows that we’re building, literally coding as we speak, now you’re facing a whole different experience and ballgame where precision and currency really matters. SO: Yeah, and I think I mean, we’ve really only scraped the surface of this. Both of the articles that you’ve mentioned, the one that I started with and the one that you mentioned in this context, we’ll make sure we get those into the show notes. I believe they are on your is it Medium? On your website. So we’ll get those links in there. Any final parting words in the last? I don’t know. Fifteen seconds or so. MI: No, that’s good. I want to give, I want to tell you the good news and the bad news for tech doc professionals. What I’m seeing in the industry hurts me. I think there’s a lot of excuse right now, not just in the tech doc space, but in all jobs where we’re seeing AI being used as an excuse to make business decisions, to scale back. It may take some time until the impact of some poor business decisions that are being made will reflect themselves, but there’s going to be reality that hits. And the question is, is how do we navigate the interim? I’m confident that we will. I’m confident that those of us that are building the AI, I feel like I’m evil and a savior at the same time. I’m evil because I’m building automation that can speed up and make people much more productive, meaning you need less people potentially. At the same time, I feel like we’re in a position when we do it, rather than an engineer that doesn’t even know the documentation space, we’re getting to redefine our space ourselves and not leave it to the whims of people that don’t understand the incredible intricacy and dependencies of creating what we know as high-quality content. So we’re in this tumult right now, I think we’re going to come out of it. I can’t tell you what that window looks like. There will be challenges through doing that, but I would rather see this community define their own, redefine their own future in this transformation that is unavoidable. It’s not going away. It’s going to accelerate and get more serious. But if we don’t define ourselves, others will. And I think that’s the message I want our community to take away. So when we go to conferences and we show what we’re doing and we’re open and we’re sharing all the stuff that we’re doing, that’s not, hi, look at us. That’s you come back to the next conference and the next webinar and show us what you took from us and made better and helped shape and mold that transformative industry that we know as knowledge and content. And I’m excited because I want to celebrate every single advance that I see as we share. And I think it’s incumbent upon us to share and be vocal. And I think when I write my articles, they’re aimed at not only our own community, they’re aimed at the executives and technologists themselves to educate them, so that if we don’t do it, who will? And it does fall on all of us to do that. SO: I think I’m going to leave it there with a call for the executives to pay attention to what you are saying, and some of the rest of this community, many of the rest of this community are saying. So, Michael, thank you very much for taking the time. I look forward to seeing you at the next conference and seeing what more you’ve come up with. And we will see you soon. MI: Thank you very much. SO: Thank you. Conclusion with ambient background music CC: Thank you for listening to Content Operations by Scriptorium. For more information, visit Scriptorium.com or check the show notes for relevant links. Want more content ops insights? Download our book, Content Transformation. The post Futureproof your content ops for the coming knowledge collapse appeared first on Scriptorium.

November 3, 202527 min

The five stages of content debt

Your organization’s content debt costs more than you think. In this podcast, host Sarah O’Keefe and guest Dipo Ajose-Coker unpack the five stages of content debt from denial to action. Sarah and Dipo share how to navigate each stage to position your content—and your AI—for accuracy, scalability, and global growth. The blame stage: “It’s the tools. It’s the process. It’s the people.” Technical writers hear, “We’re going to put you into this department, and we’ll get this person to manage you with this new agile process,” or, “We’ll make you do things this way.” The finger-pointing begins. Tech teams blame the authors. Authors blame the CMS. Leadership questions the ROI of the entire content operations team. This is often where organizations say, “We’ve got to start making a change.” They’re either going to double down and continue building content debt, or they start looking for a scalable solution. — Dipo Ajose-Coker Related links: Scriptorium: Technical debt in content operations Scriptorium: AI and content: Avoiding disaster RWS: Secrets of Successful Enterprise AI Projects: What Market Leaders Know About Structured Content RWS: Maximizing Your CCMS ROI: Why Data Beats Opinion RWS: Accelerating Speed to Market: How Structured Content Drives Competitive Advantage (Medical Devices) RWS: The all-in-one guide to structured content: benefits, technology, and AI readiness LinkedIn: Dipo Ajose-Coker Sarah O’Keefe Transcript: Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. SO: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction Sarah O’Keefe: Hey, everyone. I’m Sarah O’Keefe and I’m here today with Dipo Ajose-Coker. He is a Solutions Architect and Strategy at RWS and based in France. His strategy work is focused on content technology. Hey, Dipo. Dipo Ajose-Coker: Hey there, Sarah. Thanks for having me on. SO: Yeah, how are you doing? DA-C: Hanging in there. It’s a sunny, cold day, but the wind’s blowing. SO: So in this episode, we wanted to talk about moving forward with your content and how you can make improvements to it and address some of the gaps that you have in terms of development and delivery and all the rest of it. And Dipo’s come up with a way of looking at this that is a framework that I think is actually extremely helpful. So Dipo, tell us about how you look at content debt. DA-C: Okay, thanks. First of all, I think before I go into my little thing that I put up, what is content debt? I think it’d be great to talk about that. It’s kind of like technical debt. It refers to that future work that you keep storing up because you’ve been taking shortcuts to try and deliver on time. You’ve let quality slip. You’ve had consultants come in and out every three months, and they’ve just been putting… I mean writing consultants. SO: These consultants. DA-C: And they’ve been basically doing stuff in a rush to try and get your product out on time. And over time, those sort of little errors, those sort of shortcuts will build up and you end up with missing metadata or inconsistent styles. The content is okay for now, but as you go forward, you find you’re building up a big debt of all these little fixes. And these little fixes will eventually add up and then end up as a big debt to pay. SO: And I saw an interesting post just a couple of days ago where somebody said that tech debt or content debt, you could think of it as having principle and interest and the interest accumulates over time. So the less work you do to pay down your content debt, the bigger and bigger and bigger it gets, right? It just keeps snowballing and eventually you find yourself with an enormous problem. So as you were looking at this idea of content debt, you came up with a framework for looking at this that is at once shiny and new and also very familiar. So what was it? DA-C: Yeah, really familiar. I think everyone’s heard of the five stages of grief, and I thought, “Well, how about applying that to content debt?” And so I came up with the five stages of content debt. So let’s go into it. I’m not going to keep referring to the grief part of it. You can all look it up, but the first stage is denial. “Our content is fine. We just need a better search engine. We can actually put it into this shiny new content delivery platform and it’s got this type of search,” and so on and so forth. Basically what you’re doing is you’re ignoring the growing mess. You’re duplicating content. You’ve got outdated docs. You’re building silos, and then you’re ignoring that these silos are actually getting even further and further apart. No one wants to admit that the CMS or whatever system, bespoke system that you’ve put into place, is just a patchwork of workarounds. This quietly builds your content debt until, actually the longer denial lasts, the more expensive that cleanup is. As we said in that first bit, you want to pay off the capital of your debt as quickly as possible. Anyone with a mortgage knows that. You come into a little bit of money, pay off as much capital as you can so that you stop accruing that debt, the interest on the debt. SO: And that is where when we talk about AI-based workflows, I feel like that is firmly situated in denial. Basically, “Yeah, we’ve got some issues, but the AI will fix it. The AI will make it all better.” Now, we painfully know that that’s probably not true, so we move ourselves out of denial. And then what? DA-C: There we go into anger. SO: Of course. DA-C: “Why can’t we find anything? Why does every update take two weeks?” And that was a question we used to get regularly where I used to work at a global medical device manufacturer. We had to change one short sentence because a spec change and it took weeks to do that. Authors are wasting time looking for reusable content if they don’t have an efficient CCMS. Your review cycles drag through because all you’re doing is giving the entire 600-page PDF to the reviewer without highlighting what’s in there. Your translation costs balloon and your project managers or leadership gets angry because, “Well, we only changed one word. Can’t you just use Google Translate? It should only cost like five cents.” Compliance teams then start raising flags. And if you’re in a regulated industry, you don’t want the compliance teams on your back, and especially you don’t want to start having defects out in the field. So eventually, productivity drops, your teams feel like they’re stuck. And the cracks are now starting to show across other departments and you’re putting a bad name on your doc team. SO: Yeah. And a lot of this, what you’ve got here, is the anger that’s focused inward to a certain extent. It’s the authors that are angry at everybody. I’ve also seen this play out as management saying, “Where are our docs? We have this team, we’re spending all this money, and updates take six months.” Or people submit update requests, tickets, something, the content doesn’t get into the docs, the docs don’t get updated. There’s a six-month lag. Now the SOP, the standard operating procedure, is out of sync with what people are actually doing on the factory floor, which it turns out, again, if you’re in medical devices, is extremely bad and will lead to your factory getting shut down, which is not what you want generally. DA-C: Yeah, it’s not a good position to be in. SO: And then there’s anger. DA-C: Yeah. SO: “Why aren’t they doing their job?” And yet you’ve got this group that’s doing the best that they can within their constraints, which are, as you said, in a lot of cases, very inefficient workflows, the wrong tool sets, not a lot of support, etc. Okay, so everybody’s mad. And then what? DA-C: Everyone’s mad, and eventually, actually this is a closed little loop because all you then do is say, “Okay, well, we’re going to take a shortcut,” and you’ve just added to your content debt. So this stage is actually one of the most dangerous of the parts of it because all you end up trying to do without actually solving the problem is just add to the debt. “Let’s take a shortcut here, let’s do this.” The next stage is now the blame stage. “It’s the tools. It’s the process. It’s the people.” These here and then you get calls of technical writers or, “Well, we’re going to put you into this department and we’ll get this person to rule you with this new agile process,” or, “We’ll get you to be doing it in this way.” The finger-pointing begins. Tech teams will blame the authors. Authors will blame the CMS. Leadership questions the ROI of the entire content operations team. This is often where organizations see that we’ve got to start making a change. They’re either going to double down and continue building that content debt or they start looking for a scalable solution. SO: Right. And this is the point at which people look at it and say, “Why can’t we just use AI to fix all of this?” DA-C: Yep, and we all know what happens when you point AI at garbage in. We’ve got the saying, and this saying has been true from the beginning of computing, garbage in, garbage out, GIGO. SO: Time. DA-C: Yeah. I changed that to computing. SO: Yeah. It’s really interesting though because the blame that goes around, I’ve talked to a lot of executives who, and we’re right back to anger too, it is sort of like, “We’ve never had to invest in this before. Why are you telling us that this organization, this group, this tech writers, content ops,” whatever you want to call it, “that they are going to need enterprise tools just like everybody else?” And they are just halfway astounded and halfway offended that these worker bees that were running around doing their thing… DA-C: Glorified secretaries. SO: Yeah, that whole thing, like, “How dare they?” And it can be helpful, sometimes it is and sometimes it isn’t, to say, “Well, you’ve invested in tools for your developers. You wouldn’t dream of writing software without source control, I assume,” although let’s not go down the rabbit hole of vibe coding. DA-C: Let’s not go down that one. SO: And the fact that there are already people with the job title of vibe coding remediation specialist. DA-C: Nice. SO: Yeah. So that’s going to be a growth industry. DA-C: That’s what, if you can get it. SO: But this blame thing is we are saying, “This is an asset. You need to invest in it. You need to manage it. You need to depreciate it just like anything else. And if you don’t invest properly, you’re going to have some big problems.” And to your- DA-C: A lot of that- SO: Yeah, they don’t want to do it. They’re horrified. DA-C: Yeah. A lot of that comes to looking at docs departments as cost centers. They’re costing us money. We’re paying all these people to produce this stuff that people don’t read. The users don’t want to. But if you look at it properly, deeply, the documentation department can be classed as a revenue generator. What are your sales teams pointing prospects at? They’re pointing at docs. Where are they getting the information about how things work? They’re pointing at the docs. What are you using? Especially if you’re having people looking through trying to find a solution? I know I do this. I go and look at the user manuals. And first thing that I want to see in there that is properly written, if I see something that does not describe the gadget or whatever I’m trying to buy properly, then I’m like, “Well, if you’ve taken shortcuts there, you’ve probably done the same with the actual thing that I’m going to buy.” So I’m going to walk away. Reducing costs for online centers. If your customers can find the information very quickly that describes the exact problem that they’re trying to solve, then you’ve got fewer calls to your online help center. And then while escalating onto the next person, because the level, I don’t know how this goes, level three, two, one, let’s say the level three is the lowest level, if that person can not find the information that is true, clear, one source of truth, then they’re going to escalate it onto that person who you’re paying a lot more, is at that level two, that person can’t find it, moved on. So it’s basically costing you a lot of money not to have good documentation. It’s a revenue generator. SO: So my experience has been that the blame phase is perhaps the longest of all the phases. DA-C: Yeah. SO: And some organizations just get stuck there forever and they blame different people every year. I’ve also, I’m sure you’ve seen this as well, we were talking about reorganizing. “Well, okay, the tech writers are all in one group. Let’s burst them out and put them all on the product team.” DA-C: Yes. SO: “So you go on product team A and you go on product team B and you go on product team C.” And I talk to people about this and they say, “This is terrible and I don’t want to do it.” I’m like, “It’s fine, just wait two years.” DA-C: Yeah. SO: Because it won’t work, and then they’ll put them all back together. Ultimately, I’m not sure it matters whether they’re together or apart because we fall into this sort of weird intermediate thing. What matters is that somebody somewhere understands the value, to your point, and isn’t making the investment. I don’t care if you do that in a single group or in a cross-functional matrix, blah, blah, but here we are. All right. So eventually, hopefully, we exit blame. DA-C: And then we move into acceptance. SO: Do we? DA-C: “Okay, we need a better way to manage that.” And this is like when people start contacting you, Sarah, it’s like, “I’ve heard there’s a better way to manage this. Somebody’s talked to me about there’s something called the component content management system or the structured content,” and all of this. So teams start to acknowledge, one, that they’ve got debt and that debt is growing. Then they start auditing that content and then really seeing that, “Oh, well, yes, things are really going bad. We’ve got 15 versions of this same document living in different spaces in different countries. The translations always cost us a bomb.” So leadership then starts budgeting for a transformation. This is where they then start doing their research to find structured content, competent reuse, they enter the conversation. If they look at their software departments, software departments reuse stuff. You’ve got libraries of objects. Variables is the simplest form of that reuse. And they’ve been using this for years. And so, “Well, why aren’t we doing this? Oh, there’s DITA, there’s metadata. We can govern our content better. We can collaborate using this tool.” So there is a better way to do this. And then we know what to do. SO: I feel like a lot of times the people that reach out to us are in stage four, they’ve reached acceptance, but their management is still back in anger and bargaining and denial and all the rest of that. DA-C: They’re still blaming and trying to find a reason. SO: Yeah, blaming and all of it, just, “How dare you?” All right, so we acknowledge that we have a problem, which I think is actually the first step in a different step process, but okay. DA-C: Yeah. SO: And then what? DA-C: And then there’s action. Let’s start fixing this before it gets totally out of control, before it gets worse. Then they start investing in structured content authoring platforms like Tridion Docs, I work for RWS, I’ve got to mention it. They start speaking with experts, doing that research, listening to their documentation team leaders, speaking with content strategists to define what the content model is, first of all, and then where can we optimize efficiency by having a reuse strategy? A reuse without a strategy is just asking for trouble. You’re basically going to end up duplicating content. And then you’ve got to govern how that is used. What rules have you got in place and what ways have you got to implement those rules? The old job of having an editor used to work in the good old days where you’d print something off and somebody would sign it off and so on and so forth. Now, we’re having to deliver content really quickly and we’re using a lot of technology to do that. And so, well, you need to use technology to govern how that content is being created. Then your content becomes an asset. It’s no longer a liability. This is where that transformation happens, and then you start paying down your content debt. You’re able to scale the content that you’re creating a lot faster without raising the number of the headcount, without having to hire more people. And if you want to then really expand, let’s say, because you’ve got this really great operation now and you’re able to create that content that takes hours and not weeks, then you’re able to expand your market. You’re able to say, “Okay, well, now we’re going to tackle the Brazilian market. Now, we can move into China because they’ve got different regulations.” Again, I speak a lot on the regulatory side of things. That’s where I passed most of my time as a technical writer. Having different content for different regulatory regimes and so on is just such a headache where you don’t have something that is helping you with that structure, applying structure to that content, applying rules to that content, making sure that your workflows are carried out in the way that you set it out six months ago and people have changed and so on and they’re not doing their own thing again. If your organization is stuck at stages one to three, as I just mentioned it, it’s basically time to move. SO: Yeah, I think it’s interesting thinking about this in the larger context of when we talk about writers, the act of writing, right? DA-C: Yes. SO: Culturally, that word or that process is really loaded with this idea of a single human in an attic somewhere writing the great American or French or British novel, writing a great piece of literature or creating a piece of art on their own, by themselves, in solitude. And of course, we know that technical writing- DA-C: Starting at A and going all the way to Z. SO: And we know that technical writing is not that at all, but it does really feel as though when we describe what it means to be a writer or a content creator in a structured content environment, it is just the 180 degree opposite of what it means to be a writer. It’s not the same thing. You are a creator of these little components. They all get put together. We need consistent voice and tone. You have to kind of subordinate your own voice and your own style to the corporate style and to the regulatory and to all the rest of it. And so it’s just this sort of… I think we maybe sometimes underestimate the level of cultural push and pull that there is between what it is to be a writer and what it is to be a technical writer. DA-C: Yes. SO: Or a technical communicator or content creator, whatever you want to call that role. Okay, so we’ve talked about a lot of this and then we’ve not talked a lot about AI, but a big chunk of this is that when you move into an environment where you are using AI for end users to access your content, so they go through a chatbot to get to the content or they’re consulting ChatGPT or something like that, and asking, “Tell me about X.” All of the things that you’re describing in terms of content debt play into the AI not performing, the content not getting in there, not being delivered. So what does it look like? What are some of the specifics of good source content, of paying down the debt and moving into this environment where the content is getting better? What does that mean? What do I actually have to do? We’ve talked about tools. DA-C: Yeah. So first, you’ve got to understand how AI accesses content and how large language models get trained. AI interprets patterns as meaning. If your content deviates from pattern predictability, then you’re going to get what we call hallucinations. And so asking the ChatGPT without having it plugged as an enterprise AI thing where you’ve really trained it on your own content, you get all sorts of hallucinations. Basically, they’ve taken two PDFs that have similar information, but two different conclusions. And so you’re looking for a conclusion in document A, but ChatGPT has given you the one in B. And it’s mixed and matched those because it does not know how one bit of information relates to the other. So good source content needs to be accurate. Your facts are correct. They reflect the current state of the product or subject. It needs to be kept up to date. You need to have single copies of it, that’s what we talk about, a single source of truth. You can not have two sources of truth. It’s either black or it’s white. There are no gray zones with AI, it will hallucinate. You’ve got to have that consistency in style and tone. How do you get that? Well, you’ve got the brand and the way we speak. In French, you would say, “Do you vouvoie or do you tutoie?” Do use the formal voice, formal tone, or do you speak like you’re speaking with your friends? How do you enforce some of that? Well, you can use controlled terminology. These are special terms that you’ve defined, a special voice. But the gold part of it is having that structured formatting and presentation. There’s always a logical structure and sequence to the way that you present that information. Your heading, subheading, steps, lists, are always displayed in the same way. You’ve defined an information architecture to then give that pattern. And the way AI then understands or creates relationships with those patterns is from the metadata that you’re adding onto it. And so good source content is accurate, up to date, consistent in style and tone, uses control terminology, has structure in formatting. Forget the presentation because that you put on the end of things, in that what it looks like, how pretty it is. But the presentation in terms of I always start with a short description and then I follow up with the required tools. And then I describe any prerequisites, and that is the way every one of my writers are contributing towards this central repository of knowledge, this single repository of knowledge. And you can do that as well if you’ve got a great CCMS by using templates, building templates into that CCMS so that it guides the author. And the author no longer has to think about, “Oh, how is this going to look? Should I be coloring my tables green, red, blue? Should they be this wide?” They’re basically filling in a template form. And some of the standards that we’ve developed like DITA allow you to do this, allow you to have a particular pattern for creating that information and the ability to put it into a template which is managed by your CCMS. SO: Yeah, and that’s the roadmap, right? We talk about how as a human, if I’m looking at content and I notice that it’s formatted differently, like, “Oh, they bolded this word here but not there,” and I start thinking, “Well, was that meaningful?” DA-C: Yeah. SO: And at some point, I decide, “No, it was just sloppy and somebody screwed up and didn’t bold the thing.” But AI will infer meaning from pattern deviations. DA-C: Yeah. SO: And so the more consistent the information is in all the levels that you’ve described, the more likely it is that it will process it correctly and give you the right outcome. Okay, so that seems like maybe the place that we need to wrap this up and say, folks, you have content debt. Dipo is giving you a handy roadmap for how to understand your content debt and understand the process of coming to terms with your content debt, and then figuring out how and where to move forward. So any closing thoughts on that before we say good luck to everybody? DA-C: Basically before, or, I mean, most enterprises today have already jumped on the AI bandwagon. They’re already trying to put it in, but at the same time, start taking a look at your content to ensure that it is structured and has semantic meaning to it. Because the day that you then start training your large language model on that, if you’ve not built those relationships into it, it’s like teaching a kid bad habits. They’re going to just continue doing it. It’s basically train your AI right the first time by having content that is structured and semantic, and you’ll find your AI outcomes are a lot more successful. SO: So I’m hearing that AI is basically a toddler? Okay. Well, I think we’ll leave it there. Dipo, thanks, it’s great to see you as always. DA-C: Thanks for having me. SO: Everybody, thank you for joining us, and we’ll see you on the next one. Conclusion with ambient background music CC: Thank you for listening to Content Operations by Scriptorium. For more information, visit Scriptorium.com or check the show notes for relevant links. Want more content ops insights? Download our book, Content Transformation. The post The five stages of content debt appeared first on Scriptorium.

Is this your show?

Claim this listing to keep it up to date, reach guests who want to pitch you, and manage bookings with Guestify.

Claim this listing

More Business podcasts