Content Operations

Hosted by Scriptorium - The Content Strategy Experts

BusinessInterviews guests

Website RSS feed

Episodes

204

Latest episode

Oct 2026

Language

EN-US

About the show

The Content Operations podcast from Scriptorium delivers industry-leading insights for scalable, global, AI-optimized content.

Listen to episodes

60 recent

October 26, 202628 min

From clay tablets to LLMs: Applying communication theory to AI-optimized content ops

In the second episode of The Uninvited Author podcast series, authors Sarah O’Keefe and Carlos Evia examine how foundational communication theories help structure modern content operations. They unpack how AI adds new intentionality to creating and delivering content, providing the theoretical groundwork creators and leaders need to manage workflow risks and build sustainable AI-optimized content ops. Sarah O’Keefe: I t’s not that you have to go talk to your management or your CEO or whoever you’re dealing with about Umberto Echo, although that might work. It is that this needs to be in the back of your mind so that when somebody pushes back with something that is clearly nonsensical and is just simply not going to work, you can explain why. You have the foundation that allows you to move forward and say, “Okay, so as we’re building out the system, as we’re bringing AI into the equation, here’s where it’s going to do well and here’s where it’s risky.” Related links: Terje Hillesund Marshall T Poe Umberto Eco Daniel Dennett Digital sovereignty in the age of AI From ad hoc to autonomous: The AI content ops maturity model Check out other episodes in The Uninvited Author podcast series LinkedIn: Sarah O’Keefe Carlos Evia Transcript: Disclaimer: This is a machine-generated transcript with edits. Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction SO: Hey everybody, welcome back. This is Sarah O’Keefe again, and I am here once again with Carlos Evia. CE: Hello. SO: And this is part two of our podcast series about the book that we’ve just released called The Uninvited Author. And we want to talk in this episode about some of the core concepts that we used in breaking down how we look at the process of creating, managing, and delivering content and what that looks like. So Carlos, what were some of the key concepts that we that we brought into this discussion? CE: Well, we were looking at things that were a bit broader than just one type of content. We talk about content a lot in our lives. I mean, you in your work and you do as well in my work from slightly different perspectives, but with a core of what unites our experiences is technical content or enabling content, that content that at the end of the day, you’re gonna read not for entertainment, but when you need assistance, you need information to complete or perform a task. But we wanted this book to be a bit broader than just specific cases about technical or enabling content. So instead of just jumping to the life cycle of technical content, that many versions of that exist. I think you were the one who found this text cycle that came from a professor in Norway. SO: Yeah. CE: And my Norwegian is pretty bad, so I will attempt to pronounce Terje Hillesund. I hope that is correct. We might get some tomatoes thrown at us for mispronouncing this media studies professor’s name. and he does research from where we can tell he’s still active. He does research on digital publishing, especially when it comes to electronic books and EPUBs. So he proposed a text cycle that had some very clear stages. What he calls them stages, right? Or phases? What does he? I think he’s stages. SO: I’m not actually sure. And also it probably sounded better in the original Norwegian. CE: We need to do an AI translation of this episode into Norwegian and see how it goes. So those let’s call them stages, this is our adaptation of it, are pretty much life cycle like, but they’re not just specific about one type or one combination of elements of content. So they’re very broad for any type of communication that is particularly being written, but not just delivered in text. We also applied it to multimedia and other digital approaches to text. And we have a whole section in the book saying: what do we mean by text? It’s not just words written on a page, but it can be audio and visual content, anything that is contained in bits and is delivered through electronic channels, even if you at the end of the day get it in a piece of paper that had to come from a file that was printed. So do you want to talk about the stages or should I talk about the stages, whatever you wanna do? SO: So breaking down the stages. As Carlos is saying, it’s not just text, words on a page per se. and in fact, you could apply the so-called text cycle to spoken communication, which kind of makes me twitch a little bit. CE: Mm-hmm. SO: If you think about this, and of course we’ve got a bunch of images for this in the book, but there are six phases to this text process or cycle. The first one is write, which is really having the idea. Then there’s produce, which is something like writing it down. There’s storing, which, you know, in a digital content world would be parking it in a file or a database. There’s representation. To represent it is to format it and make it look a certain way. And then you distribute it, and finally you read it, or you consume it. Now, most of us, if you’ve ever thought about this, which most regular people don’t have any need to, think about writing and reading. There’s an author; they write some stuff, and then there’s a reader, and they read the stuff. But as you start breaking down this process and thinking about what really goes into that, if you go back to the earliest communication, so now we’re talking about spoken communication. Well, the process of writing, of creating the idea, of packaging that idea of storing it, of representing it, which is to say, to use your words to speak it. And then you use your words also, your voice to distribute. And then you’ve got the reader, the consumer, they’re sitting right there listening to you. And so this entire six-phase process is really just two, right? It’s two actions. I speak, you listen. And then all these other pieces that are represented in this text cycle are all embedded in that little two-step process. The minute you introduce actual writing, the process of putting something on a clay tablet or piece of paper, which of course came much, much later, well, now I have an idea in my head, and then I take a pen or a stylus or a chisel or something, and I actually encode that information onto a tablet or we’ll go with paper, right? We put it on a piece of paper. So I’ve now produced it using visible marks, and I’ve stored it on that piece of paper, and I’ve represented it on that piece of paper. But the key thing is that I have not yet distributed it because I haven’t distributed the idea. Now if I hand you the piece of paper and you drive across town and hand it to somebody else, that’s distribution, and then they read it. SO: And so you can see this process of decoupling the phases starting to happen. And the key thing is when you look at digital content, we decouple more pieces. It gets more and more fragmented, and there are more and more people involved, more and more different roles involved in the process of having an idea, encoding that idea in some way and then representing it in some way and delivering it downstream to the person who finally consumes it. And so that was kind of the fundamental starting point for this book was to look at this text cycle and ask the question of: okay, well, what does it look like to have this text cycle in typically now a digital workflow? And now we introduce AI. And when you look at the six phases that we have here, it turns out that you can use AI in different ways at every point in the text cycle. And I think this is why, as authors, as content creators, looking at AI is so difficult because there are lots and lots of different things we can do with it. It’s not just go online, open up a chat bot and ask a question, which is how most people, the vast majority of everybody, that is how they interact with AI. But as an author, there’s a whole bunch of other stuff going on. so did you want to talk a little bit about what it looks like at some of the points in this text cycle to introduce AI. CE: Yeah, but I think what something that I would like to mention is that we didn’t just use that one lens to analyze how content and communication has evolved based on this life cycle, the the text cycle, to get us to where we are right now working with artificial intelligence. So instead of that one lens, we combine it with other perspectives from different disciplines and particularly from history, we borrow the work of Marshall T. Poe about the ages of human communication. So Poe, who by the way is from Alabama, not from Norway, the develop a framework for studying human communication across history. So Poe had five ages of communication, which include the age of speech, the age of manuscripts, the age of print, the age of audio and visual media, and the age of the internet. So what we did is to put a Hillison’s text cycle across the ages that Poe documents in his model to expand that and get to propose a new age of human communication that we surprised called the age of human communication with AI. And we were able to I want to say map the stages of the text cycle into what communication looks like now with artificial intelligence. And we start the book in very, I don’t want to say naive, but conservative ways of presenting artificial intelligence as an assistant with a, you know, we understand that people are using AI across the stages of the text cycle as an assistant for writing, as an assistant for producing, storing, and eventually as an assistant for reading, because readers are using AI chatbots and AI agents to consume content. And it gets to the point in which as the book progresses and you go to different chapters, and the title of the book I will remind the listeners is The Uninvited Author, because it’s precisely based on the idea that we have the tech cycle with humans, and humans have traditionally been the developers and readers of the tech cycle using machines and tools at different stages of the text cycle, but at the ends, here comes the little cart again outside. SO: Yeah. Is this what this is? The uninvited cart. CE: Buy a little cart. It’s the uninvited little cart. So at the end and at the beginning of the tech cycle across all of the ages of human communication in history, there’s a human being and a human being. But now it’s a human being and/or a machine. It’s a fancy silicon machine and not just a mechanical machine, but they are machines. And that has really modified the approaches for developing and interpreting content in this new age of human communication with artificial intelligence. And that’s why we took the liberty of proposing our new age following the model of Marshall T Poe. And a funny anecdote is that I’m gonna reveal all the dirty laundry. When we were writing the book, I heard a rumor that we forgot to include one of the listed stages of the tech cycle, and also one of our very effective and kind reviewers noticed that one of our invented stages of the era, the age of human communication with AI was wrong and we had to rewrite it. But I promise you that the version that was published incorporates all those comments and if you identify any other gaps. We blame it on the AI. SO: The worst part about all of this is that because we really limited our use of AI in writing this book, we don’t get to blame the AI for anything. So we’d like to, but sadly it’s all us. So the text cycle, and then we’ve got Marshall Poe. And then, you know, the third thing is because like Norway and Alabama I mean, what’s the average of Norway and Alabama? It’s Italy, obviously. And so we brought in Umberto Echo, who well, most of us probably know about him because he wrote a bunch of very literary detective novels that involved medieval friars, and that was super fun. But he also wrote about communication and fundamentally that as a writer, I am trying to communicate. I have a certain intent in what I’m trying to do. And the reader has maybe a different intent in using a document or in using the content that I wrote. And then the work itself, the document, the podcast, the thing, the article of communication has its own intent. And of course, what we’re finding now is that you add to that the fourth intention, which is the AI, the algorithm sitting inside the AI. Now, AI is not people. CE: Not people. SO: AI is perhaps made of people or made of people’s ideas, but AI is not is not people. And it doesn’t have intent in the same way that a human has motivation and some other things. But algorithms have input, and they have processing, and that leads to certain kinds of outputs and outcomes. And so an algorithm can have its own intent in the sense that there’s a certain kind of result that it is pushing towards. CE: Yeah. And one of the components that I want to mention, and the book is kind of a geeky book because we write about topics related to technology, and that’s true. But it has a very strong humanities core. And humanity is not just… SO: And really bad puns. CE: And really bad jokes about people with weird names, but that’s a topic for another episode. I’m gonna completely go in that direction in another episode, I promise you, because at one point the book started as a love story. There are going to be two characters, Margarita and Cosmo, who fall in love, but then we went in a different direction because it was not a fiction book. Anywho, the humanities base of what we analyze in the book, and this is where I want to really emphasize: this is not an AI slop book that was written by AI. It was written by people and incorporating traditions across different disciplines. So we have the historical component; we have the history from Poe, we have the media studies component from Hillesund. And we’re gonna get, and Sarah already talked about it to the semiotic component from Umberto Echo. But from a philosophical perspective, Daniel Dennett, who died a couple of years ago, was a philosopher from Massachusetts. And one of the many things that Dennett wrote in his career was a theory of interpretation that says that we as human beings, when we encounter a phenomenon that we don’t understand, we tend to treat it as if it were a being that has intentions, that has intent. So combining the perspectives of Dennett with what Echo wrote about the limits of interpretation and how readers have their intentions when they encounter a text, and here again we go back to text being not just words, that come with intentions from the human authors. Echo was very against the idea of the school of deconstruction; I’m getting super geeky on this. I promise you, I’m not gonna continue on this path. Was a very strange school of thinking that said that anything goes. And if you take a book, you can interpret it to be whatever it is that you want it. If you find a poem, you can interpret it to be a recipe for making cake. And I’m making very stupid examples here. I apologize. Echo was very against that. And Echo said, yes, go ahead and interpret things in different ways, but there is a point in which your interpretation of a text is going to be silly and nonsensical. So that’s why he proposed the text itself has its intentions that have the primary purpose of avoiding or preventing that people are going to interpret a chicken recipe to be a love letter. So that’s nonsensical. We don’t go there. If you read the chicken recipe, it has steps. It has ingredients, it has prerequisites, and it has pictures of a chicken that is not a love letter unless you love chickens. But we’re not getting into that by any means. SO: Too late. CE: And that’s what we came up with our contribution of the fourth layer of intentions, which, like we said, if you take a look at how the text cycle in the age of human communication with AI now includes AI agents, AI tools as readers and as authors, they come with their own intentions. And their intentions, not human intentions, are a hodgepodge combination of the training materials that the agents came from, the version of the model that you’re using. If your model is allowed to go out and perform web searches on the fly. If it keeps any record of interactions with the reader. So those are intentions that get imprinted into the text cycle in the age of human communication with AI. And that’s where we really created this prism of different approaches coming from the work that we do in technical communication, but also based on bigger conversations about history and communication studies and philosophy without making it sound too nerdy. I promise you, if you have read the book, you will see that there’s some nerdy elements, no doubt. But we try to make it accessible for our peers, our colleagues, managers in our industry, and not just in technical communication, but people who work across different sectors of content. And have to acknowledge, or have come to terms that machines are now being authors in the text cycles whether they invited them. SO: it’s important to understand some of this underlying theoretical work and some of these frameworks, because as you move into this as a content creator or as a leader, as somebody that’s responsible for some sort of content, if you don’t understand how these things interoperate and fundamentally how they break down and how you can and cannot break them down, then you don’t have any possibility of actually bringing your AI-based or AI-enabled content operations under control. So it’s not that you have to go talk to your management or your CEO or whoever you’re dealing with about Umberto Echo, although, depending that might work. It is that this needs to be in the back of your mind so that when somebody pushes back with something that is clearly nonsensical and is just simply not going to work, that you can explain why, that you have that foundation that then allows you to move forward and say, okay, so as we’re building out the system, as we’re bringing AI into the equation, here’s where it’s going to do well and here’s where it’s risky. These are the pieces that it will not do well, and you know why. So fundamentally, we weren’t that interested in the question of what can Claude do today, or what kinds of ridiculous things can you make ChatGPT say if you feel like it. CE: Mm-hmm. SO: The fundamental question is much more how are large language models put together. And what kinds of things are they going to be good at and bad at? And how do you bring that into a workflow? And how do you understand the risks that you’re introducing and the benefits? So what’s the cost-benefit analysis of having an LLM in your in your process? And, you know, we touched on this at the very top of the podcast when we made some joke about it’s the AI’s fault, right? You know, if there’s a mistake, it’s the AI’s fault. Well, so first of all, it’s not because we didn’t really use much AI, but fundamentally and secondly, we don’t get to disclaim responsibility. if you’re in techcomm or e-learning or anything else and you put out a document and it was 100% AI-generated and then it’s wrong, the organization, I mean you may get to say, well, you made me use AI, it’s not my fault. CE: Mm-hmm. SO: But the organization doesn’t get to disclaim responsibility for the document because it was AI-generated. They’re going to have significant product liability. And I believe, on no evidence whatsoever, but I believe that when the product liability lawsuits come, and they will, right? The organization that goes up in front of a jury and says it’s the AI’s fault is going to lose. And I think they’re going to lose more than they would if they said, well, our humans did our we tried our best and we made a mistake, as opposed to, yeah, we offloaded it onto the AI. We didn’t read the results. We fired all the people that are creating content. And we’re sorry that you were horrifically injured in some way. But we weren’t sorry enough to do the work to prevent it, right? So there’s going to be huge product liability. And I think the jury sitting there is going to say, well you tried to save money, you tried to cut corners, and this is the result. And so I believe that ultimately the legal outcomes of excessive AI use are going to be you know, problematic enough to kind of reel people in. CE: Mm-hmm. SO: Now that’s a very US-centric perspective because in Europe there are different regulatory issues. But even in the US with our current general lack of regulation, I think you’re going to see that product liability lever applied to force organizations to do better. CE: I remember that when we were writing the book, we were in very early stages of the book, you posted to LinkedIn that we were writing the book, and you kinda opened it to people to propose if you were writing a book about AI and content, what would you want to see? And a few of the people who responded to that post were expecting concrete formulas, concrete prompts or recipes for creating agentic AI workflows or what should I ask my chatbot to have good content. And that’s not the book that we wanted to write. We wanted to write a book that will make people we hope think and consider those implications of using AI for creativity, for productivity, and like you said, for legal consequences of delegating everything to the artificial intelligence agents and expecting magic. And that’s what I keep going to. I keep saying that the problem is when people see AI as this oracle that is going to just magically solve all the problems without understanding how it works. And it is a monster. AI is applied to mean many different things. And we write about this in the book too. But also, it’s how what are the consequences of bringing AI as an invited component into our professional tech cycles. And by the way, going back to your LinkedIn activity, I think a couple of days ago you published something that said we are not anti-AI, we are anti-slop. And I think that’s something that is reflected in this book. We are not anti-AI. We have a whole section in the book that I think it presents in very good and honest detail how we use different tools that include artificial intelligence in the production of the book, but we were very cautious about avoiding perspectives of presenting AI slop, passing it as human products. There are so many books. If we go back to our main corner of technical communication, there are many books that have appeared on Amazon in the past few months about the DITA standard. That you can see that no humans were involved in the writing of those books. Somebody just prompted a chatbot and said, go fetch information about DITA. And that’s not the book that we wanted to create. This is not a book with concrete recipes on how to use things with AI. And it is not a book that we delegated to AI. It’s a book written by humans coming from different workplace, practitioners, and academic disciplines, asking questions about what does this mean and how do we go to the next stage and learn to live with these synthetic authors without losing our humanity and without getting in trouble. SO: And I hope that this book is valuable to the people that are going to read it because it was painful to write. I mean, it hurt. So I, you know, if I’m gonna go through this pain and suffering, then I wish for it to be useful. And if people look at it and say, you know, you could have prompted your way to this with AI, then I’m probably just gonna like curl up in the corner and cry. CE: Yeah. SO: So all right, on that cheery note, we will leave it there for this episode, and we will return and talk a little less about theory and a little more about fun things you can do with synthetic content and what that means to our operations. So thank you, Carlos, and I’ll see you on the next one. Bye. CE: Thank you. Bye bye. The post From clay tablets to LLMs: Applying communication theory to AI-optimized content ops appeared first on Scriptorium .

October 12, 202632 min

Bridging the gap between content design, content ops, and AI

Is AI the catalyst that will finally bring content design and content ops together? In this episode, Sarah O’Keefe speaks with Kate Agena, Vice President of Content Design at SAP, about why fluency is cheap while meaning remains expensive in an AI-driven world . Kate Agena: But users don’t care if content is coming from marketing, product, or whatever: they want whatever it is that they need at the time they need it. If everything is just kind of in one pile of information, that’s not going to come out correctly. So I think ops is just so incredibly important now, and content ops in particular are going to be far more important than they were. Related links: KateAgena.com Paz Parez: Context architecture Digital sovereignty in the age of AI From ad hoc to autonomous: The AI content ops maturity model LinkedIn: Sarah O’Keefe Kate Agena Transcript: Disclaimer: This is a machine-generated transcript with edits. Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction SO: Hi everyone, I’m Sarah O’Keefe. I’m here today with Kate Agena, who is Vice President of Content Design at SAP and also founder of the Content Design Leaders Membership Community. She has a PhD from Purdue, and in doing her PhD, she studied how tech writers could use iterative development processes to impact software design, otherwise known as, at least to me, basically waterfall versus agile. So she’s got some really interesting background as a tech writer and kind of started out there via I think information development at IBM and then landed in content design. So welcome, Kate. Kate Agena: Thank you so much, Sarah. Happy to be here. SO: Yeah, we’re glad to have you and to connect some of the maybe more technical content enabling content side of things to content design. our focus today is actually on that, on the fact that content design and content ops are related, or maybe should be, but somehow aren’t connected. And Kate has a theory that AI is actually going to for once be a force for good and maybe bring these disciplines together. So we should talk about that. But first, give us your definition, your short definition of content design. What is that? KA: Yeah, so for me, UX content design is a UX design practice, first of all. And it is the practice of designing the information and the language within a product experience so people can understand what’s happening, they can make decisions better, and they can accomplish whatever it is that they’re trying to do. SO: Yeah, and I think I would argue that content ops is the tooling and the processes that sit underneath that, that this content design process or discipline flows over or inside. so I was reading up on some of your stuff, and I’ve included Kate’s personal website in our show notes. One of the things that you said, and I’ve seen this in longer phrasing from other people, but you had a very succinct definition of this. You said fluency is now cheap and meaning is not. So, we’ve said for many, many years, that it is not true that anyone can write. However, it turns out that with AI, if you know, if writing counts as let the AI generate the text, then I guess it is actually true that anyone can write. The problem is that we evaluate content on fluency. If it’s poorly written, if it’s hard to understand, if it’s ungrammatical, we assume it’s low quality. But that isn’t true anymore because AI will always generate something that looks good, looks plausible. So it sounds good, it’s well written for values of the mechanics are correct, but it’s not necessarily accurate. So when you say that fluency is cheap because of AI, but meaning is not. What what is meaning and why is it still expensive? KA: Yeah. So I mean, for me, I think coming from the techcomm world where, you know, and from rhetoric where obviously I’m always worried about things like audience purpose and context. What is it that the person is needing? What are they trying to do? And then design really makes you think about psychology and you know, where is this in a user’s journey and maybe in the user’s day and additional, you know, a lot of different context around whatever the content is. And that emphasizes so much user research. You understand if you’re, you know, if you have experience doing content design, you have to like, you know, changing one phrase, changing one word can make a huge difference on, you know, even product success, but certainly on how a user reacts to that content. And so it’s, you know, it’s very different for an AI to just kind of generate a phrase that has the general meaning, supposedly, that you need in that piece of UI, than to really do all of the thinking about the human context that’s around that. And I think there’s a lot of discernment that happens that a content designer brings in, or even a technical writer brings in to documentation, maybe. As they’re kind of interpreting a lot of times what devs have put in, that any tech writer will tell you that they do a lot of work to get maybe specs or you know, JIRA information or a description that a developer gave them into a format that users can actually understand. And with content design, it’s the same that you really have a lot of human context that you need to take into account. And that’s a lot of discernment that I think. You know, you can’t give all of that context to the AI so easily. and if you try to, I think you it’s it’s gonna get lost in the context window at some point. SO: So then as we’re working through this and we’re saying, well, you know, writing and discernment and humans are more important than ever, you know, I had yet another note from a colleague who’s facing a layoff. And it’s pretty clear that at the individual level, people are losing their jobs because of AI. the AI can do it, we’re getting rid of our entire team. The AI can do it, you’re out. So there are lots of content people looking for work on sort of both sides of this fence that we’re talking about, but there are also new opportunities. So acknowledging that there’s huge disruption and a lot of pain, where do you see some potential and some new jobs and some new specializations evolving? KA: Yeah, I think there’s a lot. And I I feel like there should be kind of niches that people can find something that feels a little bit closer to their own skill set to move into. In content design, a lot of folks are moving into content systems, which is a little closer to content ops. Also, content ops, I think, should be something that people could be moving into more content strategies, you know, kind of ideas that we would have thought about as content strategy a while back. So there’s definitely the system side, which is just expanding because now we’re talking about things like SAP calls it a compositional design system that would allow a machine to use that design system to generate UIs on the fly based on user intent. And to do that, you need an awful lot of information to help that machine kind of go through a tree of decision-making. And find patterns. What are the good patterns for doing this specific thing? How do you understand what the user is actually looking to do? And then things that you know used to be a human would be involved, and so they would bring in all of their human knowledge. You need to encode that into components and things as well. So there’s, you know, kind of system-level voice and tone, personality, behavioral things, but also, you know, on the component level, there needs to be a pattern for what the copy should be on that specific component. So I think content systems is a huge new area. Where, you know, before I think we always wanted to put resource into that kind of work and there wasn’t any appetite for resourcing it. And now it’s kind of table stakes that we need to get there. If they want AI to do these things, we need to have a lot more effort put into content systems and governance. You know, it’s one thing to build it, and it’s another thing to keep it going. SO: A bad word. KA: So content systems, yes, content systems is a big one. A second big one for content designers is model design, or there’s a lot of different names going around for it now. Paz Paris called it model design when she first kind of identified it when she was at Google working on, actually, I think it was working on Bard at the time, before it was Gemini. And that’s work that content designers have been kind of really a natural fit for. So that’s things like system prompts and skills and evals and determining all of that stuff around kind of how the model will behave with folks. so model design is a second one, content engineering people call it, that kind of thing. So that’s a second big one, I think, for content designers. I don’t think static UI content design is going to fully go away, but I don’t think anyone will be doing exclusively that. I think there’s a new need that you have to have patterns and guidelines that fit your static UI, but you have to have all of those also articulated for the machines. And then everything has to go together. So I think on the content design side, that’s where most people are trying to learn skills. And then I see this additional area that I feel from my background would be ideal for folks with a tech writing background, which is, you know, you just have such a greater need for structured content and for kind of interpreting what maybe a developer or a data scientist or a PM or someone else, even a designer, articulates as this is the need, and we need this put into clear structured content. That the machine can read. And I think that there’s a big space for that. Because I also feel like there’s, you know, I mean, I see in SAP, we have a ton of data scientists, we have a huge amount of folks that are working on ontologies and knowledge graphs and things like that. But from the design perspective, I feel like coming from the user’s side of things, there’s probably a gap in between somewhere where there needs to be some kind of like interpretation layer that connects with how the user sees things to how the data sees things. And I have a feeling that, you know, that needs to be designed. It’s not, it’s not objective. And so I feel like there’s a space for designing that level, that’s a new level that I am not sure until we’re really trying these things and finding that we’re not quite connecting. I don’t think companies will realize they need that. But I have a feeling that’s going to come. And I think content folks are best equipped to do that. SO: So one of the things that sends me into sort of incandescent rage is, you know, we talk about all these content groups and all this investment that’s being done and structured content and all the rest of it. And then you go talk to the AI team and what you’ll discover is that they go over to SharePoint and they grab a bunch of PDFs and they dump them in their LLM. And then they’re like, yeah, we have to figure out how to do the metadata and it’s real challenge. Really, you don’t say. okay, allow me to show you the back end of where these PDFs come from. And they’re like, that looks hard, just I need a PDF-based solution. And then I just start foaming at the mouth, and that doesn’t go well. So I’m still working on that one. But as we, you know, as content design, it sounds like evolves a little bit towards systems, which means content ops to a certain extent. And then content ops, you know, needs to connect better with content design because just automating it, see my example of PDFs being dumped, is not actually thoughtful necessarily, and it needs to be done better. So where do you see opportunities for these things to connect via the I guess like catalyst of AI or something? KA: Yeah, I mean, I think so content designers can bring language, they can bring interaction patterns, they can bring meaning into structured content, terminology, taxonomy, and the context layer. And I think that, you know, all of that needs to go into systems, and those systems need to be scalable, they need that they’re that bad word again. They need to be governed, they need to be maintained. there’s, you know, I mean. We know how difficult it is in any org, I think, of any size to get any kind of handle on all of the content. And there’s this idea that, you know, with AI, you can just like magically throw it all in and it’s gonna come out in some way that makes sense. And that’s, you know, just not going to be the case, I think. so I I think that there’s there’s room already in content design for there to be more content ops. I think that content designers kind of define what good content experience is. And at this point, to make that scalable, it has to show up in a lot of different places. And there has to be a really complex architecture for that context to be given to the machine in a way that’s going to be effective and efficient. So, you know, there’s this architecture of there’s, you know, system level prompts that are going to apply to everything. There’s skills that need to be pulled at the moment that that specific skill is needed. And, you know, and we’ve got a lot of UX writing skills and things like that. But I think that this like large UX writing skill is not the answer. It’s going to be pretty granular skills that are needed and an architecture to help the machine know when to use which one. And then evals and also the whole process of doing evals and getting results and doing things that will improve the results next time around. To me, those are all really ops-centered things. And we’re going to need better systems to run all of that. And I think we also are going to need better systems to help deal with those silos and silos of content and disagreement on content, because you know, if a user is interacting with an AI from a product, you know, it’s not so different. I in so many ways nothing has changed. But they don’t care if it’s you know coming from marketing or from the product or from whatever, they want whatever it is that they need at the time. And if you know everything is just kind of in one pile of information, that’s not going to come out. Correctly. So I think ops is just so incredibly important now, and content ops in particular are going to be far more important than they were. SO: Mm. KA: I think, you know, at least in orgs that I’ve been in, that’s been kind of more, you know, there’s marketing content ops, and they have marketing CMS and all of a whole system there, and maybe, maybe also with like experience, like journeys and things like that. So they know all about the customer and what they’ve done and what they haven’t done and what question they asked and et cetera. But that’s completely siloed from product content. There’s customer support content, that’s a whole other thing. You know, all of these things have been separate. And we’re going to have to find more ways to unify that entire picture in a way that actually can make sense. SO: Right. And the interesting thing to me is that fundamentally, if you think about it, the reason that they’re siloed is because in the olden days they were siloed, right? KA: Mm-hmm. SO: You go to a trade show, and you get marketing paper, collateral, handouts, and then you buy the product and the product comes with documentation like a book on paper. And maybe you read it and maybe you don’t, but that’s where the technical documentation lives. And then you have a problem. So you call up on the phone or your field services tech comes on site and helps you out. So then along comes digital, just broadly digital web online stuff. Well, now everything is on the web page, but it’s like training.yada.com and docs.blah.com and well www.yada.com, which is of course the marketing content. So even there though, once you sort of find yourself in the learning management system, the experience is different. And so you’re signaled that that’s where you are. Fine. KA: Right. SO: And then there’s a docs portal, we hope. And then there’s the marketing, the dot-com. But now the way this is going to work is that I, as a reader, as a content consumer, am going to ask the AI for information. And it’s going to give me this flat text. And by flat, I mean not designed by the originator. KA: Mm-hmm. SO: Right? It just shoves it through the LLM and spews out a bunch of text at me and says, here’s the answer you’re looking for, here’s the information you’re looking for, maybe. And so therefore, that whole signaling mechanism that says, I am training content, or I am marketing content, or I am tech support content is basically gone. It’s just here’s the answer to your question, and it’s conversational, it’s interesting, and maybe it’s accurate; we don’t know. It’s probably accurate at a high level in the sense that the first question you ask is pretty accurate. As you start drilling down, it gets less and less and less accurate because it has less and less and less information to average in order to give you that answer. So with all of that, I was gonna ask you how the next couple of years are gonna go, but then I decided that was dumb. So how do you think the next six months, or maybe six weeks, or you know, even next week? Like pick a time period and tell us what you think. KA: Yeah, I mean, from my point of view, and I talk to a lot of folks from across content design, and that’s kind of been my focus the last months is to really think about future of UX content and where we’re headed. I do see what you were saying earlier of you know, there are have been a lot of layoffs. I expect there will probably continue to be a lot of layoffs in content roles and design roles and everything else probably as well. So I think that in the short run, things are gonna be pretty bumpy. I do think that companies are getting to the point where the development that they’ve been doing on their products is getting further along. They’re closer to releasing something or at least using something with customers. And I also have a feeling that there’s going to be we’re getting closer to this point where companies are going to start seeing the way that we thought we could do this is not working. And they’re gonna have to start trying to figure out where are the points that it’s breaking. Now, whether or not they will have the folks at the table who will know where to point as, you know, these are the kinds of skills that we need back here. I don’t know. I guess I’m in some ways I’m not terribly optimistic about that in the short term. but I do think eventually they’re going to have to figure out what’s actually required. To build these products that they want to build. And I think that a lot of content people are going to be required to do that work. And so I’m hopeful, I don’t know if in the next six months, but in the next year anyway, that we will see a major turnaround of companies starting to really hire back content folks. I’m not sure if they will have the same titles they had before, or there’ll be new titles. But I do think within a year, we’re going to start seeing the turnaround on content folks having a lot of work. but I I have a feeling that the success of the products themselves are going to be the thing that’s going to cause that turnaround to happen. SO: What’s your best advice to the people that are being displaced right now? KA: Yeah, I mean, I think there’s so much hype about AI, and a lot of us would probably wish that we could tell folks, don’t worry about it. You don’t have to learn it, you can choose not to. And I mean, there’s a lot of talk now in design at least of people maybe wanting to leave tech. And I think that can be a valid choice. But if you do want to stay in tech and you do want to stay in user experience or even in techcomm, I think you really need to be upskilling, a lot of upskilling. And there’s a ton of information out there. I think there’s courses are just starting to come out. A lot of them are not necessarily that good. now I think the better courses are starting to come out. SO: Because ironically, if you write them with AI, they’re not going to be that good. KA: That’s the problem. That is the problem. So find the folks that have actually done it who are teaching things because they’re gonna be the ones that have, you know, better advice for you and more realistic advice of kind of, you know, this is what this is what I’m seeing while I’m doing it, or this is how it came about, as you know, I mean, model design is a whole new field. It’s not that they’ve necessarily figured out the best way to do these things, but the same kinds of skills that we have as content folks are the things that are going to stay the same, even if the format changes. And you know, now we’re thinking about all these markdown files. And, you know, I think in design systems there’s a lot of talk about adding context to JSON files and things like that. All of that stuff can change, but the ability to know what good looks like doesn’t change. So I think it’s a combination of trying to get really clear on those fundamental skills that you have. So, you know, are you, are you a good writer? Are you a good designer? And what makes you a good writer or a good designer? Can you articulate those things? And can you get better at being able to articulate those things? Because I think that’s what’s going to be needed. We’re going to need to define what good looks like and be able to create an eval for what good looks like in a way that we didn’t have to before. So one thing is double down on your fundamentals and make sure that you’re really you know, not just kind of a surface-level writer, but you’re getting down into the depths of why what you’re doing is right. And then second is, you know, that upskilling on could be on model design, could be on content systems, could be on, you know, I mean, Mike Iantosca is constantly putting out new stuff on how DITA can be expanded to use AI. So I think depending on where your kind of comfort is, there’s plenty of ways that you can find things, even for free. And that’s one of the things I would tell people is there’s lots of courses, but you don’t have to spend a fortune, especially if you don’t have work right now. There’s so much good information out there for free. So just, you know, go out there, start digging into it. And also use AI in your, you know, daily activities because a lot of times you it’s something you’ve said many times, like if you talk to an AI for five minutes, it might sound really smart, but the more you use it, you’ll figure out that it’s it doesn’t get it right so easily. SO: Yeah, I think the best advice I have right now is that while I think we have a recognition that there’s some really problematic stuff going on, and we haven’t talked about, you know, the problems with bias and ethics, but you cannot in in this universe that we live in, which is largely, you know, tech and software, you cannot be the anti-AI person. That is not going to work. KA: Mm-hmm. SO: So my advice to people is to do two things at the same time, which are try nudge your AI strategy, your organization, assuming you still have a job, into a place of using it intelligently and away from a place of using it unthinkingly and unethically, without putting yourself in the position of becoming known as the person who is sort of sounding the alarm. So it’s like look like an advocate, but nudging it in a direction it needs to go. KA: Absolutely. I love that. SO: Because people are making some really, really dumb decisions. I don’t mean us people, but at a at a very high level, there’s a lot of people that perceive AI as the solution to all of their problems, which are basically, goody, we’ll never have to hire anybody again. And that’s real dumb. And I worry a whole lot about the products that we work on and what would happen if and when those products are AI-ified in non-strategic ways and what could happen. KA: Mm. SO: I think about medical devices and bad AI, and it kind of makes me twitch. KA: Mm-hmm. Mm-hmm. SO: So you know I see risks there. And I think I wanted to close with that. Either what do you see as the biggest risk, or maybe that, and also what’s the biggest opportunity here? Like what are the two sides of the coin of this, you know, innovation, new thing that we’re in the middle of? KA: Yeah, I think there’s a ton of risks. As you say, it’s not popular to be the person who’s talking about those all the time. But all those things you mentioned are huge risks. I’m not even sure I would call them risks. They’re inevitable at this moment. A lot of, you know, the issues with ethics, issues with inclusion, issues. I mean, in content design, I feel like content designers care so much about the users. They’re just completely full of empathy for users. And they’re often the ones who come in and argue with a PM or argue with someone about the way that they’re, you know, suggesting something is done in a product. And, you know, I don’t know that we can even, if we try to encode that into some kind of discernment for a machine, I’m not sure that it can do that. So I think there’s a huge risk of taking humans out of these processes because I just think humans bring so much to them on that kind of risk side. I also think there’s a huge risk that these products will never quite get to the level that the companies need them to get to. You know, if you’ve worked in content, you know all the things you need to deal with legal and all the compliance issues that you have and how difficult it is to get people to trust your product enough to want to adopt it. If if you’re in consumer work, even more so that you’re, you know, constantly focused on how do we get people to trust this product and want to use this product. and I’m just not sure. Well, I just feel like it’s we’re very, very far from the orchestration, the mechanisms, the context, the data: All the different pieces that would need to be in place and working pretty well, working pretty smoothly in a way that orgs have never been able to make all of those things work really smoothly. And I don’t think that AI can magically clean that all up. And so I think there’s definitely a risk that companies are putting so much effort into AI and expecting it to get to a level that it’s just not going to get to. So I think that’s a huge risk. In the meantime, my take on it and my take for my team and others in content is use this time to be learning and we’ll see where it goes. You know, but in the meantime, if you’re learning and, like you say, I think if you have an opportunity to nudge something in a better direction, do it. You know, speak up with some ideas. Yes, you don’t want to be the person who’s anti-AI at this moment in your company, probably. But that doesn’t mean you don’t have small opportunities where you can nudge things in a positive place. so I think that’s a lot of risk, a lot of danger. And that’s not even touching on all of the real dangers that I think we all can kind of see coming with AI. And then to move on to the biggest opportunity, I think I mean it’s a challenging moment for people in content. And I feel like we have our work cut out for us. And if this is ever going to get to a functioning place, it’s going to require a lot of really smart people who work in content and who have the skills that we have. And so in that way, I feel there is a ton of opportunity for for folks who have a background in content fields. And so I think you know, I think that’s wonderful and to maybe tie them together. One thing I love about that is that content folks tend to care about ethics and tend to care, you know, they have a good set of values that they want to bring to things. SO: Yup KA: And so if a future necessity at some point will be a lot of content folks, I think that’s going to be a positive for all of these AI experiences eventually. SO: I think that I heard this I think over the weekend, and I can’t remember from whom, but it was 100% not me. So let’s start there. There was an interesting discussion around the idea that code is effectively an artificial language. It’s a language in a box. It has a set number of things that you can do with it, which means that effectively, it’s a solvable problem for a machine. Content language, not code, but language, is not that. Because people can use language in interesting ways, you can use it in ways that’s never been used before. You can analogize the orca sitting off of Gibraltar to something interesting that nobody’s done before. That’s not a thing you do with software code, right? So language itself has simile and metaphor and all these other fun things, and it has different forms and shapes and things that you can do with it. And so fundamentally, solving for code automation is a different and much easier problem than solving for language automation. But we’re accustomed to thinking that code is hard and language is easy. And I think what we’re gonna find in an AI world is that the opposite is actually true. So that’s the opportunity that I see is once we get to that point, and I hear you saying a year, and I’m, you know, it could be next week, who knows? KA: Who knows? SO: Who knows? Okay, so yes, so our concluding thought is who knows? Yeah. It depends. That’s always KA: It depends. That’s always the content answer, right? SO: So Kate, well, thank you. This has been really, really interesting. And I hope you’ll come back at some point in six months or six weeks or whenever the next, you know, disaster strikes us, and we’ll take it from there. KA: Perfect. Absolutely. There’s so many more things we could chat about. SO: So many. Thank you. Conclusion with ambient background music CC: Thank you for listening to Content Operations by Scriptorium. For more information, visit Scriptorium.com or check the show notes for relevant links. The post Bridging the gap between content design, content ops, and AI appeared first on Scriptorium .

September 14, 202613 min

The Uninvited Author: AI, intent, and the future of content

In this podcast, Sarah O’Keefe and Carlos Evia discuss their upcoming book: The Uninvited Author: AI, Intent, and the Future of Content. Carlos Evia: One of the things that we really want to invite people who end up reading this book to do is engage critically with AI tools across the text cycle. It’s not magic. It’s a collection of ugly computers in water-chugging data centers that work primarily on predictability approaches. And I truly think that we need to be critical in our engagement with AI as authors and as readers of content. Related links: Digital sovereignty in the age of AI From ad hoc to autonomous: The AI content ops maturity model Subscribe to our newsletter for updates on Sarah O’Keefe and Carlos Evia’s book: The Uninvited Author LinkedIn: Sarah O’Keefe Carlos Evia Transcript: Disclaimer: This is a machine-generated transcript with edits. Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction Sarah O’Keefe: Hey everyone, this is Sarah O’Keefe. I’m here with Carlos Evia. Hi, Carlos. Carlos Evia: Hello! SO: So, Carlos, for those of you who don’t know, is the co-author of this book that we’re working on and also professor at Virginia Tech. His official title, his official titles are Associate Dean of Strategic Initiatives at Virginia Tech, and he is also the CTO of the College of Liberal Arts and Human Sciences. I think, Carlos, that means you’re the computer guy in the humanities world. CE: That means that people come and complain to me about curriculum and also about their computers. So yes. SO: Okay, cool. And so with that, Carlos and I have written this book. And in this first episode to this special series, we wanted to give you an overview of what it is that we’ve done, or what it is we hope we’ve done, we are trying to achieve here. And probably the easiest way to do that is actually to break down the official book titles. So what is our official book title? CE: The Uninvited Author: AI, Intent, and the Future of Content. And I don’t know if I got it right from memory. SO: No, I think that’s right, only because I wrote it down. So I’ve been staring at it nonstop for, you know, four to six months now, but still. CE: Yeah. yeah. I see it now. Ha ha ha. SO: You know, can we remember our own book title at this point in the writing process? And we should tell you that we’re recording this as we’re approaching sort of ninety percent done, which means that we still have approximately ninety percent of the work to do. And we’re a little punchy. CE: My co-author today woke up being very optimistic about progress on the books. But I agree, 90% sounds accurate. Yes, I’m not gonna argue. SO: Yes, you’re well, your co-author is slightly delusional, so we’ll just set that aside and move on. All right, so let’s talk about this. The main title is the uninvited author. So who is the uninvited author? What are we talking about here? CE: Well, the uninvited author, surprise, is the AI author. We have seen how AI has been taking over the generation of content at different types of content, content for marketing purposes, content for technical purposes, content in video, in text, in audio. And in some of those cases, it’s creeping into people’s workflow without an invitation. And now, and we know people, I mean, across the different facets of content work that now are, on purpose, inviting AI agents to contribute or to take over and start driving the car of the content operations. But particularly for the type of content that we do in our little corner of technical communication, AI, authors kind of started coming as uninvited guests that will appear when a user will use a chatbot, say going to ChatGPT or going to Claude asking how do I do this instead of going to the published official manual of the name of the appliance or name, the software application. And what we realized is that what was happening is that as the company, I write my documentation, I put it on the website, I put it on a book if I’m in 1985, and I give that to my users. And I hope that that’s what they use. I am the author. I bring it to you. But now when a user prompts a chat button and says, how do I turn on this machine? How do I save a file in this software application? Now there’s a new author that is going to give information that might be accurate, that might reflect what I wrote, or might be inaccurate. It might be, like you said, delulu and inventing some other content. So that is the uninvited author. An author that is already here and is adding to what other humans have created or written or flat-out making up stuff out of nowhere that didn’t exist and that was not created by humans before. SO: And I think that this this theme of loss of control for the humans returns in maybe every page of this every page of this book. So we have the AI that is now delivering… well, it was delivering content, right? And when you had a search engine, you would ask it a question and it would return answers, but in general, those answers were, “Here’s what I found in this document,” or “Here’s a collection of links,” or whatever. The what’s new is that the AI is synthesizing answers from its sources, and we don’t know what those sources are necessarily. Sometimes they’re cited, sometimes they’re not. I asked the chatbot something the other day, and it said it was something very straightforward like how old is somebody, like famous person. And it gave me an answer, and I said, cite your sources. And it said, this is just general knowledge. I’m sorry. How is this general knowledge? It it would not and then it said, but you know, it got kind of passive-aggressive. It was like, well, if you insist, I can give you a couple of websites that list this kind of information. but essentially it was like, how dare you question my knowledge of just the general universe? Well, I I use computers for a living, so I question everything. All right, so the uninvited author is the AI, and now the chatbot is authoring content, accurately or not, whether we like it or not. So our subtitle then is AI Intent and the Future of Content . When we look at AI in the context of content, what are we looking at here? How does AI get involved in content operations, in the content process? CE: Well, we have been chatting, ha ha, we have been chatting about chatbots so far, but the chatbot, it’s really only scratching the surface of what AI already is for content operations. I don’t even want to think about right now, let’s save it for the next section of the title of what it’s going to do or what it could do. CE: And the chatbot is the easiest representation of it. And yesterday I was teaching, I’m teaching this semester a senior seminar in communication on the topic of what AI is doing to the profession of communication, for good or for bad. And something that my students tell me is that they use ChatGPT or a similar chatbot as a replacement for the Google search box. So they go there and they go ahead and ask and prompt without prompting, ask, they query things that up to two or three years ago, people would just ask Uncle Google. SO: Now, in their defense, it’s getting very difficult to actually find the search box. CE: If you go to Google, yeah, you don’t know what’s Google, and you don’t know what’s Gemini. They’re all married and combined together. And I think that that is the very interesting part. But the chatbot in that box that resembles the Google Omni box is only one component of what we mean by AI. AI comes in many other flavors that I think my co-author in this book even has a fancy table inserted on page number here about the different forms of AI engagement in content operations. And at the end of that spectrum, we have agentic approaches to AI in which you, as the developer or you as the author of the content, are going to be sending out these AI agents to be taking care of processes or supporting processes that human beings are doing throughout the life cycle of your content. So I want to emphasize that what we’re talking about here is not just ChatGPT, not just Gemini or even Claude, but many in-house developed and maintained in a little or not so little computer in one organization implementation of not just a large language model, but many other machine learning and data mining approaches that could create something that smells or looks like artificial intelligence beyond basic automation. So AI means many things, and we have more than one section in the book that tries to address that AI is many things, even though for the majority of people who probably are listening to this AI is ChatGPT in that box. SO: And one of I think one of the really tricky parts about this is that, as content people, we see AI in lots of different places in our in our job roles. So what I mean by that is that mean, we’ll start at the end. The content consumer, it used to be get your content on the website, do some search engine optimization, keywords, whatever, make sure that people find your content. Or you might be privileged that your content is ships with the product as online help or a companion document. Again, if it’s nineteen eighty five, but you know, it’s the supporting content that goes with the content, or you’re working on knowledge bases, that type of thing. So as a content creator, I’m thinking about what does it mean SO: To get my content in front of the end customer. And if that is now mediated by a chatbot, how do I make sure that it survives the trip through the chatbot so that the content, the instructions, the information still reaches that end user instead of them getting, you know, something potentially inaccurate? So that’s the front end, right? There’s also the issue that LLMs and chatbots have the ability to create what we call synthetic content. Content that doesn’t previously exist, but when I ask the chatbot for it, it gets generated. There are lots of examples of this, but machine translation is one. Hey, can you give this to me in a different language, please? And it says okay, and it magically machine translates it and gives you what you’re asking for. Make it shorter, make it longer, give me more information, give me less information. Write it in a different style. Those are all cases where the chatbot is intervening and putting its own layer on top of that. And then as you work your way backwards, there’s authoring assistance, you know, fix my grammar, refactor my sentences, that kind of thing. And as a content creator, the availability of AI across the entire what you know, the text cycle. The process of creating and editing and managing and delivering content, it affects every single one of those pieces. And we have to really think carefully about how and where and whether or not to integrate that into content operations. What are appropriate and inappropriate ways of doing that? CE: I think you said something very interesting when you refer to a magical process. The assumption is that a magical process happens and then voila, you have something that the AI agent or the AI chatbot generated or assisted in the generation. If we go back to that separation of the backend or the front end. And that’s where I think that one of the things that we really want to invite people who end up reading this book is to engage critically with AI tools across the text cycle. And we haven’t even talked about the concept of the text cycle, which is a major component of the book, because it might be that you, as an author or you as a content reader, as a consumer of content, prompt a chatbot for information on how to do this, either to write something or to read something and solve a problem. But it’s not magic. It’s a collection of ugly computers in water-chugging data centers that work primarily on predictability approaches. And I truly think that we need to be critical in our engagement with AI as authors and as readers of content and not just about technical content. If you’re doomscrolling on Instagram, and let’s not even mention TikTok because I don’t use TikTok, I’m very old. And you see these AI sloppy videos from dozens of people fighting in like every Mortal Combat. Let’s ask why… SO: You seem to know a lot about this for someone that doesn’t use TikTok. CE: I doomscroll a lot. You’re like, what is this, and what is happening behind the scenes? How do we think critically about the concept of AI as it becomes an invited or uninvited component in communication processes? And it’s not magic. It does things fast. Sometimes it does it in an accurate way. The models are getting very interesting, and they tend to be way more accurate than they were just a year ago. But I think that’s an invitation that engage with this critically. And I want to peddle this book as an approach for people in our profession, particularly in technical communication, in content operations, to read this and start thinking in terms that allowed him to expand that idea of this is the oracle. I will post a question, and I hope I get something good in return, and my life can be better. SO: Yeah, I think one of the most useful things you can do with AI is sit down and ask it a question about something that you are an actual expert on. That might be a hobby in your life. It might be that you have an interest in, you know, eighteenth century literature. It could be a lot of things, but pick something and ask the AI a question about something that you truly know a lot about. And pay attention to the results you get, and you know it’ll say, would you like me to give you more information about something? And just kind of follow that trail for for a bit. What I found in doing that is that nearly always the first answer, the high-level answer, the sort of overview is pretty accurate. And the farther I chase it, the worse it gets. As I ask it for more and more detail, the details degenerate. It’s still very confident. But it starts just going completely off the rails. Now, then you try the same thing with a topic you don’t really know anything about and you’re like, well, this looks pretty accurate. So essentially, it will get you to a sort of mediocre level of expertise, but if you need to go deeper than that, you run into all sorts of problems. So intent. We have intent in the title. What is intent, and why do we care? CE: I’ve been thinking about this for a very long time, and to a degree, this is something that I wanted to write even back in the early 2000s, not about AI, but in the context of single sourcing of content and how we were embracing, in the early 2000s, approaches for reusing pieces of content that will appear in different deliverables and will adapt on the fly to the needs of different audiences or different versions of a product or something like that. I started thinking about in the tradition of, I’m gonna say a bad word that Sarah doesn’t like: hermeneutics, which is the study of interpretations. SO: Boo. CE: And it’s not Herman Munster. Hermeneutics. As an author, as a human author, I write something, and I imprint in that something my intentions, my intent. I want to say something. If you compare it to the rhetorical tradition, that’s your purpose. You have a purpose when you send something out to the world. You have an intention when you put something out in the world. And at the other end of the spectrum of the communication model, there will be somebody who, up to a few years ago, it was most likely a human being and not a machine, who was going to receive that message that you were sending with your intentions and was going to make an interpretation of said message based on their environment, on their cultural background, on their languages, and extract something that will make sense for them. So that is what we’re talking about here when we mention intentions or intent, because surprising nobody, if we go back to that communication process that today includes the human author, the human reader, and the possibility of a human, sorry, not human, of an AI author or co-author in an AI reader or assistant to the human reader, well, those AI players bring their own intentions. And you mentioned this when you were talking about where is the information that they have coming from? And there are many layers of the training data, models that go out and perform web searches on the fly, models that are looking at your memory and your previous interactions with them, all of those components create the intentions of these AI authors or readers. So we think that intent or intention is a very complicated topic when it comes to AI in content operations, because it can really take things on a completely different path of what that traditional approach of a human talking or a human writing to another human is conveying. But at the same time, also, intention is a good thing from the author’s perspective, because if I imprint my intention and I set some guardrails on my communication processes, that might be my only hope, my only way to battle the production of AI slop. That is completely taken over by the intentions of the “AI authors.” And I am still not super comfortable calling the AI authors, even though in the book we call them the AI authors. And that’s what we mean by intentions. It’s going to that very basic fundamental concept of I write something, I publish a video, I publish a picture that I took; I have a purpose, I have an intention. What is my intent in publishing that? And what happens when that message gets to a human receiver and that human receiver makes the interpretation out of it? That cycle is truly affected by AI players at all points in the cycle. SO: Yeah. And then, you know, the last chunk of this is the future of content, which is, you know, pretty much what we might expect. I think I’m gonna wrap it up here because this is getting too long already. We have a lot to say, and much of it we’ve captured, but we’re trying very, very hard to provide a roadmap to, you know, as Carlos is saying, to engage critically with AI, with the tools that are now available to us as authors, as content creators, as content consumers, and to understand, you know, what does it mean to use AI? What kind of biases does AI introduce? We’ve seen a lot of discussions about, it uses too many em dashes and, you know, now I can’t use em dashes anymore because people see that as a proxy for AI-written content. But what else is it introducing? What kinds of biases are in the training data? What are the ethics of using data that is largely English-centric, biased towards things that are available online, not just biased towards if it’s not available online, then the AI can’t consume it, right? Which is why they’re buying up books and shredding them and scanning them wholesale, which is a whole other thing. CE: Mm-hmm. SO: But the digital footprint of what an AI has ingested is different from reality. It’s the AI’s reality. And then you have to ask the question of how far does that deviate from actual reality? And does it do so in predictable ways that we can potentially remediate? So we’d have a fair amount of discussion about algorithms and bias and what it means to work in an environment like this and, you know, are there ways to remediate it? Are there ways to bring this under control? And then finally talked, you know, a fair amount about what it means to live in a world where your chatbot has access to, you know, truly a stupendous amount of information, which means it’s at your fingertips and you can potentially automate and agentify these workflows. what does that mean? Our last sort of section is currently called the next 30 years. We’ve debated changing it to the next 30 months, or I feel like the next 30 days is the most accurate way to portray it. But we are trying to step back and take that longer view of where is this going and what are the implications of all of what’s going on here? So with that, we are mostly done. CE: Optimistic. SO: Our goal, our plan is that this book will be available by the end of October, which is to say for LavaCon. if you want to get more information about it, I would say follow this podcast series or sign up for our newsletter. We’ll be sending out updates that way. And I expect there will be some fun special offers as we launch. We expect this thing to be in the vicinity of probably 300 to 350 pages. And we’ve tried hard to make it an actually fun and entertaining read as opposed to something that you have to slog through. And I hope that you will find information in there that you currently cannot get from your chatbot CE: And it’s not written by AI, it’s written by us in painful writing sessions and lots of online meetings and what have you. SO: We have suffered greatly, and I would really like for people to read this stuff, and I don’t know about agree with, but at least engage with and have that conversation. I am actually much more interested in finding out what other people think about all these developments than I am in asking the chatbot. So so I hope it’s useful and I look forward to being done. CE: Me too. SO: That’s the most exciting part of all of this. We’re gonna do a couple of these podcasts to talk through some of the pieces and parts that are there. I would like to state for the record that we are not going to generate synthetic podcasts off the book content, although we totally could. CE: Tempting, tempting. SO: Not very tempting, but it is possible. Given the shape that my voice is in this week, it’s actually, yeah, the listeners might have appreciated that. But I think that’s it. We’ll see you on the next one. And Carlos, thanks, and I’ll talk to you in our next torture meeting. CE: Thank you very much. SO: Okay, bye. Conclusion with ambient background music CC: Thank you for listening to Content Operations by Scriptorium. For more information, visit Scriptorium.com or check the show notes for relevant links. The post The Uninvited Author: AI, intent, and the future of content appeared first on Scriptorium .

August 31, 202623 min

The limits of automation and AI in content workflows

What are the limits of automation and AI in content systems? In this episode, Sarah O’Keefe and Bill Swallow dive into traditional formatting, workflow challenges, and what happens when AI is introduced. The limit of automation here is not really the automation. It’s the people who say, “I don’t like the way this looks, and I’m gonna find a way to fix it. I know all sorts of ways of bending this stuff to my will.” If you’re generating content at scale, you can’t afford to do any of this stuff. Related links: Digital sovereignty in the age of AI From ad hoc to autonomous: The AI content ops maturity model Subscribe to our newsletter for updates on Sarah O’Keefe and Carlos Evia’s book: The Uninvited Author LinkedIn: Sarah O’Keefe Bill Swallow Transcript: Disclaimer: This is a machine-generated transcript with edits. Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction BS: Hi, I’m Bill Swallow. SO: And I’m Sarah O’Keefe. BS: And today we’re gonna talk a bit about automation, and more specifically what the limits are for automation. SO: Yeah, they let us out of our cages. And so here we are. And I think that in this world of I’m not going to get through the first sentence without saying AI. In this world of AI, we are suddenly facing, you know, increasing automation across every facet of everything that you could imagine. And I think it’s important to take a step back and talk a little bit about limits and what automation cannot do and maybe more importantly, why it cannot do certain things. So before we get to, you know, the AI in the room, Bill, what are the limits when we talk about generalized publishing in the pre-AI world where we’ve done a ton of formatting automation and we talk about all those kinds of things, what are some of the limits that we’ve run up against in terms of automation? BS: Yeah, automation is something that we’ve been doing quite a bit of for the past, what, twenty years, I think, or more. I’ve only been around for thirteen years. I’m thirteen years young, so… SO: Yeah, this is lies. BS: But no, a lot of our work around automation is around publishing. That is, I think first and foremost, the most common case. And with automation and publishing, you can save a lot of time and save a lot of effort, but a lot goes into building that capability because you can’t really work around edge cases or you know, fancy hand-cobbled formatting, hand page breaks, all of that thing. You have to kind of rely on the software to understand the content that you’re feeding it and it’s going to produce something that you know is a finished product: a PDF, a web page, what have you. And you know, if you deviate from the structures and conventions that it’s expecting, then all bets are off as to whether or not your automation will be successful. SO: Right. And so, from our point of view, doing lots and lots and lots of structured content work, i setting aside the tools and the technologies, what structured content really does is limit variability. Right? It limits the variability of the information or of the markup really going into your system, which then means in turn that the system can be automated and can automatically output all the outputs, all the deliverables, all the file formats that you need. And so when you have weird edge cases, and I am of course the worst offender in terms of actually finding ways to bend the software to my will, but only when I’m an author, right? As a system configuration, I’m, yeah, I think you should all follow the rules. Yeah. So here we are, and we basically say we’re gonna limit the variability of the input and standardize the input and therefore we will get standardized output. Cool. But as you said, that only works if you don’t have so what is a weird edge case or what are some of the things you’ve run into that just bollocks up automation? BS: One good case is working with content that maybe has valid structures in place. So they’re not, you know, the content itself validates against a validator. So there’s nothing wrong with it. But they’re using, let’s say, different in this case, DITA elements in a let’s say creative way. You know, so they’re you know I was on vacation last week, so that wasn’t me. SO: I see you were looking over my shoulder last week again. Yeah, that’s true. Well, what I actually ran into was I had a DITA map; it was valid, it validated, and then I used Oxygen’s validator, which you know really goes a little bit deeper and looks at things. Everything was fine, but it crashed. And eventually what I realized or what I found after some digging was that somebody who was definitely me had inserted a draft comment and the draft comment was in a table, maybe under the title, but before the table group kind of thing. It was in an unusual location and it just the processor just died. It just laid down and gave up. now I don’t know exactly whether that was because of the, you know, the core processing or something that we did in the the plugin that I was running. It doesn’t really matter. The point is it was it was valid. But it didn’t pass the processor because the processor was like, Why would anybody put a draft comment in this location? This is dumb. And then I, you know, felt berated by the processor and I moved it and then it all worked. BS: Interesting. Yeah. I would have enjoyed being there for that. SO: Yeah. Mm hmm. So anyway, we fixed it, and it was fine, and you know, and off we go. Let’s talk about pagination! BS: Pagination’s a fun one. SO: Pagination’s my favorite. I get very upset with bad line breaks or bad page breaks. BS: Yeah. And you know, we can build in rules that say, you know, only, you know, to control widows and orphans, that type of thing. You know, tell it, you know, break a table leaving X many cells. If you don’t have X many cells to break, then move the entire table to the next page, all that fun stuff. But if you have, you know, specific places where you need to break to a new page for whatever reason, that always can’t be necessarily automated, depending on what the rules are for producing your output. You know, the processor’s not going to know that, you know, you may arbitrarily want to break a page at this particular location. So you know, in many cases we cobble together a little tag that says, you know, essentially break the page here, and the processor knows to you know, when it sees that to say, okay, stop processing this page, move to the next one. And you know, it works most of the time. SO: I would never. Yeah, the problem with inserting page breaks, as I’ve learned to my great sorrow, is that, of course, later you add more content and now you have a page break a third of the way down the page because you hard-coded it in. So this is bad, and you shouldn’t do it. But if you insist on doing it, do it l as late in the process as possible. Don’t try to fix your pages when you’re 80% of the way there, because you will have to reinsert them over and over and over again. also. BS: Right. SO: Inserting empty tags that have a non-breaking space in them to introduce vertical space is wrong, and you shouldn’t do it. BS: Ha ha. I will agree with that one. Also, I mean, doing these types of things to force a page break or to force extra space, it really flies in the face of automation because technically you have to create the output in order to know where you need to insert your page breaks and then go back and add them and then automate your output again. SO: Ha ha. It looks bad. BS: Yeah. SO: So yeah. So really the limit of automation here is not really the automation, right? It’s, well, it’s me, right? It’s the people who are like, I don’t like the way this looks, and I’m gonna find a way to fix it. And I have lots more demented tricks up my sleeve. I know all sorts of ways of bending this stuff to my will. And if you’re generating content at scale, you basically can’t afford to do any of this stuff, right? It’s one thing if you’re producing, you know, one document and it’s short and/or it’s marketing content, and we’re really concerned about the appearance because of people making buying decisions. But if you’re producing, you know, ten, twenty, fifty thousand pages a year, then you just need an engine that produces this stuff. So BS: Mm-hmm. Yeah. And the page break problem is actually a good one to speak to at scale because if you’re in an environment where you’re sharing a bunch of different content and you’re reusing pieces, if you insert a page break for your own personal preference, suddenly that’s going to be in everyone else’s document that also uses that particular topic. SO: All right. So I’m hearing that page breaks are bad and I shouldn’t do them. BS: No, they’re great. It’s just that you have to be smart about it. SO: Okay. Sneak them in. Don’t get caught. Got it. So, while we’re on the subject of recalcitrant authors. Such is definitely not me. What about automation in a scenario where your content production system is not being used by the authors? BS: Yes. That’s a completely different problem. Yeah, that’s an entirely different problem because then you’re in a situation where essentially you need a production team that are also, you know, data entry people, you know, and they need to somehow bring the content that’s being produced into the system in order to produce the output. SO: Okay, but can’t we just use AI for that? BS: Sure, why not? AI’s great for that, right? SO: Huh. Right. Yeah. So ultimately, I mean, sure. If the authors are authoring and they follow some sort of reasonable template or or they’re reasonably consistent in some way, we can bring the content in automatically. I hope you heard the first part of that about the reasonably consistent part because that’s where this breaks, right? You’re gonna get all sorts of things. BS: Mm-hmm. SO: And our experience on this has been that this is a problem or a challenge across a variety of different factors. Are your authors volunteers or are they paid? If they’re paid, you have, you know, some degree of leverage to say you need to work in the system. So are they volunteers or are they paid? Is content their full-time job or is it a small part of their responsibility? So, you know, 90% of the time I’m an engineer and 10% of the time I’m a writer. Or 90% of the time I’m a subject matter expert of some sort, and 10% of the time I’m writing content. The less time a given person is spending on content production, the more likely it is that they’ll push back on learning your weird content system. Right. BS: Mm-hmm. SO: And then the third question, the third factor that goes into this is sort of level of expertise. I mean, we make jokes about rocket scientists, but it’s kind of that question. If you have a person who is responsible for contributing content and they are a world expert in their field, unless that field happens to be something like content markup and formats, they’re not gonna be that interested in spending their time learning about content. They’re focused on, you know, curing cancer. And I’m not prepared to argue that, you know, learning some tags is actually a useful use of their time, right? So you take the content that this world expert produces in whatever format you get it. Scribbled on a napkin, yucky Word file, whatever. We’ll take it. And you find a way to bring it into the system. Now, I will say there are some subject matter experts who are actually very interested in the content production process and will jump right in and kind of, you know, play around with it and learn what they need to learn and contribute directly in the system because they think it’s interesting and fun. But when you start talking about volunteers versus people that are being paid and/or experts versus, you know, people who are dedicated to this role and/or it’s part-time, the intersection of those three, the Venn diagram or actually the anti-Venn diagram of those three things is where you will find people that will not be working in your system. BS: Yeah. And you know, with any kind of an external contributor to content, you also run the risk of having content that doesn’t necessarily follow the conventions that you’re expecting, you know, or that your automation pipeline is expecting. So you may get something from someone who absolutely knows their subject matter inside out and sideways, but they’ll give you a document that has, I don’t know, three levels of nested lists, and your system only accounts for two; then you have to figure out how you’re gonna get that content to play well within the construct that you have. Otherwise, you have to reengineer your pipeline. SO: And then you have to balance the cost of reengineering the pipeline for the edge case against the value that the edge case brings to the situation. And, you know, very early in my career, I ran into somebody who has since passed away, unfortunately, who was terrible and I mean awful at I I shouldn’t say it that way. He wasn’t terrible at following templates. He followed the templates until it didn’t suit him. And when it didn’t suit him, he would come up with some workaround to do what he was trying to do. So your nested lists are a good example, stuff like that. And I was, I don’t know, 20-something, and now I’m slightly older than that. It was my job to push back on this guy and say you didn’t follow the templates. And he would literally yell, which was a fun experience. BS: Mm-hmm. SO: But the thing was that in every single case, he had thought about it. He never deviated from the templates by accident. It was always, I need it this way because of this other thing. I also learned the hard way that he had thought really carefully about word choice. And so if I ever changed any of his words, we would have well, we would have words. And he would explain to me why I was wrong and why changing his word to my word or, you know, follow the style guide, whatever, was wrong in that instance. And he was always right. Always. Now, I’ve been doing this a really long time, and I’ve only run into one person like that in my entire career. But I have not forgotten that at least once I ran into somebody who would work outside the provided, you know, color outside the lines in order to make his document better, and always had a defensible reason now. Do I wish he had yelled a little bit less? Yes, but you know, that’s life. BS: Mm-hmm. All right, so we talked a bit about the automation pipeline and what people can inject into it. Should we turn our attention back toward our good friend Mr. AI? SO: Mm. All right. So it’s interesting because a lot of the edge cases that we’re talking about can actually be addressed and flattened out by AI. So AI and AI tooling actually provide a potential solution to all of this. But I wanted to focus a little bit on a different problem with AI and automation. So when we talk about the limits of automation in, like a scripting workflow or a traditional non-AI workflow. A big part of the response to that is, well, the AI could actually address what you’re describing and solve for those things. Okay, fine. But now let’s talk about what AI can’t do, the limits of automation with AI. And it’s actually a really different problem than what we’ve been talking about. We’ve been talking about sort of following the template, don’t color outside the lines. With AI, what we’re running into is that. BS: Mm-hmm. SO: AI and AI systems only have visibility into the digital world. And so if there’s information floating around an organization that gets passed around in a hallway conversation, typically, and I say typically with great caution, the AI is not listening. And so if I, you know, if I run into Bill in the hallway and I say, which would be unusual because we’re always remote, so but never mind. We run into each other in the hall, and I say, hey, don’t forget to update that one thing. Or I ran into so-and-so at lunch, and they told me this other little tidbit of information. Okay. Well, I’ve now transmitted that information to you and you’ve got to get it into your whatever you’re working on, right? BS: Right. SO: But the AI is not aware of that because it didn’t make it into a JIRA ticket or anything else, any sort of digital workflow. Now, this is also in many cases, it’s similar to the problem that you actually run into with remote employees that they miss out on the so-called water cooler conversations, right? Like the conversations that happen on break or at lunch or this or that. But so the AI can only see what it has. BS: Mm-hmm. SO: And what it has access to is digital content. So your existing documents, your meeting recordings, your digital footprint, your email. I mean, it’s super creepy, but what it doesn’t have access to is the person-to-person interactions that are not being recorded, or conversations that are out there, or a note scribbled on a piece of paper or a couple of these Post-it notes that are sitting in front of my laptop, right? BS: Yep. SO: The AI doesn’t see those and therefore it can’t automate them. And because it can’t automate them, they don’t get injected into the workflow. And all of that sort of detritus, because if it’s important, I’m going to pass it on, right? I’ll put it in an email, or I’ll put it in a Jira ticket, or I’ll put it somewhere where it belongs. BS: Right. SO: In some sort of defect tracking. But it’s those little like, hey, I just saw this kind of side conversations. That cannot be automated. I mean, unless we move into an always-on surveillance cap surveillance world, which I mean I’m sure somebody’s working on it, but at the moment, if it’s not captured digitally, the AI can’t see it. And I think that we underestimate the amount of work and the amount of information that is sitting out there in the not-yet-digital world. BS: Mm-hmm. SO: So that’s the limit of AI automation, which is very different from “It can’t do really nice page breaks.” It probably actually can do really nice page breaks. BS: So basically the quality and the efficiency of what you’re gonna get out of an AI automated workflow really depends on whether or not the information itself is available and being captured. SO: Right, which is really quite different. I mean, AI is much better at taking sort of questionable inputs and doing things with them, right? Because it’s not a pass-fail script, but it can only see what it can see. And so we have to understand that there’s still limits there. And and it’s it’s a different limit. It’s a qualitatively different type of limit. BS: Mm-hmm. SO: And I think that’s important to recognize as we move forward. BS: Right. And of course all that stuff that does need to be captured, it needs to be done in a way that AI can access it and can access it in the right way. So if it’s input wrong or whatever, then AI is not gonna have the right info. SO: Yep. And so, yeah, this is where I’ve been spending a lot of my time because, as you probably know, in the world’s worst-kept secret, Carlos Evia and I are working on a book on content and AI. BS: That’s my shocked face. SO: And yeah, I know, please contain your shocked face. It’s coming soon. She says hopefully, with many fingers crossed. We’re getting there. And what we’re hoping to do is get something out there that talks about some of these issues, right? We’re not gonna write a book that tells you how to optimize for a specific AI model or specifically how to build agentic workflows. BS: Mm-hmm. SO: What we want to do is give you the tools to ask the right questions so that you can start looking at how to put these things together and how to move forward with a reasonable content operations approach that also involves AI as appropriate. SO: I look forward to being done. Also, for the record, the AI is not writing the book. BS: Like, yeah. SO: That seems important to say in this context today. BS: That is quite important to say, given all of the AI-generated stuff we’re being force-fed at this point. SO: No, I can assure you that my pain is human and personal when it comes to this writing. It has been real. It has been a slog. As perhaps it should be, because the process of putting something like this together forces you to really think carefully about what’s going on. And the process of doing that with a co-author really forces you to think carefully about what’s going on. BS: And real? SO: And so it’s been great working with Carlos and also extremely painful because that’s just how books are. BS: Yep. Cool. I think we’ll leave it there. Thanks, Sarah. SO: Thank you. Conclusion with ambient background music CC: Thank you for listening to Content Operations by Scriptorium. For more information, visit Scriptorium.com or check the show notes for relevant links. The post The limits of automation and AI in content workflows appeared first on Scriptorium .

August 3, 202622 min

Digital sovereignty in the age of AI

Are you in control of your digital destiny? In this episode, Alan Pringle and Sarah O’Keefe define digital sovereignty. They break down what organizations need to consider as they bring AI into content operations, from data leakage and competitive intelligence to shifting international regulations. Sarah O’Keefe: The working definition that I’m using is that digital sovereignty is control over your own digital assets, digital destiny. That could be you personally, it could be you as an organization, or it could be you as a country or as a group of nations. And when I say group of nations, probably 98% of the time I’m talking about the EU, which has some laws in this regard. Digital sovereignty is your ability to control, own, and manage your digital assets and how they are used, reused, processed, resold, repurposed, and all the rest of it. Related links: AI in the content lifecycle: three years later From ad hoc to autonomous: The AI content ops maturity model Upcoming AI book: The Uninvited Author LinkedIn: Alan Pringle Sarah O’Keefe Transcript: Disclaimer: This is a machine-generated transcript with edits. Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction AP: Hey everybody, I’m Alan Pringle. SO: And I’m Sarah O’Keefe, hello. AP: Hey there. And today Sarah and I want to talk about something that’s really starting to come to the forefront with all of the AI things that are going on in our world. And that is digital sovereignty. And before we get too deep in that, I need to throw up many, many disclaimers. Sarah and I are not lawyers, nor do we play them on television. And we are absolutely not lawyers or experts on anything in regard to international intellectual property. So with those disclaimers out there, Sarah, if you would, would you define what digital sovereignty is? SO: The working definition that I’m using is that digital sovereignty is control over your own digital assets, digital destiny. That could be you personally, it could be you, the organization, or it could be you as a country or as a group of nations. And when I say group of nations, probably 98% of the time I’m talking about the EU, which has some laws in this regard. So it is your ability to control, own, and manage your digital assets and how they are used, reused, processed, resold, repurposed, and all the rest of it. AP: And I’m going to give a very basic example from my personal life in regard to email. Years ago, actually decades ago, when I got set up with an internet service provider, they provided me with an email address. It had their company name in the domain.com. And I used it for years. But when I switched my ISP, guess what? I had to make a decision. Do I continue to pay that old ISP, basically rent, to maintain that old email address? Or should I move to one of maybe the free email providers? So I did a little research, and my ultimate decision was I created my own domain with my name, and I kind of just decided not to ever use again the email provided by an ISP because if you switch it, you’re going to possibly lose control of that email address. And at the time that I made the switch, a lot of the free providers, they would scan your email to provide targeted ads and some other things that was kind of unsavory to me. So I ultimately made the decision I was going to own the domain and set that up myself and pick my own software that I hooked up to it to use to to read it. And I did not use a third-party email provider to basically pull in or reference my account. I am using an open-source standalone email client to kind of minimize who can poke into my email. So that’s one very basic example of how I kind of took control of my digital life as it were. SO: Right. And if we apply that to content ops, again, staying pretty general, you think about cloud systems, like the cloud universe versus on-prem. AP: Yes. SO: And when you talk about, let’s say, a CCMS, a component content management system that is on-premises, the the argument was always, well, that way all of our content lives on our servers, in our organization, we have complete control over it. Along come the cloud services, the cloud-based CCMSs and everything else. And they say, well, yes, but it’s much cheaper for you to put this into the cloud on our systems, which are shared. And you know, there’s advantages of upgrading, and there’s all sorts of advantages to cloud systems, mostly around IT overhead. from a digital sovereignty point of view. You are delegating to that cloud system and you have some sort of a contract or a service level agreement. And again, we are not lawyers, but you have this agreement that says we the cloud provider promise to not scan your information or not use it for evil or not, you know, there’s a bunch of stuff in that contract that governs how cloud provider is or is not allowed to handle your content, your personal information, your credit card information that you might be putting in there and all the rest of it. And with software as a service, with cloud systems, we have for the most part cut over into cloud. You know, that’s that’s kind of the default these days. There’s hardly anybody that is still putting their content, their content and their content management systems on their own proprietary in-house servers. AP: Yes, correct. SO: So we have decided that for cloud, you know, cloud writ large generally, that the advantages of cloud outweigh the disadvantages. Now, moving this a little bit more towards content and slowly towards AI, where things get really interesting with digital sovereignty, if we talk about machine translation for a minute. There are a couple of different ways of doing machine translation, obviously, but big picture, you can have your in-house machine translation system and database, and you can control that and govern it and do things with it. Or you can take your content and you can throw it at a public-facing machine translation system. And the risk that you run when you throw something at a public system is that they will take your proprietary confidential content. And use it. So I throw at it a sentence that says, the XYZ company has developed a special new thing, right? And I need that translated into various languages and I get it translated. But as a result of that, I am leaking information. I am leaking my confidential, potentially information into the machine translation services. And there are some really interesting security issues around that and how you might be able to. As a competitor, extract that back out. But just dialing it back for a for a minute, we understand the concept of leaking information by using public-facing machine translation, right? Publicly available machine translation. But one thing that I think is very often overlooked is that machine translation, the fact that I ask for a specific language, never mind the content, But the fact that I am now asking for a new language provides competitive intelligence in the sense that that means that I or my organization now cares about that locale, that that language. So let’s say that I’ve been consistently submitting European languages for machine translation, right? If you but if you look at my record of what I’m asking for, you will see that all of a sudden, about three months ago, I started adding a bunch of Asian languages. Well, what does that tell you about what I’m up to? Either I’ve decided that Asian languages are interesting and fun, or my organization, I mean, presumably I’m doing this for work and not fun, or my organization is launching into Asia. And I don’t actually need to see the content to sort of get that piece of competitive intelligence out of it. So essentially. The way that I’m using the machine translation, even if we protect or have a contract that says you you are not allowed to look at what I’m processing, but if you can look at the parameters of what I’m processing, that might give you enough information to tell you something about what I’m up to. AP: Yeah, so basically what you request or don’t request, as the case may be, can give away clues that you may not want floating out in the world. SO: Right, exactly. So now we take this to AI and we think about digital sovereignty for AI, and it gets very much more complicated. So t first, taking this example of machine translation, if you think about AI chatbots and prompting and public-facing models, then in the same way that requesting a particular language so well, let’s back up. Let’s say that we have a contract with XYZ model provider and it says we will not use your input as training data. We will not use your output as training data. Cool. But are you going to use my prompts as competitive intelligence? Because think about what I’m prompting on. Hey, tell me about the intellectual property laws in Vietnam. Tell me about how to go to market in a particular country. Tell me about strategy for pricing tiers, right? If I’m doing those kinds of prompts, then you, as my competitor or adversary or whatever we’re dealing with here, can get an awful lot of information out of what I’m up to, right? You can figure out what I’m up to just by looking at my prompts. So we have to worry not just about the protection of the input and the output, but also the actual prompting that I’m using to get the inputs and the outputs, because the prompting itself has competitive information in it. So then when we start thinking about digital sovereignty, you say, okay, well, then what we should probably do is have a restrictions on usage of input, output, or prompting data by the vendor, right? The vendor who’s providing the AI model should have a contract that says we are now not allowed to use this stuff. But then we take that another step forward. Nearly every contract I’ve seen in the past, you know, whatever years, decades says we promise not to use this unless we’re required to by law. So in other words, if a government subpoenas us, we will cough up this information as we are legally obligated to do, that sends us right down the road to something called zero data retention. Which is the idea that you process your AI prompts in such a way that they are not retained, that the inputs and outputs are not retained by the provider, so that if they are subpoenaed, they cannot in fact cough up the information because they don’t have it. And then if you’re more paranoid than that, and some of you, you know, depending on what industry you’re in, should be, you start thinking about bringing the model in-house. So instead of using public-facing with a an enterprise contract of some sort, you think about bringing the model onto again, in-house on premises in the same way. This is right back to cloud versus not cloud. And then we have some questions about what model are you using and do you know what’s going on in the internals of that model, which points you maybe at using something open source or maybe not. It depends. But there’s all these layers of decisions that you have to make around how you’re going to use AI and how concerned you are about leakage from your use of AI to you know your competitors or something else. Now, over the top of that, we start thinking about nations rather than organizations. And we have some of this with just generalized cloud systems. GDPR, the European Data Protection Regulation, says a lot of things about how you process personal information and what the requirements are and what you are and are not allowed to do. Now If you’re a European company inside the EU, you’re clearly subject to GDPR. But many non-European companies are also subject to GDPR because you have a server in the European Union, or you have customers in the European Union, or you operate in some way in the EU, which then that’s enough to have you sort of folded into GDPR. On the AI side, we have something very similar going on where companies are looking at AI models and making decisions based on what jurisdiction does that model belong to. Now, the most prominent of these is that, for example, currently, as of right now, as we’re recording this, the American models, like a ChatGPT, a Claude, that kind of thing, are largely not available in China. and it’s a little bit tricky in terms of is it actually banned or is it just restricted, but broadly not available. The Chinese models are available in the US and are open source, but if I’m an American or actually, let’s say I’m a European company and I’m trying to figure out my AI strategy. Do I use an American model? Do I use a Chinese model? What happens if I’m using a Chinese model and then the US government bans that model in the US and I have operations in the US? And I’m suddenly now what? In reverse, if I’m a an American company and I’m using American US models, and I have, let’s say, a chatbot, and I go to market in the EU. I am now subject to the EU AI Act. And the EU AI Act, with some complications for when it goes into effect, et cetera, has rules that say things like: if you put up a chatbot, you have to disclose that it’s AI. You can’t pretend that people are talking to a human. You have to say this chatbot is AI. This image was generated by AI. there are disclosure requirements in the EU. That are much more stringent than the disclosure requirements, which are none in the in the US. AP: Yeah. In the US. And again, if you were making decisions about what model to get, where you should place your servers, etc., we highly recommend that you speak to your intellectual property attorneys and do not take advice from the two of us. Thank you very much. SO: Right. This entire podcast is just an ad for IP lawyers. You’re welcome, IP lawyers. AP To me, this whole thing is so interesting. You know, we talk about the digital world and globalization and you know, there’s no boundaries anymore. Guess what? They very much apply here because, like you said, if you have customers in the EU, and you may be a US-based company, but that does not give you clearance to absolutely ignore GDPR. So we still have to be aware of geographical boundaries and the laws of all the nations inside those boundaries. SO: Yeah, the thing that keeps me awake at night is the question of what if I build out an entire thing on some model? It it kind of doesn’t matter which one, but I build an entire infrastructure based on some AI vendor and then and then that AI vendor gets banned by a government whose jurisdiction I or my operations are subject to. AP: But there are significant costs. SO: Right. And I’m not picking on any particular country. It it could go every which way that you can imagine. So then as a as an organization, especially if you’re a big international organization, you start thinking about well, maybe we should bring all this stuff in house so that we can control it. AP: And infrastructure involved with that decision. SO: Right, because now we’re talking about you operating your own data center, and that makes you, you know, not so beloved by literally anybody. So I mean, this is a really hard problem. And what’s fascinating to me is that we generally in software, software as a service, generally cloud has pretty much won with some minor exceptions for air-gapped systems, high security systems, network operations centers, or you know, software that runs utilities, that kind of thing. Things where you really, really, really do not want it taken down by external things. So you have this idea of secure systems, air-gapped systems, systems down at the bottom of a mine that are cut off because they’re literally down at the bottom of a mine. AP: Yes, inside the earth, correct. SO: We used to talk about airplane help, but that’s much less of an issue anymore because now, for better or for worse, we have Wi Fi on the planes. AP: Yes. SO: Mostly for the worse. Anyway… AP: At least I haven’t heard a meeting yet, an online meeting. I’m sure it will happen at some point, but I have yet to see that happen. Thank goodness. SO: On a plane? Yeah. Hmm. AP: Yeah. SO: So anyway. So cloud, the risk-reward for cloud versus on-prem has pretty much tilted towards the cloud systems in general, with you know, very specific kinds of exceptions for I would say unique use cases, but it’s kind of like a ninety ten or an eighty-twenty kind of split. Back in the day, it was everything was on-prem and cloud was the exception, but it’s it’s very much shifted. And now with all this AI stuff, everybody’s using currently big picture. As everybody’s adopting AI, they’re using public-facing models, public-facing chatbots, maybe with like some sort of an enterprise license. But I just have this feeling that a lot of this stuff is going to be brought into in-house at least managed models. AP: Exactly. Yeah. SO: So still sort of cloud software as a service, but with a big enterprise contract wrapped around it that says, here’s what you you, the vendor, the AI vendor, can and cannot do with our data. But it’s a it’s just a really interesting problem because we in content we don’t deal with these sort of national issues that much, like sovereignty at the national level. And I think that with AI systems, we’re seeing a lot of concern around, well, what does it mean that this might be regulated differently in different countries? And how do we manage that? I mean, how do you put together an AI content ops strategy if you are not sure whether your model will be available in that country over there next week. AP: Yeah, and I’m just thinking, think about audits are often a component of a contract, and you add AI on top of this, and all of the jurisdictional things you’re talking about, all of the facets of those audits just got even more complicated with AI than they already are, which that is probably a whole other discussion. Again, not a lawyer, don’t want to be, but I can see that being a very contentious something that’s gonna have to be ironed out in the very near future. SO: Yeah, so digital sovereignty, it’s it’s very hard to say and even harder to spell. but I think that it’s something that we should be thinking about as content people and we should at least understand the big picture of what’s going on so that when we go to our corporate legal team and say, Hey, we’re worried about this, we can at least have a reasonable conversation about where this is going. AP: And again, we don’t know where it’s going, but it’s definitely something to keep an eye on. This is a great place to wrap up. Sarah, thank you very much for a conversation that I frankly haven’t heard a lot about in the content world. SO: Well thank you. Not a lawyer. AP: Nor am I. Until the next time everyone, thanks. Bye. SO: Bye everybody. Subscribe to our monthly newsletter for more insights on AI, content ops, and more! var gform;gform||(document.addEventListener("gform_main_scripts_loaded",function(){gform.scriptsLoaded=!0}),document.addEventListener("gform/theme/scripts_loaded",function(){gform.themeScriptsLoaded=!0}),window.addEventListener("DOMContentLoaded",function(){gform.domLoaded=!0}),gform={domLoaded:!1,scriptsLoaded:!1,themeScriptsLoaded:!1,isFormEditor:()=>"function"==typeof InitializeEditor,callIfLoaded:function(o){return!(!gform.domLoaded||!gform.scriptsLoaded||!gform.themeScriptsLoaded&&!gform.isFormEditor()||(gform.isFormEditor()&&console.warn("The use of gform.initializeOnLoaded() is deprecated in the form editor context and will be removed in Gravity Forms 3.1."),o(),0))},initializeOnLoaded:function(o){gform.callIfLoaded(o)||(document.addEventListener("gform_main_scripts_loaded",()=>{gform.scriptsLoaded=!0,gform.callIfLoaded(o)}),document.addEventListener("gform/theme/scripts_loaded",()=>{gform.themeScriptsLoaded=!0,gform.callIfLoaded(o)}),window.addEventListener("DOMContentLoaded",()=>{gform.domLoaded=!0,gform.callIfLoaded(o)}))},hooks:{action:{},filter:{}},addAction:function(o,r,e,t){gform.addHook("action",o,r,e,t)},addFilter:function(o,r,e,t){gform.addHook("filter",o,r,e,t)},doAction:function(o){gform.doHook("action",o,arguments)},applyFilters:function(o){return gform.doHook("filter",o,arguments)},removeAction:function(o,r){gform.removeHook("action",o,r)},removeFilter:function(o,r,e){gform.removeHook("filter",o,r,e)},addHook:function(o,r,e,t,n){null==gform.hooks[o][r]&&(gform.hooks[o][r]=[]);var d=gform.hooks[o][r];null==n&&(n=r+"_"+d.length),gform.hooks[o][r].push({tag:n,callable:e,priority:t=null==t?10:t})},doHook:function(r,o,e){var t;if(e=Array.prototype.slice.call(e,1),null!=gform.hooks[r][o]&&((o=gform.hooks[r][o]).sort(function(o,r){return o.priority-r.priority}),o.forEach(function(o){"function"!=typeof(t=o.callable)&&(t=window[t]),"action"==r?t.apply(null,e):e[0]=t.apply(null,e)})),"filter"==r)return e[0]},removeHook:function(o,r,t,n){var e;null!=gform.hooks[o][r]&&(e=(e=gform.hooks[o][r]).filter(function(o,r,e){return!!(null!=n&&n!=o.tag||null!=t&&t!=o.priority)}),gform.hooks[o][r]=e)}}); Name * First Last Email * Data collection * I consent to my submitted data being collected and stored. Review our privacy policy Consent to subscribe * I consent to the use of my submitted data for marketing emails. I understand that I can unsubscribe at any time. Review our privacy policy Submit The post Digital sovereignty in the age of AI appeared first on Scriptorium .

July 13, 202622 min

The debt crisis: AI edition

What happens when you feed years of messy content into AI? In this episode, Bill Swallow and Alan Pringle dig into the content debt crisis, including increased system costs, neglected localization, and the fallout of “just use AI” mandates. They share practical insights to help organizations get back on track. Alan Pringle: Is your content updated? Does it reflect the latest information? Is it created for all the different locales that your company serves? Is it in different languages? That is another pile of debt that when you start looking at AI, all the problems will be very brutally magnified, and you’re going to have to address them to really have a large language model that works at all. Related links: Balancing automation, accuracy, and authenticity: AI in localization (podcast) Forbes: AI Costs More Than The People It Replaced Taming AI: Using AI for content conversion at scale (podcast) LinkedIn: Bill Swallow Alan Pringle Transcript: Disclaimer: This is a machine-generated transcript with edits. Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction Bill Swallow: Hi everybody, I’m Bill Swallow. Alan Pringle: And I’m Alan Pringle. BS: And today’s episode is going to focus more on the content debt crisis, AI edition. AP: And it’s also going to be the complaint edition, surprise, surprise, because there’s a lot of things in the AI world right now that are still making me cranky. And I am sure we will talk about them at some point. BS: Yeah. So with the rise of AI, I think it’s kind of holding a microscope to a lot of the technical debt that we’ve been seeing over the years in content operations in general, whether you have outdated authoring formats or content that’s not being updated on a regular basis, new delivery formats not necessarily meeting the needs of the users, and so forth. And all of that is kind of compiling or snowballing into a bigger problem once you start feeding all of this stuff into AI. AP: Right. And it’s interesting to me how everyone’s talking about AI as being this productivity tool. In a lot of ways it is, but that’s not what the focus of this is. In a way, it is also sort of a consultant for you. As you just mentioned, Bill, when you start looking at AI and delivering it, treating it as a delivery endpoint for your content, a distribution endpoint, you are going to start to discover that your processes on the back end for creating and distributing your content are not what they should be. So it’s kind of like this consultant saying, Hey, you need to do better over here. And that is where a lot of this debt is coming from, from my point of view. BS: Mm-hmm. The unfortunate part of that consultant is that it’s not offering advice on how to fix it, but it certainly is pointing out the issues. AP: It’s like, “This is screwed up. Full stop.” So, I mean, part of why we’re here is to talk about some of those kinds of debt. And let’s just start with one technical debt. And I’m saying technical in the sense of the way that you perhaps use software to put together your content. Let’s kind of focus on the content operations world. BS: Mm-hmm. AP: For example, if you are delivering content via unstructured desktop authoring tools, of which there are many, and you can templatize things and make your content seem more consistent, but there’s a problem with a lot of desktop published or content generated from the desktop publishing world. It’s more focused on look and feel and fit and finish, particularly if you’re delivering for PDF. And yes, people are still doing that. So there’s a lot of time and effort spent on that look and feel, that fit and finish. And frankly, that time should have been invested in adding intelligence to the content to explain, you know, things under the covers about w hat user is this for? What is the model of this particular thing? All of that kind of metadata, that kind of categorization. Desktop publishing, at least from my point of view, doesn’t really do a great job of helping you catalog that kind of stuff. So that’s a problem. BS: No. It is a problem. Also, with desktop publishing, you can kind of confuse AI a bit if you’re using desktop publishing inconsistently. So if you’re using formatting tools to override formatting to make things look like headings or make certain paragraphs look like children of another paragraph, doing those manual finesses is great for print because, as you know, most people will look at that and understand the hierarchy of information, understand what’s going on with the content that they’re reading. But anything digital isn’t necessarily going to pick that up. AP: Right. Yeah. BS: Especially if you’re looking at something like bare bones HTML, if you’re using CSS to override the size and prominence of a standard paragraph as opposed to using a heading, that is not necessarily going to be picked up as a heading, even though a reader would actually see that as a heading while looking at the HTML page. AP: Right. What a human reader can figure out from formatting cues, if those cues aren’t set up in a way that a large language model, a computer, can understand, there’s that huge disconnect, and that’s where that debt starts piling up. And then another angle here in this content creation world, if you are using multiple different tools to create your content, there’s a good chance that content under the covers is not going to be processed the same by a large language model. So there’s another deficiency right there on top of that. And this is very common, for example, if you have had mergers, acquisitions, and you’ve basically created a larger company from many different companies. And they all still have, especially in legacy content, things created the “old way,” and all the old ways start to pile up and cause problems because your large language model can’t properly basically figure out what is the heading in this particular chunk of docs versus what it is over here. So it can’t parse it as well. And again, this all goes all the way back to the way that you created that content. And it’s a clue, hey, you need you need to fix this. And I I think beyond that more technical, the way that you create it, there also there are also issues with the content itself. BS: Mm-hmm. AP: Is it updated? Does it reflect the latest information? Is it created for all the different locales that your company serves? Is it in different languages? That is another pile of debt that when you start looking at AI, it’s gonna be all the problems in regard to that are also gonna be just very brutally magnified, and you’re going to have to address them to really have a large language model that works at all. BS: Yeah, because not only do you have the technical debt on the source content side and on the published content side, but you also now have technical debt growing on the AI side because you need to spend more time and energy refining the how that model works with your content in order to achieve the correct results. AP: Right. So you’re having to do a lot of overrides and we I don’t know if overrides is the right word, but you’re having to do a lot of additional processing and figuring out so it will parse things correctly. And there are a lot of companies today that still have problems keeping content updated to the latest and greatest. So unfortunately, people turn around and call support. Or today they start hitting up the chatbot. But guess what? BS: Mm-hmm. AP: If the chatbot doesn’t have access to the latest and greatest because frankly it doesn’t exist or it’s not hooked up to it. It’s it’s just like the poor people in support. It’s not going to know what to do and it’s going to spit out probably very authoritatively wrong outdated information. Again, yeah, it’s be it’s goes all the way back to BS: That’s yeah. AP: In your content creation process, how are you accounting for updates? How quickly are you getting them in place? How are you handling them in both your source language and how are you handling them in your other languages? It’s one thing I do want to bring up here, and and and I may be biased here, but I don’t think localization is getting enough attention on the AI distribution side. It’s been talked about for a very long time in regard to machine translation, AI assisted translation. BS: Mm-hmm. AP: But I don’t just like I think sometimes localization is a second thought for a lot of companies, which still blows my mind in 2026. And by the way, we’re recording this in July 2026. So what we say right now may be outdated next month. Who knows? BS: Who knows? AP: There is that issue. I’m curious, you have a more of a localization background than I do, but I do see that being a potential debt problem, part of this debt crisis that we’re talking about here. BS: Definitely, because if you’re pushing your content out to AI, do you have targeted audiences in mind? Or is it going to be a free-for-all of people going in and using that AI to get answers to their questions? if you are localizing your content, I think probably the best practice here would be to somehow bundle for consumption all of the guides in all of the different languages together. So you have product XYZ and you have it translated in three languages. Then you put all of those copies together and give that to the AI so it can draw its associations as well as it builds out you know its understanding or I hate to use understanding because the AI doesn’t understand things. It’s very easy to make slips in that way. But so that AI can actually draw those relationships. So, you know, if you’re looking in English or you’re looking in Spanish or you’re looking in German, the same query in those three languages will retrieve pretty much the same result. That’s proper for that language. And we talked with someone last year on the podcast, Steve Maule from Acclaro, about AI in translation. AP: Yeah. BS: And his focus was more on using AI as not so much a generative tool, but you know, a tool for aiding in AI, basically the next step from machine translation to neural machine translation to AI translation. And kind of the benefits and drawbacks at the time. Again, that was a year ago. So many of these things have probably changed again. But if you are translating content you may want to go back to that podcast and take a listen to what he had to say. and we’ll put a link to that in the show notes for this one. AP: Yeah, again, I keep going back to this idea that we talked about at the beginning, how delivering via AI just puts this magnifying glass to what you’re doing and uncovers things that are wrong. But as you had mentioned, it doesn’t necessarily offer advice on how to fix it. And we will talk about that probably to wrap up the podcast. But what I also want to mention too. More and more, there’s actual real debt involved, not just the more indirect debt we’re talking about with the content debt, the technical debt. The AI companies, the providers, at one time had very generous offerings as far as the amount you could hit their APIs, the amount of tokens that they offered, whatever else. BS: Definitely. AP: But now, the days of I guess you could say subsidized token use, they’re pretty much over. And now the actual real cost is being passed on to the people using these large language models. And as a result, the actual price, the cost of using AI is skyrocketing in all these organizations. And there have been a lot of news reports lately about very BS: Mm-hmm. AP: And there have been a lot of news reports lately about very big firms, and I will not name names, but you know them all, their household names, basically having to do a 180 and tell their employees, hey, you need to chill a little bit on your AI use because you’re burning up the tokens, converting PDF files to PowerPoint. And yeah, I get it. BS: Mm-hmm. AP: Doing that conversion is probably not the most efficient use, productive use of AI tokens. But this gets into kind of, I don’t know if there’s such a thing as cultural debt, but if your company is sending this message, and a lot of companies are, use AI for everything. And you know, they’ve got these leaderboards up showing these people use this much AI this week, you know, that sort of thing. You are encouraging a free-for-all. So, instead of saying, here are the good uses, this is where we think you need to focus your use of AI in coding, in content creation, in whatever else, this whole idea of just use it. You got to use it a lot. And now that’s coming back and biting people in the backside. So companies are scrambling and telling people, calm down. BS: Mm-hmm. AP: Chill out. Don’t use the AI as much. So it, there’s a mixed message there. And it’s because there was not good communication, nor was there good thought put into how workflows should incorporate AI. And I think that cultural thing right now is a huge problem. And everybody’s focus on the cost and the non-subsidized token use when they need to be going back and looking at, well, look at your comms. Look at what you were telling people six, twelve months ago about AI. It’s kinda on you. BS: Mm-hmm. Yeah, basically don’t let your nine-year-old run around in a toy store with your credit card. AP: Yeah, pretty much. BS: That’s pretty much what’s been happening. And yeah, we’ve been seeing lots of reports. There was one in Forbes last week, where, you know, companies are saying that the cost of using the compute power is more expensive now than the people it was supposed to augment. AP: Exactly. So I think the reality of what AI can do maybe is starting to sink in, maybe a little, but at least exposing people to the true cost of it on a corporate level, it may BS: Mm-hmm. AP: I’m probably being too positive here. What? Me being positive? It may result in some recalibration about how companies are telling people to use AI and maybe thinking a little more on a cultural level, all the way starting with communication. Here are the workflows you need to apply AI to. This is where you can apply it. BS: Mm-hmm. AP: And then there’s also the whole vibe coding discussion. I don’t know how much we want to get into it, but we all know, right, there is now an industry, you know, people are now going in to clean up vibe coding because, yeah, just because you can vibe code shouldn’t mean doesn’t mean you should be vibe coding. Because who was going to maintain and clean up that mess? So again. BS: Well, we’re cleaning up a lot of vibe coding now. AP: It’s another cultural issue that should have been addressed, but in this AI rush, everybody was like, use it, use it, use it, without thinking about how they should really how they should really apply it and basically sort of reimagine the way they do work in a productive way. And that does not mean let the AI do everything from my point of view. BS: Mm-hmm. AP: So I guess we probably need to kind of wrap up and talk about what people can do when all of their ugliness is amplified and magnified by AI. And it really it’s about rewinding and going back and looking at from the start, especially in the content world and the content operations world, how are you creating that content? Are you doing it in a consistent way? Are you building in intelligence into your content that an LLM can pick up to better understand the context of that content you’re providing it? Is your localization workflow efficient? Is it getting content turned around quickly so people in other markets are not six or eight months behind in the latest and greatest information. And by the way, yes, that still happens today, believe it or not, that people are months out and getting it because they are in another location. It’s shameful, but it still happens. It does. BS: Mm-hmm. Yeah. And there’s also, you know, how are you feeding your content over to AI? How are you providing it? Are you just having the AI team you know scrape web pages and PDFs, or are you supplying something that’s a little bit more targeted and tied to the content itself, and maybe supplies some of that intelligence over? AP: Yeah, because I think Sarah’s talked about this on a previous podcast. A company was noticing that their LLM was kind of not using PDFs or not weighing them, or and I’m again I’m personifying big time here. my apologies. BS: Yeah. It’s easy. AP: Was yeah, was not really it was not giving the weight to the content in PDFs. And when the company kind of did some reverse engineering, they realized it was because that PDF what they didn’t have a lot of context. BS: Mm-hmm. AP: There wasn’t a lot of basically metadata built in that the LLM could basically parse. So it’s like, I’m gonna kind of put that to the side because it doesn’t have the richness that I need to do things well. So it it’s about looking at how you write. How are you delivering this content? And one possibility here, and it’s, well, there’s several structured content where you have metadata built into your content. You have tagging that offers semantic context. BS: Mm-hmm. AP: That’s one way to do it. And it’s not the only way. I mean, we have a lot of clients who use it, but it absolutely is not the only way to do it. Knowledge graphs can also be part of this solution. And sometimes knowledge graphs and structured content can play together to provide that rich feed of information that LLMs prefer and can do a better job with. So it’s not a one-size-fits-all solution when it comes to how to create that content or how to best hand it over to AI, but I think it is kind of a one-size-fits-all that if you aren’t doing things right foundationally, AI is going to kick your tail. BS: Mm-hmm. Pretty much, yeah. And I think the best way to look at it is to consider AI another delivery target, just like you would a portal, just like you would a PDF or what have you, a help system. When you consider it as another endpoint for your content, another, you know, place to deliver to, it makes it a lot easier to start scoping what you need to do to reach, the requirements for that particular target. AP: Agreed. So content people look at it as a delivery point and also look at as a job aid too to help enforce style guides, to help enforce taxonomy, whatever else. So it’s not just about content creation. BS: Mm-hmm. AP: It’s also that part of your thinking needs to be a content distribution point, like Bill mentioned. And it can be hard to think of it as both of those things, but in the content world, it absolutely is. BS: And I think that’s a good place to leave it. So thank you, Alan. AP: Thanks, Bill. BS: And we’ll see you on the next one. Get more insights on AI and content operations in our monthly newsletter! Name * First Last Email * Data collection * I consent to my submitted data being collected and stored. Review our privacy policy Consent to subscribe * I consent to the use of my submitted data for marketing emails. I understand that I can unsubscribe at any time. Review our privacy policy Submit The post The debt crisis: AI edition appeared first on Scriptorium .

June 22, 202618 min

From ad hoc to autonomous: The AI content ops maturity model

There are five levels of maturity for AI-driven content operations. Which level are you in? In this episode, Sarah O’Keefe and Bill Swallow walk through the AI content ops maturity model, from ad hoc experimentation to fully autonomous workflows. Sarah O’Keefe: We want this automation, right? We want the ability to go in and extract release notes and do something with them. We have to have a certain level of maturity on the software development process so that we can grab the appropriate information. The same thing is true on the content side. You have to have a certain level of maturity in your content development processes, in your content management, so that you can identify the right things to process and the right things to access. Related links: Want to know more about Sarah’s nifty little side project? Register for our upcoming webinar . AI in the content lifecycle Enterprise content strategy maturity model LinkedIn: Sarah O’Keefe Bill Swallow Transcript: Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction Bill Swallow: I am Bill Swallow. Sarah O’Keefe: And I’m Sarah O’Keefe. BS: And today we’re going to talk about AI in content operations, or more specifically, a maturity model for AI. SO: Everything needs a maturity model, even AI. BS: Even me. SO: I have no comment. BS: My maturity model is written in crayon, what can I say? So okay, so we need a maturity model for AI as far as content operations are concerned, and probably in you know, many different degrees, but we’ll focus on content operations. So what might that look like? SO: I’ve been thinking about this and what it looks like to employ AI as a tool to help you with content. And as I was thinking about what this looks like, you know, you always fall back on that standard five-step model where one is basically mass chaos, and five is the perfect world, generally. also, one is nearly always cheap, and five is nearly always expensive enterprise things. But, you know, let’s go a little beyond mass chaos versus governed, regulated, etc., and sort of sort of back up a little bit and talk about what this might look like. So level one in every maturity model typically is ad hoc. And what that means is that in this case, AI is being used sporadically by some people. It’s inconsistent. And I would say that when we look at AI and content specifically, BS: Mm-hmm. SO: This is going to be things like reprocessing your content using public-facing models. So I wrote a draft of something, I shove it into ChatGPT and I ask it to shorten it or tighten it up or identify areas that are problematic. Or I just say, hey, you know, write my article for me. The outcome that you’re gonna get on an ad hoc model is going to depend on an ad hoc level one. BS: Mm-hmm. SO: AI thing is going to depend on how good you are on the individual’s expertise and their level of interest. So if you want to just go in there and say, hey, I have a bio and it’s too long, and I’ve been asked to produce one that’s only 50 words for a particular conference, for example, then, you know, this is this is actually a really good example of ad hoc, right? BS: Mm-hmm. SO: We have these long multi-paragraph bios and every conference I’ve been to has a different requirement for how that bio needs to be shaped. And the fastest way to success is to just shove it into a chatbot and say, give me a 50-word version. And and then read it and make sure it didn’t invent things or give you a PhD or anything like that, and then ship it off to the conference organizer. But this is much, much faster than rewriting it from scratch by hand. And also I think, it’s a good example of something where I have the extended version and I’m going to summarize down. And that usually works pretty well. So level one is ad hoc. It’s kind of sporadic. There’s no standard across the organization. It’s just me saying, this looks useful, or you’ve probably got some use cases in this space as well. BS: Right. So it it kind of aligns with I guess level one of the the content maturity model that we talked about a while back, where level one is is simply content exists. Could be, you know, someone typing stuff up in Word or, you know, using a myriad of different tools, no style guide, just kind of getting content out there because people need it. SO: Yep. So level two is tactical. And tactical is sort of like we’re using this tool to solve some specific problems. And what you’re going to see here is something like that Bill has invented some nifty time saving tool and he has shared it with other people. Or in a larger organization, maybe somebody invented a nifty validate or or something like that and they’ve rolled it out across maybe the department, probably not the entire organization. Aomething like AI support is being rolled out. Maybe the organization has created a chatbot internally for customers, right? So there’s a chatbot, it’s sitting on the company website, people can use it to get answers, but it’s really bad. And the reason it’s really bad is because nobody thought too carefully about the content going into the chatbot, because again, we’re tactical. So probably this looked like the AI team just raided the local SharePoint, grabbed a bunch of content, did not pay a whole lot of attention to the question of whether this content was up to date in release status. Those things don’t exist, right? It’s just, look, a bucket of PDFs. Cool. Let’s dump them into the AI and go for it. BS: Mm-hmm. SO: And tragically, in many cases, the techcomm team is sitting on rigorous, structured, vetted, approved content, and nobody remembered to go ask them, can we have your content? Or where is your official content source? Or how do I know what version belongs with which document? BS: Right, because you know, in in their point of view there’s a PDF of it, so I don’t need to ask them. SO: Yeah, it’s it’s just PDF. How hard could it be? So something like AI was support was rolled out, but nobody really thought about it. Maybe it’s at a departmental level, probably it’s not enterprise-wide. And nobody has really thought about connecting this AI thing to the assets inside the organization in a reasonable, rigorous, governed, organized kind of manner. BS: And I suppose that’s where you get to the next tier. SO: Right. So the next tier after tactical comes strategic, right? So we have an actual strategy. Now, one of the difficulties in talking about AI is that AI is a tool and it’s kind of like talking about electricity. You can apply it to lots of places and it’s more sensible in some places than others. But when we say what’s your AI strategy, like how do you use water? I mean, come on, and the answer is of course to drive the AI and you know destroy the environment. But there are things that you can do with AI that are useful for content. There are also things that you can do that are not. So if you have a strategic approach to this, a strategic approach to use of AI, backing up to the authors again rather than the delivery side, maybe this looks like a collection of prompts that have been built that are shared. BS: Not with electricity. SO: Maybe this looks like saying this is the workflow that you employ. These are the kinds of things that we do to actually test whether this thing is working. these are the metrics that we’re following. So there’s an actual overarching bigger picture that somebody’s thinking about that goes beyond, let me go shove this into chatbot of the day. BS: Mm-hmm. Right, right. SO: So there’s an actual strategy for the public-facing chatbots. Somebody has thought about the back end. The authors have useful AI tools that add to their you know their productivity. One of the things that I’m hearing a lot now, you know, low-hanging fruit, release notes. Nobody wants to write release notes. It’s a terrible drudge task. It’s and it needs to be done. Well, BS: Mm-hmm. SO: There’s now there are now a lot of solutions that look like look at the diff in the code, look at the delta from you know version one to version one dot one, find the diff in the code, find the changes that have been made, look at the JIRA tickets that have been addressed, that have been solved in release one dot one, and then consolidate that all into a set of release notes that say, here’s what’s been done. And that’s probably 90% of the work, and the last 10% of the work is read that and make sure it’s accurate. Right? Don’t please don’t skip that step. Like actually look at what the thing is generating. Now, what’s interesting to me about level three, this sort of more strategic approach, is that what you’re gonna start to see is that you have prerequisites for this. You can’t do this. BS: Yes. SO: So release notes are good example. Let’s say that hypothetically, and this is gonna sound insane, but let’s say that hypothetically, you have software development and you have no source control. BS: Hmm. SO: Everybody’s screaming, right? Because this is nuts, and why would you ever do this? Okay. But hypothetically, you have no source control. Okay. How do you know what’s changed between version one and version one point one? BS: It’s up here in my head. SO: Excellent, great. Ha okay, cool. so I’m gonna need to connect the AI to your head so that we can pull those changes out of your head. BS: That sounds fun. SO: Yeah. Amazing. Right. So all of a sudden, because we want this automation, right? We want the ability to go in and extract release notes and do something with them. We have to have a certain level of maturity on the software development process so that we can grab the appropriate information. Now, the same thing is of course true on the content side. You have to have a certain level of maturity in your content development processes, in your content management, so that you can identify, you know, the right things to process and the right things to access. And why it is that, you know, we know that software has to be governed, but we’re not so sure about content is a mystery to me. BS: I had never understood that. SO: Yeah. So there we are. Okay, so that’s kind of like a level three. With there’s some sort of strategy emerging across the enterprise. There are some useful tools and they’re shared. This is kind of like in content when you start thinking about templates. We’re gonna have some templates and we’re gonna give them to people and they’re gonna use them and it’s gonna be great. All right, so level four is governed, managed. And so now good understanding of AI. BS: Mm-hmm. SO: It is being applied in a useful, intelligent manner, by which I mean don’t apply it to the wrong problem sets, right? Apply it to the things where it makes sense to apply it. Thinking about governance, thinking about metrics, thinking about success. And then your data sources and your content sources are being managed in such a way that the AI gets good input and can actually generate good output. So I’m actually not a big fan of the term human in the loop because human in the loop implies that the AI is doing all the work and then like the human eventually gets around to QAing it. You know what? We’re terrible at QA. You know who’s good at QA? BS: Mm-hmm. AI. SO: No, computers, not AI. AI is all about probability and whatever. It is actually not very good at QA. What’s good at QA is traditional software, right? One plus one is always two. In an AI, one plus one, sometimes it’s not two. So you manage that stuff and you put those guardrails up and you start putting up the guardrails that say, okay, when the AI kind of wanders off into the wilderness, we’re gonna like bring it back to reality. We’re gonna have, we’re gonna put it in a box, right? Make the AI think inside the box, and we’re gonna govern what that box is. AI is great at thinking outside the box. Unfortunately, that’s usually not what we want from technical content. So it needs to be in in the box and it needs to be consistent and needs to be managed and all the rest of it. BS: Mm-hmm. SO: So we govern it, right? We go in there and we make sure that the processes and the tooling that’s being put in place and the automation that’s being put in place where we’re leveraging or using AI to do things is managed. And so the human in the loop thing. I don’t want the human in the loop to fix things on the back end. I want the human in the loop to fix things on the front end so that what goes in is better, so that there’s less work to do when it comes out. You know, fix it beforehand. Don’t remediate it afterwards. That’s a boatload of work and it is not fun. So fix it ahead of time. BS: Right. Yeah. And likewise you probably wanna have, you know, some guardrails in there so that, you know, your AI, whatever it is, doesn’t go playing around with content that has been approved and released and is not slated for updating. SO: Yeah, you know, don’t fix that. That one’s done. That one’s and you know, we’re not even talking here about what it means to be in a regulated industry or in a regulatory environment. there if you are shipping or sorry, if you are a large organization and you are doing things in Europe, then you are likely subject to the European, the EU AI Act. BS: That’s a completely different beast. SO: And you have to think about what that means for what you’re doing, because the fun gold rush wild, wild west strategy of just throw AI at everything is not gonna fly in Europe. Okay, so that’s governed, you know, hypothetically. And then level five is agentic, which is basically that everything, everything or a lot of it is running autonomously. BS: Mm-hmm. SO: You know, the layman’s explanation of what is agentic AI, the difference is that instead of saying I need to put a prompt into the chatbot, it does it itself because you’ve built out the systems that drive all of that happening. BS: It understands what needs to happen at what point in time. SO: Well, let’s not say understands, but yes. I’m trying so hard. BS: Well, yeah, not understands, but there’s a workflow in place that the AI is following. SO: And it’s so difficult. I think that, you know, there’s, as a side note, the why do we think, why do we impose personality on the chatbots? And the answer is I think that psychologically it’s very, very difficult to interact with something that play acts at human interaction. What a great idea! Good for you. I love your thinking, blah, blah, blah. So it makes you think you’re interacting with a human. And I don’t think that our brains are equipped to say, no, actually, this is a machine. BS: It’s like a scary version of Teddy Ruxpin. SO: It, well, it passes the Turing test. And so we just can’t separate if it feels like you’re interacting with a person, you know you’re not, but it feels as though you are, and feeling is always gonna win over knowledge. So BS: Mm-hmm. Well yeah, the interaction is a lot more organic than you get from, you know, traditional tools. SO: Or, you know, yeah. I mean, think about the difference between a search, typing in a search string, and you know, a conversational search, a conversational interface. It’s quite, quite troubling, actually. Yeah, so this is kind of the five-level model, right? From big mess ad hoc, some things are happening, th some things aren’t, up to it’s completely autonomous. Now, if it’s going to be the more autonomy you want, the better your inputs have to be, which circles us right back to, and therefore, you have to do the work on the content side, because if you don’t do the work on the content side, the AI is going to go off the rails in interesting, unexpected, and potentially disastrous ways. BS: It will play with the mess you leave it. SO: Yep. So that’s where we’re going with this. That’s the AI content ops maturity model as it stands today. I reserve the right to change it tomorrow. BS: Today. Of course.So this model came out of I guess some little nifty side project you’ve been working on recently. SO: I am working on a nifty little side project. We’re not quite ready to announce it. but I’ve got a a co-author and we’re working on a thing. BS: Fair. SO: I could say more but then I’d, you know, be in trouble. BS: When might you be able to say more? SO: I believe that we have a webinar coming July 22nd , where we will say some more things. BS: Alrighty. Well we will learn more things then. SO: I too will learn more things and probably we’ll have to we’ll probably we’ll have to change everything we’ve done up until this point because everything will change by then. BS: Of course. Guess that’s a good place to leave this podcast. Thank you, Sarah. SO: Thank you. Want to know more about Sarah’s nifty little side project? Register for our upcoming webinar . The post From ad hoc to autonomous: The AI content ops maturity model appeared first on Scriptorium .

June 1, 202642 min

Tool selection and the unpredictable variable

How do you really choose the right documentation tool? In this podcast episode, Sarah O’Keefe (Scriptorium) talks with Paweł Kowaluk and Michał Skowron (Guidewire Software) about building a successful tool selection process, the realities of docs as code, and what happens when the technology becomes the unpredictable variable. Paweł Kowaluk: It’s funny how programming used to be deterministic, and it was the people who were messy. We always knew that people are going to be whimsical and maybe harder to rein in, but the technology is going to be predictable. Whereas now, technology is not predictable anymore, and you give it a prompt and you hope it’s going to do what you want. You adjust the system prompts and change the weight of things which are retrieved versus metadata, et cetera, and it doesn’t always work the way you expect it to. Sarah O’Keefe: And now the people are being asked to be the deterministic layer, right? To be the QA on top of the AI. Paweł Kowaluk: That’s actually very insightful. I like that. That is true. The human in the loop or whatever you call it, that’s supposed to be the voice of reason. Related links: Scriptorium: AI in the content lifecycle Tech Writer Koduje podcast Tech Writer Koduje: DITA as code – a modern approach to the classic standard Tech Writer Koduje: Are people abandoning docs as code? Tech Writer Koduje: A tech writing CCMS can also be a broken promise LinkedIn: Host: Sarah O’Keefe Guest: Paweł Kowaluk Guest: Michał Skowron Tech Writer Koduje LinkedIn profile Transcript: Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction Sarah O’Keefe: Hey, everyone. I’m Sarah O’Keefe, and welcome to the podcast. In this episode, we are going to talk about tool selection with a couple of special guests. With me today are Paweł Kowaluk, who is a software architect at Guidewire Software, and Michał Skowron, who is a documentation tools developer, also at Guidewire. Both of them are based in Poland. Welcome. Paweł Kowaluk: Hi. Michał Skowron: Hello. SO: I am glad to have you. For those of you on this podcast that speak Polish, you’re probably already aware that they have the one and only techcomm podcast in Polish that is available out there, and Michał and Paweł are also experts on doc process and tool selection, so that’s what we wanted to focus on today. So I will start and throw it to Michał and ask you the big picture question, which is what does a good tool selection process actually look like? MS: For me, good selection tool process would be divided in three stages. The first one would be gathering requirements, looking what’s out there, defining what you want to basically achieve with this new tool. Then I would go to a pilot project where you can actually test the selected tool in the real world. Manufacturers and producers of software will tell you that it can do anything and it will promise that, “Okay, you can meet all your requirements easily and we can fix that, we can improve that, we can adjust that,” so everything can be done is usually what we hear, but then you want to test it in real world on a real project, so that will be a pilot project for you and your team. And the third phase that depends on the outcome of the second phase, which is you either productize the selected solution or you just say, “Okay, that was a bad choice and we don’t need that.” Then we need to go back to the first stage and then say, “Okay, we need to select another tool,” and again, requirements, et cetera, et cetera. So for me, that’s the whole process, and the first stage would be probably the longest one because you need to make sure that you are meeting all your goals. SO: So what’s the most common reason that a pilot doesn’t succeed, that you have to go back and say, “That didn’t work. We have to try something different”? MS: It’s usually because you didn’t see everything when you were planning. For example, you have some projects that are very specific or you didn’t see all the problems or things that are coming your way. It’s hard to say exactly what the reason is, but it can be multiple reasons. For example, using of, I don’t know, branching, let’s say, in a specific tool. When you have multiple versions of your product and you want to keep them separate when it comes to documentation, it can turn out that the feature says, “Okay, you can use branching and then you can do it easily,” and then you start using it and it turns out that it doesn’t work the way you expect it. This is actually a real life example because we had a system that… I’m not going to mention any names or anything like that, but there was a system and they promised us… That was years ago and it was a vendor that promised us that they’re going to introduce a feature called branching, and it turned out after they did that that it wasn’t what we expected. So it can turn out in many cases, in many ways, it can be the problem, but branching is just an example, but it can be many other things that can go wrong. PK: Hey, if I can jump in here, I got a couple of examples. One is I could call it releasing strategy or versioning strategy overall, which is very hard to test in a pilot project. It’s very hard to scope for requirements because the little problems come out after a while, after a year of publishing, after two years of publishing. And another example which is related is reuse, and this one is down to formulating the requirements correctly. Because I think just saying, “We want to reuse something,” is not enough, because you have to say exactly what you want to reuse and how you want to reuse it and what you want the result to be. So for example, if you say, “I want to reuse notes and warnings and things like that.” We sometimes call them admonition. So, “I want to reuse these notes in my docs, and if I update a note, I want it to update in every published version of the doc.” Then only if you have these details, like I want to update it once and then want it to automatically update and publish docs, then you will see it’s not working the way you expect it. Because if you just say, “I want to reuse notes,” every system can reuse notes. Even in docs as code, there’s scripts and macros that allow you to reuse notes. MS: It can be also another thing that, for example, you compare the benefits with the actual cost of implementation, and it can turn out it’s not worth it because people are, for example, reluctant to use your new tool. The training, the cost of licensing, the cost of support is too big, and then you realize, okay, we want to achieve a goal like, I don’t know, reduce the time to market, and then it turns out it doesn’t work because people are struggling with using the product on a real project. And on paper, everything looks cool and you have all these features, you can use them, like Paweł mentioned, for example, reuse conditional formatting and things like that. And then it turns out it’s very hard to use, it doesn’t serve its purpose, and then you have to pay for every additional stuff and people don’t want to use it, so what’s the point? SO: Yeah. And I think we find that the technical problems in general, if you’ve done your requirements work, then usually, the technical problems are solvable if the people engaged in the project want to solve them, and that’s where you run into the change management issues that you’re talking about, that if the team that is being asked to pilot, to test, to try things out is sufficiently disinterested in making it succeed, they will find a way to make it fail. And the reverse is true as well. If they want it to succeed, you can implement tools that are … Well, all tools are imperfect, but you can implement tools that are not perfect solutions and succeed if the team is behind you, and if they’re not, bad, bad things will happen. PK: Oh yeah, that’s true. And I’ve been on projects where we did not do proper change management and I’ve been on projects where we really did it well. If you start early and you involve people, like get the biggest troublemakers, people who are the most opposed to any change, get them on the team, and if you can convince them, that means, one, you are making the right choice because you’re convincing people who are skeptical, and then two, you are set up for success. These people are going to be your biggest champions of the new solution. MS: But it’s good that you mentioned it because I think it’s worth emphasizing that the goal of the pilot project is not to succeed. That’s not the actual goal. The goal is to verify the requirements against the real project, and so the failure is also a success to some extent. It’s not like you have to do everything to prove that the selected tool is the right one. No, you should be aware that the pilot can end up with your let’s call it failure, and then you realize that it’s either a bad choice or you don’t need that at all. For example, you don’t need that tool at all. It can be also the outcome of the pilot project. So there are many different outcomes, so don’t be fixated on the success path that is the only right way. It means, okay, the pilot project was a success because everybody agreed that that was the right tool that we want to use, so keep it in mind. SO: Yeah. I think that there’s … I got into actually a debate with somebody about this. They were telling me that pilot project, proof of concept, and what was the third one? Prototype are not the same thing. Okay. Well, so a pilot project is, “We think this is the right answer and we’re going to try it small and then we’ll expand.” A prototype is, “We’re just going to try it and throw it away,” and a proof of concept is, “We think this is the right answer, but we’re not sure and we’re prepared to throw it away.” And I thought, “This is way too specific for me, but okay, sure.” But to your point, the project, whatever category it belongs in, has different purposes, and it’s important to be clear about what kind of a project is this? Is this essentially beta testing, this is step one, or is this more speculative? We’re just really, really not sure. PK: I think it has to be falsifiable. Like they say in the scientific method, there has to be a failure criteria somewhere. This will fail if ABC happens, because if you don’t have that- MS: Or a success criteria, right? PK: Or a success criteria, but I’ll be more focused on the failure criteria because you can always show, “Yeah, we accomplished this, 30%,” but if your failure criteria is below 50% is fail, then you will say, “I failed this.” You fail five out of six and the project is a no-go. SO: So I wanted to switch gears a little bit and talk about some specific tool chains and problem sets. I think it’s fair to say that the two of you are, I’m going to say mostly but not exclusively, focused on docs as code, so what does that look like? And there’s a lot of debate about docs as code versus structured content and they’re very much pitted against each other, but ultimately, what’s your perspective on the situations where docs as code is appropriate or maybe is not appropriate, and what do you look for in a project or in a problem set that matches the docs as code model? PK: I think the reason people associate us with docs as code is the last eight years, we’ve been working at Guidewire and that’s the strategy we’ve chosen there, so I think we’ve been entrenched in this worldview. My previous job before Guidewire was a consultant, and I would go from a company to company and set up different systems for documentation. That was my specialty, systems for publishing and updating documentation. So yeah, circling back to your question of when it works, when it doesn’t, I guess docs as code is more about it works when the right mix of people are creating the docs and the mix of people includes software developers. So for example, at Guidewire, we have several dozen static websites which are maintained by software teams without any technical writers involved, and those are a variety of internal tools, tools which are almost external, and then tools which go out to customers, and all of these little websites integrate very well with our publication system. And I think this is the main criterion for docs as code fitting is you are giving tools for writing to people where they work. So software developers work in code. You give them tools for writing in their codes so they don’t have to buy extra licenses, get training on anything external, use some alien process, alien to them, just because they follow SDLC, the software development lifecycle to update their docs. And it’s what they’ve been doing, and then it’s easier to convince or it’s easier to fit in a smaller team of technical writers, I don’t know, like 40 technical writers versus 2000 software developers. It’s kind of everyone contributing to the same documentation system. It’s easier for the tech writers to adapt and join the docs as code platform than the other way around. MS: Just like Paweł mentioned, docs as code makes sense in certain environment. It’s not like we are tribal about it. We love this solution because it works for us. So after a few years at Guidewire, we realized that this is the way we want to go, because as I said, it makes sense and it just works. We tried different things before, and before I joined Guidewire, there were different solutions. We used, for example, CCMS and we had more a traditional approach to producing docs, let’s call it this way, and then we started building our own pipelines, our own solutions. We started integrating with what was there that other devs used for their work, not related to documentation, but we decided, “Hey, maybe we can use what’s out there and just plug into the same infrastructure.” So as I said, it’s not for everybody, it doesn’t work in every environment, so I wouldn’t say, “Okay, if you’re in a factory, you should use docs as code.” No, I wouldn’t say that. So maybe we positioned ourselves as docs as code proponents, let’s call it this way, but this is because we use it every day, we build it, and this just works for us. And also maybe I can mention that we decided to end this holy war by putting DITA into Git and CICD pipelines, so we have DITA as code, and now everybody’s happy. PK: Oh, this is actually a great point because Sarah, also mentioned docs as code versus structured. Now we’re doing docs as code in a structured way, and where technical writers are working on the docs, they are free to use DITA and a lot of teams do that with all the reuse and all of that, but even if it’s something simple like some markdown files in a repo, we still impose a metadata structure which allows us to integrate the doc into our publishing pipeline, and the two key aspects of what we’re integrating with our authentication and search, and that needs metadata, right? It needs to know who has access to this content and it needs to know how to filter and direct the searches, and from that, our next project emerged. Like everyone else in this industry, we started working on AI solutions, and the AI solution is a third integration point where this structured approach to content also pays off. SO: Yeah, I did want to touch on AI and so we should probably just jump to that. What are the implications that you’re seeing? What are the effects that you’re seeing on your current processes of the use of AI or the requirement to deliver to AI, and how do you see that working? MS: Paweł, you want to start? PK: Yeah. So the content remains the same that we’ve been working on for years, and the structure, the metadata is all the same. We could not change this overnight, but overnight, we had to introduce this AI which is going to look into the content, find the topics, the chunks we call them. It’s going to find the right chunks of documentation and generate answers based on those chunks. So what happened is we had to adjust the AI, but since we were so structured, it was pretty easy, and then we gained a new source of feedback from customers through the AI. Because when people started using our chatbot, we built a chatbot which is available on docs@guidewire.com, but it’s only available to customers and partners, so you need a login on the website. When you go there, the AI answers questions, and then we monitor the exchanges that people have with the AI and we see what needs to improve as far as content ingestion, as far as content structure. We generate a lot of internal tickets to our tech writing team and identifying gaps, because we’re seeing now that this AI interaction is possible, here’s what people are asking of the content. And other people are asking, “How do I implement X, Y, and Z in the scenario where ABCD?” Which we didn’t know people were looking for before because they had no interaction with the docs. So they just come to the website, they either find or don’t find what they need, and we don’t know anything about that, and we could ask them, but we’re just going to ask a small group of people, whereas now, it’s as if we’re asking everyone. So we’re seeing these new patterns and something emerges out of those patterns, like everyone’s asking for code samples in this specific area, so we go back to the team and we work on adding more code samples, or people are looking for illustrations of these particular workflows, so people create these illustrations. It’s amazing how rich of a source of feedback this has become. MS: And it also adds complexity to even testing all the solutions that we have right now, because with keyword search, I’m not saying it was easy because it wasn’t easy, but now we have another layer where you have this middleman, let’s call this chatbot, let’s say. It’s a middleman that can hallucinate, so you can either have a problem with your content or you can have a problem with the agent that responds. So before, when you had a keyword search and you were missing results, you were going usually to the content and see, “Okay, I’m missing this topic. It wasn’t indexed properly, or I just need to move some knobs in my search engine and just see the fuzziness or something like that.” And now you have this content, it’s being processed by this AI tool, and then it gives you some answers and you don’t know why the answer is wrong. Because it didn’t find the content? Because it’s not there? Because it’s not the right content? Because the content is right, but there is something that it missed? There are so many moving parts right now, more than before that it adds complexity to our job, and this is just one side of it. The second thing is writers using AI for creating content, and we’re also exploring this path because everybody’s trying to incorporate AI solutions into their work. We code mostly on a daily basis and we use some tools that help us with coding that are AI-based, but we also want our writers to benefit from all this development in technology, and we’re exploring, let’s say Oxygen, XML, Positron. Just to clarify, we’re not sponsored anything. We’re just using it so I’m just mentioning the name specifically, but even if you don’t have any specific tool for writing that integrates directly with a AI tool, you can still use, let’s say, Copilot or any other solution that you like and you can just ask it to help you with the content, but it comes with a lot of caveats. It’s not like you just throw a prompt and just get what you need. It involves a lot of fine-tuning, a lot of working on instructions, giving it context, giving it information that it needs to use for producing code or docs. Because I use it for both, for coding and for documenting, for internal purposes mostly, but it takes time to make it work the way you want it. PK: Yeah. It’s funny how programming used to be deterministic and it was the people who are messy and the processes, and working with setting up the process for the doc tools, we always knew, people are going to be whimsical and maybe harder to reign in but the technology is going to be predictable. Whereas now, technology is not predictable anymore, and you give it a prompt and you hope it’s going to do what you want. You adjust the system prompts and change the weight of things which are retrieved versus metadata, et cetera, and it doesn’t always work the way you expect it to. SO: And now the people are being asked to be the deterministic layer, right? To be the QA on top of the AI. PK: That’s actually very insightful. I like that. That is true. The human in the loop or whatever you call it, that’s supposed to be the voice of reason. SO: I don’t know about you, I’m not good at voice of reason. I’m much better at causing trouble. So what you’re describing is a fairly complex and time-consuming approach to this in that you’re saying we have this AI chatbot and we’re looking at the metrics and we’re adjusting accordingly and making changes, and then using the AI on the backend for some productivity kinds of things, but not as a replacement. We’ve seen in the US and in North America, we’ve seen a significant number of people losing their jobs because the idea is that, oh, the AI can just do it, and what you’re describing is not that at all. So are you seeing any of this sort of, “Oh, this will make us more efficient, and therefore, we need fewer people,” or is it a different perspective in your piece of the tech com world? MS: I’m aware of all the layoffs and of all the bad things that are happening right now in the IT world, let’s call it, because it’s not only technical writers because it’s also developers and different jobs, but I think I’m still in this kind of a bubble right now where it’s not happening directly next to me so maybe I’m being too optimistic. But I keep saying maybe I’m going to eat my shirt after some time because I keep saying, “Show me one tech writer that doesn’t have a backlog that is not too long.” So we usually have too much work, and I hope that with the right approach and with a sensible management, these AI tools will be doing all the things that we don’t want to do or we don’t have time to do, and then it will give us time to do something more and something more significant. I’m looking at my job and I’m not saying it’s perfect because of AI, because it has so many challenges right now and it gives you different problems, but I can see that I can speed up my cumbersome tasks very, very easily. And before, I had to … This is a simple example, and I’m doing a lot of infrastructure work and a lot of backend work, so many things go wrong, and very often, I had to debug difficult problems. It was taking me weeks sometimes to nail the actual place where it happened. Now, I can do it much faster. If I have to debug a problem, it takes me, I don’t know, an hour, sometimes even minutes, and I know that I would spend a day, two, or even a week if I didn’t have these tools. But these are good examples, but there are also a lot of bad examples, but I don’t think we have time for that. SO: We’ll take the optimistic view. PK: Yeah, I agree with Michal. The approach we have is these tools can give us tremendous productivity boosts, but not in the sense of getting rid of people. We can redirect people’s work to where it’s more meaningful. And what I mentioned about feedback, identifying these content gaps. Some of these content gaps are going to be very mechanistic and you solve them by generating a bunch of docs out of code, for example, and then you put those docs in the repository where the chatbot can find them, because they’re not going to require a lot of thinking. It’s kind of like API docs where you create the swagger spec and then you generate the dogs out of that. You just grab the source code and you generate some samples, and you use that to generate answers to people. Without people, this wouldn’t work because you need, like we said, the voice of reason, but yeah, people are still required in the process. SO: I think that looking at it across the industry, if you take AI, take a chatbot, especially a public facing chatbot, and ask it for answers on literally anything, it will give you the average of what’s out there in the world because it’s math, so it gives you the average essentially. And so from a content creation point of view, I think the fact of the matter is that there is in fact a lot of below average content out there. Roughly half is below average, or perhaps exactly half. If the information is bad enough, if the content being produced is not of high quality or ungrammatical, and we’ve all seen terrible, terrible documentation that was badly translated and is just incomprehensible, then the AI as a tool for creating content may be able to produce something that is better than terrible. It’s not going to replace a professional, well-trained, highly experienced and knowledgeable product documentation group, or it shouldn’t, but that’s not every company. Not every company has a really good group of tech writers that understand the product and are adding value as they produce the content that goes with that product. And so I suspect that what we’re going to see is a split with commoditization on one end, just auto generated, it’s not great but it’s adequate maybe, and then the higher end stuff, which needs to be done well. PK: Well, there’s definitely documentation which only exists because it absolutely has to, and companies like that can easily generate just some kind of documentation and just use it, right? But in cases where … I think the worth, the value of a technical writer is not just writing and generating the content from below their fingertips. It’s more about the research and understanding, like you said, understanding the needs of the users and understanding how to meet them. You throw AI at a problem which sounds like a generic problem, it’s going to give you a generic answer, not necessarily one that is rooted in the organization you’re in. The list of products that you have, the way those products interact with one another, it’s going to miss all that. It’s just going to give you … It’s like ordering a hamburger and you say, “So what’s in the burger?” And the AI is going to tell you, “Well, usually in a burger, it’s a patty and lettuce.” And it doesn’t mention that at your restaurant, you use pear and avocado. It doesn’t know that. You know that. You’re the chef back in the kitchen and you know why your burger is special. SO: Yeah, I think that’s right. So I did want to touch briefly on the question of, because this is a rare opportunity to talk to some folks that are not based in the US or North America, what differences, if any, do you see in tech writing in the market? Now, you’re based in Poland but working for a, I think, US company, but what kinds of things do you see that are maybe different that especially the people listening to this in the US would not be aware of coming out of Eastern Europe? MS: For me, always, it was the fact that tech com appeared much later than in the United States, in Poland of course, because I’m not talking about Europe in general because there are also differences between countries in Europe. For example, Germany is totally different from I would say the Polish market, because in Germany, you usually have factories and you have technical writing associated with hardware, let’s say. SO: Yeah, heavy industry, machinery. MS: Yeah, heavy industry, that’s right. And in Poland, it’s very often about software, maybe because we have a lot of companies that outsource to Poland when it comes to software development, R&D and stuff like that, so that’s the first thing. And don’t get me wrong, people were doing tech com way before it appeared in the mainstream, but sometimes they were not even aware that this is called technical writing or technical communication. Actually, it happened to me because I moved to techcomm from some kind of IT support job, and then after a year, I was like, “I’m wondering if this is anything regulated or there are some rules or it’s actually a profession.” Then I started digging and here I am. But it turned out there are so many things, so I was also surprised. Okay, this is called that. These are standards. There are books. Well, actually, one of the first books that I read was your book, Technical Writing 101, which I’ll still recommend, although it’s been years since it was published the first time, but I think it still holds a lot of truth about techcomm. The core values of techcomm are still there. So going back to your question, the techcomm scene is relatively young, let’s call it this way, in Poland, which gives us a big opportunity to skip some stages, let’s say. So we don’t have the legacy, we don’t have some things that we used to do a certain way and we like it or we are just accustomed to it. We can just start fresh and just jump. In 2000s, we just jump into techcomm and we say, “Okay, let’s do it this way because now this is the way we do it.” And I think that people are flexible when it comes to looking at things a certain way. Not everybody because there are also people who don’t like changes. But I think also what is unique about Polish techcomm scene is it’s relatively small, so if you do something outside your work, you don’t treat your job as only a means to earn money and just survive and you want to do something else, let’s say just write an article, like give a presentation, go to a meetup. After a year or two, you just keep meeting the same people, you know basically everyone who is in the circle, so it’s much easier to be visible if you do something outside your work. So I think that would be something unique about our techcomm scene. PK: I think what might be a downside of the Polish techcomm scene is a lack of veteran experts, because we don’t have people who have been doing this for 40 years. I’ve been in the industry for 18 years. SO: But you do understand that you two are the veteran experts. MS: That’s what he’s trying to say, that- PK: I’m getting to this. So yeah- MS: Imagine that. PK: Putting on my clown makeup every morning. What I mean is there’s nobody to look to who wrote the book on technical writing and has been there for 40 years. I’ve been here for 18 years doing this thing, and out of the people who are visible publicly and who contribute to the scene, I don’t know if there’s anyone who has more experience than me. Because I know there are people who have more experience but they’re just not visible. They don’t share their knowledge with anyone, and we miss that, so that’s why we look to the West, to people like you guys, like Scriptorium, and we read books which were created there and we see there’s definitely value, even though something is as old as the scene here or older. MS: And people also have this tendency to look at things that were done in the past as like, “This is the old stuff. We don’t need that. This is the old way. Let’s do it this way because it’s better now, because we have all these tools and et cetera.” But I think it’s a big mistake, because what they say? History doesn’t repeat but it rhymes, something like that. So in order to do your job in the present, you just need to look at the past, and this way, you can also be ready for the future. I think it’s worth looking at all this, let’s call it legacy stuff, all the experience that people with 30, 40 years of experience in the field have because it’s valuable. Because usually, when you look at it much deeper, it’s nothing new. It’s usually something that already happened but it’s dressed up as something else. So if you look closer, it’s usually something that let’s say you can understand what happened and then you can apply the same rules, maybe slightly change them or bend them. But my experience is the same happens in software development, in coding. People are inventing new things, and when you look closer, it turns out somebody already said that in the past. It was already done this way, but it’s dressed up as something else or named differently or is hyped more, or it was invented too early and now is the time to use it. So the history gives you a lot of perspective, a lot of things that you can use, so don’t dismiss it. SO: Well, I think that’s a good point, and as we close this out, I want to ask you what you see in the past or where you’re gaining perspective on the introduction of AI. Whether it’s from a delivery point of view or a backend productivity point of view, where do you see that pattern previously, or do you? PK: There are similarities, but they end. They’re not perfect analogies. So going digital was on example, and when I started, it was just on the brink of companies going from print to web, and that was a huge paradigm shift. And we’re seeing reverberations of this even today where teams are still thinking in books and chapters instead of thinking in pages and thinking about even every page is page one, which is, I don’t know, it’s older than me, the idea, but it still hasn’t been adapted to by teams. Now, the AI thing is probably a similar earthquake where it’s the AI reading your docs, giving you answers. You’re using AI to generate these docs, et cetera, et cetera. I don’t think we’re going to get 20 years of leeway to adapt to that and still see people doing things the old way. I think the world maybe moves faster nowadays, but I don’t know, we’ll see. MS: It may be something similar to the invention of computer. So instead of writing let’s say manually, people have got this new device that gives you more power and you can do more things and then programming languages and everything, but as Paweł mentioned, the pace was much, much slower. So we had years of development to, let’s say slowly, maybe this is the bad word, let’s say slowly adjust to the change, and now it’s an earthquake. So every day, every week, there is something new and you need to keep up, keep up, so the pace of development I think is the biggest challenge. Because we tech writers survived many revolutions. I just started reading a book by Sharon Barton about the women in technical communication, and women who talk about their careers, they mentioned many, they start very early because there are a lot of examples from the States. So they started working when they were not even treated equally with men, which was years ago, and they mentioned all those changes, all those revolutions, all those evolutions. And I’m saying, “Okay, this is what we are witnessing right now, but the pace is definitely faster, so we need to just keep up,” which is hard, of course. PK: Well, I have another one which is a good one. I don’t want to not say it. The dot-com bubble. I think maybe we’re in a bubble again, maybe. You’re seeing the inflation of AI prices right now, and I think we’re going to end up in a position where you cannot do all these things with AI which we’re hoping to do with AI, so the surface of usage is going to shrink and there’s only going to be specialized uses and special case scenarios when you use AI because it’s going to be expensive, and a whole lot of AI applications are just going to disappear. This might be a parallel, but I don’t know, it’s always hard to predict, especially the future. MS: I’m also predicting some corrections. Let’s call them corrections. SO: And I three am also predicting some corrections. You touched earlier on the fuzzy. AI is very good at fuzzy problem solving, but it’s being thrown at things that are old problems that we know how to solve with scripting. And scripting from a computing power point of view is cheap or maybe free. You write your script and you run it, and every time you run it, you get the same predictable result. Now, sometimes you don’t want that. Sometimes you need that fuzzy pattern matching thing, but in the cases where that’s not what you want and we’re still using AI, that is part of the bubble, right? Everything looks like an AI-shaped problem. So yeah, I agree with you. I think that any predictions we make on specifics as to where this is going are guaranteed to be wrong, because I’ve tried this before and I’m always wrong. So I want to thank you both. This has been very entertaining and very interesting, and I hope that we will see you again and you’ll come back and tell us more about what you’re up to over there. PK: That would be lovely. I would like that very much. Thank you. MS: Yeah, sure. No problem. Thank you very much. That was fun. SO: Yeah. Thank you both. We will see you soon. Want to learn more? Download our book, Content Transformation. The post Tool selection and the unpredictable variable appeared first on Scriptorium .

May 18, 202624 min

Taming AI: Using AI for content conversion at scale

AI promises to transform content conversion, but what does it actually look like when you’re processing thousands of documents a day? In this episode, Sarah O’Keefe (Scriptorium) and Rich Dominelli (DCL) dig into the real-world challenges of using AI for large-scale structured content conversion. Rich Dominelli: If you have millions of articles and you’re asking the AI, ‘What did we do for this project six months ago?” The AI has to find those articles, pull the relevant information out of those articles, summarize it, and hand it back to you. The best way of doing that is to give extra signals to the AI, structured relevant bits of information, front matter, back matter, publication date, keywords, abstract, that allows the AI to query the corpus and get the relevant chunks out of that corpus in a very quick manner. Then, it can summarize what those chunks are. So the AI almost becomes the user interface over that corpus. But to find that data in the first place, structured content is key. Structured content is key when you’re dealing with big indexes and the web, and it’s the same with AI. Related links: Defeating Nondeterminism in LLM Inference (white paper) Data Conversion Laboratory (DCL) Scriptorium, Machine experience (MX): Making content work for humans and machines (podcast) LinkedIn: Host: Sarah O’Keefe Guest: Rich Dominelli Transcript: Disclaimer: This is a machine-generated transcript with edits. Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction Sarah O’Keefe: Hey everyone, I’m Sarah O’Keefe and I’m here today with Rich Dominelli who is a Senior Developer and Architect at DCL. Rich, welcome. Rich Domineli: Hi, thank you for having me. SO: Glad to have you. We were talking before we hit the record button, and you described yourself as a perhaps hopeful AI evangelist. RD: Yeah, I am well and thoroughly immersed in the AI game at DCL and using it and plus I play with AI assistants at home. I’m enthusiastic about the future of AI, sometimes disappointed about the present. SO: So DCL, as I think many of our listeners know, is focused on conversion at scale, which to me makes a great use case for AI because ultimately conversion is about edge cases and about inconsistency, right? If everything was 100% consistent, conversion would be pretty easy. RD: Yeah, no, DCL does a lot of structured content generation out of unstructured data, and the creativity, especially in the academic space, of what that unstructured data looks like is sometimes nightmarish. So the AI lets us, does a lot of the heavy lifting for us when it comes to looking for particular items, identifying concrete data points within the documents, pulling things like authors and affiliation, front matter type information, and back matter type information out of the documents and in automated fashion. It can be painful from time to time, but it’s definitely helped. SO: Yeah, so this is, think, you know, the reality of working with AI and working with it in a production environment in order to address all these weird edge cases and what’s going on. So tell us a little bit about how you’re using AI in, you know, these conversion use cases. What does it look like to go in there and start applying some of these tools that we have? RD: So, I mean, typically our flows work in a way where we’re coming in with a PDF or a Word document or some other unstructured format. We take it, we reformat it into a version that’s more AI-friendly, like Markdown, for example. And that’s usually the first step we’re doing when we’re looking for information to pull out of it like front matter. It’s a very common use case. If you look at academic papers, the front matter, the authors and the affiliations that are on that paper can be formatted in more ways than I could list out during the course of this podcast. It’s kind of crazy. So what we’ve started doing, and we’ve been doing this for a couple of years now, is we’re using the AI, we’re handing it the Markdown document, and we’re saying we need to list authors and affiliations, please extract it for us. Now, naively, when we started that process, we assumed that the AI would give us a consistent list of authors and affiliations. And sometimes it does. But every time you do that call, you’ll get it in a different format. So then you have to start tightening things down. So OK, give me a list of authors and affiliations. I want it to be structured exactly like this. And typically, we have a JSON structure that we’re presenting to the AI, along with our prompt, and saying, give it to us. Well, okay, and that gets you a good chunk of the way there. And that was very exciting when we had that working consistently, we were getting things out of the system on a consistent basis. Awesome. But then you start looking at the results, and every once in a while, you get an author that was missed, or there would be too many authors on that paper. We had one test paper, which I loved, which had 600 collaborative authors in it. And the AI would just choke after about 280-ish. So then you have to start dealing with things like paging through the data and formatting the data. And then you have to figure out, well, did it miss anything? You have 600 authors. Good luck. So now you have to take what the AI did and compare it against your own representation of it and write a program to do that comparison to say, OK, is it good? Is it good? You have to take a step back and you look at it and you say, okay, we have the information that’s in the non-structured format. We’re handing it to the AI. The AI is gonna give us a structured version of it and we need to validate it. Well, the first validation is very easy. Does that structured version match the schema that we gave it? Yes or no, that’s easy. Well, then you have to say, okay, is everybody there? Well, is there anybody added? Because the nice thing about AI is they occasionally get very creative. Even if you have that temperature dial turned all the way down to zero, it will pull names out of thin air and then come back to you with some random name and stick it in the middle of the data where it’s not obvious, of course, and then hand it back to you. So then you have to start saying, are all the names that appear in this list actually in the document? Are the counts matching? And if it’s not, you go back to the AI and you ask it again, and usually you’ll get a better answer the second or sometimes the third or fourth time. But you need to be able to catch that, especially if you’re doing this at scale, because if you’re doing a few, it’s easy, you can eyeball it. If you’re doing 1,000 of these a day, you can eyeball all of them. You can say, you can ask the AI, OK, give me a confidence level, but if you can’t trust it in the first place about what it’s returning, yeah, I’m very confident about what I’m giving you right now. It’s really the truth, I promise you this time. I don’t know how trustworthy that would be. So you have to write tools to validate what the AI is producing, or you have to use the AI to validate what it’s producing. So coming in the first time, obviously, we did the count, we did the schema validation. We then said, okay, we’re going to check to make sure all the names appear in the document, we’re going to have landmarks in the document that we can refer back to. So if you start with Microsoft Word and you have track changes on, you can have paragraph IDs that are supplied. So you can make sure that you can find all of the authors in that list and they all have a paragraph ID and you can have your landmarks and that’s great. Or you can even hand the results to a separate AI call and say, proofread this. Is this accurate? Is this the best answer that could be for each of these? I know we’ll come back with an answer. And you can use that as a signal to gauge accuracy and to gauge repeatability and make sure it’s correct. SO: So you’re, let’s see, generating an AI, not a test bed, but an AI environment that’s doing this conversion or that’s processing the files for you for conversion. And then you have to go in and do all this validation to make sure that the output that you’re getting is actually correct. As compared to, I’m gonna say old-fashioned, but you know, as compared to scripting, deterministic, pretty straightforward, if A then B kinds of scripting. What are the differences between that and AI-driven conversion in testing and validation? What are the test plans? How are they different conceptually? RD: So from our perspective, the frustrating thing sometimes is the AI is completely non-deterministic. SO: Mm-hmm. RD: It can give you a name formatted one way today, and then tomorrow, its formatting might be subtly different, where in the paper it has “Richard Dominelli, Junior.” The AI may decide, well, that comma probably shouldn’t be there, or junior should be followed by a period, and it wasn’t in the paper originally. And you can try prompting around that and tell it to prompt around that and make sure that it’s accurate. But it doesn’t always follow your instructions exactly when that’s the case. SO: And why is that? Why is it non-deterministic? RD: Because AIs are built on a neural network, the neural network itself has fuzzy fields within that, mostly due to floating-point arithmetic. So when you’re looking at it and it’s that weight on that particular key might be out to like 16 digits of a number and it might shift it slightly one way or the other. There is a fantastic paper from, I wanna say it’s anthropic, that goes through the different reasons why AIs are non-deterministic. It goes through repeatedly querying for the AI and who Richard Hyman is and getting back a different answer every single time. They’re all correct. However, they’re all slightly different. The other thing that will lean into that is if the AI is being heavily used, the memory and model weights will shift ever so slightly and you’ll get a different result. So you’ll end up having an issue where today I’m getting accurately this way and it’s relatively consistent, not perfectly, but close enough. And then tomorrow, it may just give you a dumpster fire of random information and you need to be able to detect that. Okay, the other challenge we hit fairly early on is more and more people are aggressively using AI right now. So we’re actually starting to hit issues where the LLM providers are overwhelmed. So you have to be able to code in sale over because you’ll literally get too many, you’ll get 429 errors, which are basically, I’m too busy. I can’t deal with your request right now. Call me back. And you’ll have to go back and repeatedly query to get around that. I am hoping at some day in the near future, we’ll be able to have in-house AI at scale and have these wonderful models that are so intelligent that we can run on our local hardware. And so I won’t have to deal with that, but right now, that’s not the case. SO: So given all of this, I mean, I’ve asked you the leading question about the issues and the negatives, but what then makes an AI-driven conversion appealing versus a sort of scripted, deterministic, if I plug in AI, I will always get B output? RD: So part of it is the type of data we’re dealing with. We’re dealing with unstructured information and the unstructured, the creativity of the unstructured information is rather astonishing. You’ll have people format things, know, we’ll get papers in where the entire paper is placed in different cells of the table. It’s not tabular information at all. They just, you know, we wanted this particular section to be in this cell and this particular section to be in this cell and this particular section. And the AI, I don’t want to say is immune to that, but it’s a lot more forgiving than having to write those reg ex or traditional programming or word interrupt things to try to extract that information, because the AI can address it in a much more fuzzy fashion. I know approximately what an author’s name looks like. I know approximately what a reference looks like. Even though today they decided to do it in Comic Sans or with Wingdings fonts, I can still read that and move on. So that’s really the wonderful aspect of it, is it gets around a lot of that fuzzy logic coding. You’re not dealing with having to address each of these nuances in a generic switch or state machine to try to figure out, OK, this paper should be classified this way and this approach used. Instead, the AI does a lot of that heavy lifting for you. SO: Okay, so it gives us that sort of fuzzier, more, I’m gonna say more flexible, I know if that’s exactly the right word. And then the outcome, what you’re describing is you’re ingesting unstructured word, PDF, those kinds of things, and turning them into structured content, presumably fundamentally XML of some sort, but also some other downstream formats. So I wanted to switch gears a little bit. There’s been a lot of conversation about using structured content as an input for AI. So this, guess, is the scenario where you’ve already ingested the unstructured content, have remediated it in various ways. We now have structured content, and we’re gonna take that and feed it into, I guess, AI part two, right? So we’re past conversion. And there’s a lot of people saying, you should feed structured content into AI, it will make the AI better. And so my question for you is, you know, is that the case, and also maybe why and what goes into structured content that makes it produce better AI outcomes, potentially, assuming that it does. RD: So there’s a bunch of guides out there. There are two pieces of conversation. First, there’s a bunch of guides out there for prompting AIs where they suggest using XML or simplified XML tagging to give the AI signals about your prompt that aren’t verbally expressible. So here is my question. Here is an example. Here’s how I want my output to look like. And you can put tags around that when you’re actually prompting the AI and the AI will know that those signals mean that it should pay attention to it. Okay, so that putting that aside, what I think you’re really asking though, is how does structured content, structured documents, the JATs and the DITAs and the S1000Ds and how does that help the world of AI? And to answer that question, we have to go through two things. One, we have to go through retrieval augmented generation and context rot. So let’s talk about context rot first, because that’s a really interesting topic and people don’t talk about it enough. You have these large language models that are coming out right now and they’re advertising this sticker shock value of, can ingest a million tokens and it has this tremendous memory so you can stick the entire encyclopedia botanica in it, and it will be able to ingest it and regurgitate it. There’s a whole lot of academic work out there that basically says that, hold on a second, practically speaking, once you exceed a certain size, even though they can technically hold that million tokens of data in memory, they’re not gonna be answering as accurately as a smaller model. The most common example or the most easy test for that is needle in the haystack test, where you take a document, you stick a random fact in the middle of it, and you hand the AI the document, and then you ask them for that random fact. Nine times out of 10, it will answer incorrectly. An even easier test is there’s a website which I actually like called A Thousand Names. And all this website is is a thousand randomly generated human names. The thousand randomly generated human names. You take that, you give it to the AI, say, how many names are there? And more often than not, you’ll get, well, when you do 100, you’ll get an accurate answer. 200, accurate answer. 300, things start to break down. You might get 300, or you might get 280, 320. You might get a random answer. And then it gets progressively worse as it gets bigger and bigger. So if you’re working in the context world, content world, you’re looking at ingesting documents into a corpus of some sort. You’re making these structured documents in such a way for the sole purpose of making them retrievable. You want the AI to be able to retrieve those documents and the relevant documents from the corpus so that I can answer the question. A, because your corpus is probably bigger than that million tokens. And B, because the less data you send the AI, the more accurate the answer is. So the better way of thinking. SO: And so a token is roughly a character, right? RD: No, a token is actually roughly a word. It’s less than a word. It’s kind of a lot, but it’s still not like a PubMed-sized corpus or anything like that. It’s roughly the size of the New and Old Testament of the Bible, roughly a million words. So just give everybody that mental picture. But that’s just one book. SO: Roughly a word. So a million tokens is kind of a lot. It’s a lot of words. RD: So if you have millions of articles, or and you’re asking the AI, you know, what did we do for this project six months ago that involved JAPs in this solution? And the AI has to say, okay, it has to find those articles, and then it has to find the relevant information out of those articles to be able to summarize it and hand it back to you. And the best way of doing that, and the best way we know how to do that is to giving extra signals to the AI, giving those structured relevant bits of information, front matter, back matter, publication date, keywords, abstract, that allows the AI to query the corpus and get the relevant chunks out of that corpus in a very quick manner. And then summarize what those chunks are. So the AI almost becomes the user interface over that corpus, because it’s going to summarize the data. But to find that data in the first place, structured content is key because for the same reason, structured content is key when you’re dealing with big indexes and web, same with AI. SO: So then structured content is potentially helpful. And I guess then circling back, let’s say I’m sitting on a pile of content of varying degrees of structured or unstructured, varying degrees of quality or lack thereof. What kinds of things should be happening before that content gets ingested into some sort of an LLM or some sort of a corpus to be used in AI-generated outputs? RD: So these are the same type of things you would do to make them easily retrievable ahead of time. So the standard approach that was being espoused about two years ago, a year and half ago, was something called Naive RAG. You can just take your PDFs and throw them at the AI, and the AI will ingest them into a vector database, and it will do semantic similarity and find the documents that you care about, not the best approach when you start talking about large amounts of documents. And there are issues with semantic similarities, where the AI will have a hard time distinguishing negative cases, will have a hard time peeling out the best documents, and that type of thing. So the best approach to take is you want to take those documents, you want to turn them into structured information in such a way that it’s easy for the AI to ingest. So typically that involves chunking it, into topic-level pieces or semantic chunking, coming up with summaries to make them easy for the AI to find, and whatever other information you may want to chase out of those. So, for example, if I’m handing a PDF to an AI and saying, I want to be able to search this PDF later, well, six months from now, if I get a new version of that PDF and I want to search it, search against the two of them, I really want my answers coming out of the second PDF. That’s metadata, that’s structured information that doesn’t appear in the text of the PDF or may not appear in the text of the PDF. You wanna be able to do things like versioning, you wanna be able to do things like dates, you wanna be able to give these signals to the AI to be able to pull that information back quickly. And that’s really where structured content comes in. So for the purposes of preparing your own corpus, you want to convert them into an easy to ingest format, which typically means Markdown or XML or something that the AI can deal with. You want to give it whatever other signals you can so that it’s easy to find. And then you want to hand it to something that first does chunking and then text embedding, which is basically turning the information into numbers so that you can do those cosine similarity searches. And then you want everything handed off to some kind of object store like a hybrid brand database or the hybrid factor database or graph database so that they’re easy to pull out. SO: Awesome. So you started this off talking about being the hopeful evangelist, and now having gone through all of this, it sounds as though you’re really thinking about these issues and dealing with them at scale. What are some of the top things that you’re thinking about going forward, whether hopeful or not, the good, the bad, and the ugly? RD: So one of the interesting aspects of my job is I get to do a lot of interactions with AI from an R &D perspective and do some in-house programming and do some in-house tool use. And what we’re finding is developing our own internal mechanisms for AI to call third-party tools, to be able to call Crossref or Grovid or some of these reference facilities out there through like model context protocol or through API calls so we can execute those calls and get that information back and do validation before it hands back the results is a very interesting topic for us because that would let us do things like any AI have it do the first few rounds of validation before it ever comes back to us without having it go to the next step, do a validation step and then the next step and then possibly do a round trip. It would be a much faster interaction. We use right now, of course, like most of the world, we’re using a lot of AI coding tools to tighten up our code bases to make sure things are working well, to basically act as a force multiplier when we’re doing development on projects, which is phenomenal. I can’t say enough good things about Cloud Code, you know, because it’s really become an essential tool in my day-to-day life. But I’m also seeing a lot of people out there using these tools to help analyze their own and improve their own workflow and that day-to-day work. We talked with one of our customers recently, and they use cloud code, even though the person giving the demo was not a developer; they use cloud code to answer the RFP. And Cloud Code does a tool use call against their document corpus, answers the RFP correctly, and what used to take two or three days of slogging through documents and finding things are now being done in an hour by one person instead of having multiple people working on this project. So it’s great to start seeing that type of stuff in the enterprise just blossom because it’s really exciting. SO: Well, Rich, I really appreciate your insights on this. I learned a few things and I think that it’s great to hear from people who are actually using this stuff, you know, in a production world, in a high stakes world where you’re actually, you know, need to get the content right, get the information right as opposed to just, you know, that we’ll play around with it and not worry about it too much. So thank you, and we’ll look forward to hearing more from you and what you’re doing at DCL. RD: Sounds great. Thanks for having me. Conclusion with ambient background music CC: Thank you for listening to Content Operations by Scriptorium. For more information, visit Scriptorium.com or check the show notes for relevant links. Want to learn more? Download our book, Content Transformation. The post Taming AI: Using AI for content conversion at scale appeared first on Scriptorium .

May 4, 202619 min

Machine experience (MX): Making content work for humans and machines

Your website may look great to humans, but can machines understand it? In this episode, Sarah O’Keefe (Scriptorium) and Tom Cranstoun (Digital Domain Technologies) explore the emerging discipline of machine experience (MX). Sarah and Tom discuss what AI agents actually encounter when they visit your web pages, why microdata and metadata are critical, and what content creators must do to ensure content is consumable for both human and machine audiences. Tom Cranstoun: Humans are looking for pictures, they’re looking for text, and they can infer. You may think, “Well, we’ve already added information on the page,” but by putting it in as microdata, it doesn’t appear on the page for the humans. It appears on the page for the machine. I think that that’s a critical distinction. We are trying to design for both. We don’t want to overload a human with information, but we do want to give the machine as much information as it can take. Related links: The Gathering Digital Domain Technologies MX books The Scriptorium Content Ops manifesto LinkedIn: Host: Sarah O’Keefe Guest: Tom Cranstoun Transcript: Disclaimer: This is a machine-generated transcript with edits. Introduction with ambient background music Christine Cuellar: From Scriptorium, this is Content Operations, a show that delivers industry-leading insights for global organizations. Bill Swallow: In the end, you have a unified experience so that people aren’t relearning how to engage with your content in every context you produce it. Sarah O’Keefe: Change is perceived as being risky; you have to convince me that making the change is less risky than not making the change. Alan Pringle: And at some point, you are going to have tools, technology, and processes that no longer support your needs, so if you think about that ahead of time, you’re going to be much better off. End of introduction Sarah O’Keefe: Hey, everyone. I’m Sarah O’Keefe. Today, our guest is Tom Cranstoun, who is founder of a machine experience, or MX community, called The Gathering. He has a couple of books on MX and is currently a consultant operating as Digital Domain Technologies. Tom, after 53 years in the business, some experience with AEM at very, very large companies, including a huge project at Nissan, has turned his attention to the question of how machines, which is to say AI agents, interoperate with the current public-facing web. And so today, Tom, I’m delighted to have you on to talk with you about machine experience, or MX, and what this all means as we move forward in this brave new AI world. So welcome. Tom Cranstoun: Thank you, Sarah. I’m very pleased to be with you today. SO: I am delighted to have you. So I guess we’ll start with the extreme basics here, which is what is machine experience, or MX? TC: Yeah. MX, well, to my definition, machine experience is like user experience, but it’s for machines. Machines cannot ask a friend for help if something goes wrong when they’re browsing a website. They can’t turn to a partner and say, “What do you think this means?” They can’t retry a failing form input because they will just go through the same mechanical patterns to try and carry on throughout the web journeys. Therefore, machine experience is thinking about what elements one must put on a webpage to help a machine understand and action the final goal of the webpage, whether that be a CTA that lets you purchase something, or an information document that lets you know about a government policy, or a charity good, whatever the author of the page is trying to get across to the audience. SO: And so at a high level, what does it look like to build out machine experience? What are some examples of things that you need to put onto a webpage to accommodate the machine that’s reading it? TC: Well, the very first level is the disabilities angle, things like the Americans with Disabilities Act, that kind of WCAG, W-C-A-G, the accessibility work. The more accessibility information is on the page, the more the machine can understand the background of the page. So machine experience and accessibility are pretty much at the top level, the same sort of thing. If you put in JSON-LD, microdata, and you enrich your pages with the things that Americans with Disabilities Act would like, you’re actually helping a machine understand the page. So that is the top-level constraint. When you go below that level, you need to give the machine lots of information about your product, not just the thing that a human wants when it’s glancing at the page now, and as you go through the journeys, things will be added on. Humans can only take in two or three items at a time, so we design pages to reveal what is happening. You go to a catalog, to a product, to a variation, to a purchase, four different steps. Each step introduces different pricing and concepts. It’s best to feed the machine on the page that the machine lands on with all of the information that it needs. This may not necessarily be surfaced to the human reading the page, but it’s there for the machine. This helps the machine when it arrives at your webpage. SO: So I’m really enjoying this concept that a properly organized page with proper accessibility WCAG or ADA compliance and support then results in the machine being better able to parse the page for essentially the same reason, right? It’s properly structured, it’s predictable. The things that are labeled are labeled correctly. I don’t know that we should be driving accessibility in order to enable AI, but on the other hand, if it gets us more accessible pages, then let’s certainly do that. Can you give some examples of what happens when pages are not machine-compatible? What are the kinds of problems that people run… Or not people. What are the kinds of problems that the AIs run into when they try to parse a page that has not been labeled properly or encoded properly? TC: Yeah, I collect these examples from real life. Whenever I use the web as a normal person, I say, “Well, how would a machine interpret this?” Recently, I was looking for a holiday, and I asked an LLM to give me a list of five companies that offer cruises up the Mekong Delta. The machine came back with one offer at $200,000 for a week’s holiday, and the rest of them were $2,000 for a week’s holiday. What had happened there was that the machine had found a European website. Now, the Europeans changed the comma and the dot in monetary labels differently from what the Anglo-Americans do. We use a comma separator between thousands and a full stop between fractions. The Europeans actually put the full stop as the thousand separator and a comma between the fractions. This meant that when the LLM built a table of prices for holidays, it didn’t understand the distinction, and it tripped up. The agent hadn’t been instructed to compare prices and make sure that they were all within the same range and were reasonable. It just produced them as a matter of a fact. “Here’s a holiday for you. One of them is $200,000. The rest of them are 2,000.” There was no knowledge, no information that could tell the agents what was happening. If those pages had been decorated with currency and they had microdata with the… microdata always says that you should use commas as a separator and full stops as the fractional separator. If these things had been in the page, the machine wouldn’t have flipped up. Now, a human could have read a page and seen the locale values shown on the page, and both people would be able to understand what was going on. So that’s a typical trip-up from an undecorated page. SO: And so essentially, the presentational component that says, because I’m serving this page to somebody in, for example, Germany, they are expecting a comma separator between the full Euro amount and the cents, the Euro cents. But that comma is essentially formatting, as opposed to data, and so here we are. TC: Yes, correct. And the microdata has got the thing in a proper machine-readable way. The other things that we always get problems with in the world are English and American date formats. We swap the month and year around when doing short form. The machine-readable version uses ISO dates, and ISO dates put in as a microdata tells the machine categorically. It doesn’t matter what the locale is, this is the date and time. SO: Yeah. And so as the expression of the date, whether April 1st is 1-4 or 4-1 is essentially a formatting problem. TC: Correct. And these are not visibility problems. These are machine experience problems. So it’s layering up. You start with fixing the disability by doing machine experience, and then you fix the locality and the community values, the human factors, display factors. SO: And so I think we’re all familiar with the concept of a customer journey, but you’re now talking about a machine or an MX journey. What does that look like? I mean, how is the machine processing of a website? How do you explore that journey and what it looks like? TC: The machines will not discover your website, come in through your landing page, and then look for offers or products. A machine will have an idea of where it wants to go and will land straight in at a page. It will arrive five pages into your journey, and read the webpage as it is. The owner of the website has lost all of the signals about what the dwell time was on each page, how’s the reader arrived at the end location. Did they go sideways and look at other things? Those things don’t happen with machines. They go straight in, see if they can get what they can. If they can get what they can, they will action it. If they can’t, they will move on, and go to another page or another person’s website and do exactly the same to them. So when a machine arrives at your webpage, it will not be giving you any referral details. It will not tell you what the journey it is, and it won’t tell you what else it’s interested in. You’ll just get a cold caller who will arrive and disappear. I call them invisible users. They’re invisible to your analytics, they’re invisible to your tracking, and they’re invisible to your future. You cannot tickle them and say, “Hey, you left something in the basket.” You cannot use those parts of the journey. A machine comes in and goes, gets what it wants or it doesn’t. So you must give it, front load it as much information as possible on any and every page that a machine may land on. SO: So then coming at this from the perspective of structured content people, because a lot of what you’re talking about, I mean, is web experience, like how does what we view as the end state result of the content that we’re creating. So if I have an enormous DITA CCMS full of stuff and then I output it to some semblance of a website, your focus is on what needs to be on that website so that it is describing itself in such a way that the machine, that an AI or a crawler can go in there and pick up what it needs to and process it accurately and not offer you a vacation for $200,000. I assume you did not pick that one. So what are the opportunities? When you look at MX and then also DITA as a backend, what kinds of opportunities do you see there to map those things across and take advantage of some of the structure that perhaps is already in the XML and/or structured content systems? TC: Yeah, I see the backend is full of good content operation stuff. Everybody has got details about pricing and dates and frequency, and there’s lots of backend information, which often doesn’t make it into the front end for people. Humans are looking for pictures, and they’re looking for text, and they can infer. They can infer if two prices are on a page and it says, “Was $200, Now $180.” A human understands that. A machine, well, depends on the quality of the machine, whether it can read and infer those things. So the backend information has to be made more visible and in a redundant manner. You may think, well, we’ve done this on the page before. We’re doing this on the page after. But by putting it in as microdata, it doesn’t appear on the page for the humans, but it appears on the page for the machine. And I think that that’s a critical distinction. We are trying to design for both. We don’t want to overload a human with information, but we do want to give the machine as much information as it can take. We don’t necessarily have to surface all of the information within the page, but we have to carry it with the page. So a page taken in isolation contains the entire story, not just the fraction that a human is looking at, which does mean that a lot more pushing off the backend from data to the front end. And some people will think that’s a waste of time, but I don’t think so. I think giving that extra material to the machine is what makes the journey successful for the machine. SO: I’ve been to a lot of conferences in the past couple of weeks, and the conversation around what is needed for successful LLM processing, or crawling, or ingestion, or agents for that matter and what is already provided in a metadata-rich structured content system is sort of, well, we have all of this. Now, what do we do with it, and where do we put it, and how do we make sure that this all works? So it seems like this discussion around machine experience is going to help to maybe close that gap and connect the pieces such that we can do this successfully. And so as we move into this, I know that you have some material out there, but also a community. Can you talk a little bit about the MX community, and what you’re looking for there, and what it’s called? We will put all of the links in the show notes. But what does it look like to participate in that community, and what sort of participants are you looking for? TC: Yeah, we are looking for content creators. We are looking for business owners. We are looking for technical writers. It’s called The Gathering, gathering being a Scottish term for the gathering of the clans. We all get together to do something that’s good for the combined grouping. And then after we’ve created whatever we’re going to create, we go away and do our own things. Now The Gathering is tg.community. That’s https//tg.community. We are building a set of community-led standards to try and make it easier for machines to understand documents. The Gathering is not just interested in HTML. We’re talking about documents of all types, and we’re talking about keeping the metadata that you have in the backend of the content creation systems, whether that be data or other content creation systems, and passing it through into the end documents. You have metadata in PowerPoint slides. You have metadata in Word documents. You have metadata in JPEGs. These, too, deserve the machine experience. If you can tell the machine details about an image inside a JPEG, then the machine doesn’t have to try and scan and interpret the image to find out what it is. It makes things so much better. And The Gathering is a community that is trying to build these as open community-led standards. One of the first things that I am proposing for the community, which was just launched on the 2nd of April, 2026, by the way, it’s very young, and we hope to build at the speed of LLMs. We need to work fast. The key point and the key thing that helps LLMs understand your website, there’s a thing called llms.txt, which people don’t really understand and machines don’t really use. It’s a standard for describing your website in a way that a machine can help to understand, know what’s going on without reading your site map. It is not used by the machines because, one, it’s not served as HTML, and, two, it’s not in your site map. Therefore, the crawlers that build your training material do not pick it up and do not ingest it. I am suggesting, and I have it in my books, I talk about this, if you wrap the llms.txt in HTML and serve it as HTML and put it in your site map, then you will get a better response from the training stage and from the inference stage. So you are seeding the machines with the information about your website, something that is currently missing from the world, and that’s step one. There are five steps that you’ve got to go through before you can do a successful e-commerce position. And that is feed the machine, get noticed, be descriptive, be MX-aware and be citable, and MX lets all of those things happen. SO: Perfect. Well, Tom, I know that there’s a lot to discuss here, and we could go on for a very long time, but I hope this gives people a little bit of an introduction to this idea and an opportunity, if they’re interested to reach out to you and to the community that you have. And there’s also a book or three. Any closing thoughts that you want to pass on before we close this out? TC: My personal opinion is that I think that we should treat the machines as first-class citizens and not block them from our content and to create content that works for them. The more that we do for them, the more they will do for us. And if we start treating them as an afterthought, it’s not going to be such a good web as we could build. SO: Okay. Well, thank you so much. I’m glad we had an opportunity to talk. And we will, again, put the links to the various resources that Tom mentioned, including the community. There’s some RFC, some standards drafts and a manifesto and a book. We will put all of that in the show notes. So Tom, thank you again for being here, and I look forward to hearing more on this effort. TC: Thank you very much, Sarah. Conclusion with ambient background music CC: Thank you for listening to Content Operations by Scriptorium. For more information, visit Scriptorium.com or check the show notes for relevant links. Want to learn more? Download our book, Content Transformation. The post Machine experience (MX): Making content work for humans and machines appeared first on Scriptorium .

Is this your show?

Claim this listing to keep it up to date, reach guests who want to pitch you, and manage bookings with Guestify.

Claim this listing

More Business podcasts

The Bottom Line

Alexander Spencer