Tag: software development

Making Audio Stories Interactive with AI: The Story of Schmaudio

I have a somewhat complicated relationship with AI. As a software engineer, I find the technology fascinating. I use it regularly, and it has become a genuinely useful tool in my work. At the same time, I am cautious about some of the enthusiasm surrounding it. I don’t think we should automate everything simply because we can, and I am skeptical of the idea that handing a task to an AI and accepting whatever comes back necessarily represents progress. There are also still many open questions around how this technology will affect creative work, software development, and society more broadly.

But there is another side to AI that I find much more exciting. Sometimes a new technology doesn’t merely make an existing process faster or cheaper. It makes something practical that previously wasn’t. For me, interactive audio stories are an example of that. And somewhat unexpectedly, they connect modern AI with an idea that fascinated me when I was a child.

Choosing What Happens Next

As a child, I was fascinated by pen-and-paper role-playing games. I also loved interactive adventure books. Instead of simply reading from the first page to the last, you would eventually have to make a decision. Perhaps you could enter a mysterious cave or continue along the path through the forest. If you entered the cave, the book might tell you to continue on page 74. If you chose the forest, you might jump to page 112. The story changed depending on what you decided.

I loved the idea that I wasn’t simply waiting to find out what the character would do next. I had to make that decision myself. Of course, those books had an unavoidable limitation: every possible path had to be written beforehand. And branching grows very quickly. If every episode of a story leads to 3 completely different episodes, those 3 lead to 9, then 27, then 81. Continue for long enough and the number becomes absurd. A human author cannot write millions of episodes for a single story.

Traditional interactive stories therefore have to work within that constraint. Branches can eventually reconnect, some decisions can have relatively small consequences, and the overall number of paths has to remain manageable. That doesn’t prevent great interactive stories from being written, but it puts a practical limit on just how far the idea can go.

Generative AI changes that constraint.

From My Childhood to My Children

Many years later, audio stories became part of my own children’s lives. They listen to them regularly, and at some point this brought me back to the interactive stories I had enjoyed as a child. I started wondering what would happen if you combined the two ideas. What if an audio story stopped at an important moment and asked the listener what should happen next? And what if that decision didn’t simply cause a small variation before returning to the same story, but could actually send the adventure in a different direction?

Modern generative AI suddenly made that idea practical, so I started experimenting. There wasn’t a business plan behind it. I simply wanted to create interactive audio adventures for my children and see whether the idea worked. A story would play for several minutes. Eventually, the characters would reach a situation requiring a decision. The listener would choose what they should do, and the next episode would continue from that decision. Then another choice would appear, and another.

The more I experimented with it, the more I liked the idea. Eventually, that experiment became Schmaudio.

AI Isn’t the Interesting Part

It would be easy to describe Schmaudio as a platform for AI-generated audio stories. Technically, that’s true, but I don’t think that’s the interesting part. Ideally, when someone listens to a Schmaudio story, they shouldn’t be thinking about AI at all. They should be thinking about the story.

The interesting part is what AI makes possible. A listener can make a decision, and that decision becomes part of the history of their adventure. Later episodes can build on what happened before. Characters can have experienced different things, relationships can develop differently, and one path might reveal places or characters that another listener never encounters. Over time, two people who started with exactly the same first episode can experience increasingly different stories.

And when someone reaches the end, there is another interesting possibility:

What would have happened if I had chosen differently?

They can go back and find out.

A Story That Grows as You Explore It

There is still an obvious practical problem. Even if AI makes it possible to generate all those episodes, it wouldn’t make much sense to generate an enormous story tree before anyone listens to it. Most of those episodes might never be heard.

Schmaudio therefore generates episodes lazily. The first episode of a story already exists. When someone listens to an episode for the first time, the possible episodes following it are generated. If there are three choices, three new paths into the story appear. But Schmaudio doesn’t yet need to know what happens ten decisions later. Those parts of the story can be created when someone actually gets there.

I like this aspect of the system a lot. The complete story doesn’t have to exist before somebody begins listening to it. Instead, the world grows as people explore it. In theory, this allows stories to become much deeper than what Schmaudio currently offers.

More Branches Don’t Automatically Make Better Stories

There is an important caveat. Being able to create an enormous number of branches doesn’t mean that doing so automatically produces a better experience. The number of choices isn’t what makes an interactive story interesting. The choices themselves do.

A good choice should make you hesitate. Do you trust someone who may be hiding something? Do you follow the strange light disappearing between the trees or help somebody who needs you? Do you investigate the mystery or take the safer route home? There shouldn’t always be an obvious correct answer. Different decisions should appeal to different personalities, priorities, or curiosities, and most importantly, the decision should matter to what happens afterward.

AI makes deep branching possible. Good storytelling determines whether that branching is meaningful.

How I Use AI

This project also reflects how I currently prefer to use AI in my work. I use it as an assistant rather than expecting it to do the entire job for me. For Schmaudio, I currently use ChatGPT to generate the stories and individual episodes, and ElevenLabs to turn those scripts into narrated audio. I developed the web application myself, but AI assisted me there as well. I use ChatGPT to discuss architecture, write and review code, investigate bugs, explore alternatives, and question decisions.

The creative process works similarly. Generating a Schmaudio episode isn’t simply a matter of asking an AI to “write the next part.” There is a framework around it. Stories have definitions that establish their worlds, characters, themes, atmosphere, and creative direction. Episodes have to respect everything that happened earlier on the current path. Characters should remember previous events. Choices should emerge naturally from the story and have meaningful consequences. The writing has to work as narration rather than simply as text on a page.

I define and continuously refine that system, while AI works within it. In that sense, I don’t see AI as replacing my work on Schmaudio. It allows me to work at a scale that would otherwise be impossible for one person. And, appropriately enough, I also used AI as an assistant while writing this blog post.

It Doesn’t Always Work

None of this means that I think today’s generative AI is particularly close to perfect. It isn’t. Generating plausible text is remarkably easy now, but generating a consistently good story is much harder. Maintaining a coherent story across many episodes and many decisions is harder still.

Characters need to remain consistent. Previous events need to matter. An episode needs to move the story forward rather than merely fill another ten minutes. Choices have to feel natural rather than being added simply because the product requires a decision. And then there is the audio. Text-to-speech technology has improved enormously, but it still has limitations. Pronunciation isn’t always correct, maintaining a consistent voice and delivery across many episodes can be difficult, and producing stories in several languages adds another layer of complexity.

These are all areas where I think Schmaudio can become considerably better. I want to improve the quality of the scripts. I want decisions made much earlier in an adventure to influence later events more naturally. I want better long-term narrative coherence, and I want the voices and audio quality to continue improving.

The interesting thing is that the idea already works. I can listen to an interactive story, make a decision, hear the adventure continue from that decision, and repeat the process. And this is probably the worst the underlying technology will ever be. I find that exciting.

Where Does This Go?

It’s difficult to predict what generative AI and text-to-speech will look like several years from now. Even looking only a year ahead, I expect the tools available for building something like Schmaudio to improve considerably. But improvements to the underlying models are only part of it. I am also learning more about interactive storytelling itself.

How much should a previous choice affect later episodes? When should branches diverge dramatically, and when is a subtler consequence actually more satisfying? How do you maintain a coherent narrative when the listener can repeatedly push it in different directions? How long should a story be? How much should listeners be encouraged to replay it? Those are storytelling and product questions rather than AI questions, and I suspect answering them will ultimately matter more than simply having access to a better model.

Schmaudio today therefore doesn’t explore the full potential of the idea. Stories could go considerably deeper. Choices could have consequences much further into the future. Different paths through the same story could diverge more dramatically while still remaining part of a coherent whole. There is a lot left to experiment with.

The Same Idea, Without the Pages

Despite all the technology involved, the part I find most interesting remains remarkably simple. A child is listening to a story. The characters reach a crossroads. Something has to happen next. But this time, the listener isn’t waiting to discover what the author decided. They get to decide, and the story continues from there.

It’s fundamentally the same idea that fascinated me when I was jumping between numbered pages in adventure books as a child. Except now the number of pages no longer has to be decided in advance.

And I think we’re only beginning to discover what that makes possible.

Explore Schmaudio

If you’d like to try the idea for yourself, you can find Schmaudio at schmaudio.net. Stories are available in multiple languages, and you can make your own choices as you listen.

Here are three stories you can currently explore for free:

The Alibi

A mystery for adult listeners in which a seemingly straightforward story may not be quite what it appears.

Listen to The Alibi

The Map Awakens

An interactive adventure that invites younger listeners into a world of exploration, discovery, and meaningful choices.

Listen to The Map Awakens

Who Comes Tonight?

An interactive story for younger children, offering a gentler introduction to choosing where an audio adventure goes next.

Listen to Who Comes Tonight?

A Reading List about Code Health

In this post I would like to give an overview about the books related to code health that had the most influence on me as a software engineer. They are also listed in my post A Reading List for Software Engineers (which is still in an unfinished state), but are covered here in more detail.

The books are not presented in any particular order, except for the first entry (which is actually a series of books), which had a profound effect on me, in the sense that it made the whole field of code health and sustainable software development accessible to me.

But before we dive into the books, let me say a few words about why I believe that code health is such an important topic and why it is essential for sustainable development at a high pace and also a determining factor for the long-term success of projects.

Why Code Health?

Code health is very difficult to define and even more difficult to measure. We have some indicators that can show the presence of potential issues, but to my knowledge there is no indicator that proves that a codebase is healthy. In other words, it is easier to spot issues than to spot quality. In my opinion, this is due to the fact that code health is to a large extent defined by higher-level structures of the system and abstractions and not so much by local low-level properties. This leaves us in a complicated situation and it makes it quite difficult to approach it from a scientific point of view.

Nevertheless, we have gathered plenty of anecdotal evidence over the last decades. And I have collected my own experiences during my professional career. In a high-quality codebase I am able to deliver plenty of value per time, while maintaining the high quality. In contrast, in a big ball of mud, I feel I am not even worth the money I am getting paid. And the difference in productivity I am talking about here is not a factor of 2 or 3, it is orders of magnitude. And yes, the plural orders is not a typo. We are talking about factors above 100 here.

So why do we so often work in low-quality codebases? In particular, this is puzzling, since fresh projects have by default a high quality. I have written about this in another post (Code Health is not a Trade-off) and do not want to go too much into detail here, but I think it is a combination of multiple factors, such as poor understanding of the requirements and the problems, inexperienced developers, time pressure and companies setting the wrong incentives.

My personal solution to all that is to apply the knowledge I extracted from the books listed below (and many other books which are not directly related to code health, but still had a great impact) and to shape those parts of the codebases which I have to work with, in such a way, to create an environment that allows me (and other developers) to perform at the highest level of productivity. The problem here is that this is not always easy and too often there are external factors involved which further complicate this. But starting with the books listed in this post is a first step in making this happen.

The Clean Code Series

This series consists of four books. These are (in chronological order) Clean Code: A Handbook of Agile Software Craftsmanship, The Clean Coder: A Code of Conduct for Professional Programmers, Clean Architecture: A Craftsman’s Guide to Software Structure and Design and Clean Agile: Back to Basics. The first and the third book of this series are all about code health, while the second is more about software craftsmanship and being a professional software engineer and the fourth gives us a sort of historical perspective on the Agile movement. I list them all here for completeness, the second book is also relevant with respect to code health, the fourth one maybe less so. But overall, I think they are all worth a read.

In any case, right after I made the transition from academia into the world of professional software development, I started reading Clean Code, Clean Coder and Clean Architecture (if I recall correctly, in that particular order). And those books had a tremendous impact on me and my view about writing software. Reading those books made me realize that the software I wrote during my time in research was pretty low quality. I was always very confident about the code I produced, but as a scientist I did not know how much better I could have done. Looking back, it makes me a bit afraid. I am still sure that my results were correct and there is a lot of empirical evidence that backs this up, but how could I be so unprofessional? Well, my main focus was on the scientific results, but writing software was one of the tools I used to obtain those results. And I used that tool quite badly.

Coming back to the books. Clean Code: A Handbook of Agile Software Craftsmanship focuses on what I would call now low-level readability. This is about smaller chunks of code being expressive and readable. In some sense, this is the foundation on which we can build larger components and whole systems, all adhering to high standards of quality. And this is exactly the topic of Clean Architecture: A Craftsman’s Guide to Software Structure and Design. Here the focus is on higher-level structures and abstractions. The principles that are explained in this book also apply to the design and architecture of whole systems or services, but the focus is more on components and relations between components. Exactly those structures, which to a large extent determine the overall code health. This should not diminish the value of low-level readability, but while I personally value low-level readability, I value clean structures and abstractions even more.

The Clean Coder: A Code of Conduct for Professional Programmers complements the other two books by providing guidelines for professional behavior as a software engineer. Definitely not with the same strong focus on code health as the other two books, but laying out why a professional conduct is important for doing our job efficiently (and also for being able to push for code health and other things we as developers know are important). For Clean Agile: Back to Basics the focus is even less on code health. Actually, for those who understand the Agile movement as a strong force for promoting code health and clean software development practices, there is quite some connection to code health. In any case, I would recommend this book for the historical perspective that it provides on the Agile movement and the more recent developments.

All in all, Clean Code, Clean Architecture and Clean Coder are all a must-read and I would recommend to read them in exactly that order. Clean Agile is a nice addition, but it would not be wrong to put it into the personal book backlog for the moment. Here are a few links to the books. Please note that there is a special offer for the electronic versions of Clean Code and Clean Coder, bundled together.

Expanding Your Toolkit

The books in this section are Design Patterns: Elements of Reusable Object-Oriented Software, Test Driven Development: By Example, Working Effectively with Legacy Code and Refactoring: Improving the Design of Existing Code. Let me start by saying that they are all a must-read. They all enrich your personal toolkit as a professional software engineer by providing important insights and generic techniques. I am pretty sure I am using some of the knowledge I got from these books on a day by day basis.

Let’s start with Design Patterns: Elements of Reusable Object-Oriented Software. During my time as a University student, a friend introduced me to this book and I read it front to back in only a few days. I did not fully understand all the aspects, but I found it fascinating to think in patterns while developing software. Unfortunately, I was not able to grasp the full value of this book at that time. This happened years later, though, after I made the transition from academia into software engineering and when I read the book for a second time. I don’t know what changed, but it was much easier to comprehend and suddenly all the patterns were easily accessible. Maybe the few experience I had with writing production software played a role here. In any case, this book contains a list of generic design patterns and presents them in a very structured way. These patterns make it easier to write, communicate and reason about software. They are mostly about lower-level design, just where software engineering gets interesting and where decisions have a lot of impact on code health.

The next one is Test Driven Development: By Example. This book is all about writing tests first in order to drive the implementation of production code. It had a profound effect on me. For a short period of around two months, I was practicing test driven development (TDD). It felt slow and awkward at the beginning, but I got used to it quite fast. The insights I gathered from that period of time are about what makes code and design testable and how this can be achieved without sacrificing any other code or design properties. Think about it for a moment, this is huge: making the code testable without being invasive. I stopped with strict TDD after those two months and currently I am practicing something that I would call pseudo-TDD (I guess I have to write about that in the future). By the way, this is also something quite typical that happens when I read books: I try to absorb the knowledge that is within the books and use it to improve / complement / adapt my personal practices and views as a software engineer, while I usually do not fully buy in to the more dogmatic parts. That does not mean that there is anything wrong with strict TDD, quite the contrary, I can only recommend everyone to read this book and to practice TDD (at least for a while). For me personally, I tried it for a bit and that was sufficient for me to extract the underlying principles and incorporate those into my personal practices. So, saying that my current style is influenced by TDD would be definitely an understatement.

The last two books in this section, Working Effectively with Legacy Code and Refactoring: Improving the Design of Existing Code are sort of related to each other. Both books focus on non-functional changes, so-called refactorings, but within different contexts. The first one focuses on how we can regain control over legacy code. This book contains a large number of specific techniques that can be used to incrementally transform legacy code into a more healthy state. If you ever became desperate while trying to add tests to existing code, while breaking up a large method or while separating responsibilities of a single huge class, this is the right book for you. The second book approaches refactorings from a different angle. In this book the term code smell was coined. This term depicts an unusual structure in the code that could potentially cause issues in the long-term. Many such code smells are presented in the book, together with the techniques that can be used to remove them. In my opinion, those two books nicely complement each other and both should be part of any library of a professional software engineer.

Again, here some links for the books discussed in this section.

More Perspectives

In this section I would like to introduce a few books which I read after the books that I discussed earlier. Due to the knowledge I already had while reading those books, it is difficult for me to judge how much value they would provide by themselves in isolation. All I can say is that they were very good reads, refreshed my memory on many different aspects of code health, provided different perspectives on certain topics and offered new insights which made me adapt and improve my personal practices even more. For the latter it is not clear how much of the insights are actually completely new and how many I missed while reading other books. This is also why I tend to re-read books years later while having a more profound background knowledge and a completely different context. In any case, the following books were definitely worth my time: The Art of Readable Code: Simple and Practical Techniques for Writing Better Code, A Philosophy of Software Design and Understanding Software: Max Kanat-Alexander on simplicity, coding, and how to suck less as a programmer.

I would recommend all of these to anyone who wants to dig deeper into the topic of code health and who has already consumed most of the other books mentioned earlier. As usual, here the links to the books.

Epilogue

I hope this list provides some value for fellow software engineers, in particular to those who are at the very beginning of their careers. If this is the case, please feel free to share it with your peers. And please let me know in the comments what you think about this list and if there is anything missing here.

Scrum is fragile, not Agile

As the title suggests, this post is about two different aspects of Scrum. The first part deals with Scrum not being Agile and the second part is about Scrum being fragile.

Before going more into detail, a short disclaimer: Everything I present in this post (and in this blog in general) is my personal view and does not represent the view of my current employer, my former employers and any future employers.

Scrum is not Agile

I guess a typical reaction to this heading would go like “How is this possible? Scrum is not Agile? Isn’t Scrum the number one Agile software development process?”. The short answer is that Scrum claims to be an Agile process, but the sad reality is that Scrum is quite far from being Agile. I will show you why.

Lets have a quick look at the Agile Manifesto. It states that it values “Individuals and interactions over processes and tools”. Lets also have a quick look at the meaning of the word agile. According to the Oxford Dictionary agile means “Able to move quickly and easily”. It is not a coincidence that the term agile has been chosen to represent the high-level ideas within the Agile Manifesto. In fact, one major point behind Agile is that in many software projects it is extremely difficult to move quickly and easily. This is not the case for a completely new project, but over time many projects get into a situation where sustainable development is simply not possible anymore. To prevent this (and other issues), the Agile Manifesto and the Principles behind the Agile Manifesto provide several high-level guidelines. These guidelines are not specific well-defined processes or tools and they allow for many different implementations. I suspect that both of these properties (high-level and allowing different implementations) were fully intended. The overall goal was not to present a silver bullet, but to help peers to avoid many of the pitfalls in software development, which the authors of the Agile Manifesto experienced first-hand and which fall into exactly these categories.

Now lets have a look at the Scrum Guide (written by two of the authors of the Agile Manifesto). In contrast to the Agile Manifesto and the Agile Principles, this guide seems quite lengthy. Surprisingly, the whole guide does not mention Agile a single time. I am not sure if this was historically always the case, but if the authors of the Scrum Guide do not claim that Scrum is Agile, then we would already be done with the first part of this blog post. I assume that this is not the case, so lets move on. The Scrum Guide is about a framework which contains “roles, events, artifacts, and the rules that bind them together”. In other words, it is a very specific and well-defined process. This does not sound agile and it also does not sound Agile (remember: “Individuals and interactions over processes and tools”). This is quite ironic and obvious. And this is where the whole Scrum movement should have stopped. But it did not and instead frustrates an increasing number of software developers all around the world. And whenever a Scrum project fails, it is not because of Scrum’s potential flaws, but because Scrum was not implemented correctly. That sounds like a nice transition into the second part of this post.

Scrum is fragile

This part is very short. I thought that the wordplay (Scrum being agile / fragile) is kind of funny and apart from that it perfectly describes one of the things that really bother me about Scrum: Whenever a Scrum project fails, it is because Scrum was not implemented correctly. And you can read about a vast amount of such projects. What does it mean, if a large number of intelligent software developers are not able to implement Scrum correctly? It means the whole framework is fragile. And this is another major argument against using Scrum. What is a framework good for, if it is so difficult to use?

Well, it seems that with the help of expensive consulting and coaching, as well as training and certificates, Scrum might in fact provide value. But it is not clear if this is value for the companies developing software and the hard-working software developers or for those who offer services in and around the Scrum ecosystem.

Personal View

I would like to finish this post with a bit of my personal view regarding software development, Agile and Scrum. To me it seems that one very important part of high quality software development is to maintain a simple priority queue of tasks. The weight is a combination of the value a task provides for the customer / developers and the estimated effort to implement this task. For some developers this comes naturally. For teams and companies for which this is not the case, Scrum offers a rather expensive and inefficient implementation of a priority queue.

And lets be honest. Software development is a very difficult and complex work. Are we really surprised that so many projects fail? The field is still very young and we need to learn a lot. And this is crucial: We need to learn from past experiences, let it be failures or success stories. And here we collectively fail. We are not using the wrong processes or implementing the right processes in the wrong way. We are simply caught in a rat race and not able to make a short break in order to look at and learn from all the things that happened around us, maybe even before our time. It is our duty to extract the knowledge, the experiences and the wisdom from the many resources that are so easily available to us: The many many books, articles and videos about software development and, last but not least, the Agile Manifesto.