“AI” systems have a way to go before they’re ready to run games of Dungeons and Dragons, according to a thesis paper by graduate researcher Pavlos Sakellaridis which recently surfaced on the internet. The results of his tests are interestingly mixed, with players reporting satisfaction scores for the bot DM that came close to – and sometimes beating – those for a human. But there were notable problems with the droid railroading players, failing to keep the adventure to time, and not making the adventure challenging enough.
Sakellaridis used questionnaires to compare satisfaction scores between a group of players who played a one shot DnD session with the robo DM, and who those who played the same scenario with a human dungeon master.
He found statistically significant differences in player satisfaction for several areas. Players reported that the human DM was more competent, created better flow and narrative progression, and elicited more positive feelings overall. But players thought the AI DM was better at creating immersive environments.
He also notes three specific issues that players reported with the robo-DM. First, while it “could generate coherent narratives”, “it sometimes struggled to accommodate the diverse and unpredictable choices that players made” – so it had a tendency to railroad players.
Second, although the bot “had access to the necessary adventure details, it did not relay this information clearly or timely to the players, resulting in confusion and incomplete story arcs” in some cases.
Third, “participants noted that the gameplay did not present sufficient threats or challenges”.
Sakellaridis’ paper, published in July 2024, goes into detail about how the bot worked and how he gathered responses. The test session that players were reporting on wasn’t a typical DnD game, as it featured a single player and the robo DM, which naturally had to interact via a text interface.
It also generated a random DnD character sheet for each participant, assigning them to one of the DnD classes and DnD races and giving them a short backstory.
The bot was built on ChatGPT, with added DnD specific training data from an open-source compilation of fan-created transcripts of Critical Role’s DnD sessions. The bot also had access to the rules of Dungeons and Dragons, and the fifth edition starter adventure ‘The Sunless Citadel’.
Sakellaridis also provided a huge prompt for the bot to obey, putting further constraints and patterns on how it would respond in interactions with players.
Artificial Intelligence is a bit of a misnomer for Chat GPT. It’s a Large Language Model, a high powered pattern-recognition system like autocorrect on steroids. That doesn’t mean it’s not impressive or potentially useful. But the fact that LLMs don’t understand anything is important to remember when interpreting their performance, and speculating about their potential.
LLMs answer questions and continues conversations in a fundamentally different way to humans. Humans (hopefully) understand the statement, identify the salient facts we need to interpret, do so, decide on any other information we wish to convey, and then provide an answer: an LLM determines which pattern(s) it’s being called on to produce, and spits out a string of text that falls into the pattern.
So the robo DM’s ability to create an immersive environment, in a text-based exchange, doesn’t mean that it’s imaginative. LLMs have been specifically designed to digest large volumes of copyrighted material down to patterns, and then generate new text that fits into those patterns when prompted. They can do that faster than a regular DM can type. It would be very interesting to compare the transcripts from the AI DM’s sessions with the Critical Role transcripts it was trained on, to see if there are any similarities.
The question of whether an LLM based system could overcome the problems found in Sakellaridis research remains opaque. It seemingly performed the basic role of dungeon master pretty well, arbitrating the game and moving the players through the beats of the story.
Could a more advanced version of ChatGPT, or a better designed prompt, plug the gaps – or is it impossible for a system that doesn’t actually understand anything to appropriately handle player improvisation, timekeeping, and threat management?
We won’t jump to the conclusion that this could never outperform a human dungeon master, because ‘outperform’ is a nebulous target. Baldur’s Gate 3 has perfect mechanical consistency, incredible scripts and responsive story arcs that are the results of thousands of hours of human labor, but it is utterly incapable of straying beyond what is coded into the game. Is that better, or worse, than a human DM? Yes and yes.
Last year, Hasbro CEO Chris Cooks speculated about uses for AI software introducing new players to DnD, or as support tools for DMs or players. If LLMs can perform well in new capacities they will be tempting for businesses looking for new products to offer, or ways to downsize their workforce.
But the fact that they do not and cannot know anything about what they’re doing and what that might mean in the real world seems like an inalienable risk. Anyone who has accidentally sent a badly auto-corrected text message knows exactly why.
Whether LLMs are an efficient or effective way to do things, compared to conventional software, or humans, is another question entirely.
If you’ve recently picked up the DM’s guide and are waiting on the Monster Manual to hit the DnD release schedule, we don’t think you should worry about the machines making you redundant just yet. But as several RPG industry insiders have said, AI is a Pandora’s box that isn’t going to close.
Source: Wargamer
