Is it possible to create a simple point-and-click adventure that changes its look and story each time it’s played, thanks to generative AI? During Prototyp Week 2024, one team set out to explore this very question.
Press to view video
From the role-playing craze of the 1980s, purely text-based adventure games became popular as home computers spread. By the 1990s, these had evolved into graphical point-and-click games, with Sierra as the leading name in the market.
The Prototyp team wanted to explore whether large language models (LLMs) and image-generation technology could be used to create unique adventures in this genre. The goal was to develop fully generated experiences that would differ each time a player hit “play,” with as little human involvement as possible in the story and puzzles.
With a large team of seven people, it became clear that the work needed to be split up efficiently. Half of the team focused on script generation, while the other half worked on turning that script into an actual game.
They started with the idea of multiple scenes that a player would progress through to complete the game. Based on this concept, the team chose a simple JSON (JavaScript Object Notation) structure, consisting of a list of scenes where each scene contained different objects and tasks.
Script generation primarily involved refining prompt engineering for various AI models. The aim was to create game data in the correct format from a single sentence of input, such as: “An adventure game with a pirate crew searching for treasure.”
Using that prompt, the team could generate the game’s structure and the puzzles the player would need to solve.
To create graphics, the team built a pipeline that took a game script − including its scenes and objects − as input. Leveraging image-generation and image-segmentation models, they produced images with the correct objects placed in the right positions. These were then stored in the game’s data structure.
For an added layer of immersion, a final step was introduced in the generation pipeline: creating audio for descriptions and text, giving the game its own narrator.
Ultimately, they had a functioning web application where everything from storytelling and content, to graphics and audio − was fully generated by AI.
The biggest challenge was getting the AI models to produce coherent stories and puzzle solutions. Multiple approaches were tested, and the team realized that working with large, complex prompts was tough. As a result, they structured the process more like a pipeline with multiple steps, each step generating a specific portion of the game’s content.
While this solved the issue of producing output in the right format, it was difficult to make sure that all parts fit together into a seamless gaming experience.
In just a week, the team succeeded in realizing their initial idea. The game works and almost always leaves players with a smile after completing the brief experience.
The most important takeaway was that, although generative AI enables rapid creation, the results aren’t always predictable and can be incoherent or quirky. While that adds charm to a project like this, it can be a serious drawback in many other applications where predictability is crucial.
The game could be enhanced by allowing non-linear movement between scenes, resulting in richer adventures and more engaging puzzles. There’s also huge potential to refine the generation process to produce more compelling narratives and an overall improved gaming experience.
Play the game in it's current state: https://lla.prototyp.se/
We use cookies to give you a better experience when visiting our website. Read more about how we handle cookies