Scientists from Google have developed an AI-based engine and used it to run Doom

Specialists from Google's research division, Google Research, have demonstrated the GameNGen game engine, which is based on artificial intelligence. In a video, the authors showed how it recreates the gameplay of the original Doom.

Everything works in real-time, but due to the raw nature of the technology, the frame rate is currently around 20 FPS. Neural networks, including Stable Diffusion, have to create a sequence of frames depending on the player's actions. This is similar to real-time video generation.

To ensure everything works correctly, the AI was first made to play Doom, and then the neural network was given the ability to create accurate frames based on the obtained data. Thus, the artificial intelligence does not generate the game in the usual sense but dynamically recreates the levels around the player as they explore them.

At the moment, the technology has several shortcomings that they plan to fix in the future. For example, GameNGen has extremely limited memory. Additionally, the AI agent (which helps the neural network learn to play the game for future gameplay reproduction) is very limited in its variety of behaviors in the game and in imitating the actions of a real player.

The project involves scientists Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter.