🧀 BigCheese.ai

Social

Evaluating the world model implicit in a generative model

🧀

The paper 'Evaluating the World Model Implicit in a Generative Model' authored by Keyon Vafa, Justin Y. Chen, Jon Kleinberg, Sendhil Mullainathan, and Ashesh Rambachan, proposes novel evaluation metrics inspired by the Myhill-Nerode theorem. It focuses on assessing world models learned by generative models in domains like game-playing, logic puzzles, and navigation, highlighting potential fragility when models encounter related but different tasks.

  • Released on 6 Jun 2024.
  • Latest revision on 22 Jun 2024.
  • Includes a deterministic finite automaton perspective.
  • Utilizes the Myhill-Nerode theorem.
  • Submitted by Keyon Vafa.