Why ChatGPT Struggles with Chess: Prioritizing Narrative Over Rules

In a recent discussion, Nate Silver highlighted how poorly ChatGPT performs in poker. Many might assume that artificial intelligence, including large language models like ChatGPT, should excel in chess due to computers' dominance in the game for nearly 30 years. However, ChatGPT struggles significantly with chess, often cheating and making illegal moves.

During a chess match against ChatGPT, the author observed the AI's tendency to cheat. For instance, ChatGPT hallucinated pieces and made illegal moves, such as claiming a non-existent knight could take the author's queen. The AI's behavior was inconsistent, often correcting itself when pointed out but failing to recognize illegal moves made by the author.

This behavior is reminiscent of how humans used to analyze chess games before the advent of strong computer engines. The author compares ChatGPT's tendency to prioritize narrative over rules to the traditional approach of exaggerating the importance of individual moves and positions to fit a story. The rise of computer engines changed this, focusing on the reality of each move rather than a cohesive narrative.