AI Poker Challenge: Why It's a Misleading but Entertaining Spectacle

The recent AI Poker Challenge on Kaggle has sparked debate and entertainment among poker enthusiasts. The challenge pitted Large Language Models (LLMs) such as ChatGPT, Gemini, Grok, and Claude against each other in a tournament featuring three games: chess, werewolf, and poker. Doug Polk, a renowned poker expert, was invited to participate in the challenge and analyze hands played by the LLMs.

One of the most striking observations from the challenge is that LLMs are really bad at poker. They struggled to read their hole cards, the board, or even understand basic rules of Texas Hold’em—flushes seem to be a particular pain point. This is not surprising to those who have followed similar challenges, as LLM chess matches often feature illegal moves, adding new pieces to the board, and more. While entertaining, these performances highlight the limitations of LLMs in tasks they were not designed for.

The Kaggle challenge is not representative of AI on a large scale — it shows a small subset of it (LLMs), placed in an environment for which they weren’t developed. Simply put, if you’re using an excellent tool to do a job it’s not meant to do, you’ll be disappointed. That doesn’t make the tool bad, though. In fact, we could go as far as to say that such an application provides almost zero relevant information about the tool. AI has been superior to humans in chess for a while now, and this is no longer disputed by anyone. Even the best chess players in the world refuse to play AI because it’s pointless. They can’t win. With the rise of poker solvers and the development of GTO strategy, we now have very good, almost perfect mathematical models for Hold’em. A player capable of reproducing this strategy 100% would crush it in the long run. However, LLMs are very bad at “thinking” or analyzing things, as they were not built for that purpose. LLMs represent a subset of the larger AI network, whose primary purpose is gathering and interpreting text inputs and generating credible, reliable text outputs. LLMs are not using solvers or crunching numbers, their knowledge is based on what they picked up from various sources across the internet.