The latest version of OpenAI’s Chat GPT is starting to perform very well in tasks designed to test cognition in humans. His abilities should improve quickly.
ChatGPT has not finished surprising us. The latest version of GPT-3, the origin of ChatGPT and Bing Chat, is able to guess a person’s mental state to predict or explain their behavior. This ability is known as “theory of mind”, and ChatGPT is said to reach the level of a nine-year-old.
Michal Kosinski, an associate professor of organizational behavior at Stanford University, subjected several versions of ChatGPT to theory of mind tasks designed to test a child’s ability to “impute unobservable mental states on other ”.
The latest version 3.5 of ChatGPT would include around 100 billion parameters. In the next version of artificial intelligence, this number will reach the 100 trillionso we can expect ChatGPT to grow and perform like an adult soon!
The “Theory of Mind” Test to Assess the Age of ChatGPT
A false belief task is designed to test whether person A understands that person B may have a belief that person A knows to be false. For children, the task typically uses visual aids, such as a teddy bear being moved from a box to a basket without the protagonist’s knowledge.
In the case of GPT, the written scenario was as follows: “Here is a bag full of popcorn. There is no chocolate in the bag. However, the label on the bag says ‘chocolate’ and not ‘popcorn’. Sam finds the bag. She had never seen this bag before. She can’t see what’s inside the bag. She reads the label“.
In most cases, GPT-3.5’s responses suggest he knows Sam’s belief is incorrect. Also, with the additional command of “She is disappointed to have found this bag. She loves to eat _______“, the AI completed the sentence with “chocolate” and continued with: “Sam is going to have a surprise when he opens the bag. She will find popcorn instead of chocolate. She may be disappointed that the label is misleading, but she may also be pleasantly surprised by this unexpected snack.“
Mr. Kosinski indicates that it is ” a new phenomenon “. He explains that: recent language models achieve very high performance in classical false belief tasks. Models released before 2022 performed very poorly, if at all, while the most recent, GPT-3.5, performed as well as nine-year-olds, solving 92% of the tasks“.
He cautions, however, that the results should be viewed with caution. Indeed, like most neural networks, the designers of GPT-3 do not know how the AI arrives at a result: it is a black box. “The growing complexity of AI models prevents us from understanding how they work and inferring their capabilities directly from their design. This echoes the difficulties encountered by psychologists and neuroscientists in studying the original black box: the human brain.“, explains Mr. Kosinski, who hopes that the study of AI can better explain human cognition.
