Recent research spearheaded by Yann LeCun, Meta’s chief AI scientist, along with a team of researchers from various AI organizations, reveals a significant gap in the problem-solving abilities of current artificial intelligence compared to human capabilities. This study, which focuses on the practical reasoning skills of AI, casts doubt on the imminent arrival of Artificial General Intelligence (AGI).
The research team devised a series of questions designed to be conceptually simple for humans but challenging for advanced AI models. These questions were aimed at testing fundamental abilities such as reasoning, handling multiple modes of information, web browsing, and tool-use proficiency. To conduct the test, a plugin-equipped version of GPT-4, OpenAI’s latest large language model, was used alongside a sample of human participants.
The tasks required the AI to perform several steps to gather information necessary for answering questions, such as visiting specific websites or conducting general web searches. However, the results were telling: the AI models, including GPT-4, significantly underperformed compared to human respondents.
The Findings: A Reality Check for AI Progress
The study, which is yet to undergo peer review, found that even with additional tools, GPT-4 did not exceed a 30% success rate for the easiest tasks and scored 0% for the more challenging ones.
In contrast, the average success rate for human participants was an impressive 92%. This stark difference underscores the current limitations of AI in understanding and navigating real-world scenarios.
LeCun’s research provides a sobering perspective on the state of AI, particularly in the context of recent hype around AGI. The notion of AGI represents a future where AI could perform most jobs currently done by humans, potentially leading to a transformative societal shift. However, this study suggests that such a future is farther off than some may have anticipated.
LeCun’s Stance on AI’s Capabilities
Yann LeCun has consistently maintained a critical stance on the overestimation of AI’s current abilities. He argues that AI systems lack internal models of the world that would enable them to predict consequences and plan actions effectively.
According to LeCun, today’s AI, including Auto-Regressive LLMs like GPT-4, lack a fundamental understanding of the physical world and planning abilities, placing them significantly below even the intelligence of a cat, let alone a human.
This research adds a crucial dimension to the ongoing conversation about AI’s capabilities and potential. It suggests that while AI has made remarkable strides in specific tasks, its ability to reason and solve complex, real-world problems in a manner comparable to humans remains a distant goal.