How Intelligent Is AI, Really?...

The ARC Prize Foundation operates as a tech-forward nonprofit, dedicated to pushing open progress toward systems capable of generalizing intelligence similar to humans. This mission is rooted in a specific definition of intelligence, diverging from conventional views focused on task mastery. The foundation's work aims to inspire research and development that addresses the core challenge of learning and adaptability in AI.

The foundation adopts a highly opinionated definition of intelligence, proposed by Francois in his 2019 paper 'On the Measure of Intelligence.' Instead of equating intelligence with high scores on tests like the SAT or complex math problems, Francois defines it as the ability to efficiently learn new things. This perspective contrasts with AI's superhuman performance in specific domains like chess, Go, or self-driving, where learning a new, different skill remains a significant hurdle for these systems.

The ARC AGI benchmark was created to test an AI's ability to learn new things, aligning with Francois's definition of intelligence. Unlike traditional benchmarks that often involve increasingly difficult problems within a single domain (e.g., MMLU, MMLU+, 'Humanities Last Exam'), ARC AGI focuses on tasks that normal humans can solve but challenge AI's generalizing capabilities. The benchmark ensures human solvability for all its tasks.

Initially, base models like GPT-4 (without reasoning) performed poorly, scoring only 4-5% on the ARC benchmark in 2019. However, the introduction of reasoning paradigms significantly improved results. For example, 01 preview saw a jump to 21% after five years, demonstrating the impact of new approaches on generalization. Major AI labs, including OpenAI, XAI (Grok 4), Google (Gemini 3 Pro, DeepThink), and Anthropic (Opus 45), now use ARC AGI as a standard for reporting model performance, validating its role in assessing AI progress.

ARC AGI has evolved through different versions to better assess generalize intelligence. ARC AGI 1, introduced in 2019 by Francois, comprised 800 tasks designed to test foundational learning abilities. This initial version and its successor, ARC AGI 2 (released in March 2025), are both 'static' or 'metastatic' benchmarks, meaning the tasks are presented in a predefined, non-interactive manner.

ARC AGI 3, planned for next year, represents a significant shift towards interactive testing. This version will feature approximately 150 video game-like environments. The key innovation in V3 is the absence of instructions; test-takers (both humans and AI) must figure out the environment's goals and rules by taking actions and observing feedback. This interactive approach mirrors real-world learning and interaction, where actions produce responses, allowing for continuous adaptation and problem-solving through novel experiences. Human solvability is a critical filter for V3 games; if ten general public participants cannot exceed a minimum solvability threshold, the game is excluded, ensuring tasks remain within human cognitive reach.

Evaluating AI intelligence extends beyond mere accuracy; it incorporates factors of data and energy efficiency, vital for assessing human-like learning. While 'wall clock time' is considered arbitrary due to its dependency on compute power, the amount of training data and energy required to execute a task are crucial metrics. Human benchmarks exist for both: the data points a human needs to perform a task and the energy consumption of the human brain for task execution.

ARC AGI 3 will measure efficiency by comparing the number of actions required for an AI to solve a turn-based video game against the average human performance. This method prevents brute-force solutions, prevalent in earlier AI approaches to video games that consumed millions of frames and actions. By normalizing AI performance to human action counts, ARC AGI 3 aims to identify systems that learn efficiently rather than relying on overwhelming computational resources, thereby highlighting true generalization and intelligence.

Solving the ARC AGI benchmarks is considered a necessary but not sufficient condition for achieving Artificial General Intelligence (AGI). Francois maintains that a system solving ARC AGI 1 and 2 would not be AGI, but rather an 'authoritative source of generalization.' For ARC AGI 3, even if a system solves all tasks, it would still represent the 'most authoritative evidence' of generalization to date, not AGI itself. The intent is to continually refine benchmarks to guide the field toward a complete understanding of AGI.

If a team achieves 100% on ARC AGI tomorrow, the ARC Prize Foundation would analyze the system to identify remaining failure points. The goal is to evolve benchmarks to keep pace with AI advancements. Ultimately, the foundation aims to position itself to fully understand and declare when true AGI is achieved, initiating dialogue with any team that reaches that milestone to collaboratively assess the breakthrough.