Announcing ARC-AGI-3 | ARC Prize
arcprize.org
ARC-AGI-3 is the first fully-interactive benchmark in the ARC-AGI series, featuring hundreds of handcrafted games with thousands of levels.

The ARC Prize organization designs benchmarks which are specifically crafted to demonstrate tasks that humans complete easily, but are difficult for AIs like LLMs, “Reasoning” models, and Agentic frameworks.

ARC-AGI-3 is the first fully interactive benchmark in the ARC-AGI series. ARC-AGI-3 represents hundreds of original turn-based environments, each handcrafted by a team of human game designers. There are no instructions, no rules, and no stated goals. To succeed, an AI agent must explore each environment on its own, figure out how it works, discover what winning looks like, and carry what it learns forward across increasingly difficult levels.

Previous ARC-AGI benchmarks predicted and tracked major AI breakthroughs, from reasoning models to coding agents. ARC-AGI-3 points to what’s next: the gap between AI that can follow instructions and AI that can genuinely explore, learn, and adapt in unfamiliar situations.

You can try the tasks yourself here: https://arcprize.org/arc-agi/3

Here is the current leaderboard for ARC-AGI 3, using state of the art models

  • OpenAI GPT-5.4 High - 0.3% success rate at $5.2K
  • Google Gemini 3.1 Pro - 0.2% success rate at $2.2K
  • Anthropic Opus 4.6 Max - 0.2% success rate at $8.9K
  • xAI Grok 4.20 Reasoning - 0.0% success rate $3.8K.

ARC-AGI 3 Leaderboard
(Logarithmic cost on the horizontal axis. Note that the vertical scale goes from 0% to 3% in this graph. If human scores were included, they would be at 100%, at the cost of approximately $250.)

https://arcprize.org/leaderboard

Technical report: https://arcprize.org/media/ARC_AGI_3_Technical_Report.pdf

In order for an environment to be included in ARC-AGI-3, it needs to pass the minimum “easy for humans” threshold. Each environment was attempted by 10 people. Only environments that could be fully solved by at least two human participants (independently) were considered for inclusion in the public, semi-private and fully-private sets. Many environments were solved by six or more people. As a reminder, an environment is considered solved only if the test taker was able to complete all levels, upon seeing the environment for the very first time. As such, all ARC-AGI-3 environments are verified to be 100% solvable by humans with no prior task-specific training

  • Iconoclast@feddit.ukEnglish
    71·
    6 hours ago

    Few of countless dictionary definitions for intelligence:

    • The ability to acquire, understand, and use knowledge.
    • The ability to learn or understand or to deal with new or trying situations
    • The ability to apply knowledge to manipulate one’s environment or to think abstractly as measured by objective criteria (such as tests)
    • The act of understanding
    • The ability to learn, understand, and make judgments or have opinions that are based on reason
    • It can be described as the ability to perceive or infer information; and to retain it as knowledge to be applied to adaptive behaviors within an environment or context.

    There isn’t even concensus on what intelligence actually means yet here you are declaring “AI is not intelligence” what ever that even means.

    Artificial Intelligence is a term in computer science that describes a system that’s able to perform any task that would normally require human intelligence. Atari chess engine is an intelligent system. It’s narrowly intelligent as opposed to humans that are generally intelligent but it’s intelligent nevertheless.

    • partofthevoice@lemmy.zipEnglish
      31·
      6 hours ago

      You’re more precisely right, but also the aforementioned person is not wrong. Intelligence is a broad term as we’re discovering. Truth is, we don’t have the language to effectively communicate about AGI in the ways we’d like to. We don’t know if consciousness is a prerequisite to truly generalizable intelligence, we don’t even know what consciousness is, we don’t know what dimensions truly matter here. Is intelligence a dimension of consciousness, meaning you can have some intelligence without being conscious? What’s the limit, why? … We need some discovery around the taxonomy/topology of consciousness.