If you want to evaluate an AGI claim or an AGI benchmark score, ignore the word AGI for a minute. Ask five questions: What exactly is being claimed? What system took the test? What was it allowed to use? Who is the human comparison? What result would prove the claim wrong? If those answers are fuzzy, the headline is ahead of the evidence.
I learned this the uncomfortable way. My September 4 LinkedIn post asked if GPT-6 Astra was AGI or just the best model of the month. It reached 35,401 impressions, 276 reactions and 149 comments. One number in that post already needs correcting: OpenAI's current page reports 99.9% on ARC-AGI-3, rather than 98.6%.
in
“Is this AGI, or just the best model of the month?”
How to Evaluate AGI Claims in 30 Seconds
- Define what AGI means in this specific claim.
- Name the exact model, agent or product that was tested.
- Check the tools, retries, time and help it received.
- Read the benchmark and human baseline before reading the score.
- Ask what the system still cannot do and what would falsify the claim.
That is the whole method. The rest of this article shows you how to use it without needing a research degree. The goal is not to make every breakthrough sound small. It is to keep an impressive result attached to the exact thing that was measured.
1. Make Them Define AGI
AGI can mean at least four different things: learning unfamiliar tasks, doing most paid knowledge work, matching people across many cognitive skills or operating alone for long periods. Those are different claims. They need different proof. If nobody defines the finish line first, the finish line moves every time a new benchmark drops.
Make the definition concrete. Which tasks? Which people? How much help? What failure rate? A model that beats casual users on one puzzle is different from a system that matches experts across hundreds of jobs. Both can be impressive. Only one might support the claim you were sold.
Stanford HAI's guide to validating AI claims connects the claim, the test and the evidence behind the interpretation.
2. Check What Actually Took the Test
Here is the part most launch posts make easy to miss: a model, an agent and a product are not the same thing. An agent can add tools, memory, search, retries and a second model that checks the answer. A product can add permissions and human help. The final score belongs to that complete setup.
Ask for the model version, reasoning setting, tool access, number of attempts and selection method. If the system tried a problem one hundred times and somebody kept the best answer, say that. It does not make the result fake. It tells you what resources were needed to produce it.
3. Read the AGI Benchmark Before the Percentage
OpenAI currently reports 99.9% on ARC-AGI-3 for Astra. That is a huge result. It shows the system is extremely strong on that evaluation under OpenAI's stated conditions. It does not automatically prove the system can run a company, understand every unfamiliar situation or replace most human work.
OpenAI's GPT-6 Astra announcement contains the reported scores and evaluation details.
The human comparison matters too. ARC Prize built its ARC-AGI-3 baseline from a controlled study with 458 participants. Good. Now we know who was tested and inside which environment. That is far more useful than a chart that simply says human level, as if every person and every job were interchangeable.
ARC Prize explains the participants and setup in its ARC-AGI-3 human performance study.
My rule is simple: never let a percentage travel without its test. Ask what the benchmark includes, what it excludes and whether the examples resemble the work you care about. A perfect score on the wrong thing is still the wrong evidence for your decision.
4. Look for Transfer Beyond the Demo
Real generality shows up when the environment changes. New instructions. An unfamiliar interface. Missing information. A task that was never in the examples. If performance collapses as soon as the surface looks different, you may have an excellent specialist. You do not yet have proof of a general one.
This is why one leaderboard cannot settle the AGI question. You want several kinds of tasks, held-out problems and human comparisons. You also want repeated results. One spectacular run can start the conversation. Reliable transfer is what moves the claim forward.
Google DeepMind's framework for measuring AGI shows why breadth and staged evaluation matter.
5. Separate Capability From Permission
A system can be capable of doing something and still be the wrong system to authorize. OpenAI says Astra reaches its Critical cybersecurity capability threshold. It also says monitoring the model's written reasoning becomes harder under adversarial conditions. Those are two important facts. Neither one answers the whole AGI question.
OpenAI's Astra safety overview describes the capability threshold, safeguards and monitoring limits.
For a business, this distinction is practical. A model may be good enough to draft a contract, move a deal or change production code. That does not mean it should do those things without approval. Intelligence tells you what it can attempt. Permission tells you what it is allowed to touch.
My One-Page AGI Claim Scorecard
When the next AGI announcement lands, put six fields on one page. This takes ten minutes and kills most of the confusion:
- Claim: copy the exact sentence being promoted.
- Definition: write what AGI means here in plain English.
- System: name the model, tools, retries and human help.
- Evidence: link the dated primary source and benchmark.
- Limits: list what the result does not establish.
- Decision: state what you will test or change because of it.
Here is what that looks like for Astra. Claim: this is AGI. Evidence: a named system scores 99.9% on ARC-AGI-3 under reported conditions. Supported conclusion: Astra is exceptionally strong on that benchmark. Still unknown: whether the result transfers across the broader tasks and operating conditions required by your definition of AGI.
What the GPT-6 Astra Evidence Actually Says
The honest answer is less dramatic than the headline and more useful. Astra shows major progress across reasoning, computer use, mathematics and professional tasks. The ARC result deserves attention. The safety threshold deserves attention too. Together they justify serious testing. They do not force every reasonable person to accept one universal definition of AGI.
I also left four claims from my original post out of this article: the GPU count, exact number of browser vulnerabilities, price comparison and attributed executive statements. I could not independently verify all four from current primary sources. Repeating them here would make the article sound more exciting and make the evidence worse.
So, Is It AGI?
Maybe under a benchmark-based definition. Not proven under every broader definition. That is not a dodge. It is the only answer that keeps the conclusion tied to the evidence we actually have. The better question for your company is whether this system completes your real work more reliably, safely and economically than the alternatives.
If that is the decision you need to make, use my guide to evaluating AI models on real work and build a small side-by-side pilot.
The next time somebody says AGI has arrived, do not choose a team from the headline. Run the five questions. Keep the score, the test conditions and the limits together. Then decide what deserves a real pilot. That is how you stay excited about the progress without letting the marketing make the decision for you.
