Sakana Fugu Benchmarks: What "Shoulder-to-Shoulder With Fable 5" Actually Means

Sakana’s Fugu benchmarks are vendor-reported parity claims, not independently verified scorecards. Per Sakana’s release page, Fugu Ultra “stands shoulder-to-shoulder with leading models like Fable 5 and Mythos Preview” on engineering, scientific, and reasoning tasks, and Fugu “consistently outperforms” Gemini 3.1 Pro, Opus 4.8, and GPT 5.5 on a named set of applications. The catch worth understanding before you read any number: Fugu is an orchestrator that calls other vendors’ frontier models, so its results are not single-model wins in the way Fable 5’s are.

What Fugu actually is, and why it changes how you read benchmarks

Fugu is not a single foundation model. It is a multi-agent orchestration system presented as one model behind one OpenAI-compatible API. Sakana describes it as a trained language model specialized in delegation, agent communication, and work synthesis. It dynamically coordinates multiple LLMs, including recursive instances of itself, and decides whether to answer directly or assemble a team. The release headline is “One Model to Command Them All.”

That design detail is the whole story for benchmarks. When a normal model posts a score, the number reflects that model’s own weights doing the work. When Fugu posts a score, the number may reflect Fugu calling Opus 4.8, or GPT 5.5, or Gemini 3.1 Pro, then synthesizing their outputs. So a “beats Opus 4.8” result can come from a system that calls Opus and combines it with other models. That is a model-of-models result, not a single-model result. If you want the deeper architecture context, our explainer on what Sakana Fugu is walks through the orchestration loop.

The parity claim: “shoulder-to-shoulder with Fable 5 and Mythos Preview”

Here is the first claim, stated carefully.

Per Sakana, Fugu Ultra “stands shoulder-to-shoulder with leading models like Fable 5 and Mythos Preview” across engineering, scientific, and reasoning benchmarks. Read the verb. This is a parity claim, not a “beats” claim. Sakana is positioning Fugu Ultra as a frontier peer, not a frontier leader.

Two things deserve a flag.

First, the named rival is “Mythos Preview,” the April frontier model that Anthropic described as too dangerous to release. It is not the current generally available Mythos 5. If you have read about the Mythos-class model, you know Preview and the shipped line are different artifacts. Pinning a parity claim to Preview rather than the current model is a choice, and it matters for how impressive the claim reads.

Second, no benchmark table backs this up in a form anyone outside Sakana can rerun. The claim is qualitative on the release page. There is no published methodology, no per-task score grid, and no third-party reproduction. Treat “shoulder-to-shoulder” as a vendor’s framing of its own internal results.

The stronger claim: “consistently outperforms” on named applications

Sakana makes a second, bolder claim, and it is worth separating from the first.

Per Sakana, Fugu “consistently outperforms” three configured competitors on a specific list of applications:

• Gemini 3.1 Pro (high)
• Opus 4.8 (max)
• GPT 5.5 (xhigh)

The named applications are AutoResearch, Rubik’s Cube, Mechanical Design, Japanese Handwriting Analysis, One-Shot Chess, and Financial Time Series Prediction.

This is application-level performance, not a standard academic benchmark suite. These are end-to-end tasks where an orchestration system has room to shine, because it can route sub-problems to whichever underlying model handles them best and then stitch the results together. That is exactly where a conductor should beat any single player.

But apply the honesty spine again. Several of those competitors are models Fugu can call. A “beats Opus 4.8 (max)” result on AutoResearch may be Fugu calling Opus, calling other models, and synthesizing a stronger combined answer. That is a real capability, and it may genuinely help you. It is not evidence that a single Sakana model out-reasons Opus. Never read these numbers as a single-model win, and never phrase it as “Fugu beats Fable 5,” because Sakana did not even claim that. The parity claim and the outperform claim target different rivals.

Why these numbers cannot be independently verified yet

No independent replication yet. Every Fugu benchmark figure on this page is vendor-reported, measured on Sakana’s own setup, with competitor configurations Sakana chose (the “high,” “max,” and “xhigh” effort settings). As of 2026-06-22, no third party has rerun these tasks, no per-task score grid has been published, and no evaluation harness has been released. The correct posture is to treat all of it as a claim, not a measurement.

This is not a knock on Sakana specifically. It is the default state for any model on launch day. The difference with Fugu is that the orchestration design makes independent replication harder, not easier.

To reproduce a single model’s benchmark, you need the model and the test. To reproduce Fugu’s, you need Fugu plus access to every underlying model it routes to, at the same versions and effort settings, plus the same orchestration topology Sakana ran. The system “dynamically routes around” provider restrictions and adapts its agent topology per task, so two runs of the same prompt may not even use the same internal team. That adaptivity is a feature for users and a headache for reproducibility.

So you will not find clean head-to-head tables here, and you should be skeptical of any floating “Fugu scored X” numbers circulating from secondary sources. Several of those secondary write-ups name the wrong rival versions (current Mythos instead of Mythos Preview, for example). An empty-number state is the honest outcome right now. Our Fugu Ultra vs Fable 5 vs Mythos comparison stays qualitative for the same reason.

The research records behind the claims

Sakana’s marketing rests on real, citable research. Two ICLR 2026 papers describe the lineage. Neither is asserted as a product benchmark, so read them as research records, not Fugu spec sheets.

The first is Trinity, “An Evolved LLM Coordinator” (arXiv:2512.04695). Trinity is a sub-20,000-parameter coordinator optimized by derivative-free evolution, with Thinker, Worker, and Verifier roles. It is tiny and evolved, not trained by gradient descent.

The second is Conductor, “Learning to Orchestrate Agents in Natural Language” (arXiv:2512.04388). Conductor is a 7B model trained with reinforcement learning that learns the communication structure between agents. The paper claims it beats Mixture-of-Agents at lower cost.

These are different methods and different sizes. Trinity uses evolution at sub-20K parameters. Conductor uses RL at 7B parameters. Do not conflate them. And do not assume either paper’s exact specs describe the shipped product. Mapping the 7B figure, or any specific base model, onto the released Fugu is third-party inference. The official release gives no product parameter count.

A specs sidebar to keep next to the claims

Here is what is reasonably established versus what is still unconfirmed. Treat the architecture line as unverified.

The naming note is worth repeating once: the small variant was called “Fugu Mini” during the roughly 500-user beta that opened around April 24-25, 2026. The release page uses “Fugu” and “Fugu Ultra.” Lead with the current names.

What this means for your own testing

You cannot verify Sakana’s benchmarks. You can run your own.

Because Fugu speaks the OpenAI chat-completions protocol, you point an existing OpenAI client at the Fugu base URL and send your real tasks. No SDK migration. The base URL is not published on any public page as of 2026-06-22, so copy it from your console at console.sakana.ai and never trust an invented host. The pattern below mirrors the standard OpenAI chat completions request:

from openai import OpenAI

# Copy the real base URL from console.sakana.ai after you sign in.
client = OpenAI(
    api_key="YOUR_FUGU_API_KEY",
    base_url="<YOUR_FUGU_BASE_URL_FROM_CONSOLE>",
)

resp = client.chat.completions.create(
    model="fugu-ultra",  # 'fugu' for the balanced variant; ids reported, verify in console
    messages=[
        {"role": "system", "content": "You are a precise code reviewer."},
        {"role": "user", "content": "Review this function for security issues:\n<paste code>"},
    ],
)

print(resp.choices[0].message.content)

The model id strings reported so far are fugu and fugu-ultra, possibly with a dated form. Confirm the exact ids in the console rather than baking one into your config. Because Fugu decides per request whether to answer directly or assemble a team, the same prompt can produce different latency and cost on different runs. Log both.

This is where running your own evaluation matters more than usual. Send the tasks you actually care about, not AutoResearch or one-shot chess, and measure latency, cost, and output quality against the single models you already use. Vendor benchmarks tell you what Sakana measured. Your own runs tell you what you will get.

How this fits your Apidog workflow

You do not need a new tool to pressure-test a vendor’s benchmark claims. You need a way to fire the same prompt at several endpoints and compare the responses side by side.

Apidog lets you register the Fugu endpoint as an OpenAI-compatible API, save your real evaluation prompts as requests, and run them as a test scenario. Put Fugu, Fable 5, and an Opus endpoint in the same environment, send identical inputs, and capture the outputs, status codes, latency, and token usage in one place. That is a far more useful comparison than a parity claim with no methodology behind it. When you want to track cost drift from Fugu’s adaptive routing, assertions on response time and token counts surface it run over run. Download Apidog and build the comparison once, then rerun it whenever a new model version ships.

Frequently Asked Questions

Does Fugu beat Fable 5 on benchmarks?

No, and Sakana never claimed that. The claim is parity: Fugu Ultra “stands shoulder-to-shoulder with” Fable 5 and Mythos Preview, per Sakana. The separate “outperforms” claim targets Gemini 3.1 Pro, Opus 4.8, and GPT 5.5 on specific applications, not Fable 5. For the single-model side of that comparison, see the Claude Fable 5 benchmarks.

Are the Fugu benchmark numbers independently verified?

No. As of 2026-06-22, every figure is vendor-reported on Sakana’s own setup, with competitor effort settings Sakana chose. No third party has rerun the tasks, and no evaluation harness has been published. Treat the claims as claims until someone outside Sakana reproduces them.

Why does it matter that Fugu is an orchestrator?

Because Fugu calls other vendors’ frontier models, recursively including itself, a “beats Opus 4.8” result may come from Fugu calling Opus and synthesizing. That is a model-of-models win, not a single-model win. Fable 5 and the Mythos line are single Anthropic models, which makes a direct head-to-head an apples-to-oranges read.

Which Mythos did Sakana compare against?

The older Mythos Preview from April, the frontier model Anthropic described as too dangerous to release, not the current Mythos 5. Some secondary write-ups name the wrong version. The Mythos-class explainer covers the difference between Preview and the shipped line.

What is the difference between Trinity and Conductor?

They are two separate ICLR 2026 papers. Trinity (arXiv:2512.04695) is a sub-20,000-parameter coordinator optimized by evolution. Conductor (arXiv:2512.04388) is a 7B model trained with reinforcement learning. Different methods, different sizes. Neither is asserted as the shipped product’s spec sheet.

How can I test Fugu’s performance myself?

Point an OpenAI-compatible client at the Fugu base URL from console.sakana.ai, send your own tasks, and measure quality, latency, and cost. Register the endpoint in Apidog to compare Fugu against the single models you already use, with identical prompts and captured metrics.