Initial Evaluation of GPT-5.6 on ChatIBD's Internal Benchmarks
GPT-5.6 was released last week and we benchmarked the variants Luna, Terra and Sol, against our current default, GPT-5.4-mini, on a 21-question synthetic benchmark: same prompt, knowledge base, and tools, one sample each. Every variant ran at medium reasoning; following that, we also ran Luna at low reasoning to isolate how much of the cost is reasoning-token overhead.
Frontier models have advanced so quickly that ChatIBD's setup can be adequately served by smaller models with lower reasoning budgets.
This is an initial evaluation. We thought it'll be interesting to share a sneak peek behind the scenes of the work that goes into updates to the underlying ChatIBD architecture.
Cost and latency
The clearest way to see the trade-off is to plot total cost against latency. On the core 21-question benchmark assessing general IBD queries, we find no significant differences in answer quality here across all models tested.
| Model | Completed | Total cost | Median latency | Median reasoning tokens |
|---|---|---|---|---|
| GPT-5.4-mini · medium (current default) | 21 / 21 | $0.37 | 12.1s | 657 |
| GPT-5.6 Luna · low | 21 / 21 | $0.35 | 12.4s | 112 |
| GPT-5.6 Luna · medium | 21 / 21 | $0.44 | 18.8s | 248 |
| GPT-5.6 Terra · medium | 20 / 21 | $0.82 | 17.2s | 45 |
| GPT-5.6 Sol · medium | 20 / 21 | $2.36 | 34.3s | 305 |
Cost is the total across all 21 questions. The two incompletes were both 60-second timeouts.
At medium reasoning, GPT-5.6 is broadly at parity on answer quality but runs more expensive and slower: 1.2× to 6.4× the cost and 40–180% slower at the median. Dropping Luna to low reasoning brings cost to $0.35 and median latency to 12.4s, close to the current default on both, with median reasoning tokens falling from 248 to 112.
Tool behaviour
| Model | Knowledge searches | Drug-safety searches | Questions with >2 searches | Avg steps |
|---|---|---|---|---|
| GPT-5.4-mini · medium (current default) | 43 | 8 | 6 | 2.71 |
| GPT-5.6 Luna · low | 22 | 10 | 0 | 2.14 |
| GPT-5.6 Luna · medium | 28 | 11 | 1 | 2.33 |
| GPT-5.6 Terra · medium | 22 | 10 | 0 | 2.10 |
| GPT-5.6 Sol · medium | 34 | 9 | 2 | 2.40 |
Every GPT-5.6 configuration retrieves less than GPT-5.4-mini and takes fewer steps. Luna-low is the tightest — roughly half the knowledge searches and never more than two per question. Its speed-up over Luna-medium came from reasoning tokens (248 → 112 at the median) rather than tool calls, which were already below the current default's.
Multilingual robustness
We also ran a small multilingual set that checks whether non-English answers stay fully in the user's language. Our current model occasionally leaves section headings or the "no information found" fallback in English inside an otherwise-fluent non-English answer. Both Luna configurations stayed fully in-language on every case, including non-Latin scripts, at similar cost and latency to today's default.
| Model | Passed | Total cost | Median latency |
|---|---|---|---|
| GPT-5.4-mini · medium (current default) | 4 / 6 | $0.077 | 11.1s |
| GPT-5.6 Luna · low | 6 / 6 | $0.087 | 10.5s |
| GPT-5.6 Luna · medium | 6 / 6 | $0.100 | 16.6s |
Reliability check
To gauge whether the single-run numbers were a lucky sample, we re-ran Luna-low three times per question on both sets. Reliability, cost, latency, and the multilingual result all held: 63/63 completions, and every localisation check passed.
Where this leaves us
GPT-5.6 Luna with low reasoning, maintains general answer quality at comparable costs & latency against our current baseline, while improving multilingual performance. We are sharing these numbers as a first look rather than a decision. Caveats: The headline figures come from single runs, the reliability metrics are averaged across three repeats. For now, this is what the new GPT-5.6 models look like on our setup. We are in the process of assessing Luna at low vs medium, weighing the tradeoff of slightly higher costs and latency against more reasoning tokens which can provide a buffer for more complex questions.
All figures are directional. If you have feedback or questions, reach us at contact@chatibd.com.