Beyond the price per token: Choosing the right OpenAI model on Amazon Bedrock for your workload
aws ·
An AWS Machine Learning Blog post argues that organizations should evaluate OpenAI models on Amazon Bedrock by cost per successful outcome rather than by price per token, since production workloads buy results such as resolved tickets or completed briefs. The authors used an open-source benchmarking harness (openai-on-aws/benchmarks-openai) that runs one identical code path through the OpenAI Responses API against five models: gpt-5.6-luna, gpt-5.6-terra and gpt-5.6-sol on Amazon Bedrock, plus gpt-5.4-mini and gpt-5.4-nano on the OpenAI API as cost-efficient baselines. The Bedrock models ran with reasoning disabled while the OpenAI API baselines ran at defaults, so the authors describe the results as a comparison of practical deployment configurations rather than intrinsic model capability. On single-call benchmarks (AIME, GPQA Diamond, MMLU-Pro), capability tiers were visible: sol solved 75% of AIME problems versus mini's 37%, and led GPQA Diamond 68% versus 43% and MMLU-Pro 82% versus 59%. After the July 30, 2026 price reduction on Amazon Bedrock for Luna and Terra, the recorded cost per correct AIME answer was $0.0021 for luna versus $0.0139 for mini. In a live multi-turn DeepSearchQA agent workload, turn count drove input-token volume, with mini taking the most turns (7.6) and reaching 2.3 times terra's input-token volume, costing $0.40 per passing answer versus terra's $0.31 and luna's $0.05. On a 48-task GDPval slice of professional deliverables, luna had the lowest recorded cost per passing deliverable at $0.010 versus mini's $0.030 and nano's $0.012, passing 27 tasks to mini's 20, though outputs were capped at 8,192 tokens and smaller per-category samples are exploratory. The post closes with a workload-shape decision framework favoring luna for high-volume and document production, luna and terra for tool-chaining agents, and sol where accuracy is a hard gate, while noting that sample sizes of 48–198 items mean small gaps are directional and that latency figures are point-in-time snapshots.
aws