Reduce RAG costs on Amazon Bedrock with query-aware compression
aws ·
AWS published a blog post describing a query-aware compression pattern to reduce RAG costs on Amazon Bedrock. The pattern inserts a smaller, cheaper model between retrieval and the final answer call, which filters retrieved chunks against the user's query and outputs only verbatim relevant spans. This compressed context is then sent to the primary model, reducing input tokens and cost. The implementation uses a single AWS Lambda function that calls the Bedrock Converse API twice, once for compression and once for the answer. The economics depend on the price ratio between the small and primary models and the achieved compression ratio. In their benchmark, the compression pipeline reduced costs by 33% and tokens sent to the primary model by 8.6x, while retaining 97.5% of baseline composite quality and reducing hallucination rates. Adding a rerank step before compression achieved a 36% cost saving and a 10.1x token reduction, with slightly different quality trade-offs. The post also covers the compression prompt design, cost model, latency trade-offs, and evaluation methodology.
aws