Pathway’s brain-inspired architecture development on Amazon SageMaker HyperPod
aws ·
Pathway is developing BDH, a brain-inspired post-transformer architecture, and uses Amazon SageMaker HyperPod to scale its training. BDH performs reasoning in latent space instead of externalizing chain-of-thought, with sparse local interactions among neuron-like units and memory encoded in synapse-like connections. It adapts in context without test-time weight updates and avoids the fixed context-window limits of transformers, using Hebbian learning and linear attention over fixed high-dimensional states. The article says transformers face limitations including systematic generalization problems, costly dense computation, catastrophic forgetting, attention scalability constraints, growing KV-cache, and limited interpretability. Pathway built BDH-CQ on top of BDH to add in-context learning and latent iterative reasoning for visual problem-solving, using recurrent latent computation to explore candidate solutions at fixed memory cost for demonstrations. Pathway integrated BDH with PyTorch and trained on SageMaker HyperPod, using Amazon Managed Service for Prometheus and Grafana for observability, EFA networking, and Amazon EC2 p5en.48xlarge instances with NVIDIA H200 GPUs in an UltraCluster. BDH-CQ achieved 29.5 percent pass@2 on ARC-AGI at a cost of US $0.0007 per task, changing the cost-accuracy Pareto frontier on ARC-AGI-1 as of August 2026.
aws