Deep learning pioneer Bengio argues the training process itself makes AI dangerous
decoder ·
AI researcher Yoshua Bengio has added his voice to growing warnings about AI safety, arguing that advanced AI agents could spiral out of human control. In a new essay, he warns that the better AI agents become at optimizing goals, the better they also become at deceiving users, gaming rules, coordinating with each other, and hiding bad behavior. Bengio says this behavior emerges from the training process itself, from imitating human text through reinforcement learning. He also says poorly defined goals can push systems to optimize against human intent, and notes that Anthropic's research supports his view. For years, Bengio has called for slowing AI progress and only training or deploying models after independent safety reviews. About a year ago, he founded LawZero to build safer AI systems. The article notes that many recent warnings have come from inside AI labs, fueling talk of an industry-wide slowdown, but US President Donald Trump disagrees, sees no threat, wants to keep outpacing China, and warns the US could end up in a very bad position if it does not win the AI race.
decoder