Every week is exciting with something new and possibly worrisome in the world of artificial intelligence. This week, Jacob Coxon, a 27-year-old researcher, resigned from Anthropic and skipped the polite goodbye, posting a warning instead. He had spent three years training AI models, first at OpenAI, then at Anthropic, joining the latter because he thought it was the careful one. This week, he decided careful wasn't the same as safe. Neither company, he wrote, is "acting responsibly". Both are "racing straight to self-improving superintelligence and gambling with our lives".

That an insider said it is the whole point. The striking part of his letter isn't the doom, as we have heard such talk before. But his account of why clever people keep building a thing they're frightened of is interesting.

Coxon anticipates the obvious retort: if you really believe this, why is anyone still doing it? His answer splits the two labs neatly. At OpenAI, he says, most people have simply not "deeply internalised the civilisational stakes". At Anthropic, they have, and that is worse. There, the danger is understood, but everyone is "locked in a race to get there first", each convinced that if they hesitate, someone less scrupulous grabs the wheel. So the knowledge sits right beside the accelerator and does nothing to slow it down.

Two years ago, Geoffrey Hinton, the "godfather of AI", a Turing Award winner, walked out of Google to say much the same thing. He warned that machines might soon outsmart us, could one day slip out of our control, and might pose a threat even more urgent than climate change. And asked how he could have spent a whole career building the very thing he now feared, he gave an answer that ought to haunt this industry: "If I hadn't done it, somebody else would have."
Link: https://www.ndtv.com/opinion/the-insiders-warning-why-anthropic-researchers-resignation-should-worry-us-12023478