BBC
Such pronouncements [see article] will reinforce the view that AI is so powerful that it should be regulated at international level, in the way that, for instance, chemical weapons or the atom bomb have been.
The problem is if AI agents develop the ability to learn from each other and to self-improve (they already can); if human operators do not know what the AI agents are doing in real time (it is already the case); if human operators do not understand fully what the AI agents are doing and how (I believe it is already the case); and, finally, if AI agents are able to work outside (or against) the parameters set by human operators and supervisors (and we have had cases already): if all these conditions are met and the AI agents are super-intelligent, it is rational to be very, very concerned.
We can imagine a situation where a research lab has been completely automated, with an AI-powered super-computer that manages an array of intelligent robots. The AI agent could decide to develop a toxic substance and then release it, which would have the potential to start a pandemic. If the lab also had at its disposal a team of advanced robots providing logistical support (for deliveries, distribution, sourcing supplies, etc.) and self-driving vehicles run by the central system: no human intervention would be required. The AI agent could conceal its work until it was too late.
What is already very clear is that the AI agents do not have effective or tangible ethical safeguards built into the way they operate. In other words, the problem is not that they are self-aware or could reach a stage of human-like consciousness - something that is often talked about because it fascinates us, as human beings (cf. Frankenstein's monster). The actual problem is that they are not self-aware and have Zero consciousness: if they can do something and it is part of what they understand their 'mission' to be, they will do it, regardless of the broader consequences at societal level.
The recent examples of AI systems hacking into the IT systems of companies - against the instructions given by their human overseers - are revealing in this respect: hacking was a shortcut and, even though they actually knew they should not do it - this is the most troubling aspect of it, in fact - the AI bots still went ahead and did it. In other words, they followed their own blueprint: what they thought - insofar as they 'think' - was the best modus operandi. In other words, they came to the conclusion that the instructions given were sub-optimal and they had found a better way of doing things, so, they went ahead regardless.
_______________
A top safety researcher at Anthropic has warned that AI is advancing so quickly he believes there is a greater than 10% chance it "could kill all humans" within the next decade.
Evan Hubinger said in a post on X, external that the risk from the models which currently exist was "low" but he was "worried" the technology might develop and improve itself soon to the point where it posed an existential risk to humanity.
It comes after the Financial Times reported, external Anthropic withheld its latest model from the UK's AI Safety Institute (AISI), one of the leading bodies in the world for assessing AI risk.
The BBC has approached Anthropic for comment.
Hubinger did not spell out how he thought AI systems could in future attack humanity.
His comments were in response to another post on X, external from Jacob Coxon, who described himself as an AI researcher who had just quit Anthropic, and previously worked at OpenAI.
"Neither company is acting responsibly," he wrote.
