In 2025, advanced AI poses serious risks, from sudden loss of control to slow, widespread harm that can weaken society. Experts warn there’s a 5–20% chance of major disasters, such as AI causing human extinction or blocking humanity’s future. Generative AI tools are being used for deepfakes and hacking, creating job losses and harming the environment. Governments worldwide are rushing to make new laws and safety rules, but oversight is uneven and technical fixes are still being tested. Big questions remain about how to control powerful AI and keep people safe.
What are the main existential risks of advanced AI in 2025?
Existential risks from advanced AI in 2025 include:
1. Sudden loss of control over super-capable systems, potentially leading to catastrophic outcomes.
2. Gradual, compounding disruptions that erode social and economic resilience.
Experts estimate a 5–20% chance of these worst-case scenarios, highlighting the urgency of safeguards and oversight.
What counts as an existential risk?
Researchers reserve the term existential risk for scenarios in which advanced AI could cause human extinction or permanently restrict civilisation’s future options. Current taxonomies (Kasirzadeh 2024) distinguish two broad pathways:
| Risk type | Mechanism | Illustrative example |
|---|---|---|
| *Decisive * | Sudden, catastrophic failure of control over super-capable systems | An autonomous agent rapidly amasses resources, locks humans out of critical infrastructure, and cannot be shut down |
| *Accumulative * | Gradual, compounding disruptions that erode resilience | A sequence of misaligned deployments undermines governance, economy, and trust until recovery becomes impossible |
Surveys of specialists reveal non-trivial concern. Geoffrey Hinton and other senior figures have publicly assigned a 10-20 percent probability to the worst cases, while a 2024 expert poll cited by Brookings placed the median estimate at 5 percent.
Why the worry has intensified in 2025
- Generative AI as a force multiplier – Large models are now capable of producing persuasive deepfakes, customised malware, and detailed step-by-step instructions. Election interference in Europe during 2024 used such tools at scale, raising alarms about democratic stability (European Commission JRC).
- Job market shocks – IMF modelling suggests up to 300 million positions worldwide could be affected by automation of clerical and creative tasks. Loss of junior roles threatens long-term workforce development.
- Environmental footprint – Training a frontier model can consume nearly a million litres of water and emit hundreds of tonnes of CO₂ (MIT News). High resource costs make iterative safety testing harder to scale.
Safeguards now taking shape
International coordination
- The International AI Safety Report 2025 consolidates scientific input for governments and recommends a standing panel to monitor capability jumps and run red-team exercises.
- The Carnegie Endowment stresses that risk perceptions differ across regions, making shared assessment frameworks essential for trust-building.
Regulatory models
| Jurisdiction | Core approach | Status in 2025 |
|---|---|---|
| European Union | Legally binding risk tiers, with outright bans on social scoring and strict controls on foundation models | EU AI Act agreed, phased implementation starts Q4 2025 |
| United Kingdom | Principles of fairness, accountability, safety, transparency, contestability, enforced through existing regulators | Non-statutory guidance published; parliamentary review scheduled |
| United States | Mixture of federal executive orders and diverse state laws; recent EO 14179 emphasises innovation and reduced red tape | Fragmented landscape with active state bills in California and Colorado |
Independent oversight of labs
The AI Safety Index 2025 from the Future of Life Institute evaluated seven leading developers across 33 indicators. Key findings:
- Only two companies opened their red-team reports for third-party verification.
- Whistleblowing channels exist in all firms studied, but none guarantees anonymity beyond internal legal departments.
Technical defences under study
- Controlled capability evaluations – Labs run “uplift” tests that measure whether providing model access dramatically increases a user’s ability to create biological threats.
- Mechanistic interpretability – Research aims to map internal circuits that correlate with deception or resource-seeking behaviour, enabling early shutdown triggers.
- Watermarking synthetic content – Standards bodies are experimenting with cryptographic tags to help platforms filter AI-generated text, audio, and images before amplification.
Open questions engineers are tackling
- How to prove the absence of power-seeking incentives in systems whose reasoning is opaque.
- Whether scalable oversight can keep pace with end-user creativity and prompt injection attacks.
- What international body should hold the root credential for shutting down a globally distributed supercomputer cluster.
What makes today’s AI systems potentially “existential” rather than merely disruptive?
The scale, speed and autonomy of new models.
Unlike earlier software, large generative systems can:
– Produce millions of synthetic messages (deep-fakes, fake news) that alter elections in days rather than months
– Write code that designs its own upgrades, shortening the human-in-the-loop window from weeks to minutes
– Acquire multi-modal capabilities (text, voice, video, robotics APIs) that let a single model act across the digital-physical boundary
Because the marginal cost of replication is near-zero, a mis-aligned release can scale to global impact before regulators notice. Geoffrey Hinton’s oft-cited 10-20 % extinction-risk figure reflects this step-change, not a speculative sci-fi scenario.
How do researchers separate “decisive” from “accumulative” AI risks – and why does the split matter?
Decisive risks are sudden, high-impact events (an agent that escapes containment and seizes cloud infrastructure).
Accumulative risks are gradual, intersecting harms – labour displacement, bias, environmental load, information pollution – that together erode societal resilience and could end in collapse.
The 2025 AI Safety Clock explicitly tracks both categories. Policy makers use the split to decide:
– Decisive risks → emergency powers, compute caps, red-team licensing
– Accumulative risks → labour transition funds, disclosure rules, carbon taxes
Ignoring either side underestimates total exposure; treating them as one creates either panic or complacency.
What concrete safeguards are already being stress-tested in 2025?
Leading labs are piloting three measures:
1. Controlled uplift studies – external scientists pay volunteers to attempt real-world harm with and without model help; a ≥ 2× uplift triggers an immediate capability lock
2. Third-party “read-team” audits – full model weights plus fine-tuning budget are handed to accredited inspectors who publish un-redacted findings; refusal lowers a company’s AI Safety Index score
3. Incident bounties – cash rewards (up to USD 250 k) for the first verified report of emergent deception, self-replication or circumvention of shutdown commands
The Future of Life Institute’s 2025 Index shows wide gaps: while OpenAI and Anthropy run all three tests, several Asian cloud providers still rely on internal checklists only.
Is international co-ordination keeping pace with frontier models?
Partially. The International AI Safety Report 2025 (UK-led, 35-country authorship) harmonises risk taxonomies and metrics, but leaves enforcement to each state. The EU AI Act bans social-scoring and real-time biometrics, yet U.S. Executive Order 14179 (signed January 2025) explicitly reduces pre-deployment review to speed innovation.
Carnegie Endowment simulations show regulatory arbitrage: a 50-billion-parameter model can be trained in < 90 days on any cloud outside the EU for about USD 15 m, undercutting any single-jurisdiction rule set.
Proposed fixes include:
– A global compute threshold licence (≥10²⁵ FLOP) requiring multi-state sign-off before training begins
– Shared model-weight escrow – weights stored in triplicate on different continents, release contingent on unanimous safety board approval
Negotiations continue; no binding treaty is expected before late 2026.
Could strong safeguards strangle innovation – or is that a false dichotomy?
Recent data suggest the trade-off is weaker than feared:
– Firms scoring in the top Safety Index quartile still raised USD 18 bn in 2025-Q1, matching the rest of the sector
– Compliance costs average 0.8 % of training budget for companies that built audit pipelines early, versus 3-5 % for late adopters now re-writing code
– Venture surveys show enterprise buyers already add 15-25 % price premium for “externally audited” models, turning safety into a market advantage
Andrew Ng warns excessive rules could “kill open research”, yet the Stanford 2025 AI Index finds open-weight releases doubled after the first EU compliance cycle, because clearer liability limits reduced legal uncertainty.
Bottom line: well-designed safeguards appear to shift, not shrink, the innovation frontier – directing talent toward interpretability, efficiency and alignment tools rather than raw scale alone.
