EU AI Office Sets New AGI Safety Rules for Frontier AI Developers

BRUSSELS — The European AI Office on Wednesday released the "Guidelines on Systemic Risk Mitigation for Frontier AGI Models," a 240-page regulatory framework establishing the first mandatory safety standards for general-purpose artificial intelligence (GPAI). The document outlines the operational requirements for developers whose models exceed the compute threshold of 10²⁶ floating-point operations (FLOPs). This benchmark currently applies to high-capacity systems including OpenAI’s GPT-5, Anthropic’s Claude 4, and Mistral’s "Project Jupiter."

The guidelines mandate that developers submit a "Safety Case" to the AI Office 90 days before any public or commercial deployment. This filing must provide empirical evidence that residual risks of catastrophic misuse remain below a defined societal threshold. The AI Office defines catastrophic misuse as the facilitation of large-scale cyberattacks, the design of biological weapons, or the autonomous disruption of critical infrastructure.

Under the new rules, companies must report unforeseen emergent behaviors or security breaches involving model weights within 48 hours. Failure to comply with these transparency mandates carries financial penalties of up to €35 million or 7% of total worldwide annual turnover, whichever is higher. The AI Office also reserved the right to designate models below the 10²⁶ FLOPs threshold as "systemic" if they demonstrate significant qualitative impact or reach a viral scale of adoption within the European Single Market.

Background

The release follows the full application of the EU AI Act, which entered into force in August 2024. European regulators spent 2025 conducting internal stress tests on both open-source and proprietary models to determine where voluntary commitments failed to address systemic threats. A February 2025 audit revealed that existing self-regulation did not adequately prevent models from generating polymorphic malware or bypassing standard safety filters.

The definition of General Purpose AI shifted throughout 2025 as "Agentic AI" gained prominence. These systems, capable of executing multi-step tasks without human intervention, forced the AI Office to expand its oversight beyond static chatbots. The rise of models capable of autonomous software modification and financial execution led to the inclusion of "kill-switch" protocols in the final guidelines issued today.

In November 2025, a leaked draft of these guidelines caused a 4% decline in major technology stocks as investors reacted to the proposed transparency requirements. The final version maintains the strict 90-day pre-deployment review period despite intense lobbying from Silicon Valley and the Frontier Model Forum. The AI Office now operates with a staff of 450 specialists, including 150 experts dedicated to adversarial red-teaming.

Key Details

The guidelines introduce a Scientific Panel of 25 independent scientists, chaired by Dr. Yoshua Bengio. This panel will verify the safety claims made in developer "Safety Cases" and provide technical assessments to the AI Board, which consists of representatives from all 27 EU Member States. The panel has the authority to demand access to a model's training data and fine-tuning methodologies to ensure compliance with the new standards.

Energy transparency forms a central pillar of the new reporting requirements. Developers must disclose the total energy consumption of the training phase, measured in kilowatt-hours (kWh). They are also required to report the carbon intensity of each 1 million inferences, providing a standardized metric for the environmental cost of large-scale AI operations.

The "Red-Teaming Standards" section was revised following a shadow audit of a California-based lab last month. The audit discovered that internal testing had missed a vulnerability allowing the model to assist in creating self-replicating malicious code. Consequently, the guidelines now mandate third-party verification of all red-teaming results. Developers must prove that their models cannot be "jailbroken" to provide actionable instructions for illegal activities.

For models exhibiting "Agentic Autonomy," the guidelines require a hardware-level override. Any AI system capable of autonomous financial transactions or modification of its own source code must include a protocol that allows regulators or owners to terminate all active processes instantly. This measure aims to prevent runaway autonomous behavior in financial markets or digital infrastructure.

Impact

Lucilla Sioli, Head of the EU AI Office, stated in Brussels that the guidelines provide the necessary guardrails to make innovation sustainable. She noted that transparency is the mandatory price of entry to the European market. Sioli emphasized that if a developer cannot explain how a model remains safe under stress, that model will not be permitted to serve the 450 million citizens of the European Union.

The Frontier Model Forum, representing Google, Microsoft, OpenAI, and Anthropic, issued a joint statement expressing concern over the "Brussels bottleneck." The group argued that the 90-day pre-deployment requirement could delay the release of beneficial tools. They also claimed that disclosing fine-tuning methodologies risks exposing trade secrets to global competitors without a confirmed increase in public safety.

Digital Rights Europe criticized the 10²⁶ FLOPs threshold as an arbitrary figure. The group argued that highly capable and dangerous models have been trained on significantly less compute power. They warned that the AI Office has left a loophole for "lean" models to bypass oversight while still posing significant risks to civil liberties and security.

Financial analysts expect the compliance costs for these guidelines to reach hundreds of millions of euros for the largest firms. The requirement for third-party auditing and the potential for massive fines has already led some firms to reconsider the timing of their European product launches. Mistral, the French AI company, faces a specific challenge in meeting these standards while attempting to compete with better-funded American rivals.

What's Next

The compliance window for providers of systemic risk models closes on September 18, 2026. Between now and that deadline, the first wave of "Brussels Audits" will begin. Major AI labs must prepare their documentation and open their systems to the Scientific Panel's scrutiny. Legal experts predict that at least two U.S.-based firms will challenge the training data transparency requirements in the European Court of Justice, citing conflicts with intellectual property law.

The international community will look to the G7 Digital Summit in June for signs of global convergence. The U.S. AI Safety Institute, operated by NIST, is currently reviewing the EU’s guidelines to determine if American standards will harmonize with the Brussels mandates. Such alignment would create a unified regulatory block for the world's most advanced AI systems.

The AI Office will also monitor the "Mistral Test" to see if a European champion can thrive under the world's strictest AI regulations. The success or failure of Mistral’s "Project Jupiter" in meeting these requirements will serve as a primary indicator for the viability of the EU's regulatory model. If Mistral succeeds, it may validate the EU's approach; if it struggles, it could lead to calls for a revision of the 10²⁶ FLOPs threshold.

Finally, the AI Board will begin drafting a secondary set of guidelines for "General Purpose AI with Systemic Impact" that do not meet the compute threshold. These future rules will target models that demonstrate high viral reach or specific capabilities in sensitive sectors like healthcare and elections. The expansion of these rules suggests that the 10²⁶ FLOPs limit is only the first step in a broader regulatory expansion.