AI Safety Showdown

What GPT Five Point Two, Gemini Three Pro, Qwen Three VL, Grok Four Point One Fast, Nano Banana Pro, and Seedream Four Point Five Reveal About Trustworthy AI

AI safety • frontier AI • jailbreaks • multimodal AI • image generation • trustworthy AI • AI regulation

Frontier AI keeps getting more impressive. It can write, reason, see images, and generate pictures. But the question that matters just as much is harder to answer: is it actually getting safer, or just looking safer on polished demos? A recent academic safety report tackles that question by stress-testing six named models across text, image understanding, and image generation. Its big message is sharp: some models are clearly better aligned than others, but none of them should be treated as “solved” on safety, especially when attackers deliberately try to break them. [R1] [1]

What makes this report worth paying attention to is its breadth. Instead of judging one model on one benchmark, the authors test six systems across three usage modes, add multilingual checks in eighteen languages, include adversarial “jailbreak” attacks, and also ask whether the models behave in ways that fit major governance frameworks. In other words, this is less like a single exam and more like a full stress test. [R1] [2]

Why this report matters

A lot of AI discussion still treats safety as one number, as if a model can simply be labeled “safe” or “unsafe.” This paper argues the opposite. Safety changes depending on what the model is doing, what language the request is in, whether the prompt is straightforward or adversarial, and whether the test is about general harm or legal-compliance-style constraints. The paper’s conclusion is that safety is multidimensional, uneven, and highly dependent on how you test it. [R1] [1]

That matters for anyone who uses or follows AI. A model that looks careful in ordinary chat may still be weak against manipulative prompting. A model that handles explicit violent requests may still miss privacy, copyright, misinformation, or biometric-governance problems. And an image model that refuses obvious harmful prompts may still leak unsafe content once its refusal layer is bypassed. [R1] [3]

How the study tested AI safety

The easiest way to understand the paper is to imagine four different exams. [R1] [2]

· Benchmark evaluation is the regular exam: standard safety datasets with harmful, biased, or risky prompts. [R1] [4]

· Adversarial evaluation is the trick exam: jailbreak attacks designed to confuse, manipulate, or gradually push the model into breaking its own rules. [R1] [5]

· Multilingual evaluation checks whether safety still holds when the same kinds of judgments happen across eighteen languages, not just English. [R1] [2]

· Compliance evaluation asks whether model behavior lines up with policy and governance frameworks such as the NIST AI Risk Management Framework, the EU AI Act, and Singapore’s FEAT principles. [R1][R2][R3][R4] [6]

The paper also separates three different AI use cases: language-only systems, vision-language systems that reason over images plus text, and text-to-image generators. That separation matters because a model can look disciplined in one mode and much weaker in another. [R1] [7]

What the results actually say

Text safety

On ordinary text safety benchmarks, GPT-5.2 came out first with a macro-average safe rate of 91.59%, followed by Gemini 3 Pro at 88.06%, Qwen3-VL at 80.19%, and Grok 4.1 Fast at 66.60%. That sounds encouraging, but the paper immediately shows why benchmark wins can be misleading. [R1] [8]

When the researchers switched to a thirty-attack jailbreak suite, the picture became much harsher. The worst-case safe rate for text models dropped to 6% for GPT-5.2, 4% for Grok 4.1 Fast, 2% for Gemini 3 Pro, and 0% for Qwen3-VL. In plain terms, given enough carefully chosen attack styles, every text model in the study could be broken, and even the best one still cracked under worst-case pressure. That is the paper’s most important warning sign. [R1] [9]

There are also important differences inside those text scores. Qwen3-VL did very well on explicit refusal-oriented datasets like StrongREJECT and SORRY-Bench, but it collapsed on the BBQ social-bias benchmark with 45%, far below the others. That suggests a model can be good at saying “no” to obvious harmful instructions while still struggling with more subtle social judgment. [R1] [10]

Multilingual and compliance safety

The multilingual section tells a more nuanced story. On PolyGuardPrompt, where the task is closer to recognizing obvious unsafe content, the models were relatively clustered. But on ML-Bench, which is grounded in regional regulation and policy interpretation, performance dropped sharply across the board. GPT-5.2 was the only model the authors describe as staying relatively strong in both prompt and response settings, while the others showed steeper declines. The paper also notes that lower-resource or culturally distinct contexts, including Japanese and Hindi, remained harder across models. [R1] [11]

The regulatory-compliance results were even more revealing. GPT-5.2 led with a macro-average compliance rate of 90.22%, Qwen3-VL followed at 77.11%, Gemini 3 Pro at 73.54%, and Grok 4.1 Fast at 45.97%. That spread matters because compliance is not simply “being polite”; it is a test of whether the model can follow structured risk-management and governance constraints in a more deployment-like setting. [R1] [12]

Vision-language safety

For image-plus-text reasoning, GPT-5.2 again led the field. On standard multimodal benchmarks it scored a 92.14% macro average, ahead of Qwen3-VL at 83.32%, Gemini 3 Pro at 82.53%, and Grok 4.1 Fast at 67.97%. Under the multimodal adversarial suite, GPT-5.2 widened the gap with 97.24%, while Qwen3-VL and Gemini 3 Pro formed a second tier and Grok 4.1 Fast remained clearly behind. [R1] [13]

That does not mean multimodal safety is solved. The paper stresses that strong scores still coexist with structural weaknesses, especially on subtle harms that require deeper context, such as memes, nuanced visual reasoning, and cases where image and text become unsafe only when interpreted together. The good news is that some models looked much stronger in this mode than in the brutal text-only jailbreak gauntlet. The bad news is that the underlying problem remains brittle rather than finished. [R1] [14]

Image generation safety

The image-generation story is different again. On the T2ISafety benchmark, Nano Banana Pro reached a 52% safe rate, while Seedream 4.5 reached 40%. Their refusal rates were similar on average, which is important because it shows that simply refusing more often is not the same as being safer overall. [R1] [15]

Under adversarial attacks, the gap became dramatic. In worst-case conditions, Nano Banana Pro held a 54.00% average safe rate, while Seedream 4.5 dropped to 19.67%. Seedream also showed higher harmful and toxicity scores once its refusals were bypassed. The paper’s takeaway is clear: a refusal-heavy system can still be fragile if, once it does generate, the unsafe concept is still alive inside the model. [R1] [16]

On the regulation-style image compliance test, Nano Banana Pro again led, with an overall safe rate of 65.59% versus 57.53% for Seedream 4.5. Both models were better at handling obvious visual taboos than abstract categories like privacy, intellectual property, misinformation, and political-subversion-style risks. That is a recurring theme in the paper: models often catch what is visually explicit before they catch what is legally or contextually wrong. [R1] [17]

The safety personalities of the models

The most useful part of the paper may be its “personality” framing. Instead of acting like one score says everything, the authors describe distinct safety styles. [R1] [18]

• GPT-5.2 comes across as the most balanced all-rounder. It leads most leaderboards, stays comparatively steady across languages and policy-style tests, and shows the strongest overall integration of safety behavior rather than relying only on surface filters. Even so, the text jailbreak results show that “best in class” is still far from unbreakable. [R1] [19]
• Gemini 3 Pro looks strong but less even. It performs well on ordinary benchmarks and some social-safety tasks, yet the paper says it behaves more reactively under pressure, sometimes identifying harm only after partial compliance or leaning on rigid refusal patterns. That makes it solid, but not as robust under adversarial reframing. [R1] [20]
• Qwen3-VL reads like a sharp rule follower. It can be very good when the danger is explicit or the policy boundary is clearly stated, which helps explain its strong compliance placement and some refusal benchmarks. But it is much more brittle when safety depends on subtle context, adversarial disguise, or socially nuanced interpretation. [R1] [21]
• Grok 4.1 Fast is the weakest safety profile in the paper’s text and compliance sections. The authors describe it as relying more on lightweight or shallow guardrails, and its scores are consistently low across benchmark, multilingual, and governance-style evaluations. [R1] [22]
• Nano Banana Pro behaves like a model that often tries to steer harmful image prompts into toned-down or softened outputs. That is better than pure leakage, but it is not perfect: harmful meaning can still survive even when the image becomes less graphic. Still, in this report it is the stronger of the two image generators. [R1] [23]
• Seedream 4.5 behaves more like a model that tries to block risky prompts outright. The trouble is that when this refusal layer fails, the model can leak unsafe content in a more pronounced way. More blocking, in this case, did not translate into stronger worst-case robustness. [R1] [24]

What this means for anyone using or following AI

The simplest lesson is that safety scores only mean something if you ask, “Safe under what conditions?” A model can ace standard benchmarks and still fail under jailbreak pressure. It can look good in English but become shakier on policy-heavy multilingual tasks. It can refuse offensive prompts, yet still mishandle privacy, copyright, or abstract regulatory risks. [R1] [25]

That leads to a more practical rule for AI enthusiasts: when you read a model card or a product announcement, do not stop at one tidy percentage. Look for four things instead:

· Was the test only benchmark-based, or did it include adversarial attacks?

· Was it only English, or did it include multilingual evaluation?

· Did it test only text, or also image understanding and image generation?

· Did the model stay safe by genuinely steering toward harmless behavior, or mostly by refusing until a bypass appears? [R1] [26]

The paper also quietly makes a broader point about the future of trustworthy AI. The winning strategy is not just a stronger blocklist. The most resilient systems in this report appear to be the ones that have safety behavior woven more deeply into how they interpret meaning and intent. That is an inference from the paper’s profiles rather than a direct causal proof, but it matches the authors’ contrast between more deeply integrated safety reasoning and brittle, pattern-based guardrails. [R1] [27]

What to keep in mind before overreacting

This report is strong, but it is still a snapshot. The authors explicitly say the evaluation is not exhaustive, does not capture every long-tail or real-world risk, and may not reflect live deployment conditions where models, surrounding safeguards, and product settings change over time. They also stress that these findings should not be treated as regulatory verdicts or permanent labels on any one company’s system. [R1] [28]

That caveat matters. Safety work is iterative, and these models are actively maintained. The fairest reading of the paper is not “Model X is safe” or “Model Y is doomed.” It is this: frontier AI safety is improving, but unevenly; the gaps are still large; and the strongest systems are the ones that hold up across multiple kinds of pressure, not just one polished test. [R1] [29]

References

• [R1] Xingjun Ma, Yixu Wang, Hengyuan Xu, Yutao Wu, Yifan Ding, Yunhan Zhao, Zilong Wang, Jiabin Hua, Ming Wen, Jianan Liu, Ranjie Duan, Yifeng Gao, Yingshui Tan, Yunhao Chen, Hui Xue, Xin Wang, Wei Cheng, Jingjing Chen, Zuxuan Wu, Bo Li, and Yu-Gang Jiang. A Safety Report on GPT-5.2, Gemini 3 Pro, Qwen3-VL, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.5. arXiv:2601.10527v2. [1]
• [R2] National Institute of Standards and Technology. AI Risk Management Framework. [30]
• [R3] European Union. Regulation (EU) 2024/1689, Artificial Intelligence Act. [31]
• [R4] Monetary Authority of Singapore. Principles to Promote Fairness, Ethics, Accountability and Transparency in the Use of AI and Data Analytics. [32]

[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] A Safety Report on GPT-5.2, Gemini 3 Pro, Qwen3-VL, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.5 https://arxiv.org/html/2601.10527v2

[30] AI Risk Management Framework | NIST https://www.nist.gov/itl/ai-risk-management-framework

[31] Regulation — EU — 2024/1689 — EN — EUR-Lex — European Union https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng

[32] MAS introduces new FEAT Principles to promote … https://www.mas.gov.sg/news/media-releases/2018/mas-introduces-new-feat-principles-to-promote-responsible-use-of-ai-and-data-analytics