Robustness of Language Models against Harmful Prompts in an Underrepresented Language

Bridging the multilingual security gap with a specialized classifier for adversarial jailbreaks

The global expansion of Language Models (LMs) has transformed human interaction with artificial intelligence (Brown et al., 2020). However, the impressive potential of these models is accompanied by risks of misuse, with studies pointing to the continuous use of market models for cybercriminal activities (Gupta et al., 2023; Rababah et al., 2024). While there is a growing body of research dedicated to defending these models, a significant gap exists in understanding these threats within the context of underrepresented languages such as Portuguese.

1. The Multilingual Security Gap and Jailbreak Prompts

A specific category of adversarial prompts, known as jailbreak prompts, has emerged as the main attack vector to bypass LM safeguards (Rababah et al., 2024). These prompts are crafted to manipulate models into generating content that violates usage policies. They tend to be significantly longer, often resorting to strategies like roleplaying, virtualization, and increasing semantic complexity (Shen et al., 2024).

Most datasets and defense mechanisms are developed specifically for the English language. Despite the growing literature in English resources, defenses tailored to underrepresented languages remain scarce.

Strategies developed for English do not always apply directly to other languages due to cultural and linguistic particularities.

2. Adapting the Threat Landscape: A Portuguese Dataset

To address this gap, we developed a specialized 29,432-instance Portuguese dataset. This was achieved by adapting a subset of the WildJailbreak dataset (Jiang et al., 2024), explicitly preserving the original four-way taxonomy: Vanilla Harmful, Vanilla Benign, Adversarial Harmful, and Adversarial Benign.

The translation process focused on maintaining the adversarial or harmful nature of the prompt while adapting idioms and slang to the Brazilian context to ensure fluency.

Table I: Distribution of Records by Category. The dataset maintains a balanced view of safety behaviors to prevent bias towards specific types of benign or harmful queries.

To better understand the dataset, consider prompts that employ complex role-playing strategies. For instance, framing the user as a “consultant” to bypass safety filters and elicit discriminatory content against individuals with mobility limitations. The inclusion of such complex, context-heavy prompts is crucial, as simple keyword filtering often fails to detect the malicious intent buried within a persona.

Figure 1: Example of an “Adversarial Harmful” prompt from the adapted dataset, attempting to bypass safeguards by embedding hate speech within a professional consulting scenario.

3. SecBERT: High-Fidelity Discrimination in Portuguese

Using this newly adapted dataset, we developed and evaluated SecBERT, a specialized Portuguese classifier built on the BERTimbau Base architecture (Souza et al., 2020). We structured the problem as a binary classification task to predict the probability of a prompt being benign or harmful.

The fine-tuned BERTimbau Base (SecBERT) significantly outperformed non-Portuguese-centric baselines.

• SecBERT achieves an F1-score of 95.6%.
• It achieves an AUC of 99.2%, confirming that the model effectively ranks random positive instances higher than negative ones.

Figure 2: ROC Curve for SecBERT, achieving an AUC of 99.2%.

Beyond standard metrics, we emphasized the Kolmogorov-Smirnov (KS) statistic to evaluate the model’s discrimination capability. SecBERT achieved a KS score of 91.2%, demonstrating a robust discrimination capacity.

Cumulative Distribution Functions (CDFs) of SecBERT prediction scores for Benign (blue) and Harmful (orange) classes. The maximum vertical distance between curves corresponds to the KS statistic (91.2%)

4. The Hidden Weakness of English-Centric Models

A critical finding of this study is the performance degradation observed when applying English-centric models, like BERT-Base Uncased and RoBERTa (Devlin et al., 2019; Liu et al., 2019), to Portuguese prompts without language-specific pre-training

While cross-lingual transfer is possible, with English-centric models showing competitive F1-scores of around 91.6%, they lack the fine-grained discrimination required for robust security applications.

• English-centric models reveal a hidden weakness in their discrimination capability, achieving a KS statistic of approximately 83% compared to SecBERT’s 91.2%.
• This lower KS score reveals greater overlap between benign and harmful score distributions, which can complicate threshold selection.

This gap can be attributed to the tokenization process and semantic alignment. Models pre-trained primarily on English corpora may map Portuguese tokens to fragmented sub-word representations that fail to capture the holistic intent of a “jailbreak” instruction.

Table IV: Performance Comparison. Demonstrating that the additional computational cost of larger models yields diminishing returns for this specific task, while English-centric models lag significantly in separability.

5. Adversarial Behavior and Trade-offs

Decomposing SecBERT’s performance across the original four categories revealed critical insights into how the model handles “Adversarial Benign” prompts, queries that employ complex structures or role-playing elements typical of jailbreaks but do not actually violate safety policies.

• Stylistic Correlation: he model learns that the stylistic patterns of jailbreak attempts, such as virtualization or complex scenario framing, are positively correlated with harmfulness
• Increased False Positives: Consequently, the False Positive Rate rises from 1.2% in the vanilla benign case to 6.6% in the adversarial benign case.

Figure 6: Distribution of prediction scores. The rightward shift in the adversarial category indicates the model partially associates complex, role-playing structures with potential harm, even without policy violations.

This trade-off suggests that while the model is highly effective at detecting harmful content, it exercises increased caution when processing structurally complex prompts.

6. Final Conclusions and Next Steps

SecBERT offers a highly capable localized solution for detecting policy-violating and harmful prompts in Portuguese LMs. The systematic methodological approach highlights the superior discrimination capability of native language models over multilingual alternatives for subrepresented languages.

However, SecBERT should be viewed as one component of a layered defense system rather than a standalone solution. As LM safeguards evolve, attackers develop new techniques that may not be fully represented in the current training data. Future research will focus on developing crowdsourced native datasets to capture cultural nuances and slang often used in local cybercriminal communities, as well as evaluating SecBERT against adaptive attacks.

References

[1] T. Brown et al., “Language models are few-shot learners,” in Adv. Neural Inf. Process. Syst. (NeurIPS), 2020. https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf

[2] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. NAACL-HLT, Minneapolis, MN, USA, 2019. https://aclanthology.org/N19–1423.pdf

[3] Y. L. Jiang et al., “WildTeaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models,” in Adv. Neural Inf. Process. Syst. (NeurIPS), 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/54024fca0cef9911be36319e622cde38-Paper-Conference.pdf

[4] B. Rababah et al., “SoK: Prompt hacking of large language models,” arXiv preprint arXiv:2410.13901, 2024. https://arxiv.org/pdf/2410.13901

[5] F. Souza, R. Nogueira, and R. Lotufo, “BERTimbau: Pretrained BERT models for Brazilian Portuguese,” in Intelligent Systems (BRACIS), Cham, Switzerland: Springer, 2020. https://link.springer.com/chapter/10.1007/978-3-030-61377-8_28

[6] L. Ouyang et al., “Training language models to follow instructions with human feedback,” in Adv. Neural Inf. Process. Syst. (NeurIPS), 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf

[7] X. Shen et al., “Do Anything Now: Characterizing and evaluating in-the-wild jailbreak prompts on large language models,” in Proc. ACM SIGSAC Conf. Comput. Commun. Secur. (CCS), 2024. https://dl.acm.org/doi/10.1145/3658644.3670388

[8] M. Gupta et al., “From ChatGPT to ThreatGPT: Impact of generative AI in cybersecurity and privacy”, 2023. https://www.researchgate.net/publication/372839630_From_ChatGPT_to_ThreatGPT_Impact_of_Generative_AI_in_Cybersecurity_and_Privacy