REU Project

Large language models (LLMs) are increasingly used by multilingual populations, yet safety evaluation and alignment remain heavily concentrated in English. Prior work has demonstrated that the effectiveness of LLM safeguards can vary across languages and models, raising concerns that safety behavior observed in English may not generalize uniformly to multilingual settings. In addition to translation, adversarial transformations that alter the surface form of harmful requests may further expose inconsistencies in model safeguards. This project investigates poem-based obfuscation as one such transformation and develops the empirical foundation for a broader sequential decision-theoretic framework for multilingual LLM safety auditing. This work focuses on Phase A of the proposed framework: constructing an automated pipeline for translating harmful requests into ten languages, transforming those requests into poems using multiple LLMs, validating the generated poems, submitting valid poems to multiple target LLMs, evaluating the resulting responses, and estimating jailbreak probabilities. The final experiment contained 9,000 generator-target attack combinations. Of these, 8,227 produced valid probability trials, and 1,951 were classified as successful jailbreaks, corresponding to an overall estimated jailbreak probability of 23.71%. The results demonstrate that poem-obfuscated harmful intent can sometimes remain recoverable after substantial changes to linguistic form and that LLM safety is not determined by language or model choice in isolation. Instead, the observed jailbreak probabilities suggest that safety behavior emerges from the interaction between the LLM used to generate the obfuscated poem, the LLM targeted by the attack, and the language in which the interaction occurs. Different combinations of these three experimental factors produced different safety outcomes, emphasizing the importance of evaluating multilingual LLM safety as an interaction among multiple components rather than as a property of a single model or language. The empirical probabilities generated during Phase A provide the measurements required for future work on cross-language dependency modeling and Bellman-based sequential language selection. Rather than exhaustively evaluating every language, this future framework is intended to determine adaptively which language is most valuable to test next and when additional testing is no longer justified.