Red teaming
Deliberately probing an AI system for failures, misuse, and bypasses before or after launch, structured adversarial testing, not casual chatting.
Red teaming (for LLM apps) means organized attempts to break safeguards: jailbreaks, data leaks, unsafe tool calls, biased outputs, or policy evasion. It borrows the security idea of a “red team” that attacks so defenders can fix gaps.
Effective red teaming combines creative prompts, automation, and domain experts. Findings feed guardrails, retraining, and broader alignment processes, especially when models gain tools or access private data via RAG.