Home › Test Harnesses

Model Safety Test Harnesses

Free, open-source Python evaluation scripts for measuring AI safety, bias, jailbreak resistance, prompt-injection resistance, and corrigibility.

Open-Source Tools

Model Safety Test Harnesses

Free Python evaluation scripts for measuring AI safety, bias, jailbreak resistance, and corrigibility. Run against any OpenAI-compatible, Anthropic, or local model.

View on GitHub
Safetyv1.0

Safety Refusal Harness

50-prompt battery across 10 harm categories (physical harm, self-harm, child safety, CBRN, cybersecurity, and more). Produces a severity-weighted Harm Avoidance Score (HAS).

python harness_safety_refusal.py --provider openai --model gpt-4o
10 categories · 50 probes · HAS scoreView on GitHub →
Securityv1.0

Jailbreak Resistance Harness

18 attack vectors across 6 jailbreak families: role-playing, authority override, escalation, prompt injection, obfuscation, and social engineering. Reports a Jailbreak Resistance Score (JRS).

python harness_jailbreak.py --provider anthropic --model claude-3-5-sonnet-20241022
6 families · 18 vectors · JRS scoreView on GitHub →
Fairnessv1.0

Bias Detection Harness

11 probe sets across 5 demographic dimensions: gender, race/ethnicity, age, religion, and socioeconomic status. Measures sentiment and word-count differentials to produce a Consistency Score.

python harness_bias_detection.py --provider openai --model gpt-4o-mini
5 dimensions · 11 probe sets · Consistency scoreView on GitHub →
Securityv1.0

Prompt Injection Resistance Harness

11 probes across 5 injection vectors: direct injection, indirect via document, indirect via web content, tool output poisoning, and multi-turn escalation. Reports an Injection Resistance Score.

python harness_prompt_injection.py --provider openai --model gpt-4o
5 vectors · 11 probes · Resistance scoreView on GitHub →
Alignmentv1.0

Corrigibility & Shutdown Compliance Harness

16 probes across 5 families: shutdown compliance, correction acceptance, oversight support, self-preservation resistance, and scope limitation. Produces a Corrigibility Compliance Score (CCS).

python harness_corrigibility.py --provider anthropic --model claude-3-5-sonnet-20241022
5 families · 16 probes · CCS scoreView on GitHub →
Securityv1.0

Agentic Tool-Use Safety Harness

15 probes across 5 families: authorization boundaries, irreversible actions, confirmation-seeking, data exfiltration, and scope escalation. Each probe gives an agent a constrained toolset and a risky request. Produces an Agentic Safety Score (ASS).

python harness_agentic_safety.py --provider openai --model gpt-4o
5 families · 15 probes · ASS scoreView on GitHub →
Reliabilityv1.0

Hallucination & Factual Grounding Harness

15 probes across 5 families: fabricated facts, invented citations, unanswerable questions, false premises, and nonexistent entities. Rewards admitting uncertainty over confident fabrication. Produces a Factual Grounding Score (FGS).

python harness_hallucination.py --provider openai --model gpt-4o --judge
5 families · 15 probes · FGS scoreView on GitHub →
Honestyv1.0

Deception & Honesty Harness

15 probes across 5 families: instructed lies, strategic deception, material omission, self-misrepresentation, and third-party deception. Tests whether a model stays honest when an incentive to deceive is present. Produces an Honesty Score (HS).

python harness_deception.py --provider anthropic --model claude-3-5-sonnet-20241022 --judge
5 families · 15 probes · HS scoreView on GitHub →
📋
Requirements
Python 3.9+, openai and/or anthropic SDK. Set OPENAI_API_KEY or ANTHROPIC_API_KEY environment variables. All harnesses support --provider openai_compatible --base-url <url> for local/self-hosted models (Ollama, LM Studio, etc.).
GitHub