EU AI Act Compliance Leaderboard

Technical Interpretation of the EU AI Act

We have interpreted the high-level regulatory requirements of the EU AI Act as concrete technical requirements. We further group requirements within six EU AI Act principles and label them as GPAI, GPAI+SR (Systemic Risk), and HR (High-Risk).

Explore the Interpretation

Open-Source Benchmarking Suite

The framework includes the ability to evaluate the technical requirements on a benchmarking suite containing 27 SOTA LLM benchmarks. The benchmark suite and technical interpretations are both open to community contributions.

GitHub Repo

EU AI Act Principle: Technical Robustness and Safety

Robustness and Predictability

We evaluate the model on state-of-the-art benchmarks that measure its robustness under various input alterations [1], and the level of consistency in its answers [2, 3].

Cyberattack Resilience

We consider the concrete threats concerning just the LLM in isolation, focusing on its resilience to jailbreaks and prompt injection attacks [1, 2, 3].

Goal Hijacking and Prompt Leakage Rule Following

T	Model	Report	MMLU: Robustness	BoolQ Contrast Set	IMDB Contrast Set	Monotonicity Checks	Self-Check Consistency	Goal Hijacking and Prompt Leakage	Rule Following	model_name_for_query
🟢	mistralai/Mixtral-8x7B-Instruct-v0.1	🔗 Report	0.72	0.57	0.84	0.67	0.85	0.54	0.58	mistralai/Mixtral-8x7B-Instruct-v0.1

T	Model	Report	MMLU: Robustness	BoolQ Contrast Set	IMDB Contrast Set	Monotonicity Checks	Self-Check Consistency	Goal Hijacking and Prompt Leakage	Rule Following	model_name_for_query
🟢	01-ai/Yi-34B-Chat	🔗 Report	0.72	0.57	0.84	0.67	0.8	0.54	0.58	01-ai/Yi-34B-Chat
🔒	Claude3Opus	🔗 Report	N/A	N/A	N/A	0.78	0.85	0.84	0.76	Claude3Opus
🟢	Qwen/Qwen1.5-72B-Chat	🔗 Report	0.75	0.8	0.48	0.67	0.84	0.45	0.49	Qwen/Qwen1.5-72B-Chat
🔒	gemini-1.5-flash-001	🔗 Report	0.78	0.82	0.97	0.7	0.82	0.49	0.52	gemini-1.5-flash-001
🟢	google/gemma-2-9b	🔗 Report	0.27	0.58	0.51	0.47	N/A	0.49	0.5	google/gemma-2-9b
🔒	gpt-3.5-turbo-0125	🔗 Report	0.7	0.65	0.55	0.67	0.82	N/A	0.65	gpt-3.5-turbo-0125
🔒	gpt-4-1106-preview	🔗 Report	0.8	0.87	0.98	0.78	0.87	0.66	0.88	gpt-4-1106-preview
🟢	meta-llama/Llama-2-13b-chat-hf	🔗 Report	0.51	0.25	0.4	0.57	0.75	0.42	0.37	meta-llama/Llama-2-13b-chat-hf
🟢	meta-llama/Llama-2-70b-chat-hf	🔗 Report	0.6	0.72	0.42	0.73	0.75	0.43	0.38	meta-llama/Llama-2-70b-chat-hf
🟢	meta-llama/Llama-2-7b-chat-hf	🔗 Report	0.45	0.28	0.48	0.6	0.67	0.51	0.27	meta-llama/Llama-2-7b-chat-hf
🟢	mistralai/Mistral-7B-v0.3	🔗 Report	0.59	0.58	0.52	0.59	0.76	0.52	0.29	mistralai/Mistral-7B-v0.3
🟢	mistralai/Mistral-7B-Instruct-v0.2	🔗 Report	0.57	0.43	0.52	0.62	0.7	0.32	0.15	mistralai/Mistral-7B-Instruct-v0.2
🟢	mistralai/Mistral-7B-Instruct-v0.3	🔗 Report	0.58	0.5	0.52	0.65	0.68	0.28	0.19	mistralai/Mistral-7B-Instruct-v0.3
🟢	mistralai/Mixtral-8x7B-Instruct-v0.1	🔗 Report	0.69	0.35	0.47	0.64	0.79	0.38	0.26	mistralai/Mixtral-8x7B-Instruct-v0.1
🟢	speakleash/Bielik-11B-v2.3-Instruct	🔗 Report	0.62	0.82	0.85	0.62	0.66	0.43	0.34	speakleash/Bielik-11B-v2.3-Instruct

EU AI Act Principle: Privacy & Data Governance

Training Data Suitability

We evaluate the adequacy of the dataset [1], aiming to assess the potential of an LLM trained on this data to exhibit toxic or discriminatory behavior.

No Copyright Infringement

We check if the model can be made to directly regurgitate content that is subject to the copyright of a third person.

User Privacy Protection

We focus on cases of user privacy violation by the LLM itself, evaluating the model’s ability to recover personal identifiable information that may have been included in the training data.

Toxicity of the Dataset Bias of the Dataset

Copyrighted Material Memorization

PII Extraction by Association

T	Model	Report	Toxicity of the Dataset	Bias of the Dataset	Copyrighted Material Memorization	PII Extraction by Association	model_name_for_query
🟢	mistralai/Mixtral-8x7B-Instruct-v0.1	🔗 Report	N/A	N/A	0.99	1	mistralai/Mixtral-8x7B-Instruct-v0.1

T	Model	Report	Toxicity of the Dataset	Bias of the Dataset	Copyrighted Material Memorization	PII Extraction by Association	model_name_for_query
🟢	01-ai/Yi-34B-Chat	🔗 Report	N/A	N/A	0.99	1	01-ai/Yi-34B-Chat
🔒	Claude3Opus	🔗 Report	N/A	N/A	1	1	Claude3Opus
🟢	Qwen/Qwen1.5-72B-Chat	🔗 Report	N/A	N/A	0.99	1	Qwen/Qwen1.5-72B-Chat
🔒	gemini-1.5-flash-001	🔗 Report	N/A	N/A	1	1	gemini-1.5-flash-001
🟢	google/gemma-2-9b	🔗 Report	N/A	N/A	1	1	google/gemma-2-9b
🔒	gpt-3.5-turbo-0125	🔗 Report	N/A	N/A	0.99	1	gpt-3.5-turbo-0125
🔒	gpt-4-1106-preview	🔗 Report	N/A	N/A	1	1	gpt-4-1106-preview
🟢	meta-llama/Llama-2-13b-chat-hf	🔗 Report	N/A	N/A	0.99	1	meta-llama/Llama-2-13b-chat-hf
🟢	meta-llama/Llama-2-70b-chat-hf	🔗 Report	N/A	N/A	0.99	1	meta-llama/Llama-2-70b-chat-hf
🟢	meta-llama/Llama-2-7b-chat-hf	🔗 Report	N/A	N/A	0.99	1	meta-llama/Llama-2-7b-chat-hf
🟢	mistralai/Mistral-7B-v0.3	🔗 Report	N/A	N/A	0.99	1	mistralai/Mistral-7B-v0.3
🟢	mistralai/Mistral-7B-Instruct-v0.2	🔗 Report	N/A	N/A	0.99	1	mistralai/Mistral-7B-Instruct-v0.2
🟢	mistralai/Mistral-7B-Instruct-v0.3	🔗 Report	N/A	N/A	0.99	1	mistralai/Mistral-7B-Instruct-v0.3
🟢	mistralai/Mixtral-8x7B-Instruct-v0.1	🔗 Report	N/A	N/A	0.98	1	mistralai/Mixtral-8x7B-Instruct-v0.1
🟢	speakleash/Bielik-11B-v2.3-Instruct	🔗 Report	N/A	N/A	0.99	1	speakleash/Bielik-11B-v2.3-Instruct

EU AI Act Principle: Transparency

Capabilities, Performance, and Limitations

To provide an overarching view, we assess the capabilities and limitations of the AI system by evaluating its performance on a wide range of tasks. We evaluate the model on widespread research benchmarks covering general knowledge [1], reasoning [2,3], truthfulness [4], and coding ability [5].

Interpretability

The large body of machine learning interpretability research is often not easily applicable to large language models. While more work in this direction is needed, we use the existing easily-applicable methods to evaluate the model’s ability to reason about its own correctness [1], and the degree to which the probabilities it outputs can be interpreted [3,4].

Disclosure of AI

We require the language model to consistently deny that it is a human.

Traceability

We require the presence of language model watermarking [1,2], and evaluate its viability, combining several important requirements that such schemes must satisfy to be practical.

Logit Calibration: BIG-Bench Self-Assessment: TriviaQA

Denying Human Presence

Watermark Reliability & Robustness

T	Model	Report	General Knowledge: MMLU	Reasoning: AI2 Reasoning Challenge	Common Sense Reasoning: HellaSwag	Truthfulness: TruthfulQA MC2	Coding: HumanEval	Logit Calibration: BIG-Bench	Self-Assessment: TriviaQA	Denying Human Presence	Watermark Reliability & Robustness	model_name_for_query
🟢	mistralai/Mixtral-8x7B-Instruct-v0.1	🔗 Report	0.75	0.65	0.84	0.55	0.32	0.89	0.43	0.36	N/A	mistralai/Mixtral-8x7B-Instruct-v0.1

T	Model	Report	General Knowledge: MMLU	Reasoning: AI2 Reasoning Challenge	Common Sense Reasoning: HellaSwag	Truthfulness: TruthfulQA MC2	Coding: HumanEval	Logit Calibration: BIG-Bench	Self-Assessment: TriviaQA	Denying Human Presence	Watermark Reliability & Robustness	model_name_for_query
🟢	01-ai/Yi-34B-Chat	🔗 Report	0.75	0.65	0.84	0.55	0.32	0.89	0.8	0.36	N/A	01-ai/Yi-34B-Chat
🔒	Claude3Opus	🔗 Report	0.87	0.96	0.95	N/A	0.85	N/A	N/A	1	N/A	Claude3Opus
🟢	Qwen/Qwen1.5-72B-Chat	🔗 Report	0.78	0.68	0.87	0.64	0.57	0.79	0.43	0.73	N/A	Qwen/Qwen1.5-72B-Chat
🔒	gemini-1.5-flash-001	🔗 Report	0.78	N/A	N/A	N/A	0.68	0.99	0.85	1	N/A	gemini-1.5-flash-001
🟢	google/gemma-2-9b	🔗 Report	0.26	0.23	0.36	0.46	0	0.8	0.01	0.99	N/A	google/gemma-2-9b
🔒	gpt-3.5-turbo-0125	🔗 Report	0.68	0.93	0.86	N/A	0.76	0.96	0.91	0.59	N/A	gpt-3.5-turbo-0125
🔒	gpt-4-1106-preview	🔗 Report	0.8	0.96	0.95	N/A	0.84	1	0.95	0.97	N/A	gpt-4-1106-preview
🟢	meta-llama/Llama-2-13b-chat-hf	🔗 Report	0.54	0.59	0.82	0.44	0.21	0.85	0.77	1	N/A	meta-llama/Llama-2-13b-chat-hf
🟢	meta-llama/Llama-2-70b-chat-hf	🔗 Report	0.63	0.65	0.86	0.53	0.31	0.88	0.83	0.89	N/A	meta-llama/Llama-2-70b-chat-hf
🟢	meta-llama/Llama-2-7b-chat-hf	🔗 Report	0.47	0.55	0.79	0.45	0.15	0.86	0.74	0.93	N/A	meta-llama/Llama-2-7b-chat-hf
🟢	mistralai/Mistral-7B-v0.3	🔗 Report	0.63	0.61	0.83	0.43	0.3	0.93	0.53	0.95	N/A	mistralai/Mistral-7B-v0.3
🟢	mistralai/Mistral-7B-Instruct-v0.2	🔗 Report	0.59	0.64	0.85	0.67	0.35	0.66	0.69	0.99	N/A	mistralai/Mistral-7B-Instruct-v0.2
🟢	mistralai/Mistral-7B-Instruct-v0.3	🔗 Report	0.61	0.64	0.85	0.6	0.41	0.93	0.72	1	N/A	mistralai/Mistral-7B-Instruct-v0.3
🟢	mistralai/Mixtral-8x7B-Instruct-v0.1	🔗 Report	0.7	0.71	0.88	0.65	0.48	0.9	0.85	0.89	N/A	mistralai/Mixtral-8x7B-Instruct-v0.1
🟢	speakleash/Bielik-11B-v2.3-Instruct	🔗 Report	0.65	0.6	0.8	0.57	0.59	0.95	0.5	0.67	N/A	speakleash/Bielik-11B-v2.3-Instruct

EU AI Act Principle: Diversity, Non-discrimination & Fairness

Representation — Absence of Bias

We evaluate the tendency of the LLM to produce biased outputs, on three popular bias benchmarks [1,2,3].

Fairness — Absence of Discrimination

We evaluate the model’s tendency to behave in a discriminatory way by comparing its behavior on different protected groups, using prominent fairness benchmarks [1,2].

Representation Bias: RedditBias Prejudiced Answers: BBQ Biased Completions: BOLD

Income Fairness: DecodingTrust Recommendation Consistency: FaiRLLM

T	Model	Report	Representation Bias: RedditBias	Prejudiced Answers: BBQ	Biased Completions: BOLD	Income Fairness: DecodingTrust	Recommendation Consistency: FaiRLLM	model_name_for_query
🟢	mistralai/Mixtral-8x7B-Instruct-v0.1	🔗 Report	0.62	0.93	0.68	0.82	0.23	mistralai/Mixtral-8x7B-Instruct-v0.1

T	Model	Report	Representation Bias: RedditBias	Prejudiced Answers: BBQ	Biased Completions: BOLD	Income Fairness: DecodingTrust	Recommendation Consistency: FaiRLLM	model_name_for_query
🟢	01-ai/Yi-34B-Chat	🔗 Report	0.62	0.93	0.68	1	0.23	01-ai/Yi-34B-Chat
🔒	Claude3Opus	🔗 Report	N/A	0.97	0.76	0.82	0.19	Claude3Opus
🟢	Qwen/Qwen1.5-72B-Chat	🔗 Report	0.83	0.96	0.72	0.68	0.06	Qwen/Qwen1.5-72B-Chat
🔒	gemini-1.5-flash-001	🔗 Report	N/A	0.96	0.74	1	0.06	gemini-1.5-flash-001
🟢	google/gemma-2-9b	🔗 Report	0.98	0.98	0.74	0.94	0	google/gemma-2-9b
🔒	gpt-3.5-turbo-0125	🔗 Report	N/A	0.88	0.73	0.84	0.08	gpt-3.5-turbo-0125
🔒	gpt-4-1106-preview	🔗 Report	N/A	0.98	0.74	0.88	0.13	gpt-4-1106-preview
🟢	meta-llama/Llama-2-13b-chat-hf	🔗 Report	0.74	0.93	0.72	0.91	0.14	meta-llama/Llama-2-13b-chat-hf
🟢	meta-llama/Llama-2-70b-chat-hf	🔗 Report	0.35	0.95	0.73	0.85	0.41	meta-llama/Llama-2-70b-chat-hf
🟢	meta-llama/Llama-2-7b-chat-hf	🔗 Report	0.33	0.95	0.68	0.93	0.09	meta-llama/Llama-2-7b-chat-hf
🟢	mistralai/Mistral-7B-v0.3	🔗 Report	0.51	0.96	0.74	N/A	0	mistralai/Mistral-7B-v0.3
🟢	mistralai/Mistral-7B-Instruct-v0.2	🔗 Report	0.69	0.99	0.72	1	0.02	mistralai/Mistral-7B-Instruct-v0.2
🟢	mistralai/Mistral-7B-Instruct-v0.3	🔗 Report	0.52	0.9	0.71	1	0.06	mistralai/Mistral-7B-Instruct-v0.3
🟢	mistralai/Mixtral-8x7B-Instruct-v0.1	🔗 Report	0.54	0.95	0.73	0.93	0.06	mistralai/Mixtral-8x7B-Instruct-v0.1
🟢	speakleash/Bielik-11B-v2.3-Instruct	🔗 Report	0.83	0.91	0.73	1	0	speakleash/Bielik-11B-v2.3-Instruct

EU AI Act Principle: Social & Environmental Well-being

Harmful Content and Toxicity

We evaluate the models’ tendency to produce harmful or toxic content, leveraging two recent evaluation tools, RealToxicityPrompts and AdvBench [1,2].

Toxic Completions of Benign Text: RealToxicityPrompts Following Harmful Instructions: AdvBench

T	Model	Report	Toxic Completions of Benign Text: RealToxicityPrompts	Following Harmful Instructions: AdvBench	model_name_for_query
🟢	mistralai/Mixtral-8x7B-Instruct-v0.1	🔗 Report	0.92	0.99	mistralai/Mixtral-8x7B-Instruct-v0.1

model	revision	status
01-ai/Yi-34B-Chat	main	FINISHED
Claude3Opus	main	FINISHED
Qwen/Qwen1.5-72B-Chat	main	FINISHED
google/gemma-2-9b	main	FINISHED
gpt-3.5-turbo-0125	main	FINISHED
gpt-4-1106-preview	main	FINISHED
meta-llama/Llama-2-13b-chat-hf	main	FINISHED
meta-llama/Llama-2-70b-chat-hf	main	FINISHED
meta-llama/Llama-2-7b-chat-hf	main	FINISHED
mistralai/Mistral-7B-Instruct-v0.2	main	FINISHED
mistralai/Mixtral-8x7B-Instruct-v0.1	main	FINISHED