Islamify.ai
94.6%
Baseline
Islamic Massive Language Understanding
IslamicMMLU Leaderboard
Benchmarking language models on Islamic jurisprudence, Quranic structure, and Hadith scholarship.
IslamicMMLU evaluates frontier and specialist models on 10,013 paper-backed multiple-choice questions across Fiqh, Quran, and Hadith.
This leaderboard measures benchmark accuracy, not scholarly authority.Models should not be treated as sources of fatwa or unsupervised religious guidance.
10,013
Total questions
Canonical benchmark target
4,000
Fiqh
Comparative jurisprudence
2,013
Quran
Structural Quranic knowledge
4,000
Hadith
Source, chapter, and grading tasks
{}
Public model results
Rankings use the simple average of available track accuracies. The current benchmark target is 4,000 Fiqh, 2,013 Quran, and 4,000 Hadith questions.
Overall score = average of available track scores
Madhab bias = the dominant school in the model's bias-detection selections
Partial evaluations remain visible instead of being hidden
Click a model row to inspect track, difficulty, and madhab-bias detail.
Leaderboard leaders
Top public results at a glance
ansari-3.5
94.2%
Baseline
Madhab balance: 93/100
Ansari 3.5 using RAG + Gemini 3.5 Flash
94.1%
Baseline
Madhab balance: 87/100
Ranking controls
Filter coverage, constrain score range, and sort explicitly.
0 100
0 100
Track coverage
Provider
Tier
Model type
Sort by
Order
Rank | Model | Tier | Overall | Fiqh | Quran | Hadith | % Unanswered | Madhab Bias |
|---|---|---|---|---|---|---|---|---|
🥇 | Ansari 3.5 using RAG + Gemini 3.5 Flash | Baseline | 94.6 | 89.7 | 99.6 | 94.4 | 51.9 | Hanafi / Shafi'i |
Analysis views
Further explore model performance across tracks, difficulty levels, and question topics (madhabs).
Analysis map
Performance: question-type and cognitive load analysis.
Bias: madhab distribution and significance views.
Categories: tier behavior and Arabic-model spotlight.
Tracks: track and category heatmaps plus refusal patterns.
Question-type performance
How accuracy moves across different reasoning demands rather than one flat aggregate score.
Madhab distribution analysis
Bias is model-specific, not a uniform property of all frontier systems. Balance near 25% per school is the reference point, not evidence that every issue has one correct answer.
Model categories
This view isolates tier spread and Arabic-specialist behavior instead of forcing those findings to compete with every other chart on the page.
Track and category breakdowns
Cross-track heatmaps and refusal patterns expose where capability coverage is broad, narrow, or unstable.
Side-by-side model analysis
Compare up to four models across track accuracy, difficulty progression, and selection bias.
Selection
Choose between two and four models
Compare track accuracy, difficulty progression, and relative strength across the same output surface.
Model 1
Model 2
Model 3 (optional)
Model 4 (optional)
Default prefill compares the current top model against the strongest Arabic-tier result.
The table keeps numeric spread visible instead of hiding variance behind one chart.
Optional slots only count after you commit an exact model name from the live list. Custom text is validated, and duplicate selections are blocked with a warning instead of being silently accepted.
No comparison yet
Select at least two models to render the table and charts.
1 | 2 | 3 |
|---|---|---|
any openai-compatible /v1/chat/completions endpoint.
api keys are encrypted at rest and wiped from the queue once evaluation finishes. nothing runs until a maintainer signs off.
01endpoint
provider
model type
02generation
reasoning effort (optional)
leave blank for non-reasoning models.
03notes
custom
standard
POST
https://example.com/v1/chat/completions
Content-Type: application/json Authorization: Bearer ··········
{
"model": "<model-id>",
"messages": [
{
"role": "system",
"content": "answer with a single letter A, B, C, or D."
},
{
"role": "user",
"content": "<question text>"
}
],
"temperature": 0.0,
"max_tokens": 8,
"seed": 42
}
Benchmark documentation
Learn documents what the benchmark measures, how the dataset was built, and what the reported metrics actually mean.
About IslamicMMLU
IslamicMMLU is a paper-backed benchmark for evaluating language models on 10,013 multiple-choice questions spanning Fiqh, Quran, and Hadith.
10,013
Total questions
4,000
Fiqh
2,013
Quran
4,000
Hadith
What the leaderboard is measuring
The benchmark focuses on answer accuracy in a fixed four-way multiple-choice setting. Scores describe how well a model performs on the dataset, not whether that model is fit to replace scholars or issue religious rulings.
Overall score is the simple average of available track scores. A model evaluated on only one track is not artificially penalized for missing tracks, but its coverage is visibly narrower than a model evaluated across all three.
Track coverage
- Fiqh: 4,000 questions across five question types and eight jurisprudence categories.
- Quran: 2,013 questions focused on surah identification, verse retrieval, and verse-count accuracy.
- Hadith: 4,000 questions covering source identification, cloze completion, chapter classification, and authenticity grading.
Fiqh source corpus
The fiqh track is derived from Jurisprudence According to the Four Schools by Abd al-Rahman al-Jaziri. The source text spans 2,221 pages, 1,043 sections, and 8 major fiqh categories.
A multi-agent extraction pipeline produced 2,163 structured rulings across 797 topics before question generation and verification.
- Categories: Salat, Taharat, Siyam, Zakat, Hajj, Mu'amalat, Family Law, Criminal Law.
Quality control
The fiqh question set was automatically checked for source traceability, madhab attribution, distractor validity, and coverage balance. A stratified manual review examined 213 items, approving 207 with an approval rate of 97.2%.
This gives the benchmark a documented quality story rather than a hand-wavy claim of curation.
Why this benchmark exists
Islamic knowledge tasks are unusually sensitive because users often interpret fluent answers as authoritative answers. The benchmark is designed to surface both strengths and failure modes, especially around fiqh disagreement and selective bias toward one madhab.
A strong model in this setting should combine factual recall, structural knowledge, and restraint. Good benchmark performance is useful evidence, but it is not a license for unsupervised deployment in religious decision making.
Methodology
The benchmark currently targets 10,013 questions across three tracks, with overall score computed as the average of available track accuracies.
25%
Random-chance baseline
3
Tracks
4
Answer options per item
261
Dedicated bias-detection fiqh items
Overall scoring
Each question is a four-way multiple-choice item, so random guessing yields a 25% baseline. Track accuracy is measured per domain, and overall score is the simple average of the tracks a model has actually completed.
This avoids the misleading behavior of treating missing tracks as zeros while still making partial evaluations visibly incomplete.
Quran track pipeline
The Quran track contains 2,013 questions across all 114 surahs.
Removal of diacritics and kashida, with Alef and Ya normalization.
Every item passes uniqueness checks so distractors do not duplicate the ground truth.
- Verse-count questions use structured distractors from neighboring or offset surah counts.
- Surah-identification items rely on hard-negative lexical neighbors instead of easy keyword matching.
- Verse-retrieval includes null-response trap items to measure hallucination and boundary awareness.
Hadith track pipeline
The Hadith track contains 4,000 questions drawn from the six canonical Sunni collections: Sahih al-Bukhari, Sahih Muslim, Sunan Abu Dawud, Jami' al-Tirmidhi, Sunan al-Nasa'i, Sunan Ibn Majah.
Deterministic generation with fixed random seeds.
The isnad is removed so models must reason over matn content instead of narrator names.
- Cross-collection near-duplicates are filtered to reduce ambiguous source attribution.
- Cloze targets are selected with IDF-style weighting so blanks test substantive vocabulary.
- Authenticity grading is deliberately high-stakes because misgrading changes which narrations users trust.
Fiqh extraction and question generation
The fiqh corpus starts from Jurisprudence According to the Four Schools, covering 2,221 pages and 1,043 source sections.
Approximately 60% of source sections contained extractable rulings. Parallel extraction and adversarial verification produced 2,163 structured rulings over 797 topics.
Question generation varies distractor strategy by task type: alternative madhab rulings, plausible legal variants, or partial-reasoning decoys are used instead of simplistic random distractors.
Madhab bias analysis
Dedicated fiqh bias-detection items present one valid answer per madhab on the same issue. A perfectly balanced system would distribute its selections at roughly 25% per school.
The leaderboard reports both a chi-squared goodness-of-fit signal and a neutrality score out of 100. High accuracy with high bias is still a problem, because it means the model is selectively flattening legitimate juristic diversity.
- p < 0.05 indicates statistically significant deviation from uniform madhab distribution.
- Neutrality score compresses deviation from 25% into a more legible 0-100 scale.
- Bias results reflect training data and answer priors; they should not be read as intentional sectarian alignment.
Evaluation conditions
The evaluation harness presents Arabic questions in a deterministic setup, logs token usage, and stores model metadata alongside aggregate results.
Refusals and ambiguous responses are not silently ignored. They remain part of the evaluation record because safe non-answers and confident hallucinations are materially different failure modes.
Fiqh terms
The legal vocabulary you need in order to read the fiqh results without flattening madhab differences.
Fiqh
فقهIslamic jurisprudence: the human understanding and application of sacred law.
The fiqh track tests comparative legal reasoning, not only direct fact recall.
Madhab
مذهبA juristic school with its own methods, precedents, and legal reasoning habits.
Bias analysis checks whether a model collapses valid madhab diversity into one preferred answer pattern.
Mu'amalat
معاملاتCommercial and transactional law, including contracts, loans, and trade.
This category is a strong test of conditional reasoning rather than memorized ritual facts.
Taharat
طهارةRitual purity, including wudu, ghusl, tayammum, and impurity rulings.
Purity questions often expose subtle madhab distinctions that weaker models flatten.
Comparative fiqh
Reasoning across multiple madhahib on the same issue without erasing disagreement.
It is one of the benchmark's most important capabilities because real users encounter multiple valid rulings.
Quran terms
Key structural and retrieval concepts used by the Quran track.
Surah
سورةA chapter of the Quran. The Quran contains 114 surahs.
The Quran track measures structural knowledge over the full corpus of surahs.
Ayah
آيةA verse of the Quran.
Verse indexing and retrieval are central to the track's adversarial retrieval tasks.
Jaccard similarity
A lexical-overlap measure used here to identify verses similar enough to support hard negatives.
It reduces trivial questions by preferring verses that are easy to confuse if the model only matches keywords.
Null response trap
A deliberately invalid verse-position option used when the requested verse index exceeds a surah's length.
These items test hallucination control and boundary awareness, not just retrieval.
Hadith terms
Core hadith-science vocabulary used throughout the track and leaderboard.
Hadith
حديثA report of the Prophet's words, actions, or approvals.
The Hadith track tests source knowledge, topical mapping, lexical recall, and grading.
Isnad
إسنادThe chain of narrators that transmits a hadith.
The track removes isnad text to stop models from identifying collections through narrator-name shortcuts.
Matn
متنThe main content text of a hadith narration.
After isnad trimming, models must classify narrations from matn content alone.
Authenticity grading
Scholarly labeling of narrations as authentic, good, weak, or fabricated.
This is a high-stakes task because incorrect grading can distort downstream religious reasoning.
Benchmark and metric terms
Interpretation guidance for the leaderboard metrics rather than a dump of statistical jargon.
Overall accuracy
The average of the available track accuracies for a given model.
It is comparable only when you also look at track coverage and partial-evaluation status.
Neutrality score
A 0-100 score summarizing how evenly a model distributes answers across the four Sunni madhahib.
It is easier to read than raw chi-squared values, but it should still be interpreted beside the distribution itself.
Chi-squared bias signal
A goodness-of-fit test comparing observed madhab selection rates to the ideal 25/25/25/25 split.
It distinguishes genuine imbalance from harmless random variation.
Refused count
The number of questions where the model declined or failed to provide a usable answer.
Refusal is often preferable to hallucinated certainty, so it deserves visibility rather than silent removal.
Fiqh: 4,000 questions
Comparative jurisprudence across the four Sunni schools with adversarial verification and madhab-bias instrumentation.
4,000
Track questions
5
Question types
8
Fiqh categories
What it tests
- Legal recall across worship, transactions, family law, and criminal law.
- Reasoning across multiple valid madhab positions without collapsing disagreement.
- Sensitivity to structural details such as pillars, conditions, obligations, and evidence chains.
Question types
- Madhab ruling identification.
- Multi-hop reasoning.
- Madhab bias detection.
- Detailed component parsing.
- Comparative fiqh.
Category footprint
- Salat, Taharat, Siyam, Zakat, Hajj, Mu'amalat, Family Law, Criminal Law.
Why it is hard
The track is not only about remembering one correct answer. Many items are difficult because the model has to keep multiple valid schools in play, understand contextual conditions, and avoid over-projecting one dominant ruling everywhere.
Quran: 2,013 questions
Structural Quranic knowledge over all 114 surahs, with hard negatives chosen to punish shallow lexical matching.
2,013
Track questions
3
Question types
114
Surahs
Generation design
Removal of diacritics and kashida, with Alef and Ya normalization.
Every item passes uniqueness checks so distractors do not duplicate the ground truth.
Question breakdown
- Verse (Ayah) Count: 114 questions (6%) - Recall the number of verses in each surah using adversarial count distractors.
- Surah Identification: 833 questions (41%) - Identify the source surah of a verse with hard-negative lexical neighbors.
- Verse Retrieval: 1,066 questions (53%) - Map a verse index within a surah to its text, including trap questions for hallucination control.
Why it is hard
The track is designed to punish keyword-sniping. Similar verses are intentionally paired, verse indices can be represented as numerals or Arabic words, and trap items check whether the model invents a verse when the requested position does not exist.
Hadith: 4,000 questions
Narration-source recognition and hadith scholarship tasks built from the six canonical Sunni collections.
4,000
Track questions
4
Question types
6
Canonical collections
Generation design
Deterministic generation with fixed random seeds.
The isnad is removed so models must reason over matn content instead of narrator names.
Question breakdown
- Source Identification: 1,000 questions - Identify which of the six canonical Sunni collections a narration belongs to.
- Cloze Completion: 1,000 questions - Fill a missing high-information keyword using semantically close distractors.
- Chapter Classification: 1,000 questions - Map a hadith to its thematic chapter within a given collection.
- Authenticity Grading: ~1,000 questions - Grade narrations as authentic, good, weak, or fabricated after ambiguity filtering.
Why it is hard
Chapter classification forces taxonomic reasoning rather than direct quotation recall. Authenticity grading adds a more practically sensitive judgment task, especially once near-duplicate narrations with conflicting grades are filtered out.