Jegue 0.2

Jegue is "donkey" in Portuguese. Version 0.2 is a 9B typed-decision model: given a question with options (choice or yes/no), it returns a calibrated probability for each option. Most questions are answered in a single forward pass; a learned router sends the hard ones through a short rationale first. It runs on one consumer GPU.

This repository is self-contained: the Qwen3.5-9B backbone files (safetensors, config, tokenizer, chat template) sit at the repository root, next to the adapter, the heads and the code. Nothing is downloaded from the base-model repository. Version 0.1 stays available as the git tag v0.1.

What changed since 0.1

• Decision Index (0.3 public suite) 59.59, up from 54.13 (raw 68.78 against 65.39).
• A three-level answer path per question: direct (slot readout, no text), brief (a stock-model rationale of at most 64 tokens, then the slot is read with the adapter on) and a router that picks between them per question.
• Self-contained repository, a plain Hugging Face class and a vLLM class (code/jegue.py).
• Weaker than 0.1 on ACOS (35.9 against 50.6 skill) and a little on FinEntity and BFCL; see the table.

What it is

• Base model: Qwen/Qwen3.5-9B (bf16), a LoRA adapter (rank 32, alpha 64), an answer-slot readout head (weights/head.pt) and a router head (weights/router-direct-brief.npz).
• Answers are read at an answer slot: the next token through the LM-head rows of the option tokens plus a trained residual. Probabilities are not temperature-scaled.
• Router. Two small ridge heads on the direct pass's slot hidden state and its confidence estimate the chance that direct is wrong and that a rationale would fix it; their product is compared with a threshold (tau = 0.055). Only single-slot choice questions are routed; per-option and yes/no questions always stay direct. In the benchmark run 15.2% of questions took the brief path.
• Prompts longer than 32,768 tokens are refused as unsupported and never truncated.
• Training method is not described in this release.

Decision Index 0.3, public suite (complete run, all 140,620 requests answered, none failed)

Decision Index 59.59 (raw 68.78, breadth 58.04). Jegue 0.1: 54.13 (65.39, 51.78).

Area	Benchmarks	Raw	Skill (chance-corrected)	Skill, Jegue 0.1
Knowledge & Reasoning	11	56.1	44.1	33.5
Language Understanding	9	75.6	65.6	62.5
Retrieval & Classification	6	77.5	72.0	65.5
Tools & Automation	5	75.3	73.1	72.3
Arts & Human Taste	6	54.5	34.6	29.8

Skill is chance-corrected and coverage-adjusted, in points. "Raw" is the metric of the benchmark (accuracy or F1). Full numbers: results/benchmark-summary.json, results/index.json, results/scores.json.

Per benchmark

Area	Benchmark	Metric	Raw 0.2	Skill 0.2	Skill 0.1
Knowledge & Reasoning	GPQA Diamond	accuracy	45.4	27.2	18.4
Knowledge & Reasoning	GSM8K	accuracy	86.9	84.2	52.5
Knowledge & Reasoning	ChessBench	accuracy	20.9	13.9	4.1
Knowledge & Reasoning	MuSR	accuracy	58.5	34.0	33.4
Knowledge & Reasoning	SATA-Bench	case exact accuracy	32.7	31.8	31.6
Knowledge & Reasoning	CRUXEval	accuracy	64.0	42.9	26.8
Knowledge & Reasoning	CLadder	accuracy	84.6	69.1	30.7
Knowledge & Reasoning	HLE	accuracy	8.4	0.0	0.0
Knowledge & Reasoning	MMLU-Pro	accuracy	61.1	56.2	49.8
Knowledge & Reasoning	BBH fixed-option tasks	accuracy	72.6	60.3	52.3
Knowledge & Reasoning	WinoGrande	accuracy	83.6	67.2	65.0
Language Understanding	ContractNLI	macro-F1	84.1	77.0	73.0
Language Understanding	ANLI	macro-F1	70.2	55.3	49.2
Language Understanding	HellaSwag	accuracy	94.1	92.2	92.3
Language Understanding	ACOS	per-review F1	37.9	35.9	50.6
Language Understanding	FinEntity	macro-F1	92.6	89.1	91.6
Language Understanding	iSarcasmEval	Sarcasm F1 · track A, English	59.4	47.8	45.7
Language Understanding	VAST	macro-F1	78.6	67.8	34.3
Language Understanding	NLI4CT	macro-F1	78.5	58.1	58.8
Language Understanding	RAGTruth response-level hallucination	F1 on hallucinated class	82.7	64.1	64.1
Retrieval & Classification	BANKING77	macro-F1	87.8	87.7	87.3
Retrieval & Classification	CLINC150+OOS	macro-F1	93.7	93.7	93.5
Retrieval & Classification	BRIGHT	nDCG@10	44.2	36.9	37.9
Retrieval & Classification	Amazon ESCI	macro-F1	57.8	47.1	45.3
Retrieval & Classification	PhishNChips phishing decisions	accuracy	97.6	95.2	94.0
Retrieval & Classification	HoVer claim verification	accuracy	85.4	70.7	30.6
Tools & Automation	BFCL	case exact accuracy	93.8	91.6	93.7
Tools & Automation	ToolRet	nDCG@10	66.6	61.4	60.2
Tools & Automation	API-Bank	accuracy	78.3	77.9	77.7
Tools & Automation	Home appliance simulator	case exact accuracy	43.2	43.2	40.9
Tools & Automation	When2Call MCQ	accuracy	90.2	86.9	83.7
Arts & Human Taste	BPoMP	accuracy	90.0	80.0	55.9
Arts & Human Taste	Humicroedit	accuracy	65.3	30.7	30.9
Arts & Human Taste	POP909-CL	accuracy	0.6	0.0	0.9
Arts & Human Taste	cfcolor	accuracy	58.7	17.3	11.7
Arts & Human Taste	Habermas Machine	accuracy	45.4	20.7	22.5
Arts & Human Taste	New Yorker caption matching	accuracy	67.2	59.0	56.9

Weak spots (skill under 30): HLE, POP909-CL, cfcolor, Habermas Machine, ChessBench and GPQA Diamond. ACOS is down from 0.1.

Decision Index 0.4

The live board is Decision Index 0.4. Its Full score is 60% private tests of the same skills and 40% private new-domain tasks; the public index no longer counts toward it, but a complete public run is still required. The public suite did not change from 0.3, so the numbers above are the public index under 0.4 as well. The Full score is computed privately by the maintainers and is not available for this release; it has not been measured.

Latency

Measured end to end with the vLLM class (routed mode), one request at a time, on one RTX 5090 (32 GB), over all 140,620 requests of the benchmark: mean 293 ms, median 89 ms, p80 557 ms, p90 773 ms, p95 1,166 ms. The board rule (mean, median and p80 each under 1,000 ms on one RTX PRO 6000) is met on this hardware. Not measured on a PRO 6000. The Hugging Face class in direct mode is a single forward pass; routed mode with Hugging Face generation is slower than vLLM and is meant for accuracy checks.

Try it

One CUDA GPU (about 20 GB for the Hugging Face class; 24 GB or more for vLLM). Install torch, transformers (5.x), peft, safetensors, numpy, tqdm; for the vLLM class also vllm (tested with 0.32). The Decision Index kit is not needed.

Hugging Face class (mode="direct": one forward pass, no text generated; mode="routed": adds the router):

import sys
sys.path.insert(0, "code")
from jegue import Jegue

model = Jegue(mode="direct")

state = "Email from a customer: 'My parcel was due on Monday and it is Friday. Tracking has not changed since Tuesday. I want my money back.'"
questions = {
    "intent": {
        "type": "choice",
        "instructions": "What does the customer mainly want?",
        "criteria": {"refund": "A refund", "status": "An update on the delivery", "complaint": "To complain about the service", "other": "Something else"},
    },
    "urgent": {
        "type": "noul",   # yes/no: returns the probability of yes
        "instructions": "Should this email be escalated to a human agent today?",
    },
}

response = model.ask(state, questions)
print(response["answers"]["intent"]["probabilities"])
print(response["answers"]["urgent"]["noul"])

vLLM class (the fast path; run in a Python process of its own, because it sets VLLM_ENABLE_V1_MULTIPROCESSING=0 to read the hidden states):

import sys
sys.path.insert(0, "code")
from jegue import JegueVLLM

model = JegueVLLM(mode="routed")        # or "direct" / "brief"
response = model.ask(state, questions)  # same state and questions as above

Output of examples/ask_hf.py (direct) and examples/ask_vllm.py (routed), the same: intent: refund refund=1.00 ..., urgent: P(yes)=0.78. ask_vllm.py is the same example with the vLLM class.

Run the benchmark

From a checkout of the Decision Index kit (the engine uses the kit's Engine base class when it is installed):

PYTHONPATH=<this repo>/code python -m decision_index run \
  --engine decision_engine_vr:VllmRoutedEngine \
  --option model=<this repo> --option run=<this repo>/weights --option ckpt=<this repo>/weights --option mode=routed \
  --suite-dir <suite-0.3> --out runs/jegue-0.2

Use the Python of a vLLM environment. decision_engine_v6:BriefSlotEngine (options context=direct|brief, parallel_context=direct, temperature=none) and decision_engine_v6r:RoutedSlotEngine run the same model with Hugging Face.

Layout

• Repository root: config.json, model.safetensors-0000N-of-00004.safetensors and model.safetensors.index.json, tokenizer.json, tokenizer_config.json, vocab.json, merges.txt, chat_template.jinja, preprocessor configs: the Qwen3.5-9B backbone, unchanged.
• weights/: lora/ (PEFT adapter), lora-vllm/ (the same adapter with the key names vLLM expects), head.pt (answer-slot residual), router-direct-brief.npz (router), meta.json.
• code/: jegue.py (the two classes), the engines and the modules they import.
• results/: the complete run (results.jsonl, scores.json, index.json, benchmark-summary.json, environment.json, status.json); in environment.json the local weight directory was replaced by weights, everything else is untouched.
• examples/: ask_hf.py, ask_vllm.py. assets/: the logo.

Training data note

The model was trained on public train/dev splits of upstream datasets plus synthetic examples. Every training example was screened against all public-suite rows (exact state match and 8-word-window overlap), and no public-suite row was used for training. Information about the public suite's distributions (unlabeled inputs and class proportions of some benchmarks) informed the design of the training; no public-suite labels or answers were used.

The routing head, a small ridge regression that only decides which questions get a short rationale, was also calibrated on 1,400 public-suite rows (700 cfcolor, 700 BPoMP). The adapter and the answer-slot head did not see them.

Limitations

• Weak on open-ended expert knowledge (HLE, GPQA Diamond), chess, music (POP909-CL) and on cfcolor.
• The rationale is generated by the stock model, with the adapter off, and is capped at 64 tokens.
• The router escalates by a threshold fitted on development data; on a 38-benchmark audit, always using the rationale was +0.3 points better than routing, at about twice the latency.

License

Apache License 2.0 (see LICENSE), the same license as the base model Qwen/Qwen3.5-9B, whose files are redistributed here unchanged.