Instructions to use zym1105/Qwen3.5FundusReasoner_base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use zym1105/Qwen3.5FundusReasoner_base with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="zym1105/Qwen3.5FundusReasoner_base")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("zym1105/Qwen3.5FundusReasoner_base")
model = AutoModelForMultimodalLM.from_pretrained("zym1105/Qwen3.5FundusReasoner_base", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use zym1105/Qwen3.5FundusReasoner_base with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "zym1105/Qwen3.5FundusReasoner_base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zym1105/Qwen3.5FundusReasoner_base",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/zym1105/Qwen3.5FundusReasoner_base

SGLang

How to use zym1105/Qwen3.5FundusReasoner_base with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "zym1105/Qwen3.5FundusReasoner_base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zym1105/Qwen3.5FundusReasoner_base",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "zym1105/Qwen3.5FundusReasoner_base" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zym1105/Qwen3.5FundusReasoner_base",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use zym1105/Qwen3.5FundusReasoner_base with Docker Model Runner:
```
docker model run hf.co/zym1105/Qwen3.5FundusReasoner_base
```

FundusReasoner

FundusReasoner: An Interpretable Vision-Language Reasoning Model for Fundus Diagnosis

中文 | English

中文介绍

模型简介

FundusReasoner 是一个面向彩色眼底照片分析与诊断的可解释视觉语言模型（Vision-Language Model, VLM）。

该模型旨在不仅给出最终诊断结果，还能够将眼底图像中的局部视觉证据、病灶描述、诊断分析与最终诊断联系起来，从而提供更具可检查性的眼底图像分析过程。

FundusReasoner 基于 Qwen3.5 多模态模型构建，并通过逐步增加任务难度和证据粒度的训练方式，使模型学习：

眼科语义理解（Ophthalmic Semantic Adaptation）
学习眼底疾病、正常/异常状态、眼科术语及相关视觉问答任务。
区域级视觉定位（Region-level Visual Grounding）
使用边界框或坐标形式，将眼科解剖结构和病灶概念与眼底图像中的具体区域进行对齐。
结构化证据链推理（Structured Evidence-chain Reasoning）
在诊断过程中依次生成病灶区域、影像表现描述、证据分析以及最终诊断，使诊断结论能够与图像证据建立显式联系。

模型的结构化诊断输出形式如下：

<thinking>
<region>(x1, y1, x2, y2)</region>
<description>眼底影像表现描述</description>
...
<analyse>基于上述影像证据进行诊断分析</analyse>
</thinking>

<answer>最终诊断</answer>

坐标统一采用：

(x1, y1, x2, y2)

其中 (x1, y1) 为左上角，(x2, y2) 为右下角，坐标范围为 0~512。

项目资源

代码仓库 / Code Repository
https://gitee.com/guoyitongxue/fundus-reasoner-qwen35
Hugging Face 用户主页 / Hugging Face Profile
https://huggingface.co/zym1105
当前模型仓库 / Model Repository
https://huggingface.co/zym1105/Qwen3.5FundusReasoner_base

代码仓库包含 FundusReasoner 的项目介绍、训练与评测代码，以及相关实验配置。后续模型与项目更新也将优先在上述仓库中维护。

预定义提示词 / Prompt Templates

下面给出 FundusReasoner 训练和评测过程中使用的几类典型提示词。

<image> 表示输入的眼底图像。在实际推理时，应按照所使用的多模态推理框架传入图像，而不是将 <image> 当作普通文本单独输入。

1. 解剖结构定位 / Anatomical Structure Localization

<image>请标出视盘的位置，并提供对应的边界框坐标（[x1,y1,x2,y2]，左上-右下）。

该任务要求模型根据输入眼底照片定位指定解剖结构，并输出对应的区域坐标。

2. 候选区域结构识别 / Region Recognition

<image>请判断候选框 (0,261,53,316) 内对应的主要生理结构是什么？只输出名称。

该任务给定一个候选区域，要求模型识别其中主要的眼底解剖结构。

3. 区域-概念验证 / Region-Concept Verification

<image>下列候选区域是否为(0,261,53,316)？候选框：视盘。请回答是或否。

该任务用于判断给定区域与指定眼科概念是否匹配。

4. 结构化证据链诊断 / Structured Evidence-chain Diagnosis

<image>
你是一名经验丰富的眼科医生，需要按指定格式对眼底图像进行分析与诊断。

请严格按照以下结构输出：

<thinking>
<region>(x1,y1,x2,y2)</region>
<description>描述该区域的可疑病变；若该区域无异常，则使用坐标(0,0,512,512)，并将描述固定写为“未见异常”。</description>
<!-- 可多组：每个病灶一组。若病灶呈全局性变化（涉及整个眼底），也需单独作为一组，并统一使用区域(0,0,512,512)。 -->

<analyse>基于上述病灶，给出“病灶特征 → 推理 → 诊断”的分析过程。</analyse>
</thinking>

<answer>给出最终诊断；若整体无异常，请将内容固定写为“正常眼底”。</answer>

【坐标格式为 (x1,y1,x2,y2)，分别表示左上角与右下角，数值范围 0~512】
【整体无异常时，仅输出一组：region=(0,0,512,512)、description=“未见异常”，answer=“正常眼底”】
【全局性病灶需以 region=(0,0,512,512) 单独标注，并在 description 中描述】
【每个 region 必须紧跟对应的 description】

这是 FundusReasoner 的核心诊断形式。模型首先定位可疑视觉证据，然后描述对应影像表现，并在 <analyse> 中整合证据，最终在 <answer> 中给出诊断。

5. 直接诊断 / Direct Diagnosis

<image>该患者的眼底情况如何？请直接诊断。

该任务用于直接生成眼底诊断结果，不要求输出完整的结构化证据链。

使用说明

FundusReasoner 面向眼底图像相关的视觉语言任务，包括但不限于：

眼底疾病诊断；
解剖结构定位；
病灶与区域识别；
候选区域验证；
图像级眼科视觉问答；
基于局部影像证据的结构化诊断推理。

对于需要可解释分析的场景，推荐使用结构化证据链诊断提示词，以获得：

区域定位
    ↓
影像表现描述
    ↓
证据分析
    ↓
最终诊断

相比仅输出疾病名称，该形式可以直接检查模型最终诊断所依据的图像区域和视觉证据。

版本说明与当前适用范围

当前发布的 Qwen3.5FundusReasoner_base 为 FundusReasoner 的基础版本，主要面向粗粒度眼底图像预标注、候选标签生成与后续人工筛选，并不建议直接用于真实临床诊断。

需要特别说明的是，当前 base 版本在训练过程中保留了不同语义粒度和不同诊断层级的标签，因此部分标签之间可能存在一定程度的重叠。例如，同一处黄白色病灶在较粗粒度下可以被描述为“黄白色斑块”，而在结合其形态、分布和局部特征进一步判断后，又可能被识别为“玻璃膜疣”。这类标签并非完全互斥，而是反映了从一般影像表现到更具体病理概念的逐级细化过程。

为了提高模型在数据预标注阶段的覆盖率，并为后续人工审核、标签筛选和数据整理提供更充分的候选信息，当前 base 版本有意保留了不同尺度和不同层级的标签体系。因此，在直接使用模型进行图像分析时，模型可能输出部分语义重叠、层级不同或相互包含的标签。

基于上述原因，本版本更适合用于：

大规模眼底数据的粗粒度预标注；
候选病灶和候选诊断标签生成；
人工审核前的数据筛选与辅助整理；
后续指令数据和结构化证据链数据的构建。

本版本不应被视为最终的临床诊断模型。

我们正在进一步整理和修订更加严格的眼科诊断标准，并基于这些标准开发后续的指令微调版本（instruction-tuned version）。该版本将进一步规范标签体系、诊断粒度、输出格式和临床判定标准，以减少标签冗余和语义层级混杂问题。

相关诊断标准目前仍在由眼科医生进行修订和审核。待标准完成后，我们将尽快发布对应的指令微调模型及更完整的使用说明。

研究用途声明

本模型主要用于科研与学术交流。

FundusReasoner 尚未作为医疗器械完成临床验证，其输出可能存在错误、遗漏或不准确的医学判断。因此：

不应将模型输出作为独立的临床诊断依据；
不应替代具有资质的眼科医生进行诊断或治疗决策；
在医学研究和临床相关实验中，应由专业人员对模型输出进行独立审核。

English

Model Overview

FundusReasoner is an interpretable vision-language model designed for color fundus image analysis and diagnosis.

Rather than producing only a final disease label, FundusReasoner is designed to explicitly connect localized visual evidence, descriptions of ophthalmic findings, diagnostic analysis, and the final diagnosis, making the evidence supporting a prediction more directly inspectable.

FundusReasoner is built on a Qwen3.5 multimodal backbone and is trained with progressively increasing task complexity and evidence granularity. The framework consists of three main stages:

Ophthalmic Semantic Adaptation
The model learns fundus disease semantics, ophthalmic terminology, normal/abnormal concepts, and domain-specific visual question answering.
Region-level Visual Grounding
Anatomical structures and pathological findings are aligned with localized image regions using bounding boxes or coordinate-based representations.
Structured Evidence-chain Reasoning
The model generates localized regions, descriptions of visible findings, evidence analysis, and the final diagnosis in sequence, explicitly linking diagnostic conclusions to visual evidence.

The structured diagnostic output follows the general format:

<thinking>
<region>(x1, y1, x2, y2)</region>
<description>Visible ophthalmic finding</description>
...
<analyse>Evidence-based diagnostic analysis</analyse>
</thinking>

<answer>Final diagnosis</answer>

Coordinates follow the format:

(x1, y1, x2, y2)

where (x1, y1) and (x2, y2) represent the top-left and bottom-right corners, respectively. Coordinates are normalized to the 0–512 image coordinate space used by FundusReasoner.

Resources

Code Repository
https://gitee.com/guoyitongxue/fundus-reasoner-qwen35
Hugging Face Profile
https://huggingface.co/zym1105
Model Repository
https://huggingface.co/zym1105/Qwen3.5FundusReasoner_base

The code repository contains the FundusReasoner project description, training and evaluation code, experiment configurations, and related resources.

Supported Task Types

FundusReasoner is designed for several fundus-oriented vision-language tasks, including:

fundus disease diagnosis;
anatomical structure localization;
lesion and region recognition;
region-concept verification;
ophthalmic visual question answering;
structured evidence-based diagnostic reasoning.

For interpretable diagnostic analysis, we recommend using the Structured Evidence-chain Diagnosis prompt shown above.

Its output can be interpreted as:

Region localization
        ↓
Finding description
        ↓
Evidence analysis
        ↓
Final diagnosis

This formulation makes it possible to inspect the visual regions and ophthalmic findings used by the model to support its diagnostic conclusion.

Version Notice and Current Scope

The currently released Qwen3.5FundusReasoner_base is the base version of FundusReasoner. It is primarily intended for coarse-grained fundus image pre-annotation, candidate label generation, and subsequent human screening, rather than direct use for real-world clinical diagnosis.

An important characteristic of the current base model is that it preserves labels at different semantic granularities and diagnostic levels. As a result, some labels may partially overlap. For example, the same yellow-white retinal lesion may first be described at a coarse level as a yellow-white plaque, and may subsequently be identified more specifically as drusen after considering its morphology, distribution, and local characteristics.

Such labels are not necessarily mutually exclusive. Instead, they may reflect a progressive refinement process from a general visual appearance to a more specific pathological concept.

To improve recall during large-scale data pre-annotation and to provide richer candidate information for later manual review, label filtering, and dataset curation, the current base version intentionally retains labels across multiple semantic scales and diagnostic levels. Consequently, direct model outputs may contain partially redundant, hierarchically related, or semantically overlapping labels.

For this reason, the current version is more suitable for:

coarse-grained pre-annotation of large-scale fundus datasets;
generation of candidate lesion and diagnosis labels;
data screening and organization before expert review;
construction of instruction-tuning data and structured evidence-chain annotations.

This release should not be regarded as a finalized clinical diagnostic model.

We are currently developing a subsequent instruction-tuned version based on a more strictly curated ophthalmic diagnostic standard. The upcoming version will further standardize label definitions, diagnostic granularity, output formats, and clinical decision criteria, with the goal of reducing redundant labels and ambiguity across semantic levels.

The corresponding diagnostic standards are still being revised and reviewed by ophthalmologists. Once this process is completed, we plan to release the instruction-tuned model together with more detailed usage guidelines as soon as possible.

Research-use Notice

FundusReasoner is released primarily for academic and research purposes.

The model has not been clinically validated as a medical device and may produce incorrect, incomplete, or misleading medical outputs. Therefore:

model outputs should not be used as an independent basis for clinical diagnosis;
the model must not replace qualified ophthalmologists or other healthcare professionals;
outputs used in medical or clinical research should be independently reviewed by qualified experts.

Citation

Citation information for the FundusReasoner paper will be added after publication.

If you use this model before the formal publication of the paper, please cite or link to the FundusReasoner code repository and this Hugging Face model repository.

Downloads last month: 13

Safetensors

Model size

9B params

Tensor type

BF16