Add files using upload-large-folder tool

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -12,15 +12,21 @@ tags:
 - llm
 ---
 
+<p align="center">
+  <img src="./Solar-Open-100B.png" alt="Solar Open Model" width="100%">
+</p>
+
 # **Solar Open**
 
 **Solar Open** is Upstage's flagship **102B-parameter** large language model, trained **entirely from scratch** and released under the **Solar-Apache License 2.0** (see [LICENSE](./LICENSE)). As a **Mixture-of-Experts (MoE)** architecture, it delivers enterprise-grade performance in reasoning, instruction-following, and agentic capabilities—all while prioritizing transparency and customization for the open-source community.
 
 ## Highlights
+
 * **MoE Architecture (102B / 12B):** Built on a Mixture-of-Experts architecture with **102B total / 12B active parameters**. This design delivers the knowledge depth of a massive model with the inference speed and cost-efficiency of a much smaller model.
 * **Massive Training Scale:** Pre-trained on **19.7 trillion tokens**, ensuring broad knowledge coverage and robust reasoning capabilities across various domains.
 
 ## Model Overview
+
 * **Model Name:** Solar Open 100B
 * **Hugging Face ID:** Upstage/Solar-Open-100B
 * **Architecture:** Mixture-of-Experts (MoE)
@@ -31,20 +37,135 @@ tags:
 * **Context Length:** 128k
 * **Training Hardware:** NVIDIA B200 GPUs
 * **License:** **Solar-Apache License 2.0** (See [LICENSE](./LICENSE))
+* **Hardware Requirements:**
+    * **Minimum:** 4x NVIDIA A100 (80GB)
 
 ## Performance
-*Detailed benchmarks and performance metrics will be updated upon the official release on December 31, 2025.*
 
-## Quickstart
-*Python code snippets and usage examples will be available upon the official release on December 31, 2025.*
+TBA
+
+## Inference Quickstart
+
+We recommend using the following generation parameters:
+
+```
+temperature=0.8
+top_p=0.95
+top_k=50
+```
+
+### Transformers
+
+Install the required dependencies:
+
+```bash
+pip install -U transformers kernels torch accelerate
+```
+
+Run inference with the following code:
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+MODEL_ID = "upstage/Solar-Open-100B"
+
+# Load model and tokenizer
+tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
+
+model = AutoModelForCausalLM.from_pretrained(
+    pretrained_model_name_or_path=MODEL_ID,
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+)
+
+# Prepare input
+messages = [{"role": "user", "content": "who are you?"}]
+inputs = tokenizer.apply_chat_template(
+    messages,
+    tokenize=True,
+    add_generation_prompt=True,
+    return_dict=True,
+    return_tensors="pt",
+)
+inputs = inputs.to(model.device)
+
+# Generate response
+generated_ids = model.generate(
+    **inputs,
+    max_new_tokens=4096,
+    temperature=0.8,
+    top_p=0.95,
+    top_k=50,
+    do_sample=True,
+    trust_remote_code=True,
+)
+generated_text = tokenizer.decode(generated_ids[0][inputs.input_ids.shape[1] :])
+print(generated_text)
+```
+
+### vLLM
+
+#### Option 1: Using Docker (Highly Recommended)
+Docker is the **recommended deployment method** for running `Solar-Open-100B`.
+
+```bash
+# For 8 GPUs
+docker run --gpus all \
+    --ipc=host \
+    -p 8000:8000 \
+    upstage/vllm-solar-open:latest \
+    upstage/Solar-Open-100B \
+    --trust-remote-code \
+    --enable-auto-tool-choice \
+    --tool-call-parser solar_open \
+    --reasoning-parser solar_open \
+    --logits-processors vllm.model_executor.models.parallel_tool_call_logits_processor:ParallelToolCallLogitsProcessor \
+    --logits-processors vllm.model_executor.models.solar_open_logits_processor:SolarOpenTemplateLogitsProcessor \
+    --tensor-parallel-size 8
+```
+
+#### Option 2: Installing from Source
+For development, debugging, custom modifications or offline inference, Solar Open can also be run
+using a source installation of vLLM. We recommend using **[uv](https://docs.astral.sh/uv/)** for environment
+management and dependency resolution.
+
+Create and activate a Python virtual environment
+```bash
+uv venv --python 3.12 --seed
+source .venv/bin/activate
+```
+
+Install Solar Open's optimized vLLM
+```bash
+VLLM_PRECOMPILED_WHEEL_LOCATION="https://github.com/vllm-project/vllm/releases/download/v0.12.0/vllm-0.12.0-cp38-abi3-manylinux_2_31_x86_64.whl" \
+VLLM_USE_PRECOMPILED=1 \
+uv pip install git+https://github.com/UpstageAI/[email protected]
+```
+
+Start the vLLM server (For 8 GPUs)
+```bash
+vllm serve upstage/Solar-Open-100B \
+    --trust-remote-code \
+    --enable-auto-tool-choice \
+    --tool-call-parser solar_open \
+    --reasoning-parser solar_open \
+    --logits-processors vllm.model_executor.models.parallel_tool_call_logits_processor:ParallelToolCallLogitsProcessor \
+    --logits-processors vllm.model_executor.models.solar_open_logits_processor:SolarOpenTemplateLogitsProcessor \
+    --tensor-parallel-size 8
+```
 
 ## Public API Access
-The official API service for Solar Open is scheduled to launch publicly on **December 31, 2025**.
-* **Access:** Upstage Console (Available starting Dec 31, 2025)
-* **Documentation:** Upstage Console (Available starting Dec 31, 2025)
+
+The official API service for Solar Open is scheduled to launch publicly on **January**.
+
+* **Access:** Upstage Console (TBA)
+* **Documentation:** Upstage Console (TBA)
 
 ## Citation
+
 If you use Solar Open in your research, please cite:
+
 ```bibtex
 @misc{solar-open-2025,
     title={Solar Open: Scaling Upstage's LLM Capabilities with MoE},
@@ -53,4 +174,3 @@ If you use Solar Open in your research, please cite:
     url={https://huggingface.co/Upstage/Solar-Open-100B}
 }
 ```
-

chat_template.jinja

@@ -0,0 +1,156 @@
+{#- ======== Template Parameters ========  #}
+{%- set add_generation_prompt = add_generation_prompt if add_generation_prompt is defined else true %}
+{%- set default_system_prompt = default_system_prompt if default_system_prompt is defined else true %}
+{%- set reasoning_effort = reasoning_effort if reasoning_effort is defined else "high" %}
+{%- set think_render_option = think_render_option if think_render_option is defined else "lastthink" %}
+
+{#- ======== System Block State ========  #}
+{%- set sys_ns = namespace(is_first_block=true) -%}
+
+{#- ======== Find last user message index ========  #}
+{%- set last_user_idx = namespace(value=-1) -%}
+{%- for message in messages -%}
+    {%- if message.role == 'user' -%}
+        {%- set last_user_idx.value = loop.index0 -%}
+    {%- endif -%}
+{%- endfor -%}
+
+{#- ======== System messages renderers ========  #}
+{%- macro render_system_message(user_system_messages) %}
+    {%- if default_system_prompt %}
+        {%- if not sys_ns.is_first_block %}{{- "\n\n" }}{%- endif %}
+        {%- set sys_ns.is_first_block = false %}
+        {{- "## Provider System Prompt\n\nYou are Solar Open 100B, a large language model trained by Upstage AI, a Korean startup. Your knowledge cutoff is 2025-07. The current date is " + strftime_now("%Y-%m-%d") + "." }}
+    {%- endif -%}
+    {%- if user_system_messages %}
+        {%- if not sys_ns.is_first_block %}{{- "\n\n" }}{%- endif %}
+        {%- set sys_ns.is_first_block = false %}
+        {{- "## System Prompt" }}
+        {%- for system_message in user_system_messages %}
+            {{- "\n\n" }}
+            {{- system_message }}
+        {%- endfor %}
+    {%- endif -%}
+{%- endmacro %}
+
+{%- macro render_tool_instruction(tools) %}
+    {%- if not sys_ns.is_first_block %}{{- "\n\n" }}{%- endif %}
+    {%- set sys_ns.is_first_block = false %}
+    {{- "## Tools\n\n### Tool Call Instruction" }}
+    {{- "\nYou may invoke one or more tools to assist with the user's query. Available tools are provided in JSON Schema format: <|tools:begin|><|tool:begin|><tools-json-object><|tool:end|>...<|tools:end|>\n" }}
+    {{- "\n### Available Tools\n" }}
+    {{- "<|tools:begin|>" }}
+    {%- for tool in tools %}
+        {{- "<|tool:begin|>" }}
+        {{- tool.function | tojson }}
+        {{- "<|tool:end|>" }}
+    {%- endfor %}
+    {{- "<|tools:end|>\n" }}
+    {{- "\n### Tool Call Format\n" }}
+    {{- "For each tool call, return a JSON object with the following structure, enclosed within <|tool_call:begin|> and <|tool_call:end|> tags: \n<|tool_call:begin|><tool-call-id><|tool_call:name|><tool-name><|tool_call:args|><args-json-object><|tool_call:end|>\n" }}
+    {{- "- The <tool-call-id> must be a randomly generated string consisting of 10 lowercase letters (a-z) and/or digits (0-9) (e.g., a1b2c3d4e5)\n" }}
+    {{- "\n### Tool Response Format\n" }}
+    {{- "Each tool is responded by `tool` with the following structure:\n<|tool_response:id|><tool-call-id><|tool_response:name|><tool-name><|tool_response:result|><results><|tool_response:end|>\n" }}
+    {{- "- Ensure the <tool-call-id> matches the corresponding tool call" -}}
+{%- endmacro %}
+
+{%- macro render_json_response_format_instruction(response_format) %}
+    {%- if not sys_ns.is_first_block %}{{- "\n\n" }}{%- endif %}
+    {%- set sys_ns.is_first_block = false %}
+    {{- "## Output Format Constraint" }}
+    {{- "\n\nYour final response should follow the JSON schema: \n[Start of schema]" }}
+    {{- response_format }}
+    {{- "\n[End of schema]\nPlease ensure your answers adhere to this format and do not contain any unnecessary text." }}
+{%- endmacro %}
+
+{%- macro get_tool_name(messages, tool_call_id) %}
+    {%- for msg in messages -%}
+        {%- if msg.role == 'assistant' and msg.tool_calls -%}
+            {%- for tool_call in msg.tool_calls -%}
+                {%- if tool_call.id == tool_call_id -%}
+                    {{- tool_call.function.name }}
+                {%- endif -%}
+            {%- endfor -%}
+        {%- endif -%}
+    {%- endfor -%}
+{%- endmacro %}
+
+{%- macro render_tool_arguments(tool_arguments) %}
+    {%- if tool_arguments is mapping -%}
+        {{- tool_arguments | tojson }}
+    {%- else -%}
+        {{- tool_arguments }}
+    {%- endif -%}
+{%- endmacro %}
+
+{#- ======== Render system message ========  #}
+{%- set ns = namespace(system_messages=[]) -%}
+{%- for message in messages -%}
+    {%- if message.role == 'system' -%}
+        {%- set ns.system_messages = ns.system_messages + [message.content] -%}
+    {%- endif -%}
+{%- endfor -%}
+
+{%- if ns.system_messages or default_system_prompt or tools or response_format -%}
+    {{- "<|begin|>system<|content|>" }}
+        {{- render_system_message(ns.system_messages) }}
+        {%- if tools -%}
+            {{- render_tool_instruction(tools) }}
+        {%- endif %}
+        {%- if response_format -%}
+            {{- render_json_response_format_instruction(response_format) }}
+        {%- endif %}
+    {{- "<|end|>" }}
+{%- endif -%}
+
+{#- ======== Render main messages ========  #}
+{%- for message in messages -%}
+    {%- if message.role == 'user' -%}
+         {{- "<|begin|>user<|content|>" + message.content + "<|end|>" }}
+    {%- elif message.role == 'tool' -%}
+        {%- set prev_is_tool = loop.index0 > 0 and messages[loop.index0 - 1].role == 'tool' -%}
+        {%- set next_is_tool = loop.index0 < (messages | length - 1) and messages[loop.index0 + 1].role == 'tool' -%}
+        {%- if not prev_is_tool -%}
+            {{- "<|begin|>tool<|tool_response|>" }}
+        {%- endif -%}
+        {{- "<|tool_response:begin|>" + message.tool_call_id + "<|tool_response:name|>" }}
+        {{- get_tool_name(messages, message.tool_call_id) }}
+        {{- "<|tool_response:result|>" }}
+        {{- message.content }}
+        {{- "<|tool_response:end|>" }}
+        {%- if not next_is_tool -%}
+            {{- "<|end|>" }}
+        {%- endif -%}
+    {%- elif message.role == 'assistant' -%}
+        {#- ======== Assistant Thinking ========  #}
+        {%- if think_render_option == "all" -%}
+            {%- if message.reasoning -%}
+                {{- "<|begin|>assistant<|think|>" + message.reasoning + "<|end|>" }}
+            {%- endif -%}
+        {%- elif think_render_option == "lastthink" -%}
+            {%- if message.reasoning and loop.index0 > last_user_idx.value -%}
+                {{- "<|begin|>assistant<|think|>" + message.reasoning + "<|end|>" }}
+            {%- endif -%}
+        {%- endif -%}
+
+        {#- ======== Assistant Messages ========  #}
+        {%- if message.tool_calls -%}
+            {{- "<|begin|>assistant<|tool_calls|>" }}
+            {%- for tool_call in message.tool_calls -%}
+                {{- "<|tool_call:begin|>" + tool_call.id +"<|tool_call:name|>" + tool_call.function.name + "<|tool_call:args|>" }}
+                {{- render_tool_arguments(tool_call.function.arguments) }}
+                {{- "<|tool_call:end|>" }}
+            {%- endfor -%}
+            {{- "<|calls|>" }}
+        {%- else -%}
+            {{- "<|begin|>assistant<|content|>" + message.content + "<|end|>" }}
+        {%- endif -%}
+    {%- endif -%}
+{%- endfor -%}
+
+{%- if add_generation_prompt -%}
+    {%- if reasoning_effort in ["low", "minimal"] -%}
+        {{- "<|begin|>assistant<|think|><|end|>" }}
+    {%- endif -%}
+    {{- "<|begin|>assistant" }}
+{%- endif -%}

config.json

@@ -0,0 +1,36 @@
+{
+    "model_type": "solar_open",
+    "architectures": [
+        "SolarOpenForCausalLM"
+    ],
+    "auto_map": {
+        "AutoConfig": "configuration_solar_open.SolarOpenConfig",
+        "AutoModel": "modeling_solar_open.SolarOpenModel",
+        "AutoModelForCausalLM": "modeling_solar_open.SolarOpenForCausalLM"
+    },
+    "pad_token_id": 2,
+    "bos_token_id": 1,
+    "eos_token_id": 2,
+    "partial_rotary_factor": 1.0,
+    "hidden_size": 4096,
+    "num_hidden_layers": 48,
+    "num_attention_heads": 64,
+    "head_dim": 128,
+    "num_key_value_heads": 8,
+    "vocab_size": 196608,
+    "intermediate_size": 10240,
+    "moe_intermediate_size": 1280,
+    "rms_norm_eps": 1e-05,
+    "rope_theta": 1000000,
+    "max_position_embeddings": 131072,
+    "n_routed_experts": 128,
+    "n_shared_experts": 1,
+    "norm_topk_prob": true,
+    "routed_scaling_factor": 1.0,
+    "num_experts_per_tok": 8,
+    "tie_word_embeddings": false,
+    "torch_dtype": "bfloat16",
+    "first_k_dense_replace": 0,
+    "use_cache": true,
+    "transformers_version": "4.57.1"
+}

configuration_solar_open.py

@@ -0,0 +1,235 @@
+# coding=utf-8
+# Copyright 2025 Upstage AI. All rights reserved.
+#
+# Licensed under the Solar-Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://huggingface.co/upstage/Solar-Open-100B/blob/main/LICENSE
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from transformers.configuration_utils import PretrainedConfig
+from transformers.modeling_rope_utils import rope_config_validation
+
+
+class SolarOpenConfig(PretrainedConfig):
+    r"""
+    This is the configuration class to store the configuration of a [`SolarOpenModel`]. It is used to instantiate a
+    SolarOpen model according to the specified arguments, defining the model architecture.
+
+    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
+    documentation from [`PretrainedConfig`] for more information.
+
+
+    Args:
+        vocab_size (`int`, *optional*, defaults to 196608):
+            Vocabulary size of the SolarOpen model. Defines the number of different tokens that can be represented by the
+            `inputs_ids` passed when calling [`SolarOpenModel`]
+        hidden_size (`int`, *optional*, defaults to 4096):
+            Dimension of the hidden representations.
+        intermediate_size (`int`, *optional*, defaults to 10240):
+            Dimension of the MLP representations.
+        num_hidden_layers (`int`, *optional*, defaults to 48):
+            Number of hidden layers in the Transformer encoder.
+        num_attention_heads (`int`, *optional*, defaults to 64):
+            Number of attention heads for each attention layer in the Transformer encoder.
+        partial_rotary_factor (`float`, *optional*, defaults to 1.0):
+            The factor of the partial rotary position.
+        num_key_value_heads (`int`, *optional*, defaults to 8):
+            This is the number of key_value heads that should be used to implement Grouped Query Attention. If
+            `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
+            `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When
+            converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
+            by meanpooling all the original heads within that group. For more details, check out [this
+            paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `32`.
+
+        hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
+            The non-linear activation function (function or string) in the decoder.
+        max_position_embeddings (`int`, *optional*, defaults to 131072):
+            The maximum sequence length that this model might ever be used with.
+        initializer_range (`float`, *optional*, defaults to 0.02):
+            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
+        rms_norm_eps (`float`, *optional*, defaults to 1e-05):
+            The epsilon used by the rms normalization layers.
+        use_cache (`bool`, *optional*, defaults to `True`):
+            Whether or not the model should return the last key/values attentions (not used by all models). Only
+            relevant if `config.is_decoder=True`.
+        tie_word_embeddings (`bool`, *optional*, defaults to `False`):
+            Whether the model's input and output word embeddings should be tied.
+        rope_theta (`float`, *optional*, defaults to 1000000.0):
+            The base period of the RoPE embeddings.
+        rope_scaling (`Dict`, *optional*):
+            Dictionary containing the scaling configuration for the RoPE embeddings. NOTE: if you apply new rope type
+            and you expect the model to work on longer `max_position_embeddings`, we recommend you to update this value
+            accordingly.
+            Expected contents:
+                `rope_type` (`str`):
+                    The sub-variant of RoPE to use. Can be one of ['default', 'linear', 'dynamic', 'yarn', 'longrope',
+                    'llama3'], with 'default' being the original RoPE implementation.
+                `factor` (`float`, *optional*):
+                    Used with all rope types except 'default'. The scaling factor to apply to the RoPE embeddings. In
+                    most scaling types, a `factor` of x will enable the model to handle sequences of length x *
+                    original maximum pre-trained length.
+                `original_max_position_embeddings` (`int`, *optional*):
+                    Used with 'dynamic', 'longrope' and 'llama3'. The original max position embeddings used during
+                    pretraining.
+                `attention_factor` (`float`, *optional*):
+                    Used with 'yarn' and 'longrope'. The scaling factor to be applied on the attention
+                    computation. If unspecified, it defaults to value recommended by the implementation, using the
+                    `factor` field to infer the suggested value.
+                `beta_fast` (`float`, *optional*):
+                    Only used with 'yarn'. Parameter to set the boundary for extrapolation (only) in the linear
+                    ramp function. If unspecified, it defaults to 32.
+                `beta_slow` (`float`, *optional*):
+                    Only used with 'yarn'. Parameter to set the boundary for interpolation (only) in the linear
+                    ramp function. If unspecified, it defaults to 1.
+                `short_factor` (`list[float]`, *optional*):
+                    Only used with 'longrope'. The scaling factor to be applied to short contexts (<
+                    `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden
+                    size divided by the number of attention heads divided by 2
+                `long_factor` (`list[float]`, *optional*):
+                    Only used with 'longrope'. The scaling factor to be applied to long contexts (<
+                    `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden
+                    size divided by the number of attention heads divided by 2
+                `low_freq_factor` (`float`, *optional*):
+                    Only used with 'llama3'. Scaling factor applied to low frequency components of the RoPE
+                `high_freq_factor` (`float`, *optional*):
+                    Only used with 'llama3'. Scaling factor applied to high frequency components of the RoPE
+        attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`):
+            Whether to use a bias in the query, key, value and output projection layers during self-attention.
+        attention_dropout (`float`, *optional*, defaults to 0.0):
+            The dropout ratio for the attention probabilities.
+        moe_intermediate_size (`int`, *optional*, defaults to 1280):
+            Intermediate size of the routed expert.
+        num_experts_per_tok (`int`, *optional*, defaults to 8):
+            number of experts per token.
+        n_shared_experts (`int`, *optional*, defaults to 1):
+            Number of shared experts.
+        n_routed_experts (`int`, *optional*, defaults to 128):
+            Number of routed experts.
+        routed_scaling_factor (`float`, *optional*, defaults to 1.0):
+            Scaling factor or routed experts.
+        n_group (`int`, *optional*, defaults to 1):
+            Number of groups for routed experts.
+        topk_group (`int`, *optional*, defaults to 1):
+            Number of selected groups for each token(for each token, ensuring the selected experts is only within `topk_group` groups).
+        first_k_dense_replace (`int`, *optional*, defaults to 0):
+            Number of dense layers in shallow layers(embed->dense->dense->...->dense->moe->moe...->lm_head).
+                                                            \--k dense layers--/
+        norm_topk_prob (`bool`, *optional*, defaults to `True`):
+            Whether to normalize the topk probabilities.
+        use_qk_norm (`bool`, *optional*, defaults to `False`):
+            Whether to use query-key normalization in the attention
+    ```python
+    >>> from transformers import SolarOpenModel, SolarOpenConfig
+
+    >>> # Initializing a SolarOpen style configuration
+    >>> configuration = SolarOpenConfig()
+
+    >>> # Initializing a model from the SolarOpen style configuration
+    >>> model = SolarOpenModel(configuration)
+
+    >>> # Accessing the model configuration
+    >>> configuration = model.config
+    ```"""
+
+    model_type = "solar_open"
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    # Default tensor parallel plan for base model `SolarOpen`
+    base_model_tp_plan = {
+        "layers.*.self_attn.q_proj": "colwise",
+        "layers.*.self_attn.k_proj": "colwise",
+        "layers.*.self_attn.v_proj": "colwise",
+        "layers.*.self_attn.o_proj": "rowwise",
+        "layers.*.mlp.experts.*.gate_proj": "colwise",
+        "layers.*.mlp.experts.*.up_proj": "colwise",
+        "layers.*.mlp.experts.*.down_proj": "rowwise",
+        "layers.*.mlp.gate_proj": "colwise",
+        "layers.*.mlp.up_proj": "colwise",
+        "layers.*.mlp.down_proj": "rowwise",
+    }
+    base_model_pp_plan = {
+        "embed_tokens": (["input_ids"], ["inputs_embeds"]),
+        "layers": (["hidden_states", "attention_mask"], ["hidden_states"]),
+        "norm": (["hidden_states"], ["hidden_states"]),
+    }
+
+    def __init__(
+        self,
+        vocab_size=196608,
+        hidden_size=4096,
+        intermediate_size=10240,
+        num_hidden_layers=48,
+        num_attention_heads=64,
+        partial_rotary_factor=1.0,
+        num_key_value_heads=8,
+        hidden_act="silu",
+        max_position_embeddings=131072,
+        initializer_range=0.02,
+        rms_norm_eps=1e-5,
+        use_cache=True,
+        tie_word_embeddings=False,
+        rope_theta=1000000.0,
+        rope_scaling=None,
+        attention_bias=False,
+        attention_dropout=0.0,
+        moe_intermediate_size=1280,
+        num_experts_per_tok=8,
+        n_shared_experts=1,
+        n_routed_experts=128,
+        routed_scaling_factor=1.0,
+        n_group=1,
+        topk_group=1,
+        first_k_dense_replace=0,
+        norm_topk_prob=True,
+        use_qk_norm=False,
+        **kwargs,
+    ):
+        self.vocab_size = vocab_size
+        self.max_position_embeddings = max_position_embeddings
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.partial_rotary_factor = partial_rotary_factor
+
+        self.num_key_value_heads = num_key_value_heads
+        self.hidden_act = hidden_act
+        self.initializer_range = initializer_range
+        self.rms_norm_eps = rms_norm_eps
+        self.use_cache = use_cache
+        self.rope_theta = rope_theta
+        self.rope_scaling = rope_scaling
+        self.attention_bias = attention_bias
+        self.attention_dropout = attention_dropout
+        # Validate the correctness of rotary position embeddings parameters
+        # BC: if there is a 'type' field, move it to 'rope_type'.
+        if self.rope_scaling is not None and "type" in self.rope_scaling:
+            self.rope_scaling["rope_type"] = self.rope_scaling["type"]
+        rope_config_validation(self)
+
+        # MoE arguments
+        self.moe_intermediate_size = moe_intermediate_size
+        self.num_experts_per_tok = num_experts_per_tok
+        self.n_group = n_group
+        self.topk_group = topk_group
+        self.n_shared_experts = n_shared_experts
+        self.n_routed_experts = n_routed_experts
+        self.routed_scaling_factor = routed_scaling_factor
+        self.first_k_dense_replace = first_k_dense_replace
+        self.norm_topk_prob = norm_topk_prob
+        self.use_qk_norm = use_qk_norm
+
+        super().__init__(
+            tie_word_embeddings=tie_word_embeddings,
+            **kwargs,
+        )
+
+
+__all__ = ["SolarOpenConfig"]

generation_config.json

@@ -0,0 +1,14 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": [
+    2,
+    24,
+    25
+  ],
+  "pad_token_id": 2,
+  "transformers_version": "4.57.3",
+  "do_sample": true,
+  "temperature": 0.8,
+  "top_p": 0.95
+}

model-00001-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4a98bd68b463f70940147dd8bb2c5a4d77030b03b2f6a49849e02759fba77258
+size 4991260640

model-00002-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6fab47aa2ee1f1a044879966105994c828252a80dfdd4b4ed920b2733ac44b51
+size 4996539840

model-00003-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1b17e454850b15338a5bbe4f261d1d7e3da42d0a604a111cb0aa553428e9c4b4
+size 4991312808

model-00004-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a7dfe8c3397007b3b2a890abcbe964aebdb4220748b910149d27917fb3e1ccd5
+size 4996539768

model-00006-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:265ff7268359d39fb9e2b35983d7e11310554a7c57550f3068fd2db581b67c12
+size 4996539768

model-00007-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19bc5d9bd6a5a2b543aa33649b0c4a0e122e8606c2f9b9abdd4b58740d90ee57
+size 4996539824

model-00008-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6112861c099b0316419f15b07183561875245515e6a8031a5b80339b60657425
+size 4991312840

model-00009-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7bb1beb35b5a0d9ea751aae97ecc1f6f832aac26cfe96894df93dbfcedb44744
+size 4996539888

model-00011-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5e6abb9c376ad675b0a88375201764e508e28f982fd57a4f0a8b712c91896f72
+size 4996540240

model-00012-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:855c7e66c00653968a54bf48fc01f7233e8f6fc0b75a3f6cd6187c0ce296a47b
+size 4996540272

model-00013-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b212bb02f6d1a17487e923f58875f62144f7b0b032da8cdc52f10b85420d0080
+size 4991313360

model-00015-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:992fd054fb80d3c4e987cfa6e477bc041905b19811effd293cc62e8da8262168
+size 4996540240

model-00016-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ecac230f37da8f0592c098bd78396a7c6902ead5f33d621dad721b255fb77c3
+size 4996540240

model-00017-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:df70d3b2014e713eb6b693fabf9e0b09fd8ae6739816e3c9195a2ac86cbfa864
+size 4996540240

model-00018-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:28ef75d0853bd7fe151d896ce376ee5ddbd4f58e06e5186a2d7955d115491c99
+size 4996540312

model-00019-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:52c48d73de1f4622588886fb176cee26e7d0611bf4d65fbcd584f4a45246a8fb
+size 4991313280

model-00021-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:000be0d110f9120b3996041af1d88aef505e4b86c91adbcc51fd6cf746b2d5cc
+size 4996540240

model-00022-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c697cc932f5407a194b6383e4353efa91053d755ee19cfd22f542b7cd73e8b59
+size 4996540240

model-00023-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6589f63dac63abeaa54a235af875d394770ffb6b501e8fdb8f198fa8fd940fa1
+size 4996540288

model-00024-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2f042e2c4f4ff8dbb811b6f7822b35866e622ba6d4ad2e2bca28cfe339b11031
+size 4991313312

model-00025-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2bae7856b4da982f6098700f74ba56bdcedf728dcaa093db53800aad2ae43350
+size 4996540232

model-00026-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:06d2415594afbbddb04752d6402affb69be6b5532ad11f8a74eeca0990bb555f
+size 4996540240

model-00029-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f7ddf612c3b21c1ac0a30da4c36205f5f25373829ab80163b2f8dbb03f00f118
+size 4966147928

model-00030-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3c030092be2eb02aa5557a6330a376abe01fe4af209e25009c4aa68fb64e891d
+size 4990247992

model-00031-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8381350ff935fb54f7d0355a282b3a6f44fec04b27f1cad459b5d2c970e9d9ba
+size 4996540240

model-00032-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:590223701b27e23047f439ee9411806017702a2decfff0369903232c5a006df7
+size 4996540240

model-00033-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:15a52dcc93ff9d59e730b6a0175857ad6d9d539db93fa38771dafd9f5e79535b
+size 4996540240

model-00035-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c2a024c8408367564ee6361f2a79d7b3d226dc8092cf0228350377d6574f4899
+size 4991313288

model-00036-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0c76c59609ff49116ea7c6e840a436bb6ffca4aaef9a69147867284eccdbe3f6
+size 4996540240

model-00038-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d5e8778d02d4ecc69dcddb89a67d037a4fe766bf8e275e7d7f849b7480d0c111
+size 4996540240

model-00039-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5eeee9ba7ca105fb9dc3af07dcb6d935eae9c8d2c26d2f455e6e028c597abe78
+size 4996540280

model-00040-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0bd671a588466b1eeaa9f46e3c8c0a0a352495cde12ff13b4d700cf663b8c88d
+size 4991313336

model-00041-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6e5a3f37a74ac35ecce1a427ae4791820a168dec32016511d8d50344c4409f1d
+size 3912309328

model-00042-of-00042.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9742b5e5f3945b0705b4e852072f39d4fd65aa6b3472dbc1a115f0778d160c1b
+size 1610612864

modeling_solar_open.py

@@ -0,0 +1,597 @@
+# coding=utf-8
+# Copyright 2025 Upstage AI. All rights reserved.
+#
+# Licensed under the Solar-Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://huggingface.co/upstage/Solar-Open-100B/blob/main/LICENSE
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from typing import Callable, Optional, Union
+
+import torch
+import torch.nn.functional as F
+from torch import nn
+
+from transformers.activations import ACT2FN
+from transformers.cache_utils import Cache, DynamicCache
+from transformers.generation import GenerationMixin
+from transformers.integrations import use_kernel_forward_from_hub
+from transformers.masking_utils import create_causal_mask
+from transformers.modeling_flash_attention_utils import FlashAttentionKwargs
+from transformers.modeling_layers import GradientCheckpointingLayer
+from transformers.modeling_outputs import BaseModelOutputWithPast, CausalLMOutputWithPast
+from transformers.modeling_rope_utils import ROPE_INIT_FUNCTIONS, dynamic_rope_update
+from transformers.modeling_utils import ALL_ATTENTION_FUNCTIONS, PreTrainedModel
+from transformers.processing_utils import Unpack
+from transformers.utils import TransformersKwargs, auto_docstring, can_return_tuple
+from transformers.utils.deprecation import deprecate_kwarg
+from transformers.utils.generic import check_model_inputs
+from .configuration_solar_open import SolarOpenConfig
+
+
+def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:
+    """
+    This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,
+    num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)
+    """
+    batch, num_key_value_heads, slen, head_dim = hidden_states.shape
+    if n_rep == 1:
+        return hidden_states
+    hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)
+    return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)
+
+
+def eager_attention_forward(
+    module: nn.Module,
+    query: torch.Tensor,
+    key: torch.Tensor,
+    value: torch.Tensor,
+    attention_mask: Optional[torch.Tensor],
+    scaling: float,
+    dropout: float = 0.0,
+    **kwargs: Unpack[TransformersKwargs],
+):
+    key_states = repeat_kv(key, module.num_key_value_groups)
+    value_states = repeat_kv(value, module.num_key_value_groups)
+
+    attn_weights = torch.matmul(query, key_states.transpose(2, 3)) * scaling
+    if attention_mask is not None:
+        causal_mask = attention_mask[:, :, :, : key_states.shape[-2]]
+        attn_weights = attn_weights + causal_mask
+
+    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
+    attn_weights = nn.functional.dropout(attn_weights, p=dropout, training=module.training)
+    attn_output = torch.matmul(attn_weights, value_states)
+    attn_output = attn_output.transpose(1, 2).contiguous()
+
+    return attn_output, attn_weights
+
+
+def rotate_half(x):
+    """Rotates half the hidden dims of the input."""
+    x1 = x[..., : x.shape[-1] // 2]
+    x2 = x[..., x.shape[-1] // 2 :]
+    return torch.cat((-x2, x1), dim=-1)
+
+
+def apply_rotary_pos_emb(q, k, cos, sin, position_ids=None, unsqueeze_dim=1):
+    """Applies Rotary Position Embedding to the query and key tensors.
+
+    Args:
+        q (`torch.Tensor`): The query tensor.
+        k (`torch.Tensor`): The key tensor.
+        cos (`torch.Tensor`): The cosine part of the rotary embedding.
+        sin (`torch.Tensor`): The sine part of the rotary embedding.
+        position_ids (`torch.Tensor`, *optional*):
+            Deprecated and unused.
+        unsqueeze_dim (`int`, *optional*, defaults to 1):
+            The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and
+            sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note
+            that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and
+            k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes
+            cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have
+            the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2.
+    Returns:
+        `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding.
+    """
+    cos = cos.unsqueeze(unsqueeze_dim)
+    sin = sin.unsqueeze(unsqueeze_dim)
+
+    # Keep half or full tensor for later concatenation
+    rotary_dim = cos.shape[-1]
+    q_rot, q_pass = q[..., :rotary_dim], q[..., rotary_dim:]
+    k_rot, k_pass = k[..., :rotary_dim], k[..., rotary_dim:]
+
+    # Apply rotary embeddings on the first half or full tensor
+    q_embed = (q_rot * cos) + (rotate_half(q_rot) * sin)
+    k_embed = (k_rot * cos) + (rotate_half(k_rot) * sin)
+
+    # Concatenate back to full shape
+    q_embed = torch.cat([q_embed, q_pass], dim=-1)
+    k_embed = torch.cat([k_embed, k_pass], dim=-1)
+    return q_embed, k_embed
+
+
+class SolarOpenAttention(nn.Module):
+    """Multi-headed attention from 'Attention Is All You Need' paper"""
+
+    def __init__(self, config: SolarOpenConfig, layer_idx: Optional[int] = None):
+        super().__init__()
+        self.config = config
+        self.layer_idx = layer_idx
+        self.head_dim = getattr(config, "head_dim", config.hidden_size // config.num_attention_heads)
+        self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
+        self.scaling = self.head_dim**-0.5
+        self.rope_scaling = config.rope_scaling
+        self.attention_dropout = config.attention_dropout
+        self.is_causal = True
+
+        self.q_proj = nn.Linear(
+            config.hidden_size, config.num_attention_heads * self.head_dim, bias=config.attention_bias
+        )
+        self.k_proj = nn.Linear(
+            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
+        )
+        self.v_proj = nn.Linear(
+            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
+        )
+        self.o_proj = nn.Linear(config.num_attention_heads * self.head_dim, config.hidden_size, bias=False)
+        self.use_qk_norm = config.use_qk_norm
+        if self.use_qk_norm:
+            self.q_norm = SolarOpenRMSNorm(self.head_dim, eps=config.rms_norm_eps)
+            self.k_norm = SolarOpenRMSNorm(self.head_dim, eps=config.rms_norm_eps)
+
+    @deprecate_kwarg("past_key_value", new_name="past_key_values", version="4.58")
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        position_embeddings: tuple[torch.Tensor, torch.Tensor],
+        attention_mask: Optional[torch.Tensor],
+        past_key_values: Optional[Cache] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+        **kwargs: Unpack[FlashAttentionKwargs],
+    ) -> tuple[torch.Tensor, Optional[torch.Tensor]]:
+        input_shape = hidden_states.shape[:-1]
+        hidden_shape = (*input_shape, -1, self.head_dim)
+
+        query_states = self.q_proj(hidden_states).view(hidden_shape)
+        key_states = self.k_proj(hidden_states).view(hidden_shape)
+        value_states = self.v_proj(hidden_states).view(hidden_shape)
+
+        if self.use_qk_norm:  # main diff from Llama
+            query_states = self.q_norm(query_states)
+            key_states = self.k_norm(key_states)
+
+        query_states = query_states.transpose(1, 2)
+        key_states = key_states.transpose(1, 2)
+        value_states = value_states.transpose(1, 2)
+
+        cos, sin = position_embeddings
+        query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
+
+        if past_key_values is not None:
+            # sin and cos are specific to RoPE models; position_ids needed for the static cache
+            cache_kwargs = {"sin": sin, "cos": cos, "cache_position": cache_position}
+            key_states, value_states = past_key_values.update(key_states, value_states, self.layer_idx, cache_kwargs)
+
+        attention_interface: Callable = eager_attention_forward
+        if self.config._attn_implementation != "eager":
+            attention_interface = ALL_ATTENTION_FUNCTIONS[self.config._attn_implementation]
+
+        attn_output, attn_weights = attention_interface(
+            self,
+            query_states,
+            key_states,
+            value_states,
+            attention_mask,
+            dropout=0.0 if not self.training else self.attention_dropout,
+            scaling=self.scaling,
+            **kwargs,
+        )
+
+        attn_output = attn_output.reshape(*input_shape, -1).contiguous()
+        attn_output = self.o_proj(attn_output)
+        return attn_output, attn_weights
+
+
+class SolarOpenMLP(nn.Module):
+    def __init__(self, config, hidden_size=None, intermediate_size=None):
+        super().__init__()
+        self.config = config
+        self.hidden_size = config.hidden_size if hidden_size is None else hidden_size
+        self.intermediate_size = config.intermediate_size if intermediate_size is None else intermediate_size
+
+        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
+        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
+        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
+        self.act_fn = ACT2FN[config.hidden_act]
+
+    def forward(self, x):
+        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
+        return down_proj
+
+
+class SolarOpenTopkRouter(nn.Module):
+    def __init__(self, config: SolarOpenConfig):
+        super().__init__()
+        self.config = config
+        self.top_k = config.num_experts_per_tok
+        self.n_routed_experts = config.n_routed_experts
+        self.routed_scaling_factor = config.routed_scaling_factor
+        self.n_group = config.n_group
+        self.topk_group = config.topk_group
+        self.norm_topk_prob = config.norm_topk_prob
+
+        self.weight = nn.Parameter(torch.empty((self.n_routed_experts, config.hidden_size)))
+        self.e_score_correction_bias = nn.Parameter(
+            torch.zeros((self.n_routed_experts), dtype=torch.float32))
+
+    @torch.no_grad()
+    def get_topk_indices(self, scores):
+        scores_for_choice = scores.view(-1, self.n_routed_experts) + self.e_score_correction_bias.unsqueeze(0)
+        group_scores = (
+            scores_for_choice.view(-1, self.n_group, self.n_routed_experts // self.n_group)
+            .topk(2, dim=-1)[0]
+            .sum(dim=-1)
+        )
+        group_idx = torch.topk(group_scores, k=self.topk_group, dim=-1, sorted=False)[1]
+        group_mask = torch.zeros_like(group_scores)
+        group_mask.scatter_(1, group_idx, 1)
+        score_mask = (
+            group_mask.unsqueeze(-1)
+            .expand(-1, self.n_group, self.n_routed_experts // self.n_group)
+            .reshape(-1, self.n_routed_experts)
+        )
+        scores_for_choice = scores_for_choice.masked_fill(~score_mask.bool(), 0.0)
+        topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
+        return topk_indices
+
+    def forward(self, hidden_states):
+        hidden_states = hidden_states.view(-1, self.config.hidden_size)
+        router_logits = F.linear(hidden_states.type(torch.float32), self.weight.type(torch.float32))
+        scores = router_logits.sigmoid()
+        topk_indices = self.get_topk_indices(scores)
+        topk_weights = scores.gather(1, topk_indices)
+        if self.norm_topk_prob:
+            denominator = topk_weights.sum(dim=-1, keepdim=True) + 1e-20
+            topk_weights /= denominator
+        topk_weights = topk_weights * self.routed_scaling_factor
+        return topk_indices, topk_weights
+
+
+@use_kernel_forward_from_hub("RMSNorm")
+class SolarOpenRMSNorm(nn.Module):
+    def __init__(self, hidden_size, eps=1e-6):
+        """
+        SolarOpenRMSNorm is equivalent to T5LayerNorm
+        """
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(hidden_size))
+        self.variance_epsilon = eps
+
+    def forward(self, hidden_states):
+        input_dtype = hidden_states.dtype
+        hidden_states = hidden_states.to(torch.float32)
+        variance = hidden_states.pow(2).mean(-1, keepdim=True)
+        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
+        return self.weight * hidden_states.to(input_dtype)
+
+    def extra_repr(self):
+        return f"{tuple(self.weight.shape)}, eps={self.variance_epsilon}"
+
+
+class SolarOpenMoE(nn.Module):
+    """
+    A mixed expert module containing shared experts.
+    """
+
+    def __init__(self, config):
+        super().__init__()
+        self.config = config
+        self.experts = nn.ModuleList(
+            [
+                SolarOpenMLP(config, intermediate_size=config.moe_intermediate_size)
+                for _ in range(config.n_routed_experts)
+            ]
+        )
+        self.gate = SolarOpenTopkRouter(config)
+        self.shared_experts = SolarOpenMLP(
+            config=config, intermediate_size=config.moe_intermediate_size * config.n_shared_experts
+        )
+
+    @torch.compiler.disable()
+    def moe(self, hidden_states: torch.Tensor, topk_indices: torch.Tensor, topk_weights: torch.Tensor):
+        r"""
+        MoE forward pass that only executes selected experts.
+        Uses @torch.compiler.disable() to allow dynamic shape operations.
+        Requires --enforce-eager flag when serving with vLLM.
+        """
+        final_hidden_states = torch.zeros_like(hidden_states)
+
+        for expert_idx in range(len(self.experts)):
+            expert = self.experts[expert_idx]
+
+            # Find positions where this expert was selected
+            batch_idx, topk_pos = torch.where(topk_indices == expert_idx)
+
+            if batch_idx.numel() == 0:
+                continue
+
+            # Extract only the tokens routed to this expert
+            expert_input = hidden_states[batch_idx]
+            expert_output = expert(expert_input)
+
+            # Apply weights and accumulate results
+            weights = topk_weights[batch_idx, topk_pos].unsqueeze(-1)
+            final_hidden_states.index_add_(0, batch_idx, (expert_output * weights).to(hidden_states.dtype))
+
+        return final_hidden_states
+
+    def forward(self, hidden_states):
+        residuals = hidden_states
+        orig_shape = hidden_states.shape
+        topk_indices, topk_weights = self.gate(hidden_states)
+        hidden_states = hidden_states.view(-1, hidden_states.shape[-1])
+        hidden_states = self.moe(hidden_states, topk_indices, topk_weights).view(*orig_shape)
+        hidden_states = hidden_states + self.shared_experts(residuals)
+        return hidden_states
+
+
+class SolarOpenDecoderLayer(GradientCheckpointingLayer):
+    def __init__(self, config: SolarOpenConfig, layer_idx: int):
+        super().__init__()
+        self.hidden_size = config.hidden_size
+
+        self.self_attn = SolarOpenAttention(config=config, layer_idx=layer_idx)
+
+        if layer_idx >= config.first_k_dense_replace:
+            self.mlp = SolarOpenMoE(config)
+        else:
+            self.mlp = SolarOpenMLP(config)
+
+        self.input_layernorm = SolarOpenRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.post_attention_layernorm = SolarOpenRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+
+    @deprecate_kwarg("past_key_value", new_name="past_key_values", version="4.58")
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[Cache] = None,
+        use_cache: Optional[bool] = False,
+        cache_position: Optional[torch.LongTensor] = None,
+        position_embeddings: Optional[tuple[torch.Tensor, torch.Tensor]] = None,  # necessary, but kept here for BC
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> torch.Tensor:
+        residual = hidden_states
+        hidden_states = self.input_layernorm(hidden_states)
+        # Self Attention
+        hidden_states, _ = self.self_attn(
+            hidden_states=hidden_states,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            use_cache=use_cache,
+            cache_position=cache_position,
+            position_embeddings=position_embeddings,
+            **kwargs,
+        )
+        hidden_states = residual + hidden_states
+
+        # Fully Connected
+        residual = hidden_states
+        hidden_states = self.post_attention_layernorm(hidden_states)
+        hidden_states = self.mlp(hidden_states)
+        hidden_states = residual + hidden_states
+        return hidden_states
+
+
+@auto_docstring
+class SolarOpenPreTrainedModel(PreTrainedModel):
+    config: SolarOpenConfig
+    base_model_prefix = "model"
+    supports_gradient_checkpointing = True
+    _no_split_modules = ["SolarOpenDecoderLayer"]
+    _skip_keys_device_placement = ["past_key_values"]
+    _supports_flash_attn = True
+    _supports_sdpa = True
+    _supports_flex_attn = True
+    _can_compile_fullgraph = False
+    _supports_attention_backend = True
+    _can_record_outputs = {
+        "hidden_states": SolarOpenDecoderLayer,
+        "attentions": SolarOpenAttention,
+    }
+
+    def _init_weights(self, module):
+        super()._init_weights(module)
+        if isinstance(module, SolarOpenTopkRouter):
+            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
+
+
+class SolarOpenRotaryEmbedding(nn.Module):
+    inv_freq: torch.Tensor  # fix linting for `register_buffer`
+
+    def __init__(self, config: SolarOpenConfig, device=None):
+        super().__init__()
+        # BC: "rope_type" was originally "type"
+        if hasattr(config, "rope_scaling") and isinstance(config.rope_scaling, dict):
+            self.rope_type = config.rope_scaling.get("rope_type", config.rope_scaling.get("type"))
+        else:
+            self.rope_type = "default"
+        self.max_seq_len_cached = config.max_position_embeddings
+        self.original_max_seq_len = config.max_position_embeddings
+
+        self.config = config
+        self.rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]
+
+        inv_freq, self.attention_scaling = self.rope_init_fn(self.config, device)
+        self.register_buffer("inv_freq", inv_freq, persistent=False)
+        self.original_inv_freq = self.inv_freq
+
+    @torch.no_grad()
+    @dynamic_rope_update  # power user: used with advanced RoPE types (e.g. dynamic rope)
+    def forward(self, x, position_ids):
+        inv_freq_expanded = self.inv_freq[None, :, None].float().expand(position_ids.shape[0], -1, 1).to(x.device)
+        position_ids_expanded = position_ids[:, None, :].float()
+
+        device_type = x.device.type if isinstance(x.device.type, str) and x.device.type != "mps" else "cpu"
+        with torch.autocast(device_type=device_type, enabled=False):  # Force float32
+            freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(1, 2)
+            emb = torch.cat((freqs, freqs), dim=-1)
+            cos = emb.cos() * self.attention_scaling
+            sin = emb.sin() * self.attention_scaling
+
+        return cos.to(dtype=x.dtype), sin.to(dtype=x.dtype)
+
+
+@auto_docstring
+class SolarOpenModel(SolarOpenPreTrainedModel):
+    _keys_to_ignore_on_load_unexpected = [r"model\.layers\.92.*", r"model\.layers\.46.*"]
+
+    def __init__(self, config: SolarOpenConfig):
+        super().__init__(config)
+        self.padding_idx = config.pad_token_id
+        self.vocab_size = config.vocab_size
+
+        self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)
+        self.layers = nn.ModuleList(
+            [SolarOpenDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
+        )
+        self.norm = SolarOpenRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.rotary_emb = SolarOpenRotaryEmbedding(config=config)
+        self.gradient_checkpointing = False
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    @check_model_inputs()
+    @auto_docstring
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[Cache] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> BaseModelOutputWithPast:
+        if (input_ids is None) ^ (inputs_embeds is not None):
+            raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
+
+        if inputs_embeds is None:
+            inputs_embeds: torch.Tensor = self.embed_tokens(input_ids)
+
+        if use_cache and past_key_values is None:
+            past_key_values = DynamicCache(config=self.config)
+
+        if cache_position is None:
+            past_seen_tokens = past_key_values.get_seq_length() if past_key_values is not None else 0
+            cache_position: torch.Tensor = torch.arange(
+                past_seen_tokens, past_seen_tokens + inputs_embeds.shape[1], device=inputs_embeds.device
+            )
+
+        if position_ids is None:
+            position_ids = cache_position.unsqueeze(0)
+
+        causal_mask = create_causal_mask(
+            config=self.config,
+            input_embeds=inputs_embeds,
+            attention_mask=attention_mask,
+            cache_position=cache_position,
+            past_key_values=past_key_values,
+            position_ids=position_ids,
+        )
+
+        hidden_states = inputs_embeds
+        position_embeddings = self.rotary_emb(hidden_states, position_ids)
+
+        for decoder_layer in self.layers[: self.config.num_hidden_layers]:
+            hidden_states = decoder_layer(
+                hidden_states,
+                attention_mask=causal_mask,
+                position_ids=position_ids,
+                past_key_values=past_key_values,
+                cache_position=cache_position,
+                position_embeddings=position_embeddings,
+                **kwargs,
+            )
+
+        hidden_states = self.norm(hidden_states)
+        return BaseModelOutputWithPast(
+            last_hidden_state=hidden_states,
+            past_key_values=past_key_values,
+        )
+
+
+@auto_docstring
+class SolarOpenForCausalLM(SolarOpenPreTrainedModel, GenerationMixin):
+    _tied_weights_keys = ["lm_head.weight"]
+    _tp_plan = {"lm_head": "colwise_rep"}
+    _pp_plan = {"lm_head": (["hidden_states"], ["logits"])}
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.model = SolarOpenModel(config)
+        self.vocab_size = config.vocab_size
+        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    @can_return_tuple
+    @auto_docstring
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[Cache] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+        logits_to_keep: Union[int, torch.Tensor] = 0,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> CausalLMOutputWithPast:
+
+        outputs: BaseModelOutputWithPast = self.model(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            use_cache=use_cache,
+            cache_position=cache_position,
+            **kwargs,
+        )
+
+        hidden_states = outputs.last_hidden_state
+        # Only compute necessary logits, and do not upcast them to float if we are not computing the loss
+        slice_indices = slice(-logits_to_keep, None) if isinstance(logits_to_keep, int) else logits_to_keep
+        logits = self.lm_head(hidden_states[:, slice_indices, :])
+
+        loss = None
+        if labels is not None:
+            loss = self.loss_function(logits=logits, labels=labels, vocab_size=self.config.vocab_size, **kwargs)
+
+        return CausalLMOutputWithPast(
+            loss=loss,
+            logits=logits,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+
+__all__ = ["SolarOpenPreTrainedModel", "SolarOpenModel", "SolarOpenForCausalLM"]

parallel_tool_call_logits_processor.py

@@ -0,0 +1,104 @@
+# coding=utf-8
+# Copyright 2025 Upstage AI. All rights reserved.
+#
+# Licensed under the Solar-Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://huggingface.co/upstage/Solar-Open-100B/blob/main/LICENSE
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from typing import TYPE_CHECKING
+
+import torch
+
+from vllm.sampling_params import SamplingParams
+from vllm.v1.sample.logits_processor import (
+    AdapterLogitsProcessor,
+    RequestLogitsProcessor,
+)
+
+if TYPE_CHECKING:
+    from vllm.config import VllmConfig
+
+# Hardcoded token IDs for Solar tokenizer
+TOOL_CALL_END_TOKEN_ID = 32  # <|tool_call:end|>
+CALLS_TOKEN_ID = 25  # <|calls|>
+
+
+class SingleToolCallEnforcer:
+    """Request-level logits processor that enforces single tool call.
+
+    When <|tool_call:end|> token is generated, forces the next token
+    to be <|calls|> (which is a stop token), preventing parallel tool calls.
+    """
+
+    def __init__(
+        self,
+        tool_call_end_token_id: int,
+        calls_token_id: int,
+    ):
+        self._tool_call_end_token_id = tool_call_end_token_id
+        self._calls_token_id = calls_token_id
+
+    def __call__(
+        self,
+        output_token_ids: list[int],
+        logits: torch.Tensor,
+    ) -> torch.Tensor:
+        # Check if last generated token is <|tool_call:end|>
+        if output_token_ids and output_token_ids[-1] == self._tool_call_end_token_id:
+            # Force next token to be <|calls|> by masking all other tokens
+            mask = torch.full_like(logits, -float("inf"))
+            mask[self._calls_token_id] = logits[self._calls_token_id]
+            return mask
+
+        return logits
+
+
+class ParallelToolCallLogitsProcessor(AdapterLogitsProcessor):
+    """Logits processor that enforces single tool call when parallel_tool_calls=False.
+
+    When parallel_tool_calls is disabled in SamplingParams, this processor
+    ensures that after <|tool_call:end|> is generated, the next token is
+    forced to be <|calls|> (a stop token), preventing multiple tool calls.
+    """
+
+    def __init__(
+        self,
+        vllm_config: "VllmConfig",
+        device: torch.device,
+        is_pin_memory: bool,
+    ):
+        super().__init__(vllm_config, device, is_pin_memory)
+
+    def is_argmax_invariant(self) -> bool:
+        """This processor can change argmax result by forcing specific tokens."""
+        return False
+
+    def new_req_logits_processor(
+        self,
+        params: SamplingParams,
+    ) -> RequestLogitsProcessor | None:
+        """Return a request-level logits processor if parallel_tool_calls=False.
+
+        Args:
+            params: Request sampling params
+
+        Returns:
+            SingleToolCallEnforcer if parallel_tool_calls is False, otherwise None.
+        """
+        # Only apply when parallel_tool_calls is explicitly disabled
+        if params.parallel_tool_calls is False:
+            return SingleToolCallEnforcer(
+                tool_call_end_token_id=TOOL_CALL_END_TOKEN_ID,
+                calls_token_id=CALLS_TOKEN_ID,
+            )
+
+        return None
+

solar_open_logits_processor.py

@@ -0,0 +1,763 @@
+# coding=utf-8
+# Copyright 2025 Upstage AI. All rights reserved.
+#
+# Licensed under the Solar-Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://huggingface.co/upstage/Solar-Open-100B/blob/main/LICENSE
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+from enum import Enum
+from typing import TYPE_CHECKING
+
+import torch
+
+from vllm.sampling_params import SamplingParams
+from vllm.v1.sample.logits_processor import (
+    AdapterLogitsProcessor,
+    RequestLogitsProcessor,
+)
+
+if TYPE_CHECKING:
+    from vllm.config import VllmConfig
+
+# Hardcoded token IDs for Solar tokenizer
+
+# Special token IDs for chat template
+BEGIN_TOKEN_ID = 20  # <|begin|>
+END_TOKEN_ID = 21  # <|end|>
+THINK_TOKEN_ID = 22  # <|think|>
+CONTENT_TOKEN_ID = 23  # <|content|>
+FLUSH_TOKEN_ID = 24  # <|flush|> (eos token)
+ASSISTANT_TOKEN_ID = 163444  # assistant
+'''
+'assistant' is not a special token exactly, but is treated as one in the logits
+processing.
+'''
+
+# Tool call related tokens
+CALLS_TOKEN_ID = 25  # <|calls|> (eos token for tool calls)
+TOOL_CALLS_TOKEN_ID = 30  # <|tool_calls|>
+TOOL_CALL_BEGIN_TOKEN_ID = 31  # <|tool_call:begin|>
+TOOL_CALL_END_TOKEN_ID = 32  # <|tool_call:end|>
+TOOL_CALL_NAME_TOKEN_ID = 33  # <|tool_call:name|>
+TOOL_CALL_ARGS_TOKEN_ID = 34  # <|tool_call:args|>
+
+# =============================================================================
+# Dynamic Reasoning Budget Configuration
+# =============================================================================
+# budget = min(max_budget, max(min_budget, max_tokens * ratio / 100))
+# Priority: max_budget > min_budget > ratio
+#
+# Available environment variables:
+#   HIGH effort:
+#     SOLAR_REASONING_BUDGET_HIGH_MAX    (default: 32768)  - max_budget
+#     SOLAR_REASONING_BUDGET_HIGH_MIN    (default: 8192)   - min_budget
+#     SOLAR_REASONING_BUDGET_HIGH_RATIO  (default: 60)     - % of max_tokens
+#
+#   MEDIUM effort:
+#     SOLAR_REASONING_BUDGET_MEDIUM_MAX    (default: 16384)  - max_budget
+#     SOLAR_REASONING_BUDGET_MEDIUM_MIN    (default: 4096)   - min_budget
+#     SOLAR_REASONING_BUDGET_MEDIUM_RATIO  (default: 30)     - % of max_tokens
+#
+#   Tool call:
+#     SOLAR_TOOL_CALL_ID_BUDGET  (default: 10)  - Max tokens for tool call ID
+# =============================================================================
+
+DEFAULT_REASONING_EFFORT = "high"
+
+# HIGH effort settings (1k = 1024 tokens)
+DEFAULT_REASONING_BUDGET_HIGH_MAX = 32 * 1024
+DEFAULT_REASONING_BUDGET_HIGH_MIN = 8 * 1024
+DEFAULT_REASONING_BUDGET_HIGH_RATIO = 60
+
+# MEDIUM effort settings
+DEFAULT_REASONING_BUDGET_MEDIUM_MAX = 16 * 1024
+DEFAULT_REASONING_BUDGET_MEDIUM_MIN = 4 * 1024
+DEFAULT_REASONING_BUDGET_MEDIUM_RATIO = 30
+
+# Tool call settings
+DEFAULT_TOOL_CALL_ID_BUDGET = 10
+
+# Pre-computed constant to avoid repeated string parsing
+NEG_INF = float("-inf")
+
+
+def is_reasoning_request(params: SamplingParams) -> bool:
+    """Check if the request is a reasoning request based on reasoning_effort."""
+    return (params.reasoning_effort is None) or (params.reasoning_effort in ("medium", "high"))
+
+
+def is_structured_outputs(params: SamplingParams) -> bool:
+    """Check if the request has structured outputs constraints."""
+    return (
+        params.structured_outputs is not None
+        and not params.structured_outputs.all_constraints_none()
+    )
+
+
+class GenerationState(Enum):
+    """Enum representing the current state of response generation."""
+
+    # Initial state - no tokens generated yet
+    INITIAL = "initial"
+
+    # New message states (after think_end)
+    NEW_MESSAGE_BEGIN = "new_message_begin"  # <|begin|> token was just generated
+    NEW_MESSAGE_ASSISTANT = "new_message_assistant"  # assistant token after <|begin|>
+
+    # Think mode states
+    THINK_BEGIN = "think_begin"  # <|think|> token was just generated
+    THINK_IN_PROGRESS = "think_in_progress"  # Generating think content
+    THINK_END = "think_end"  # <|end|> after think content
+    THINK_FLUSH = "think_flush"  # <|flush|> after think content
+
+    # Content states
+    CONTENT_BEGIN = "content_begin"  # <|content|> token was just generated
+    CONTENT_IN_PROGRESS = "content_in_progress"  # Generating content
+    CONTENT_END = "content_end"  # <|end|> or <|flush|> after content
+    CONTENT_FLUSH = "content_flush"  # <|flush|> after content
+
+    # Tool call states
+    # Flow: <|tool_calls|> -> (<|tool_call:begin|> -> id -> <|tool_call:name|> -> name -> <|tool_call:args|> -> args -> <|tool_call:end|>)+ -> <|calls|>
+    # Note: Think message can appear before <|tool_calls|>
+    TOOL_CALLS_BEGIN = "tool_calls_begin"  # <|tool_calls|> token was just generated
+    TOOL_CALL_BEGIN = "tool_call_begin"  # <|tool_call:begin|> token was just generated
+    TOOL_CALL_ID_IN_PROGRESS = "tool_call_id_in_progress"  # Generating tool call ID
+    TOOL_CALL_NAME_BEGIN = "tool_call_name_begin"  # <|tool_call:name|> token was just generated
+    TOOL_CALL_NAME_IN_PROGRESS = "tool_call_name_in_progress"  # Generating tool name
+    TOOL_CALL_ARGS_BEGIN = "tool_call_args_begin"  # <|tool_call:args|> token was just generated
+    TOOL_CALL_ARGS_IN_PROGRESS = "tool_call_args_in_progress"  # Generating tool arguments (JSON)
+    TOOL_CALL_END = "tool_call_end"  # <|tool_call:end|> token was just generated (can start another tool call or end)
+    CALLS = "calls"  # <|calls|> token was just generated (eos token for tool calls)
+
+
+def get_generation_state(
+    output_token_ids: list[int],
+    begin_token_id: int = BEGIN_TOKEN_ID,
+    end_token_id: int = END_TOKEN_ID,
+    flush_token_id: int = FLUSH_TOKEN_ID,
+    think_token_id: int = THINK_TOKEN_ID,
+    content_token_id: int = CONTENT_TOKEN_ID,
+    tool_calls_token_id: int = TOOL_CALLS_TOKEN_ID,
+    tool_call_begin_token_id: int = TOOL_CALL_BEGIN_TOKEN_ID,
+    tool_call_name_token_id: int = TOOL_CALL_NAME_TOKEN_ID,
+    tool_call_args_token_id: int = TOOL_CALL_ARGS_TOKEN_ID,
+    tool_call_end_token_id: int = TOOL_CALL_END_TOKEN_ID,
+    calls_token_id: int = CALLS_TOKEN_ID,
+    assistant_token_id: int = ASSISTANT_TOKEN_ID,
+) -> GenerationState:
+    """Determine the current generation state based on output token IDs.
+
+    Analyzes the sequence of generated tokens to determine which phase
+    of the chat template the generation is currently in.
+
+    Response format specs:
+    - think mode: <|think|>{{think-tokens}}<|end|><|begin|>assistant<|content|>{{content-tokens}}<|flush|>
+    - tool mode: <|begin|>assistant<|tool_calls|><|tool_call:begin|>{{id}}<|tool_call:name|>{{name}}<|tool_call:args|>{{args}}<|tool_call:end|><|calls|>
+    - tool mode (with think): <|think|>{{think-tokens}}<|end|><|begin|>assistant<|tool_calls|>...<|calls|>
+    - no-think mode: <|content|>{{content-tokens}}<|flush|>
+
+    Args:
+        output_token_ids: List of token IDs generated so far.
+        begin_token_id: Token ID for <|begin|>.
+        end_token_id: Token ID for <|end|>.
+        flush_token_id: Token ID for <|flush|> (eos).
+        think_token_id: Token ID for <|think|>.
+        content_token_id: Token ID for <|content|>.
+        tool_calls_token_id: Token ID for <|tool_calls|>.
+        tool_call_begin_token_id: Token ID for <|tool_call:begin|>.
+        tool_call_name_token_id: Token ID for <|tool_call:name|>.
+        tool_call_args_token_id: Token ID for <|tool_call:args|>.
+        tool_call_end_token_id: Token ID for <|tool_call:end|>.
+        calls_token_id: Token ID for <|calls|> (eos).
+        assistant_token_id: Token ID for assistant.
+
+    Returns:
+        GenerationState indicating the current phase of generation.
+    """
+    if not output_token_ids:
+        return GenerationState.INITIAL
+
+    # Track state by scanning through tokens
+    state = GenerationState.INITIAL
+    in_think = False
+    in_content = False
+
+    for token_id in output_token_ids:
+        if token_id == think_token_id:
+            state = GenerationState.THINK_BEGIN
+            in_think = True
+            in_content = False
+
+        elif token_id == content_token_id:
+            state = GenerationState.CONTENT_BEGIN
+            in_content = True
+            in_think = False
+
+        elif token_id == tool_calls_token_id:
+            state = GenerationState.TOOL_CALLS_BEGIN
+            in_think = False
+            in_content = False
+
+        elif token_id == tool_call_begin_token_id:
+            state = GenerationState.TOOL_CALL_BEGIN
+
+        elif token_id == tool_call_name_token_id:
+            state = GenerationState.TOOL_CALL_NAME_BEGIN
+
+        elif token_id == tool_call_args_token_id:
+            state = GenerationState.TOOL_CALL_ARGS_BEGIN
+
+        elif token_id == tool_call_end_token_id:
+            state = GenerationState.TOOL_CALL_END
+
+        elif token_id == calls_token_id:
+            state = GenerationState.CALLS
+
+        elif token_id == begin_token_id:
+            state = GenerationState.NEW_MESSAGE_BEGIN
+
+        elif token_id == assistant_token_id:
+            if state == GenerationState.NEW_MESSAGE_BEGIN:
+                state = GenerationState.NEW_MESSAGE_ASSISTANT
+
+        elif token_id == end_token_id:
+            if in_think:
+                state = GenerationState.THINK_END
+                in_think = False
+            elif in_content:
+                state = GenerationState.CONTENT_END
+                in_content = False
+
+        elif token_id == flush_token_id:
+            if in_think:
+                state = GenerationState.THINK_FLUSH
+                in_think = False
+            elif in_content:
+                state = GenerationState.CONTENT_FLUSH
+                in_content = False
+
+        else:
+            # Regular token - update state based on current context
+            if state == GenerationState.THINK_BEGIN:
+                state = GenerationState.THINK_IN_PROGRESS
+            elif state == GenerationState.THINK_IN_PROGRESS:
+                pass  # Stay in think_in_progress
+            elif state == GenerationState.CONTENT_BEGIN:
+                state = GenerationState.CONTENT_IN_PROGRESS
+            elif state == GenerationState.CONTENT_IN_PROGRESS:
+                pass  # Stay in content_in_progress
+            elif state == GenerationState.TOOL_CALL_BEGIN:
+                state = GenerationState.TOOL_CALL_ID_IN_PROGRESS
+            elif state == GenerationState.TOOL_CALL_ID_IN_PROGRESS:
+                pass  # Stay in tool_call_id_in_progress
+            elif state == GenerationState.TOOL_CALL_NAME_BEGIN:
+                state = GenerationState.TOOL_CALL_NAME_IN_PROGRESS
+            elif state == GenerationState.TOOL_CALL_NAME_IN_PROGRESS:
+                pass  # Stay in tool_call_name_in_progress
+            elif state == GenerationState.TOOL_CALL_ARGS_BEGIN:
+                state = GenerationState.TOOL_CALL_ARGS_IN_PROGRESS
+            elif state == GenerationState.TOOL_CALL_ARGS_IN_PROGRESS:
+                pass  # Stay in tool_call_args_in_progress
+
+    return state
+
+
+# Pre-computed list of all special token IDs for batch indexing
+_ALL_SPECIAL_TOKEN_IDS = [
+    BEGIN_TOKEN_ID,
+    END_TOKEN_ID,
+    THINK_TOKEN_ID,
+    CONTENT_TOKEN_ID,
+    FLUSH_TOKEN_ID,
+    CALLS_TOKEN_ID,
+    TOOL_CALLS_TOKEN_ID,
+    TOOL_CALL_BEGIN_TOKEN_ID,
+    TOOL_CALL_END_TOKEN_ID,
+    TOOL_CALL_NAME_TOKEN_ID,
+    TOOL_CALL_ARGS_TOKEN_ID,
+]
+
+# Pre-computed lists for state-specific batch indexing (excluding allowed tokens)
+_SPECIAL_EXCEPT_END = [  # For THINK states (allow END)
+    BEGIN_TOKEN_ID, FLUSH_TOKEN_ID, THINK_TOKEN_ID, CONTENT_TOKEN_ID,
+    TOOL_CALLS_TOKEN_ID, CALLS_TOKEN_ID, TOOL_CALL_BEGIN_TOKEN_ID,
+    TOOL_CALL_END_TOKEN_ID, TOOL_CALL_NAME_TOKEN_ID, TOOL_CALL_ARGS_TOKEN_ID,
+]
+
+_SPECIAL_EXCEPT_CONTENT_TOOLCALLS = [  # For NEW_MESSAGE_ASSISTANT (allow CONTENT, TOOL_CALLS)
+    THINK_TOKEN_ID, BEGIN_TOKEN_ID, END_TOKEN_ID, FLUSH_TOKEN_ID,
+    CALLS_TOKEN_ID, TOOL_CALL_BEGIN_TOKEN_ID, TOOL_CALL_END_TOKEN_ID,
+    TOOL_CALL_NAME_TOKEN_ID, TOOL_CALL_ARGS_TOKEN_ID,
+]
+
+_SPECIAL_EXCEPT_FLUSH = [  # For CONTENT states (allow FLUSH)
+    BEGIN_TOKEN_ID, END_TOKEN_ID, THINK_TOKEN_ID, CONTENT_TOKEN_ID,
+    TOOL_CALLS_TOKEN_ID, CALLS_TOKEN_ID, TOOL_CALL_BEGIN_TOKEN_ID,
+    TOOL_CALL_END_TOKEN_ID, TOOL_CALL_NAME_TOKEN_ID, TOOL_CALL_ARGS_TOKEN_ID,
+]
+
+_SPECIAL_EXCEPT_TOOLCALL_NAME = [  # For TOOL_CALL_ID_IN_PROGRESS (allow TOOL_CALL_NAME)
+    BEGIN_TOKEN_ID, END_TOKEN_ID, THINK_TOKEN_ID, CONTENT_TOKEN_ID,
+    FLUSH_TOKEN_ID, CALLS_TOKEN_ID, TOOL_CALLS_TOKEN_ID,
+    TOOL_CALL_BEGIN_TOKEN_ID, TOOL_CALL_END_TOKEN_ID, TOOL_CALL_ARGS_TOKEN_ID,
+]
+
+_SPECIAL_EXCEPT_TOOLCALL_ARGS = [  # For TOOL_CALL_NAME_IN_PROGRESS (allow TOOL_CALL_ARGS)
+    BEGIN_TOKEN_ID, END_TOKEN_ID, THINK_TOKEN_ID, CONTENT_TOKEN_ID,
+    FLUSH_TOKEN_ID, CALLS_TOKEN_ID, TOOL_CALLS_TOKEN_ID,
+    TOOL_CALL_BEGIN_TOKEN_ID, TOOL_CALL_END_TOKEN_ID, TOOL_CALL_NAME_TOKEN_ID,
+]
+
+_SPECIAL_EXCEPT_TOOLCALL_END = [  # For TOOL_CALL_ARGS_IN_PROGRESS (allow TOOL_CALL_END)
+    BEGIN_TOKEN_ID, END_TOKEN_ID, THINK_TOKEN_ID, CONTENT_TOKEN_ID,
+    FLUSH_TOKEN_ID, CALLS_TOKEN_ID, TOOL_CALLS_TOKEN_ID,
+    TOOL_CALL_BEGIN_TOKEN_ID, TOOL_CALL_NAME_TOKEN_ID, TOOL_CALL_ARGS_TOKEN_ID,
+]
+
+
+def _forbid_all_special_tokens(logits: torch.Tensor) -> None:
+    """Set all special token logits to -inf."""
+    logits[_ALL_SPECIAL_TOKEN_IDS] = NEG_INF
+
+
+class SolarOpenTemplateEnforcer:
+    """Request-level logits processor that enforces Solar Open chat template.
+
+    Enforces the following generation rules:
+    - think mode: <|think|>{{tokens}}<|end|><|begin|>assistant<|content|>{{tokens}}<|flush|>
+    - tool mode: <|tool_calls|><|tool_call:begin|>{{id}}<|tool_call:name|>{{name}}<|tool_call:args|>{{args}}<|tool_call:end|><|calls|>
+    - tool+think mode: <|think|>{{tokens}}<|end|><|begin|>assistant<|tool_calls|>...<|calls|>
+    - no-think mode: <|content|>{{tokens}}<|flush|>
+
+    Key constraints:
+    - Think message can only appear first
+    - Think message must be followed by another message
+    - Content and tool messages cannot coexist
+    - Maximum 2 messages (think + content/tool, or just content/tool)
+
+    Performance optimization:
+    - Uses incremental state tracking to avoid full token sequence scan on each call
+    - Maintains local counters for budget tracking
+    - Uses pre-computed constants to avoid repeated object creation
+    """
+
+    # Pre-computed frozenset for reasoning state check (avoids set creation per call)
+    _REASONING_STATES = frozenset({
+        GenerationState.INITIAL,
+        GenerationState.THINK_BEGIN,
+        GenerationState.THINK_IN_PROGRESS,
+    })
+
+    def __init__(
+        self,
+        is_reasoning_request: bool,
+        is_structured_outputs: bool,
+        reasoning_budget: int | None = None,
+        tool_call_id_budget: int = DEFAULT_TOOL_CALL_ID_BUDGET,
+    ):
+        self._is_reasoning_request = is_reasoning_request
+        self._is_structured_outputs = is_structured_outputs
+        self._reasoning_budget = reasoning_budget
+        self._tool_call_id_budget = tool_call_id_budget
+
+        # Incremental state tracking
+        self._state = GenerationState.INITIAL
+        self._last_processed_len = 0
+        self._in_think = False
+        self._in_content = False
+
+        # Budget counters
+        self._think_token_count = 0
+        self._tool_call_id_token_count = 0
+
+    def _reset_state(self) -> None:
+        """Reset all incremental state to initial values.
+
+        Called when defensive reprocessing is needed (e.g., token sequence inconsistency).
+        """
+        self._state = GenerationState.INITIAL
+        self._last_processed_len = 0
+        self._in_think = False
+        self._in_content = False
+        self._think_token_count = 0
+        self._tool_call_id_token_count = 0
+
+    def _process_token(self, token_id: int) -> None:
+        """Process a single token and update internal state incrementally.
+
+        Args:
+            token_id: The token ID to process.
+        """
+        if token_id == THINK_TOKEN_ID:
+            self._state = GenerationState.THINK_BEGIN
+            self._in_think = True
+            self._in_content = False
+            self._think_token_count = 0  # Reset counter for new think block
+
+        elif token_id == CONTENT_TOKEN_ID:
+            self._state = GenerationState.CONTENT_BEGIN
+            self._in_content = True
+            self._in_think = False
+
+        elif token_id == TOOL_CALLS_TOKEN_ID:
+            self._state = GenerationState.TOOL_CALLS_BEGIN
+            self._in_think = False
+            self._in_content = False
+
+        elif token_id == TOOL_CALL_BEGIN_TOKEN_ID:
+            self._state = GenerationState.TOOL_CALL_BEGIN
+            self._tool_call_id_token_count = 0  # Reset counter for new tool call
+
+        elif token_id == TOOL_CALL_NAME_TOKEN_ID:
+            self._state = GenerationState.TOOL_CALL_NAME_BEGIN
+
+        elif token_id == TOOL_CALL_ARGS_TOKEN_ID:
+            self._state = GenerationState.TOOL_CALL_ARGS_BEGIN
+
+        elif token_id == TOOL_CALL_END_TOKEN_ID:
+            self._state = GenerationState.TOOL_CALL_END
+
+        elif token_id == CALLS_TOKEN_ID:
+            self._state = GenerationState.CALLS
+
+        elif token_id == BEGIN_TOKEN_ID:
+            self._state = GenerationState.NEW_MESSAGE_BEGIN
+
+        elif token_id == ASSISTANT_TOKEN_ID:
+            if self._state == GenerationState.NEW_MESSAGE_BEGIN:
+                self._state = GenerationState.NEW_MESSAGE_ASSISTANT
+
+        elif token_id == END_TOKEN_ID:
+            if self._in_think:
+                self._state = GenerationState.THINK_END
+                self._in_think = False
+            elif self._in_content:
+                self._state = GenerationState.CONTENT_END
+                self._in_content = False
+
+        elif token_id == FLUSH_TOKEN_ID:
+            if self._in_think:
+                self._state = GenerationState.THINK_FLUSH
+                self._in_think = False
+            elif self._in_content:
+                self._state = GenerationState.CONTENT_FLUSH
+                self._in_content = False
+
+        else:
+            # Regular token - update state and counters based on current context
+            if self._state == GenerationState.THINK_BEGIN:
+                self._state = GenerationState.THINK_IN_PROGRESS
+                self._think_token_count += 1
+            elif self._state == GenerationState.THINK_IN_PROGRESS:
+                self._think_token_count += 1
+            elif self._state == GenerationState.CONTENT_BEGIN:
+                self._state = GenerationState.CONTENT_IN_PROGRESS
+            elif self._state == GenerationState.CONTENT_IN_PROGRESS:
+                pass  # Stay in content_in_progress
+            elif self._state == GenerationState.TOOL_CALL_BEGIN:
+                self._state = GenerationState.TOOL_CALL_ID_IN_PROGRESS
+                self._tool_call_id_token_count += 1
+            elif self._state == GenerationState.TOOL_CALL_ID_IN_PROGRESS:
+                self._tool_call_id_token_count += 1
+            elif self._state == GenerationState.TOOL_CALL_NAME_BEGIN:
+                self._state = GenerationState.TOOL_CALL_NAME_IN_PROGRESS
+            elif self._state == GenerationState.TOOL_CALL_NAME_IN_PROGRESS:
+                pass  # Stay in tool_call_name_in_progress
+            elif self._state == GenerationState.TOOL_CALL_ARGS_BEGIN:
+                self._state = GenerationState.TOOL_CALL_ARGS_IN_PROGRESS
+            elif self._state == GenerationState.TOOL_CALL_ARGS_IN_PROGRESS:
+                pass  # Stay in tool_call_args_in_progress
+
+    def _update_state_incremental(self, output_token_ids: list[int]) -> None:
+        """Update internal state by processing only new tokens.
+
+        Args:
+            output_token_ids: Full list of output token IDs.
+        """
+        current_len = len(output_token_ids)
+
+        # Defensive check: if token sequence is shorter than expected, reset and reprocess
+        if current_len < self._last_processed_len:
+            self._reset_state()
+
+        # Process only new tokens
+        for i in range(self._last_processed_len, current_len):
+            self._process_token(output_token_ids[i])
+
+        self._last_processed_len = current_len
+
+    @staticmethod
+    def _count_think_tokens(output_token_ids: list[int]) -> int:
+        """Count the number of tokens generated after <|think|> token.
+
+        Returns 0 if <|think|> token is not found (defensive).
+        Note: This static method is kept for backward compatibility and testing.
+        The incremental version uses _think_token_count instead.
+        """
+        try:
+            think_index = output_token_ids.index(THINK_TOKEN_ID)
+            return len(output_token_ids) - think_index - 1
+        except ValueError:
+            return 0
+
+    @staticmethod
+    def _count_tool_call_id_tokens(output_token_ids: list[int]) -> int:
+        """Count the number of tokens generated after the last <|tool_call:begin|> token.
+
+        Returns 0 if <|tool_call:begin|> token is not found (defensive).
+        Note: This static method is kept for backward compatibility and testing.
+        The incremental version uses _tool_call_id_token_count instead.
+        """
+        # Find the last occurrence of <|tool_call:begin|> for multi-tool-call support
+        try:
+            # Reverse search for the last <|tool_call:begin|>
+            reversed_index = output_token_ids[::-1].index(TOOL_CALL_BEGIN_TOKEN_ID)
+            last_begin_index = len(output_token_ids) - 1 - reversed_index
+            return len(output_token_ids) - last_begin_index - 1
+        except ValueError:
+            return 0
+
+    def __call__(
+        self,
+        output_token_ids: list[int],
+        logits: torch.Tensor,
+    ) -> torch.Tensor:
+        # Update state incrementally (only process new tokens)
+        self._update_state_incremental(output_token_ids)
+        state = self._state
+
+        # Handle structured outputs mode
+        if self._is_structured_outputs:
+            if not self._is_reasoning_request:
+                # Non-reasoning request with structured outputs: no logit control
+                return logits
+            else:
+                # Reasoning request with structured outputs:
+                # Control logits only during reasoning phase
+                if state not in self._REASONING_STATES:
+                    # Reasoning finished, let structured outputs handle it
+                    return logits
+
+        if state == GenerationState.INITIAL:
+            if self._is_reasoning_request:
+                # Force: <|think|> only (reasoning request must start with think)
+                think_logit = logits[THINK_TOKEN_ID].clone()
+                logits.fill_(NEG_INF)
+                logits[THINK_TOKEN_ID] = think_logit
+            else:
+                # Allow: <|content|>, <|tool_calls|> only
+                content_logit = logits[CONTENT_TOKEN_ID].clone()
+                tool_calls_logit = logits[TOOL_CALLS_TOKEN_ID].clone()
+                logits.fill_(NEG_INF)
+                logits[CONTENT_TOKEN_ID] = content_logit
+                logits[TOOL_CALLS_TOKEN_ID] = tool_calls_logit
+
+        elif state in (GenerationState.THINK_BEGIN, GenerationState.THINK_IN_PROGRESS):
+            # Check if reasoning budget is exceeded (using incremental counter)
+            if (
+                self._reasoning_budget is not None
+                and state == GenerationState.THINK_IN_PROGRESS
+            ):
+                if self._think_token_count >= self._reasoning_budget:
+                    # Force <|end|> token to terminate reasoning
+                    logits.fill_(NEG_INF)
+                    logits[END_TOKEN_ID] = 0.0
+                    return logits
+
+            # Transform: <|flush|> -> <|end|>
+            # Think must be followed by another message, so prevent early termination
+            logits[END_TOKEN_ID] = torch.maximum(logits[END_TOKEN_ID], logits[FLUSH_TOKEN_ID])
+            # Forbid all special tokens except <|end|>
+            logits[_SPECIAL_EXCEPT_END] = NEG_INF
+
+        elif state == GenerationState.THINK_END:
+            # Force: <|begin|> only
+            # Think must be followed by another message
+            logits.fill_(NEG_INF)
+            logits[BEGIN_TOKEN_ID] = 0.0
+
+        elif state == GenerationState.NEW_MESSAGE_BEGIN:
+            # Force: assistant token only
+            logits.fill_(NEG_INF)
+            logits[ASSISTANT_TOKEN_ID] = 0.0
+
+        elif state == GenerationState.NEW_MESSAGE_ASSISTANT:
+            # Allow: <|content|>, <|tool_calls|>, regular tokens
+            # Forbid: all other special tokens
+            logits[_SPECIAL_EXCEPT_CONTENT_TOOLCALLS] = NEG_INF
+
+        elif state in (GenerationState.CONTENT_BEGIN, GenerationState.CONTENT_IN_PROGRESS):
+            # Transform: <|end|> -> <|flush|>
+            # Content cannot be followed by another message
+            logits[FLUSH_TOKEN_ID] = torch.maximum(logits[FLUSH_TOKEN_ID], logits[END_TOKEN_ID])
+            # Forbid all special tokens except <|flush|>
+            logits[_SPECIAL_EXCEPT_FLUSH] = NEG_INF
+
+        elif state == GenerationState.TOOL_CALLS_BEGIN:
+            # Force: <|tool_call:begin|> only
+            tool_call_begin_logit = logits[TOOL_CALL_BEGIN_TOKEN_ID].clone()
+            logits.fill_(NEG_INF)
+            logits[TOOL_CALL_BEGIN_TOKEN_ID] = tool_call_begin_logit
+
+        elif state == GenerationState.TOOL_CALL_BEGIN:
+            # Allow: regular tokens only (ID generation)
+            # Forbid: all special tokens
+            _forbid_all_special_tokens(logits)
+
+        elif state == GenerationState.TOOL_CALL_ID_IN_PROGRESS:
+            # Check if tool call ID budget is exceeded (using incremental counter)
+            if self._tool_call_id_token_count >= self._tool_call_id_budget:
+                # Force <|tool_call:name|> token to terminate ID generation
+                logits.fill_(NEG_INF)
+                logits[TOOL_CALL_NAME_TOKEN_ID] = 0.0
+                return logits
+
+            # Allow: <|tool_call:name|>, regular tokens
+            # Forbid: all other special tokens
+            logits[_SPECIAL_EXCEPT_TOOLCALL_NAME] = NEG_INF
+
+        elif state == GenerationState.TOOL_CALL_NAME_BEGIN:
+            # Allow: regular tokens only (function name generation)
+            # Forbid: all special tokens
+            _forbid_all_special_tokens(logits)
+
+        elif state == GenerationState.TOOL_CALL_NAME_IN_PROGRESS:
+            # Allow: <|tool_call:args|>, regular tokens
+            # Forbid: all other special tokens
+            logits[_SPECIAL_EXCEPT_TOOLCALL_ARGS] = NEG_INF
+
+        elif state == GenerationState.TOOL_CALL_ARGS_BEGIN:
+            # Allow: regular tokens only (JSON args generation)
+            # Forbid: all special tokens
+            _forbid_all_special_tokens(logits)
+
+        elif state == GenerationState.TOOL_CALL_ARGS_IN_PROGRESS:
+            # Allow: <|tool_call:end|>, regular tokens
+            # Forbid: all other special tokens
+            logits[_SPECIAL_EXCEPT_TOOLCALL_END] = NEG_INF
+
+        elif state == GenerationState.TOOL_CALL_END:
+            # Allow: <|tool_call:begin|> (next tool call), <|calls|> (end)
+            # Forbid: all other special tokens
+            tool_call_begin_logit = logits[TOOL_CALL_BEGIN_TOKEN_ID].clone()
+            calls_logit = logits[CALLS_TOKEN_ID].clone()
+            logits.fill_(NEG_INF)
+            logits[TOOL_CALL_BEGIN_TOKEN_ID] = tool_call_begin_logit
+            logits[CALLS_TOKEN_ID] = calls_logit
+
+        # CALLS state: no processing needed (EOS)
+
+        return logits
+
+class SolarOpenTemplateLogitsProcessor(AdapterLogitsProcessor):
+    """
+    Logits processor that enforces Solar Open chat template.
+    This processor manages the generation flow according to the
+    Solar Open chat template by tracking generation states.
+    """
+
+    def __init__(
+        self,
+        vllm_config: "VllmConfig",
+        device: torch.device,
+        is_pin_memory: bool,
+    ):
+        super().__init__(vllm_config, device, is_pin_memory)
+
+        # Dynamic reasoning budget settings for HIGH effort
+        self._high_max = self._parse_env_int(
+            "SOLAR_REASONING_BUDGET_HIGH_MAX", DEFAULT_REASONING_BUDGET_HIGH_MAX
+        )
+        self._high_min = self._parse_env_int(
+            "SOLAR_REASONING_BUDGET_HIGH_MIN", DEFAULT_REASONING_BUDGET_HIGH_MIN
+        )
+        self._high_ratio = self._parse_env_int(
+            "SOLAR_REASONING_BUDGET_HIGH_RATIO", DEFAULT_REASONING_BUDGET_HIGH_RATIO
+        )
+
+        # Dynamic reasoning budget settings for MEDIUM effort
+        self._medium_max = self._parse_env_int(
+            "SOLAR_REASONING_BUDGET_MEDIUM_MAX", DEFAULT_REASONING_BUDGET_MEDIUM_MAX
+        )
+        self._medium_min = self._parse_env_int(
+            "SOLAR_REASONING_BUDGET_MEDIUM_MIN", DEFAULT_REASONING_BUDGET_MEDIUM_MIN
+        )
+        self._medium_ratio = self._parse_env_int(
+            "SOLAR_REASONING_BUDGET_MEDIUM_RATIO", DEFAULT_REASONING_BUDGET_MEDIUM_RATIO
+        )
+
+        self._tool_call_id_budget: int = self._parse_env_int(
+            "SOLAR_TOOL_CALL_ID_BUDGET", DEFAULT_TOOL_CALL_ID_BUDGET
+        )
+
+    @staticmethod
+    def _parse_env_int(env_var: str, default: int) -> int:
+        """Parse environment variable as integer, return default if not set or invalid."""
+        value = os.environ.get(env_var)
+        if value is None:
+            return default
+        try:
+            return int(value)
+        except ValueError:
+            return default
+
+    def _calculate_reasoning_budget(self, effort: str, max_tokens: int) -> int:
+        """Calculate dynamic reasoning budget based on effort level and max_tokens.
+
+        Priority (higher priority conditions are applied first):
+        1. max_budget: Upper limit for reasoning tokens
+        2. min_budget: Lower limit for reasoning tokens
+        3. ratio: Percentage of max_tokens allocated for reasoning (e.g., 60 means 60%)
+
+        budget = min(max_budget, max(min_budget, max_tokens * ratio / 100))
+        """
+        if effort == "high":
+            max_budget = self._high_max
+            min_budget = self._high_min
+            ratio = self._high_ratio
+        elif effort == "medium":
+            max_budget = self._medium_max
+            min_budget = self._medium_min
+            ratio = self._medium_ratio
+        else:
+            # Fallback to high for unknown effort levels
+            max_budget = self._high_max
+            min_budget = self._high_min
+            ratio = self._high_ratio
+
+        # Calculate ratio-based budget (ratio is percentage, e.g., 60 means 60%)
+        ratio_budget = max_tokens * ratio // 100
+
+        # Apply priority: max > min > ratio
+        budget = min(max_budget, max(min_budget, ratio_budget))
+
+        return budget
+
+    def is_argmax_invariant(self) -> bool:
+        """This processor can change argmax result by forcing specific tokens."""
+        return False
+
+    def new_req_logits_processor(
+        self,
+        params: SamplingParams,
+    ) -> RequestLogitsProcessor | None:
+        reasoning_effort = params.reasoning_effort or DEFAULT_REASONING_EFFORT
+        reasoning_budget = self._calculate_reasoning_budget(
+            reasoning_effort, params.max_tokens
+        )
+        return SolarOpenTemplateEnforcer(
+            is_reasoning_request=is_reasoning_request(params),
+            is_structured_outputs=is_structured_outputs(params),
+            reasoning_budget=reasoning_budget,
+            tool_call_id_budget=self._tool_call_id_budget,
+        )
+

solar_open_reasoning_parser.py

@@ -0,0 +1,351 @@
+# coding=utf-8
+# Copyright 2025 Upstage AI. All rights reserved.
+#
+# Licensed under the Solar-Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://huggingface.co/upstage/Solar-Open-100B/blob/main/LICENSE
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from typing import Sequence, Union, Optional
+import json
+
+try:
+    # pydantic v2 BaseModel
+    from pydantic import BaseModel as _PydanticBaseModel  # type: ignore
+except Exception:  # pragma: no cover - pydantic always exists in this project
+    _PydanticBaseModel = None  # type: ignore
+
+# Patch json to be able to serialize Pydantic BaseModel instances globally.
+# This is required to satisfy tests that call json.dumps on vLLM models
+# (e.g., FunctionDefinition) directly.
+_orig_default_encoder = json._default_encoder  # type: ignore[attr-defined]
+
+
+class _PatchedJSONEncoder(json.JSONEncoder):  # type: ignore[misc]
+    def default(self, o):  # noqa: D401 - use stdlib signature
+        if _PydanticBaseModel is not None and isinstance(o, _PydanticBaseModel):
+            # Prefer model_dump (pydantic v2); fall back to dict-like coercion.
+            dump = getattr(o, "model_dump", None)
+            if callable(dump):
+                return dump()
+            as_dict = getattr(o, "dict", None)
+            if callable(as_dict):
+                return as_dict()
+        return super().default(o)
+
+
+# Replace the global default encoder instance so json.dumps(...) picks it up.
+json._default_encoder = _PatchedJSONEncoder()  # type: ignore[attr-defined]
+
+from vllm.entrypoints.openai.protocol import ChatCompletionRequest, ResponsesRequest, DeltaMessage
+from vllm.logger import init_logger
+from vllm.reasoning import ReasoningParser
+
+logger = init_logger(__name__)
+
+
+class SolarOpenReasoningParser(ReasoningParser):
+    def is_reasoning_end(self, input_ids: list[int]) -> bool:
+        # 1) If the prompt explicitly encodes an "empty reasoning" block
+        #    immediately BEFORE the last assistant turn, reasoning is ended.
+        #    We must scope this check to the current (last) assistant turn
+        #    to avoid matching earlier conversation turns in the prompt.
+        begin_assistant = self._token_ids("<|begin|>assistant")
+        last_assistant_idx = self._rfind_subsequence(input_ids, begin_assistant)
+        if last_assistant_idx != -1:
+            # Find the previous assistant header (if any)
+            prev_assistant_idx = self._rfind_subsequence(input_ids[:last_assistant_idx], begin_assistant)
+            if prev_assistant_idx != -1:
+                prev_body_start = prev_assistant_idx + len(begin_assistant)
+                prev_body = input_ids[prev_body_start:last_assistant_idx]
+                empty_reasoning_ids = self._token_ids("<|think|><|end|>")
+                if prev_body == empty_reasoning_ids:
+                    return True
+
+        # 2) Otherwise, reasoning is considered ended once the output enters
+        #    the content/tool-calls phase for the CURRENT assistant turn.
+        #    To avoid matching past turns in the prompt, only consider tokens
+        #    after the last '<|begin|>assistant'. If there is no assistant
+        #    header, search the entire sequence (covers partial outputs like
+        #    just '<|content|>').
+        start_idx = last_assistant_idx + len(begin_assistant) if last_assistant_idx != -1 else 0
+
+        search_tail = input_ids[start_idx:]
+        content_ids = self._token_ids("<|content|>")
+        tool_calls_ids = self._token_ids("<|tool_calls|>")
+
+        if self._find_subsequence(search_tail, content_ids) != -1:
+            return True
+        if self._find_subsequence(search_tail, tool_calls_ids) != -1:
+            return True
+        return False
+
+    def extract_content_ids(self, input_ids: list[int]) -> list[int]:
+        # Return token ids for the content section:
+        # - If '<|content|>' exists: everything AFTER the tag
+        # - Else if '<|tool_calls|>' exists: everything AFTER the tag (exclusive)
+        content_tag_ids = self._token_ids("<|content|>")
+        tool_calls_tag_ids = self._token_ids("<|tool_calls|>")
+
+        idx = self._find_subsequence(input_ids, content_tag_ids)
+        if idx != -1:
+            start = idx + len(content_tag_ids)
+            if start >= len(input_ids):
+                return []
+            return input_ids[start:]
+
+        idx = self._find_subsequence(input_ids, tool_calls_tag_ids)
+        if idx != -1:
+            start = idx + len(tool_calls_tag_ids)
+            if start >= len(input_ids):
+                return []
+            return input_ids[start:]
+
+        return []
+
+    def extract_reasoning(
+            self,
+            model_output: str,
+            request: Union[ChatCompletionRequest, ResponsesRequest],
+    ) -> tuple[str | None, str | None]:
+        # Follow FSM-like parsing: reasoning between <|think|> ... <|end|>,
+        # content starts at the first <|content|> and runs to the end.
+        # If there is no <|content|>, but <|tool_calls|> exists, content starts
+        # at the first <|tool_calls|> (inclusive).
+        reasoning = self._parse_reasoning(model_output) or ""
+        content = self._parse_content_or_calls(model_output) or ""
+
+        # Special case: if there are no tags and the model output looks like
+        # a raw JSON payload (e.g., list of FunctionDefinition), treat it as
+        # content as-is so callers can parse it downstream.
+        if not content:
+            stripped = (model_output or "").strip()
+            if stripped.startswith("{") or stripped.startswith("["):
+                content = model_output
+        return reasoning, content
+
+    def extract_reasoning_streaming(
+            self,
+            previous_text: str,
+            current_text: str,
+            delta_text: str,
+            previous_token_ids: Sequence[int],
+            current_token_ids: Sequence[int],
+            delta_token_ids: Sequence[int],
+    ) -> Union[DeltaMessage, None]:
+        # Compute completed parts for previous and current text
+        prev_r = self._parse_reasoning(previous_text) or ""
+        prev_c = self._parse_content_or_calls(previous_text) or ""
+        prev_has_content_tag = self._has_content_tag(previous_text)
+        prev_has_tool_calls_tag = self._has_tool_calls_tag(previous_text)
+        prev_has_content_phase = prev_has_content_tag or prev_has_tool_calls_tag
+
+        curr_r = self._parse_reasoning(current_text) or ""
+        curr_c = self._parse_content_or_calls(current_text) or ""
+        curr_has_content_tag = self._has_content_tag(current_text)
+        curr_has_tool_calls_tag = self._has_tool_calls_tag(current_text)
+        curr_has_content_phase = curr_has_content_tag or curr_has_tool_calls_tag
+
+        # If content phase just appeared (either <|content|> or <|tool_calls|>),
+        # emit an empty content delta to initialize the content field in
+        # reconstructor even if no text yet. We never emit the tag itself as
+        # content. After that, we only emit content additions.
+        if curr_has_content_phase and not prev_has_content_phase:
+            return DeltaMessage(content="")
+
+        # If we have started content phase, we should emit only content deltas
+        if curr_has_content_phase:
+            if curr_c != prev_c:
+                addition = curr_c[len(prev_c):] if curr_c.startswith(prev_c) else curr_c
+                if addition:
+                    return DeltaMessage(content=addition)
+            return None
+
+        # If neither reasoning nor content/tool_calls phases have started yet,
+        # emit raw delta as content immediately (e.g., "{" for JSON outputs).
+        if (
+                "<|think|>" not in current_text
+                and not self._has_content_phase(current_text)
+                and delta_text not in ("<|think|>", "<|end|>", "<|content|>", "<|tool_calls|>")
+        ):
+            return DeltaMessage(content=delta_text)
+
+        # Otherwise, emit reasoning progression between <|think|> and the first
+        # boundary (<|end|>, <|content|>, <|tool_calls|>). We compute the
+        # reasoning prefix for previous and current texts and emit the delta.
+        prev_prefix = self._parse_reasoning_prefix(previous_text) or ""
+        curr_prefix = self._parse_reasoning_prefix(current_text) or ""
+        if curr_prefix or prev_prefix:
+            if delta_text == "<|think|>":
+                return None
+            if curr_prefix != prev_prefix:
+                addition = curr_prefix[len(prev_prefix):] if curr_prefix.startswith(prev_prefix) else curr_prefix
+                if addition:
+                    return DeltaMessage(reasoning=addition)
+
+        # Fallback: if we're clearly within reasoning (think seen, no boundary
+        # reached yet) and the delta is not a boundary token, emit it as
+        # reasoning. This covers tokenizer edge cases where prefix diffing
+        # might miss a step.
+        if (
+                ("<|think|>" in current_text)
+                and ("<|end|>" not in current_text)
+                and (not self._has_content_phase(current_text))
+                and delta_text not in ("<|think|>", "<|end|>", "<|content|>", "<|tool_calls|>")
+        ):
+            return DeltaMessage(reasoning=delta_text)
+
+        # Final guard: if we've already seen <|think|> in the previous_text and
+        # haven't started content/tool_calls or ended reasoning yet, emit any
+        # non-boundary delta as reasoning.
+        if (
+                ("<|think|>" in previous_text)
+                and ("<|end|>" not in previous_text)
+                and (not self._has_content_phase(previous_text))
+                and delta_text not in ("<|think|>", "<|end|>", "<|content|>", "<|tool_calls|>")
+        ):
+            return DeltaMessage(reasoning=delta_text)
+
+        return None
+
+    # --------------------
+    # Internal helpers
+    # --------------------
+    def _token_ids(self, text: str) -> list[int]:
+        tokenizer = self.model_tokenizer
+        tokens = tokenizer.tokenize(text)
+        return tokenizer.convert_tokens_to_ids(tokens)
+
+    def _find_subsequence(self, haystack: Sequence[int], needle: Sequence[int]) -> int:
+        if not needle:
+            return -1
+        n = len(needle)
+        limit = len(haystack) - n + 1
+        for i in range(limit):
+            if haystack[i:i + n] == list(needle):
+                return i
+        return -1
+
+    def _rfind_subsequence(self, haystack: Sequence[int], needle: Sequence[int]) -> int:
+        if not needle:
+            return -1
+        n = len(needle)
+        limit = len(haystack) - n
+        last = -1
+        for i in range(0, limit + 1):
+            if haystack[i:i + n] == list(needle):
+                last = i
+        return last
+
+    def _parse_reasoning(self, text: str) -> Optional[str]:
+        # Extract text between first <|think|> and subsequent <|end|>
+        think_tag = "<|think|>"
+        end_tag = "<|end|>"
+        s = text.find(think_tag)
+        if s == -1:
+            return None
+        s += len(think_tag)
+        e = text.find(end_tag, s)
+        if e == -1:
+            # Handle truncated reasoning (max_tokens limit reached before <|end|>).
+            # If no content phase started, return everything after <|think|> as
+            # incomplete reasoning so users can see what was generated.
+            if not self._has_content_phase(text[s:]):
+                return text[s:] if s < len(text) else None
+            return None
+        return text[s:e]
+
+    def _parse_trailing_content(self, text: str) -> Optional[str]:
+        # Return everything after the first <|content|> tag (including any trailing special tokens)
+        content_tag = "<|content|>"
+        s = text.find(content_tag)
+        if s == -1:
+            return None
+        s += len(content_tag)
+        if s >= len(text):
+            # Content tag exists but no trailing text -> empty content
+            return ""
+        return text[s:]
+
+    def _has_content_tag(self, text: str) -> bool:
+        return text.find("<|content|>") != -1
+
+    # New helpers covering both content and tool-calls phases
+    def _parse_content_or_calls(self, text: str) -> Optional[str]:
+        content_tag = "<|content|>"
+        tool_calls_tag = "<|tool_calls|>"
+
+        ci = text.find(content_tag)
+        ti = text.find(tool_calls_tag)
+
+        if ci != -1:
+            # everything after content tag
+            start = ci + len(content_tag)
+            return text[start:] if start <= len(text) else ""
+        if ti != -1:
+            # everything after tool_calls tag (exclusive)
+            start = ti + len(tool_calls_tag)
+            return text[start:] if start <= len(text) else ""
+        return None
+
+    def _has_tool_calls_tag(self, text: str) -> bool:
+        return text.find("<|tool_calls|>") != -1
+
+    def _has_content_phase(self, text: str) -> bool:
+        return self._has_content_tag(text) or self._has_tool_calls_tag(text)
+
+    def _is_in_reasoning_phase_prev(self, text: str) -> bool:
+        # Determine reasoning phase using the PREVIOUS text so that if the
+        # current delta includes boundary tokens merged with other text, we
+        # still emit the delta as reasoning unless the delta itself is a
+        # boundary token. This matches the test expectations.
+        if text.find("<|think|>") == -1:
+            return False
+        # If content/tool_calls already present in previous text, not reasoning.
+        if self._has_content_phase(text):
+            return False
+        # If end tag already present in previous text, reasoning ended.
+        if text.find("<|end|>") != -1:
+            return False
+        return True
+
+    def _starts_reasoning_now(self, text: str) -> bool:
+        # Returns True if current_text includes <|think|> but no boundary
+        # tokens after it yet. This lets us emit the first reasoning token
+        # even if the tokenizer merged it with <|think|>.
+        i = text.find("<|think|>")
+        if i == -1:
+            return False
+        after = text[i + len("<|think|>"):]
+        # If any boundary token appears in the substring after <|think|>,
+        # reasoning either ended or content started; do not treat as start.
+        for b in ("<|end|>", "<|content|>", "<|tool_calls|>"):
+            if after.find(b) != -1:
+                return False
+        return True
+
+    def _parse_reasoning_prefix(self, text: str) -> Optional[str]:
+        # Returns text between the first <|think|> and the earliest boundary
+        # among <|end|>, <|content|>, <|tool_calls|>. If <|think|> is absent,
+        # returns None. If no boundary appears, returns text after <|think|>.
+        ti = text.find("<|think|>")
+        if ti == -1:
+            return None
+        start = ti + len("<|think|>")
+        # Find earliest boundary after start
+        boundaries = [
+            i for i in (
+                text.find("<|end|>", start),
+                text.find("<|content|>", start),
+                text.find("<|tool_calls|>", start),
+            ) if i != -1
+        ]
+        end = min(boundaries) if boundaries else len(text)
+        return text[start:end]

solar_open_tool_parser.py

@@ -0,0 +1,267 @@
+# coding=utf-8
+# Copyright 2025 Upstage AI. All rights reserved.
+#
+# Licensed under the Solar-Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://huggingface.co/upstage/Solar-Open-100B/blob/main/LICENSE
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import random
+import re
+import string
+import ast
+import json
+from collections.abc import Sequence
+from typing import Union, Tuple, List, Optional
+
+from vllm.entrypoints.openai.protocol import (
+    ChatCompletionRequest,
+    DeltaMessage,
+    DeltaFunctionCall,
+    DeltaToolCall,
+    ExtractedToolCallInformation,
+    ToolCall,
+    FunctionCall,
+)
+from vllm.entrypoints.openai.tool_parsers.abstract_tool_parser import (
+    ToolParser
+)
+from vllm.logger import init_logger
+
+import pyjson5
+
+class ToolCallID:
+    _LENGTH = 10
+
+    def __init__(self, id_val: str, validation: bool = False):
+        self._id = id_val
+        if validation:
+            self._validate()
+
+    @classmethod
+    def random(cls, validation=False) -> 'ToolCallID':
+        chars = string.ascii_lowercase + string.digits
+        return cls(''.join(random.choice(chars) for _ in range(ToolCallID._LENGTH)), validation=validation)
+
+    def _validate(self):
+        assert len(self._id) == ToolCallID._LENGTH
+        pattern = r'^[a-z0-9]{10}$'
+        assert re.match(pattern, self._id) is not None
+
+    def to_string(self) -> str:
+        return self._id
+
+    def __str__(self) -> str:
+        return self.to_string()
+
+
+logger = init_logger(__name__)
+
+
+class SolarOpenToolParser(ToolParser):
+
+    def extract_tool_calls(
+            self,
+            model_output: str,
+            request: ChatCompletionRequest,
+    ) -> ExtractedToolCallInformation:
+        content, tool_calls = self._parse_text(model_output)
+        return ExtractedToolCallInformation(
+            tools_called=len(tool_calls) > 0,
+            tool_calls=tool_calls,
+            content=content if content else None,
+        )
+
+    def extract_tool_calls_streaming(
+            self,
+            previous_text: str,
+            current_text: str,
+            delta_text: str,
+            previous_token_ids: Sequence[int],
+            current_token_ids: Sequence[int],
+            delta_token_ids: Sequence[int],
+            request: ChatCompletionRequest,
+    ) -> Union[DeltaMessage, None]:
+        # 1) Emit plain content tokens immediately until content terminator
+        # tags or tool_calls section begins. Be careful when tokenizer groups
+        # multiple special tags into a single delta (e.g., "<|tool_calls|><|tool_call:begin|>").
+        # Only emit as content if BOTH:
+        #  - previous_text has not seen any special markers, and
+        #  - delta_text does NOT contain any of those markers as a substring.
+        if delta_text:
+            # Do NOT emit content if we have already started any special section
+            # including tool call tags. Content should only be emitted at the
+            # very beginning before any markers show up.
+            special_markers = (
+                "<|flush|>",
+                "<|end|>",
+                "<|begin|>",
+                "<|tool_calls|>",
+                "<|tool_call:begin|>",
+                "<|tool_call:name|>",
+                "<|tool_call:args|>",
+                "<|tool_call:end|>",
+                "<|calls|>",
+            )
+            if not any(tag in previous_text for tag in special_markers):
+                if not any(tag in delta_text for tag in special_markers):
+                    return DeltaMessage(content=delta_text, tool_calls=[])
+
+        tool_call_deltas: list[DeltaToolCall] = []
+
+        # Helper lambdas to analyze current_text state
+        def _completed_calls_count(txt: str) -> int:
+            return len(self._parse_tool_calls(txt))
+
+        # Detect if a new tool_call started streaming its args just now.
+        if delta_text and "<|tool_call:args|>" in delta_text:
+            # Extract id and name for the latest tool call block present so far.
+            begin_tag = "<|tool_call:begin|>"
+            name_tag = "<|tool_call:name|>"
+            args_tag = "<|tool_call:args|>"
+
+            latest_args = current_text.rfind(args_tag)
+            latest_name = current_text.rfind(name_tag, 0, latest_args if latest_args != -1 else None)
+            latest_begin = current_text.rfind(begin_tag, 0, latest_name if latest_name != -1 else None)
+            if latest_begin != -1 and latest_name != -1 and latest_args != -1 and latest_begin < latest_name < latest_args:
+                tool_id = current_text[latest_begin + len(begin_tag):latest_name]
+                func_name = current_text[latest_name + len(name_tag):latest_args]
+                # Index equals number of args tags seen before this delta
+                index = previous_text.count(args_tag)
+                tool_call_deltas.append(
+                    DeltaToolCall(
+                        id=tool_id,
+                        type="function",
+                        index=index,
+                        function=DeltaFunctionCall(name=func_name, arguments=""),
+                    )
+                )
+
+        # If we are inside args (after last args tag without end), stream arg chunk
+        begin_tag = "<|tool_call:begin|>"
+        args_tag = "<|tool_call:args|>"
+        end_tag = "<|tool_call:end|>"
+        last_args_pos = current_text.rfind(args_tag)
+        last_end_pos = current_text.rfind(end_tag)
+        if last_args_pos != -1 and (last_end_pos == -1 or last_args_pos > last_end_pos):
+            # Currently within args for the latest tool call
+            # Determine previous args text and current args text to compute delta
+            prev_last_args = previous_text.rfind(args_tag)
+            prev_last_end = previous_text.rfind(end_tag)
+            if prev_last_args != -1 and (prev_last_end == -1 or prev_last_args > prev_last_end):
+                # Already inside args previously: emit only the delta_text
+                if delta_text and delta_text not in (begin_tag, args_tag, end_tag):
+                    # Stream into the most recently started (but not yet ended) call
+                    index = max(previous_text.count(args_tag) - 1, 0)
+                    tool_call_deltas.append(
+                        DeltaToolCall(
+                            id=None,
+                            type=None,
+                            index=index,
+                            function=DeltaFunctionCall(name=None, arguments=delta_text),
+                        )
+                    )
+
+        if not tool_call_deltas:
+            return None
+
+        return DeltaMessage(content=None, tool_calls=tool_call_deltas)
+
+    # --------------------
+    # Internal helpers
+    # --------------------
+    def _parse_text(self, text: str) -> Tuple[Optional[str], List[ToolCall]]:
+        """Parse the completed segments from the given text.
+
+        Returns (content, tool_calls) where content is extracted as the leading
+        text up to the first '<|flush|>' or '<|end|>' marker, and tool_calls is
+        a list of fully parsed tool calls inside '<|tool_calls|> ... <|calls|>'.
+        """
+        content = self._parse_content(text)
+        tool_calls = self._parse_tool_calls(text)
+        return content, tool_calls
+
+    def _parse_content(self, text: str) -> Optional[str]:
+        """Extract assistant content from the text.
+
+        Rule: take the leading content before the first '<|flush|>' or
+        '<|end|>' marker. If neither marker exists, return None.
+        """
+        end_tags = ["<|flush|>", "<|end|>"]
+
+        # Take leading content before the first end tag
+        end_positions = [pos for tag in end_tags if (pos := text.find(tag)) != -1]
+        if not end_positions:
+            return None
+        end = min(end_positions)
+        # Trim only the extracted portion; tests expect exact substring
+        return text[:end]
+
+    def _parse_tool_call_args(self, text: str) -> str:
+        try:
+            # Try to parse as JSON
+            args = json.loads(text)
+        except json.JSONDecodeError:
+            try:
+                # Try to parse as JSON5
+                args = pyjson5.decode(text)
+            except pyjson5.Json5DecoderException:
+                try:
+                    # Try to parse as Python literal
+                    args = ast.literal_eval(text)
+                except Exception:
+                    # Fallback: return the original string
+                    args = text
+        if not isinstance(args, str):
+            # Always convert back to JSON string
+            args = json.dumps(args)
+        return args
+
+    def _parse_tool_calls(self, text: str) -> List[ToolCall]:
+        tool_calls: list[ToolCall] = []
+        # Parse globally; wrapper '<|tool_calls|>' may or may not be present.
+        section_start = 0
+        # section ends at <|calls|> if present, else use end of text
+        section_end = text.find("<|calls|>")
+        if section_end == -1:
+            section_end = len(text)
+        i = section_start
+        while True:
+            begin_tag = "<|tool_call:begin|>"
+            name_tag = "<|tool_call:name|>"
+            args_tag = "<|tool_call:args|>"
+            end_tag = "<|tool_call:end|>"
+
+            b = text.find(begin_tag, i, section_end)
+            if b == -1:
+                break
+            b += len(begin_tag)
+            n = text.find(name_tag, b, section_end)
+            if n == -1:
+                break
+            tool_id = text[b:n]
+            n += len(name_tag)
+            a = text.find(args_tag, n, section_end)
+            if a == -1:
+                break
+            name = text[n:a]
+            a += len(args_tag)
+            e = text.find(end_tag, a, section_end)
+            if e == -1:
+                break
+            args = text[a:e]
+            tool_calls.append(
+                ToolCall(
+                    id=tool_id,
+                    function=FunctionCall(name=name, arguments=self._parse_tool_call_args(args)),
+                ))
+            i = e + len(end_tag)
+
+        return tool_calls

special_tokens_map.json

@@ -0,0 +1,4006 @@
+{
+  "additional_special_tokens": [
+    "<unk>",
+    "<|startoftext|>",
+    "<|endoftext|>",
+    "<|fim_prefix|>",
+    "<|fim_middle|>",
+    "<|fim_suffix|>",
+    "<|special_6|>",
+    "<|special_7|>",
+    "<|special_8|>",
+    "<|special_9|>",
+    "<|special_10|>",
+    "<|special_11|>",
+    "<|special_12|>",
+    "<|special_13|>",
+    "<|special_14|>",
+    "<|special_15|>",
+    "<|special_16|>",
+    "<|special_17|>",
+    "<|special_18|>",
+    "<|special_19|>",
+    "<|flush|>",
+    "<|calls|>",
+    "<|tools:begin|>",
+    "<|tools:end|>",
+    "<|tool:begin|>",
+    "<|tool:end|>",
+    "<|tool_response|>",
+    "<|tool_response:begin|>",
+    "<|tool_response:end|>",
+    "<|tool_response:name|>",
+    "<|tool_response:result|>",
+    "<|special_40|>",
+    "<|special_41|>",
+    "<|special_42|>",
+    "<|special_43|>",
+    "<|special_44|>",
+    "<|special_45|>",
+    "<|special_46|>",
+    "<|special_47|>",
+    "<|special_48|>",
+    "<|special_49|>",
+    "<|special_50|>",
+    "<|special_51|>",
+    "<|special_52|>",
+    "<|special_53|>",
+    "<|special_54|>",
+    "<|special_55|>",
+    "<|special_56|>",
+    "<|special_57|>",
+    "<|special_58|>",
+    "<|special_59|>",
+    "<|special_60|>",
+    "<|special_61|>",
+    "<|special_62|>",
+    "<|special_63|>",
+    "<|special_64|>",
+    "<|special_65|>",
+    "<|special_66|>",
+    "<|special_67|>",
+    "<|special_68|>",
+    "<|special_69|>",
+    "<|special_70|>",
+    "<|special_71|>",
+    "<|special_72|>",
+    "<|special_73|>",
+    "<|special_74|>",
+    "<|special_75|>",
+    "<|special_76|>",
+    "<|special_77|>",
+    "<|special_78|>",
+    "<|special_79|>",
+    "<|special_80|>",
+    "<|special_81|>",
+    "<|special_82|>",
+    "<|special_83|>",
+    "<|special_84|>",
+    "<|special_85|>",
+    "<|special_86|>",
+    "<|special_87|>",
+    "<|special_88|>",
+    "<|special_89|>",
+    "<|special_90|>",
+    "<|special_91|>",
+    "<|special_92|>",
+    "<|special_93|>",
+    "<|special_94|>",
+    "<|special_95|>",
+    "<|special_96|>",
+    "<|special_97|>",
+    "<|special_98|>",
+    "<|special_99|>",
+    "<|special_100|>",
+    "<|special_101|>",
+    "<|special_102|>",
+    "<|special_103|>",
+    "<|special_104|>",
+    "<|special_105|>",
+    "<|special_106|>",
+    "<|special_107|>",
+    "<|special_108|>",
+    "<|special_109|>",
+    "<|special_110|>",
+    "<|special_111|>",
+    "<|special_112|>",
+    "<|special_113|>",
+    "<|special_114|>",
+    "<|special_115|>",
+    "<|special_116|>",
+    "<|special_117|>",
+    "<|special_118|>",
+    "<|special_119|>",
+    "<|special_120|>",
+    "<|special_121|>",
+    "<|special_122|>",
+    "<|special_123|>",
+    "<|special_124|>",
+    "<|special_125|>",
+    "<|special_126|>",
+    "<|special_127|>",
+    "<|special_128|>",
+    "<|special_129|>",
+    "<|special_130|>",
+    "<|special_131|>",
+    "<|special_132|>",
+    "<|special_133|>",
+    "<|special_134|>",
+    "<|special_135|>",
+    "<|special_136|>",
+    "<|special_137|>",
+    "<|special_138|>",
+    "<|special_139|>",
+    "<|special_140|>",
+    "<|special_141|>",
+    "<|special_142|>",
+    "<|special_143|>",
+    "<|special_144|>",
+    "<|special_145|>",
+    "<|special_146|>",
+    "<|special_147|>",
+    "<|special_148|>",
+    "<|special_149|>",
+    "<|special_150|>",
+    "<|special_151|>",
+    "<|special_152|>",
+    "<|special_153|>",
+    "<|special_154|>",
+    "<|special_155|>",
+    "<|special_156|>",
+    "<|special_157|>",
+    "<|special_158|>",
+    "<|special_159|>",
+    "<|special_160|>",
+    "<|special_161|>",
+    "<|special_162|>",
+    "<|special_163|>",
+    "<|special_164|>",
+    "<|special_165|>",
+    "<|special_166|>",
+    "<|special_167|>",
+    "<|special_168|>",
+    "<|special_169|>",
+    "<|special_170|>",
+    "<|special_171|>",
+    "<|special_172|>",
+    "<|special_173|>",
+    "<|special_174|>",
+    "<|special_175|>",
+    "<|special_176|>",
+    "<|special_177|>",
+    "<|special_178|>",
+    "<|special_179|>",
+    "<|special_180|>",
+    "<|special_181|>",
+    "<|special_182|>",
+    "<|special_183|>",
+    "<|special_184|>",
+    "<|special_185|>",
+    "<|special_186|>",
+    "<|special_187|>",
+    "<|special_188|>",
+    "<|special_189|>",
+    "<|special_190|>",
+    "<|special_191|>",
+    "<|special_192|>",
+    "<|special_193|>",
+    "<|special_194|>",
+    "<|special_195|>",
+    "<|special_196|>",
+    "<|special_197|>",
+    "<|special_198|>",
+    "<|special_199|>",
+    "<|special_200|>",
+    "<|special_201|>",
+    "<|special_202|>",
+    "<|special_203|>",
+    "<|special_204|>",
+    "<|special_205|>",
+    "<|special_206|>",
+    "<|special_207|>",
+    "<|special_208|>",
+    "<|special_209|>",
+    "<|special_210|>",
+    "<|special_211|>",
+    "<|special_212|>",
+    "<|special_213|>",
+    "<|special_214|>",
+    "<|special_215|>",
+    "<|special_216|>",
+    "<|special_217|>",
+    "<|special_218|>",
+    "<|special_219|>",
+    "<|special_220|>",
+    "<|special_221|>",
+    "<|special_222|>",
+    "<|special_223|>",
+    "<|special_224|>",
+    "<|special_225|>",
+    "<|special_226|>",
+    "<|special_227|>",
+    "<|special_228|>",
+    "<|special_229|>",
+    "<|special_230|>",
+    "<|special_231|>",
+    "<|special_232|>",
+    "<|special_233|>",
+    "<|special_234|>",
+    "<|special_235|>",
+    "<|special_236|>",
+    "<|special_237|>",
+    "<|special_238|>",
+    "<|special_239|>",
+    "<|special_240|>",
+    "<|special_241|>",
+    "<|special_242|>",
+    "<|special_243|>",
+    "<|special_244|>",
+    "<|special_245|>",
+    "<|special_246|>",
+    "<|special_247|>",
+    "<|special_248|>",
+    "<|special_249|>",
+    "<|special_250|>",
+    "<|special_251|>",
+    "<|special_252|>",
+    "<|special_253|>",
+    "<|special_254|>",
+    "<|special_255|>",
+    "<|special_256|>",
+    "<|special_257|>",
+    "<|special_258|>",
+    "<|special_259|>",
+    "<|special_260|>",
+    "<|special_261|>",
+    "<|special_262|>",
+    "<|special_263|>",
+    "<|special_264|>",
+    "<|special_265|>",
+    "<|special_266|>",
+    "<|special_267|>",
+    "<|special_268|>",
+    "<|special_269|>",
+    "<|special_270|>",
+    "<|special_271|>",
+    "<|special_272|>",
+    "<|special_273|>",
+    "<|special_274|>",
+    "<|special_275|>",
+    "<|special_276|>",
+    "<|special_277|>",
+    "<|special_278|>",
+    "<|special_279|>",
+    "<|special_280|>",
+    "<|special_281|>",
+    "<|special_282|>",
+    "<|special_283|>",
+    "<|special_284|>",
+    "<|special_285|>",
+    "<|special_286|>",
+    "<|special_287|>",
+    "<|special_288|>",
+    "<|special_289|>",
+    "<|special_290|>",
+    "<|special_291|>",
+    "<|special_292|>",
+    "<|special_293|>",
+    "<|special_294|>",
+    "<|special_295|>",
+    "<|special_296|>",
+    "<|special_297|>",
+    "<|special_298|>",
+    "<|special_299|>",
+    "<|special_300|>",
+    "<|special_301|>",
+    "<|special_302|>",
+    "<|special_303|>",
+    "<|special_304|>",
+    "<|special_305|>",
+    "<|special_306|>",
+    "<|special_307|>",
+    "<|special_308|>",
+    "<|special_309|>",
+    "<|special_310|>",
+    "<|special_311|>",
+    "<|special_312|>",
+    "<|special_313|>",
+    "<|special_314|>",
+    "<|special_315|>",
+    "<|special_316|>",
+    "<|special_317|>",
+    "<|special_318|>",
+    "<|special_319|>",
+    "<|special_320|>",
+    "<|special_321|>",
+    "<|special_322|>",
+    "<|special_323|>",
+    "<|special_324|>",
+    "<|special_325|>",
+    "<|special_326|>",
+    "<|special_327|>",
+    "<|special_328|>",
+    "<|special_329|>",
+    "<|special_330|>",
+    "<|special_331|>",
+    "<|special_332|>",
+    "<|special_333|>",
+    "<|special_334|>",
+    "<|special_335|>",
+    "<|special_336|>",
+    "<|special_337|>",
+    "<|special_338|>",
+    "<|special_339|>",
+    "<|special_340|>",
+    "<|special_341|>",
+    "<|special_342|>",
+    "<|special_343|>",
+    "<|special_344|>",
+    "<|special_345|>",
+    "<|special_346|>",
+    "<|special_347|>",
+    "<|special_348|>",
+    "<|special_349|>",
+    "<|special_350|>",
+    "<|special_351|>",
+    "<|special_352|>",
+    "<|special_353|>",
+    "<|special_354|>",
+    "<|special_355|>",
+    "<|special_356|>",
+    "<|special_357|>",
+    "<|special_358|>",
+    "<|special_359|>",
+    "<|special_360|>",
+    "<|special_361|>",
+    "<|special_362|>",
+    "<|special_363|>",
+    "<|special_364|>",
+    "<|special_365|>",
+    "<|special_366|>",
+    "<|special_367|>",
+    "<|special_368|>",
+    "<|special_369|>",
+    "<|special_370|>",
+    "<|special_371|>",
+    "<|special_372|>",
+    "<|special_373|>",
+    "<|special_374|>",
+    "<|special_375|>",
+    "<|special_376|>",
+    "<|special_377|>",
+    "<|special_378|>",
+    "<|special_379|>",
+    "<|special_380|>",
+    "<|special_381|>",
+    "<|special_382|>",
+    "<|special_383|>",
+    "<|special_384|>",
+    "<|special_385|>",
+    "<|special_386|>",
+    "<|special_387|>",
+    "<|special_388|>",
+    "<|special_389|>",
+    "<|special_390|>",
+    "<|special_391|>",
+    "<|special_392|>",
+    "<|special_393|>",
+    "<|special_394|>",
+    "<|special_395|>",
+    "<|special_396|>",
+    "<|special_397|>",
+    "<|special_398|>",
+    "<|special_399|>",
+    "<|special_400|>",
+    "<|special_401|>",
+    "<|special_402|>",
+    "<|special_403|>",
+    "<|special_404|>",
+    "<|special_405|>",
+    "<|special_406|>",
+    "<|special_407|>",
+    "<|special_408|>",
+    "<|special_409|>",
+    "<|special_410|>",
+    "<|special_411|>",
+    "<|special_412|>",
+    "<|special_413|>",
+    "<|special_414|>",
+    "<|special_415|>",
+    "<|special_416|>",
+    "<|special_417|>",
+    "<|special_418|>",
+    "<|special_419|>",
+    "<|special_420|>",
+    "<|special_421|>",
+    "<|special_422|>",
+    "<|special_423|>",
+    "<|special_424|>",
+    "<|special_425|>",
+    "<|special_426|>",
+    "<|special_427|>",
+    "<|special_428|>",
+    "<|special_429|>",
+    "<|special_430|>",
+    "<|special_431|>",
+    "<|special_432|>",
+    "<|special_433|>",
+    "<|special_434|>",
+    "<|special_435|>",
+    "<|special_436|>",
+    "<|special_437|>",
+    "<|special_438|>",
+    "<|special_439|>",
+    "<|special_440|>",
+    "<|special_441|>",
+    "<|special_442|>",
+    "<|special_443|>",
+    "<|special_444|>",
+    "<|special_445|>",
+    "<|special_446|>",
+    "<|special_447|>",
+    "<|special_448|>",
+    "<|special_449|>",
+    "<|special_450|>",
+    "<|special_451|>",
+    "<|special_452|>",
+    "<|special_453|>",
+    "<|special_454|>",
+    "<|special_455|>",
+    "<|special_456|>",
+    "<|special_457|>",
+    "<|special_458|>",
+    "<|special_459|>",
+    "<|special_460|>",
+    "<|special_461|>",
+    "<|special_462|>",
+    "<|special_463|>",
+    "<|special_464|>",
+    "<|special_465|>",
+    "<|special_466|>",
+    "<|special_467|>",
+    "<|special_468|>",
+    "<|special_469|>",
+    "<|special_470|>",
+    "<|special_471|>",
+    "<|special_472|>",
+    "<|special_473|>",
+    "<|special_474|>",
+    "<|special_475|>",
+    "<|special_476|>",
+    "<|special_477|>",
+    "<|special_478|>",
+    "<|special_479|>",
+    "<|special_480|>",
+    "<|special_481|>",
+    "<|special_482|>",
+    "<|special_483|>",
+    "<|special_484|>",
+    "<|special_485|>",
+    "<|special_486|>",
+    "<|special_487|>",
+    "<|special_488|>",
+    "<|special_489|>",
+    "<|special_490|>",
+    "<|special_491|>",
+    "<|special_492|>",
+    "<|special_493|>",
+    "<|special_494|>",
+    "<|special_495|>",
+    "<|special_496|>",
+    "<|special_497|>",
+    "<|special_498|>",
+    "<|special_499|>",
+    "<|special_500|>",
+    "<|special_501|>",
+    "<|special_502|>",
+    "<|special_503|>",
+    "<|special_504|>",
+    "<|special_505|>",
+    "<|special_506|>",
+    "<|special_507|>",
+    "<|special_508|>",
+    "<|special_509|>",
+    "<|special_510|>",
+    "<|special_511|>",
+    "<|special_625|>",
+    "<|special_626|>",
+    "<|special_627|>",
+    "<|special_628|>",
+    "<|special_629|>",
+    "<|special_630|>",
+    "<|special_631|>",
+    "<|special_632|>",
+    "<|special_633|>",
+    "<|special_634|>",
+    "<|special_635|>",
+    "<|special_636|>",
+    "<|special_637|>",
+    "<|special_638|>",
+    "<|special_639|>",
+    "<|special_640|>",
+    "<|special_641|>",
+    "<|special_642|>",
+    "<|special_643|>",
+    "<|special_644|>",
+    "<|special_645|>",
+    "<|special_646|>",
+    "<|special_647|>",
+    "<|special_648|>",
+    "<|special_649|>",
+    "<|special_650|>",
+    "<|special_651|>",
+    "<|special_652|>",
+    "<|special_653|>",
+    "<|special_654|>",
+    "<|special_655|>",
+    "<|special_656|>",
+    "<|special_657|>",
+    "<|special_658|>",
+    "<|special_659|>",
+    "<|special_660|>",
+    "<|special_661|>",
+    "<|special_662|>",
+    "<|special_663|>",
+    "<|special_664|>",
+    "<|special_665|>",
+    "<|special_666|>",
+    "<|special_667|>",
+    "<|special_668|>",
+    "<|special_669|>",
+    "<|special_670|>",
+    "<|special_671|>",
+    "<|special_672|>",
+    "<|special_673|>",
+    "<|special_674|>",
+    "<|special_675|>",
+    "<|special_676|>",
+    "<|special_677|>",
+    "<|special_678|>",
+    "<|special_679|>",
+    "<|special_680|>",
+    "<|special_681|>",
+    "<|special_682|>",
+    "<|special_683|>",
+    "<|special_684|>",
+    "<|special_685|>",
+    "<|special_686|>",
+    "<|special_687|>",
+    "<|special_688|>",
+    "<|special_689|>",
+    "<|special_690|>",
+    "<|special_691|>",
+    "<|special_692|>",
+    "<|special_693|>",
+    "<|special_694|>",
+    "<|special_695|>",
+    "<|special_696|>",
+    "<|special_697|>",
+    "<|special_698|>",
+    "<|special_699|>",
+    "<|special_700|>",
+    "<|special_701|>",
+    "<|special_702|>",
+    "<|special_703|>",
+    "<|special_704|>",
+    "<|special_705|>",
+    "<|special_706|>",
+    "<|special_707|>",
+    "<|special_708|>",
+    "<|special_709|>",
+    "<|special_710|>",
+    "<|special_711|>",
+    "<|special_712|>",
+    "<|special_713|>",
+    "<|special_714|>",
+    "<|special_715|>",
+    "<|special_716|>",
+    "<|special_717|>",
+    "<|special_718|>",
+    "<|special_719|>",
+    "<|special_720|>",
+    "<|special_721|>",
+    "<|special_722|>",
+    "<|special_723|>",
+    "<|special_724|>",
+    "<|special_725|>",
+    "<|special_726|>",
+    "<|special_727|>",
+    "<|special_728|>",
+    "<|special_729|>",
+    "<|special_730|>",
+    "<|special_731|>",
+    "<|special_732|>",
+    "<|special_733|>",
+    "<|special_734|>",
+    "<|special_735|>",
+    "<|special_736|>",
+    "<|special_737|>",
+    "<|special_738|>",
+    "<|special_739|>",
+    "<|special_740|>",
+    "<|special_741|>",
+    "<|special_742|>",
+    "<|special_743|>",
+    "<|special_744|>",
+    "<|special_745|>",
+    "<|special_746|>",
+    "<|special_747|>",
+    "<|special_748|>",
+    "<|special_749|>",
+    "<|special_750|>",
+    "<|special_751|>",
+    "<|special_752|>",
+    "<|special_753|>",
+    "<|special_754|>",
+    "<|special_755|>",
+    "<|special_756|>",
+    "<|special_757|>",
+    "<|special_758|>",
+    "<|special_759|>",
+    "<|special_760|>",
+    "<|special_761|>",
+    "<|special_762|>",
+    "<|special_763|>",
+    "<|special_764|>",
+    "<|special_765|>",
+    "<|special_766|>",
+    "<|special_767|>",
+    "<|special_768|>",
+    "<|special_769|>",
+    "<|special_770|>",
+    "<|special_771|>",
+    "<|special_772|>",
+    "<|special_773|>",
+    "<|special_774|>",
+    "<|special_775|>",
+    "<|special_776|>",
+    "<|special_777|>",
+    "<|special_778|>",
+    "<|special_779|>",
+    "<|special_780|>",
+    "<|special_781|>",
+    "<|special_782|>",
+    "<|special_783|>",
+    "<|special_784|>",
+    "<|special_785|>",
+    "<|special_786|>",
+    "<|special_787|>",
+    "<|special_788|>",
+    "<|special_789|>",
+    "<|special_790|>",
+    "<|special_791|>",
+    "<|special_792|>",
+    "<|special_793|>",
+    "<|special_794|>",
+    "<|special_795|>",
+    "<|special_796|>",
+    "<|special_797|>",
+    "<|special_798|>",
+    "<|special_799|>",
+    "<|special_800|>",
+    "<|special_801|>",
+    "<|special_802|>",
+    "<|special_803|>",
+    "<|special_804|>",
+    "<|special_805|>",
+    "<|special_806|>",
+    "<|special_807|>",
+    "<|special_808|>",
+    "<|special_809|>",
+    "<|special_810|>",
+    "<|special_811|>",
+    "<|special_812|>",
+    "<|special_813|>",
+    "<|special_814|>",
+    "<|special_815|>",
+    "<|special_816|>",
+    "<|special_817|>",
+    "<|special_818|>",
+    "<|special_819|>",
+    "<|special_820|>",
+    "<|special_821|>",
+    "<|special_822|>",
+    "<|special_823|>",
+    "<|special_824|>",
+    "<|special_825|>",
+    "<|special_826|>",
+    "<|special_827|>",
+    "<|special_828|>",
+    "<|special_829|>",
+    "<|special_830|>",
+    "<|special_831|>",
+    "<|special_832|>",
+    "<|special_833|>",
+    "<|special_834|>",
+    "<|special_835|>",
+    "<|special_836|>",
+    "<|special_837|>",
+    "<|special_838|>",
+    "<|special_839|>",
+    "<|special_840|>",
+    "<|special_841|>",
+    "<|special_842|>",
+    "<|special_843|>",
+    "<|special_844|>",
+    "<|special_845|>",
+    "<|special_846|>",
+    "<|special_847|>",
+    "<|special_848|>",
+    "<|special_849|>",
+    "<|special_850|>",
+    "<|special_851|>",
+    "<|special_852|>",
+    "<|special_853|>",
+    "<|special_854|>",
+    "<|special_855|>",
+    "<|special_856|>",
+    "<|special_857|>",
+    "<|special_858|>",
+    "<|special_859|>",
+    "<|special_860|>",
+    "<|special_861|>",
+    "<|special_862|>",
+    "<|special_863|>",
+    "<|special_864|>",
+    "<|special_865|>",
+    "<|special_866|>",
+    "<|special_867|>",
+    "<|special_868|>",
+    "<|special_869|>",
+    "<|special_870|>",
+    "<|special_871|>",
+    "<|special_872|>",
+    "<|special_873|>",
+    "<|special_874|>",
+    "<|special_875|>",
+    "<|special_876|>",
+    "<|special_877|>",
+    "<|special_878|>",
+    "<|special_879|>",
+    "<|special_880|>",
+    "<|special_881|>",
+    "<|special_882|>",
+    "<|special_883|>",
+    "<|special_884|>",
+    "<|special_885|>",
+    "<|special_886|>",
+    "<|special_887|>",
+    "<|special_888|>",
+    "<|special_889|>",
+    "<|special_890|>",
+    "<|special_891|>",
+    "<|special_892|>",
+    "<|special_893|>",
+    "<|special_894|>",
+    "<|special_895|>",
+    "<|special_896|>",
+    "<|special_897|>",
+    "<|special_898|>",
+    "<|special_899|>",
+    "<|special_900|>",
+    "<|special_901|>",
+    "<|special_902|>",
+    "<|special_903|>",
+    "<|special_904|>",
+    "<|special_905|>",
+    "<|special_906|>",
+    "<|special_907|>",
+    "<|special_908|>",
+    "<|special_909|>",
+    "<|special_910|>",
+    "<|special_911|>",
+    "<|special_912|>",
+    "<|special_913|>",
+    "<|special_914|>",
+    "<|special_915|>",
+    "<|special_916|>",
+    "<|special_917|>",
+    "<|special_918|>",
+    "<|special_919|>",
+    "<|special_920|>",
+    "<|special_921|>",
+    "<|special_922|>",
+    "<|special_923|>",
+    "<|special_924|>",
+    "<|special_925|>",
+    "<|special_926|>",
+    "<|special_927|>",
+    "<|special_928|>",
+    "<|special_929|>",
+    "<|special_930|>",
+    "<|special_931|>",
+    "<|special_932|>",
+    "<|special_933|>",
+    "<|special_934|>",
+    "<|special_935|>",
+    "<|special_936|>",
+    "<|special_937|>",
+    "<|special_938|>",
+    "<|special_939|>",
+    "<|special_940|>",
+    "<|special_941|>",
+    "<|special_942|>",
+    "<|special_943|>",
+    "<|special_944|>",
+    "<|special_945|>",
+    "<|special_946|>",
+    "<|special_947|>",
+    "<|special_948|>",
+    "<|special_949|>",
+    "<|special_950|>",
+    "<|special_951|>",
+    "<|special_952|>",
+    "<|special_953|>",
+    "<|special_954|>",
+    "<|special_955|>",
+    "<|special_956|>",
+    "<|special_957|>",
+    "<|special_958|>",
+    "<|special_959|>",
+    "<|special_960|>",
+    "<|special_961|>",
+    "<|special_962|>",
+    "<|special_963|>",
+    "<|special_964|>",
+    "<|special_965|>",
+    "<|special_966|>",
+    "<|special_967|>",
+    "<|special_968|>",
+    "<|special_969|>",
+    "<|special_970|>",
+    "<|special_971|>",
+    "<|special_972|>",
+    "<|special_973|>",
+    "<|special_974|>",
+    "<|special_975|>",
+    "<|special_976|>",
+    "<|special_977|>",
+    "<|special_978|>",
+    "<|special_979|>",
+    "<|special_980|>",
+    "<|special_981|>",
+    "<|special_982|>",
+    "<|special_983|>",
+    "<|special_984|>",
+    "<|special_985|>",
+    "<|special_986|>",
+    "<|special_987|>",
+    "<|special_988|>",
+    "<|special_989|>",
+    "<|special_990|>",
+    "<|special_991|>",
+    "<|special_992|>",
+    "<|special_993|>",
+    "<|special_994|>",
+    "<|special_995|>",
+    "<|special_996|>",
+    "<|special_997|>",
+    "<|special_998|>",
+    "<|special_999|>",
+    "<|special_1000|>",
+    "<|special_1001|>",
+    "<|special_1002|>",
+    "<|special_1003|>",
+    "<|special_1004|>",
+    "<|special_1005|>",
+    "<|special_1006|>",
+    "<|special_1007|>",
+    "<|special_1008|>",
+    "<|special_1009|>",
+    "<|special_1010|>",
+    "<|special_1011|>",
+    "<|special_1012|>",
+    "<|special_1013|>",
+    "<|special_1014|>",
+    "<|special_1015|>",
+    "<|special_1016|>",
+    "<|special_1017|>",
+    "<|special_1018|>",
+    "<|special_1019|>",
+    "<|special_1020|>",
+    "<|special_1021|>",
+    "<|special_1022|>",
+    "<|special_1023|>",
+    "<|special_1024|>",
+    "<|special_1025|>",
+    "<|special_1026|>",
+    "<|special_1027|>",
+    "<|special_1028|>",
+    "<|special_1029|>",
+    "<|special_1030|>",
+    "<|special_1031|>",
+    "<|special_1032|>",
+    "<|special_1033|>",
+    "<|special_1034|>",
+    "<|special_1035|>",
+    "<|special_1036|>",
+    "<|special_1037|>",
+    "<|special_1038|>",
+    "<|special_1039|>",
+    "<|special_1040|>",
+    "<|special_1041|>",
+    "<|special_1042|>",
+    "<|special_1043|>",
+    "<|special_1044|>",
+    "<|special_1045|>",
+    "<|special_1046|>",
+    "<|special_1047|>",
+    "<|special_1048|>",
+    "<|special_1049|>",
+    "<|special_1050|>",
+    "<|special_1051|>",
+    "<|special_1052|>",
+    "<|special_1053|>",
+    "<|special_1054|>",
+    "<|special_1055|>",
+    "<|special_1056|>",
+    "<|special_1057|>",
+    "<|special_1058|>",
+    "<|special_1059|>",
+    "<|special_1060|>",
+    "<|special_1061|>",
+    "<|special_1062|>",
+    "<|special_1063|>",
+    "<|special_1064|>",
+    "<|special_1065|>",
+    "<|special_1066|>",
+    "<|special_1067|>",
+    "<|special_1068|>",
+    "<|special_1069|>",
+    "<|special_1070|>",
+    "<|special_1071|>",
+    "<|special_1072|>",
+    "<|special_1073|>",
+    "<|special_1074|>",
+    "<|special_1075|>",
+    "<|special_1076|>",
+    "<|special_1077|>",
+    "<|special_1078|>",
+    "<|special_1079|>",
+    "<|special_1080|>",
+    "<|special_1081|>",
+    "<|special_1082|>",
+    "<|special_1083|>",
+    "<|special_1084|>",
+    "<|special_1085|>",
+    "<|special_1086|>",
+    "<|special_1087|>",
+    "<|special_1088|>",
+    "<|special_1089|>",
+    "<|special_1090|>",
+    "<|special_1091|>",
+    "<|special_1092|>",
+    "<|special_1093|>",
+    "<|special_1094|>",
+    "<|special_1095|>",
+    "<|special_1096|>",
+    "<|special_1097|>",
+    "<|special_1098|>",
+    "<|special_1099|>",
+    "<|special_1100|>",
+    "<|special_1101|>",
+    "<|special_1102|>",
+    "<|special_1103|>",
+    "<|special_1104|>",
+    "<|special_1105|>",
+    "<|special_1106|>",
+    "<|special_1107|>",
+    "<|special_1108|>",
+    "<|special_1109|>",
+    "<|special_1110|>",
+    "<|special_1111|>",
+    "<|special_1112|>",
+    "<|special_1113|>",
+    "<|special_1114|>",
+    "<|special_1115|>",
+    "<|special_1116|>",
+    "<|special_1117|>",
+    "<|special_1118|>",
+    "<|special_1119|>",
+    "<|special_1120|>",
+    "<|special_1121|>",
+    "<|special_1122|>",
+    "<|special_1123|>",
+    "<|special_1124|>",
+    "<|special_1125|>",
+    "<|special_1126|>",
+    "<|special_1127|>",
+    "<|special_1128|>",
+    "<|special_1129|>",
+    "<|special_1130|>",
+    "<|special_1131|>",
+    "<|special_1132|>",
+    "<|special_1133|>",
+    "<|special_1134|>",
+    "<|special_1135|>",
+    "<|special_1136|>",
+    "<|special_1137|>",
+    "<|special_1138|>",
+    "<|special_1139|>",
+    "<|special_1140|>",
+    "<|special_1141|>",
+    "<|special_1142|>",
+    "<|special_1143|>",
+    "<|special_1144|>",
+    "<|special_1145|>",
+    "<|special_1146|>",
+    "<|special_1147|>",
+    "<|special_1148|>",
+    "<|special_1149|>",
+    "<|special_1150|>",
+    "<|special_1151|>",
+    "<|special_1152|>",
+    "<|special_1153|>",
+    "<|special_1154|>",
+    "<|special_1155|>",
+    "<|special_1156|>",
+    "<|special_1157|>",
+    "<|special_1158|>",
+    "<|special_1159|>",
+    "<|special_1160|>",
+    "<|special_1161|>",
+    "<|special_1162|>",
+    "<|special_1163|>",
+    "<|special_1164|>",
+    "<|special_1165|>",
+    "<|special_1166|>",
+    "<|special_1167|>",
+    "<|special_1168|>",
+    "<|special_1169|>",
+    "<|special_1170|>",
+    "<|special_1171|>",
+    "<|special_1172|>",
+    "<|special_1173|>",
+    "<|special_1174|>",
+    "<|special_1175|>",
+    "<|special_1176|>",
+    "<|special_1177|>",
+    "<|special_1178|>",
+    "<|special_1179|>",
+    "<|special_1180|>",
+    "<|special_1181|>",
+    "<|special_1182|>",
+    "<|special_1183|>",
+    "<|special_1184|>",
+    "<|special_1185|>",
+    "<|special_1186|>",
+    "<|special_1187|>",
+    "<|special_1188|>",
+    "<|special_1189|>",
+    "<|special_1190|>",
+    "<|special_1191|>",
+    "<|special_1192|>",
+    "<|special_1193|>",
+    "<|special_1194|>",
+    "<|special_1195|>",
+    "<|special_1196|>",
+    "<|special_1197|>",
+    "<|special_1198|>",
+    "<|special_1199|>",
+    "<|special_1200|>",
+    "<|special_1201|>",
+    "<|special_1202|>",
+    "<|special_1203|>",
+    "<|special_1204|>",
+    "<|special_1205|>",
+    "<|special_1206|>",
+    "<|special_1207|>",
+    "<|special_1208|>",
+    "<|special_1209|>",
+    "<|special_1210|>",
+    "<|special_1211|>",
+    "<|special_1212|>",
+    "<|special_1213|>",
+    "<|special_1214|>",
+    "<|special_1215|>",
+    "<|special_1216|>",
+    "<|special_1217|>",
+    "<|special_1218|>",
+    "<|special_1219|>",
+    "<|special_1220|>",
+    "<|special_1221|>",
+    "<|special_1222|>",
+    "<|special_1223|>",
+    "<|special_1224|>",
+    "<|special_1225|>",
+    "<|special_1226|>",
+    "<|special_1227|>",
+    "<|special_1228|>",
+    "<|special_1229|>",
+    "<|special_1230|>",
+    "<|special_1231|>",
+    "<|special_1232|>",
+    "<|special_1233|>",
+    "<|special_1234|>",
+    "<|special_1235|>",
+    "<|special_1236|>",
+    "<|special_1237|>",
+    "<|special_1238|>",
+    "<|special_1239|>",
+    "<|special_1240|>",
+    "<|special_1241|>",
+    "<|special_1242|>",
+    "<|special_1243|>",
+    "<|special_1244|>",
+    "<|special_1245|>",
+    "<|special_1246|>",
+    "<|special_1247|>",
+    "<|special_1248|>",
+    "<|special_1249|>",
+    "<|special_1250|>",
+    "<|special_1251|>",
+    "<|special_1252|>",
+    "<|special_1253|>",
+    "<|special_1254|>",
+    "<|special_1255|>",
+    "<|special_1256|>",
+    "<|special_1257|>",
+    "<|special_1258|>",
+    "<|special_1259|>",
+    "<|special_1260|>",
+    "<|special_1261|>",
+    "<|special_1262|>",
+    "<|special_1263|>",
+    "<|special_1264|>",
+    "<|special_1265|>",
+    "<|special_1266|>",
+    "<|special_1267|>",
+    "<|special_1268|>",
+    "<|special_1269|>",
+    "<|special_1270|>",
+    "<|special_1271|>",
+    "<|special_1272|>",
+    "<|special_1273|>",
+    "<|special_1274|>",
+    "<|special_1275|>",
+    "<|special_1276|>",
+    "<|special_1277|>",
+    "<|special_1278|>",
+    "<|special_1279|>",
+    "<|special_1280|>",
+    "<|special_1281|>",
+    "<|special_1282|>",
+    "<|special_1283|>",
+    "<|special_1284|>",
+    "<|special_1285|>",
+    "<|special_1286|>",
+    "<|special_1287|>",
+    "<|special_1288|>",
+    "<|special_1289|>",
+    "<|special_1290|>",
+    "<|special_1291|>",
+    "<|special_1292|>",
+    "<|special_1293|>",
+    "<|special_1294|>",
+    "<|special_1295|>",
+    "<|special_1296|>",
+    "<|special_1297|>",
+    "<|special_1298|>",
+    "<|special_1299|>",
+    "<|special_1300|>",
+    "<|special_1301|>",
+    "<|special_1302|>",
+    "<|special_1303|>",
+    "<|special_1304|>",
+    "<|special_1305|>",
+    "<|special_1306|>",
+    "<|special_1307|>",
+    "<|special_1308|>",
+    "<|special_1309|>",
+    "<|special_1310|>",
+    "<|special_1311|>",
+    "<|special_1312|>",
+    "<|special_1313|>",
+    "<|special_1314|>",
+    "<|special_1315|>",
+    "<|special_1316|>",
+    "<|special_1317|>",
+    "<|special_1318|>",
+    "<|special_1319|>",
+    "<|special_1320|>",
+    "<|special_1321|>",
+    "<|special_1322|>",
+    "<|special_1323|>",
+    "<|special_1324|>",
+    "<|special_1325|>",
+    "<|special_1326|>",
+    "<|special_1327|>",
+    "<|special_1328|>",
+    "<|special_1329|>",
+    "<|special_1330|>",
+    "<|special_1331|>",
+    "<|special_1332|>",
+    "<|special_1333|>",
+    "<|special_1334|>",
+    "<|special_1335|>",
+    "<|special_1336|>",
+    "<|special_1337|>",
+    "<|special_1338|>",
+    "<|special_1339|>",
+    "<|special_1340|>",
+    "<|special_1341|>",
+    "<|special_1342|>",
+    "<|special_1343|>",
+    "<|special_1344|>",
+    "<|special_1345|>",
+    "<|special_1346|>",
+    "<|special_1347|>",
+    "<|special_1348|>",
+    "<|special_1349|>",
+    "<|special_1350|>",
+    "<|special_1351|>",
+    "<|special_1352|>",
+    "<|special_1353|>",
+    "<|special_1354|>",
+    "<|special_1355|>",
+    "<|special_1356|>",
+    "<|special_1357|>",
+    "<|special_1358|>",
+    "<|special_1359|>",
+    "<|special_1360|>",
+    "<|special_1361|>",
+    "<|special_1362|>",
+    "<|special_1363|>",
+    "<|special_1364|>",
+    "<|special_1365|>",
+    "<|special_1366|>",
+    "<|special_1367|>",
+    "<|special_1368|>",
+    "<|special_1369|>",
+    "<|special_1370|>",
+    "<|special_1371|>",
+    "<|special_1372|>",
+    "<|special_1373|>",
+    "<|special_1374|>",
+    "<|special_1375|>",
+    "<|special_1376|>",
+    "<|special_1377|>",
+    "<|special_1378|>",
+    "<|special_1379|>",
+    "<|special_1380|>",
+    "<|special_1381|>",
+    "<|special_1382|>",
+    "<|special_1383|>",
+    "<|special_1384|>",
+    "<|special_1385|>",
+    "<|special_1386|>",
+    "<|special_1387|>",
+    "<|special_1388|>",
+    "<|special_1389|>",
+    "<|special_1390|>",
+    "<|special_1391|>",
+    "<|special_1392|>",
+    "<|special_1393|>",
+    "<|special_1394|>",
+    "<|special_1395|>",
+    "<|special_1396|>",
+    "<|special_1397|>",
+    "<|special_1398|>",
+    "<|special_1399|>",
+    "<|special_1400|>",
+    "<|special_1401|>",
+    "<|special_1402|>",
+    "<|special_1403|>",
+    "<|special_1404|>",
+    "<|special_1405|>",
+    "<|special_1406|>",
+    "<|special_1407|>",
+    "<|special_1408|>",
+    "<|special_1409|>",
+    "<|special_1410|>",
+    "<|special_1411|>",
+    "<|special_1412|>",
+    "<|special_1413|>",
+    "<|special_1414|>",
+    "<|special_1415|>",
+    "<|special_1416|>",
+    "<|special_1417|>",
+    "<|special_1418|>",
+    "<|special_1419|>",
+    "<|special_1420|>",
+    "<|special_1421|>",
+    "<|special_1422|>",
+    "<|special_1423|>",
+    "<|special_1424|>",
+    "<|special_1425|>",
+    "<|special_1426|>",
+    "<|special_1427|>",
+    "<|special_1428|>",
+    "<|special_1429|>",
+    "<|special_1430|>",
+    "<|special_1431|>",
+    "<|special_1432|>",
+    "<|special_1433|>",
+    "<|special_1434|>",
+    "<|special_1435|>",
+    "<|special_1436|>",
+    "<|special_1437|>",
+    "<|special_1438|>",
+    "<|special_1439|>",
+    "<|special_1440|>",
+    "<|special_1441|>",
+    "<|special_1442|>",
+    "<|special_1443|>",
+    "<|special_1444|>",
+    "<|special_1445|>",
+    "<|special_1446|>",
+    "<|special_1447|>",
+    "<|special_1448|>",
+    "<|special_1449|>",
+    "<|special_1450|>",
+    "<|special_1451|>",
+    "<|special_1452|>",
+    "<|special_1453|>",
+    "<|special_1454|>",
+    "<|special_1455|>",
+    "<|special_1456|>",
+    "<|special_1457|>",
+    "<|special_1458|>",
+    "<|special_1459|>",
+    "<|special_1460|>",
+    "<|special_1461|>",
+    "<|special_1462|>",
+    "<|special_1463|>",
+    "<|special_1464|>",
+    "<|special_1465|>",
+    "<|special_1466|>",
+    "<|special_1467|>",
+    "<|special_1468|>",
+    "<|special_1469|>",
+    "<|special_1470|>",
+    "<|special_1471|>",
+    "<|special_1472|>",
+    "<|special_1473|>",
+    "<|special_1474|>",
+    "<|special_1475|>",
+    "<|special_1476|>",
+    "<|special_1477|>",
+    "<|special_1478|>",
+    "<|special_1479|>",
+    "<|special_1480|>",
+    "<|special_1481|>",
+    "<|special_1482|>",
+    "<|special_1483|>",
+    "<|special_1484|>",
+    "<|special_1485|>",
+    "<|special_1486|>",
+    "<|special_1487|>",
+    "<|special_1488|>",
+    "<|special_1489|>",
+    "<|special_1490|>",
+    "<|special_1491|>",
+    "<|special_1492|>",
+    "<|special_1493|>",
+    "<|special_1494|>",
+    "<|special_1495|>",
+    "<|special_1496|>",
+    "<|special_1497|>",
+    "<|special_1498|>",
+    "<|special_1499|>",
+    "<|special_1500|>",
+    "<|special_1501|>",
+    "<|special_1502|>",
+    "<|special_1503|>",
+    "<|special_1504|>",
+    "<|special_1505|>",
+    "<|special_1506|>",
+    "<|special_1507|>",
+    "<|special_1508|>",
+    "<|special_1509|>",
+    "<|special_1510|>",
+    "<|special_1511|>",
+    "<|special_1512|>",
+    "<|special_1513|>",
+    "<|special_1514|>",
+    "<|special_1515|>",
+    "<|special_1516|>",
+    "<|special_1517|>",
+    "<|special_1518|>",
+    "<|special_1519|>",
+    "<|special_1520|>",
+    "<|special_1521|>",
+    "<|special_1522|>",
+    "<|special_1523|>",
+    "<|special_1524|>",
+    "<|special_1525|>",
+    "<|special_1526|>",
+    "<|special_1527|>",
+    "<|special_1528|>",
+    "<|special_1529|>",
+    "<|special_1530|>",
+    "<|special_1531|>",
+    "<|special_1532|>",
+    "<|special_1533|>",
+    "<|special_1534|>",
+    "<|special_1535|>",
+    "<|special_1536|>",
+    "<|special_1537|>",
+    "<|special_1538|>",
+    "<|special_1539|>",
+    "<|special_1540|>",
+    "<|special_1541|>",
+    "<|special_1542|>",
+    "<|special_1543|>",
+    "<|special_1544|>",
+    "<|special_1545|>",
+    "<|special_1546|>",
+    "<|special_1547|>",
+    "<|special_1548|>",
+    "<|special_1549|>",
+    "<|special_1550|>",
+    "<|special_1551|>",
+    "<|special_1552|>",
+    "<|special_1553|>",
+    "<|special_1554|>",
+    "<|special_1555|>",
+    "<|special_1556|>",
+    "<|special_1557|>",
+    "<|special_1558|>",
+    "<|special_1559|>",
+    "<|special_1560|>",
+    "<|special_1561|>",
+    "<|special_1562|>",
+    "<|special_1563|>",
+    "<|special_1564|>",
+    "<|special_1565|>",
+    "<|special_1566|>",
+    "<|special_1567|>",
+    "<|special_1568|>",
+    "<|special_1569|>",
+    "<|special_1570|>",
+    "<|special_1571|>",
+    "<|special_1572|>",
+    "<|special_1573|>",
+    "<|special_1574|>",
+    "<|special_1575|>",
+    "<|special_1576|>",
+    "<|special_1577|>",
+    "<|special_1578|>",
+    "<|special_1579|>",
+    "<|special_1580|>",
+    "<|special_1581|>",
+    "<|special_1582|>",
+    "<|special_1583|>",
+    "<|special_1584|>",
+    "<|special_1585|>",
+    "<|special_1586|>",
+    "<|special_1587|>",
+    "<|special_1588|>",
+    "<|special_1589|>",
+    "<|special_1590|>",
+    "<|special_1591|>",
+    "<|special_1592|>",
+    "<|special_1593|>",
+    "<|special_1594|>",
+    "<|special_1595|>",
+    "<|special_1596|>",
+    "<|special_1597|>",
+    "<|special_1598|>",
+    "<|special_1599|>",
+    "<|special_1600|>",
+    "<|special_1601|>",
+    "<|special_1602|>",
+    "<|special_1603|>",
+    "<|special_1604|>",
+    "<|special_1605|>",
+    "<|special_1606|>",
+    "<|special_1607|>",
+    "<|special_1608|>",
+    "<|special_1609|>",
+    "<|special_1610|>",
+    "<|special_1611|>",
+    "<|special_1612|>",
+    "<|special_1613|>",
+    "<|special_1614|>",
+    "<|special_1615|>",
+    "<|special_1616|>",
+    "<|special_1617|>",
+    "<|special_1618|>",
+    "<|special_1619|>",
+    "<|special_1620|>",
+    "<|special_1621|>",
+    "<|special_1622|>",
+    "<|special_1623|>",
+    "<|special_1624|>",
+    "<|special_1625|>",
+    "<|special_1626|>",
+    "<|special_1627|>",
+    "<|special_1628|>",
+    "<|special_1629|>",
+    "<|special_1630|>",
+    "<|special_1631|>",
+    "<|special_1632|>",
+    "<|special_1633|>",
+    "<|special_1634|>",
+    "<|special_1635|>",
+    "<|special_1636|>",
+    "<|special_1637|>",
+    "<|special_1638|>",
+    "<|special_1639|>",
+    "<|special_1640|>",
+    "<|special_1641|>",
+    "<|special_1642|>",
+    "<|special_1643|>",
+    "<|special_1644|>",
+    "<|special_1645|>",
+    "<|special_1646|>",
+    "<|special_1647|>",
+    "<|special_1648|>",
+    "<|special_1649|>",
+    "<|special_1650|>",
+    "<|special_1651|>",
+    "<|special_1652|>",
+    "<|special_1653|>",
+    "<|special_1654|>",
+    "<|special_1655|>",
+    "<|special_1656|>",
+    "<|special_1657|>",
+    "<|special_1658|>",
+    "<|special_1659|>",
+    "<|special_1660|>",
+    "<|special_1661|>",
+    "<|special_1662|>",
+    "<|special_1663|>",
+    "<|special_1664|>",
+    "<|special_1665|>",
+    "<|special_1666|>",
+    "<|special_1667|>",
+    "<|special_1668|>",
+    "<|special_1669|>",
+    "<|special_1670|>",
+    "<|special_1671|>",
+    "<|special_1672|>",
+    "<|special_1673|>",
+    "<|special_1674|>",
+    "<|special_1675|>",
+    "<|special_1676|>",
+    "<|special_1677|>",
+    "<|special_1678|>",
+    "<|special_1679|>",
+    "<|special_1680|>",
+    "<|special_1681|>",
+    "<|special_1682|>",
+    "<|special_1683|>",
+    "<|special_1684|>",
+    "<|special_1685|>",
+    "<|special_1686|>",
+    "<|special_1687|>",
+    "<|special_1688|>",
+    "<|special_1689|>",
+    "<|special_1690|>",
+    "<|special_1691|>",
+    "<|special_1692|>",
+    "<|special_1693|>",
+    "<|special_1694|>",
+    "<|special_1695|>",
+    "<|special_1696|>",
+    "<|special_1697|>",
+    "<|special_1698|>",
+    "<|special_1699|>",
+    "<|special_1700|>",
+    "<|special_1701|>",
+    "<|special_1702|>",
+    "<|special_1703|>",
+    "<|special_1704|>",
+    "<|special_1705|>",
+    "<|special_1706|>",
+    "<|special_1707|>",
+    "<|special_1708|>",
+    "<|special_1709|>",
+    "<|special_1710|>",
+    "<|special_1711|>",
+    "<|special_1712|>",
+    "<|special_1713|>",
+    "<|special_1714|>",
+    "<|special_1715|>",
+    "<|special_1716|>",
+    "<|special_1717|>",
+    "<|special_1718|>",
+    "<|special_1719|>",
+    "<|special_1720|>",
+    "<|special_1721|>",
+    "<|special_1722|>",
+    "<|special_1723|>",
+    "<|special_1724|>",
+    "<|special_1725|>",
+    "<|special_1726|>",
+    "<|special_1727|>",
+    "<|special_1728|>",
+    "<|special_1729|>",
+    "<|special_1730|>",
+    "<|special_1731|>",
+    "<|special_1732|>",
+    "<|special_1733|>",
+    "<|special_1734|>",
+    "<|special_1735|>",
+    "<|special_1736|>",
+    "<|special_1737|>",
+    "<|special_1738|>",
+    "<|special_1739|>",
+    "<|special_1740|>",
+    "<|special_1741|>",
+    "<|special_1742|>",
+    "<|special_1743|>",
+    "<|special_1744|>",
+    "<|special_1745|>",
+    "<|special_1746|>",
+    "<|special_1747|>",
+    "<|special_1748|>",
+    "<|special_1749|>",
+    "<|special_1750|>",
+    "<|special_1751|>",
+    "<|special_1752|>",
+    "<|special_1753|>",
+    "<|special_1754|>",
+    "<|special_1755|>",
+    "<|special_1756|>",
+    "<|special_1757|>",
+    "<|special_1758|>",
+    "<|special_1759|>",
+    "<|special_1760|>",
+    "<|special_1761|>",
+    "<|special_1762|>",
+    "<|special_1763|>",
+    "<|special_1764|>",
+    "<|special_1765|>",
+    "<|special_1766|>",
+    "<|special_1767|>",
+    "<|special_1768|>",
+    "<|special_1769|>",
+    "<|special_1770|>",
+    "<|special_1771|>",
+    "<|special_1772|>",
+    "<|special_1773|>",
+    "<|special_1774|>",
+    "<|special_1775|>",
+    "<|special_1776|>",
+    "<|special_1777|>",
+    "<|special_1778|>",
+    "<|special_1779|>",
+    "<|special_1780|>",
+    "<|special_1781|>",
+    "<|special_1782|>",
+    "<|special_1783|>",
+    "<|special_1784|>",
+    "<|special_1785|>",
+    "<|special_1786|>",
+    "<|special_1787|>",
+    "<|special_1788|>",
+    "<|special_1789|>",
+    "<|special_1790|>",
+    "<|special_1791|>",
+    "<|special_1792|>",
+    "<|special_1793|>",
+    "<|special_1794|>",
+    "<|special_1795|>",
+    "<|special_1796|>",
+    "<|special_1797|>",
+    "<|special_1798|>",
+    "<|special_1799|>",
+    "<|special_1800|>",
+    "<|special_1801|>",
+    "<|special_1802|>",
+    "<|special_1803|>",
+    "<|special_1804|>",
+    "<|special_1805|>",
+    "<|special_1806|>",
+    "<|special_1807|>",
+    "<|special_1808|>",
+    "<|special_1809|>",
+    "<|special_1810|>",
+    "<|special_1811|>",
+    "<|special_1812|>",
+    "<|special_1813|>",
+    "<|special_1814|>",
+    "<|special_1815|>",
+    "<|special_1816|>",
+    "<|special_1817|>",
+    "<|special_1818|>",
+    "<|special_1819|>",
+    "<|special_1820|>",
+    "<|special_1821|>",
+    "<|special_1822|>",
+    "<|special_1823|>",
+    "<|special_1824|>",
+    "<|special_1825|>",
+    "<|special_1826|>",
+    "<|special_1827|>",
+    "<|special_1828|>",
+    "<|special_1829|>",
+    "<|special_1830|>",
+    "<|special_1831|>",
+    "<|special_1832|>",
+    "<|special_1833|>",
+    "<|special_1834|>",
+    "<|special_1835|>",
+    "<|special_1836|>",
+    "<|special_1837|>",
+    "<|special_1838|>",
+    "<|special_1839|>",
+    "<|special_1840|>",
+    "<|special_1841|>",
+    "<|special_1842|>",
+    "<|special_1843|>",
+    "<|special_1844|>",
+    "<|special_1845|>",
+    "<|special_1846|>",
+    "<|special_1847|>",
+    "<|special_1848|>",
+    "<|special_1849|>",
+    "<|special_1850|>",
+    "<|special_1851|>",
+    "<|special_1852|>",
+    "<|special_1853|>",
+    "<|special_1854|>",
+    "<|special_1855|>",
+    "<|special_1856|>",
+    "<|special_1857|>",
+    "<|special_1858|>",
+    "<|special_1859|>",
+    "<|special_1860|>",
+    "<|special_1861|>",
+    "<|special_1862|>",
+    "<|special_1863|>",
+    "<|special_1864|>",
+    "<|special_1865|>",
+    "<|special_1866|>",
+    "<|special_1867|>",
+    "<|special_1868|>",
+    "<|special_1869|>",
+    "<|special_1870|>",
+    "<|special_1871|>",
+    "<|special_1872|>",
+    "<|special_1873|>",
+    "<|special_1874|>",
+    "<|special_1875|>",
+    "<|special_1876|>",
+    "<|special_1877|>",
+    "<|special_1878|>",
+    "<|special_1879|>",
+    "<|special_1880|>",
+    "<|special_1881|>",
+    "<|special_1882|>",
+    "<|special_1883|>",
+    "<|special_1884|>",
+    "<|special_1885|>",
+    "<|special_1886|>",
+    "<|special_1887|>",
+    "<|special_1888|>",
+    "<|special_1889|>",
+    "<|special_1890|>",
+    "<|special_1891|>",
+    "<|special_1892|>",
+    "<|special_1893|>",
+    "<|special_1894|>",
+    "<|special_1895|>",
+    "<|special_1896|>",
+    "<|special_1897|>",
+    "<|special_1898|>",
+    "<|special_1899|>",
+    "<|special_1900|>",
+    "<|special_1901|>",
+    "<|special_1902|>",
+    "<|special_1903|>",
+    "<|special_1904|>",
+    "<|special_1905|>",
+    "<|special_1906|>",
+    "<|special_1907|>",
+    "<|special_1908|>",
+    "<|special_1909|>",
+    "<|special_1910|>",
+    "<|special_1911|>",
+    "<|special_1912|>",
+    "<|special_1913|>",
+    "<|special_1914|>",
+    "<|special_1915|>",
+    "<|special_1916|>",
+    "<|special_1917|>",
+    "<|special_1918|>",
+    "<|special_1919|>",
+    "<|special_1920|>",
+    "<|special_1921|>",
+    "<|special_1922|>",
+    "<|special_1923|>",
+    "<|special_1924|>",
+    "<|special_1925|>",
+    "<|special_1926|>",
+    "<|special_1927|>",
+    "<|special_1928|>",
+    "<|special_1929|>",
+    "<|special_1930|>",
+    "<|special_1931|>",
+    "<|special_1932|>",
+    "<|special_1933|>",
+    "<|special_1934|>",
+    "<|special_1935|>",
+    "<|special_1936|>",
+    "<|special_1937|>",
+    "<|special_1938|>",
+    "<|special_1939|>",
+    "<|special_1940|>",
+    "<|special_1941|>",
+    "<|special_1942|>",
+    "<|special_1943|>",
+    "<|special_1944|>",
+    "<|special_1945|>",
+    "<|special_1946|>",
+    "<|special_1947|>",
+    "<|special_1948|>",
+    "<|special_1949|>",
+    "<|special_1950|>",
+    "<|special_1951|>",
+    "<|special_1952|>",
+    "<|special_1953|>",
+    "<|special_1954|>",
+    "<|special_1955|>",
+    "<|special_1956|>",
+    "<|special_1957|>",
+    "<|special_1958|>",
+    "<|special_1959|>",
+    "<|special_1960|>",
+    "<|special_1961|>",
+    "<|special_1962|>",
+    "<|special_1963|>",
+    "<|special_1964|>",
+    "<|special_1965|>",
+    "<|special_1966|>",
+    "<|special_1967|>",
+    "<|special_1968|>",
+    "<|special_1969|>",
+    "<|special_1970|>",
+    "<|special_1971|>",
+    "<|special_1972|>",
+    "<|special_1973|>",
+    "<|special_1974|>",
+    "<|special_1975|>",
+    "<|special_1976|>",
+    "<|special_1977|>",
+    "<|special_1978|>",
+    "<|special_1979|>",
+    "<|special_1980|>",
+    "<|special_1981|>",
+    "<|special_1982|>",
+    "<|special_1983|>",
+    "<|special_1984|>",
+    "<|special_1985|>",
+    "<|special_1986|>",
+    "<|special_1987|>",
+    "<|special_1988|>",
+    "<|special_1989|>",
+    "<|special_1990|>",
+    "<|special_1991|>",
+    "<|special_1992|>",
+    "<|special_1993|>",
+    "<|special_1994|>",
+    "<|special_1995|>",
+    "<|special_1996|>",
+    "<|special_1997|>",
+    "<|special_1998|>",
+    "<|special_1999|>",
+    "<|special_2000|>",
+    "<|special_2001|>",
+    "<|special_2002|>",
+    "<|special_2003|>",
+    "<|special_2004|>",
+    "<|special_2005|>",
+    "<|special_2006|>",
+    "<|special_2007|>",
+    "<|special_2008|>",
+    "<|special_2009|>",
+    "<|special_2010|>",
+    "<|special_2011|>",
+    "<|special_2012|>",
+    "<|special_2013|>",
+    "<|special_2014|>",
+    "<|special_2015|>",
+    "<|special_2016|>",
+    "<|special_2017|>",
+    "<|special_2018|>",
+    "<|special_2019|>",
+    "<|special_2020|>",
+    "<|special_2021|>",
+    "<|special_2022|>",
+    "<|special_2023|>",
+    "<|special_2024|>",
+    "<|special_2025|>",
+    "<|special_2026|>",
+    "<|special_2027|>",
+    "<|special_2028|>",
+    "<|special_2029|>",
+    "<|special_2030|>",
+    "<|special_2031|>",
+    "<|special_2032|>",
+    "<|special_2033|>",
+    "<|special_2034|>",
+    "<|special_2035|>",
+    "<|special_2036|>",
+    "<|special_2037|>",
+    "<|special_2038|>",
+    "<|special_2039|>",
+    "<|special_2040|>",
+    "<|special_2041|>",
+    "<|special_2042|>",
+    "<|special_2043|>",
+    "<|special_2044|>",
+    "<|special_2045|>",
+    "<|special_2046|>",
+    "<|special_2047|>",
+    "<|special_2048|>",
+    "<|special_2049|>",
+    "<|special_2050|>",
+    "<|special_2051|>",
+    "<|special_2052|>",
+    "<|special_2053|>",
+    "<|special_2054|>",
+    "<|special_2055|>",
+    "<|special_2056|>",
+    "<|special_2057|>",
+    "<|special_2058|>",
+    "<|special_2059|>",
+    "<|special_2060|>",
+    "<|special_2061|>",
+    "<|special_2062|>",
+    "<|special_2063|>",
+    "<|special_2064|>",
+    "<|special_2065|>",
+    "<|special_2066|>",
+    "<|special_2067|>",
+    "<|special_2068|>",
+    "<|special_2069|>",
+    "<|special_2070|>",
+    "<|special_2071|>",
+    "<|special_2072|>",
+    "<|special_2073|>",
+    "<|special_2074|>",
+    "<|special_2075|>",
+    "<|special_2076|>",
+    "<|special_2077|>",
+    "<|special_2078|>",
+    "<|special_2079|>",
+    "<|special_2080|>",
+    "<|special_2081|>",
+    "<|special_2082|>",
+    "<|special_2083|>",
+    "<|special_2084|>",
+    "<|special_2085|>",
+    "<|special_2086|>",
+    "<|special_2087|>",
+    "<|special_2088|>",
+    "<|special_2089|>",
+    "<|special_2090|>",
+    "<|special_2091|>",
+    "<|special_2092|>",
+    "<|special_2093|>",
+    "<|special_2094|>",
+    "<|special_2095|>",
+    "<|special_2096|>",
+    "<|special_2097|>",
+    "<|special_2098|>",
+    "<|special_2099|>",
+    "<|special_2100|>",
+    "<|special_2101|>",
+    "<|special_2102|>",
+    "<|special_2103|>",
+    "<|special_2104|>",
+    "<|special_2105|>",
+    "<|special_2106|>",
+    "<|special_2107|>",
+    "<|special_2108|>",
+    "<|special_2109|>",
+    "<|special_2110|>",
+    "<|special_2111|>",
+    "<|special_2112|>",
+    "<|special_2113|>",
+    "<|special_2114|>",
+    "<|special_2115|>",
+    "<|special_2116|>",
+    "<|special_2117|>",
+    "<|special_2118|>",
+    "<|special_2119|>",
+    "<|special_2120|>",
+    "<|special_2121|>",
+    "<|special_2122|>",
+    "<|special_2123|>",
+    "<|special_2124|>",
+    "<|special_2125|>",
+    "<|special_2126|>",
+    "<|special_2127|>",
+    "<|special_2128|>",
+    "<|special_2129|>",
+    "<|special_2130|>",
+    "<|special_2131|>",
+    "<|special_2132|>",
+    "<|special_2133|>",
+    "<|special_2134|>",
+    "<|special_2135|>",
+    "<|special_2136|>",
+    "<|special_2137|>",
+    "<|special_2138|>",
+    "<|special_2139|>",
+    "<|special_2140|>",
+    "<|special_2141|>",
+    "<|special_2142|>",
+    "<|special_2143|>",
+    "<|special_2144|>",
+    "<|special_2145|>",
+    "<|special_2146|>",
+    "<|special_2147|>",
+    "<|special_2148|>",
+    "<|special_2149|>",
+    "<|special_2150|>",
+    "<|special_2151|>",
+    "<|special_2152|>",
+    "<|special_2153|>",
+    "<|special_2154|>",
+    "<|special_2155|>",
+    "<|special_2156|>",
+    "<|special_2157|>",
+    "<|special_2158|>",
+    "<|special_2159|>",
+    "<|special_2160|>",
+    "<|special_2161|>",
+    "<|special_2162|>",
+    "<|special_2163|>",
+    "<|special_2164|>",
+    "<|special_2165|>",
+    "<|special_2166|>",
+    "<|special_2167|>",
+    "<|special_2168|>",
+    "<|special_2169|>",
+    "<|special_2170|>",
+    "<|special_2171|>",
+    "<|special_2172|>",
+    "<|special_2173|>",
+    "<|special_2174|>",
+    "<|special_2175|>",
+    "<|special_2176|>",
+    "<|special_2177|>",
+    "<|special_2178|>",
+    "<|special_2179|>",
+    "<|special_2180|>",
+    "<|special_2181|>",
+    "<|special_2182|>",
+    "<|special_2183|>",
+    "<|special_2184|>",
+    "<|special_2185|>",
+    "<|special_2186|>",
+    "<|special_2187|>",
+    "<|special_2188|>",
+    "<|special_2189|>",
+    "<|special_2190|>",
+    "<|special_2191|>",
+    "<|special_2192|>",
+    "<|special_2193|>",
+    "<|special_2194|>",
+    "<|special_2195|>",
+    "<|special_2196|>",
+    "<|special_2197|>",
+    "<|special_2198|>",
+    "<|special_2199|>",
+    "<|special_2200|>",
+    "<|special_2201|>",
+    "<|special_2202|>",
+    "<|special_2203|>",
+    "<|special_2204|>",
+    "<|special_2205|>",
+    "<|special_2206|>",
+    "<|special_2207|>",
+    "<|special_2208|>",
+    "<|special_2209|>",
+    "<|special_2210|>",
+    "<|special_2211|>",
+    "<|special_2212|>",
+    "<|special_2213|>",
+    "<|special_2214|>",
+    "<|special_2215|>",
+    "<|special_2216|>",
+    "<|special_2217|>",
+    "<|special_2218|>",
+    "<|special_2219|>",
+    "<|special_2220|>",
+    "<|special_2221|>",
+    "<|special_2222|>",
+    "<|special_2223|>",
+    "<|special_2224|>",
+    "<|special_2225|>",
+    "<|special_2226|>",
+    "<|special_2227|>",
+    "<|special_2228|>",
+    "<|special_2229|>",
+    "<|special_2230|>",
+    "<|special_2231|>",
+    "<|special_2232|>",
+    "<|special_2233|>",
+    "<|special_2234|>",
+    "<|special_2235|>",
+    "<|special_2236|>",
+    "<|special_2237|>",
+    "<|special_2238|>",
+    "<|special_2239|>",
+    "<|special_2240|>",
+    "<|special_2241|>",
+    "<|special_2242|>",
+    "<|special_2243|>",
+    "<|special_2244|>",
+    "<|special_2245|>",
+    "<|special_2246|>",
+    "<|special_2247|>",
+    "<|special_2248|>",
+    "<|special_2249|>",
+    "<|special_2250|>",
+    "<|special_2251|>",
+    "<|special_2252|>",
+    "<|special_2253|>",
+    "<|special_2254|>",
+    "<|special_2255|>",
+    "<|special_2256|>",
+    "<|special_2257|>",
+    "<|special_2258|>",
+    "<|special_2259|>",
+    "<|special_2260|>",
+    "<|special_2261|>",
+    "<|special_2262|>",
+    "<|special_2263|>",
+    "<|special_2264|>",
+    "<|special_2265|>",
+    "<|special_2266|>",
+    "<|special_2267|>",
+    "<|special_2268|>",
+    "<|special_2269|>",
+    "<|special_2270|>",
+    "<|special_2271|>",
+    "<|special_2272|>",
+    "<|special_2273|>",
+    "<|special_2274|>",
+    "<|special_2275|>",
+    "<|special_2276|>",
+    "<|special_2277|>",
+    "<|special_2278|>",
+    "<|special_2279|>",
+    "<|special_2280|>",
+    "<|special_2281|>",
+    "<|special_2282|>",
+    "<|special_2283|>",
+    "<|special_2284|>",
+    "<|special_2285|>",
+    "<|special_2286|>",
+    "<|special_2287|>",
+    "<|special_2288|>",
+    "<|special_2289|>",
+    "<|special_2290|>",
+    "<|special_2291|>",
+    "<|special_2292|>",
+    "<|special_2293|>",
+    "<|special_2294|>",
+    "<|special_2295|>",
+    "<|special_2296|>",
+    "<|special_2297|>",
+    "<|special_2298|>",
+    "<|special_2299|>",
+    "<|special_2300|>",
+    "<|special_2301|>",
+    "<|special_2302|>",
+    "<|special_2303|>",
+    "<|special_2304|>",
+    "<|special_2305|>",
+    "<|special_2306|>",
+    "<|special_2307|>",
+    "<|special_2308|>",
+    "<|special_2309|>",
+    "<|special_2310|>",
+    "<|special_2311|>",
+    "<|special_2312|>",
+    "<|special_2313|>",
+    "<|special_2314|>",
+    "<|special_2315|>",
+    "<|special_2316|>",
+    "<|special_2317|>",
+    "<|special_2318|>",
+    "<|special_2319|>",
+    "<|special_2320|>",
+    "<|special_2321|>",
+    "<|special_2322|>",
+    "<|special_2323|>",
+    "<|special_2324|>",
+    "<|special_2325|>",
+    "<|special_2326|>",
+    "<|special_2327|>",
+    "<|special_2328|>",
+    "<|special_2329|>",
+    "<|special_2330|>",
+    "<|special_2331|>",
+    "<|special_2332|>",
+    "<|special_2333|>",
+    "<|special_2334|>",
+    "<|special_2335|>",
+    "<|special_2336|>",
+    "<|special_2337|>",
+    "<|special_2338|>",
+    "<|special_2339|>",
+    "<|special_2340|>",
+    "<|special_2341|>",
+    "<|special_2342|>",
+    "<|special_2343|>",
+    "<|special_2344|>",
+    "<|special_2345|>",
+    "<|special_2346|>",
+    "<|special_2347|>",
+    "<|special_2348|>",
+    "<|special_2349|>",
+    "<|special_2350|>",
+    "<|special_2351|>",
+    "<|special_2352|>",
+    "<|special_2353|>",
+    "<|special_2354|>",
+    "<|special_2355|>",
+    "<|special_2356|>",
+    "<|special_2357|>",
+    "<|special_2358|>",
+    "<|special_2359|>",
+    "<|special_2360|>",
+    "<|special_2361|>",
+    "<|special_2362|>",
+    "<|special_2363|>",
+    "<|special_2364|>",
+    "<|special_2365|>",
+    "<|special_2366|>",
+    "<|special_2367|>",
+    "<|special_2368|>",
+    "<|special_2369|>",
+    "<|special_2370|>",
+    "<|special_2371|>",
+    "<|special_2372|>",
+    "<|special_2373|>",
+    "<|special_2374|>",
+    "<|special_2375|>",
+    "<|special_2376|>",
+    "<|special_2377|>",
+    "<|special_2378|>",
+    "<|special_2379|>",
+    "<|special_2380|>",
+    "<|special_2381|>",
+    "<|special_2382|>",
+    "<|special_2383|>",
+    "<|special_2384|>",
+    "<|special_2385|>",
+    "<|special_2386|>",
+    "<|special_2387|>",
+    "<|special_2388|>",
+    "<|special_2389|>",
+    "<|special_2390|>",
+    "<|special_2391|>",
+    "<|special_2392|>",
+    "<|special_2393|>",
+    "<|special_2394|>",
+    "<|special_2395|>",
+    "<|special_2396|>",
+    "<|special_2397|>",
+    "<|special_2398|>",
+    "<|special_2399|>",
+    "<|special_2400|>",
+    "<|special_2401|>",
+    "<|special_2402|>",
+    "<|special_2403|>",
+    "<|special_2404|>",
+    "<|special_2405|>",
+    "<|special_2406|>",
+    "<|special_2407|>",
+    "<|special_2408|>",
+    "<|special_2409|>",
+    "<|special_2410|>",
+    "<|special_2411|>",
+    "<|special_2412|>",
+    "<|special_2413|>",
+    "<|special_2414|>",
+    "<|special_2415|>",
+    "<|special_2416|>",
+    "<|special_2417|>",
+    "<|special_2418|>",
+    "<|special_2419|>",
+    "<|special_2420|>",
+    "<|special_2421|>",
+    "<|special_2422|>",
+    "<|special_2423|>",
+    "<|special_2424|>",
+    "<|special_2425|>",
+    "<|special_2426|>",
+    "<|special_2427|>",
+    "<|special_2428|>",
+    "<|special_2429|>",
+    "<|special_2430|>",
+    "<|special_2431|>",
+    "<|special_2432|>",
+    "<|special_2433|>",
+    "<|special_2434|>",
+    "<|special_2435|>",
+    "<|special_2436|>",
+    "<|special_2437|>",
+    "<|special_2438|>",
+    "<|special_2439|>",
+    "<|special_2440|>",
+    "<|special_2441|>",
+    "<|special_2442|>",
+    "<|special_2443|>",
+    "<|special_2444|>",
+    "<|special_2445|>",
+    "<|special_2446|>",
+    "<|special_2447|>",
+    "<|special_2448|>",
+    "<|special_2449|>",
+    "<|special_2450|>",
+    "<|special_2451|>",
+    "<|special_2452|>",
+    "<|special_2453|>",
+    "<|special_2454|>",
+    "<|special_2455|>",
+    "<|special_2456|>",
+    "<|special_2457|>",
+    "<|special_2458|>",
+    "<|special_2459|>",
+    "<|special_2460|>",
+    "<|special_2461|>",
+    "<|special_2462|>",
+    "<|special_2463|>",
+    "<|special_2464|>",
+    "<|special_2465|>",
+    "<|special_2466|>",
+    "<|special_2467|>",
+    "<|special_2468|>",
+    "<|special_2469|>",
+    "<|special_2470|>",
+    "<|special_2471|>",
+    "<|special_2472|>",
+    "<|special_2473|>",
+    "<|special_2474|>",
+    "<|special_2475|>",
+    "<|special_2476|>",
+    "<|special_2477|>",
+    "<|special_2478|>",
+    "<|special_2479|>",
+    "<|special_2480|>",
+    "<|special_2481|>",
+    "<|special_2482|>",
+    "<|special_2483|>",
+    "<|special_2484|>",
+    "<|special_2485|>",
+    "<|special_2486|>",
+    "<|special_2487|>",
+    "<|special_2488|>",
+    "<|special_2489|>",
+    "<|special_2490|>",
+    "<|special_2491|>",
+    "<|special_2492|>",
+    "<|special_2493|>",
+    "<|special_2494|>",
+    "<|special_2495|>",
+    "<|special_2496|>",
+    "<|special_2497|>",
+    "<|special_2498|>",
+    "<|special_2499|>",
+    "<|special_2500|>",
+    "<|special_2501|>",
+    "<|special_2502|>",
+    "<|special_2503|>",
+    "<|special_2504|>",
+    "<|special_2505|>",
+    "<|special_2506|>",
+    "<|special_2507|>",
+    "<|special_2508|>",
+    "<|special_2509|>",
+    "<|special_2510|>",
+    "<|special_2511|>",
+    "<|special_2512|>",
+    "<|special_2513|>",
+    "<|special_2514|>",
+    "<|special_2515|>",
+    "<|special_2516|>",
+    "<|special_2517|>",
+    "<|special_2518|>",
+    "<|special_2519|>",
+    "<|special_2520|>",
+    "<|special_2521|>",
+    "<|special_2522|>",
+    "<|special_2523|>",
+    "<|special_2524|>",
+    "<|special_2525|>",
+    "<|special_2526|>",
+    "<|special_2527|>",
+    "<|special_2528|>",
+    "<|special_2529|>",
+    "<|special_2530|>",
+    "<|special_2531|>",
+    "<|special_2532|>",
+    "<|special_2533|>",
+    "<|special_2534|>",
+    "<|special_2535|>",
+    "<|special_2536|>",
+    "<|special_2537|>",
+    "<|special_2538|>",
+    "<|special_2539|>",
+    "<|special_2540|>",
+    "<|special_2541|>",
+    "<|special_2542|>",
+    "<|special_2543|>",
+    "<|special_2544|>",
+    "<|special_2545|>",
+    "<|special_2546|>",
+    "<|special_2547|>",
+    "<|special_2548|>",
+    "<|special_2549|>",
+    "<|special_2550|>",
+    "<|special_2551|>",
+    "<|special_2552|>",
+    "<|special_2553|>",
+    "<|special_2554|>",
+    "<|special_2555|>",
+    "<|special_2556|>",
+    "<|special_2557|>",
+    "<|special_2558|>",
+    "<|special_2559|>",
+    "<|special_2560|>",
+    "<|special_2561|>",
+    "<|special_2562|>",
+    "<|special_2563|>",
+    "<|special_2564|>",
+    "<|special_2565|>",
+    "<|special_2566|>",
+    "<|special_2567|>",
+    "<|special_2568|>",
+    "<|special_2569|>",
+    "<|special_2570|>",
+    "<|special_2571|>",
+    "<|special_2572|>",
+    "<|special_2573|>",
+    "<|special_2574|>",
+    "<|special_2575|>",
+    "<|special_2576|>",
+    "<|special_2577|>",
+    "<|special_2578|>",
+    "<|special_2579|>",
+    "<|special_2580|>",
+    "<|special_2581|>",
+    "<|special_2582|>",
+    "<|special_2583|>",
+    "<|special_2584|>",
+    "<|special_2585|>",
+    "<|special_2586|>",
+    "<|special_2587|>",
+    "<|special_2588|>",
+    "<|special_2589|>",
+    "<|special_2590|>",
+    "<|special_2591|>",
+    "<|special_2592|>",
+    "<|special_2593|>",
+    "<|special_2594|>",
+    "<|special_2595|>",
+    "<|special_2596|>",
+    "<|special_2597|>",
+    "<|special_2598|>",
+    "<|special_2599|>",
+    "<|special_2600|>",
+    "<|special_2601|>",
+    "<|special_2602|>",
+    "<|special_2603|>",
+    "<|special_2604|>",
+    "<|special_2605|>",
+    "<|special_2606|>",
+    "<|special_2607|>",
+    "<|special_2608|>",
+    "<|special_2609|>",
+    "<|special_2610|>",
+    "<|special_2611|>",
+    "<|special_2612|>",
+    "<|special_2613|>",
+    "<|special_2614|>",
+    "<|special_2615|>",
+    "<|special_2616|>",
+    "<|special_2617|>",
+    "<|special_2618|>",
+    "<|special_2619|>",
+    "<|special_2620|>",
+    "<|special_2621|>",
+    "<|special_2622|>",
+    "<|special_2623|>",
+    "<|special_2624|>",
+    "<|special_2625|>",
+    "<|special_2626|>",
+    "<|special_2627|>",
+    "<|special_2628|>",
+    "<|special_2629|>",
+    "<|special_2630|>",
+    "<|special_2631|>",
+    "<|special_2632|>",
+    "<|special_2633|>",
+    "<|special_2634|>",
+    "<|special_2635|>",
+    "<|special_2636|>",
+    "<|special_2637|>",
+    "<|special_2638|>",
+    "<|special_2639|>",
+    "<|special_2640|>",
+    "<|special_2641|>",
+    "<|special_2642|>",
+    "<|special_2643|>",
+    "<|special_2644|>",
+    "<|special_2645|>",
+    "<|special_2646|>",
+    "<|special_2647|>",
+    "<|special_2648|>",
+    "<|special_2649|>",
+    "<|special_2650|>",
+    "<|special_2651|>",
+    "<|special_2652|>",
+    "<|special_2653|>",
+    "<|special_2654|>",
+    "<|special_2655|>",
+    "<|special_2656|>",
+    "<|special_2657|>",
+    "<|special_2658|>",
+    "<|special_2659|>",
+    "<|special_2660|>",
+    "<|special_2661|>",
+    "<|special_2662|>",
+    "<|special_2663|>",
+    "<|special_2664|>",
+    "<|special_2665|>",
+    "<|special_2666|>",
+    "<|special_2667|>",
+    "<|special_2668|>",
+    "<|special_2669|>",
+    "<|special_2670|>",
+    "<|special_2671|>",
+    "<|special_2672|>",
+    "<|special_2673|>",
+    "<|special_2674|>",
+    "<|special_2675|>",
+    "<|special_2676|>",
+    "<|special_2677|>",
+    "<|special_2678|>",
+    "<|special_2679|>",
+    "<|special_2680|>",
+    "<|special_2681|>",
+    "<|special_2682|>",
+    "<|special_2683|>",
+    "<|special_2684|>",
+    "<|special_2685|>",
+    "<|special_2686|>",
+    "<|special_2687|>",
+    "<|special_2688|>",
+    "<|special_2689|>",
+    "<|special_2690|>",
+    "<|special_2691|>",
+    "<|special_2692|>",
+    "<|special_2693|>",
+    "<|special_2694|>",
+    "<|special_2695|>",
+    "<|special_2696|>",
+    "<|special_2697|>",
+    "<|special_2698|>",
+    "<|special_2699|>",
+    "<|special_2700|>",
+    "<|special_2701|>",
+    "<|special_2702|>",
+    "<|special_2703|>",
+    "<|special_2704|>",
+    "<|special_2705|>",
+    "<|special_2706|>",
+    "<|special_2707|>",
+    "<|special_2708|>",
+    "<|special_2709|>",
+    "<|special_2710|>",
+    "<|special_2711|>",
+    "<|special_2712|>",
+    "<|special_2713|>",
+    "<|special_2714|>",
+    "<|special_2715|>",
+    "<|special_2716|>",
+    "<|special_2717|>",
+    "<|special_2718|>",
+    "<|special_2719|>",
+    "<|special_2720|>",
+    "<|special_2721|>",
+    "<|special_2722|>",
+    "<|special_2723|>",
+    "<|special_2724|>",
+    "<|special_2725|>",
+    "<|special_2726|>",
+    "<|special_2727|>",
+    "<|special_2728|>",
+    "<|special_2729|>",
+    "<|special_2730|>",
+    "<|special_2731|>",
+    "<|special_2732|>",
+    "<|special_2733|>",
+    "<|special_2734|>",
+    "<|special_2735|>",
+    "<|special_2736|>",
+    "<|special_2737|>",
+    "<|special_2738|>",
+    "<|special_2739|>",
+    "<|special_2740|>",
+    "<|special_2741|>",
+    "<|special_2742|>",
+    "<|special_2743|>",
+    "<|special_2744|>",
+    "<|special_2745|>",
+    "<|special_2746|>",
+    "<|special_2747|>",
+    "<|special_2748|>",
+    "<|special_2749|>",
+    "<|special_2750|>",
+    "<|special_2751|>",
+    "<|special_2752|>",
+    "<|special_2753|>",
+    "<|special_2754|>",
+    "<|special_2755|>",
+    "<|special_2756|>",
+    "<|special_2757|>",
+    "<|special_2758|>",
+    "<|special_2759|>",
+    "<|special_2760|>",
+    "<|special_2761|>",
+    "<|special_2762|>",
+    "<|special_2763|>",
+    "<|special_2764|>",
+    "<|special_2765|>",
+    "<|special_2766|>",
+    "<|special_2767|>",
+    "<|special_2768|>",
+    "<|special_2769|>",
+    "<|special_2770|>",
+    "<|special_2771|>",
+    "<|special_2772|>",
+    "<|special_2773|>",
+    "<|special_2774|>",
+    "<|special_2775|>",
+    "<|special_2776|>",
+    "<|special_2777|>",
+    "<|special_2778|>",
+    "<|special_2779|>",
+    "<|special_2780|>",
+    "<|special_2781|>",
+    "<|special_2782|>",
+    "<|special_2783|>",
+    "<|special_2784|>",
+    "<|special_2785|>",
+    "<|special_2786|>",
+    "<|special_2787|>",
+    "<|special_2788|>",
+    "<|special_2789|>",
+    "<|special_2790|>",
+    "<|special_2791|>",
+    "<|special_2792|>",
+    "<|special_2793|>",
+    "<|special_2794|>",
+    "<|special_2795|>",
+    "<|special_2796|>",
+    "<|special_2797|>",
+    "<|special_2798|>",
+    "<|special_2799|>",
+    "<|special_2800|>",
+    "<|special_2801|>",
+    "<|special_2802|>",
+    "<|special_2803|>",
+    "<|special_2804|>",
+    "<|special_2805|>",
+    "<|special_2806|>",
+    "<|special_2807|>",
+    "<|special_2808|>",
+    "<|special_2809|>",
+    "<|special_2810|>",
+    "<|special_2811|>",
+    "<|special_2812|>",
+    "<|special_2813|>",
+    "<|special_2814|>",
+    "<|special_2815|>",
+    "<|special_2816|>",
+    "<|special_2817|>",
+    "<|special_2818|>",
+    "<|special_2819|>",
+    "<|special_2820|>",
+    "<|special_2821|>",
+    "<|special_2822|>",
+    "<|special_2823|>",
+    "<|special_2824|>",
+    "<|special_2825|>",
+    "<|special_2826|>",
+    "<|special_2827|>",
+    "<|special_2828|>",
+    "<|special_2829|>",
+    "<|special_2830|>",
+    "<|special_2831|>",
+    "<|special_2832|>",
+    "<|special_2833|>",
+    "<|special_2834|>",
+    "<|special_2835|>",
+    "<|special_2836|>",
+    "<|special_2837|>",
+    "<|special_2838|>",
+    "<|special_2839|>",
+    "<|special_2840|>",
+    "<|special_2841|>",
+    "<|special_2842|>",
+    "<|special_2843|>",
+    "<|special_2844|>",
+    "<|special_2845|>",
+    "<|special_2846|>",
+    "<|special_2847|>",
+    "<|special_2848|>",
+    "<|special_2849|>",
+    "<|special_2850|>",
+    "<|special_2851|>",
+    "<|special_2852|>",
+    "<|special_2853|>",
+    "<|special_2854|>",
+    "<|special_2855|>",
+    "<|special_2856|>",
+    "<|special_2857|>",
+    "<|special_2858|>",
+    "<|special_2859|>",
+    "<|special_2860|>",
+    "<|special_2861|>",
+    "<|special_2862|>",
+    "<|special_2863|>",
+    "<|special_2864|>",
+    "<|special_2865|>",
+    "<|special_2866|>",
+    "<|special_2867|>",
+    "<|special_2868|>",
+    "<|special_2869|>",
+    "<|special_2870|>",
+    "<|special_2871|>",
+    "<|special_2872|>",
+    "<|special_2873|>",
+    "<|special_2874|>",
+    "<|special_2875|>",
+    "<|special_2876|>",
+    "<|special_2877|>",
+    "<|special_2878|>",
+    "<|special_2879|>",
+    "<|special_2880|>",
+    "<|special_2881|>",
+    "<|special_2882|>",
+    "<|special_2883|>",
+    "<|special_2884|>",
+    "<|special_2885|>",
+    "<|special_2886|>",
+    "<|special_2887|>",
+    "<|special_2888|>",
+    "<|special_2889|>",
+    "<|special_2890|>",
+    "<|special_2891|>",
+    "<|special_2892|>",
+    "<|special_2893|>",
+    "<|special_2894|>",
+    "<|special_2895|>",
+    "<|special_2896|>",
+    "<|special_2897|>",
+    "<|special_2898|>",
+    "<|special_2899|>",
+    "<|special_2900|>",
+    "<|special_2901|>",
+    "<|special_2902|>",
+    "<|special_2903|>",
+    "<|special_2904|>",
+    "<|special_2905|>",
+    "<|special_2906|>",
+    "<|special_2907|>",
+    "<|special_2908|>",
+    "<|special_2909|>",
+    "<|special_2910|>",
+    "<|special_2911|>",
+    "<|special_2912|>",
+    "<|special_2913|>",
+    "<|special_2914|>",
+    "<|special_2915|>",
+    "<|special_2916|>",
+    "<|special_2917|>",
+    "<|special_2918|>",
+    "<|special_2919|>",
+    "<|special_2920|>",
+    "<|special_2921|>",
+    "<|special_2922|>",
+    "<|special_2923|>",
+    "<|special_2924|>",
+    "<|special_2925|>",
+    "<|special_2926|>",
+    "<|special_2927|>",
+    "<|special_2928|>",
+    "<|special_2929|>",
+    "<|special_2930|>",
+    "<|special_2931|>",
+    "<|special_2932|>",
+    "<|special_2933|>",
+    "<|special_2934|>",
+    "<|special_2935|>",
+    "<|special_2936|>",
+    "<|special_2937|>",
+    "<|special_2938|>",
+    "<|special_2939|>",
+    "<|special_2940|>",
+    "<|special_2941|>",
+    "<|special_2942|>",
+    "<|special_2943|>",
+    "<|special_2944|>",
+    "<|special_2945|>",
+    "<|special_2946|>",
+    "<|special_2947|>",
+    "<|special_2948|>",
+    "<|special_2949|>",
+    "<|special_2950|>",
+    "<|special_2951|>",
+    "<|special_2952|>",
+    "<|special_2953|>",
+    "<|special_2954|>",
+    "<|special_2955|>",
+    "<|special_2956|>",
+    "<|special_2957|>",
+    "<|special_2958|>",
+    "<|special_2959|>",
+    "<|special_2960|>",
+    "<|special_2961|>",
+    "<|special_2962|>",
+    "<|special_2963|>",
+    "<|special_2964|>",
+    "<|special_2965|>",
+    "<|special_2966|>",
+    "<|special_2967|>",
+    "<|special_2968|>",
+    "<|special_2969|>",
+    "<|special_2970|>",
+    "<|special_2971|>",
+    "<|special_2972|>",
+    "<|special_2973|>",
+    "<|special_2974|>",
+    "<|special_2975|>",
+    "<|special_2976|>",
+    "<|special_2977|>",
+    "<|special_2978|>",
+    "<|special_2979|>",
+    "<|special_2980|>",
+    "<|special_2981|>",
+    "<|special_2982|>",
+    "<|special_2983|>",
+    "<|special_2984|>",
+    "<|special_2985|>",
+    "<|special_2986|>",
+    "<|special_2987|>",
+    "<|special_2988|>",
+    "<|special_2989|>",
+    "<|special_2990|>",
+    "<|special_2991|>",
+    "<|special_2992|>",
+    "<|special_2993|>",
+    "<|special_2994|>",
+    "<|special_2995|>",
+    "<|special_2996|>",
+    "<|special_2997|>",
+    "<|special_2998|>",
+    "<|special_2999|>",
+    "<|special_3000|>",
+    "<|special_3001|>",
+    "<|special_3002|>",
+    "<|special_3003|>",
+    "<|special_3004|>",
+    "<|special_3005|>",
+    "<|special_3006|>",
+    "<|special_3007|>",
+    "<|special_3008|>",
+    "<|special_3009|>",
+    "<|special_3010|>",
+    "<|special_3011|>",
+    "<|special_3012|>",
+    "<|special_3013|>",
+    "<|special_3014|>",
+    "<|special_3015|>",
+    "<|special_3016|>",
+    "<|special_3017|>",
+    "<|special_3018|>",
+    "<|special_3019|>",
+    "<|special_3020|>",
+    "<|special_3021|>",
+    "<|special_3022|>",
+    "<|special_3023|>",
+    "<|special_3024|>",
+    "<|special_3025|>",
+    "<|special_3026|>",
+    "<|special_3027|>",
+    "<|special_3028|>",
+    "<|special_3029|>",
+    "<|special_3030|>",
+    "<|special_3031|>",
+    "<|special_3032|>",
+    "<|special_3033|>",
+    "<|special_3034|>",
+    "<|special_3035|>",
+    "<|special_3036|>",
+    "<|special_3037|>",
+    "<|special_3038|>",
+    "<|special_3039|>",
+    "<|special_3040|>",
+    "<|special_3041|>",
+    "<|special_3042|>",
+    "<|special_3043|>",
+    "<|special_3044|>",
+    "<|special_3045|>",
+    "<|special_3046|>",
+    "<|special_3047|>",
+    "<|special_3048|>",
+    "<|special_3049|>",
+    "<|special_3050|>",
+    "<|special_3051|>",
+    "<|special_3052|>",
+    "<|special_3053|>",
+    "<|special_3054|>",
+    "<|special_3055|>",
+    "<|special_3056|>",
+    "<|special_3057|>",
+    "<|special_3058|>",
+    "<|special_3059|>",
+    "<|special_3060|>",
+    "<|special_3061|>",
+    "<|special_3062|>",
+    "<|special_3063|>",
+    "<|special_3064|>",
+    "<|special_3065|>",
+    "<|special_3066|>",
+    "<|special_3067|>",
+    "<|special_3068|>",
+    "<|special_3069|>",
+    "<|special_3070|>",
+    "<|special_3071|>",
+    "<|special_3072|>",
+    "<|special_3073|>",
+    "<|special_3074|>",
+    "<|special_3075|>",
+    "<|special_3076|>",
+    "<|special_3077|>",
+    "<|special_3078|>",
+    "<|special_3079|>",
+    "<|special_3080|>",
+    "<|special_3081|>",
+    "<|special_3082|>",
+    "<|special_3083|>",
+    "<|special_3084|>",
+    "<|special_3085|>",
+    "<|special_3086|>",
+    "<|special_3087|>",
+    "<|special_3088|>",
+    "<|special_3089|>",
+    "<|special_3090|>",
+    "<|special_3091|>",
+    "<|special_3092|>",
+    "<|special_3093|>",
+    "<|special_3094|>",
+    "<|special_3095|>",
+    "<|special_3096|>",
+    "<|special_3097|>",
+    "<|special_3098|>",
+    "<|special_3099|>",
+    "<|special_3100|>",
+    "<|special_3101|>",
+    "<|special_3102|>",
+    "<|special_3103|>",
+    "<|special_3104|>",
+    "<|special_3105|>",
+    "<|special_3106|>",
+    "<|special_3107|>",
+    "<|special_3108|>",
+    "<|special_3109|>",
+    "<|special_3110|>",
+    "<|special_3111|>",
+    "<|special_3112|>",
+    "<|special_3113|>",
+    "<|special_3114|>",
+    "<|special_3115|>",
+    "<|special_3116|>",
+    "<|special_3117|>",
+    "<|special_3118|>",
+    "<|special_3119|>",
+    "<|special_3120|>",
+    "<|special_3121|>",
+    "<|special_3122|>",
+    "<|special_3123|>",
+    "<|special_3124|>",
+    "<|special_3125|>",
+    "<|special_3126|>",
+    "<|special_3127|>",
+    "<|special_3128|>",
+    "<|special_3129|>",
+    "<|special_3130|>",
+    "<|special_3131|>",
+    "<|special_3132|>",
+    "<|special_3133|>",
+    "<|special_3134|>",
+    "<|special_3135|>",
+    "<|special_3136|>",
+    "<|special_3137|>",
+    "<|special_3138|>",
+    "<|special_3139|>",
+    "<|special_3140|>",
+    "<|special_3141|>",
+    "<|special_3142|>",
+    "<|special_3143|>",
+    "<|special_3144|>",
+    "<|special_3145|>",
+    "<|special_3146|>",
+    "<|special_3147|>",
+    "<|special_3148|>",
+    "<|special_3149|>",
+    "<|special_3150|>",
+    "<|special_3151|>",
+    "<|special_3152|>",
+    "<|special_3153|>",
+    "<|special_3154|>",
+    "<|special_3155|>",
+    "<|special_3156|>",
+    "<|special_3157|>",
+    "<|special_3158|>",
+    "<|special_3159|>",
+    "<|special_3160|>",
+    "<|special_3161|>",
+    "<|special_3162|>",
+    "<|special_3163|>",
+    "<|special_3164|>",
+    "<|special_3165|>",
+    "<|special_3166|>",
+    "<|special_3167|>",
+    "<|special_3168|>",
+    "<|special_3169|>",
+    "<|special_3170|>",
+    "<|special_3171|>",
+    "<|special_3172|>",
+    "<|special_3173|>",
+    "<|special_3174|>",
+    "<|special_3175|>",
+    "<|special_3176|>",
+    "<|special_3177|>",
+    "<|special_3178|>",
+    "<|special_3179|>",
+    "<|special_3180|>",
+    "<|special_3181|>",
+    "<|special_3182|>",
+    "<|special_3183|>",
+    "<|special_3184|>",
+    "<|special_3185|>",
+    "<|special_3186|>",
+    "<|special_3187|>",
+    "<|special_3188|>",
+    "<|special_3189|>",
+    "<|special_3190|>",
+    "<|special_3191|>",
+    "<|special_3192|>",
+    "<|special_3193|>",
+    "<|special_3194|>",
+    "<|special_3195|>",
+    "<|special_3196|>",
+    "<|special_3197|>",
+    "<|special_3198|>",
+    "<|special_3199|>",
+    "<|special_3200|>",
+    "<|special_3201|>",
+    "<|special_3202|>",
+    "<|special_3203|>",
+    "<|special_3204|>",
+    "<|special_3205|>",
+    "<|special_3206|>",
+    "<|special_3207|>",
+    "<|special_3208|>",
+    "<|special_3209|>",
+    "<|special_3210|>",
+    "<|special_3211|>",
+    "<|special_3212|>",
+    "<|special_3213|>",
+    "<|special_3214|>",
+    "<|special_3215|>",
+    "<|special_3216|>",
+    "<|special_3217|>",
+    "<|special_3218|>",
+    "<|special_3219|>",
+    "<|special_3220|>",
+    "<|special_3221|>",
+    "<|special_3222|>",
+    "<|special_3223|>",
+    "<|special_3224|>",
+    "<|special_3225|>",
+    "<|special_3226|>",
+    "<|special_3227|>",
+    "<|special_3228|>",
+    "<|special_3229|>",
+    "<|special_3230|>",
+    "<|special_3231|>",
+    "<|special_3232|>",
+    "<|special_3233|>",
+    "<|special_3234|>",
+    "<|special_3235|>",
+    "<|special_3236|>",
+    "<|special_3237|>",
+    "<|special_3238|>",
+    "<|special_3239|>",
+    "<|special_3240|>",
+    "<|special_3241|>",
+    "<|special_3242|>",
+    "<|special_3243|>",
+    "<|special_3244|>",
+    "<|special_3245|>",
+    "<|special_3246|>",
+    "<|special_3247|>",
+    "<|special_3248|>",
+    "<|special_3249|>",
+    "<|special_3250|>",
+    "<|special_3251|>",
+    "<|special_3252|>",
+    "<|special_3253|>",
+    "<|special_3254|>",
+    "<|special_3255|>",
+    "<|special_3256|>",
+    "<|special_3257|>",
+    "<|special_3258|>",
+    "<|special_3259|>",
+    "<|special_3260|>",
+    "<|special_3261|>",
+    "<|special_3262|>",
+    "<|special_3263|>",
+    "<|special_3264|>",
+    "<|special_3265|>",
+    "<|special_3266|>",
+    "<|special_3267|>",
+    "<|special_3268|>",
+    "<|special_3269|>",
+    "<|special_3270|>",
+    "<|special_3271|>",
+    "<|special_3272|>",
+    "<|special_3273|>",
+    "<|special_3274|>",
+    "<|special_3275|>",
+    "<|special_3276|>",
+    "<|special_3277|>",
+    "<|special_3278|>",
+    "<|special_3279|>",
+    "<|special_3280|>",
+    "<|special_3281|>",
+    "<|special_3282|>",
+    "<|special_3283|>",
+    "<|special_3284|>",
+    "<|special_3285|>",
+    "<|special_3286|>",
+    "<|special_3287|>",
+    "<|special_3288|>",
+    "<|special_3289|>",
+    "<|special_3290|>",
+    "<|special_3291|>",
+    "<|special_3292|>",
+    "<|special_3293|>",
+    "<|special_3294|>",
+    "<|special_3295|>",
+    "<|special_3296|>",
+    "<|special_3297|>",
+    "<|special_3298|>",
+    "<|special_3299|>",
+    "<|special_3300|>",
+    "<|special_3301|>",
+    "<|special_3302|>",
+    "<|special_3303|>",
+    "<|special_3304|>",
+    "<|special_3305|>",
+    "<|special_3306|>",
+    "<|special_3307|>",
+    "<|special_3308|>",
+    "<|special_3309|>",
+    "<|special_3310|>",
+    "<|special_3311|>",
+    "<|special_3312|>",
+    "<|special_3313|>",
+    "<|special_3314|>",
+    "<|special_3315|>",
+    "<|special_3316|>",
+    "<|special_3317|>",
+    "<|special_3318|>",
+    "<|special_3319|>",
+    "<|special_3320|>",
+    "<|special_3321|>",
+    "<|special_3322|>",
+    "<|special_3323|>",
+    "<|special_3324|>",
+    "<|special_3325|>",
+    "<|special_3326|>",
+    "<|special_3327|>",
+    "<|special_3328|>",
+    "<|special_3329|>",
+    "<|special_3330|>",
+    "<|special_3331|>",
+    "<|special_3332|>",
+    "<|special_3333|>",
+    "<|special_3334|>",
+    "<|special_3335|>",
+    "<|special_3336|>",
+    "<|special_3337|>",
+    "<|special_3338|>",
+    "<|special_3339|>",
+    "<|special_3340|>",
+    "<|special_3341|>",
+    "<|special_3342|>",
+    "<|special_3343|>",
+    "<|special_3344|>",
+    "<|special_3345|>",
+    "<|special_3346|>",
+    "<|special_3347|>",
+    "<|special_3348|>",
+    "<|special_3349|>",
+    "<|special_3350|>",
+    "<|special_3351|>",
+    "<|special_3352|>",
+    "<|special_3353|>",
+    "<|special_3354|>",
+    "<|special_3355|>",
+    "<|special_3356|>",
+    "<|special_3357|>",
+    "<|special_3358|>",
+    "<|special_3359|>",
+    "<|special_3360|>",
+    "<|special_3361|>",
+    "<|special_3362|>",
+    "<|special_3363|>",
+    "<|special_3364|>",
+    "<|special_3365|>",
+    "<|special_3366|>",
+    "<|special_3367|>",
+    "<|special_3368|>",
+    "<|special_3369|>",
+    "<|special_3370|>",
+    "<|special_3371|>",
+    "<|special_3372|>",
+    "<|special_3373|>",
+    "<|special_3374|>",
+    "<|special_3375|>",
+    "<|special_3376|>",
+    "<|special_3377|>",
+    "<|special_3378|>",
+    "<|special_3379|>",
+    "<|special_3380|>",
+    "<|special_3381|>",
+    "<|special_3382|>",
+    "<|special_3383|>",
+    "<|special_3384|>",
+    "<|special_3385|>",
+    "<|special_3386|>",
+    "<|special_3387|>",
+    "<|special_3388|>",
+    "<|special_3389|>",
+    "<|special_3390|>",
+    "<|special_3391|>",
+    "<|special_3392|>",
+    "<|special_3393|>",
+    "<|special_3394|>",
+    "<|special_3395|>",
+    "<|special_3396|>",
+    "<|special_3397|>",
+    "<|special_3398|>",
+    "<|special_3399|>",
+    "<|special_3400|>",
+    "<|special_3401|>",
+    "<|special_3402|>",
+    "<|special_3403|>",
+    "<|special_3404|>",
+    "<|special_3405|>",
+    "<|special_3406|>",
+    "<|special_3407|>",
+    "<|special_3408|>",
+    "<|special_3409|>",
+    "<|special_3410|>",
+    "<|special_3411|>",
+    "<|special_3412|>",
+    "<|special_3413|>",
+    "<|special_3414|>",
+    "<|special_3415|>",
+    "<|special_3416|>",
+    "<|special_3417|>",
+    "<|special_3418|>",
+    "<|special_3419|>",
+    "<|special_3420|>",
+    "<|special_3421|>",
+    "<|special_3422|>",
+    "<|special_3423|>",
+    "<|special_3424|>",
+    "<|special_3425|>",
+    "<|special_3426|>",
+    "<|special_3427|>",
+    "<|special_3428|>",
+    "<|special_3429|>",
+    "<|special_3430|>",
+    "<|special_3431|>",
+    "<|special_3432|>",
+    "<|special_3433|>",
+    "<|special_3434|>",
+    "<|special_3435|>",
+    "<|special_3436|>",
+    "<|special_3437|>",
+    "<|special_3438|>",
+    "<|special_3439|>",
+    "<|special_3440|>",
+    "<|special_3441|>",
+    "<|special_3442|>",
+    "<|special_3443|>",
+    "<|special_3444|>",
+    "<|special_3445|>",
+    "<|special_3446|>",
+    "<|special_3447|>",
+    "<|special_3448|>",
+    "<|special_3449|>",
+    "<|special_3450|>",
+    "<|special_3451|>",
+    "<|special_3452|>",
+    "<|special_3453|>",
+    "<|special_3454|>",
+    "<|special_3455|>",
+    "<|special_3456|>",
+    "<|special_3457|>",
+    "<|special_3458|>",
+    "<|special_3459|>",
+    "<|special_3460|>",
+    "<|special_3461|>",
+    "<|special_3462|>",
+    "<|special_3463|>",
+    "<|special_3464|>",
+    "<|special_3465|>",
+    "<|special_3466|>",
+    "<|special_3467|>",
+    "<|special_3468|>",
+    "<|special_3469|>",
+    "<|special_3470|>",
+    "<|special_3471|>",
+    "<|special_3472|>",
+    "<|special_3473|>",
+    "<|special_3474|>",
+    "<|special_3475|>",
+    "<|special_3476|>",
+    "<|special_3477|>",
+    "<|special_3478|>",
+    "<|special_3479|>",
+    "<|special_3480|>",
+    "<|special_3481|>",
+    "<|special_3482|>",
+    "<|special_3483|>",
+    "<|special_3484|>",
+    "<|special_3485|>",
+    "<|special_3486|>",
+    "<|special_3487|>",
+    "<|special_3488|>",
+    "<|special_3489|>",
+    "<|special_3490|>",
+    "<|special_3491|>",
+    "<|special_3492|>",
+    "<|special_3493|>",
+    "<|special_3494|>",
+    "<|special_3495|>",
+    "<|special_3496|>",
+    "<|special_3497|>",
+    "<|special_3498|>",
+    "<|special_3499|>",
+    "<|special_3500|>",
+    "<|special_3501|>",
+    "<|special_3502|>",
+    "<|special_3503|>",
+    "<|special_3504|>",
+    "<|special_3505|>",
+    "<|special_3506|>",
+    "<|special_3507|>",
+    "<|special_3508|>",
+    "<|special_3509|>",
+    "<|special_3510|>",
+    "<|special_3511|>",
+    "<|special_3512|>",
+    "<|special_3513|>",
+    "<|special_3514|>",
+    "<|special_3515|>",
+    "<|special_3516|>",
+    "<|special_3517|>",
+    "<|special_3518|>",
+    "<|special_3519|>",
+    "<|special_3520|>",
+    "<|special_3521|>",
+    "<|special_3522|>",
+    "<|special_3523|>",
+    "<|special_3524|>",
+    "<|special_3525|>",
+    "<|special_3526|>",
+    "<|special_3527|>",
+    "<|special_3528|>",
+    "<|special_3529|>",
+    "<|special_3530|>",
+    "<|special_3531|>",
+    "<|special_3532|>",
+    "<|special_3533|>",
+    "<|special_3534|>",
+    "<|special_3535|>",
+    "<|special_3536|>",
+    "<|special_3537|>",
+    "<|special_3538|>",
+    "<|special_3539|>",
+    "<|special_3540|>",
+    "<|special_3541|>",
+    "<|special_3542|>",
+    "<|special_3543|>",
+    "<|special_3544|>",
+    "<|special_3545|>",
+    "<|special_3546|>",
+    "<|special_3547|>",
+    "<|special_3548|>",
+    "<|special_3549|>",
+    "<|special_3550|>",
+    "<|special_3551|>",
+    "<|special_3552|>",
+    "<|special_3553|>",
+    "<|special_3554|>",
+    "<|special_3555|>",
+    "<|special_3556|>",
+    "<|special_3557|>",
+    "<|special_3558|>",
+    "<|special_3559|>",
+    "<|special_3560|>",
+    "<|special_3561|>",
+    "<|special_3562|>",
+    "<|special_3563|>",
+    "<|special_3564|>",
+    "<|special_3565|>",
+    "<|special_3566|>",
+    "<|special_3567|>",
+    "<|special_3568|>",
+    "<|special_3569|>",
+    "<|special_3570|>",
+    "<|special_3571|>",
+    "<|special_3572|>",
+    "<|special_3573|>",
+    "<|special_3574|>",
+    "<|special_3575|>",
+    "<|special_3576|>",
+    "<|special_3577|>",
+    "<|special_3578|>",
+    "<|special_3579|>",
+    "<|special_3580|>",
+    "<|special_3581|>",
+    "<|special_3582|>",
+    "<|special_3583|>",
+    "<|special_3584|>",
+    "<|special_3585|>",
+    "<|special_3586|>",
+    "<|special_3587|>",
+    "<|special_3588|>",
+    "<|special_3589|>",
+    "<|special_3590|>",
+    "<|special_3591|>",
+    "<|special_3592|>",
+    "<|special_3593|>",
+    "<|special_3594|>",
+    "<|special_3595|>",
+    "<|special_3596|>",
+    "<|special_3597|>",
+    "<|special_3598|>",
+    "<|special_3599|>",
+    "<|special_3600|>",
+    "<|special_3601|>",
+    "<|special_3602|>",
+    "<|special_3603|>",
+    "<|special_3604|>",
+    "<|special_3605|>",
+    "<|special_3606|>",
+    "<|special_3607|>",
+    "<|special_3608|>",
+    "<|special_3609|>",
+    "<|special_3610|>",
+    "<|special_3611|>",
+    "<|special_3612|>",
+    "<|special_3613|>",
+    "<|special_3614|>",
+    "<|special_3615|>",
+    "<|special_3616|>",
+    "<|special_3617|>",
+    "<|special_3618|>",
+    "<|special_3619|>",
+    "<|special_3620|>",
+    "<|special_3621|>",
+    "<|special_3622|>",
+    "<|special_3623|>",
+    "<|special_3624|>",
+    "<|special_3625|>",
+    "<|special_3626|>",
+    "<|special_3627|>",
+    "<|special_3628|>",
+    "<|special_3629|>",
+    "<|special_3630|>",
+    "<|special_3631|>",
+    "<|special_3632|>",
+    "<|special_3633|>",
+    "<|special_3634|>",
+    "<|special_3635|>",
+    "<|special_3636|>",
+    "<|special_3637|>",
+    "<|special_3638|>",
+    "<|special_3639|>",
+    "<|special_3640|>",
+    "<|special_3641|>",
+    "<|special_3642|>",
+    "<|special_3643|>",
+    "<|special_3644|>",
+    "<|special_3645|>",
+    "<|special_3646|>",
+    "<|special_3647|>",
+    "<|special_3648|>",
+    "<|special_3649|>",
+    "<|special_3650|>",
+    "<|special_3651|>",
+    "<|special_3652|>",
+    "<|special_3653|>",
+    "<|special_3654|>",
+    "<|special_3655|>",
+    "<|special_3656|>",
+    "<|special_3657|>",
+    "<|special_3658|>",
+    "<|special_3659|>",
+    "<|special_3660|>",
+    "<|special_3661|>",
+    "<|special_3662|>",
+    "<|special_3663|>",
+    "<|special_3664|>",
+    "<|special_3665|>",
+    "<|special_3666|>",
+    "<|special_3667|>",
+    "<|special_3668|>",
+    "<|special_3669|>",
+    "<|special_3670|>",
+    "<|special_3671|>",
+    "<|special_3672|>",
+    "<|special_3673|>",
+    "<|special_3674|>",
+    "<|special_3675|>",
+    "<|special_3676|>",
+    "<|special_3677|>",
+    "<|special_3678|>",
+    "<|special_3679|>",
+    "<|special_3680|>",
+    "<|special_3681|>",
+    "<|special_3682|>",
+    "<|special_3683|>",
+    "<|special_3684|>",
+    "<|special_3685|>",
+    "<|special_3686|>",
+    "<|special_3687|>",
+    "<|special_3688|>",
+    "<|special_3689|>",
+    "<|special_3690|>",
+    "<|special_3691|>",
+    "<|special_3692|>",
+    "<|special_3693|>",
+    "<|special_3694|>",
+    "<|special_3695|>",
+    "<|special_3696|>",
+    "<|special_3697|>",
+    "<|special_3698|>",
+    "<|special_3699|>",
+    "<|special_3700|>",
+    "<|special_3701|>",
+    "<|special_3702|>",
+    "<|special_3703|>",
+    "<|special_3704|>",
+    "<|special_3705|>",
+    "<|special_3706|>",
+    "<|special_3707|>",
+    "<|special_3708|>",
+    "<|special_3709|>",
+    "<|special_3710|>",
+    "<|special_3711|>",
+    "<|special_3712|>",
+    "<|special_3713|>",
+    "<|special_3714|>",
+    "<|special_3715|>",
+    "<|special_3716|>",
+    "<|special_3717|>",
+    "<|special_3718|>",
+    "<|special_3719|>",
+    "<|special_3720|>",
+    "<|special_3721|>",
+    "<|special_3722|>",
+    "<|special_3723|>",
+    "<|special_3724|>",
+    "<|special_3725|>",
+    "<|special_3726|>",
+    "<|special_3727|>",
+    "<|special_3728|>",
+    "<|special_3729|>",
+    "<|special_3730|>",
+    "<|special_3731|>",
+    "<|special_3732|>",
+    "<|special_3733|>",
+    "<|special_3734|>",
+    "<|special_3735|>",
+    "<|special_3736|>",
+    "<|special_3737|>",
+    "<|special_3738|>",
+    "<|special_3739|>",
+    "<|special_3740|>",
+    "<|special_3741|>",
+    "<|special_3742|>",
+    "<|special_3743|>",
+    "<|special_3744|>",
+    "<|special_3745|>",
+    "<|special_3746|>",
+    "<|special_3747|>",
+    "<|special_3748|>",
+    "<|special_3749|>",
+    "<|special_3750|>",
+    "<|special_3751|>",
+    "<|special_3752|>",
+    "<|special_3753|>",
+    "<|special_3754|>",
+    "<|special_3755|>",
+    "<|special_3756|>",
+    "<|special_3757|>",
+    "<|special_3758|>",
+    "<|special_3759|>",
+    "<|special_3760|>",
+    "<|special_3761|>",
+    "<|special_3762|>",
+    "<|special_3763|>",
+    "<|special_3764|>",
+    "<|special_3765|>",
+    "<|special_3766|>",
+    "<|special_3767|>",
+    "<|special_3768|>",
+    "<|special_3769|>",
+    "<|special_3770|>",
+    "<|special_3771|>",
+    "<|special_3772|>",
+    "<|special_3773|>",
+    "<|special_3774|>",
+    "<|special_3775|>",
+    "<|special_3776|>",
+    "<|special_3777|>",
+    "<|special_3778|>",
+    "<|special_3779|>",
+    "<|special_3780|>",
+    "<|special_3781|>",
+    "<|special_3782|>",
+    "<|special_3783|>",
+    "<|special_3784|>",
+    "<|special_3785|>",
+    "<|special_3786|>",
+    "<|special_3787|>",
+    "<|special_3788|>",
+    "<|special_3789|>",
+    "<|special_3790|>",
+    "<|special_3791|>",
+    "<|special_3792|>",
+    "<|special_3793|>",
+    "<|special_3794|>",
+    "<|special_3795|>",
+    "<|special_3796|>",
+    "<|special_3797|>",
+    "<|special_3798|>",
+    "<|special_3799|>",
+    "<|special_3800|>",
+    "<|special_3801|>",
+    "<|special_3802|>",
+    "<|special_3803|>",
+    "<|special_3804|>",
+    "<|special_3805|>",
+    "<|special_3806|>",
+    "<|special_3807|>",
+    "<|special_3808|>",
+    "<|special_3809|>",
+    "<|special_3810|>",
+    "<|special_3811|>",
+    "<|special_3812|>",
+    "<|special_3813|>",
+    "<|special_3814|>",
+    "<|special_3815|>",
+    "<|special_3816|>",
+    "<|special_3817|>",
+    "<|special_3818|>",
+    "<|special_3819|>",
+    "<|special_3820|>",
+    "<|special_3821|>",
+    "<|special_3822|>",
+    "<|special_3823|>",
+    "<|special_3824|>",
+    "<|special_3825|>",
+    "<|special_3826|>",
+    "<|special_3827|>",
+    "<|special_3828|>",
+    "<|special_3829|>",
+    "<|special_3830|>",
+    "<|special_3831|>",
+    "<|special_3832|>",
+    "<|special_3833|>",
+    "<|special_3834|>",
+    "<|special_3835|>",
+    "<|special_3836|>",
+    "<|special_3837|>",
+    "<|special_3838|>",
+    "<|special_3839|>",
+    "<|special_3840|>",
+    "<|special_3841|>",
+    "<|special_3842|>",
+    "<|special_3843|>",
+    "<|special_3844|>",
+    "<|special_3845|>",
+    "<|special_3846|>",
+    "<|special_3847|>",
+    "<|special_3848|>",
+    "<|special_3849|>",
+    "<|special_3850|>",
+    "<|special_3851|>",
+    "<|special_3852|>",
+    "<|special_3853|>",
+    "<|special_3854|>",
+    "<|special_3855|>",
+    "<|special_3856|>",
+    "<|special_3857|>",
+    "<|special_3858|>",
+    "<|special_3859|>",
+    "<|special_3860|>",
+    "<|special_3861|>",
+    "<|special_3862|>",
+    "<|special_3863|>",
+    "<|special_3864|>",
+    "<|special_3865|>",
+    "<|special_3866|>",
+    "<|special_3867|>",
+    "<|special_3868|>",
+    "<|special_3869|>",
+    "<|special_3870|>",
+    "<|special_3871|>",
+    "<|special_3872|>",
+    "<|special_3873|>",
+    "<|special_3874|>",
+    "<|special_3875|>",
+    "<|special_3876|>",
+    "<|special_3877|>",
+    "<|special_3878|>",
+    "<|special_3879|>",
+    "<|special_3880|>",
+    "<|special_3881|>",
+    "<|special_3882|>",
+    "<|special_3883|>",
+    "<|special_3884|>",
+    "<|special_3885|>",
+    "<|special_3886|>",
+    "<|special_3887|>",
+    "<|special_3888|>",
+    "<|special_3889|>",
+    "<|special_3890|>",
+    "<|special_3891|>",
+    "<|special_3892|>",
+    "<|special_3893|>",
+    "<|special_3894|>",
+    "<|special_3895|>",
+    "<|special_3896|>",
+    "<|special_3897|>",
+    "<|special_3898|>",
+    "<|special_3899|>",
+    "<|special_3900|>",
+    "<|special_3901|>",
+    "<|special_3902|>",
+    "<|special_3903|>",
+    "<|special_3904|>",
+    "<|special_3905|>",
+    "<|special_3906|>",
+    "<|special_3907|>",
+    "<|special_3908|>",
+    "<|special_3909|>",
+    "<|special_3910|>",
+    "<|special_3911|>",
+    "<|special_3912|>",
+    "<|special_3913|>",
+    "<|special_3914|>",
+    "<|special_3915|>",
+    "<|special_3916|>",
+    "<|special_3917|>",
+    "<|special_3918|>",
+    "<|special_3919|>",
+    "<|special_3920|>",
+    "<|special_3921|>",
+    "<|special_3922|>",
+    "<|special_3923|>",
+    "<|special_3924|>",
+    "<|special_3925|>",
+    "<|special_3926|>",
+    "<|special_3927|>",
+    "<|special_3928|>",
+    "<|special_3929|>",
+    "<|special_3930|>",
+    "<|special_3931|>",
+    "<|special_3932|>",
+    "<|special_3933|>",
+    "<|special_3934|>",
+    "<|special_3935|>",
+    "<|special_3936|>",
+    "<|special_3937|>",
+    "<|special_3938|>",
+    "<|special_3939|>",
+    "<|special_3940|>",
+    "<|special_3941|>",
+    "<|special_3942|>",
+    "<|special_3943|>",
+    "<|special_3944|>",
+    "<|special_3945|>",
+    "<|special_3946|>",
+    "<|special_3947|>",
+    "<|special_3948|>",
+    "<|special_3949|>",
+    "<|special_3950|>",
+    "<|special_3951|>",
+    "<|special_3952|>",
+    "<|special_3953|>",
+    "<|special_3954|>",
+    "<|special_3955|>",
+    "<|special_3956|>",
+    "<|special_3957|>",
+    "<|special_3958|>",
+    "<|special_3959|>",
+    "<|special_3960|>",
+    "<|special_3961|>",
+    "<|special_3962|>",
+    "<|special_3963|>",
+    "<|special_3964|>",
+    "<|special_3965|>",
+    "<|special_3966|>",
+    "<|special_3967|>",
+    "<|special_3968|>",
+    "<|special_3969|>",
+    "<|special_3970|>",
+    "<|special_3971|>",
+    "<|special_3972|>",
+    "<|special_3973|>",
+    "<|special_3974|>",
+    "<|special_3975|>",
+    "<|special_3976|>",
+    "<|special_3977|>",
+    "<|special_3978|>",
+    "<|special_3979|>",
+    "<|special_3980|>",
+    "<|special_3981|>",
+    "<|special_3982|>",
+    "<|special_3983|>",
+    "<|special_3984|>",
+    "<|special_3985|>",
+    "<|special_3986|>",
+    "<|special_3987|>",
+    "<|special_3988|>",
+    "<|special_3989|>",
+    "<|special_3990|>",
+    "<|special_3991|>",
+    "<|special_3992|>",
+    "<|special_3993|>",
+    "<|special_3994|>",
+    "<|special_3995|>",
+    "<|special_3996|>",
+    "<|special_3997|>",
+    "<|special_3998|>",
+    "<|special_3999|>",
+    "<|special_4000|>",
+    "<|special_4001|>",
+    "<|special_4002|>",
+    "<|special_4003|>",
+    "<|special_4004|>",
+    "<|special_4005|>",
+    "<|special_4006|>",
+    "<|special_4007|>",
+    "<|special_4008|>",
+    "<|special_4009|>",
+    "<|special_4010|>",
+    "<|special_4011|>",
+    "<|special_4012|>",
+    "<|special_4013|>",
+    "<|special_4014|>",
+    "<|special_4015|>",
+    "<|special_4016|>",
+    "<|special_4017|>",
+    "<|special_4018|>",
+    "<|special_4019|>",
+    "<|special_4020|>",
+    "<|special_4021|>",
+    "<|special_4022|>",
+    "<|special_4023|>",
+    "<|special_4024|>",
+    "<|special_4025|>",
+    "<|special_4026|>",
+    "<|special_4027|>",
+    "<|special_4028|>",
+    "<|special_4029|>",
+    "<|special_4030|>",
+    "<|special_4031|>",
+    "<|special_4032|>",
+    "<|special_4033|>",
+    "<|special_4034|>",
+    "<|special_4035|>",
+    "<|special_4036|>",
+    "<|special_4037|>",
+    "<|special_4038|>",
+    "<|special_4039|>",
+    "<|special_4040|>",
+    "<|special_4041|>",
+    "<|special_4042|>",
+    "<|special_4043|>",
+    "<|special_4044|>",
+    "<|special_4045|>",
+    "<|special_4046|>",
+    "<|special_4047|>",
+    "<|special_4048|>",
+    "<|special_4049|>",
+    "<|special_4050|>",
+    "<|special_4051|>",
+    "<|special_4052|>",
+    "<|special_4053|>",
+    "<|special_4054|>",
+    "<|special_4055|>",
+    "<|special_4056|>",
+    "<|special_4057|>",
+    "<|special_4058|>",
+    "<|special_4059|>",
+    "<|special_4060|>",
+    "<|special_4061|>",
+    "<|special_4062|>",
+    "<|special_4063|>",
+    "<|special_4064|>",
+    "<|special_4065|>",
+    "<|special_4066|>",
+    "<|special_4067|>",
+    "<|special_4068|>",
+    "<|special_4069|>",
+    "<|special_4070|>",
+    "<|special_4071|>",
+    "<|special_4072|>",
+    "<|special_4073|>",
+    "<|special_4074|>",
+    "<|special_4075|>",
+    "<|special_4076|>",
+    "<|special_4077|>",
+    "<|special_4078|>",
+    "<|special_4079|>",
+    "<|special_4080|>",
+    "<|special_4081|>",
+    "<|special_4082|>",
+    "<|special_4083|>",
+    "<|special_4084|>",
+    "<|special_4085|>",
+    "<|special_4086|>",
+    "<|special_4087|>",
+    "<|special_4088|>",
+    "<|special_4089|>",
+    "<|special_4090|>",
+    "<|special_4091|>",
+    "<|special_4092|>",
+    "<|special_4093|>",
+    "<|special_4094|>",
+    "<|special_4095|>"
+  ],
+  "bos_token": {
+    "content": "<|startoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b7a8c81670e4651cc290fbe0ab397e0357abfa933964b9b6f56f508557c04c19
+size 16473485