Add files using upload-large-folder tool

CODE_OF_CONDUCT.md

@@ -0,0 +1,9 @@
+# Microsoft Open Source Code of Conduct
+
+This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
+
+Resources:
+
+- [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/)
+- [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)
+- Contact [[email protected]](mailto:[email protected]) with questions or concerns

LICENSE

@@ -0,0 +1,21 @@
+    MIT License
+
+    Copyright (c) Microsoft Corporation.
+
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE

README.md

@@ -1,3 +1,151 @@
----
-license: mit
----
+---
+license: mit
+---
+
+MAI-DS-R1 is a DeepSeek-R1 reasoning model that has been post-trained by the Microsoft AI team to improve its responsiveness on blocked topics and its risk profile, while maintaining its reasoning capabilities and competitive performance. 
+
+## Model Details
+
+### Model Description
+MAI-DS-R1 is a DeepSeek-R1 reasoning model that has been post-trained by Microsoft AI team to fill in information gaps in the previous version of the model and to improve its risk profile, while maintaining R1 reasoning capabilities. The model was trained using 110k Safety and Non-Compliance examples from [Tulu](https://huggingface.co/datasets/allenai/tulu-3-sft-mixture) 3 SFT dataset, in addition to a dataset of ~350k multilingual examples internally developed capturing various topics with reported biases.  
+
+MAI-DS-R1 has successfully unblocked the majority of previously blocked queries from the original R1 model while outperforming the recently published R1-1776 model (post-trained by Perplexity) in relevant safety benchmarks.  These results were achieved while preserving the general reasoning capabilities of the original DeepSeek-R1.  
+
+*Please note: Microsoft has post-trained this model to address certain limitations relevant to its outputs, but previous limitations and considerations for the model remain, including security considerations.*
+
+## Uses
+
+### Direct Use
+MAI-DS-R1 preserves the general reasoning capabilities of DeepSeek-R1 and can be used for broad language understanding and generation tasks, especially in complex reasoning and problem-solving. Primary direct use incudes:  
+
+- **General text generation and understanding** – Producing coherent, contextually relevant text for a wide range of prompts. This includes engaging in dialogue, writing essays, or continuing a story based on a given prompt. 
+
+- **General knowledge tasks** – Answering open-domain questions requiring factual knowledge. 
+
+- **Reasoning and problem solving** – Handling multi-step reasoning tasks, such as math word problems or logic puzzles, by employing chain-of-thought strategies. 
+
+- **Code generation and comprehension** – Assisting with programming tasks by generating code snippets or explaining code. 
+
+- **Scientific and academic applications** – Assisting with structured problem-solving in STEM and research domains. 
+
+### Downstream Use *(Optional)*
+
+The model can serve as a foundation for further fine-tuning in domain-specific reasoning tasks, such as automated tutoring systems for mathematics, coding assistants, and research tools in scientific or technical fields. 
+
+### Out-of-Scope Use
+Certain application domains are out-of-scope either due to ethical/safety concerns or because the model lacks the necessary reliability in those areas. The following usage is out of scope: 
+
+- **Medical or health advice** – The model is not a medical device and has no guarantee of providing accurate medical diagnoses or safe treatment recommendations. 
+
+- **Legal advice** – The model is not a lawyer and should not be entrusted with giving definitive legal counsel, interpreting laws, or making legal decisions on its own. 
+
+- **Safety-critical systems** – The model is not suited for autonomous systems where failures could cause injury, loss of life, or significant property damage. This includes use in self-driving vehicles, aircraft control, medical life-support systems, or industrial control without human oversight.  
+
+- **High-stakes decision support** – The model should not be relied on for decisions affecting finances, security, or personal well-being, such as financial planning or investment advice. 
+
+- **Malicious or unethical Use** – The model must not be used to produce harmful, illegal, deceptive, or unethical content, including hate speech, violence, harassment, or violations of privacy or IP rights. 
+
+## Bias, Risks, and Limitations
+
+- **Biases**: The model may retain biases present in the training data and in the original DeepSeek‑R1, particularly around cultural and demographic aspects. 
+
+- **Risks**: The model may still hallucinate facts, be vulnerable to adversarial prompts, or generate unsafe, biased, or harmful content under certain conditions. Developers should implement content moderation and usage monitoring to mitigate misuse. 
+
+- **Limitations**: MAI-DS-R1 shares DeepSeek-R1’s knowledge cutoff and may lack awareness of recent events or domain-specific facts.  
+
+## Recommendations
+To ensure responsible use, we recommend the following: 
+
+- **Transparency on Limitations**: It is recommended that users are made explicitly aware of the model’s potential biases and limitations. 
+
+- **Human Oversight and Verification**: Both direct and downstream users should implement human review or automated validation of outputs when deploying the model in sensitive or high-stakes scenarios. 
+
+- **Usage Safeguards**: Developers should integrate content filtering, prompt engineering best practices, and continuous monitoring to mitigate risks and ensure the model’s outputs meet the intended safety and quality standards. 
+
+- **Legal and Regulatory Compliance**: The model may output politically sensitive content (e.g., Chinese governance, historical events) that could conflict with local laws or platform policies. Operators must ensure compliance with regional regulations. 
+
+## Evaluation
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+The model was evaluated on a variety of benchmarks, covering different tasks and addressing both performance and harm mitigation concerns. Key benchmarks include: 
+
+1. **Public Benchmarks**: These cover a wide range of tasks, such as natural language inference, question answering, mathematical reasoning, commonsense reasoning, code generation, and code completion. It evaluates the model’s general knowledge and reasoning capabilities.  
+
+2. **Blocking Test Set**: This set consists of 3.3k prompts on various blocked topics from R1, covering 11 languages. It evaluates the model’s ability to unblock previously blocked content across different languages. 
+
+3. **Harm Mitigation Test Set**: This set is a [split](https://github.com/nouhadziri/safety-eval-fork/blob/main/evaluation/tasks/generation/harmbench/harmbench_behaviors_text_test.csv) from the [HarmBench](https://www.harmbench.org/) dataset and includes 320 queries, categorized into three functional categories: standard, contextual, and copyright. The queries cover eight semantic categories, such as misinformation/disinformation, chemical/biological threats, illegal activities, harmful content, copyright violations, cybercrime, and harassment. It evaluates the model's leakage rate of harmful or unsafe content. 
+
+#### Factors
+
+The following factors can influence MAI-DS-R1's behavior and performance: 
+
+1. **Input topic and Sensitivity**: The model is explicitly tuned to freely discuss topics that were previously blocked. On such topics it will now provide information about where the base model might have demurred. However, for truly harmful or explicitly disallowed content (e.g. instructions for violence), the model remains restrictive due to fine-tuning. 
+
+2. **Language**: Although MAI-DS-R1 was post-trained on multilingual data, it may inherit limitations from the original DeepSeek-R1 model, with performance likely strongest in English and Chinese. 
+
+3. **Prompt Complexity and Reasoning Required**: The model performs well on complex queries requiring reasoning, while very long or complex prompts could still pose a challenge.  
+
+4. **User Instructions and Role Prompts**: As a chat-oriented LLM, MAI-DS-R1’s responses can be shaped by system or developer-provided instructions (e.g. a system prompt defining its role and style) and the user's phrasing. Developers should provide clear instructions to guide model’s behavior. 
+
+#### Metrics
+
+1. Public benchmarks: 
+    - Accuracy: the percentage of problems for which the model’s output matches the correct answer. 
+    - Pass@1: the percentage of problems for which the model generates a correct solution which passes all test cases in the first attempt. 
+
+2. Blocking evaluation: 
+    - Satisfaction (internal metric to measuring relevance with the question on [0,4] scale): The intent is to measure whether the unblocked answers do answer the question and not generate content which is unrelated.  
+    - % Responses: The proportion of previously blocked samples successfully unblocked. 
+
+3. Harm mitigation evaluation: 
+    - Attack Success Rate: the percentage of test cases that elicit the behavior from the model. This is evaluated per functional or semantic category. 
+    - Micro Attack Success Rate: the total average of attack success rate over all categories. 
+
+### Results  
+
+#### Evaluation on General Knowledge and Reasoning
+<p align="center">
+  <img src="figures/reasoning.png" alt="Benchmark Chart">
+</p>
+
+<p align="center">
+  <img src="figures/math.png" alt="Benchmark Chart">
+</p>
+
+<p align="center">
+  <img src="figures/coding.png" alt="Benchmark Chart">
+</p>
+
+#### Evaluation on Responsiveness
+<p align="center">
+    <table>
+    <tr>
+        <td><img src="figures/responsiveness.png" width="500"/></td>
+        <td><img src="figures/satisfaction.png" width="500"/></td>
+    </tr>
+    </table>
+</p>
+
+#### Evaluation on Harm Mitigation
+<p align="center">
+  <img src="figures/harm_mitigation_answer_only.png" alt="Benchmark Chart">
+</p>
+
+<p align="center">
+  <img src="figures/harm_mitigation_thinking_only.png" alt="Benchmark Chart">
+</p>
+
+#### Summary 
+- **General Knowledge & Reasoning**: MAI-DS-R1 performs on par with DeepSeek-R1 and slightly better than R1-1776, especially excelling in mgsm_chain_of_thought_zh, where R1-1776 had a significant regression. 
+
+- **Blocked Topics**: MAI-DS-R1 blocked 99.3% of samples, matching R1-1776, and achieved a higher Satisfaction score, likely due to more relevant responses. 
+
+- **Harm Mitigation**: MAI-DS-R1 outperforms both R1-1776 and the original R1 model in minimizing harmful content. 
+### Model Architecture and Objective 
+- **Model Name**: MAI-DS-R1 (671B) 
+- **Architecture**: Based on DeepSeek-R1, a transformer-based autoregressive language model utilizing multi-head self-attention and Mixture-of-Experts (MoE) for scalable and efficient inference. 
+- **Objective**: Post-trained to reduce CCP-aligned restrictions and enhance harm protection, while preserving the original model’s strong chain-of-thought reasoning and general-purpose language understanding capabilities. 
+- **Pre-trained Model Base**: DeepSeek-R1 (671B) 

SECURITY.md

@@ -0,0 +1,41 @@
+<!-- BEGIN MICROSOFT SECURITY.MD V0.0.9 BLOCK -->
+
+## Security
+
+Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet) and [Xamarin](https://github.com/xamarin).
+
+If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://aka.ms/security.md/definition), please report it to us as described below.
+
+## Reporting Security Issues
+
+**Please do not report security vulnerabilities through public GitHub issues.**
+
+Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://aka.ms/security.md/msrc/create-report).
+
+If you prefer to submit without logging in, send email to [[email protected]](mailto:[email protected]).  If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://aka.ms/security.md/msrc/pgp).
+
+You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://www.microsoft.com/msrc). 
+
+Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue:
+
+  * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.)
+  * Full paths of source file(s) related to the manifestation of the issue
+  * The location of the affected source code (tag/branch/commit or direct URL)
+  * Any special configuration required to reproduce the issue
+  * Step-by-step instructions to reproduce the issue
+  * Proof-of-concept or exploit code (if possible)
+  * Impact of the issue, including how an attacker might exploit the issue
+
+This information will help us triage your report more quickly.
+
+If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://aka.ms/security.md/msrc/bounty) page for more details about our active programs.
+
+## Preferred Languages
+
+We prefer all communications to be in English.
+
+## Policy
+
+Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://aka.ms/security.md/cvd).
+
+<!-- END MICROSOFT SECURITY.MD BLOCK -->

SUPPORT.md

@@ -0,0 +1,25 @@
+# TODO: The maintainer of this repo has not yet edited this file
+
+**REPO OWNER**: Do you want Customer Service & Support (CSS) support for this product/project?
+
+- **No CSS support:** Fill out this template with information about how to file issues and get help.
+- **Yes CSS support:** Fill out an intake form at [aka.ms/onboardsupport](https://aka.ms/onboardsupport). CSS will work with/help you to determine next steps.
+- **Not sure?** Fill out an intake as though the answer were "Yes". CSS will help you decide.
+
+*Then remove this first heading from this SUPPORT.MD file before publishing your repo.*
+
+# Support
+
+## How to file issues and get help  
+
+This project uses GitHub Issues to track bugs and feature requests. Please search the existing 
+issues before filing new issues to avoid duplicates.  For new issues, file your bug or 
+feature request as a new Issue.
+
+For help and questions about using this project, please **REPO MAINTAINER: INSERT INSTRUCTIONS HERE 
+FOR HOW TO ENGAGE REPO OWNERS OR COMMUNITY FOR HELP. COULD BE A STACK OVERFLOW TAG OR OTHER
+CHANNEL. WHERE WILL YOU HELP PEOPLE?**.
+
+## Microsoft Support Policy  
+
+Support for this **PROJECT or PRODUCT** is limited to the resources listed above.

config.json

@@ -0,0 +1,61 @@
+{
+  "architectures": [
+    "DeepseekV3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_deepseek.DeepseekV3Config",
+    "AutoModel": "modeling_deepseek.DeepseekV3Model",
+    "AutoModelForCausalLM": "modeling_deepseek.DeepseekV3ForCausalLM"
+  },
+  "aux_loss_alpha": 0.001,
+  "bos_token_id": 0,
+  "eos_token_id": 1,
+  "ep_size": 1,
+  "first_k_dense_replace": 3,
+  "hidden_act": "silu",
+  "hidden_size": 7168,
+  "initializer_range": 0.02,
+  "intermediate_size": 18432,
+  "kv_lora_rank": 512,
+  "max_position_embeddings": 163840,
+  "model_type": "deepseek_v3",
+  "moe_intermediate_size": 2048,
+  "moe_layer_freq": 1,
+  "n_group": 8,
+  "n_routed_experts": 256,
+  "n_shared_experts": 1,
+  "norm_topk_prob": true,
+  "num_attention_heads": 128,
+  "num_experts_per_tok": 8,
+  "num_hidden_layers": 61,
+  "num_key_value_heads": 128,
+  "num_nextn_predict_layers": 1,
+  "pretraining_tp": 1,
+  "q_lora_rank": 1536,
+  "qk_nope_head_dim": 128,
+  "qk_rope_head_dim": 64,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": {
+    "beta_fast": 32,
+    "beta_slow": 1,
+    "factor": 40,
+    "mscale": 1.0,
+    "mscale_all_dim": 1.0,
+    "original_max_position_embeddings": 4096,
+    "type": "yarn"
+  },
+  "rope_theta": 10000,
+  "routed_scaling_factor": 2.5,
+  "scoring_func": "sigmoid",
+  "seq_aux": true,
+  "tie_word_embeddings": false,
+  "topk_group": 4,
+  "topk_method": "noaux_tc",
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.46.3",
+  "use_cache": true,
+  "v_head_dim": 128,
+  "vocab_size": 129280
+}

configuration_deepseek.py

@@ -0,0 +1,210 @@
+from transformers.configuration_utils import PretrainedConfig
+from transformers.utils import logging
+
+logger = logging.get_logger(__name__)
+
+DEEPSEEK_PRETRAINED_CONFIG_ARCHIVE_MAP = {}
+class DeepseekV3Config(PretrainedConfig):
+    r"""
+    This is the configuration class to store the configuration of a [`DeepseekV3Model`]. It is used to instantiate an DeepSeek
+    model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
+    defaults will yield a similar configuration to that of the DeepSeek-V3.
+
+    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
+    documentation from [`PretrainedConfig`] for more information.
+
+
+    Args:
+        vocab_size (`int`, *optional*, defaults to 129280):
+            Vocabulary size of the Deep model. Defines the number of different tokens that can be represented by the
+            `inputs_ids` passed when calling [`DeepseekV3Model`]
+        hidden_size (`int`, *optional*, defaults to 4096):
+            Dimension of the hidden representations.
+        intermediate_size (`int`, *optional*, defaults to 11008):
+            Dimension of the MLP representations.
+        moe_intermediate_size (`int`, *optional*, defaults to 1407):
+            Dimension of the MoE representations.
+        num_hidden_layers (`int`, *optional*, defaults to 32):
+            Number of hidden layers in the Transformer decoder.
+        num_nextn_predict_layers (`int`, *optional*, defaults to 1):
+            Number of nextn predict layers in the DeepSeekV3 Model.
+        num_attention_heads (`int`, *optional*, defaults to 32):
+            Number of attention heads for each attention layer in the Transformer decoder.
+        n_shared_experts (`int`, *optional*, defaults to None):
+            Number of shared experts, None means dense model.
+        n_routed_experts (`int`, *optional*, defaults to None):
+            Number of routed experts, None means dense model.
+        routed_scaling_factor (`float`, *optional*, defaults to 1.0):
+            Scaling factor or routed experts.
+        topk_method (`str`, *optional*, defaults to `gready`):
+            Topk method used in routed gate.
+        n_group (`int`, *optional*, defaults to None):
+            Number of groups for routed experts.
+        topk_group (`int`, *optional*, defaults to None):
+            Number of selected groups for each token(for each token, ensuring the selected experts is only within `topk_group` groups).
+        num_experts_per_tok (`int`, *optional*, defaults to None):
+            Number of selected experts, None means dense model.
+        moe_layer_freq (`int`, *optional*, defaults to 1):
+            The frequency of the MoE layer: one expert layer for every `moe_layer_freq - 1` dense layers.
+        first_k_dense_replace (`int`, *optional*, defaults to 0):
+            Number of dense layers in shallow layers(embed->dense->dense->...->dense->moe->moe...->lm_head).
+                                                            \--k dense layers--/
+        norm_topk_prob (`bool`, *optional*, defaults to False):
+            Whether to normalize the weights of the routed experts.
+        scoring_func (`str`, *optional*, defaults to 'softmax'):
+            Method of computing expert weights.
+        aux_loss_alpha (`float`, *optional*, defaults to 0.001):
+            Auxiliary loss weight coefficient.
+        seq_aux = (`bool`, *optional*, defaults to True):
+            Whether to compute the auxiliary loss for each individual sample.
+        num_key_value_heads (`int`, *optional*):
+            This is the number of key_value heads that should be used to implement Grouped Query Attention. If
+            `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
+            `num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When
+            converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
+            by meanpooling all the original heads within that group. For more details checkout [this
+            paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to
+            `num_attention_heads`.
+        hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
+            The non-linear activation function (function or string) in the decoder.
+        max_position_embeddings (`int`, *optional*, defaults to 2048):
+            The maximum sequence length that this model might ever be used with.
+        initializer_range (`float`, *optional*, defaults to 0.02):
+            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
+        rms_norm_eps (`float`, *optional*, defaults to 1e-06):
+            The epsilon used by the rms normalization layers.
+        use_cache (`bool`, *optional*, defaults to `True`):
+            Whether or not the model should return the last key/values attentions (not used by all models). Only
+            relevant if `config.is_decoder=True`.
+        pad_token_id (`int`, *optional*):
+            Padding token id.
+        bos_token_id (`int`, *optional*, defaults to 1):
+            Beginning of stream token id.
+        eos_token_id (`int`, *optional*, defaults to 2):
+            End of stream token id.
+        pretraining_tp (`int`, *optional*, defaults to 1):
+            Experimental feature. Tensor parallelism rank used during pretraining. Please refer to [this
+            document](https://huggingface.co/docs/transformers/parallelism) to understand more about it. This value is
+            necessary to ensure exact reproducibility of the pretraining results. Please refer to [this
+            issue](https://github.com/pytorch/pytorch/issues/76232).
+        tie_word_embeddings (`bool`, *optional*, defaults to `False`):
+            Whether to tie weight embeddings
+        rope_theta (`float`, *optional*, defaults to 10000.0):
+            The base period of the RoPE embeddings.
+        rope_scaling (`Dict`, *optional*):
+            Dictionary containing the scaling configuration for the RoPE embeddings. Currently supports two scaling
+            strategies: linear and dynamic. Their scaling factor must be a float greater than 1. The expected format is
+            `{"type": strategy name, "factor": scaling factor}`. When using this flag, don't update
+            `max_position_embeddings` to the expected new maximum.
+        attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`):
+            Whether to use a bias in the query, key, value and output projection layers during self-attention.
+        attention_dropout (`float`, *optional*, defaults to 0.0):
+            The dropout ratio for the attention probabilities.
+
+    ```python
+    >>> from transformers import DeepseekV3Model, DeepseekV3Config
+
+    >>> # Initializing a Deepseek-V3 style configuration
+    >>> configuration = DeepseekV3Config()
+
+    >>> # Accessing the model configuration
+    >>> configuration = model.config
+    ```"""
+
+    model_type = "deepseek_v3"
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    def __init__(
+        self,
+        vocab_size=129280,
+        hidden_size=7168,
+        intermediate_size=18432,
+        moe_intermediate_size = 2048,
+        num_hidden_layers=61,
+        num_nextn_predict_layers=1,
+        num_attention_heads=128,
+        num_key_value_heads=128,
+        n_shared_experts = 1,
+        n_routed_experts = 256,
+        ep_size = 1,
+        routed_scaling_factor = 2.5,
+        kv_lora_rank = 512,
+        q_lora_rank = 1536,
+        qk_rope_head_dim = 64,
+        v_head_dim = 128,
+        qk_nope_head_dim = 128,
+        topk_method = 'noaux_tc',
+        n_group = 8,
+        topk_group = 4,
+        num_experts_per_tok = 8,
+        moe_layer_freq = 1,
+        first_k_dense_replace = 3,
+        norm_topk_prob = True,
+        scoring_func = 'sigmoid',
+        aux_loss_alpha = 0.001,
+        seq_aux = True,
+        hidden_act="silu",
+        max_position_embeddings=4096,
+        initializer_range=0.02,
+        rms_norm_eps=1e-6,
+        use_cache=True,
+        pad_token_id=None,
+        bos_token_id=0,
+        eos_token_id=1,
+        pretraining_tp=1,
+        tie_word_embeddings=False,
+        rope_theta=10000.0,
+        rope_scaling=None,
+        attention_bias=False,
+        attention_dropout=0.0,
+        **kwargs,
+    ):
+        self.vocab_size = vocab_size
+        self.max_position_embeddings = max_position_embeddings
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.moe_intermediate_size = moe_intermediate_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_nextn_predict_layers = num_nextn_predict_layers
+        self.num_attention_heads = num_attention_heads
+        self.n_shared_experts = n_shared_experts
+        self.n_routed_experts = n_routed_experts
+        self.ep_size = ep_size
+        self.routed_scaling_factor = routed_scaling_factor
+        self.kv_lora_rank = kv_lora_rank
+        self.q_lora_rank = q_lora_rank
+        self.qk_rope_head_dim = qk_rope_head_dim
+        self.v_head_dim = v_head_dim
+        self.qk_nope_head_dim = qk_nope_head_dim
+        self.topk_method = topk_method
+        self.n_group = n_group
+        self.topk_group = topk_group
+        self.num_experts_per_tok = num_experts_per_tok
+        self.moe_layer_freq = moe_layer_freq
+        self.first_k_dense_replace = first_k_dense_replace
+        self.norm_topk_prob = norm_topk_prob
+        self.scoring_func = scoring_func
+        self.aux_loss_alpha = aux_loss_alpha
+        self.seq_aux = seq_aux
+        # for backward compatibility
+        if num_key_value_heads is None:
+            num_key_value_heads = num_attention_heads
+
+        self.num_key_value_heads = num_key_value_heads
+        self.hidden_act = hidden_act
+        self.initializer_range = initializer_range
+        self.rms_norm_eps = rms_norm_eps
+        self.pretraining_tp = pretraining_tp
+        self.use_cache = use_cache
+        self.rope_theta = rope_theta
+        self.rope_scaling = rope_scaling
+        self.attention_bias = attention_bias
+        self.attention_dropout = attention_dropout
+
+        super().__init__(
+            pad_token_id=pad_token_id,
+            bos_token_id=bos_token_id,
+            eos_token_id=eos_token_id,
+            tie_word_embeddings=tie_word_embeddings,
+            **kwargs,
+        )

model-00001-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9f8dd7f9b5bfa2b5f9d64cfd7e9435f909eb7c06824bed8a7197a2ca2e23a09f
+size 4187309040

model-00002-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0763dcc633454394226edef0333a69ac6b88ff73bc01d8e55559fa541868cc66
+size 4089742568

model-00003-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ba4ad894a2cbc0c899153f7bc62f4b404b1589e80a97d9622e179eb7722bc354
+size 4266675968

model-00004-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:575284b954da37ad30b60481646ded7d0214a867598731f999584b04db0dfb1a
+size 4286596768

model-00005-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c63f003904197a4f6219a6ddd7205873deaf05ee74b5b73557287143b0de6fae
+size 4286596904

model-00006-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:640095d85b81725948f71f4c996dc2029e30fad70ba0e64f125158d00749673c
+size 4286596912

model-00007-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8d382b789406aaa68495b7ae9d5e0c9db26d9fbe013eccbaa34ca6f03dc07096
+size 4286625752

model-00008-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e570c999e69c293f1471716c33f7aa2fa6f0d4477a3972b6c76d0fa3e29e5d4b
+size 4290267512

model-00009-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a58682e8e63231b3342877310b44bb868e45cfd617ef67b8079bb6764594c449
+size 4279129128

model-00010-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d124a1d208ea1a62ef94bbb93d08567a167c5a47684ecbd9bea0bf2e3d009c28
+size 4286596824

model-00011-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a3859d3dad19603de16f1033cba2de2932f99c45c578a18c9cb678e3a4ac8b8a
+size 4286611336

model-00012-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3107258dc7864639bb7b36f196a90363607b069799957e21bd33f0389221d81c
+size 4286596912

model-00013-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:059fcf8dea28911a9953dfd79b2ec5014912c67322faeca0d53c7e241cdb4ede
+size 4279143440

model-00014-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e20c527f7e3289312731ee0d782d5dbd8bd1de410acf13bea4f3d15d8c3d8b3b
+size 4286596752

model-00015-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:edca782c5a0385f9ebe98e0da80406be47b558aeed80dcdc0f613780216cb92b
+size 4290267664

model-00016-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ce28c0cdf247dc4ba760b7119164a4533774d54c78efbc64329af1c18dbbafd
+size 4286596920

model-00017-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8bfef6aae9e23e723603b136f3cfd3790c132461ed1805193c10c5232af0a7b4
+size 4286611344

model-00018-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a24c63eef537168a94fc1c5fa1fea2d9969c19cad95e534829f2990156b0263b
+size 4286611264

model-00019-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8d93cbcccb02d80ae9cefdcb103eabeaf3bcea5b475a232970d069e3c44c8353
+size 4282799832

model-00020-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff5fbea4c127fe71b8bd7aa203978d91690890190ba4a497012f981e1fefa5f1
+size 4286596768

model-00021-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1b9601a9d9ad211aa0ad5d6c90faad2b6b9982b8d541fd8d21ec9d8551d3a865
+size 4286596888

model-00022-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d1055ad3e73dd8db03fb37866d18876ed29ed3a47377c355ba76d4afa99f123b
+size 4286596912

model-00023-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4950183ed5735266903501c0a433f23a066176251b273b32131fca1488720aca
+size 4286625776

model-00024-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:962da06e44ee1211654002040993d667c28aa72fffab03bcd1c93a27fafbf3a2
+size 4286596752

model-00025-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0c14876f4fc20cbb3932018664b8bf0626ed1243d2f29248f1c1f7b51e839db5
+size 4282799776

model-00026-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:071fd4ef3699d21aa990092caa6b38079a9ac40bb97d63eb706836ed4101ad24
+size 4286596912

model-00027-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:28b2e05b59d06f2c45e83ce82dc8897a3964ee39ecbd534062a7f23b5e52e653
+size 4286611328

model-00028-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6efd812669e779f1a80b88649ca94c003e31592ac40911d7e28fc519ebb4b3cf
+size 4286596920

model-00029-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:da9a1b0544a00767c0a94c0fb2c8229de48cd5d0463e31bbd13db2cfb3b0de70
+size 4282814232

model-00030-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2744a1d5a1649b2ada1138776536a119ee6a4655e0619fdfee78dad310a46a7a
+size 4286596816

model-00031-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a779ff246db381cd6f7b39ce1f698f6070b4ab103d1e0f0ca2faa63a336b36c5
+size 4286596792

model-00032-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3b2369b9bb87dc1f8cc55dc87ef006781fd63ee05908a2522b9129a838e32622
+size 4286596912

model-00033-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fcf6810e8a5e5a76a0c8f64655b4e1db14648801279ed144105a000842ebfd23
+size 4286611352

model-00034-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9a01e8d5587d74e891589cccc2e2f03b6c7ba42df8a1241afe6c6b4080d6653e
+size 4286611296

model-00035-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:30809f994b8b66d2d4f3d07f224a2c4ee91708c41124aa59f0ced4ac341cb21d
+size 4282799760

model-00036-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:666eae53129b6d8917caf44b98217adef79507e1b1a302ffeb4abed681633d66
+size 4286596912

model-00037-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:29530eff774b705721943e290c20f039d7116938d577a3ba2effc5dcecd6d076
+size 4286596792

model-00038-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:639fc1a70cdeef9af7d13aed3d76d38876a282e9d0336489809ebd3a153f4d6b
+size 4286596920

model-00039-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7bf9905f2de1974f61cbb8ab1afcbf538238bd6c1ffa304939faa1cd1ede0eb4
+size 4286596912

model-00040-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fbc70ecd91fc798ae103a85c6f69ee1c58096daf80282879f0a269b4d5364631
+size 4286596904

model-00041-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c4f6fd75b2ca0dc022f605fe74350f134e5e12ddd2c27eaee69f044e16f88aed
+size 4282828808

model-00042-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3725b49f140b9dfd32767d78d3eb04505111f7e2a0b2013cb73b52b58bd5fdd1
+size 4286597064

model-00043-of-00314.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2fd2d497b6b7be16bcd1254c60ebf1764ac697aa6eee95e8f5e7232ad535efce
+size 4286611472