| --- |
| model-index: |
| - name: CodeLlama-7b |
| results: |
| - task: |
| type: code-generation |
| dataset: |
| name: Humaneval |
| type: humaneval |
| metrics: |
| - type: pass@1 (BASELINE) |
| value: 0.3048780487804878 |
| - type: pass@1 (BASIC) |
| value: 0.3170731707317073 |
| --- |
| This is a d-Matrix functional reference of the CODELLAMA-7B model. |
| The reference provides the following functional *configurations*: |
| Configuration | Explanation |
| :-- | :-- |
| **`BASELINE`** | a reference functionally equivalent to the original model |
| **`BASIC`** | all linear algebraic operands quantized to `MXINT8-64` |
|
|
|
|
| ### Usage |
|
|
| Install d-Matrix [Dmx_Compressor](https://github.com/d-matrix-ai/dmx-compressor) first. |
| ```sh |
| pip install dmx_compressor |
| ``` |
|
|
| The following is an example model and its evaluation. |
|
|
| ```sh |
| git clone https://github.com/bigcode-project/bigcode-evaluation-harness.git |
| cd bigcode-evaluation-harness |
| pip install -e . |
| ``` |
|
|
| ```python |
| # code_eval.py included in HuggingFace repo |
| from code_eval import evaluate_model |
| |
| eval_results = evaluate_model("d-matrix/CodeLlama-7b", dmx_config="BASELINE") # For BASELINE configuration |
| ``` |
