Intel
/

gpt-j-6b-sparse

Text Generation

Model card Files Files and versions

weiweiz1 commited on Dec 6, 2023

Commit

c8254c9

·

1 Parent(s): 9c90a40

Update README.md

Files changed (1) hide show

README.md +2 -2

README.md CHANGED Viewed

@@ -24,8 +24,8 @@ The sparse version of GPT-J 6B is a pruned variant derived from the original [GP
 | \\(n_{heads}\\)      | 16         |
 | \\(d_{head}\\)       | 256        |
 | \\(n_{ctx}\\)        | 2048       |
-| \\(n_{vocab}\\)      | 50257/50400&dagger; (same tokenizer as GPT-2/3)  |
-| Positional Encoding  | [Rotary Position Embedding (RoPE)](https://arxiv.org/abs/2104.09864) |
 | RoPE Dimensions      | [64](https://github.com/kingoflolz/mesh-transformer-jax/blob/f2aa66e0925de6593dcbb70e72399b97b4130482/mesh_transformer/layers.py#L223) |
 <figcaption><p><strong>&ast;</strong> Each layer consists of one feedforward block and one self attention block.</p>
 <p><strong>&dagger;</strong> Although the embedding matrix has a size of 50400, only 50257 entries are used by the GPT-2 tokenizer.</p></figcaption></figure>

 | \\(n_{heads}\\)      | 16         |
 | \\(d_{head}\\)       | 256        |
 | \\(n_{ctx}\\)        | 2048       |
+| \\(n_{vocab}\\)      | 50257/50400&dagger; (same tokenizer as GPT-2/3) |
+| Positional Encoding  | Rotary Position Embedding RoPE |
 | RoPE Dimensions      | [64](https://github.com/kingoflolz/mesh-transformer-jax/blob/f2aa66e0925de6593dcbb70e72399b97b4130482/mesh_transformer/layers.py#L223) |
 <figcaption><p><strong>&ast;</strong> Each layer consists of one feedforward block and one self attention block.</p>
 <p><strong>&dagger;</strong> Although the embedding matrix has a size of 50400, only 50257 entries are used by the GPT-2 tokenizer.</p></figcaption></figure>