metadata

license: mit
pipeline_tag: text-generation
library_name: transformers

Random-Llama-Small

Model Overview

Random-Llama-Small is a randomly initialized transformer-based language model with approximately 2 billion parameters, built using the LLaMA architecture. It is designed for research purposes, providing a starting point for pretraining or fine-tuning on custom datasets. The model uses the tokenizer from HuggingFaceTB/SmolLM2-1.7B-Instruct and is configured for causal language modeling. As a randomly initialized model, it produces incoherent outputs until trained, making it ideal for researchers studying transformer training dynamics or developing custom language models.

Key Details

Architecture: LLaMA (Causal Language Model)
Parameters: ~2B
Hidden Size: 2304
Layers: 22
Attention Heads: 36 (with 9 key-value heads for grouped-query attention)
Intermediate Size: 9216
Vocabulary Size: 128256
Tokenizer: Imported from HuggingFaceTB/SmolLM2-1.7B-Instruct
Precision: bfloat16
Max Context Length: 131,072 tokens (with RoPE scaling)
License: MIT

LLaMA Architecture

The LLaMA architecture, developed by Meta AI, is a family of efficient transformer-based models optimized for research. Random-Llama-Small follows this design, incorporating several key features:

Core Components

Decoder-Only Transformer: Predicts the next token in a sequence based on prior tokens, suitable for autoregressive tasks like text generation.
Grouped-Query Attention (GQA): 36 attention heads with only 9 key-value heads, improving efficiency and reducing memory/compute cost.
Rotary Position Embeddings (RoPE): Embeds positional information with scaling, enabling a context length of up to 131,072 tokens.
Swiglu Activation: Uses SiLU (Swish) activation in the FFN for improved expressiveness.
RMSNorm: Root Mean Square Layer Normalization replaces LayerNorm for stability and faster convergence.
Tied Embeddings: Input and output embeddings share weights (tie_word_embeddings=True), reducing parameter count by ~295M.

Benefits of LLaMA Architecture

Efficiency: High throughput, low memory use.
Scalability: Works well across model sizes.
Flexibility: Long-context support and task adaptability.
Research-Friendly: Great for exploring attention, positional encoding, and training dynamics.

Random-Llama-Small Specifics

This model uses random weights and:

Has ~2B parameters across 22 layers.
Uses a 2304 hidden size and 9216 FFN size.
Supports 128K+ vocab tokens and bfloat16 precision.
Supports extended context lengths of 131,072 tokens.

Intended Use

Research on transformer dynamics, optimization, or architectural changes.
Baseline for pretraining or task-specific fine-tuning.
Experimentation with scaling laws or custom architectures.

Out-of-Scope Use

Not for direct production deployment.
Not suitable for tasks needing coherence or accuracy without training.

Usage

Requirements

transformers >= 4.45.0
torch >= 2.0
GPU with ≥ 6GB VRAM (24GB+ for training)

Inference Example

# Use a pipeline as a high-level helper
from transformers import pipeline

messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe = pipeline("text-generation", model="reflex-ai/random-llama-small")
print(pipe(messages))

Note: Outputs will be random and incoherent due to the model’s untrained state.

Training Example

from transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling, LlamaForCausalLM, AutoTokenizer

model = LlamaForCausalLM.from_pretrained("your_username/random-llama-small")
tokenizer = AutoTokenizer.from_pretrained("your_username/random-llama-small")

training_args = TrainingArguments(
    output_dir="./random_llama_small_finetuned",
    per_device_train_batch_size=4,
    num_train_epochs=3,
    fp16=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=your_dataset,
    data_collator=DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False),
)

trainer.train()

Limitations

Random Initialization: Needs significant training to be useful.
Resource Intensive: High computational cost.
No Pretraining Data: Users must provide their own.
Tokenizer Constraint: May not suit all domains.

Benefits and Potential

Customizability: A blank slate for full control of objectives and data.
Research Insights: Ideal for understanding early-stage LLM behavior.
Scalable Baseline: Balances size and research feasibility.
Extended Context: Useful for long-form tasks post-training.

Model Configuration

{
  "architectures": ["LlamaForCausalLM"],
  "hidden_size": 2304,
  "num_hidden_layers": 22,
  "num_attention_heads": 36,
  "num_key_value_heads": 9,
  "intermediate_size": 9216,
  "vocab_size": 128256,
  "max_position_embeddings": 131072,
  "rope_scaling": {
    "factor": 32.0,
    "high_freq_factor": 4.0,
    "low_freq_factor": 1.0,
    "original_max_position_embeddings": 8192,
    "rope_type": "llama3"
  },
  "torch_dtype": "bfloat16",
  "tie_word_embeddings": true
}

Ethical Considerations

Untrained Safety: No immediate harmful outputs, but ethics matter during training.
Environmental Impact: Large-scale training consumes energy; optimize and use green compute.
Accessibility: Resource requirements may limit use by smaller research teams.

Contact

For questions or issues, please open an issue on the Hugging Face repository.

Model card created on April 20, 2025.