README.md

---
language:
  - en
license: mit
tags:
  - code-generation
  - transformer
  - ast
  - cfg
  - langchain
  - ollama
model_name: MiniCoderX
datasets:
  - the-stack
  - codesearchnet
  - humaneval
  - mbpp
  - bugs2fix
  - java-python
pipeline_tag: text-generation
---

# 🚀 MiniCoderX: A Lightweight Transformer for Code Generation

**MiniCoderX** is a structure-aware, transformer-based small language model (SLM) for code generation. It blends modern architectural techniques with efficient deployment using tools like **LangChain** and **Ollama**, making it ideal for rapid local experimentation.

---

## ✨ Features

- 🧠 Transformer-based encoder-decoder (TinyCodeT5 / DistilGPT2)
- 🌲 AST/CFG-aware encoding for code structure understanding
- 💾 Syntax-constrained decoding using grammar rules and trees
- 🔁 Multi-task heads: generation, summarization, translation, bug fixing
- ⚙️ LangChain + Ollama integration for fast local deployment
- 🧪 Evaluated on HumanEval, CodeXGLUE, MBPP

---

## 🏗️ Model Architecture

| Component       | Description                                               |
|----------------|-----------------------------------------------------------|
| Base           | Tiny encoder-decoder (MiniLM, DistilGPT2, TinyCodeT5)     |
| Structure-aware | AST and Control Flow Graph embeddings + positional masks |
| Heads          | Multi-task heads for flexible downstream use              |
| Decoder        | Syntax-aware beam search (grammar constraints)            |
| Tokenizer      | BPE or SentencePiece trained on code + comments           |

---

## 🔧 Architectural Additions (SOTA Techniques)

### 🌲 AST/CFG Embeddings
Enhances understanding of code structure by:
- Adding AST node/edge embeddings to token inputs
- Including path embeddings between syntactic elements
- Graph-aware position encoding

Inspired by: **StructCoder**, **AST-T5**, **Code4Struct**

### 💾 Syntax-Constrained Decoding
Improves generation accuracy and reduces invalid code by:
- Restricting token outputs using grammar constraints (BNF/PEG)
- Custom decoding logic (e.g., Tree traversal)
- Dynamic decoding masks based on token state

Inspired by: **TreeGen**, **Code4Struct**

### 🔁 Multi-Task Learning Heads
Supports multiple tasks:
- Code generation (NL → Code)
- Summarization (Code → NL)
- Translation (Java ⇄ Python)
- Code repair and completion

Inspired by: **CodeT5+**, **CoTexT**

---

## ⚡ LangChain + Ollama Integration

### 💡 Why?
To enable:
- 🧪 Local testing and chaining of models via **LangChain**
- 🦮 Fast prototyping with **Ollama** for custom transformer backends
- 🔄 Easy switch between small local models and larger remote APIs

### 🔌 Integration Plan
```python
from langchain.llms import Ollama
from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate

# Load MiniCoderX with Ollama
llm = Ollama(model="minicoderx")  # Local model via Ollama

# Define code generation prompt
prompt = PromptTemplate(
    input_variables=["instruction"],
    template="Generate Python code for the task: {instruction}",
)

chain = LLMChain(llm=llm, prompt=prompt)
result = chain.run("Sort a list of integers using quicksort")

print(result)
```

> ✅ Ollama will be used to serve your fine-tuned SLM locally  
> ✅ LangChain will wrap it with prompts, chains, and memory features for interactivity

---

## 📦 Datasets

| Dataset        | Use                        |
|----------------|----------------------------|
| The Stack (subset) | Pretraining corpus     |
| CodeSearchNet  | Summarization, Search      |
| HumanEval      | Code generation benchmark  |
| MBPP           | Python programming prompts |
| Bugs2Fix       | Code repair                |
| Java-Python    | Cross-language translation |

---

## 🔬 Training Objectives

- ✅ Span Masking (CodeT5-style)
- ✅ Contrastive pretraining
- ✅ Instruction tuning (natural prompt formatting)
- ✅ Auto-regressive generation

---

## 📊 Evaluation Benchmarks

| Benchmark  | Metric            |
|------------|-------------------|
| HumanEval  | Pass@1, BLEU      |
| MBPP       | Accuracy          |
| CodeXGLUE  | CodeBLEU, EM      |
| Unit Tests | Pass Rate         |

---

## 🧪 Project Roadmap

### ✅ Phase 1: MVP Model
- Train TinyCodeT5 model with span masking
- Evaluate on MBPP and HumanEval-lite
- Serve via Ollama + LangChain prompt chain

### 🔁 Phase 2: Structural Learning
- Add AST/CFG encodings
- Introduce grammar-constrained decoding
- Multi-task training (gen, sum, repair)

### 📦 Phase 3: Optimization & Packaging
- Distill from larger model (e.g., StarCoder)
- Add reinforcement fine-tuning via test cases
- Export to Hugging Face + Ollama integration

---

## 🛠️ Tools & Frameworks

- [Hugging Face Transformers](https://github.com/huggingface/transformers)
- [LangChain](https://github.com/langchain-ai/langchain)
- [Ollama](https://ollama.com/)
- SentencePiece / BPE
- NetworkX for AST/CFG parsing

---

## 🤝 Contributing

Want to help with grammar decoders, AST integration, or evaluation? PRs welcome!

---

## 📜 License

MIT License. Built for research and open experimentation.

---

## 📧 Contact

Drop an issue or discussion on GitHub!