Update README.md

README.md

@@ -1,13 +1,68 @@
----
-title: BAS Website Chat
-emoji: 🏆
+title: BAS Website AI
+emoji: 🌍
 colorFrom: indigo
 colorTo: green
 sdk: gradio
 sdk_version: 5.14.0
 app_file: app.py
 pinned: false
-short_description: LLM RAG Web scraper. on Hugging Face Spaces 
----
+short_description: LLM RAG Web scraper for the British Antarctic Survey on Hugging Face Spaces
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+## Overview
+This project implements a RAG (Retrieval-Augmented Generation) system that allows users to chat with website content using LLMs. It consists of two main components:
+
+1. **Web Scraper**: A robust scraping system that:
+   - Crawls websites systematically with error handling and retry logic
+   - Processes and chunks content for optimal retrieval
+   - Stores data in ChromaDB with embeddings
+   - Supports checkpoint-based resumption of scraping
+
+2. **Chat Interface**: A Gradio-based chat application that:
+   - Uses Groq's LLM API for responses
+   - Implements semantic search with ChromaDB
+   - Features cross-encoder reranking for improved result relevance
+   - Provides source citations for responses
+
+## Key Features
+- 🔄 Resumable web scraping with progress tracking
+- 💾 Persistent vector storage using ChromaDB
+- 🔍 Advanced retrieval with semantic search and reranking
+- 🤖 Integration with Groq's LLM API
+- 📱 User-friendly chat interface
+- 🔗 Source attribution for responses
+
+## Setup
+1. Install dependencies:
+   ```bash
+   pip install -r requirements.txt
+   ```
+
+2. Set up environment variables:
+   ```
+   GROQ_API_KEY=your_groq_api_key
+   HF_TOKEN=your_huggingface_token
+   ```
+
+3. Run the scraper:
+   ```bash
+   python scraper_app.py
+   ```
+   - Use `--rescrape` flag for fresh start
+
+4. Launch the chat interface:
+   ```bash
+   python app.py
+   ```
+
+## Tools & Technologies
+- ChromaDB for vector storage
+- Sentence Transformers for embeddings
+- Groq for LLM inference
+- Gradio for web interface
+- BeautifulSoup4 for web scraping
+
+## Project Structure
+- `app.py`: Main chat application
+- `scraper.py`: Web scraping logic
+- `scraper_app.py`: Scraper management
+- `chroma_explorer.ipynb`: Database exploration notebook