Spaces:
Sleeping
Sleeping
File size: 3,467 Bytes
5fdb69e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 |
# AI Web Page Summarizer
This project is a simple AI-powered web page summarizer that leverages OpenAI's GPT models and local inference with Ollama to generate concise summaries of given text. The goal is to create a "Reader's Digest of the Internet" by summarizing web content efficiently.
## Features
- Summarize text using OpenAI's GPT models or local Ollama models.
- Flexible summarization engine selection (OpenAI API, Ollama API, or Ollama library).
- Simple and modular code structure.
- Error handling for better reliability.
## Project Structure
```
ai-summarizer/
│-- summarizer/
│ │-- __init__.py
│ │-- fetcher.py # Web content fetching logic
│ │-- summarizer.py # Main summarization logic
│-- utils/
│ │-- __init__.py
│ │-- logger.py # Logging configuration
│-- main.py # Entry point of the app
│-- .env # Environment variables
│-- requirements.txt # Python dependencies
│-- README.md # Project documentation
```
## Prerequisites
- Python 3.8 or higher
- OpenAI API Key (You can obtain it from [OpenAI](https://platform.openai.com/signup))
- Ollama installed locally ([Installation Guide](https://ollama.ai))
- `conda` for managing environments (optional)
## Installation
1. **Clone the repository:**
```bash
git clone https://github.com/your-username/ai-summarizer.git
cd ai-summarizer
```
2. **Create a virtual environment (optional but recommended):**
```bash
conda create --name summarizer-env python=3.9
conda activate summarizer-env
```
3. **Install dependencies:**
```bash
pip install -r requirements.txt
```
4. **Set up environment variables:**
Create a `.env` file in the project root and add your OpenAI API key (if using OpenAI):
```env
OPENAI_API_KEY=your-api-key-here
```
## Usage
1. **Run the summarizer:**
```bash
python main.py
```
2. **Sample Output:**
```shell
Enter a URL to summarize: https://example.com
Summary of the page:
AI refers to machines demonstrating intelligence similar to humans and animals.
```
3. **Engine Selection:**
The summarizer supports multiple engines. Modify `main.py` to select your preferred model:
```python
summary = summarize_text(content, 'gpt-4o-mini', engine="openai")
summary = summarize_text(content, 'deepseek-r1:1.5B', engine="ollama-api")
summary = summarize_text(content, 'deepseek-r1:1.5B', engine="ollama-lib")
```
## Configuration
You can modify the model, max tokens, and temperature in `summarizer/summarizer.py`:
```python
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[...],
max_tokens=300,
temperature=0.7
)
```
## Error Handling
If any issues occur, the script will print an error message, for example:
```
Error during summarization: Invalid API key or Ollama not running.
```
## Dependencies
The required dependencies are listed in `requirements.txt`:
```
openai
python-dotenv
requests
ollama-api
```
Install them using:
```bash
pip install -r requirements.txt
```
## Contributing
Contributions are welcome! Feel free to fork the repository and submit pull requests.
## License
This project is licensed under the MIT License. See the `LICENSE` file for more details.
## Contact
For any inquiries, please reach out to:
- Linkedin: https://www.linkedin.com/in/khanarafat/
- GitHub: https://github.com/raoarafat
|