notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7bfc3afd-0868-4938-9b45-19b2cba1a149",
   "metadata": {},
   "source": [
    "## Setting Up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d102db92-a346-447d-8c61-3be8292adec7",
   "metadata": {
    "executionCancelledAt": null,
    "executionTime": 22644,
    "lastExecutedAt": 1744298182897,
    "lastExecutedByKernel": "b0791251-777d-414c-b5cc-636b4f317d9a",
    "lastScheduledRunId": null,
    "lastSuccessfullyExecutedCode": "%%capture\n%pip install langchain\n%pip install langchain-community \n%pip install langchainhub \n%pip install langchain-chroma \n%pip install langchain-groq\n%pip install langchain-huggingface\n%pip install unstructured[docx]"
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "%pip install langchain\n",
    "%pip install langchain-community \n",
    "%pip install langchainhub \n",
    "%pip install langchain-chroma \n",
    "%pip install langchain-groq\n",
    "%pip install langchain-huggingface\n",
    "%pip install unstructured[docx]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "356e4c03-5642-4d21-8ee4-bc32b14e98ec",
   "metadata": {},
   "source": [
    "## Groq Python API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4e8b89b9-f5bc-466b-a30f-db8e97828826",
   "metadata": {
    "executionCancelledAt": null,
    "executionTime": 3176,
    "lastExecutedAt": 1744298186074,
    "lastExecutedByKernel": "b0791251-777d-414c-b5cc-636b4f317d9a",
    "lastScheduledRunId": null,
    "lastSuccessfullyExecutedCode": "import os\nfrom groq import Groq\n\ngroq_api_key = os.environ.get(\"GROQ_API_KEY\")\n\nclient = Groq(\n   api_key=groq_api_key,\n)\n\n\nchat_streaming = client.chat.completions.create(\n    messages=[\n       {\"role\": \"system\", \"content\": \"You are a professional Data Engineer.\"},\n       {\"role\": \"user\", \"content\": \"Can you explain how the data lake works?\"},\n    ],\n    model=\"meta-llama/llama-4-scout-17b-16e-instruct\",\n    temperature=0.3,\n    max_tokens=1200,\n    top_p=1,\n    stop=None,\n    stream=True,\n)\n\nfor chunk in chat_streaming:\n    print(chunk.choices[0].delta.content, end=\"\")",
    "outputsMetadata": {
     "0": {
      "height": 469,
      "type": "stream"
     }
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "As a Data Engineer, I'd be happy to explain how a data lake works.\n",
      "\n",
      "**What is a Data Lake?**\n",
      "\n",
      "A data lake is a centralized repository that stores raw, unprocessed data in its native format. It's a scalable and flexible storage solution that allows you to store and process large amounts of structured, semi-structured, and unstructured data. The data lake is often used as a precursor to data warehousing, data analytics, and machine learning.\n",
      "\n",
      "**Key Components of a Data Lake**\n",
      "\n",
      "1. **Storage**: The storage layer is the foundation of a data lake. It's typically a distributed file system, such as Hadoop Distributed File System (HDFS), Amazon S3, Azure Data Lake Storage (ADLS), or Google Cloud Storage (GCS). This layer stores raw data in its native format, without any transformation or processing.\n",
      "2. **Data Ingestion**: Data ingestion is the process of collecting data from various sources and loading it into the data lake. This can be done through various methods, such as batch processing, streaming, or manual uploads.\n",
      "3. **Data Processing**: The data processing layer is responsible for transforming and processing the raw data into a usable format. This can be done using various processing frameworks, such as Apache Spark, Apache Flink, or Azure Databricks.\n",
      "4. **Metadata Management**: Metadata management is critical in a data lake, as it provides context and meaning to the stored data. This includes information such as data schema, data lineage, and data quality.\n",
      "\n",
      "**How a Data Lake Works**\n",
      "\n",
      "Here's a step-by-step overview of how a data lake works:\n",
      "\n",
      "1. **Data Ingestion**: Data is collected from various sources, such as databases, applications, IoT devices, or social media platforms.\n",
      "2. **Data Landing**: The ingested data is landed in the data lake's storage layer, where it's stored in its native format.\n",
      "3. **Data Processing**: The raw data is processed and transformed into a usable format using various processing frameworks.\n",
      "4. **Data Cataloging**: The processed data is cataloged, which involves creating metadata that describes the data, such as its schema, format, and quality.\n",
      "5. **Data Analysis**: The processed and cataloged data is made available for analysis, reporting, and machine learning.\n",
      "\n",
      "**Benefits of a Data Lake**\n",
      "\n",
      "The data lake offers several benefits, including:\n",
      "\n",
      "1. **Scalability**: Data lakes can store large amounts of data and scale horizontally as needed.\n",
      "2. **Flexibility**: Data lakes can store various types of data, including structured, semi-structured, and unstructured data.\n",
      "3. **Cost-Effective**: Data lakes can be more cost-effective than traditional data warehousing solutions.\n",
      "4. **Improved Data Quality**: Data lakes provide a single source of truth for data, which improves data quality and reduces data duplication.\n",
      "\n",
      "**Common Use Cases for a Data Lake**\n",
      "\n",
      "1. **Data Warehousing**: Data lakes can be used as a precursor to data warehousing, providing a centralized repository for data before it's loaded into a data warehouse.\n",
      "2. **Big Data Analytics**: Data lakes can be used for big data analytics, providing a scalable and flexible storage solution for large datasets.\n",
      "3. **Machine Learning**: Data lakes can be used as a data source for machine learning models, providing a large and diverse dataset for training and testing.\n",
      "4. **Data Archiving**: Data lakes can be used for data archiving, providing a cost-effective solution for storing historical data.\n",
      "\n",
      "I hope this helps! Do you have any specific questions about data lakes or would you like me to elaborate on any of these points?None"
     ]
    }
   ],
   "source": [
    "import os\n",
    "from groq import Groq\n",
    "\n",
    "groq_api_key = os.environ.get(\"GROQ_API_KEY\")\n",
    "\n",
    "client = Groq(\n",
    "   api_key=groq_api_key,\n",
    ")\n",
    "\n",
    "\n",
    "chat_streaming = client.chat.completions.create(\n",
    "    messages=[\n",
    "       {\"role\": \"system\", \"content\": \"You are a professional Data Engineer.\"},\n",
    "       {\"role\": \"user\", \"content\": \"Can you explain how the data lake works?\"},\n",
    "    ],\n",
    "    model=\"meta-llama/llama-4-scout-17b-16e-instruct\",\n",
    "    temperature=0.3,\n",
    "    max_tokens=1200,\n",
    "    top_p=1,\n",
    "    stop=None,\n",
    "    stream=True,\n",
    ")\n",
    "\n",
    "for chunk in chat_streaming:\n",
    "    print(chunk.choices[0].delta.content, end=\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1849fe7-4641-44c4-a91f-27976d2c1918",
   "metadata": {},
   "source": [
    "## Initiating LLM and Embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "65583358-ce96-4657-9b4b-fabc5a2f195e",
   "metadata": {
    "executionCancelledAt": null,
    "executionTime": 560,
    "lastExecutedAt": 1744298186634,
    "lastExecutedByKernel": "b0791251-777d-414c-b5cc-636b4f317d9a",
    "lastScheduledRunId": null,
    "lastSuccessfullyExecutedCode": "from langchain_groq import ChatGroq\n\nllm = ChatGroq(model=\"meta-llama/llama-4-scout-17b-16e-instruct\", api_key=groq_api_key)"
   },
   "outputs": [],
   "source": [
    "from langchain_groq import ChatGroq\n",
    "\n",
    "llm = ChatGroq(model=\"meta-llama/llama-4-scout-17b-16e-instruct\", api_key=groq_api_key)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "70675683-6a4f-4331-b8b5-6c4e348fa389",
   "metadata": {
    "executionCancelledAt": null,
    "executionTime": 661,
    "lastExecutedAt": 1744298599903,
    "lastExecutedByKernel": "b0791251-777d-414c-b5cc-636b4f317d9a",
    "lastScheduledRunId": null,
    "lastSuccessfullyExecutedCode": "from langchain_huggingface import HuggingFaceEmbeddings\nembed_model = HuggingFaceEmbeddings(model_name=\"mixedbread-ai/mxbai-embed-large-v1\")",
    "outputsMetadata": {
     "0": {
      "height": 437,
      "type": "stream"
     }
    }
   },
   "outputs": [],
   "source": [
    "from langchain_huggingface import HuggingFaceEmbeddings\n",
    "embed_model = HuggingFaceEmbeddings(model_name=\"mixedbread-ai/mxbai-embed-large-v1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff2b277e-dc31-4801-bd05-ffda3265523b",
   "metadata": {},
   "source": [
    "## Loading and spliting the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "12390e24-2c8f-4690-8060-69eea3c224a0",
   "metadata": {
    "executionCancelledAt": null,
    "executionTime": 1932,
    "lastExecutedAt": 1744298196669,
    "lastExecutedByKernel": "b0791251-777d-414c-b5cc-636b4f317d9a",
    "lastScheduledRunId": null,
    "lastSuccessfullyExecutedCode": "from langchain_community.document_loaders import DirectoryLoader\nfrom langchain.text_splitter import RecursiveCharacterTextSplitter\n\n# Initialize the text splitter\ntext_splitter = RecursiveCharacterTextSplitter(\n    chunk_size=500,\n    chunk_overlap=50,\n    separators=[\"\\n\\n\", \"\\n\"]\n)\n\n# Load the .docx files\nloader = DirectoryLoader(\"./\", glob=\"*.docx\", use_multithreading=True)\ndocuments = loader.load()\n\n# Split the documents into chunks\nchunks = text_splitter.split_documents(documents)\n\n# Print the number of chunks\nprint(len(chunks))\n"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "29\n"
     ]
    }
   ],
   "source": [
    "from langchain_community.document_loaders import DirectoryLoader\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "\n",
    "# Initialize the text splitter\n",
    "text_splitter = RecursiveCharacterTextSplitter(\n",
    "    chunk_size=1000,\n",
    "    chunk_overlap=100,\n",
    "    separators=[\"\\n\\n\", \"\\n\"]\n",
    ")\n",
    "\n",
    "# Load the .docx files\n",
    "loader = DirectoryLoader(\"./\", glob=\"*.docx\", use_multithreading=True)\n",
    "documents = loader.load()\n",
    "\n",
    "# Split the documents into chunks\n",
    "chunks = text_splitter.split_documents(documents)\n",
    "\n",
    "# Print the number of chunks\n",
    "print(len(chunks))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26085a16-42c8-4c6c-958f-9c1e2cc62b23",
   "metadata": {},
   "source": [
    "## Creating the Vector Store"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b27426d6-a218-4a44-9067-a9d4509e59c4",
   "metadata": {
    "executionCancelledAt": null,
    "executionTime": 8773,
    "lastExecutedAt": 1744298205442,
    "lastExecutedByKernel": "b0791251-777d-414c-b5cc-636b4f317d9a",
    "lastScheduledRunId": null,
    "lastSuccessfullyExecutedCode": "from langchain_chroma import Chroma\n\nvectorstore = Chroma.from_documents(\n    documents=chunks,\n    embedding=embed_model,\n    persist_directory=\"./Vectordb\",\n)",
    "outputsMetadata": {
     "0": {
      "height": 101,
      "type": "stream"
     }
    }
   },
   "outputs": [],
   "source": [
    "from langchain_chroma import Chroma\n",
    "\n",
    "vectorstore = Chroma.from_documents(\n",
    "    documents=chunks,\n",
    "    embedding=embed_model,\n",
    "    persist_directory=\"./Vectordb\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "633bbbfa-36ac-426e-8599-0f3bfb3b80ea",
   "metadata": {
    "executionCancelledAt": null,
    "executionTime": 166,
    "lastExecutedAt": 1744298366376,
    "lastExecutedByKernel": "b0791251-777d-414c-b5cc-636b4f317d9a",
    "lastScheduledRunId": null,
    "lastSuccessfullyExecutedCode": "query = \"What this tutorial about?\"\ndocs = vectorstore.similarity_search(query)\nprint(docs[0].page_content)",
    "outputsMetadata": {
     "0": {
      "height": 122,
      "type": "stream"
     }
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Learn how to Fine-tune Stable Diffusion XL with DreamBooth and LoRA on your personal images. \n",
      "\n",
      "Let’s try another prompt:\n",
      "\n",
      "Prompt:\n"
     ]
    }
   ],
   "source": [
    "query = \"What this tutorial about?\"\n",
    "docs = vectorstore.similarity_search(query)\n",
    "print(docs[0].page_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "304a6177-9f74-40a5-bac9-eb2df32a8bff",
   "metadata": {},
   "source": [
    "## Creating the RAG pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "5e3cd149-3009-4a12-98ed-8873d3bf9ab5",
   "metadata": {
    "executionCancelledAt": null,
    "executionTime": 49,
    "lastExecutedAt": 1744298865976,
    "lastExecutedByKernel": "b0791251-777d-414c-b5cc-636b4f317d9a",
    "lastScheduledRunId": null,
    "lastSuccessfullyExecutedCode": "# Create retriever\nretriever = vectorstore.as_retriever()\n\n# Import PromptTemplate\nfrom langchain_core.prompts import PromptTemplate\n\n# Define a clearer, more professional prompt template\ntemplate = \"\"\"You are an expert assistant tasked with answering questions based on the provided documents.\nUse only the given context to generate your answer.\nIf the answer cannot be found in the context, clearly state that you do not know.\nBe detailed and precise in your response, but avoid mentioning or referencing the context itself.\n\nContext:\n{context}\n\nQuestion:\n{question}\n\nAnswer:\"\"\"\n\n# Create the PromptTemplate\nrag_prompt = PromptTemplate.from_template(template)\n"
   },
   "outputs": [],
   "source": [
    "# Create retriever\n",
    "retriever = vectorstore.as_retriever()\n",
    "\n",
    "# Import PromptTemplate\n",
    "from langchain_core.prompts import PromptTemplate\n",
    "\n",
    "# Define a clearer, more professional prompt template\n",
    "template = \"\"\"You are an expert assistant tasked with answering questions based on the provided documents.\n",
    "Use only the given context to generate your answer.\n",
    "If the answer cannot be found in the context, clearly state that you do not know.\n",
    "Be detailed and precise in your response, but avoid mentioning or referencing the context itself.\n",
    "\n",
    "Context:\n",
    "{context}\n",
    "\n",
    "Question:\n",
    "{question}\n",
    "\n",
    "Answer:\"\"\"\n",
    "\n",
    "# Create the PromptTemplate\n",
    "rag_prompt = PromptTemplate.from_template(template)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "889685f6-3e5e-4abb-8391-084bdb6b7d4d",
   "metadata": {
    "executionCancelledAt": null,
    "executionTime": 48,
    "lastExecutedAt": 1744298875804,
    "lastExecutedByKernel": "b0791251-777d-414c-b5cc-636b4f317d9a",
    "lastScheduledRunId": null,
    "lastSuccessfullyExecutedCode": "from langchain_core.output_parsers import StrOutputParser\nfrom langchain_core.runnables import RunnablePassthrough\n\nrag_chain = (\n    {\"context\": retriever, \"question\": RunnablePassthrough()}\n    | rag_prompt\n    | llm\n    | StrOutputParser()\n)"
   },
   "outputs": [],
   "source": [
    "from langchain_core.output_parsers import StrOutputParser\n",
    "from langchain_core.runnables import RunnablePassthrough\n",
    "\n",
    "rag_chain = (\n",
    "    {\"context\": retriever, \"question\": RunnablePassthrough()}\n",
    "    | rag_prompt\n",
    "    | llm\n",
    "    | StrOutputParser()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "ee406271-90d6-453b-a2aa-4753f08d30e5",
   "metadata": {
    "executionCancelledAt": null,
    "executionTime": 888,
    "lastExecutedAt": 1744298897317,
    "lastExecutedByKernel": "b0791251-777d-414c-b5cc-636b4f317d9a",
    "lastScheduledRunId": null,
    "lastSuccessfullyExecutedCode": "from IPython.display import display, Markdown\n\nresponse = rag_chain.invoke(\"What this tutorial about?\")\nMarkdown(response)"
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "This tutorial is about setting up and using the Janus project, specifically Janus Pro, a multimodal model that can understand images and generate images from text prompts, and building a local solution to use the model privately on a laptop GPU. It covers learning about the Janus Series, setting up the Janus project, building a Docker container to run the model locally, and testing its capabilities with various image and text prompts."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import display, Markdown\n",
    "\n",
    "response = rag_chain.invoke(\"What this tutorial about?\")\n",
    "Markdown(response)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "Welcome to DataCamp Workspaces.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}