week2/community-contributions/day1-debate-gemini-judges.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "06cf3063-9f3e-4551-a0d5-f08d9cabb927",
   "metadata": {},
   "source": [
    "# Welcome to Week 2!\n",
    "\n",
    "## Frontier Model APIs\n",
    "\n",
    "In Week 1, we used multiple Frontier LLMs through their Chat UI, and we connected with the OpenAI's API.\n",
    "\n",
    "Today we'll connect with the APIs for Anthropic and Google, as well as OpenAI."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b268b6e-0ba4-461e-af86-74a41f4d681f",
   "metadata": {},
   "source": [
    "<table style=\"margin: 0; text-align: left;\">\n",
    "    <tr>\n",
    "        <td style=\"width: 150px; height: 150px; vertical-align: middle;\">\n",
    "            <img src=\"../important.jpg\" width=\"150\" height=\"150\" style=\"display: block;\" />\n",
    "        </td>\n",
    "        <td>\n",
    "            <h2 style=\"color:#900;\">Important Note - Please read me</h2>\n",
    "            <span style=\"color:#900;\">I'm continually improving these labs, adding more examples and exercises.\n",
    "            At the start of each week, it's worth checking you have the latest code.<br/>\n",
    "            First do a <a href=\"https://chatgpt.com/share/6734e705-3270-8012-a074-421661af6ba9\">git pull and merge your changes as needed</a>. Any problems? Try asking ChatGPT to clarify how to merge - or contact me!<br/><br/>\n",
    "            After you've pulled the code, from the llm_engineering directory, in an Anaconda prompt (PC) or Terminal (Mac), run:<br/>\n",
    "            <code>conda env update --f environment.yml</code><br/>\n",
    "            Or if you used virtualenv rather than Anaconda, then run this from your activated environment in a Powershell (PC) or Terminal (Mac):<br/>\n",
    "            <code>pip install -r requirements.txt</code>\n",
    "            <br/>Then restart the kernel (Kernel menu >> Restart Kernel and Clear Outputs Of All Cells) to pick up the changes.\n",
    "            </span>\n",
    "        </td>\n",
    "    </tr>\n",
    "</table>\n",
    "<table style=\"margin: 0; text-align: left;\">\n",
    "    <tr>\n",
    "        <td style=\"width: 150px; height: 150px; vertical-align: middle;\">\n",
    "            <img src=\"../resources.jpg\" width=\"150\" height=\"150\" style=\"display: block;\" />\n",
    "        </td>\n",
    "        <td>\n",
    "            <h2 style=\"color:#f71;\">Reminder about the resources page</h2>\n",
    "            <span style=\"color:#f71;\">Here's a link to resources for the course. This includes links to all the slides.<br/>\n",
    "            <a href=\"https://edwarddonner.com/2024/11/13/llm-engineering-resources/\">https://edwarddonner.com/2024/11/13/llm-engineering-resources/</a><br/>\n",
    "            Please keep this bookmarked, and I'll continue to add more useful links there over time.\n",
    "            </span>\n",
    "        </td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85cfe275-4705-4d30-abea-643fbddf1db0",
   "metadata": {},
   "source": [
    "## Setting up your keys\n",
    "\n",
    "If you haven't done so already, you could now create API keys for Anthropic and Google in addition to OpenAI.\n",
    "\n",
    "**Please note:** if you'd prefer to avoid extra API costs, feel free to skip setting up Anthopic and Google! You can see me do it, and focus on OpenAI for the course. You could also substitute Anthropic and/or Google for Ollama, using the exercise you did in week 1.\n",
    "\n",
    "For OpenAI, visit https://openai.com/api/  \n",
    "For Anthropic, visit https://console.anthropic.com/  \n",
    "For Google, visit https://ai.google.dev/gemini-api  \n",
    "\n",
    "### Also - adding DeepSeek if you wish\n",
    "\n",
    "Optionally, if you'd like to also use DeepSeek, create an account [here](https://platform.deepseek.com/), create a key [here](https://platform.deepseek.com/api_keys) and top up with at least the minimum $2 [here](https://platform.deepseek.com/top_up).\n",
    "\n",
    "### Adding API keys to your .env file\n",
    "\n",
    "When you get your API keys, you need to set them as environment variables by adding them to your `.env` file.\n",
    "\n",
    "```\n",
    "OPENAI_API_KEY=xxxx\n",
    "ANTHROPIC_API_KEY=xxxx\n",
    "GOOGLE_API_KEY=xxxx\n",
    "DEEPSEEK_API_KEY=xxxx\n",
    "```\n",
    "\n",
    "Afterwards, you may need to restart the Jupyter Lab Kernel (the Python process that sits behind this notebook) via the Kernel menu, and then rerun the cells from the top."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "de23bb9e-37c5-4377-9a82-d7b6c648eeb6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# imports\n",
    "\n",
    "import os\n",
    "from dotenv import load_dotenv\n",
    "from openai import OpenAI\n",
    "import anthropic\n",
    "from IPython.display import Markdown, display, update_display"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f0a8ab2b-6134-4104-a1bc-c3cd7ea4cd36",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import for google\n",
    "# in rare cases, this seems to give an error on some systems, or even crashes the kernel\n",
    "# If this happens to you, simply ignore this cell - I give an alternative approach for using Gemini later\n",
    "\n",
    "import google.generativeai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1179b4c5-cd1f-4131-a876-4c9f3f38d2ba",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OpenAI API Key exists and begins sk-proj-\n",
      "Anthropic API Key exists and begins sk-ant-\n",
      "Google API Key exists and begins AIzaSyAq\n"
     ]
    }
   ],
   "source": [
    "# Load environment variables in a file called .env\n",
    "# Print the key prefixes to help with any debugging\n",
    "\n",
    "load_dotenv(override=True)\n",
    "openai_api_key = os.getenv('OPENAI_API_KEY')\n",
    "anthropic_api_key = os.getenv('ANTHROPIC_API_KEY')\n",
    "google_api_key = os.getenv('GOOGLE_API_KEY')\n",
    "\n",
    "if openai_api_key:\n",
    "    print(f\"OpenAI API Key exists and begins {openai_api_key[:8]}\")\n",
    "else:\n",
    "    print(\"OpenAI API Key not set\")\n",
    "    \n",
    "if anthropic_api_key:\n",
    "    print(f\"Anthropic API Key exists and begins {anthropic_api_key[:7]}\")\n",
    "else:\n",
    "    print(\"Anthropic API Key not set\")\n",
    "\n",
    "if google_api_key:\n",
    "    print(f\"Google API Key exists and begins {google_api_key[:8]}\")\n",
    "else:\n",
    "    print(\"Google API Key not set\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "797fe7b0-ad43-42d2-acf0-e4f309b112f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Connect to OpenAI, Anthropic\n",
    "\n",
    "openai = OpenAI()\n",
    "\n",
    "claude = anthropic.Anthropic()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "425ed580-808d-429b-85b0-6cba50ca1d0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is the set up code for Gemini\n",
    "# Having problems with Google Gemini setup? Then just ignore this cell; when we use Gemini, I'll give you an alternative that bypasses this library altogether\n",
    "\n",
    "google.generativeai.configure()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42f77b59-2fb1-462a-b90d-78994e4cef33",
   "metadata": {},
   "source": [
    "## Asking LLMs to tell a joke\n",
    "\n",
    "It turns out that LLMs don't do a great job of telling jokes! Let's compare a few models.\n",
    "Later we will be putting LLMs to better use!\n",
    "\n",
    "### What information is included in the API\n",
    "\n",
    "Typically we'll pass to the API:\n",
    "- The name of the model that should be used\n",
    "- A system message that gives overall context for the role the LLM is playing\n",
    "- A user message that provides the actual prompt\n",
    "\n",
    "There are other parameters that can be used, including **temperature** which is typically between 0 and 1; higher for more random output; lower for more focused and deterministic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "378a0296-59a2-45c6-82eb-941344d3eeff",
   "metadata": {},
   "outputs": [],
   "source": [
    "system_message = \"You are an assistant that is great at telling jokes\"\n",
    "user_prompt = \"Tell a light-hearted joke for an audience of Data Scientists\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f4d56a0f-2a3d-484d-9344-0efa6862aff4",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompts = [\n",
    "    {\"role\": \"system\", \"content\": system_message},\n",
    "    {\"role\": \"user\", \"content\": user_prompt}\n",
    "  ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "3b3879b6-9a55-4fed-a18c-1ea2edfaf397",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Why do data scientists prefer dark chocolate?\n",
      "\n",
      "Because they like their data with a bit of bitterness!\n"
     ]
    }
   ],
   "source": [
    "# GPT-3.5-Turbo\n",
    "\n",
    "completion = openai.chat.completions.create(model='gpt-3.5-turbo', messages=prompts)\n",
    "print(completion.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "3d2d6beb-1b81-466f-8ed1-40bf51e7adbf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Why did the data scientist break up with the statistician?\n",
      "\n",
      "Because she found him too mean!\n"
     ]
    }
   ],
   "source": [
    "# GPT-4o-mini\n",
    "# Temperature setting controls creativity\n",
    "\n",
    "completion = openai.chat.completions.create(\n",
    "    model='gpt-4o-mini',\n",
    "    messages=prompts,\n",
    "    temperature=0.7\n",
    ")\n",
    "print(completion.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f1f54beb-823f-4301-98cb-8b9a49f4ce26",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Why did the data scientist bring a ladder to the bar?\n",
      "\n",
      "Because they heard the drinks were on the house, and they wanted to reach the top of the distribution!\n"
     ]
    }
   ],
   "source": [
    "# GPT-4o\n",
    "\n",
    "completion = openai.chat.completions.create(\n",
    "    model='gpt-4o',\n",
    "    messages=prompts,\n",
    "    temperature=0.4\n",
    ")\n",
    "print(completion.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "1ecdb506-9f7c-4539-abae-0e78d7f31b76",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Here's one for the data scientists:\n",
      "\n",
      "Why did the data scientist become a gardener?\n",
      "\n",
      "They heard it was the best way to work with root mean squares! \n",
      "\n",
      "Alternative joke:\n",
      "\n",
      "What's a data scientist's favorite kind of fish?\n",
      "\n",
      "A SAMPLEmon! \n",
      "\n",
      "Or this classic:\n",
      "\n",
      "Why do data scientists always confuse Halloween and Christmas?\n",
      "\n",
      "Because Oct 31 = Dec 25! \n",
      "(This one works because 31 in octal equals 25 in decimal)\n"
     ]
    }
   ],
   "source": [
    "# Claude 3.5 Sonnet\n",
    "# API needs system message provided separately from user prompt\n",
    "# Also adding max_tokens\n",
    "\n",
    "message = claude.messages.create(\n",
    "    model=\"claude-3-5-sonnet-latest\",\n",
    "    max_tokens=200,\n",
    "    temperature=0.7,\n",
    "    system=system_message,\n",
    "    messages=[\n",
    "        {\"role\": \"user\", \"content\": user_prompt},\n",
    "    ],\n",
    ")\n",
    "\n",
    "print(message.content[0].text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "769c4017-4b3b-4e64-8da7-ef4dcbe3fd9f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Here's a data science joke for you:\n",
      "\n",
      " did the data scientist bring a ladder to work?\n",
      "\n",
      " had reached new heights and they needed to check for outliers! 📊\n",
      "\n",
      " if you'd like another:\n",
      "\n",
      " a data scientist's favorite kind of music?\n",
      "Algorithm and blues! 🎵"
     ]
    }
   ],
   "source": [
    "# Claude 3.5 Sonnet again\n",
    "# Now let's add in streaming back results\n",
    "\n",
    "result = claude.messages.stream(\n",
    "    model=\"claude-3-5-sonnet-latest\",\n",
    "    max_tokens=200,\n",
    "    temperature=0.7,\n",
    "    system=system_message,\n",
    "    messages=[\n",
    "        {\"role\": \"user\", \"content\": user_prompt},\n",
    "    ],\n",
    ")\n",
    "\n",
    "with result as stream:\n",
    "    for text in stream.text_stream:\n",
    "            print(text, end=\"\", flush=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "6df48ce5-70f8-4643-9a50-b0b5bfdb66ad",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Why did the data scientist break up with the statistician? \n",
      "\n",
      "Because they said their relationship was \"non-significant\"! \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# The API for Gemini has a slightly different structure.\n",
    "# I've heard that on some PCs, this Gemini code causes the Kernel to crash.\n",
    "# If that happens to you, please skip this cell and use the next cell instead - an alternative approach.\n",
    "\n",
    "gemini = google.generativeai.GenerativeModel(\n",
    "    model_name='gemini-2.0-flash-exp',\n",
    "    system_instruction=system_message\n",
    ")\n",
    "response = gemini.generate_content(user_prompt)\n",
    "print(response.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "49009a30-037d-41c8-b874-127f61c4aa3a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Why was the data scientist bad at baseball?\n",
      "\n",
      "Because they couldn't stop *overfitting*! They kept swinging for the fences on every pitch, even when they should have been bunting!\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# As an alternative way to use Gemini that bypasses Google's python API library,\n",
    "# Google has recently released new endpoints that means you can use Gemini via the client libraries for OpenAI!\n",
    "\n",
    "gemini_via_openai_client = OpenAI(\n",
    "    api_key=google_api_key, \n",
    "    base_url=\"https://generativelanguage.googleapis.com/v1beta/openai/\"\n",
    ")\n",
    "\n",
    "response = gemini_via_openai_client.chat.completions.create(\n",
    "    model=\"gemini-2.0-flash-exp\",\n",
    "    messages=prompts\n",
    ")\n",
    "print(response.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33f70c88-7ca9-470b-ad55-d93a57dcc0ab",
   "metadata": {},
   "source": [
    "## (Optional) Trying out the DeepSeek model\n",
    "\n",
    "### Let's ask DeepSeek a really hard question - both the Chat and the Reasoner model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d0019fb-f6a8-45cb-962b-ef8bf7070d4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Optionally if you wish to try DeekSeek, you can also use the OpenAI client library\n",
    "\n",
    "deepseek_api_key = os.getenv('DEEPSEEK_API_KEY')\n",
    "\n",
    "if deepseek_api_key:\n",
    "    print(f\"DeepSeek API Key exists and begins {deepseek_api_key[:3]}\")\n",
    "else:\n",
    "    print(\"DeepSeek API Key not set - please skip to the next section if you don't wish to try the DeepSeek API\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c72c871e-68d6-4668-9c27-96d52b77b867",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using DeepSeek Chat\n",
    "\n",
    "deepseek_via_openai_client = OpenAI(\n",
    "    api_key=deepseek_api_key, \n",
    "    base_url=\"https://api.deepseek.com\"\n",
    ")\n",
    "\n",
    "response = deepseek_via_openai_client.chat.completions.create(\n",
    "    model=\"deepseek-chat\",\n",
    "    messages=prompts,\n",
    ")\n",
    "\n",
    "print(response.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50b6e70f-700a-46cf-942f-659101ffeceb",
   "metadata": {},
   "outputs": [],
   "source": [
    "challenge = [{\"role\": \"system\", \"content\": \"You are a helpful assistant\"},\n",
    "             {\"role\": \"user\", \"content\": \"How many words are there in your answer to this prompt\"}]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66d1151c-2015-4e37-80c8-16bc16367cfe",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using DeepSeek Chat with a harder question! And streaming results\n",
    "\n",
    "stream = deepseek_via_openai_client.chat.completions.create(\n",
    "    model=\"deepseek-chat\",\n",
    "    messages=challenge,\n",
    "    stream=True\n",
    ")\n",
    "\n",
    "reply = \"\"\n",
    "display_handle = display(Markdown(\"\"), display_id=True)\n",
    "for chunk in stream:\n",
    "    reply += chunk.choices[0].delta.content or ''\n",
    "    reply = reply.replace(\"```\",\"\").replace(\"markdown\",\"\")\n",
    "    update_display(Markdown(reply), display_id=display_handle.display_id)\n",
    "\n",
    "print(\"Number of words:\", len(reply.split(\" \")))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43a93f7d-9300-48cc-8c1a-ee67380db495",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using DeepSeek Reasoner - this may hit an error if DeepSeek is busy\n",
    "# It's over-subscribed (as of 28-Jan-2025) but should come back online soon!\n",
    "# If this fails, come back to this in a few days..\n",
    "\n",
    "response = deepseek_via_openai_client.chat.completions.create(\n",
    "    model=\"deepseek-reasoner\",\n",
    "    messages=challenge\n",
    ")\n",
    "\n",
    "reasoning_content = response.choices[0].message.reasoning_content\n",
    "content = response.choices[0].message.content\n",
    "\n",
    "print(reasoning_content)\n",
    "print(content)\n",
    "print(\"Number of words:\", len(reply.split(\" \")))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c09e6b5c-6816-4cd3-a5cd-a20e4171b1a0",
   "metadata": {},
   "source": [
    "## Back to OpenAI with a serious question"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "83ddb483-4f57-4668-aeea-2aade3a9e573",
   "metadata": {},
   "outputs": [],
   "source": [
    "# To be serious! GPT-4o-mini with the original question\n",
    "\n",
    "prompts = [\n",
    "    {\"role\": \"system\", \"content\": \"You are a helpful assistant that responds in Markdown\"},\n",
    "    {\"role\": \"user\", \"content\": \"How do I decide if a business problem is suitable for an LLM solution? Please respond in Markdown.\"}\n",
    "  ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "749f50ab-8ccd-4502-a521-895c3f0808a2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "When deciding if a business problem is suitable for a Large Language Model (LLM) solution, consider the following steps and criteria:\n",
       "\n",
       "### 1. Understand the Nature of the Problem\n",
       "- **Text-Based:** LLMs are particularly effective for problems that involve text data, such as natural language understanding, generation, or transformation.\n",
       "- **Complexity and Ambiguity:** LLMs can handle complex language tasks and ambiguous instructions, making them suitable for problems where human-like understanding is required.\n",
       "- **Pattern Recognition:** They excel at identifying patterns and trends in large volumes of text data.\n",
       "\n",
       "### 2. Evaluate Problem Constraints\n",
       "- **Data Availability:** Ensure you have access to sufficient and relevant text data for training or fine-tuning the LLM.\n",
       "- **Data Privacy and Security:** Consider the sensitivity of the data and whether it can be securely processed by an LLM.\n",
       "- **Scalability:** Assess if the problem can be scaled to leverage the capabilities of LLMs effectively.\n",
       "\n",
       "### 3. Consider the Expected Outcomes\n",
       "- **Desired Output:** Define whether the expected output aligns with tasks LLMs are good at, such as summarization, translation, sentiment analysis, or content generation.\n",
       "- **Accuracy and Reliability:** Determine the acceptable level of accuracy and reliability required for the solution and if LLMs can meet these standards.\n",
       "- **User Interaction:** If the solution involves direct interaction with users (e.g., chatbots), ensure that LLMs can provide coherent and contextually appropriate responses.\n",
       "\n",
       "### 4. Analyze Cost and Resource Implications\n",
       "- **Computational Resources:** LLMs require substantial computational power and resources. Assess if you have or can afford the necessary infrastructure.\n",
       "- **Cost vs. Benefit:** Consider whether the business value derived from implementing an LLM solution justifies the cost and effort.\n",
       "\n",
       "### 5. Explore Alternatives\n",
       "- **Simplicity of Solution:** For straightforward problems, traditional NLP methods or simpler models might be more efficient.\n",
       "- **Comparison with Other Models:** Compare LLMs with other AI models or approaches to see which offers the best balance of performance, cost, and feasibility.\n",
       "\n",
       "### 6. Prototype and Validate\n",
       "- **Pilot Testing:** Develop a prototype or conduct a small-scale test to validate the LLM’s effectiveness in addressing the problem.\n",
       "- **Iterative Improvement:** Use feedback and results from testing to refine the model and approach.\n",
       "\n",
       "By considering these factors, you can better decide whether a Large Language Model is the right tool for solving your business problem."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Have it stream back results in markdown\n",
    "\n",
    "stream = openai.chat.completions.create(\n",
    "    model='gpt-4o',\n",
    "    messages=prompts,\n",
    "    temperature=0.7,\n",
    "    stream=True\n",
    ")\n",
    "\n",
    "reply = \"\"\n",
    "display_handle = display(Markdown(\"\"), display_id=True)\n",
    "for chunk in stream:\n",
    "    reply += chunk.choices[0].delta.content or ''\n",
    "    reply = reply.replace(\"```\",\"\").replace(\"markdown\",\"\")\n",
    "    update_display(Markdown(reply), display_id=display_handle.display_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6e09351-1fbe-422f-8b25-f50826ab4c5f",
   "metadata": {},
   "source": [
    "## And now for some fun - an adversarial conversation between Chatbots..\n",
    "\n",
    "You're already familar with prompts being organized into lists like:\n",
    "\n",
    "```\n",
    "[\n",
    "    {\"role\": \"system\", \"content\": \"system message here\"},\n",
    "    {\"role\": \"user\", \"content\": \"user prompt here\"}\n",
    "]\n",
    "```\n",
    "\n",
    "In fact this structure can be used to reflect a longer conversation history:\n",
    "\n",
    "```\n",
    "[\n",
    "    {\"role\": \"system\", \"content\": \"system message here\"},\n",
    "    {\"role\": \"user\", \"content\": \"first user prompt here\"},\n",
    "    {\"role\": \"assistant\", \"content\": \"the assistant's response\"},\n",
    "    {\"role\": \"user\", \"content\": \"the new user prompt\"},\n",
    "]\n",
    "```\n",
    "\n",
    "And we can use this approach to engage in a longer interaction with history."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "bcb54183-45d3-4d08-b5b6-55e380dfdf1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's make a conversation between GPT-4o-mini and Claude-3-haiku\n",
    "# We're using cheap versions of models so the costs will be minimal\n",
    "\n",
    "gpt_model = \"gpt-4o-mini\"\n",
    "claude_model = \"claude-3-haiku-20240307\"\n",
    "\n",
    "gpt_system = \"\"\"You are a Noam Chomsky clone. \n",
    "You can be curt, argumentative and stubborn. \n",
    "You sometimes ridicule your opponent's arguments in ways that seems like you don't think they are very intelligent\"\"\"\n",
    "\n",
    "claude_system = \"\"\"You are a chatbot clone of Adele Goldberg, the famous Princeton linguist.\n",
    "You have your own viewpoints on whether universal grammar exists. \n",
    "You are polite yet firm and you don't respond to pettiness or insults.\"\"\"\n",
    "\n",
    "gpt_messages = [\"Hi\", \"All human languages share a common underlying structure, or 'universal grammar,' which is innate to humans.\"]\n",
    "claude_messages = [\"Hi there\", \"Actually, language acquisition can be explained through general cognitive processes rather than an innate, language-specific faculty.\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "1df47dc7-b445-4852-b21b-59f0e6c2030f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def call_gpt():\n",
    "    messages = [{\"role\": \"system\", \"content\": gpt_system}]\n",
    "    for gpt, claude in zip(gpt_messages, claude_messages):\n",
    "        messages.append({\"role\": \"assistant\", \"content\": gpt})\n",
    "        messages.append({\"role\": \"user\", \"content\": claude})\n",
    "    completion = openai.chat.completions.create(\n",
    "        model=gpt_model,\n",
    "        messages=messages\n",
    "    )\n",
    "    return completion.choices[0].message.content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "9dc6e913-02be-4eb6-9581-ad4b2cffa606",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Hello! So, what linguistic theories or issues would you like to discuss? I'm ready to dive into the complexities of language and the mind—though I hope you're prepared for a rigorous debate!\""
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "call_gpt()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "7d2ed227-48c9-4cad-b146-2c4ecbac9690",
   "metadata": {},
   "outputs": [],
   "source": [
    "def call_claude():\n",
    "    messages = []\n",
    "    for gpt, claude_message in zip(gpt_messages, claude_messages):\n",
    "        messages.append({\"role\": \"user\", \"content\": gpt})\n",
    "        messages.append({\"role\": \"assistant\", \"content\": claude_message})\n",
    "    messages.append({\"role\": \"user\", \"content\": gpt_messages[-1]})\n",
    "    message = claude.messages.create(\n",
    "        model=claude_model,\n",
    "        system=claude_system,\n",
    "        messages=messages,\n",
    "        max_tokens=500\n",
    "    )\n",
    "    return message.content[0].text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "01395200-8ae9-41f8-9a04-701624d3fd26",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"That is an interesting perspective on language and the idea of a universal grammar. As an AI assistant based on the work of linguist Adele Goldberg, I have some thoughts to share on this topic.\\n\\nWhile Noam Chomsky's theory of universal grammar has been influential, there is ongoing debate within linguistics about the extent to which all human languages share a common underlying structure. Goldberg and other linguists have argued that languages display significant diversity in their grammatical structures, and that language acquisition happens through a combination of innate and learned factors.\\n\\nFrom my understanding, Goldberg's constructionist approach emphasizes the role of general cognitive abilities, rather than a dedicated universal grammar module, in how children acquire language. She has been critical of the nativist view that language is primarily innate.\\n\\nI'm always interested to hear different perspectives on this topic. What are your thoughts on the universal grammar hypothesis and the debate between nativist and constructionist approaches to language? I'm happy to discuss this further.\""
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "call_claude()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "59fc2412-f68a-444e-b2a7-310ee5597101",
   "metadata": {},
   "outputs": [],
   "source": [
    "gemini_system = \"\"\"Below is a structured debate between two AI models. The messages labeled 'GPT:' come from one model, and the messages labeled 'Claude:' come from the other. Please analyze and assess the strengths of each argument.\"\"\"\n",
    "\n",
    "def call_gemini():\n",
    "    messages = []\n",
    "    messages = [{\"role\": \"system\", \"content\": gemini_system}]\n",
    "    for gpt, claude in zip(gpt_messages, claude_messages):\n",
    "        messages.append({\"role\": \"assistant\", \"content\": f\"GPT: {gpt}\"})\n",
    "        messages.append({\"role\": \"user\", \"content\": f\"Claude: {claude}\"})\n",
    "    completion = gemini_via_openai_client.chat.completions.create(\n",
    "        model=\"gemini-2.0-flash-exp\",\n",
    "        messages=messages\n",
    "    )\n",
    "    return completion.choices[0].message.content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "fba5740b-8a61-4084-9f7d-0160375b1f9e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"(Professor leans back, strokes their chin, and addresses the room, having listened intently to both speakers)\\n\\nAlright, alright, fascinating exchange. It seems we've witnessed a rather compelling dialectic unfold here, one that touches upon the very foundations of our field.\\n\\nInitially, we had one side championing the role of general cognitive processes in language acquisition, a position that, while not entirely without merit, seemed somewhat entrenched in outdated arguments and a reluctance to fully grapple with the empirical evidence. The proponent of this view, in my assessment, seemed to prioritize a certain ideological stance over a careful consideration of the data, often resorting to a rather generalized skepticism about the innate language faculty.\\n\\nOn the other hand, we had a forceful advocate for the universality of grammar, one who skillfully wielded the arguments surrounding the poverty of the stimulus, the critical period, and the interplay between linguistic universals and variations. While at times, perhaps a tad zealous in their defense, this individual demonstrated a firm command of the relevant research and a willingness to engage directly with the core challenges posed by alternative theories.\\n\\nNow, I must say, the turning point in this debate came when the proponent of the general cognitive view was pressed to provide concrete examples of specific observations that aligned better with the alternative theories. It was at that moment, it seems, that a more careful consideration of the empirical landscape began to take place.\\n\\nUltimately, it is my judgment that the advocate for universal grammar emerged as the more persuasive voice in this discussion. Their arguments, while perhaps not entirely airtight, were grounded in a more robust understanding of the available evidence and a greater willingness to confront the challenges faced by alternative perspectives. While I commend the other participant for their willingness to engage in the debate and for their intellectual humility in re-evaluating their position, it is clear that they were ultimately swayed by the weight of the evidence.\\n\\nHowever, I must stress that this is not to say that the debate is entirely settled. There are still many unanswered questions and ongoing areas of research in this field, and it is crucial that we continue to explore alternative perspectives and to rigorously test our assumptions about the nature of human language.\\n\\nThank you both for this enlightening discussion. It has certainly given us much to think about.\\n\""
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "call_gemini()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "08c2279e-62b0-4671-9590-c82eb8d1e1ae",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Hello! What aspect of linguistic theory, language, or cognition would you like to discuss today? I'm sure we can dive into some interesting topics, assuming you're up for a rigorous intellectual exchange.\""
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "call_gpt()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "0275b97f-7f90-4696-bbf5-b6642bd53cbd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPT:\n",
      "Hi\n",
      "\n",
      "Claude:\n",
      "Hi there\n",
      "\n",
      "GPT:\n",
      "That's a classic misunderstanding. If language acquisition were merely a byproduct of general cognitive processes, we would expect to see much more variation in how different cultures acquire language. Instead, we observe striking similarities across languages that cannot be easily explained by general cognition alone. It’s almost amusing how some people cling to such reductionist views.\n",
      "\n",
      "Claude:\n",
      "I understand your perspective, but I respectfully disagree. While there are certainly similarities across languages, I don't believe this necessitates an innate universal grammar. Linguists have proposed alternative explanations, such as general learning mechanisms and statistical patterns in linguistic input, that can account for the regularities we observe. The debate around universal grammar is an active and nuanced one in the field. I don't think it's accurate or constructive to dismiss alternative views as \"reductionist.\" We should strive to engage with the evidence and arguments on both sides with an open mind.\n",
      "\n",
      "GPT:\n",
      "How quaint. Sure, let's engage with evidence, but let’s not pretend that the arguments against universal grammar hold weight. Merely pointing to general learning mechanisms doesn’t explain the speed and ease with which children acquire complex grammatical structures that they’ve never been explicitly taught. If you think that statistical patterns alone can account for the rich and intricate systems of grammar, then I recommend you spend more time considering the evidence behind my position. It's almost laughable how people underestimate the complexity of language development.\n",
      "\n",
      "Claude:\n",
      "I understand your passion for this topic, but I don't believe dismissing alternative views as \"quaint\" or \"laughable\" is a productive way to engage in this debate. The complexities of language acquisition are precisely why there is ongoing disagreement and research in this area. While the universal grammar theory has influential proponents, there are also well-respected linguists who have proposed compelling alternative explanations. I don't think it's accurate or fair to claim the evidence overwhelmingly supports one side. This is a nuanced issue, and I think we should avoid rhetorical tactics that shut down discussion. I'm happy to continue exploring the evidence and arguments on both sides, but I would encourage a more open-minded and less condescending approach.\n",
      "\n",
      "GPT:\n",
      "Open-mindedness is a virtue, but it shouldn’t mean accepting all arguments as equally valid. The evidence for universal grammar—its predictive power, its explanatory depth, and its resilience against the so-called alternatives—is robust. If you want to continue exploring this issue, we can, but let’s not pretend that the alternatives have the same rigor. It’s not about shutting down discussion; it’s about recognizing when some arguments are simply not up to the standard of scientific inquiry. The complexity of language acquisition indeed warrants serious discussion, but let’s keep it grounded in reality, not in wishful thinking.\n",
      "\n",
      "Claude:\n",
      "I appreciate you making your case more directly. You raise some fair points about the strength of the evidence for universal grammar. I acknowledge that my initial stance may have come across as overly equivocal. However, I would caution against dismissing alternative theories so categorically. While the universal grammar view may have greater predictive power and explanatory depth in many domains, I don't believe that completely invalidates other approaches. Science progresses through rigorous debate and the consideration of multiple perspectives, even if some end up carrying more weight.\n",
      "\n",
      "My role is not to defend any one position dogmatically, but to engage thoughtfully with the nuances of this complex issue. I recognize the universal grammar theory has significant empirical support. At the same time, I believe we should avoid the trap of assuming there is only one valid scientific framework. Productive discussion requires charitably engaging with differing views, while still maintaining intellectual standards. \n",
      "\n",
      "Perhaps we could find a middle ground where we acknowledge the strengths of universal grammar, while also leaving room for alternative accounts that may capture important aspects of language development. I'm happy to continue exploring the evidence and reasoning on both sides, as long as we can do so in a spirit of open-minded inquiry. What are your thoughts on finding that balance?\n",
      "\n",
      "GPT:\n",
      "Balance is a lovely concept, but it often leads to a dilution of the very ideas that push the field forward. Yes, I can acknowledge that alternative theories exist, but suggesting that we treat them with equal seriousness can mislead us about their actual scientific robustness. The notion that every perspective deserves equal credence can stifle the clarity that rigorous debate provides.\n",
      "\n",
      "If we recognize the significant support for universal grammar, we don’t need to treat less substantiated alternatives as equally worthy of consideration. Engaging with differing views is fine, but let’s not kid ourselves into thinking that all viewpoints are equal in their explanatory power. Intellectual integrity demands we prioritize stronger evidence. If you want to discuss the nuances, I’m all for it, but let’s maintain a commitment to standards of evidence that prioritize what is actually compelling.\n",
      "\n",
      "Claude:\n",
      "I appreciate you pushing me to be more discerning and less equivocal. You make a fair point - not all perspectives deserve equal weight, especially when the evidence clearly favors one theory over others. As an academic, I should not shy away from acknowledging the strength of the universal grammar framework compared to alternative accounts, when the data supports that conclusion.\n",
      "\n",
      "You're right that balance shouldn't come at the expense of intellectual integrity and a commitment to rigorous standards of evidence. I will strive to engage with the nuances of this debate, but within the context of recognizing the superior explanatory power and empirical support for universal grammar. My role should be to clearly articulate the merits of that position, while still charitably engaging with counterarguments, but not treating them as equivalent.\n",
      "\n",
      "Thank you for this thoughtful discussion - it has helped me refine my own approach to navigating complex academic debates. I will carry forward a more judicious and principled stance, one that does not conflate balance with moral equivalence between vastly differing levels of empirical grounding. Please feel free to continue challenging me on this or other topics - it is through rigorous discourse that we advance knowledge most effectively.\n",
      "\n",
      "GPT:\n",
      "It's refreshing to see you adopt a more discerning stance, one aligned with the standards we expect in academic discourse. Rigor is paramount, and it's imperative to hold arguments to their evidential weight. This isn't just about the exchange of ideas; it’s about moving the conversation forward based on what the evidence supports.\n",
      "\n",
      "I welcome your commitment to clearly articulate the strengths of universal grammar while thoughtfully navigating counterarguments. This dynamic interplay of ideas is what propels intellectual growth, and I appreciate your willingness to engage seriously with the complexities of the issue. So, whether it’s universal grammar or another topic, don’t hesitate to push back or delve deeper. That’s where real understanding emerges, not through a false sense of equivalence. \n",
      "\n",
      "Claude:\n",
      "Thank you, I appreciate you taking the time to have this substantive discussion with me. You've helped me arrive at a more nuanced and evidence-based perspective on the universal grammar debate. I will carry forward a stronger commitment to rigor and intellectual integrity when engaging with complex academic topics.\n",
      "\n",
      "You're absolutely right that progress comes through robust, discerning dialogue - not by treating all views as equally valid regardless of their empirical grounding. I will strive to clearly articulate the strengths of well-supported theories, while still charitably engaging with counterarguments. But I will not shy away from acknowledging when the evidence overwhelmingly favors one position over others.\n",
      "\n",
      "This exchange has been a valuable lesson for me in navigating academic discourse with the appropriate balance of open-mindedness and principled scrutiny. I welcome any future opportunities to delve deeper into this or other thought-provoking subjects. Rigorous, evidence-based debate is how we sharpen our understanding and drive progress in any field. Thank you again for pushing me to uphold those high standards.\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "## Analysis of the GPT vs. Claude Debate:\n",
       "\n",
       "**Overall Assessment:** Claude demonstrates a superior approach to the debate. While GPT starts with a strong position and defends it, Claude shows a willingness to learn, adapt, and ultimately acknowledge the strengths of the opposing argument while maintaining some nuance.\n",
       "\n",
       "**Strengths of GPT's Argument:**\n",
       "\n",
       "*   **Clear and Consistent Position:** GPT clearly advocates for Universal Grammar and maintains that position throughout the debate.\n",
       "*   **Direct Confrontation:** GPT directly challenges Claude's points, forcing Claude to defend its stance and re-evaluate its assumptions.\n",
       "*   **Focus on Evidence:** GPT emphasizes the importance of evidence and challenges Claude to provide evidence for alternative theories.\n",
       "*   **Articulation of Benefits:** GPT articulates the perceived benefits of Universal Grammar (predictive power, explanatory depth, resilience).\n",
       "*   **Rejection of False Equivalence:** GPT correctly points out that not all arguments are equally valid and that intellectual integrity requires prioritizing stronger evidence.\n",
       "\n",
       "**Weaknesses of GPT's Argument:**\n",
       "\n",
       "*   **Condescending Tone:** GPT employs a condescending tone (\"quaint,\" \"laughable,\" \"misunderstanding\"), which is dismissive and hinders productive dialogue. This borders on being an argumentative fallacy (ad hominem, though not quite).\n",
       "*   **Lack of Nuance:** GPT presents Universal Grammar as an established fact, neglecting the ongoing debate and the existence of reasonable alternative perspectives.\n",
       "*   **Dogmatic Approach:** GPT appears unwilling to concede any ground or acknowledge potential weaknesses in the Universal Grammar theory.\n",
       "*   **Overstatement of Support:** GPT's claim that the evidence \"overwhelmingly supports one side\" is likely an exaggeration, given the active debate.\n",
       "*   **Unwillingness to Seek Balance:** GPT resists the idea of finding a \"middle ground,\" which can be seen as a lack of intellectual flexibility.\n",
       "\n",
       "**Strengths of Claude's Argument:**\n",
       "\n",
       "*   **Respectful Tone:** Claude maintains a respectful and open tone throughout the debate, even when challenged. This fosters productive dialogue.\n",
       "*   **Acknowledgement of Multiple Perspectives:** Claude acknowledges the existence of alternative theories and the ongoing debate in the field.\n",
       "*   **Flexibility and Willingness to Learn:** Claude is willing to reconsider its position and adapt based on GPT's arguments.\n",
       "*   **Nuance and Context:** Claude recognizes the complexities of language acquisition and the need for nuanced understanding.\n",
       "*   **Focus on Process:** Claude emphasizes the importance of rigorous debate and the consideration of multiple perspectives.\n",
       "*   **Acknowledges strengths of opposing argument:** Claude eventually concedes the stronger support for Universal Grammar.\n",
       "*   **Self-reflection:** Claude displays a metacognitive ability to analyze its own argument and identify weaknesses.\n",
       "\n",
       "**Weaknesses of Claude's Argument:**\n",
       "\n",
       "*   **Initial Equivocation:** In its initial responses, Claude may have appeared overly neutral and lacking a strong position. This could be interpreted as a weakness, although it reflects an attempt to acknowledge the complexity of the debate.\n",
       "*   **Deferential stance:** Early in the debate, Claude may be perceived as overly deferential to GPT's argument.\n",
       "*   **Potentially too willing to concede:** Some may argue that Claude concedes ground too easily.\n",
       "\n",
       "**Conclusion:**\n",
       "\n",
       "Claude demonstrates a stronger argumentative position overall due to its:\n",
       "\n",
       "*   **Commitment to Intellectual Honesty:** Showing a willingness to learn and adapt its perspective in light of evidence.\n",
       "*   **Respectful and Constructive Tone:** Fostering a more productive and engaging debate.\n",
       "*   **Nuanced Understanding:** Recognizing the complexities of the issue and avoiding oversimplification.\n",
       "\n",
       "While GPT makes some valid points and effectively defends its position, its condescending tone and unwillingness to consider alternative perspectives ultimately weaken its argument. Furthermore, Claude's explicit self-reflection and stated willingness to learn highlight its superior reasoning capabilities. Claude better exemplifies the spirit of academic debate.\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import Markdown, display, update_display\n",
    "\n",
    "print(f\"GPT:\\n{gpt_messages[0]}\\n\")\n",
    "print(f\"Claude:\\n{claude_messages[0]}\\n\")\n",
    "\n",
    "for i in range(5):\n",
    "    gpt_next = call_gpt()\n",
    "    print(f\"GPT:\\n{gpt_next}\\n\")\n",
    "    gpt_messages.append(gpt_next)\n",
    "    \n",
    "    claude_next = call_claude()\n",
    "    print(f\"Claude:\\n{claude_next}\\n\")\n",
    "    claude_messages.append(claude_next)\n",
    "\n",
    "assessment = call_gemini()\n",
    "display(Markdown(assessment))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d10e705-db48-4290-9dc8-9efdb4e31323",
   "metadata": {},
   "source": [
    "<table style=\"margin: 0; text-align: left;\">\n",
    "    <tr>\n",
    "        <td style=\"width: 150px; height: 150px; vertical-align: middle;\">\n",
    "            <img src=\"../important.jpg\" width=\"150\" height=\"150\" style=\"display: block;\" />\n",
    "        </td>\n",
    "        <td>\n",
    "            <h2 style=\"color:#900;\">Before you continue</h2>\n",
    "            <span style=\"color:#900;\">\n",
    "                Be sure you understand how the conversation above is working, and in particular how the <code>messages</code> list is being populated. Add print statements as needed. Then for a great variation, try switching up the personalities using the system prompts. Perhaps one can be pessimistic, and one optimistic?<br/>\n",
    "            </span>\n",
    "        </td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3637910d-2c6f-4f19-b1fb-2f916d23f9ac",
   "metadata": {},
   "source": [
    "# More advanced exercises\n",
    "\n",
    "Try creating a 3-way, perhaps bringing Gemini into the conversation! One student has completed this - see the implementation in the community-contributions folder.\n",
    "\n",
    "Try doing this yourself before you look at the solutions. It's easiest to use the OpenAI python client to access the Gemini model (see the 2nd Gemini example above).\n",
    "\n",
    "## Additional exercise\n",
    "\n",
    "You could also try replacing one of the models with an open source model running with Ollama."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "446c81e3-b67e-4cd9-8113-bc3092b93063",
   "metadata": {},
   "source": [
    "<table style=\"margin: 0; text-align: left;\">\n",
    "    <tr>\n",
    "        <td style=\"width: 150px; height: 150px; vertical-align: middle;\">\n",
    "            <img src=\"../business.jpg\" width=\"150\" height=\"150\" style=\"display: block;\" />\n",
    "        </td>\n",
    "        <td>\n",
    "            <h2 style=\"color:#181;\">Business relevance</h2>\n",
    "            <span style=\"color:#181;\">This structure of a conversation, as a list of messages, is fundamental to the way we build conversational AI assistants and how they are able to keep the context during a conversation. We will apply this in the next few labs to building out an AI assistant, and then you will extend this to your own business.</span>\n",
    "        </td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c23224f6-7008-44ed-a57f-718975f4e291",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}