week6/community-contributions/week6_day2_add_validation_set.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "28a0673e-96b5-43f2-8a8b-bd033bf851b0",
   "metadata": {},
   "source": [
    "# Add a Validation Set\n",
    "\n",
    "In the lecture, we created a curated dataset with **400,000 training items** and **2,000 test items**, but we did not include a validation (dev) set. This notebook demonstrates how to take Ed Donner’s dataset, [ed-donner/pricer-data](https://huggingface.co/datasets/ed-donner/pricer-data), and add a dev set to it.\n",
    "\n",
    "> **Note**: This notebook heavily uses snippets from the lectures’ `day2.ipynb` of Week 6.\n",
    "\n",
    "**Download the Updated Dataset**:  \n",
    "You can find the resulting dataset here: [antonawinkler/pricer-data](https://huggingface.co/datasets/antonawinkler/pricer-data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67cedf85-8125-4322-998e-9375fe745597",
   "metadata": {},
   "outputs": [],
   "source": [
    "# imports\n",
    "\n",
    "# Standard libraries\n",
    "import os\n",
    "import random\n",
    "from itertools import chain\n",
    "from collections import Counter, defaultdict\n",
    "\n",
    "# Third-party libraries\n",
    "from dotenv import load_dotenv\n",
    "from huggingface_hub import login\n",
    "from datasets import concatenate_datasets, load_dataset, Dataset, DatasetDict\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "# Local modules\n",
    "from items import Item\n",
    "from loaders import ItemLoader\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7390a6aa-79cb-4dea-b6d7-de7e4b13e472",
   "metadata": {},
   "outputs": [],
   "source": [
    "# environment\n",
    "\n",
    "load_dotenv()\n",
    "os.environ['HF_TOKEN'] = os.getenv('HF_TOKEN', 'your-key-if-not-using-env')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0732274a-aa6a-44fc-aee2-40dc8a8e4451",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Log in to HuggingFace\n",
    "\n",
    "hf_token = os.environ['HF_TOKEN']\n",
    "login(hf_token, add_to_git_credential=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1adcf323-de9d-4c24-a9c3-d7ae554d06ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2b6dc50-ac5c-4cf2-af2e-968ed8ef86d7",
   "metadata": {},
   "source": [
    "## Load the Original Dataset\n",
    "\n",
    "Load the original data from McAuley-Lab/Amazon-Reviews-2023."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1d06cd3-f3c2-44f0-a9f2-13b54ff8be5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_names = [\n",
    "    \"Automotive\",\n",
    "    \"Electronics\",\n",
    "    \"Office_Products\",\n",
    "    \"Tools_and_Home_Improvement\",\n",
    "     \"Cell_Phones_and_Accessories\",\n",
    "     \"Toys_and_Games\",\n",
    "     \"Appliances\",\n",
    "     \"Musical_Instruments\",\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa8fd0f0-509a-4298-8fcc-e499a061e1be",
   "metadata": {},
   "outputs": [],
   "source": [
    "items = []\n",
    "for dataset_name in dataset_names:\n",
    "    loader = ItemLoader(dataset_name)\n",
    "    items.extend(loader.load())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf6b6b66-4a4b-41c2-b366-1f598cf18351",
   "metadata": {},
   "source": [
    "# Create Balanced Dataset\n",
    "\n",
    "We apply the balancing algorithm from the course."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "549a4bad-abe7-4d36-ad77-fc70ba0f151c",
   "metadata": {},
   "outputs": [],
   "source": [
    "slots = defaultdict(list)\n",
    "for item in items:\n",
    "    slots[round(item.price)].append(item)\n",
    "\n",
    "np.random.seed(42)\n",
    "random.seed(42)\n",
    "sample = []\n",
    "for i in range(1, 1000):\n",
    "    slot = slots[i]\n",
    "    if i>=240:\n",
    "        sample.extend(slot)\n",
    "    elif len(slot) <= 1200:\n",
    "        sample.extend(slot)\n",
    "    else:\n",
    "        weights = np.array([1 if item.category=='Automotive' else 5 for item in slot])\n",
    "        weights = weights / np.sum(weights)\n",
    "        selected_indices = np.random.choice(len(slot), size=1200, replace=False, p=weights)\n",
    "        selected = [slot[i] for i in selected_indices]\n",
    "        sample.extend(selected)\n",
    "\n",
    "print(f\"There are {len(sample):,} items in the sample\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04280d2b-210a-4fad-9163-1b32a87fb990",
   "metadata": {},
   "source": [
    "The output I get is `There are 408,635 items in the sample`\n",
    "\n",
    "Since there are 400,000 items in the train set of ed-donner/pricer-data, we can aim for a 98/1/1 split."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d1e2836-0cae-4496-a5d4-d80bc14d566b",
   "metadata": {},
   "source": [
    "## Load Ed Donner's Pricer Data Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a84e5a71-fc44-4cdf-9bc2-c69f80b8ee94",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_ori = load_dataset(\"ed-donner/pricer-data\")\n",
    "train_ori = dataset_ori['train']\n",
    "test_ori = dataset_ori['test']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9c5c877-3d30-4013-9d0f-1e490755afeb",
   "metadata": {},
   "source": [
    "## Observation 1: Order of the Data Has Changed\n",
    "\n",
    "`dataset_without_devset` should be a subset of `sample`. The order however can be different. Let us check this.\n",
    "\n",
    "I see different results for the following two cells below, indicating that the order has changed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56ad8682-4d7f-4aad-9976-96eb6d9b4a5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample[0].prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e29a5ab-ca61-41cc-9b33-22d374681b85",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_ori[0]['text']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "469a5b3c-c1a2-461d-a88d-27aa08905b31",
   "metadata": {},
   "source": [
    "## Observation 2: Duplicate Items\n",
    "\n",
    "As an further challenge, the dataset shows duplicates with identical scrubbed descriptions. For some of these duplicates the prices are identical too (I see 1774), for others they differ (I see 6747).\n",
    "\n",
    "> **Note**: Below we use `defaultdict(list)` instead of `set` because it allows to inspect duplicates easily."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94adffe8-edf6-4503-9f8f-34e4dfd29da9",
   "metadata": {},
   "outputs": [],
   "source": [
    "PRICE_IS = \"\\n\\nPrice is $\"\n",
    "def get_key(text, price):\n",
    "    prefix, price_is, _price_nearest_dollar = text.partition(PRICE_IS)\n",
    "    return f\"{prefix}{price_is}{price}\"\n",
    "def get_key_without_price(text):\n",
    "    prefix, price_is, _price_nearest_dollar = text.partition(PRICE_IS)\n",
    "    return f\"{prefix}\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a015ba1b-69e0-4651-850f-d93d3f078d16",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Identify duplicates by text+price\n",
    "train_ori_dict = defaultdict(list)\n",
    "for datapoint in train_ori:\n",
    "    # Creates a key from the text and price (scrubbed)\n",
    "    key = get_key(datapoint[\"text\"], datapoint[\"price\"])\n",
    "    train_ori_dict[key].append(datapoint)\n",
    "\n",
    "# Number of exact duplicates (same text AND same price)\n",
    "exact_duplicates = len(train_ori) - len(train_ori_dict)\n",
    "print(f\"There are {exact_duplicates} duplicates with the same description and price.\")\n",
    "\n",
    "# Identify duplicates by text alone (ignoring price)\n",
    "train_ori_dict_no_price = defaultdict(list)\n",
    "for datapoint in train_ori:\n",
    "    key_no_price = get_key_without_price(datapoint[\"text\"])\n",
    "    train_ori_dict_no_price[key_no_price].append(datapoint)\n",
    "\n",
    "# Number of duplicates that differ in price but share the same text\n",
    "different_price_duplicates = len(train_ori_dict) - len(train_ori_dict_no_price)\n",
    "print(f\"In addition, there are {different_price_duplicates} data points where the description is duplicated but the price is different.\")\n",
    "\n",
    "# Total number of duplicates if we consider text alone\n",
    "overall_duplicates = len(train_ori) - len(train_ori_dict_no_price)\n",
    "print(f\"Overall number of duplicates: {overall_duplicates}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e577dd8b-be0f-4ab0-b45f-9d3459b1286a",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_ori_dict = defaultdict(list)\n",
    "for datapoint in test_ori:\n",
    "    key = get_key(datapoint['text'], datapoint['price'])\n",
    "    test_ori_dict[key].append(datapoint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0198fc23-0825-4ce1-a961-1d390d86cbdc",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_dict = defaultdict(list)\n",
    "for datapoint in sample:\n",
    "    key = get_key(datapoint.prompt, datapoint.price)\n",
    "    sample_dict[key].append(datapoint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37f24d22-51ef-472b-8c73-e969637fa925",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check if all data points in train_ori/test_ori are included in the new sample_dict.\n",
    "missing = []\n",
    "count_found = 0\n",
    "\n",
    "for datapoint in chain(train_ori, test_ori):\n",
    "    key = get_key(datapoint[\"text\"], datapoint[\"price\"])\n",
    "    if key not in sample_dict:\n",
    "        missing.append(datapoint)\n",
    "    else:\n",
    "        count_found += 1\n",
    "\n",
    "print(f\"We found {count_found} datapoints in sample_dict.\")\n",
    "print(f\"We are missing {len(missing)} datapoints that are not present in sample_dict.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60c9d186-c688-4559-9b51-f0045d16829b",
   "metadata": {},
   "source": [
    "Expected output of the previous cell\n",
    "```\n",
    "We found 402000 datapoints in sample_dict.\n",
    "We are missing 0 datapoints that are not present in sample_dict.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b05e22d-a755-4ee5-a18b-620f7ab1df8f",
   "metadata": {},
   "source": [
    "## Add Data Points to the Test and Validation Sets\n",
    "\n",
    "Since we can match all data points in the original train and test sets from `ed-donner/pricer-data`, we’ll now incorporate any *unused* items from our balanced sample into the test set and create a new validation (dev) set. Our goal is to achieve a **98/1/1** split for train, validation, and test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16638cf9-03c3-46bc-8116-cafdd9e23ac9",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_not_used_yet = [datapoint for key in sample_dict.keys() - train_ori_dict.keys() - test_ori_dict.keys() for datapoint in sample_dict[key]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58a593ad-29a1-4b35-9753-45db75e09666",
   "metadata": {},
   "outputs": [],
   "source": [
    "# As a santity check, let us visually verify that the distribution of sample_still_available is in line with the complete sample.\n",
    "\n",
    "# Plot the distribution of prices in sample\n",
    "def plot_price_distribution(items, name):\n",
    "    prices = [float(item.price) for item in items]\n",
    "    plt.figure(figsize=(15, 10))\n",
    "    plt.title(f\"{name} - Avg {sum(prices)/len(prices):.2f} and highest {max(prices):,.2f}\\n\")\n",
    "    plt.xlabel('Price ($)')\n",
    "    plt.ylabel('Count')\n",
    "    # see https://stackoverflow.com/questions/57026223/how-to-re-scale-the-counts-in-a-matplotlib-histogram\n",
    "    (counts, bins) = np.histogram(prices, bins=range(0, 1000, 10))\n",
    "    plt.hist(bins[:-1], color=\"darkblue\", bins=bins, weights=counts/len(prices))\n",
    "    plt.show()    \n",
    "\n",
    "\n",
    "def plot_category_distribution(items, name):\n",
    "    category_counts = Counter()\n",
    "    for item in items:\n",
    "        category_counts[item.category]+=1\n",
    "    categories = sorted(category_counts.keys())\n",
    "    counts = [category_counts[category] for category in categories]\n",
    "\n",
    "    # plot a pie chart\n",
    "    plt.figure(figsize=(12, 10))\n",
    "    plt.pie(counts, labels=categories, autopct='%1.0f%%', startangle=90)\n",
    "    \n",
    "    # Add a circle at the center to create a donut chart (optional)\n",
    "    centre_circle = plt.Circle((0,0), 0.70, fc='white')\n",
    "    fig = plt.gcf()\n",
    "    fig.gca().add_artist(centre_circle)\n",
    "    plt.title(f'{name} - Categories')\n",
    "    \n",
    "    # Equal aspect ratio ensures that pie is drawn as a circle\n",
    "    plt.axis('equal')  \n",
    "\n",
    "    plt.show()\n",
    "plot_price_distribution(sample, 'Complete set')\n",
    "plot_price_distribution(sample_not_used_yet, 'Not used yet')\n",
    "plot_category_distribution(sample, 'Complete set')\n",
    "plot_category_distribution(sample_not_used_yet, 'Not used yet')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba252265-b976-426a-aefc-ebc93b153fd4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# now add the unused items to the validation and test set\n",
    "random.seed(42)\n",
    "random.shuffle(sample_not_used_yet)\n",
    "validation_items = sample_not_used_yet[:4000]\n",
    "added_test_items = sample_not_used_yet[4000:]\n",
    "\n",
    "# create Huggingface dataset\n",
    "validation_dataset = Dataset.from_dict({\"text\": [item.prompt for item in validation_items], \"price\": [item.price for item in validation_items]})\n",
    "added_test_dataset = Dataset.from_dict({\"text\": [item.prompt for item in added_test_items], \"price\": [item.price for item in added_test_items]})\n",
    "\n",
    "dataset = DatasetDict({\n",
    "    \"train\": train_ori,\n",
    "    \"test\": concatenate_datasets([test_ori, added_test_dataset]),\n",
    "    \"validation\": validation_dataset,\n",
    "})\n",
    "\n",
    "print(f\"Divided into a training set of {dataset['train'].num_rows:,} items, a validation set of {dataset['validation'].num_rows:,} items, and a test set of {dataset['test'].num_rows:,} items\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c39ac5d7-84f8-4f7d-98e1-d24651ba3a80",
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you're ready to push to the hub, and fill in the dots with your HF username\n",
    "\n",
    "HF_USER = ...\n",
    "DATASET_NAME = f\"{HF_USER}/pricer-data\"\n",
    "dataset.push_to_hub(DATASET_NAME, private=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fcb2492-ef2a-468e-8bf1-deb18eef4d9c",
   "metadata": {},
   "source": [
    "## Use of Validation Sets\n",
    "\n",
    "When you train your model in Week 7.\n",
    "\n",
    "```python\n",
    "# load the train and validation set\n",
    "train = load_dataset(DATASET_NAME, split='train[:100%]') # or less than 100%\n",
    "validation = load_dataset(DATASET_NAME, split='validation[:100%]') # or less than 100% \n",
    "\n",
    "# Define training parameters\n",
    "train_parameters = SFTConfig(\n",
    "    eval_strategy=\"steps\", # or \"epoch\"\n",
    "    eval_steps=EVAL_STEPS,\n",
    "    ...\n",
    ")\n",
    "\n",
    "# Initialize fine-tuning with validation set\n",
    "fine_tuning = SFTTrainer(\n",
    "    eval_dataset=validation,\n",
    "    ...\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bceb4407-d91d-4731-9e96-189f6f953cbc",
   "metadata": {},
   "source": [
    "## A Closer Look at the Duplicates\n",
    "\n",
    "We have now created a dataset that includes a validation set and additional test data. During this process, we observed that **2% of the data contains duplicates**, where the scrubbed descriptions are identical.\n",
    "\n",
    "Duplicates can contribute to model overfitting. However, since only **2% of the dataset is duplicated**, the impact is likely minimal. Moreover, many of these duplicates actually refer to different physical objects rather than being true duplicates.\n",
    "\n",
    "### False Duplicates\n",
    "\n",
    "The “duplicates” we observe are often not duplicates in the original dataset. Minor differences in product descriptions may be removed by the scrubbing process, leading to items that *appear* identical but aren’t. For example:\n",
    "\n",
    "```\n",
    "<RinoGear Screen Protector Designed for Sony Xperia XZ Screen Protector Case Friendly Accessories Flexible Full Coverage Clear TPU Film = $0.95>\n",
    "<RinoGear (2-Pack) Screen Protector Designed for Sony Xperia XZ Screen Protector Case Friendly Accessories Flexible Full Coverage Clear TPU Film = $2.95>\n",
    "```\n",
    "\"(2-Pack)\" is removed in the scrub method.\n",
    "\n",
    "Similarly:\n",
    "```\n",
    "[<EBC Brakes USR7115 USR Series Sport Slotted Rotor = $31.22>,\n",
    " <EBC Brakes USR7314 USR Series Sport Slotted Rotor = $71.46>,\n",
    " <EBC Brakes USR7409 USR Series Sport Slotted Rotor = $88.67>,\n",
    "...\n",
    " <EBC Brakes USR7305 USR Series Sport Slotted Rotor = $406.55>,\n",
    " <EBC Brakes USR7384 USR Series Sport Slotted Rotor = $413.61>,\n",
    " <EBC Brakes USR1602 USR Series Sport Slotted Rotor = $615.1>]\n",
    "```\n",
    "These all represent different rotor models. \n",
    "\n",
    "**Even when both the scrubbed text and the price are identical**, the items may still refer to distinct products. For instance:\n",
    "```\n",
    "<5304486359 Refrigerator Door Handles Set Replacement for Frigidaire FFTR1821QW5A Refrigerator - Compatible with 5304486359 White Door Handles - UpStart Components Brand = $17.99>\n",
    "<5304486359 Refrigerator Door Handles Set Replacement for Frigidaire FFTR1831QP1 Refrigerator - Compatible with 5304486359 White Door Handles - UpStart Components Brand = $17.99>\n",
    "```\n",
    "\n",
    "### True Duplicates\n",
    "Finding *true* duplicates—where the scrubbed text, price, and underlying real-world product match—seems relatively rare. The following items in the **Appliances** set, for instance, likely refer to the same physical product:\n",
    "```python\n",
    "{'main_category': 'Tools & Home Improvement',\n",
    " 'title': 'Whirlpool 8318084 Lid Switch for Washer',\n",
    " 'average_rating': 4.6,\n",
    " 'rating_number': 511,\n",
    " 'features': ['Works with the following models: Whirlpool 1CLBR5432PQ0, Whirlpool 1CLBR5432PQ1, Whirlpool 1CLSQ9549PG0',\n",
    "  'This products adds a great value',\n",
    "  'This product is manufactured in United States',\n",
    "  'Works with the following models: Whirlpool 1CLBR5432PQ0, Whirlpool 1CLBR5432PQ1, Whirlpool 1CLSQ9549PG0',\n",
    "  'Whirlpool 1CLSQ9549PG1, Whirlpool 1CLSQ9549PW0',\n",
    "  'Whirlpool 1CLSQ9549PW1, Whirlpool 1CLSR7010PQ0',\n",
    "  'Whirlpool 1CLSR7010PQ1, Whirlpool 1CLSR7300PQ0',\n",
    "  'Genuine Replacement Part'],\n",
    " 'description': ['Product Description',\n",
    "  'Part Number 8318084 (AP3180933) replaces 1018522, AH886960, EA886960, PS886960., Easy to use and handle. This products adds a great value This product is manufactured in United States.',\n",
    "  'From the Manufacturer',\n",
    "  'Whirlpool 8318084 Lid Switch for Washer. Works with the following models: Whirlpool 1CLBR5432PQ0, Whirlpool 1CLBR5432PQ1, Whirlpool 1CLSQ9549PG0, Whirlpool 1CLSQ9549PG1, Whirlpool 1CLSQ9549PW0, Whirlpool 1CLSQ9549PW1, Whirlpool 1CLSR7010PQ0, Whirlpool 1CLSR7010PQ1, Whirlpool 1CLSR7300PQ0. Genuine Replacement Part.'],\n",
    " 'price': '25.55',\n",
    " 'images': {'hi_res': [None],\n",
    "  'large': ['https://m.media-amazon.com/images/I/31QE91zX0mL._AC_.jpg'],\n",
    "  'thumb': ['https://m.media-amazon.com/images/I/31QE91zX0mL._AC_US75_.jpg'],\n",
    "  'variant': ['MAIN']},\n",
    " 'videos': {'title': [\"Your Washer Won't Spin?\", '8318084 Washer Lid Switch'],\n",
    "  'url': ['https://www.amazon.com/vdp/09c00a975b4b46198b5703483f424981?ref=dp_vse_rvc_0',\n",
    "   'https://www.amazon.com/vdp/3c9b3dc3c93444978d542af3fab13c49?ref=dp_vse_rvc_1'],\n",
    "  'user_id': ['', '']},\n",
    " 'store': 'Whirlpool',\n",
    " 'categories': ['Appliances',\n",
    "  'Parts & Accessories',\n",
    "  'Washer Parts & Accessories'],\n",
    " 'details': '{\"Manufacturer\": \"Whirlpool\", \"Part Number\": \"8318084\", \"Item Weight\": \"1.34 ounces\", \"Product Dimensions\": \"3 x 2 x 2 inches\", \"Item model number\": \"8318084\", \"Is Discontinued By Manufacturer\": \"No\", \"Item Package Quantity\": \"1\", \"Included Components\": \"Kkk\", \"Batteries Included?\": \"No\", \"Batteries Required?\": \"No\", \"Warranty Description\": \"Kk\", \"Best Sellers Rank\": {\"Tools & Home Improvement\": 231142, \"Washer Parts & Accessories\": 1074}, \"Date First Available\": \"August 7, 2008\"}',\n",
    " 'parent_asin': 'B01CT25N26',\n",
    " 'bought_together': None,\n",
    " 'subtitle': None,\n",
    " 'author': None}\n",
    "\n",
    "{'main_category': 'Tools & Home Improvement',\n",
    " 'title': 'Whirlpool 8318084 Lid Switch for Washer',\n",
    " 'average_rating': 4.6,\n",
    " 'rating_number': 514,\n",
    " 'features': ['Works with the following models: Whirlpool 1CLBR5432PQ0, Whirlpool 1CLBR5432PQ1, Whirlpool 1CLSQ9549PG0',\n",
    "  'This products adds a great value',\n",
    "  'This product is manufactured in United States',\n",
    "  'Works with the following models: Whirlpool 1CLBR5432PQ0, Whirlpool 1CLBR5432PQ1, Whirlpool 1CLSQ9549PG0',\n",
    "  'Whirlpool 1CLSQ9549PG1, Whirlpool 1CLSQ9549PW0',\n",
    "  'Whirlpool 1CLSQ9549PW1, Whirlpool 1CLSR7010PQ0',\n",
    "  'Whirlpool 1CLSR7010PQ1, Whirlpool 1CLSR7300PQ0',\n",
    "  'Genuine Replacement Part'],\n",
    " 'description': ['Product Description',\n",
    "  'Part Number 8318084 (AP3180933) replaces 1018522, AH886960, EA886960, PS886960., Easy to use and handle. This products adds a great value This product is manufactured in United States.',\n",
    "  'From the Manufacturer',\n",
    "  'Whirlpool 8318084 Lid Switch for Washer. Works with the following models: Whirlpool 1CLBR5432PQ0, Whirlpool 1CLBR5432PQ1, Whirlpool 1CLSQ9549PG0, Whirlpool 1CLSQ9549PG1, Whirlpool 1CLSQ9549PW0, Whirlpool 1CLSQ9549PW1, Whirlpool 1CLSR7010PQ0, Whirlpool 1CLSR7010PQ1, Whirlpool 1CLSR7300PQ0. Genuine Replacement Part.'],\n",
    " 'price': '25.55',\n",
    " 'images': {'hi_res': [None],\n",
    "  'large': ['https://m.media-amazon.com/images/I/31QE91zX0mL._AC_.jpg'],\n",
    "  'thumb': ['https://m.media-amazon.com/images/I/31QE91zX0mL._AC_US75_.jpg'],\n",
    "  'variant': ['MAIN']},\n",
    " 'videos': {'title': ['AMI PARTS,Parts Specialist'],\n",
    "  'url': ['https://www.amazon.com/vdp/09a12ea79b1a4081a18909825437760b?ref=dp_vse_rvc_0'],\n",
    "  'user_id': ['']},\n",
    " 'store': 'Whirlpool',\n",
    " 'categories': ['Appliances',\n",
    "  'Parts & Accessories',\n",
    "  'Washer Parts & Accessories'],\n",
    " 'details': '{\"Manufacturer\": \"Whirlpool\", \"Part Number\": \"8318084\", \"Item Weight\": \"1.34 ounces\", \"Product Dimensions\": \"3 x 2 x 2 inches\", \"Item model number\": \"8318084\", \"Is Discontinued By Manufacturer\": \"No\", \"Item Package Quantity\": \"1\", \"Included Components\": \"kkk\", \"Batteries Included?\": \"No\", \"Batteries Required?\": \"No\", \"Warranty Description\": \"kk\", \"Best Sellers Rank\": {\"Tools & Home Improvement\": 166821, \"Washer Parts & Accessories\": 684}, \"Date First Available\": \"August 7, 2008\"}',\n",
    " 'parent_asin': 'B0050O1UR8',\n",
    " 'bought_together': None,\n",
    " 'subtitle': None,\n",
    " 'author': None}\n",
    "```\n",
    "\n",
    "### Takeaway\n",
    "2% of the dataset contains duplicates, but most of these represent different physical objects. It does not appear to be worthwhile to remove them from the dataset. In fact it can be better the keep them to have representative data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a1d7b72-a1ab-4fc4-9065-738bd11f8058",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "403a42a2-3913-4905-9475-97509fe86c5e",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}