train.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HjS1m1tE1Thl"
      },
      "source": [
        "# __Девопсная домашка по трансформерам__\n",
        "\n",
        "## __Описание__\n",
        "\n",
        "Ваш главный квест на эту домашку - сделать свой простой сервис на трансформерах. Вот прям целый сервис: начиная с данных и заканчивая графическим интерфейсом где-то в интернете. Ваш сервис может решать либо одну из предложенных ниже задач, либо любую другую (что-то более дорогое лично вам).\n",
        "\n",
        "__Стандартная задача: классификатор статей.__ Нужно построить сервис который принимает название статьи и её abstract, и выдаёт наиболее вероятную тематику статьи: скажем, физика, биология или computer science. В интерфейсе должно быть можно ввести отдельно abstract, отдельно название -- и увидеть топ-95%* тематик, отсортированных по убыванию вероятности. Если abstract не ввели, нужно классифицировать статью только по названию. Ниже вас ждут инструкции и данные именно для этой задачи.\n",
        "\n",
        "<details><summary><u> Что значит Топ-95%?</u></summary>\n",
        "    Нужно выдавать темы по убыванию вероятности, пока их суммарная вероятность не превысит 95%. В зависимости от предсказанной вероятности, это может быть одна или более тем. Например, если модель предсказала вероятности [4%, 20%, 60%, 2%, 14%], нужно вывести 3 топ-3 класса. Если один из классов имеет вероятность 96%, достаточно вывести один этот класс.\n",
        "</details>\n",
        "\n",
        "Альтернативно, вы можете отважиться сделать что-то своё, на данных из интернета или своих собственных. В вашей задаче обязательно должно быть _оправданное_ использование трансформеров. Использовать ML чтобы переводить часовые пояса - плохой план.\n",
        "\n",
        "Achtung: трансформеры круты, но не всемогущи. Далеко не любую задачу можно решить ощутимо лучше рандома. Для калибровки, вот несколько примеров решаемых задач (всё кликабельно):\n",
        "\n",
        "\n",
        "<details><summary> - <b>[medium]</b> <u>Сгенерировать youtube-комментарии по _ссылке_ на видео</u></summary>\n",
        "    Всё просто, юзер постит ссылку на видео - вы его комментируете. Можно заранее обусловиться что видео только на английском или на русском. Нужно сочинить _несколько_ комментариев. Kudos если вместе с основным комментарием вы порождаете юзернеймы и-или ответы на него.\n",
        "    \n",
        "    Датасет для файнтюна можно [взять с kaggle](https://www.kaggle.com/tanmay111/youtube-comments-sentiment-analysis/data?select=UScomments.csv) или [собрать самостоятельно](https://towardsdatascience.com/how-to-build-your-own-dataset-of-youtube-comments-39a1e57aade).\n",
        "    \n",
        "    В качестве основной модели можно использовать [GPT-2 large](https://huggingface.co/gpt2-large). Вот как её файнтюнить: https://tinyurl.com/gpt2-finetune-colab . Если хотите больше - можно взять что-то из творчества https://huggingface.co/EleutherAI . Например, вот [тут](https://tinyurl.com/gpt-j-8bit) есть пример как файнтюнить GPT-J-6B (в 8 раз больше gpt2-large). Однако, этим стоит заниматься уже после того, как у вас заработал базовый сценарий с GPT2-large или даже base.\n",
        "    \n",
        "    В итоговом сервисе можно дать пользователю вариировать параметры генерации: температура или top-p, если сэмплинг; beam size и length penalty, если beam search; сколько комментариев сгенерировать, etc. Отдельный респект если ваш код будет выводить комментарий по одному слову, прямо в процессе генерёжки - чтобы пользователь не ждал пока вы настругаете абзац целиком.\n",
        "</details>\n",
        "\n",
        "<details><summary> - <b>[medium]</b> <u>Предсказать зарплату по профилю (симулятор Дудя).</u></summary>\n",
        "    Note: <details> <summary>Причём тут Дудь?</summary> <img src=https://www.meme-arsenal.com/memes/6dd85f126bbab4f9774ced71ffadbcb3.jpg> </details>\n",
        "    \n",
        "    Главная сложность задачи - достать хорошие данные. Если хороших данных не случилось - можно и трешовые :) Задание всё-таки про технологии а не про продукт. Для начала можно взять подмножество фичей [отсюда](https://www.kaggle.com/c/job-salary-prediction/data), которые вы можете восстановить из профиля linkedin - название профессии и компании. Название компании лучше заменить на фичи из открытых источников: сфера деятельности, размер, етц.\n",
        "    \n",
        "    А дальше файнтюним на этом BERT / T5 и радуемся. Ну или хотя бы смеёмся.\n",
        "</details>\n",
        "\n",
        "\n",
        "<details><summary> - <b>[hard]</b> <u>Мнения с географической окраской.</u></summary>\n",
        "    \n",
        "    Сервис который принимает на вход тему (хэштег или ключевую фразу) и рисует карту мира, где в каждом регионе показано, с какой эмоциональной окраской о ней высказываются в социальных сетях. В качестве социальной сети можно взять VK/twitter, в случая VK ожидается детализация не по странам, а по городам стран бывшего СССР.\n",
        "    \n",
        "    В минимальном варианте достаточно определять тональность твита в режиме \"позитивно-негативно\", зафайнтюнив условный BERT/T5 на одном из десятков {vk/twitter} sentiment classification датасетах. Географическую привязку можно получить из профиля пользователя. А дальше осталось собрать данные по странам и регионам.\n",
        "\n",
        "</details>\n",
        "\n",
        "\n",
        "<details><summary> - <b>[very hard]</b> <u>Найти статью википедии по фото предмета статьи</u></summary>\n",
        "\n",
        "    Чтобы можно было сфотать какую-нибудь неведомую чешуйню на телефон и получить сумму человеческих знаний о ней в форме вики-статьи.\n",
        "    \n",
        "    В качестве функции потерь можно использовать contrastive loss. Этот лосс неплохо описан в статье [CLIP](https://arxiv.org/abs/2103.00020). Вместо обучения с нуля предлагается взять, собственно, CLIP (text transformer + image transformer) отсюда: https://huggingface.co/docs/transformers/model_doc/clip. Модель будет сопоставлять каждой статьи и\n",
        "    \n",
        "    Данные для этого квеста можно собрать через API википедии: вики-статьи о предметах обычно содержит фото этого объекта и, собственно, текст статьи. Советуем собрать как минимум 10^4 пар картинка-статья. Картинки советуем дополнительно аугментировать как минимум стандартными картиночными аугами, как максимум - поиском похожих картинок в интернете / imagenet-е по тому же CLIP image encoder-у, но с исходными весами.\n",
        "    \n",
        "    На время отладки интерфейса рекомендуем ограничиться небольшим списком статьей: условно, кошечки, собачки, птички, гаечные ключи, машины. Как станет понятно что оно работает \"на кошках\", можно расширить этот список до \"всех статей таких-то категорий\". Эмбединги статей лучше предпосчитать в файл. Если долго их перебирать - можно (но необязательно) воспользоваться быстрым поиском соседей, e.g. [faiss](https://github.com/facebookresearch/faiss) HNSW.\n",
        "</details>\n",
        "\n",
        "\n",
        "## __Как научить классификатор статей?__\n",
        "\n",
        "Данные для классификации статей можно скачать, например, [отсюда](https://www.kaggle.com/neelshah18/arxivdataset/). В этих данных есть заголовок и abstract статьи, а ещё поле __\"tag\"__: тематика статьи [по таксономии arxiv.org](https://arxiv.org/category_taxonomy). Вы можете расширить выборку, добавив в неё статьи за 2019-н.в. годы. Для этого можно [использовать arxiv API](https://github.com/lukasschwab/arxiv.py), самостоятельно распарсить arxiv с помощью [beautifulsoup](https://pypi.org/project/beautifulsoup4/), или поискать другие датасеты на kaggle, huggingface, etc.\n",
        "\n",
        "Когда данные собраны (и аккуратно нарезаны на train/test), можно что-нибудь и обучить. Мы советуем использовать для этого библиотеку `transformers`. Советуем, но не заставляем: если хочется, можно взять [fairseq roberta](https://github.com/pytorch/fairseq/blob/main/examples/roberta), [google t5](https://github.com/google-research/text-to-text-transfer-transformer) или даже написать всё с нуля.\n",
        "\n",
        "Мы разбирали transformers на [семинаре](https://lk.yandexdataschool.ru/courses/2025-spring/7.1332-machine-learning-2/classes/13138/), за любой дополнительной информацией - смотрите [документации HF](https://huggingface.co/docs).\n",
        "\n",
        "Начать лучше с простой модели, такой как [`distilbert-base-cased`](https://huggingface.co/distilbert-base-cased). Когда вы будете понимать, какие значения accuracy ожидать от базовой модели, можно поискать что-то получше. Два очевидных направления улучшения: (1) сильнее модель T5 или deberta v3, или (2) близкие данные, например взять модель которую предобучили на том же arxiv. И то и другое удобно [искать здесь](https://huggingface.co/models).\n",
        "\n",
        "## __Научили, и что теперь?__\n",
        "\n",
        "А теперь нужно сделать так, чтобы ваша обученная модель отвечала на запросы в интернете. Как и на прошлом этапе, вы можете сделать это кучей разных способов: от простого [streamlit](https://streamlit.io/) / [gradio](https://gradio.app/), минуя [TorchServe](https://pytorch.org/serve/) с [Triton/TensorRT](https://developer.nvidia.com/nvidia-triton-inference-server), и заканчивая экспортом модели в javascript с помощью [TensorFlow.js](https://www.tensorflow.org/js/tutorials) / [ONNX.js](https://github.com/elliotwaite/pytorch-to-javascript-with-onnx-js).\n",
        "\n",
        "На [семинаре](https://lk.yandexdataschool.ru/courses/2025-spring/7.1332-machine-learning-2/classes/13138/) мы разбирали основные вещи про то как работает streamlit и как сделать простое приложение с его помощью.\n",
        "\n",
        "Общая идея streamlit: вы [описываете](https://docs.streamlit.io/library/get-started/create-an-app) внешний вид приложения на питоне с помощью примитивов (кнопки, поля, любой html) -- а потом этот код выполняется на сервере и обслуживает каждого пользователя в отдельном процессе.\n",
        "\n",
        "__Для отладки__ можно запустить приложение локально, открыв консоль рядом с app.py:\n",
        "* `pip install streamlit`\n",
        "* `streamlit run app.py --server.port 8080`\n",
        "* открыть в браузере localhost:8080, если он не открылся автоматически\n",
        "\n",
        "\n",
        "## __Deployment time!__\n",
        "\n",
        "В этот раз вам нужно не просто написать код, __но и поднять ваше приложение с доступом из интернета__. И да, вы угадали, это можно сделать несколькими способами: [HuggingFace spaces](https://huggingface.co/spaces) (данный способ разбирали на [семинаре](https://lk.yandexdataschool.ru/courses/2025-spring/7.1332-machine-learning-2/classes/13138/)), [Streamlit Cloud](https://streamlit.io/cloud), а ещё вы можете купить или арендовать свой собственный сервер и захоститься там.\n",
        "\n",
        "Проще всего захостить на HF spaces, для этого вам нужно [зарегистрироваться](https://huggingface.co/join) и найти [меню создания нового приложения](https://huggingface.co/new-space). Название и лицензию можно выбрать на своё усмотрение, главное чтобы Space SDK был Streamlit, а доступ - public.\n",
        "\n",
        "Как создали - можно редактировать ваше приложение прямо на сайте, для этого откройте приложение и перейдите в Files and versions, и там в правом углу добавьте нужные файлы.\n",
        "\n",
        "На минималках вам потребуется 2 файла:\n",
        "- `app.py`, о котором мы говорили выше\n",
        "- `requirements.txt`, где вы укажете нужные вам библиотеки\n",
        "\n",
        "Вы можете разместить там же веса вашей обученной модели, любые необходимые данные, дополнительные файлы, ...\n",
        "\n",
        "После каждого изменения файлов, ваше приложение соберётся (обычно 1-5 минут) и будет доступно уже во вкладке App. Ну или не соберётся и покажет вам, где оно сломалось. И вуаля, теперь у вас есть ссылка, которую можно показать ~друзьям~ ассистентам курса и кому угодно в интернете.\n",
        "\n",
        "__Удобная работа с кодом.__ Пока у вас 2 файла, их легко редактивровать прямо в интерфейсе HF spaces. Если же у вас дюжина файлов, вам может быть удобнее редактировать их в любимом vscode/pycharm/.../emacs. Чтобы это не вызывало мучений, можно пользоваться HF spaces как git репозиторием ([подробности тут](https://huggingface.co/docs/hub/spaces#manage-app-with-github-actions)).\n",
        "\n",
        "## __Что нужно сдать__\n",
        "\n",
        "Вы сдаёте проект, который будет проверяться вручную, то что ожидается от каждого проекта:\n",
        "- Текстовое сопровождение вашего конкретного проекта в любом удобно читаемом формате (pdf, html, текст в lk, ...) - что за задачу вы решали, где/как брали данные, какие использовали модели, какие проводили эксперименты, ...\n",
        "- Ссылка на веб интерфейс, где можно протестировать демо вашего проекта - обязательно проверяйте что работает не только у вас (с другого устройства и из под incognito режима)\n",
        "- Код обучения вашей модели (желательно ipynb с заполненными ячейками и не стёртыми выходами, переведённый в pdf / html), но если вы обучали не в ноутбуке, то сдавайте код в виде файла / архива файлов / git ссылки с readme.md описанием того как именно проходило обучение с помощью этого кода.\n",
        "\n",
        "## __Оценка__\n",
        "\n",
        "Мы будем оценивать проект целиком, включая идею и реализацию. Максимум за проект можно получить 10 баллов, но мы оставляем ещё до 5 баллов, которые можем выдать как бонусные за особенно интересные и качественно реализованные проекты.\n",
        "\n",
        "### __Тонкие места, за которые могут быть снижения баллов:__\n",
        "\n",
        "__1. Скорость работы.__\n",
        "\n",
        "По умолчанию, streamlit будет выполняет весь ваш код на каждое действие пользователя. То есть всякий раз, когда пользователь меняет что-то в тексте, оно будет заново загружать модель. Чтобы исправить это безобразие, вы можете закэшировать подготовленную модель в `@st.cache`. Подробности в [семинаре](https://lk.yandexdataschool.ru/courses/2025-spring/7.1332-machine-learning-2/classes/13138/), а также [читайте тут](https://docs.streamlit.io/library/advanced-features/caching).\n",
        "\n",
        "__Как будет оцениваться:__\n",
        "\n",
        "Вы не обязаны пользоваться кэшированием, но ваше приложение не должно неоправдано тормозить дольше, чем на 3 секунды. \"Оправданые\" тормоза это те, которые вы явно оправдали текстом в ЛМС :)\n",
        "\n",
        "-----\n",
        "\n",
        "__2. Понятный фронтенд.__\n",
        "\n",
        "Наколеночный графический интерфейс с семинара - пример того, как скорее не надо делать интерфейс приложения. Как надо - сложный вопрос, причём настолько сложный, что есть даже [Школа Разработки Интерфейсов](https://academy.yandex.ru/schools/frontend). Но для начала:\n",
        "\n",
        "- Выводить нужно человекочитаемый текст, а не просто JSON с индексами и метаданными.\n",
        "- Пользователю должно быть понятно, куда и какие данные вводить. Пустые текстовые поля в вакууме - плохой тон.\n",
        "- Сервис не должен падать с не_отловленными ошибками. Даже если пользователь введёт неправильные/пустые данные, нужно это обработать и написать, где произошла ошибка.\n",
        "\n",
        "__Как будет оцениваться:__\n",
        "\n",
        "Для полного балла достаточно соблюсти эти три правила и специально не стрелять себе в ногу.\n",
        "\n",
        "-----\n",
        "\n",
        "__3. Код обучения и инференса.__\n",
        "\n",
        "Сдавая проект мы будем также получать от вас код проекта (как обучения ваших моделей, так и код веб интерфейса).\n",
        "\n",
        "__Как будет оцениваться:__\n",
        "\n",
        "Код не будет отдельно проверяться как часть задания, поэтому пишите как хотите, однако - в спорных ситуациях мы оставляем за собой право проверить ваш код, за чем могут последовать потенциальные снижения баллов при любых нарушениях.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2025-04-08T16:38:04.503049Z",
          "iopub.status.busy": "2025-04-08T16:38:04.502631Z",
          "iopub.status.idle": "2025-04-08T16:38:09.781191Z",
          "shell.execute_reply": "2025-04-08T16:38:09.779839Z",
          "shell.execute_reply.started": "2025-04-08T16:38:04.503025Z"
        },
        "id": "OKn_gfwpS5EY",
        "tags": []
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.metrics import precision_score\n",
        "from sklearn.utils.class_weight import compute_class_weight\n",
        "import torch\n",
        "from transformers import DistilBertTokenizer, DistilBertForSequenceClassification, Trainer, TrainingArguments"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ybBRCfqoTX5v"
      },
      "source": [
        "## Data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2025-04-08T16:38:09.783193Z",
          "iopub.status.busy": "2025-04-08T16:38:09.782539Z",
          "iopub.status.idle": "2025-04-08T16:38:09.786653Z",
          "shell.execute_reply": "2025-04-08T16:38:09.786050Z",
          "shell.execute_reply.started": "2025-04-08T16:38:09.783167Z"
        },
        "id": "WelaLk5zTFvP",
        "tags": []
      },
      "outputs": [],
      "source": [
        "def extract_main_category(tag):\n",
        "    tags = eval(tag)\n",
        "    first_tag = tags[0]['term']\n",
        "    return first_tag.split('.')[0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2025-04-08T16:38:09.787781Z",
          "iopub.status.busy": "2025-04-08T16:38:09.787522Z",
          "iopub.status.idle": "2025-04-08T16:38:11.161895Z",
          "shell.execute_reply": "2025-04-08T16:38:11.161064Z",
          "shell.execute_reply.started": "2025-04-08T16:38:09.787761Z"
        },
        "id": "jxQ7reE0S7l4",
        "tags": []
      },
      "outputs": [],
      "source": [
        "df = pd.read_parquet(\"data.parquet\")\n",
        "df = df[['title', 'summary', 'tag']]\n",
        "df['tag'] = df['tag'].apply(extract_main_category)\n",
        "df['text'] = df['title'] + ' ' + df['summary']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aUAe90XHgTGg"
      },
      "source": [
        "Посмотрю на распределение таргетов"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 743
        },
        "execution": {
          "iopub.execute_input": "2025-04-08T16:38:11.163765Z",
          "iopub.status.busy": "2025-04-08T16:38:11.163464Z",
          "iopub.status.idle": "2025-04-08T16:38:11.175870Z",
          "shell.execute_reply": "2025-04-08T16:38:11.175327Z",
          "shell.execute_reply.started": "2025-04-08T16:38:11.163741Z"
        },
        "id": "rQxazH-cz89W",
        "outputId": "6fa953a4-0283-4c07-bd2b-03893909a647",
        "tags": []
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tag\n",
              "cs          34597\n",
              "stat         4782\n",
              "math          612\n",
              "q-bio         320\n",
              "physics       216\n",
              "cmp-lg        110\n",
              "eess           75\n",
              "quant-ph       66\n",
              "cond-mat       65\n",
              "astro-ph       59\n",
              "nlin           47\n",
              "q-fin          30\n",
              "econ            5\n",
              "gr-qc           4\n",
              "hep-ex          4\n",
              "hep-ph          2\n",
              "adap-org        2\n",
              "hep-lat         2\n",
              "hep-th          1\n",
              "nucl-th         1\n",
              "Name: count, dtype: int64"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>count</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>tag</th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>cs</th>\n",
              "      <td>34597</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>stat</th>\n",
              "      <td>4782</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>math</th>\n",
              "      <td>612</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>q-bio</th>\n",
              "      <td>320</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>physics</th>\n",
              "      <td>216</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>cmp-lg</th>\n",
              "      <td>110</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>eess</th>\n",
              "      <td>75</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>quant-ph</th>\n",
              "      <td>66</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>cond-mat</th>\n",
              "      <td>65</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>astro-ph</th>\n",
              "      <td>59</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>nlin</th>\n",
              "      <td>47</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>q-fin</th>\n",
              "      <td>30</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>econ</th>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>gr-qc</th>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>hep-ex</th>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>hep-ph</th>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>adap-org</th>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>hep-lat</th>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>hep-th</th>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>nucl-th</th>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div><br><label><b>dtype:</b> int64</label>"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ],
      "source": [
        "df['tag'].value_counts()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oFOZcbOX3G4m"
      },
      "source": [
        "Есть ощутимый дисбаланс классов - нужно будет учесть это при обучении (в лоссе). На этом этапе объединю все редкие классы в один"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2025-04-08T16:38:11.177011Z",
          "iopub.status.busy": "2025-04-08T16:38:11.176731Z",
          "iopub.status.idle": "2025-04-08T16:38:11.185683Z",
          "shell.execute_reply": "2025-04-08T16:38:11.185103Z",
          "shell.execute_reply.started": "2025-04-08T16:38:11.176989Z"
        },
        "id": "wIxUOEOKz_0U",
        "tags": []
      },
      "outputs": [],
      "source": [
        "class_counts = df['tag'].value_counts()\n",
        "classes_to_merge = class_counts[class_counts < 10].index\n",
        "df.loc[df['tag'].isin(classes_to_merge), 'tag'] = ':)'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 523
        },
        "execution": {
          "iopub.execute_input": "2025-04-08T16:38:11.187086Z",
          "iopub.status.busy": "2025-04-08T16:38:11.186627Z",
          "iopub.status.idle": "2025-04-08T16:38:11.195550Z",
          "shell.execute_reply": "2025-04-08T16:38:11.194989Z",
          "shell.execute_reply.started": "2025-04-08T16:38:11.187054Z"
        },
        "id": "wn8LBoi40B5e",
        "outputId": "68cee6bb-6260-44cb-a73b-c5e51227b50e",
        "tags": []
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tag\n",
              "cs          34597\n",
              "stat         4782\n",
              "math          612\n",
              "q-bio         320\n",
              "physics       216\n",
              "cmp-lg        110\n",
              "eess           75\n",
              "quant-ph       66\n",
              "cond-mat       65\n",
              "astro-ph       59\n",
              "nlin           47\n",
              "q-fin          30\n",
              ":)             21\n",
              "Name: count, dtype: int64"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>count</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>tag</th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>cs</th>\n",
              "      <td>34597</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>stat</th>\n",
              "      <td>4782</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>math</th>\n",
              "      <td>612</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>q-bio</th>\n",
              "      <td>320</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>physics</th>\n",
              "      <td>216</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>cmp-lg</th>\n",
              "      <td>110</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>eess</th>\n",
              "      <td>75</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>quant-ph</th>\n",
              "      <td>66</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>cond-mat</th>\n",
              "      <td>65</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>astro-ph</th>\n",
              "      <td>59</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>nlin</th>\n",
              "      <td>47</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>q-fin</th>\n",
              "      <td>30</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>:)</th>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div><br><label><b>dtype:</b> int64</label>"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ],
      "source": [
        "df['tag'].value_counts()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2025-04-08T16:49:10.200430Z",
          "iopub.status.busy": "2025-04-08T16:49:10.200018Z",
          "iopub.status.idle": "2025-04-08T16:49:10.206017Z",
          "shell.execute_reply": "2025-04-08T16:49:10.205358Z",
          "shell.execute_reply.started": "2025-04-08T16:49:10.200406Z"
        },
        "tags": [],
        "id": "ZpOCCcGVgTGj"
      },
      "outputs": [],
      "source": [
        "num_classes = len(df['tag'].unique())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y-8KfDg5gTGk"
      },
      "source": [
        "Рука не поднимается откусывать часть на тест, а на валидацию - тем более. Обучаться буду просто по эпохам, тест сделаю поменьше и лучше руками отвалидирую итоговую модель"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2025-04-08T16:38:11.196783Z",
          "iopub.status.busy": "2025-04-08T16:38:11.196508Z",
          "iopub.status.idle": "2025-04-08T16:38:11.246887Z",
          "shell.execute_reply": "2025-04-08T16:38:11.246341Z",
          "shell.execute_reply.started": "2025-04-08T16:38:11.196762Z"
        },
        "id": "MHvx9TbX1AHe",
        "tags": []
      },
      "outputs": [],
      "source": [
        "train_df, test_df = train_test_split(df, test_size=0.15, random_state=26, stratify=df['tag'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zEXYZ5dPTTOh"
      },
      "source": [
        "## Baseline: TF-IDF + LogReg (классика)\n",
        "\n",
        "также не забываю учесть дисбаланс классов"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2025-04-08T16:58:27.215791Z",
          "iopub.status.busy": "2025-04-08T16:58:27.215328Z",
          "iopub.status.idle": "2025-04-08T16:58:31.103917Z",
          "shell.execute_reply": "2025-04-08T16:58:31.102885Z",
          "shell.execute_reply.started": "2025-04-08T16:58:27.215765Z"
        },
        "tags": [],
        "id": "ocREmLglgTGk"
      },
      "outputs": [],
      "source": [
        "vectorizer = TfidfVectorizer(max_features=1000)\n",
        "\n",
        "X_train, y_train = vectorizer.fit_transform(train_df['text']), train_df['tag']\n",
        "X_test, y_test = vectorizer.transform(test_df['text']), test_df['tag']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2025-04-08T16:41:20.930199Z",
          "iopub.status.busy": "2025-04-08T16:41:20.929875Z",
          "iopub.status.idle": "2025-04-08T16:43:34.374654Z",
          "shell.execute_reply": "2025-04-08T16:43:34.373963Z",
          "shell.execute_reply.started": "2025-04-08T16:41:20.930177Z"
        },
        "id": "hhc1p0RhfJ9Y",
        "tags": [],
        "outputId": "30988d23-e5a0-4210-841a-666892ab1bde"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/opt/app-root/lib64/python3.8/site-packages/sklearn/linear_model/_sag.py:350: ConvergenceWarning: The max_iter was reached which means the coef_ did not converge\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<style>#sk-container-id-3 {color: black;background-color: white;}#sk-container-id-3 pre{padding: 0;}#sk-container-id-3 div.sk-toggleable {background-color: white;}#sk-container-id-3 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-3 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-3 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-3 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-3 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-3 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-3 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-3 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-3 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-3 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-3 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-3 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-3 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-3 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-3 div.sk-item {position: relative;z-index: 1;}#sk-container-id-3 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-3 div.sk-item::before, #sk-container-id-3 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-3 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-3 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-3 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-3 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-3 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-3 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-3 div.sk-label-container {text-align: center;}#sk-container-id-3 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-3 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-3\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LogisticRegression(class_weight=&#x27;balanced&#x27;, penalty=&#x27;l1&#x27;, solver=&#x27;saga&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" checked><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression(class_weight=&#x27;balanced&#x27;, penalty=&#x27;l1&#x27;, solver=&#x27;saga&#x27;)</pre></div></div></div></div></div>"
            ],
            "text/plain": [
              "LogisticRegression(class_weight='balanced', penalty='l1', solver='saga')"
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "lr_model = LogisticRegression(penalty='l1', solver='saga', class_weight='balanced')\n",
        "lr_model.fit(X_train, y_train)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "axe7HeJJgTGl"
      },
      "source": [
        "Для удобства и подсчёта метрики заведу маппинги и веса классов"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2025-04-08T17:00:24.569272Z",
          "iopub.status.busy": "2025-04-08T17:00:24.568849Z",
          "iopub.status.idle": "2025-04-08T17:00:24.616599Z",
          "shell.execute_reply": "2025-04-08T17:00:24.615924Z",
          "shell.execute_reply.started": "2025-04-08T17:00:24.569244Z"
        },
        "tags": [],
        "id": "Sd5j9kYcgTGl"
      },
      "outputs": [],
      "source": [
        "class_to_idx_lr = {lr_model.classes_[i]: i for i in range(num_classes)}\n",
        "idx_to_class_lr = {i: lr_model.classes_[i] for i in range(num_classes)}\n",
        "\n",
        "weights = [1 / df[df['tag'] == idx_to_class_lr[i]].shape[0] for i in range(num_classes)]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2025-04-08T17:01:28.800179Z",
          "iopub.status.busy": "2025-04-08T17:01:28.799751Z",
          "iopub.status.idle": "2025-04-08T17:01:28.805213Z",
          "shell.execute_reply": "2025-04-08T17:01:28.804557Z",
          "shell.execute_reply.started": "2025-04-08T17:01:28.800154Z"
        },
        "tags": [],
        "id": "bnjB9e8XgTGm"
      },
      "outputs": [],
      "source": [
        "def metric(y_true, y_pred, labels):\n",
        "    positive_answers_per_class_amount = [0] * num_classes  # для каждого класса кол-во попаданий таргета в топ-95%\n",
        "    for i in range(len(y_true)):\n",
        "        probs = y_pred[i]\n",
        "        sorted_indices = np.argsort(probs)[::-1]\n",
        "        cumulative = 0\n",
        "        top_tags = []\n",
        "        for idx in sorted_indices:\n",
        "            cumulative += probs[idx]\n",
        "            top_tags.append(idx)\n",
        "            if cumulative >= 0.95:\n",
        "                break\n",
        "        top_tags = [labels[j] for j in top_tags]\n",
        "        true_tag = y_true.iloc[i]\n",
        "        if true_tag in top_tags:\n",
        "            positive_answers_per_class_amount[class_to_idx_lr[true_tag]] += 1\n",
        "\n",
        "    precision = np.dot(positive_answers_per_class_amount, weights)\n",
        "\n",
        "    return precision"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2025-04-08T17:01:28.925889Z",
          "iopub.status.busy": "2025-04-08T17:01:28.925532Z",
          "iopub.status.idle": "2025-04-08T17:01:28.986010Z",
          "shell.execute_reply": "2025-04-08T17:01:28.985402Z",
          "shell.execute_reply.started": "2025-04-08T17:01:28.925867Z"
        },
        "tags": [],
        "id": "YDKSjq_kgTGm",
        "outputId": "90de8edf-3649-46b3-cfc2-2f3bfd2a93f6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "TF-IDF + LogReg balanced precision@95%: 0.749\n"
          ]
        }
      ],
      "source": [
        "preds_lr = lr_model.predict_proba(X_test)\n",
        "print(f\"TF-IDF + LogReg balanced precision@95%: {metric(test_df['tag'], preds_lr, lr_model.classes_):.3f}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DLSO0k3qUttY"
      },
      "source": [
        "## Distilbert\n",
        "Здесь использую взвешенный по классам лосс и добавлю щепотку переобучения, считаю после каждой эпохи лосс на тестовом датасете"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 376,
          "referenced_widgets": [
            "a84ec850e42a45b0a4a7a87800ede268",
            "c12508bddd1543449f834d41b413426d",
            "4291c1f1a11e4a48878affcc234faa87",
            "a6c07eb703c94b25a41306cafb364c31",
            "b622bb5aaf6447aa87be6907ce2b2d4e",
            "8609ab94160e491bb994ff987b2d8fbc",
            "73b156ed04554f2a81ff6a528f1c97c0",
            "a3bc75e8cbfd4630a83a6088f75239ac",
            "615d6fd52b3f43e1bbc45eb53bfe0ad7",
            "de943c53880f4f3ba77bd594a4ad87bf",
            "4e631f3e10d24699bb51c6f8f9843a5d",
            "698488c9c8854f7dbebfea7ab2d7c291",
            "f5a7b6f95da54c90ac8860be963e52bd",
            "90c70b258b24430a906c88ebc64b5231",
            "61a0f825e6344eff8ba802fa80103ccb",
            "fb7c892151c049328d2d9fedcb6f41b7",
            "40338ce6822b427799638dcd30b6e141",
            "0803738966ad429da69c32dbedbdbc03",
            "11379b7d536d4a948fe47751de456f9b",
            "af9391c4b1bc439a8d4ff0276be5d315",
            "bec5c7a1bae0480f8a49d3f56b2506eb",
            "d13b6193d01c4307b9b3d89be7e0b8cf",
            "4e229143da894ac7be8ddff37144bec8",
            "033a5fcee80a46619f2dada013a96f59",
            "b7d0fa89b6cc4e3a9f7550bfc7b94dfb",
            "7a77efa8b96c406cbd0ca16b457f048b",
            "2cfc6e9d88244aab9def1cb6764e7e0d",
            "3421d4ff74214c0081f77aa092ed3f7b",
            "d989e092a6344bfa8246f8bf846b54b1",
            "69ec8d01ed27473c8f813c859b9946a8",
            "2828fbc07c90495380412cdafd66a265",
            "14d000a6673f4a83b1cfea274512ccef",
            "30a26a11d9724fd2bb90a8ac3a80b64e",
            "c2f4aa4074bd44c89d5a30ca672fa8f9",
            "24f8549ca034493392fbf35ce6b5d3b2",
            "bd3f9196c76e49ee967f30bae35c8b18",
            "a27a26bbf9fc4f688716bb2aed879a12",
            "0792befbe6c743f69c88408498eafc32",
            "c97130a925a44468b22d0c6ab3cac6e9",
            "b956c28f53434fad9fcfdb65a7c8a303",
            "eefd0b1b319a4e75b0ac3a75fb4e7f38",
            "0d6793013fa2469191d109529bc9223d",
            "8d89cb74b480405688b10b6578fea4bd",
            "1ce96b9a31e342edaa9c40304e65b73b",
            "c4c34fd4e65b44ca8837c359e2adfef5",
            "be28f610644e4902ad22ba61c8e8b294",
            "a82c407c525f493cae6caf2ecab38eec",
            "9a616045cdb14b1eb4cc7987ba7a1ebe",
            "f5ce00f8e11c4834970d38bb7a74bff7",
            "10e1bbfa8acb454cbe44f57232af0dcc",
            "57f70f785e3148dda70d77386ea98c61",
            "555bb9a9df7f4abe8f811b9ba269565f",
            "5f93e9d3639a41e08f132fc6d9612b8d",
            "e2842507f92f4f4d9b44cd8eee193f52",
            "c7d47c9fa5bc4b3ba9ddd9da9fa57550"
          ]
        },
        "execution": {
          "iopub.execute_input": "2025-04-08T17:01:37.328625Z",
          "iopub.status.busy": "2025-04-08T17:01:37.328111Z",
          "iopub.status.idle": "2025-04-08T17:01:37.566858Z",
          "shell.execute_reply": "2025-04-08T17:01:37.565994Z",
          "shell.execute_reply.started": "2025-04-08T17:01:37.328601Z"
        },
        "id": "HIiBGC7qPXRx",
        "outputId": "c35255c3-3834-497a-97a6-435626e26303",
        "tags": []
      },
      "outputs": [
        {
          "metadata": {
            "tags": null
          },
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.11/dist-packages/huggingface_hub/utils/_auth.py:94: UserWarning: \n",
            "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
            "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
            "You will be able to reuse this secret in all of your notebooks.\n",
            "Please note that authentication is recommended but still optional to access public models or datasets.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "a84ec850e42a45b0a4a7a87800ede268",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "tokenizer_config.json:   0%|          | 0.00/49.0 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "698488c9c8854f7dbebfea7ab2d7c291",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "vocab.txt:   0%|          | 0.00/213k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "4e229143da894ac7be8ddff37144bec8",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "tokenizer.json:   0%|          | 0.00/436k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "c2f4aa4074bd44c89d5a30ca672fa8f9",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "config.json:   0%|          | 0.00/465 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "metadata": {
            "tags": null
          },
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`\n",
            "WARNING:huggingface_hub.file_download:Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "c4c34fd4e65b44ca8837c359e2adfef5",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "model.safetensors:   0%|          | 0.00/263M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "metadata": {
            "tags": null
          },
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-cased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ]
        }
      ],
      "source": [
        "tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased')\n",
        "model = DistilBertForSequenceClassification.from_pretrained(\n",
        "    'distilbert-base-cased',\n",
        "    num_labels=len(df['tag'].unique()),\n",
        "    id2label={i: tag for i, tag in enumerate(df['tag'].unique())},\n",
        "    label2id={tag: i for i, tag in enumerate(df['tag'].unique())}\n",
        ")\n",
        "\n",
        "train_encodings = tokenizer(\n",
        "    train_df['text'].tolist(),\n",
        "    truncation=True,\n",
        "    padding=True,\n",
        "    max_length=512\n",
        ")\n",
        "test_encodings = tokenizer(\n",
        "    test_df['text'].tolist(),\n",
        "    truncation=True,\n",
        "    padding=True,\n",
        "    max_length=512\n",
        ")\n",
        "\n",
        "class Dataset(torch.utils.data.Dataset):\n",
        "    def __init__(self, encodings, labels):\n",
        "        self.encodings = encodings\n",
        "        self.labels = labels\n",
        "\n",
        "    def __getitem__(self, idx):\n",
        "        item = {k: torch.tensor(v[idx]) for k, v in self.encodings.items()}\n",
        "        item['labels'] = torch.tensor(self.labels[idx])\n",
        "        return item\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.labels)\n",
        "\n",
        "train_dataset = Dataset(train_encodings, train_df['tag'].map(lambda x: model.config.label2id[x]).values)\n",
        "test_dataset = Dataset(test_encodings, test_df['tag'].map(lambda x: model.config.label2id[x]).values)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 545
        },
        "id": "c67QjfrxVjZ8",
        "outputId": "ed38d123-5772-46d6-e3ab-87eedfcbaf72"
      },
      "outputs": [
        {
          "metadata": {
            "tags": null
          },
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m The `run_name` is currently set to the same value as `TrainingArguments.output_dir`. If this was not intended, please specify a different run name by setting the `TrainingArguments.run_name` parameter.\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: Using wandb-core as the SDK backend.  Please refer to https://wandb.me/wandb-core for more information.\n"
          ]
        },
        {
          "data": {
            "application/javascript": [
              "\n",
              "        window._wandbApiKey = new Promise((resolve, reject) => {\n",
              "            function loadScript(url) {\n",
              "            return new Promise(function(resolve, reject) {\n",
              "                let newScript = document.createElement(\"script\");\n",
              "                newScript.onerror = reject;\n",
              "                newScript.onload = resolve;\n",
              "                document.body.appendChild(newScript);\n",
              "                newScript.src = url;\n",
              "            });\n",
              "            }\n",
              "            loadScript(\"https://cdn.jsdelivr.net/npm/postmate/build/postmate.min.js\").then(() => {\n",
              "            const iframe = document.createElement('iframe')\n",
              "            iframe.style.cssText = \"width:0;height:0;border:none\"\n",
              "            document.body.appendChild(iframe)\n",
              "            const handshake = new Postmate({\n",
              "                container: iframe,\n",
              "                url: 'https://wandb.ai/authorize'\n",
              "            });\n",
              "            const timeout = setTimeout(() => reject(\"Couldn't auto authenticate\"), 5000)\n",
              "            handshake.then(function(child) {\n",
              "                child.on('authorize', data => {\n",
              "                    clearTimeout(timeout)\n",
              "                    resolve(data)\n",
              "                });\n",
              "            });\n",
              "            })\n",
              "        });\n",
              "    "
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "metadata": {
            "tags": null
          },
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[34m\u001b[1mwandb\u001b[0m: Logging into wandb.ai. (Learn how to deploy a W&B server locally: https://wandb.me/wandb-server)\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: You can find your API key in your browser here: https://wandb.ai/authorize\n",
            "wandb: Paste an API key from your profile and hit enter:\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m If you're specifying your api key in code, ensure this code is not shared publicly.\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m Consider setting the WANDB_API_KEY environment variable, or running `wandb login` from the command line.\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: No netrc file found, creating one.\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: Appending key for api.wandb.ai to your netrc file: /root/.netrc\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: Currently logged in as: \u001b[33mfellafrom26\u001b[0m (\u001b[33mfellafrom26-mipt\u001b[0m) to \u001b[32mhttps://api.wandb.ai\u001b[0m. Use \u001b[1m`wandb login --relogin`\u001b[0m to force relogin\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "Tracking run with wandb version 0.19.9"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "Run data is saved locally in <code>/content/wandb/run-20250408_192101-ksqmp5qp</code>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "Syncing run <strong><a href='https://wandb.ai/fellafrom26-mipt/huggingface/runs/ksqmp5qp' target=\"_blank\">./results</a></strong> to <a href='https://wandb.ai/fellafrom26-mipt/huggingface' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/developer-guide' target=\"_blank\">docs</a>)<br>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              " View project at <a href='https://wandb.ai/fellafrom26-mipt/huggingface' target=\"_blank\">https://wandb.ai/fellafrom26-mipt/huggingface</a>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              " View run at <a href='https://wandb.ai/fellafrom26-mipt/huggingface/runs/ksqmp5qp' target=\"_blank\">https://wandb.ai/fellafrom26-mipt/huggingface/runs/ksqmp5qp</a>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='2066' max='3270' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [2066/3270 50:41 < 29:34, 0.68 it/s, Epoch 1.89/3]\n",
              "    </div>\n",
              "    <table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              " <tr style=\"text-align: left;\">\n",
              "      <th>Epoch</th>\n",
              "      <th>Training Loss</th>\n",
              "      <th>Validation Loss</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>1.650900</td>\n",
              "      <td>1.064278</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table><p>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='3270' max='3270' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [3270/3270 1:22:24, Epoch 3/3]\n",
              "    </div>\n",
              "    <table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              " <tr style=\"text-align: left;\">\n",
              "      <th>Epoch</th>\n",
              "      <th>Training Loss</th>\n",
              "      <th>Validation Loss</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>1.650900</td>\n",
              "      <td>1.064278</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>1.257900</td>\n",
              "      <td>0.991267</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>0.915400</td>\n",
              "      <td>1.005465</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table><p>"
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "labels = train_df['tag'].map(model.config.label2id).values\n",
        "class_weights = compute_class_weight('balanced', classes=np.arange(num_classes), y=labels)\n",
        "class_weights_tensor = torch.tensor(class_weights, dtype=torch.float32)\n",
        "\n",
        "class CustomTrainer(Trainer):\n",
        "    def __init__(self, class_weights, *args, **kwargs):\n",
        "        super().__init__(*args, **kwargs)\n",
        "        self.class_weights = class_weights.to(self.model.device)\n",
        "\n",
        "    def compute_loss(self, model, inputs, return_outputs=False, **kwargs):\n",
        "        labels = inputs.get(\"labels\")\n",
        "        outputs = model(**inputs)\n",
        "        logits = outputs.logits\n",
        "        loss_fct = torch.nn.CrossEntropyLoss(weight=self.class_weights)\n",
        "        loss = loss_fct(logits, labels)\n",
        "        return (loss, outputs) if return_outputs else loss\n",
        "\n",
        "training_args = TrainingArguments(\n",
        "    output_dir='./results',\n",
        "    num_train_epochs=3,\n",
        "    per_device_train_batch_size=32,\n",
        "    eval_strategy='epoch',\n",
        "    logging_steps=500\n",
        ")\n",
        "\n",
        "trainer = CustomTrainer(\n",
        "    class_weights=class_weights_tensor,\n",
        "    model=model,\n",
        "    args=training_args,\n",
        "    train_dataset=train_dataset,\n",
        "    eval_dataset=test_dataset\n",
        ")\n",
        "\n",
        "trainer.train()\n",
        "model.save_pretrained('model/')"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def nn_metric(y_true, y_logits, model):\n",
        "    probs = torch.nn.functional.softmax(torch.tensor(y_logits), dim=-1).numpy()\n",
        "\n",
        "    id2label = model.config.id2label\n",
        "    class_to_idx_bert = {v: k for k, v in model.config.label2id.items()}\n",
        "\n",
        "    weights = [1/df[df['tag'] == cls].shape[0] for cls in class_to_idx_bert.values()]\n",
        "\n",
        "    positive_answers = [0] * len(class_to_idx_bert)\n",
        "\n",
        "    for i in range(len(y_true)):\n",
        "        current_probs = probs[i]\n",
        "        sorted_indices = np.argsort(current_probs)[::-1]\n",
        "\n",
        "        cumulative = 0\n",
        "        top_tags = []\n",
        "        for idx in sorted_indices:\n",
        "            cumulative += current_probs[idx]\n",
        "            top_tags.append(id2label[idx])\n",
        "            if cumulative >= 0.95:\n",
        "                break\n",
        "\n",
        "        true_tag = y_true.iloc[i]\n",
        "        if true_tag in top_tags:\n",
        "            positive_answers[model.config.label2id[true_tag]] += 1\n",
        "\n",
        "    precision = np.dot(positive_answers, weights)\n",
        "    return precision"
      ],
      "metadata": {
        "id": "AyLSGCr9Ku97"
      },
      "execution_count": 35,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "preds = trainer.predict(test_dataset)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "TDRCOtjw1S2y",
        "outputId": "8cea69d7-9db1-4660-d953-24f7a42c892e"
      },
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": []
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "logits = preds.predictions\n",
        "print(f\"DistilBERT precision@95%: {nn_metric(test_df['tag'], logits, model):.3f}\")"
      ],
      "metadata": {
        "id": "8OiPteQ8Ltnh",
        "outputId": "c8242422-a1b6-4499-a780-e6037dc085d7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "DistilBERT precision@95%: 1.441\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "5SaDCpMKL6mi"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.13"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "a84ec850e42a45b0a4a7a87800ede268": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_c12508bddd1543449f834d41b413426d",
              "IPY_MODEL_4291c1f1a11e4a48878affcc234faa87",
              "IPY_MODEL_a6c07eb703c94b25a41306cafb364c31"
            ],
            "layout": "IPY_MODEL_b622bb5aaf6447aa87be6907ce2b2d4e"
          }
        },
        "c12508bddd1543449f834d41b413426d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8609ab94160e491bb994ff987b2d8fbc",
            "placeholder": "​",
            "style": "IPY_MODEL_73b156ed04554f2a81ff6a528f1c97c0",
            "value": "tokenizer_config.json: 100%"
          }
        },
        "4291c1f1a11e4a48878affcc234faa87": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a3bc75e8cbfd4630a83a6088f75239ac",
            "max": 49,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_615d6fd52b3f43e1bbc45eb53bfe0ad7",
            "value": 49
          }
        },
        "a6c07eb703c94b25a41306cafb364c31": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_de943c53880f4f3ba77bd594a4ad87bf",
            "placeholder": "​",
            "style": "IPY_MODEL_4e631f3e10d24699bb51c6f8f9843a5d",
            "value": " 49.0/49.0 [00:00&lt;00:00, 4.86kB/s]"
          }
        },
        "b622bb5aaf6447aa87be6907ce2b2d4e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8609ab94160e491bb994ff987b2d8fbc": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "73b156ed04554f2a81ff6a528f1c97c0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a3bc75e8cbfd4630a83a6088f75239ac": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "615d6fd52b3f43e1bbc45eb53bfe0ad7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "de943c53880f4f3ba77bd594a4ad87bf": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4e631f3e10d24699bb51c6f8f9843a5d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "698488c9c8854f7dbebfea7ab2d7c291": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_f5a7b6f95da54c90ac8860be963e52bd",
              "IPY_MODEL_90c70b258b24430a906c88ebc64b5231",
              "IPY_MODEL_61a0f825e6344eff8ba802fa80103ccb"
            ],
            "layout": "IPY_MODEL_fb7c892151c049328d2d9fedcb6f41b7"
          }
        },
        "f5a7b6f95da54c90ac8860be963e52bd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_40338ce6822b427799638dcd30b6e141",
            "placeholder": "​",
            "style": "IPY_MODEL_0803738966ad429da69c32dbedbdbc03",
            "value": "vocab.txt: 100%"
          }
        },
        "90c70b258b24430a906c88ebc64b5231": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_11379b7d536d4a948fe47751de456f9b",
            "max": 213450,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_af9391c4b1bc439a8d4ff0276be5d315",
            "value": 213450
          }
        },
        "61a0f825e6344eff8ba802fa80103ccb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bec5c7a1bae0480f8a49d3f56b2506eb",
            "placeholder": "​",
            "style": "IPY_MODEL_d13b6193d01c4307b9b3d89be7e0b8cf",
            "value": " 213k/213k [00:00&lt;00:00, 3.43MB/s]"
          }
        },
        "fb7c892151c049328d2d9fedcb6f41b7": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "40338ce6822b427799638dcd30b6e141": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0803738966ad429da69c32dbedbdbc03": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "11379b7d536d4a948fe47751de456f9b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "af9391c4b1bc439a8d4ff0276be5d315": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "bec5c7a1bae0480f8a49d3f56b2506eb": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d13b6193d01c4307b9b3d89be7e0b8cf": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "4e229143da894ac7be8ddff37144bec8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_033a5fcee80a46619f2dada013a96f59",
              "IPY_MODEL_b7d0fa89b6cc4e3a9f7550bfc7b94dfb",
              "IPY_MODEL_7a77efa8b96c406cbd0ca16b457f048b"
            ],
            "layout": "IPY_MODEL_2cfc6e9d88244aab9def1cb6764e7e0d"
          }
        },
        "033a5fcee80a46619f2dada013a96f59": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3421d4ff74214c0081f77aa092ed3f7b",
            "placeholder": "​",
            "style": "IPY_MODEL_d989e092a6344bfa8246f8bf846b54b1",
            "value": "tokenizer.json: 100%"
          }
        },
        "b7d0fa89b6cc4e3a9f7550bfc7b94dfb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_69ec8d01ed27473c8f813c859b9946a8",
            "max": 435797,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_2828fbc07c90495380412cdafd66a265",
            "value": 435797
          }
        },
        "7a77efa8b96c406cbd0ca16b457f048b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_14d000a6673f4a83b1cfea274512ccef",
            "placeholder": "​",
            "style": "IPY_MODEL_30a26a11d9724fd2bb90a8ac3a80b64e",
            "value": " 436k/436k [00:00&lt;00:00, 6.76MB/s]"
          }
        },
        "2cfc6e9d88244aab9def1cb6764e7e0d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3421d4ff74214c0081f77aa092ed3f7b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d989e092a6344bfa8246f8bf846b54b1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "69ec8d01ed27473c8f813c859b9946a8": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2828fbc07c90495380412cdafd66a265": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "14d000a6673f4a83b1cfea274512ccef": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "30a26a11d9724fd2bb90a8ac3a80b64e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c2f4aa4074bd44c89d5a30ca672fa8f9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_24f8549ca034493392fbf35ce6b5d3b2",
              "IPY_MODEL_bd3f9196c76e49ee967f30bae35c8b18",
              "IPY_MODEL_a27a26bbf9fc4f688716bb2aed879a12"
            ],
            "layout": "IPY_MODEL_0792befbe6c743f69c88408498eafc32"
          }
        },
        "24f8549ca034493392fbf35ce6b5d3b2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c97130a925a44468b22d0c6ab3cac6e9",
            "placeholder": "​",
            "style": "IPY_MODEL_b956c28f53434fad9fcfdb65a7c8a303",
            "value": "config.json: 100%"
          }
        },
        "bd3f9196c76e49ee967f30bae35c8b18": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_eefd0b1b319a4e75b0ac3a75fb4e7f38",
            "max": 465,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_0d6793013fa2469191d109529bc9223d",
            "value": 465
          }
        },
        "a27a26bbf9fc4f688716bb2aed879a12": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8d89cb74b480405688b10b6578fea4bd",
            "placeholder": "​",
            "style": "IPY_MODEL_1ce96b9a31e342edaa9c40304e65b73b",
            "value": " 465/465 [00:00&lt;00:00, 30.0kB/s]"
          }
        },
        "0792befbe6c743f69c88408498eafc32": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c97130a925a44468b22d0c6ab3cac6e9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b956c28f53434fad9fcfdb65a7c8a303": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "eefd0b1b319a4e75b0ac3a75fb4e7f38": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0d6793013fa2469191d109529bc9223d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "8d89cb74b480405688b10b6578fea4bd": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1ce96b9a31e342edaa9c40304e65b73b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c4c34fd4e65b44ca8837c359e2adfef5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_be28f610644e4902ad22ba61c8e8b294",
              "IPY_MODEL_a82c407c525f493cae6caf2ecab38eec",
              "IPY_MODEL_9a616045cdb14b1eb4cc7987ba7a1ebe"
            ],
            "layout": "IPY_MODEL_f5ce00f8e11c4834970d38bb7a74bff7"
          }
        },
        "be28f610644e4902ad22ba61c8e8b294": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_10e1bbfa8acb454cbe44f57232af0dcc",
            "placeholder": "​",
            "style": "IPY_MODEL_57f70f785e3148dda70d77386ea98c61",
            "value": "model.safetensors: 100%"
          }
        },
        "a82c407c525f493cae6caf2ecab38eec": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_555bb9a9df7f4abe8f811b9ba269565f",
            "max": 263260784,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_5f93e9d3639a41e08f132fc6d9612b8d",
            "value": 263260784
          }
        },
        "9a616045cdb14b1eb4cc7987ba7a1ebe": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e2842507f92f4f4d9b44cd8eee193f52",
            "placeholder": "​",
            "style": "IPY_MODEL_c7d47c9fa5bc4b3ba9ddd9da9fa57550",
            "value": " 263M/263M [00:01&lt;00:00, 227MB/s]"
          }
        },
        "f5ce00f8e11c4834970d38bb7a74bff7": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "10e1bbfa8acb454cbe44f57232af0dcc": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "57f70f785e3148dda70d77386ea98c61": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "555bb9a9df7f4abe8f811b9ba269565f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5f93e9d3639a41e08f132fc6d9612b8d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "e2842507f92f4f4d9b44cd8eee193f52": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c7d47c9fa5bc4b3ba9ddd9da9fa57550": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}