{ "cells": [ { "cell_type": "markdown", "id": "e69e9896-4be5-4706-9b49-cb772d02e8d4", "metadata": {}, "source": [ "# Swin Transformers as a special sparsity pattern\n", "\n", "In this notebook, we will show how the recently-introduced [Swin Transformers](https://arxiv.org/abs/2103.14030) can be cast\n", "as a sparse transformer with a particular sparsity pattern.\n", "\n", "\n", "Swin Transformers is a hierarchical Transformer whose representation is computed with shifted windows.\n", "The shifted windowing scheme brings efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection\n", "\n", "\"drawing\"\n", "\n", "\n", "In this notebook, we will cover:\n", "- what type of self-attention is needed to replicate a Swin Transformer\n", "- we will show how one can modify their pre-trained Swin Transformer to use the sparse kernels from xformers instead of hand writing the Swin Transformer self-attention by hand.\n", "\n", "Let's start with a few imports. In this notebook, the vanilla Swin Transformer will be taken from `timm`." ] }, { "cell_type": "code", "execution_count": 1, "id": "1beec17c-cdec-4c54-afca-61423c1aab58", "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import copy\n", "import torch\n", "from torch import nn\n", "from torch.utils import benchmark\n", "\n", "import xformers.components.attention.attention_patterns as AP\n", "from xformers.components.attention.core import scaled_dot_product_attention\n", "from xformers.components.attention._sputnik_sparse import SparseCS\n", "\n", "import timm\n", "\n", "%matplotlib inline" ] }, { "cell_type": "markdown", "id": "9072a48e-ba89-4093-ae7e-22706602f11e", "metadata": {}, "source": [ "## What sparsity pattern does Swin Transformer correspond to?\n", "\n", "In xformers, we provide for reference a default implementation of the attention pattern that corresponds to the Swin Transformer architecture.\n", "\n", "It can be found together with the other attention patterns in `xformers.components.attention.attention_patterns`.\n", "\n", "Let's try it out on the example case from above, on an image of size 8x8, and windows of size 4:" ] }, { "cell_type": "code", "execution_count": 2, "id": "57323b78-3b3b-457a-95d8-1f55fdcdddd6", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfAAAAD6CAYAAABeQBU0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAAHZUlEQVR4nO3dsW4VRxiG4T0hSJELChQKIlmhoiQpkLkACt8sN3AkuACQkCyXR2lo0iQQiShWkmZSpALZu94Zjdef53lK2+vfxTm8GuTfsyulTABAlm+2/gEAgPUEHAACCTgABBJwAAgk4AAQSMABINC3a774+4f3ypPj+1WDDudHVc89fXZR9ZyZjObv6a/p3/LPbu1zc+/rXq+tude71zN86c/pj99LKY++/viqgD85vj+92x9X/QCnP/xc9dx+f1b1nJmM5m15U/Xc3Pu612tr7vXu9Qxfel1efbjs4/4LHQACCTgABBJwAAgk4AAQaNUvsQF3z+H86MpfHNv/elb9fed+GW3uc71mwl3jBA4AgQQcAAIJOAAEEnAACCTgABBIwAEgkIADQKBVe+Bz+6JLanc7W/Y6zew3E4BtOYEDQCABB4BAAg4AgQQcAAIJOAAEEnAACOQ6UeBKvdYba68a7TUTEjmBA0AgAQeAQAIOAIEEHAACCTgABBJwAAi0ao3s6bOLab8/qxp007eYmQnXM/e+bnltzT3ba93Le4GROIEDQCABB4BAAg4AgQQcAAIJOAAEEnAACOQ2Mhjc4fzoyvWrXuuNtStmLTPhrnECB4BAAg4AgQQcAAIJOAAEEnAACCTgABBIwAEg0Ko98Ll90SW1u50te51m9psJwLacwAEgkIADQCABB4BAAg4AgQQcAAIJOAAEcp0ocKVe6421V432mgmJnMABIJCAA0AgAQeAQAIOAIEEHAACCTgABFq1Rvb02cW0359VDbrpW8zMhG1t8bqcm7n0Huvx844yc2muf6P6cAIHgEACDgCBBBwAAgk4AAQScAAIJOAAEEjAASCQ60SBISztItf+LYaWq1Hvysylz/eaOToncAAIJOAAEEjAASCQgANAIAEHgEACDgCBVq2RHc6Pbvy6zJY1AjP7zQRgW07gABBIwAEgkIADQCABB4BAAg4AgQQcAAK5jQxgql+rnFvjbLnBK2nm0vfdYuYInMABIJCAA0AgAQeAQAIOAIEEHAACCTgABNqVUq79xc9/+q682x9XDbrpW8zMZDRvy5vpc/m0W/vcg93D8mL3ssePdKtssZI0ysyluf6NavO6vHpfSnn+9cedwAEgkIADQCABB4BAAg4AgQQcAAIJOAAEEnAACOQ6UWAIS7vItX+LoeUazbsyc+nzvWaOzgkcAAIJOAAEEnAACCTgABBIwAEgkIADQKBVa2SH86Mbvy6zZY3AzH4zAdiWEzgABBJwAAgk4AAQSMABIJCAA0AgAQeAQG4jA5jq1yrn1jhbbvBKmrn0fbeYOQIncAAIJOAAEEjAASCQgANAIAEHgEACDgCBdqWUa3/xg93D8mL3suOPw02qvcVsmvJuTxth5tvyZvpcPu3WPjfK+3qLlaRRZi7NHX3dq9Xr8up9KeX51x93AgeAQAIOAIEEHAACCTgABBJwAAgk4AAQSMABIJDrRIEhLO0i99jbH2Xm0ue3+DsMI3ACB4BAAg4AgQQcAAIJOAAEEnAACCTgABDIGtnAtrgqs2WumfNOTi+qngMyOYEDQCABB4BAAg4AgQQcAAIJOAAEEnAACGSNDGDqs/bXcoNX0syl77vFzBE4gQNAIAEHgEACDgCBBBwAAgk4AAQScAAIJOAAEMge+MC2uCqzZa6Z8w7lY9Vzo+i1Uzz33Cgzl+b2mjk6J3AACCTgABBIwAEgkIADQCABB4BAAg4AgayRAUNoWYOq/b6jzFz6/BZrnCNwAgeAQAIOAIEEHAACCTgABBJwAAgk4AAQyBrZwLa4aatlrpnzTk4vqp4DMjmBA0AgAQeAQAIOAIEEHAACCTgABBJwAAhkjQxg6rP213KDV9LMpe+7xcwROIEDQCABB4BAAg4AgQQcAAIJOAAEEnAACCTgABDIHvjAtrgqs2WumfMO5WPVc7TZYhfZzP+17MPX2mLmVZzAASCQgANAIAEHgEACDgCBBBwAAgk4AASyRgZApLm1rV5rslvMvIoTOAAEEnAACCTgABBIwAEgkIADQCABB4BA1sgGtsVNWy1zzZx3cnpR9RyQyQkcAAIJOAAEEnAACCTgABBIwAEgkIADQCABB4BA9sABuHN6/Z2L2qtGW2bee3z5x53AASCQgANAIAEHgEACDgCBBBwAAgk4AASyRjawLa7KbJlr5rxD+Vj1HKSqXfdaMvfsFjOn6ZdLP+oEDgCBBBwAAgk4AAQScAAIJOAAEEjAASCQNTIAItWue7V83y1mXsUJHAACCTgABBJwAAgk4AAQSMABIJCAA0Aga2QD2+KmrZa5Zs47Ob2oeg7I5AQOAIEEHAACCTgABBJwAAgk4AAQSMABIJCAA0Age+AA3Dm9/s5F7VWjLTPvPb78407gABBIwAEgkIADQCABB4BAAg4AgQQcAALtSinX/+Ld7rdpmj70+3GABj+WUh6tfcj7Gm69S9/bqwIOANwO/gsdAAIJOAAEEnAACCTgABBIwAEgkIADQCABB4BAAg4AgQQcAAL9B2NyMknQIrs/AAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "H, W = 8, 8\n", "window_size = 4\n", "\n", "mask = AP.swin_attention_pattern(H, W, window_size, shift_size=0)\n", "mask_shifted = AP.swin_attention_pattern(H, W, window_size, shift_size=2)\n", "\n", "fig = plt.figure(figsize=(7, 14))\n", "ax = fig.add_subplot(1, 2, 1)\n", "ax.imshow(mask)\n", "plt.xticks([])\n", "plt.yticks([])\n", "ax = fig.add_subplot(1, 2, 2)\n", "ax.imshow(mask_shifted)\n", "plt.xticks([])\n", "plt.yticks([])\n", "fig.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "26ca8db6-9897-4239-89fb-f48b1f0d75b8", "metadata": {}, "source": [ "Now let's visualize the self-attention for every pixel in the image. Every sub-image corresponds to the self-attention for one pixel" ] }, { "cell_type": "code", "execution_count": 3, "id": "10fe9b1e-a443-4c54-a141-d3d08969efe6", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1cAAANYCAYAAAAolBclAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAABD9ElEQVR4nO3dwWpcSZY/4Ii/a9piCgkkqhceMMzCdO3sgRTWA0jgF9Az5EP1M+gFBNIDyNhg964bL2Yw2JsaNajpxtWDif+iE3EXSSoiK+7NCN3vg1okHJ8KxdEvM0+R5YwppQAAAMBv8/92fQAAAIDHwHIFAABQgeUKAACgAssVAABABZYrAACACn4oKf7p6En6z+f/llX7/k+/hsXLp9m9S+q36b0fDrNq/xb+ml07dv238Pfwz/RrzG6+UjKnEMa/+5LeY93lBHP9JaX0++w/sCJT09bL1GZj3v1UmfpdfJr2wo9jnWnU3/uW8p1b/9+f/y/8cvu9OFMlcwqhnef7P7z8RzN3v0XvrjNVWi9Tm7Xy/PcYMxVL/ir241d76e3l86zaJ88+he9fX2T3LqnfpvdZPM+qvUoX2bVj19+k63CXbosDVjKnEMa/+5LeY93lBHN9n1I6zv4DKzI1bb1MbTbm3U+VqYN4lE7i6VhnGvX3vqV859a/fvM5vPv4rThTJXMKoZ3n+8svH5q5+y16d52p0nqZ2qyV57/HmCkfCwQAAKjAcgUAAFCB5QoAAKACyxUAAEAFlisAAIAKLFcAAAAVWK4AAAAqsFwBAABUYLkCAACowHIFAABQQUwpbS6IcRlCWK4eLkY/Eff2w2G4S7cxp9acdu59Suk4p9CsdkemuiJTHVi8fBreffwmU32QqQ7IVFfWZurB5Wro+NVeenv5PKv2ybNP4fvXF9m9S+q36X0Wz7Nqr9JFdu3Y9TfpOvuN4FDJnEIY/+5Leo91lxPMNftFa0impq2Xqc3GvPupMnUQj9JJPB3rTKP+3reU79z6128+Z78RHCqZUwjtPN9ffvnQzN1v0bvrTJXWy9RmrTz/PcZM+VggAABABZYrAACACixXAAAAFViuAAAAKrBcAQAAVGC5AgAAqMByBQAAUIHlCgAAoALLFQAAQAUxpbS5IMZlCGG5ergY/UTc2w+H4S7dZn1Ltznt3Npv6V7HrHZHproiUx1YvHwa3n38JlN9kKkOyFRX1mbqweVq6PjVXnp7+Tyr9smzT+H71xfZvUvqt+l9Fs+zaq/SRXbt2PU36Tr7jeBQyZxCGP/uS3qPdZcTzDX7RWtIpqatl6nNxrz7qTJ1EI/SSTwd60yj/t63lO/c+tdvPme/ERwqmVMI7TzfX3750Mzdb9G760yV1svUZq08/z3GTPlYIAAAQAWWKwAAgAosVwAAABVYrgAAACqwXAEAAFRguQIAAKjAcgUAAFCB5QoAAKACyxUAAEAFlisAAIAKYkppc0GMyxDCcvVwMfqJuLcfDsNduo05tea0c+9TSsc5hWa1OzLVFZnqwOLl0/Du4zeZ6oNMdUCmurI2Uw8uV0PHr/bS28vnWbVPnn0K37++yO5dUr9N77N4nlV7lS6ya8euv0nX2W8Eh0rmFML4d1/Se6y7nGCu2S9aQzI1bb1MbTbm3U+VqYN4lE7i6VhnGvX3vqV859a/fvM5+43gUMmcQmjn+f7yy4dm7n6L3l1nqrRepjZr5fnvMWbKxwIBAAAqsFwBAABUYLkCAACowHIFAABQgeUKAACgAssVAABABZYrAACACixXAAAAFViuAAAAKvihpPgvf/r38OY//iuz+lNBbWl9ee85KZtTCGPffUnvuZGpPsgUU+jxd+wv6X8L+vbvX/fSxt173n5YS3cvU+s9xkzFlNLGPxZjXIYQlquHi4J/I7/RfjgMd+k25tSa0869Tykd5xSa1e7IVFdkqgMy1RWZ6oBMdWVtph5croYO4lE6iadZtVfpIpzF8+zeJfW99i6tv0nX2QEbKplT6Zl6vfsJ5pr9ojUkU9PWy1RXZ5GpDs4yt0y1dJa5Zaq0vtfeLWaqtH5Gvddmyv9zBQAAUIHlCgAAoALLFQAAQAWWKwAAgAosVwAAABVYrgAAACqwXAEAAFRguQIAAKjAcgUAAFBBTCltLohxGUJYrh4uRj8R9/bDYfa3dJvTzq39lu51zGp3ZKorMtUBmeqKTHVAprqyNlMPLldDB/EoncTTrNqrdBHO4nl275L6XnuX1t+k6+yADZXMqfRMvd79BHPNftEakqlp62Wqq7PIVAdnmVumWjrL3DJVWt9r7xYzVVo/o95rM+VjgQAAABVYrgAAACqwXAEAAFRguQIAAKjAcgUAAFCB5QoAAKACyxUAAEAFlisAAIAKLFcAAAAVWK4AAAAqiCmlzQUxLkMIy9XDxegn4t5+OAx36Tbm1JrTzr1PKR3nFJrV7shUV2SqAzLVFZnqgEx1ZW2mHlyuhg7iUTqJp1m1V+kinMXz7N4l9b32Lq2/SdfZARsqmVPpmXq9+wnmmv2iNSRT09bLVFdnkakOzjK3TLV0lrllqrS+194tZqq0fka912bKxwIBAAAqsFwBAABUYLkCAACowHIFAABQgeUKAACgAssVAABABZYrAACACixXAAAAFViuAAAAKogppc0FMS5DCMvVw8XoJ+LefjjM/pZuc9q5td/SvY5Z7Y5MdUWmOiBTXZGpDshUV9Zm6sHlauggHqWTeJpVe5Uuwlk8z+5dUt9r79L6m3SdHbChkjmVnqnXu59grtkvWkMyNW29THV1Fpnq4Cxzy1RLZ5lbpkrre+3dYqZK62fUe22mfCwQAACgAssVAABABZYrAACACixXAAAAFViuAAAAKrBcAQAAVGC5AgAAqMByBQAAUIHlCgAAoALLFQAAQAUxpbS5IMZlCGG5evhzCOHPmb1/CiH8UnCWkvpee5fW/5xS2s8p/A1zKj1Tr3c/9lynmFWv9yNT9Wpb6j32WWSqj7PMLVMtnWVumSqt77V3i5kqrZ9L7/WzSimN8k8I4d1Y9b32HvssU8yqpZ+3ld5TzarX+2np7mXq8Z5l7Dm19vO2cpa5Zaqls8wtUy3dT0u9W5zV3Hv7WCAAAEAFlisAAIAKxlyu/jhifa+9S+tLe2+rlfvptfc29dvo9X5aunuZ2l29TO2uvqXe22rl5y2t77X3ttz99L231dLP8Kh6P/gXWgAAAPAwHwsEAACowHIFAABQgeUKAACgAssVAABABZYrAACACixXAAAAFViuAAAAKrBcAQAAVPBDSfFPR0/Sfz7/t6za93/6NSxePs3uXVK/Te/9cJhV+7fw1+zaseu/hb+Hf6ZfY3bzlZI5hTD+3Zf0HusuJ5jrLyml32f/gRWZmrZepjYb8+6nytTv4tO0F34c60yj/t63lO/c+v/+/H/hl9vvxZkqmVMI7Tzf/+HlP5q5+y16d52p0nqZ2qyV57/HmKmYUspucvxqL729fJ5V++TZp/D964vs3iX12/Q+i+dZtVfpIrt27PqbdB3u0m1xwErmFML4d1/Se6y7nGCu71NKx9l/YEWmpq2Xqc3GvPupMnUQj9JJPB3rTKP+3reU79z6128+h3cfvxVnqmROIbTzfH/55UMzd79F764zVVovU5u18vz3GDPlY4EAAAAVWK4AAAAqsFwBAABUYLkCAACowHIFAABQgeUKAACgAssVAABABZYrAACACixXAAAAFViuAAAAKogppc0FMS5DCMvVw8XoJ+LefjgMd+k25tSa0869Tykd5xSa1e7IVFdkqgOLl0/Du4/fZKoPMtUBmerK2kw9uFwNHb/aS28vn2fVPnn2KXz/+iK7d0n9Nr3P4nlW7VW6yK4du/4mXWe/ERwqmVMI4999Se+x7nKCuWa/aA3J1LT1MrXZmHc/VaYO4lE6iadjnWnU3/uW8p1b//rN5+w3gkMlcwqhnef7yy8fmrn7LXp3nanSepnarJXnv8eYKR8LBAAAqMByBQAAUIHlCgAAoALLFQAAQAWWKwAAgAosVwAAABVYrgAAACqwXAEAAFRguQIAAKggppQ2F8S4DCEsVw8Xo5+Ie/vhMNyl26xv6TannVv7Ld3rmNXuyFRXZKoDi5dPw7uP32SqDzLVAZnqytpMPbhcDR2/2ktvL59n1T559il8//oiu3dJ/Ta9z+J5Vu1VusiuHbv+Jl1nvxEcKplTCOPffUnvse5ygrlmv2gNydS09TK12Zh3P1WmDuJROomnY51p1N/7lvKdW//6zefsN4JDJXMKoZ3n+8svH5q5+y16d52p0nqZ2qyV57/HmCkfCwQAAKjAcgUAAFCB5QoAAKACyxUAAEAFlisAAIAKLFcAAAAVWK4AAAAqsFwBAABUYLkCAACowHIFAABQQUwpbS6IcRlCWK4eLkY/Eff2w2G4S7cxp9acdu59Suk4p9CsdkemuiJTHVi8fBreffwmU32QqQ7IVFfWZurB5Wro+NVeenv5PKv2ybNP4fvXF9m9S+q36X0Wz7Nqr9JFdu3Y9TfpOvuN4FDJnEIY/+5Leo91lxPMNftFa0impq2Xqc3GvPupMnUQj9JJPB3rTKP+3reU79z6128+Z78RHCqZUwjtPN9ffvnQzN1v0bvrTJXWy9RmrTz/PcZM+VggAABABZYrAACACixXAAAAFViuAAAAKrBcAQAAVGC5AgAAqMByBQAAUIHlCgAAoIIfSor/8qd/D2/+478yqz8V1JbWl/eek7I5hTD23Zf0nhuZ6oNMMYUef8f+kv63oG///nUvbdy95+2HtXT3MrXeY8xUTClt/GMxxmUIYbl6uCj4N/Ib7YfDcJdus76l25x2bu23dK9jVrsjU12RqQ7IVFdkqgMy1ZW1mXpwuRo6iEfpJJ5m1V6li3AWz7N7l9T32ru0/iZdZwdsqGROpWfq9e4nmGv2i9aQTE1bL1NdnUWmOjjL3DLV0lnmlqnS+l57t5ip0voZ9V6bKf/PFQAAQAWWKwAAgAosVwAAABVYrgAAACqwXAEAAFRguQIAAKjAcgUAAFCB5QoAAKACyxUAAEAFlisAAIAKYkppc0GMyxDCcvVwMfqJuLcfDsNduo05tea0c+9TSsc5hWa1OzLVFZnqgEx1RaY6IFNdWZupB5eroYN4lE7iaVbtVboIZ/E8u3dJfa+9S+tv0nV2wIZK5lR6pl7vfoK5Zr9oDcnUtPUy1dVZZKqDs8wtUy2dZW6ZKq3vtXeLmSqtn1HvtZnysUAAAIAKLFcAAAAVWK4AAAAqsFwBAABUYLkCAACowHIFAABQgeUKAACgAssVAABABZYrAACACmJKaXNBjMsQwnL1cDH6ibi3Hw6zv6XbnHZu7bd0r2NWuyNTXZGpDshUV2SqAzLVlbWZenC5GjqIR+kknmbVXqWLcBbPs3uX1Pfau7T+Jl1nB2yoZE6lZ+r17ieYa/aL1pBMTVsvU12dRaY6OMvcMtXSWeaWqdL6Xnu3mKnS+hn1XpspHwsEAACowHIFAABQgeUKAACgAssVAABABZYrAACACixXAAAAFViuAAAAKrBcAQAAVGC5AgAAqMByBQAAUEFMKW0uiHEZQliuHi5GPxH39sNhuEu3MafWnHbufUrpOKfQrHZHproiUx2Qqa7IVAdkqitrM/XgcjV0EI/SSTzNqr1KF+Esnmf3LqnvtXdp/U26zg7YUMmcSs/U691PMNfsF60hmZq2Xqa6OotMdXCWuWWqpbPMLVOl9b32bjFTpfUz6r02Uz4WCAAAUIHlCgAAoALLFQAAQAWWKwAAgAosVwAAABVYrgAAACqwXAEAAFRguQIAAKjAcgUAAFBBTCltLohxGUJYrh7+HEL4c2bvn0IIvxScpaS+196l9T+nlPZzCn/DnErP1Ovdjz3XKWbV6/3IVL3alnqPfRaZ6uMsc8tUS2eZW6ZK63vt3WKmSuvn0nv9rFJKo/wTQng3Vn2vvcc+yxSzaunnbaX3VLPq9X5aunuZerxnGXtOrf28rZxlbplq6Sxzy1RL99NS7xZnNffePhYIAABQgeUKAACggjGXqz+OWN9r79L60t7bauV+eu29Tf02er2flu5epnZXL1O7q2+p97Za+XlL63vtvS13P33vbbX0Mzyq3g/+hRYAAAA8zMcCAQAAKrBcAQAAVGC5AgAAqMByBQAAUIHlCgAAoALLFQAAQAWWKwAAgAosVwAAABVYrgAAACr4oaT4p6Mn6T+f/1tW7fs//RoWL59m9y6p36b3fjjMqv1b+Gt27dj138Lfwz/TrzG7+UrJnEIY/+5Leo91lxPM9ZeU0u+z/8CKTE1bL1ObjXn3U2Xqd/Fp2gs/jnWmUX/vW8p3bv1/f/6/8Mvt9+JMlcwphHae7//w8h/N3P0WvbvOVGm9TG3WyvPfY8xUTCllNzl+tZfeXj7Pqn3y7FP4/vVFdu+S+m16n8XzrNqrdJFdO3b9TboOd+m2OGAlcwph/Lsv6T3WXU4w1/cppePsP7AiU9PWy9RmY979VJk6iEfpJJ6OdaZRf+9byndu/es3n8O7j9+KM1UypxDaeb6//PKhmbvfonfXmSqtl6nNWnn+e4yZ8rFAAACACixXAAAAFViuAAAAKrBcAQAAVGC5AgAAqMByBQAAUIHlCgAAoALLFQAAQAWWKwAAgApiSmlzQYzLEMJy9XAx+om4tx8Ow126zfqWbnPaubXf0r2OWe2OTHVFpjqwePk0vPv4Tab6IFMdkKmurM3Ug8vV0PGrvfT28nlW7ZNnn8L3ry+ye5fUb9P7LJ5n1V6li+zasetv0nX2G8GhkjmFMP7dl/Qe6y4nmGv2i9aQTE1bL1ObjXn3U2XqIB6lk3g61plG/b1vKd+59a/ffM5+IzhUMqcQ2nm+v/zyoZm736J315kqrZepzVp5/nuMmfKxQAAAgAosVwAAABVYrgAAACqwXAEAAFRguQIAAKjAcgUAAFCB5QoAAKACyxUAAEAFlisAAIAKLFcAAAAVxJTS5oIYlyGE5erhYvQTcW8/HIa7dBtzas1p596nlI5zCs1qd2SqKzLVgcXLp+Hdx28y1QeZ6oBMdWVtph5croaOX+2lt5fPs2qfPPsUvn99kd27pH6b3mfxPKv2Kl1k145df5Ous98IDpXMKYTx776k91h3OcFcs1+0hmRq2nqZ2mzMu58qUwfxKJ3E07HONOrvfUv5zq1//eZz9hvBoZI5hdDO8/3llw/N3P0WvbvOVGm9TG3WyvPfY8yUjwUCAABUYLkCAACowHIFAABQgeUKAACgAssVAABABZYrAACACixXAAAAFViuAAAAKrBcAQAAVBBTSpsLYlyGEJarh4vRT8S9/XAY7tJt1rd0m9POrf2W7nXMandkqisy1YHFy6fh3cdvMtUHmeqATHVlbaYeXK6Gjl/tpbeXz7Nqnzz7FL5/fZHdu6R+m95n8Tyr9ipdZNeOXX+TrrPfCA6VzCmE8e++pPdYdznBXLNftIZkatp6mdpszLufKlMH8SidxNOxzjTq731L+c6tf/3mc/YbwaGSOYXQzvP95ZcPzdz9Fr27zlRpvUxt1srz32PMlI8FAgAAVGC5AgAAqMByBQAAUIHlCgAAoALLFQAAQAWWKwAAgAosVwAAABVYrgAAACqwXAEAAFTwQ0nxX/707+HNf/xXZvWngtrS+vLec1I2pxDGvvuS3nMjU32QKabQ4+/YX9L/FvTt37/upY2797z9sJbuXqbWe4yZiimljX8sxrgMISxXDxcF/0Z+o/1wGO7SbcypNaede59SOs4pNKvdkamuyFQHZKorMtUBmerK2kw9uFwNHcSjdBJPs2qv0kU4i+fZvUvqe+1dWn+TrrMDNlQyp9Iz9Xr3E8w1+0VrSKamrZeprs4iUx2cZW6Zauksc8tUaX2vvVvMVGn9jHqvzZT/5woAAKACyxUAAEAFlisAAIAKLFcAAAAVWK4AAAAqsFwBAABUYLkCAACowHIFAABQgeUKAACggphS2lwQ4zKEsFw9XIx+Iu7th8Psb+k2p51b+y3d65jV7shUV2SqAzLVFZnqgEx1ZW2mHlyuhg7iUTqJp1m1V+kinMXz7N4l9b32Lq2/SdfZARsqmVPpmXq9+wnmmv2iNSRT09bLVFdnkakOzjK3TLV0lrllqrS+194tZqq0fka912bKxwIBAAAqsFwBAABUYLkCAACowHIFAABQgeUKAACgAssVAABABZYrAACACixXAAAAFViuAAAAKrBcAQAAVBBTSpsLYlyGEJarh4vRT8S9/XAY7tJtzKk1p517n1I6zik0q92Rqa7IVAdkqisy1QGZ6sraTD24XA0dxKN0Ek+zaq/SRTiL59m9S+p77V1af5OuswM2VDKn0jP1evcTzDX7RWtIpqatl6muziJTHZxlbplq6Sxzy1Rpfa+9W8xUaf2Meq/NlI8FAgAAVGC5AgAAqMByBQAAUIHlCgAAoALLFQAAQAWWKwAAgAosVwAAABVYrgAAACqwXAEAAFRguQIAAKggppQ2F8S4DCEsVw8Xo5+Ie/vhMNyl25hTa0479z6ldJxTaFa7I1NdkakOyFRXZKoDMtWVtZl6cLkaOohH6SSeZtVepYtwFs+ze5fU99q7tP4mXWcHbKhkTqVn6vXuJ5hr9ovWkExNWy9TXZ1Fpjo4y9wy1dJZ5pap0vpee7eYqdL6GfVemykfCwQAAKjAcgUAAFCB5QoAAKACyxUAAEAFlisAAIAKLFcAAAAVWK4AAAAqsFwBAABUYLkCAACoIKaUNhfEuAwhLFcPfw4h/Dmz908hhF8KzlJS32vv0vqfU0r7OYW/YU6lZ+r17see6xSz6vV+ZKpebUu9xz6LTPVxlrllqqWzzC1TpfW99m4xU6X1c+m9flYppVH+CSG8G6u+195jn2WKWbX087bSe6pZ9Xo/Ld29TD3es4w9p9Z+3lbOMrdMtXSWuWWqpftpqXeLs5p7bx8LBAAAqMByBQAAUMGYy9UfR6zvtXdpfWnvbbVyP7323qZ+G73eT0t3L1O7q5ep3dW31Htbrfy8pfW99t6Wu5++97Za+hkeVe8H/0ILAAAAHuZjgQAAABVYrgAAACqwXAEAAFRguQIAAKjAcgUAAFCB5QoAAKACyxUAAEAFlisAAIAKfigp/unoSfrP5/+WVfv+T7+Gxcun2b1L6rfpvR8Os2r/Fv6aXTt2/bfw9/DP9GvMbr5SMqcQxr/7kt5j3eUEc/0lpfT77D+wIlPT1svUZmPe/VSZ+l18mvbCj2OdadTf+5bynVv/35//L/xy+704UyVzCqGd5/s/vPxHM3e/Re+uM1VaL1ObtfL89xgzFVNK2U2OX+2lt5fPs2qfPPsUvn99kd27pH6b3mfxPKv2Kl1k145df5Ouw126LQ5YyZxCGP/uS3qPdZcTzPV9Suk4+w+syNS09TK12Zh3P1WmDuJROomnY51p1N/7lvKdW//6zefw7uO34kyVzCmEdp7vL798aObut+jddaZK62Vqs1ae/x5jpnwsEAAAoALLFQAAQAWWKwAAgAosVwAAABVYrgAAACqwXAEAAFRguQIAAKjAcgUAAFCB5QoAAKACyxUAAEAFMaW0uSDGZQhhuXq4GP1E3NsPh+Eu3cacWnPaufcppeOcQrPaHZnqikx1YPHyaXj38ZtM9UGmOiBTXVmbqQeXq6HjV3vp7eXzrNonzz6F719fZPcuqd+m91k8z6q9ShfZtWPX36Tr7DeCQyVzCmH8uy/pPdZdTjDX7BetIZmatl6mNhvz7qfK1EE8SifxdKwzjfp731K+c+tfv/mc/UZwqGROIbTzfH/55UMzd79F764zVVovU5u18vz3GDPlY4EAAAAVWK4AAAAqsFwBAABUYLkCAACowHIFAABQgeUKAACgAssVAABABZYrAACACixXAAAAFcSU0uaCGJchhOXq4WL0E3FvPxyGu3Sb9S3d5rRza7+lex2z2h2Z6opMdWDx8ml49/GbTPVBpjogU11Zm6kHl6uh41d76e3l86zaJ88+he9fX2T3LqnfpvdZPM+qvUoX2bVj19+k6+w3gkMlcwph/Lsv6T3WXU4w1+wXrSGZmrZepjYb8+6nytRBPEon8XSsM436e99SvnPrX7/5nP1GcKhkTiG083x/+eVDM3e/Re+uM1VaL1ObtfL89xgz5WOBAAAAFViuAAAAKrBcAQAAVGC5AgAAqMByBQAAUIHlCgAAoALLFQAAQAWWKwAAgAosVwAAABVYrgAAACqIKaXNBTEuQwjL1cPF6Cfi3n44DHfpNubUmtPOvU8pHecUmtXuyFRXZKoDi5dPw7uP32SqDzLVAZnqytpMPbhcDR2/2ktvL59n1T559il8//oiu3dJ/Ta9z+J5Vu1VusiuHbv+Jl1nvxEcKplTCOPffUnvse5ygrlmv2gNydS09TK12Zh3P1WmDuJROomnY51p1N/7lvKdW//6zefsN4JDJXMKoZ3n+8svH5q5+y16d52p0nqZ2qyV57/HmCkfCwQAAKjAcgUAAFCB5QoAAKACyxUAAEAFlisAAIAKLFcAAAAVWK4AAAAqsFwBAABUYLkCAACo4IeS4r/86d/Dm//4r8zqTwW1pfXlveekbE4hjH33Jb3nRqb6IFNMocffsb+k/y3o279/3Usbd+95+2Et3b1MrfcYMxVTShv/WIxxGUJYrh4uCv6N/Eb74TDcpduYU2tOO/c+pXScU2hWuyNTXZGpDshUV2SqAzLVlbWZenC5GjqIR+kknmbVXqWLcBbPs3uX1Pfau7T+Jl1nB2yoZE6lZ+r17ieYa/aL1pBMTVsvU12dRaY6OMvcMtXSWeaWqdL6Xnu3mKnS+hn1Xpsp/88VAABABZYrAACACixXAAAAFViuAAAAKrBcAQAAVGC5AgAAqMByBQAAUIHlCgAAoALLFQAAQAUxpbS5IMZlCGG5ergY/UTc2w+H2d/SbU47t/Zbutcxq92Rqa7IVAdkqisy1QGZ6sraTD24XA0dxKN0Ek+zaq/SRTiL59m9S+p77V1af5OuswM2VDKn0jP1evcTzDX7RWtIpqatl6muziJTHZxlbplq6Sxzy1Rpfa+9W8xUaf2Meq/NlI8FAgAAVGC5AgAAqMByBQAAUIHlCgAAoALLFQAAQAWWKwAAgAosVwAAABVYrgAAACqwXAEAAFRguQIAAKggppQ2F8S4DCEsVw8Xo5+Ie/vhMNyl25hTa0479z6ldJxTaFa7I1NdkakOyFRXZKoDMtWVtZl6cLkaOohH6SSeZtVepYtwFs+ze5fU99q7tP4mXWcHbKhkTqVn6vXuJ5hr9ovWkExNWy9TXZ1Fpjo4y9wy1dJZ5pap0vpee7eYqdL6GfVemykfCwQAAKjAcgUAAFCB5QoAAKACyxUAAEAFlisAAIAKLFcAAAAVWK4AAAAqsFwBAABUYLkCAACoIKaUNhfEuAwhLFcPF6OfiHv74TD7W7rNaefWfkv3Oma1OzLVFZnqgEx1RaY6IFNdWZupB5eroYN4lE7iaVbtVboIZ/E8u3dJfa+9S+tv0nV2wIZK5lR6pl7vfoK5Zr9oDcnUtPUy1dVZZKqDs8wtUy2dZW6ZKq3vtXeLmSqtn1HvtZnysUAAAIAKLFcAAAAVWK4AAAAqsFwBAABUYLkCAACowHIFAABQgeUKAACgAssVAABABZYrAACACixXAAAAFcSU0uaCGJchhOXq4c8hhD9n9v4phPBLwVlK6nvtXVr/c0ppP6fwN8yp9Ey93v3Yc51iVr3ej0zVq22p99hnkak+zjK3TLV0lrllqrS+194tZqq0fi69188qpTTKPyGEd2PV99p77LNMMauWft5Wek81q17vp6W7l6nHe5ax59Taz9vKWeaWqZbOMrdMtXQ/LfVucVZz7+1jgQAAABVYrgAAACoYc7n644j1vfYurS/tva1W7qfX3tvUb6PX+2np7mVqd/Uytbv6lnpvq5Wft7S+197bcvfT995WSz/Do+r94F9oAQAAwMN8LBAAAKACyxUAAEAFlisAAIAKLFcAAAAVWK4AAAAqsFwBAABUYLkCAACowHIFAABQwQ8lxb+LT9Ne+DGr9m/hr2E/HGb3LqnvtXdp/bfw9/DP9GvMbr5SMqfSM/V69xPM9ZeU0u+z/8CKTE1bL1NdnUWmOjjL3DLV0lnmlqnS+l57t5ip0voZ9V6bqaLlai/8GE7iaVbtVbrIri2t77V3af1Nus7uO1Qyp9Iz9Xr3E8z1f7KLB2Rq2nqZ6uosMtXBWeaWqZbOMrdMldb32rvFTJXWz6j32kz5WCAAAEAFlisAAIAKLFcAAAAVWK4AAAAqsFwBAABUYLkCAACowHIFAABQgeUKAACgAssVAABABZYrAACACmJKaXNBjMsQwnL1cDH6ibi3Hw7DXbqNObXmtHPvU0rHOYVmtTsy1RWZ6oBMdUWmOiBTXVmbqQeXq6GDeJRO4mlW7VW6CGfxPLt3SX2vvUvrb9J1dsCGSuZUeqZe736CuWa/aA3J1LT1MtXVWWSqg7PMLVMtnWVumSqt77V3i5kqrZ9R77WZ8rFAAACACixXAAAAFViuAAAAKrBcAQAAVGC5AgAAqMByBQAAUIHlCgAAoALLFQAAQAWWKwAAgApiSmlzQYzLEMJy9XAx+om4tx8Os7+l25x2bu23dK9jVrsjU12RqQ7IVFdkqgMy1ZW1mXpwuRo6iEfpJJ5m1V6li3AWz7N7l9T32ru0/iZdZwdsqGROpWfq9e4nmGv2i9aQTE1bL1NdnUWmOjjL3DLV0lnmlqnS+l57t5ip0voZ9V6bKR8LBAAAqMByBQAAUIHlCgAAoALLFQAAQAWWKwAAgAosVwAAABVYrgAAACqwXAEAAFRguQIAAKjAcgUAAFBBTCltLohxGUJYrh4uRj8R9/bDYbhLtzGn1px27n1K6Tin0Kx2R6a6IlMdkKmuyFQHZKorazP14HI1dBCP0kk8zaq9ShfhLJ5n9y6p77V3af1Nus4O2FDJnErP1OvdTzDX7BetIZmatl6mujqLTHVwlrllqqWzzC1TpfW99m4xU6X1M+q9NlM+FggAAFCB5QoAAKACyxUAAEAFlisAAIAKLFcAAAAVWK4AAAAqsFwBAABUYLkCAACo4IeS4j+8/Ee4vPyQVfvkWQiXX/JqS+u36T0nJXMKYfy7L+k9NzLVB5liCi3lO7f+9Zt/ZPd9DC6/fGjm7j1vP6ylu5ep9R5jpmJKaeMfjDEuQwjL1cNF9r+R32w/HGZ/S7c57dzab+lex6x2R6a6IlMdWLx8Gt59/CZTfZCpDshUV9Zm6sHlauj41V56e/k8q/bJs0/h+9cX2b1L6rfpfRbPs2qv0kV27dj1N+k6+43gUMmcQhj/7kt6j3WXE8w1+0VrSKamrZepzca8+6kydRCP0kk8HetMo/7et5Tv3PrXbz5nvxEcKplTCO083//rv7K3cfdb9O46U6X1MrVZK89/jzFT/p8rAACACixXAAAAFViuAAAAKrBcAQAAVGC5AgAAqMByBQAAUIHlCgAAoALLFQAAQAWWKwAAgAosVwAAABXElNLmghiXIYTl6uFi9BNxbz8chrt0G3NqzWnn3qeUjnMKzWp3ZKorMtWBxcun4d3HbzLVB5nqgEx1ZW2mHlyuho5f7aW3l8+zap88+xS+f32R3bukfpveZ/E8q/YqXWTXjl1/k66z3wgOlcwphPHvvqT3WHc5wVyzX7SGZGraepnabMy7nypTB/EoncTTsc406u99S/nOrX/95nP2G8GhkjmF0M7z/eWXD83c/Ra9u85Uab1MbdbK899jzJSPBQIAAFRguQIAAKjAcgUAAFCB5QoAAKACyxUAAEAFlisAAIAKLFcAAAAVWK4AAAAqsFwBAABUEFNKmwtiXIYQlquHi9FPxL39cBju0m3Wt3Sb086t/Zbudcxqd2SqKzLVgcXLp+Hdx28y1QeZ6oBMdWVtph5croaOX+2lt5fPs2qfPPsUvn99kd27pH6b3mfxPKv2Kl1k145df5Ous98IDpXMKYTx776k91h3OcFcs1+0hmRq2nqZ2mzMu58qUwfxKJ3E07HONOrvfUv5zq1//eZz9hvBoZI5hdDO8/3llw/N3P0WvbvOVGm9TG3WyvPfY8yUjwUCAABUYLkCAACowHIFAABQgeUKAACgAssVAABABZYrAACACixXAAAAFViuAAAAKrBcAQAAVGC5AgAAqCCmlDYXxLgMISxXDxejn4h7++Ew3KXbmFNrTjv3PqV0nFNoVrsjU12RqQ4sXj4N7z5+k6k+yFQHZKorazP14HI1dPxqL729fJ5V++TZp/D964vs3iX12/Q+i+dZtVfpIrt27PqbdJ39RnCoZE4hjH/3Jb3HussJ5pr9ojUkU9PWy9RmY979VJk6iEfpJJ6OdaZRf+9byndu/es3n7PfCA6VzCmEdp7vL798aObut+jddaZK62Vqs1ae/x5jpnwsEAAAoALLFQAAQAWWKwAAgAosVwAAABVYrgAAACqwXAEAAFRguQIAAKjAcgUAAFCB5QoAAKCCmFLaXBDjMoSwXD38OYTw58zeP4UQfik4S0l9r71L639OKe3nFP6GOZWeqde7H3uuU8yq1/uRqXq1LfUe+ywy1cdZ5papls4yt0yV1vfau8VMldbPpff6WaWURvknhPBurPpee499lilm1dLP20rvqWbV6/20dPcy9XjPMvacWvt5WznL3DLV0lnmlqmW7qel3i3Oau69fSwQAACgAssVAABABWMuV38csb7X3qX1pb231cr99Np7m/pt9Ho/Ld29TO2uXqZ2V99S72218vOW1vfae1vufvre22rpZ3hUvR/8Cy0AAAB4mI8FAgAAVGC5AgAAqMByBQAAUIHlCgAAoALLFQAAQAWWKwAAgAosVwAAABVYrgAAACqwXAEAAFTwQ0nx7+LTtBd+zKr9W/hr2A+H2b1L6nvtXVr/Lfw9/DP9GrObr5TMqfRMvd79BHP9JaX0++w/sCJT09bLVFdnkakOzjK3TLV0lrllqrS+194tZqq0fka912aqaLnaCz+Gk3iaVXuVLrJrS+t77V1af5Ous/sOlcyp9Ey93v0Ec/2f7OIBmZq2Xqa6OotMdXCWuWWqpbPMLVOl9b32bjFTpfUz6r02Uz4WCAAAUIHlCgAAoALLFQAAQAWWKwAAgAosVwAAABVYrgAAACqwXAEAAFRguQIAAKjAcgUAAFBBTCltLohxGUJYrh4uRj8R9/bDYbhLtzGn1px27n1K6Tin0Kx2R6a6IlMdkKmuyFQHZKorazP14HI1dBCP0kk8zaq9ShfhLJ5n9y6p77V3af1Nus4O2FDJnErP1OvdTzDX7BetIZmatl6mujqLTHVwlrllqqWzzC1TpfW99m4xU6X1M+q9NlM+FggAAFCB5QoAAKACyxUAAEAFlisAAIAKLFcAAAAVWK4AAAAqsFwBAABUYLkCAACowHIFAABQgeUKAACggphS2lwQ4zKEsFw9XIx+Iu7th8Nwl25jTq057dz7lNJxTqFZ7Y5MdUWmOiBTXZGpDshUV9Zm6sHlauggHqWTeJpVe5Uuwlk8z+5dUt9r79L6m3SdHbChkjmVnqnXu59grtkvWkMyNW29THV1Fpnq4Cxzy1RLZ5lbpkrre+3dYqZK62fUe22mfCwQAACgAssVAABABZYrAACACixXAAAAFViuAAAAKrBcAQAAVGC5AgAAqMByBQAAUIHlCgAAoIKYUtpcEOMyhLBcPVyMfiLu7YfD7G/pNqedW/st3euY1e7IVFdkqgMy1RWZ6oBMdWVtph5croYO4lE6iadZtVfpIpzF8+zeJfW99i6tv0nX2QEbKplT6Zl6vfsJ5pr9ojUkU9PWy1RXZ5GpDs4yt0y1dJa5Zaq0vtfeLWaqtH5GvddmyscCAQAAKrBcAQAAVGC5AgAAqMByBQAAUIHlCgAAoALLFQAAQAWWKwAAgAosVwAAABVYrgAAACr4oaT4Dy//ES4vP2TVPnkWwuWXvNrS+m16z0nJnEIY/+5Les+NTPVBpphCS/nOrX/95h/ZfR+Dyy8fmrl7z9sPa+nuZWq9x5ipmFLa+AdjjMsQwnL1cJH9b+Q32w+H4S7dxpxac9q59yml45xCs9odmeqKTHVg8fJpePfxm0z1QaY6IFNdWZupB5eroeNXe+nt5fOs2ifPPoXvX19k9y6p36b3WTzPqr1KF9m1Y9ffpOvsN4JDJXMKYfy7L+k91l1OMNfsF60hmZq2XqY2G/Pup8rUQTxKJ/F0rDON+nvfUr5z61+/+Zz9RnCoZE4htPN8/6//yt7G3W/Ru+tMldbL1GatPP89xkz5f64AAAAqsFwBAABUYLkCAACowHIFAABQgeUKAACgAssVAABABZYrAACACixXAAAAFViuAAAAKogppc0FMS5DCMvVw8XoJ+LefjgMd+k261u6zWnn1n5L9zpmtTsy1RWZ6sDi5dPw7uM3meqDTHVAprqyNlMPLldDx6/20tvL51m1T559Ct+/vsjuXVK/Te+zeJ5Ve5UusmvHrr9J19lvBIdK5hTC+Hdf0nusu5xgrtkvWkMyNW29TG025t1PlamDeJRO4ulYZxr1976lfOfWv37zOfuN4FDJnEJo5/n+8suHZu5+i95dZ6q0XqY2a+X57zFmyscCAQAAKrBcAQAAVGC5AgAAqMByBQAAUIHlCgAAoALLFQAAQAWWKwAAgAosVwAAABVYrgAAACqwXAEAAFQQU0qbC2JchhCWq4eL0U/Evf1wGO7SbcypNaede59SOs4pNKvdkamuyFQHFi+fhncfv8lUH2SqAzLVlbWZenC5Gjp+tZfeXj7Pqn3y7FP4/vVFdu+S+m16n8XzrNqrdJFdO3b9TbrOfiM4VDKnEMa/+5LeY93lBHPNftEakqlp62VqszHvfqpMHcSjdBJPxzrTqL/3LeU7t/71m8/ZbwSHSuYUQjvP95dfPjRz91v07jpTpfUytVkrz3+PMVM+FggAAFCB5QoAAKACyxUAAEAFlisAAIAKLFcAAAAVWK4AAAAqsFwBAABUYLkCAACowHIFAABQQUwpbS6IcRlCWK4eLkY/Eff2w2G4S7dZ39JtTju39lu61zGr3ZGprshUBxYvn4Z3H7/JVB9kqgMy1ZW1mXpwuRo6frWX3l4+z6p98uxT+P71RXbvkvptep/F86zaq3SRXTt2/U26zn4jOFQypxDGv/uS3mPd5QRzzX7RGpKpaetlarMx736qTB3Eo3QST8c606i/9y3lO7f+9ZvP2W8Eh0rmFEI7z/eXXz40c/db9O46U6X1MrVZK89/jzFTPhYIAABQgeUKAACgAssVAABABZYrAACACixXAAAAFViuAAAAKrBcAQAAVGC5AgAAqMByBQAAUIHlCgAAoIKYUtpcEOMyhLBcPfw5hPDnzN4/hRB+KThLSX2vvUvrf04p7ecU/oY5lZ6p17sfe65TzKrX+5GperUt9R77LDLVx1nmlqmWzjK3TJXW99q7xUyV1s+l9/pZpZRG+SeE8G6s+l57j32WKWbV0s/bSu+pZtXr/bR09zL1eM8y9pxa+3lbOcvcMtXSWeaWqZbup6XeLc5q7r19LBAAAKACyxUAAEAFYy5XfxyxvtfepfWlvbfVyv302nub+m30ej8t3b1M7a5epnZX31LvbbXy85bW99p7W+5++t7baulneFS9H/wLLQAAAHiYjwUCAABUYLkCAACowHIFAABQgeUKAACgAssVAABABZYrAACACixXAAAAFViuAAAAKvihpPh38WnaCz9m1f4t/DXsh8Ps3iX1vfYurf8W/h7+mX6N2c1XSuZUeqZe736Cuf6SUvp99h9Ykalp62Wqq7PIVAdnmVumWjrL3DJVWt9r7xYzVVo/o95rM1W0XO2FH8NJPM2qvUoX2bWl9b32Lq2/SdfZfYdK5lR6pl7vfoK5/k928YBMTVsvU12dRaY6OMvcMtXSWeaWqdL6Xnu3mKnS+hn1XpspHwsEAACowHIFAABQgeUKAACgAssVAABABZYrAACACixXAAAAFViuAAAAKrBcAQAAVGC5AgAAqMByBQAAUEFMKW0uiHEZQliuHi5GPxH39sNhuEu3MafWnHbufUrpOKfQrHZHproiUx2Qqa7IVAdkqitrM/XgcjV0EI/SSTzNqr1KF+Esnmf3LqnvtXdp/U26zg7YUMmcSs/U691PMNfsF60hmZq2Xqa6OotMdXCWuWWqpbPMLVOl9b32bjFTpfUz6r02Uz4WCAAAUIHlCgAAoALLFQAAQAWWKwAAgAosVwAAABVYrgAAACqwXAEAAFRguQIAAKjAcgUAAFBBTCltLohxGUJYrh4uRj8R9/bDYfa3dJvTzq39lu51zGp3ZKorMtUBmeqKTHVAprqyNlMPLldDB/EoncTTrNqrdBHO4nl275L6XnuX1t+k6+yADZXMqfRMvd79BHPNftEakqlp62Wqq7PIVAdnmVumWjrL3DJVWt9r7xYzVVo/o95rM+VjgQAAABVYrgAAACqwXAEAAFRguQIAAKjAcgUAAFCB5QoAAKACyxUAAEAFlisAAIAKLFcAAAAVWK4AAAAqiCmlzQUxLkMIy9XDxegn4t5+OAx36Tbm1JrTzr1PKR3nFJrV7shUV2SqAzLVFZnqgEx1ZW2mHlyuhg7iUTqJp1m1V+kinMXz7N4l9b32Lq2/SdfZARsqmVPpmXq9+wnmmv2iNSRT09bLVFdnkakOzjK3TLV0lrllqrS+194tZqq0fka912bKxwIBAAAqsFwBAABUYLkCAACowHIFAABQgeUKAACgAssVAABABZYrAACACixXAAAAFViuAAAAKogppc0FMS5DCMvVw8XoJ+LefjjM/pZuc9q5td/SvY5Z7Y5MdUWmOrB4+TS8+/hNpvogUx2Qqa6szdSDy9XQ8au99PbyeVbtk2efwvevL7J7l9Rv0/ssnmfVXqWL7Nqx62/SdfYbwaGSOYUw/t2X9B7rLieYa/aL1pBMTVsvU5uNefdTZeogHqWTeDrWmUb9vW8p37n1r998zn4jOFQypxDaeb6//PKhmbvfonfXmSqtl6nNWnn+e4yZ8rFAAACACixXAAAAFViuAAAAKrBcAQAAVGC5AgAAqMByBQAAUIHlCgAAoALLFQAAQAWWKwAAgAosVwAAABXElNLmghiXIYTl6uFi9BNxbz8chrt0G3NqzWnn3qeUjnMKzWp3ZKorMtWBxcun4d3HbzLVB5nqgEx1ZW2mHlyuho5f7aW3l8+zap88+xS+f32R3bukfpveZ/E8q/YqXWTXjl1/k66z3wgOlcwphPHvvqT3WHc5wVyzX7SGZGraepnabMy7nypTB/EoncTTsc406u99S/nOrX/95nP2G8GhkjmF0M7z/eWXD83c/Ra9u85Uab1MbdbK899jzJSPBQIAAFRguQIAAKjAcgUAAFCB5QoAAKACyxUAAEAFlisAAIAKLFcAAAAVWK4AAAAqsFwBAABUYLkCAACoIKaUNhfEuAwhLFcPF6OfiHv74TDcpduYU2tOO/c+pXScU2hWuyNTXZGpDixePg3vPn6TqT7IVAdkqitrM/XgcjV0/Govvb18nlX75Nmn8P3ri+zeJfXb9D6L51m1V+kiu3bs+pt0nf1GcKhkTiGMf/clvce6ywnmmv2iNSRT09bL1GZj3v1UmTqIR+kkno51plF/71vKd2796zefs98IDpXMKYR2nu8vv3xo5u636N11pkrrZWqzVp7/HmOmfCwQAACgAssVAABABZYrAACACixXAAAAFViuAAAAKrBcAQAAVGC5AgAAqMByBQAAUIHlCgAAoIKYUtpcEOMyhLBcPVyMfiLu7YfDcJdus76l25x2bu23dK9jVrsjU12RqQ4sXj4N7z5+k6k+yFQHZKorazP14HI1dPxqL729fJ5V++TZp/D964vs3iX12/Q+i+dZtVfpIrt27PqbdJ39RnCoZE4hjH/3Jb3HussJ5pr9ojUkU9PWy9RmY979VJk6iEfpJJ6OdaZRf+9byndu/es3n7PfCA6VzCmEdp7vL798aObut+jddaZK62Vqs1ae/x5jpnwsEAAAoALLFQAAQAWWKwAAgAosVwAAABVYrgAAACqwXAEAAFRguQIAAKjAcgUAAFCB5QoAAKACyxUAAEAFMaW0uSDGZQhhuXr4cwjhz5m9fwoh/FJwlpL6XnuX1v+cUtrPKfwNcyo9U693P/Zcp5hVr/cjU/VqW+o99llkqo+zzC1TLZ1lbpkqre+1d4uZKq2fS+/1s0opjfJPCOHdWPW99h77LFPMqqWft5XeU82q1/tp6e5l6vGeZew5tfbztnKWuWWqpbPMLVMt3U9LvVuc1dx7+1ggAABABZYrAACACsZcrv44Yn2vvUvrS3tvq5X76bX3NvXb6PV+Wrp7mdpdvUztrr6l3ttq5ectre+197bc/fS9t9XSz/Coej/4F1oAAADwMB8LBAAAqMByBQAAUIHlCgAAoALLFQAAQAWWKwAAgAosVwAAABVYrgAAACqwXAEAAFTwQ0nx7+LTtBd+zKr9W/hr2A+H2b1L6nvtXVr/Lfw9/DP9GrObr5TMqfRMvd79BHP9JaX0++w/sCJT09bLVFdnkakOzjK3TLV0lrllqrS+194tZqq0fka912aqaLnaCz+Gk3iaVXuVLrJrS+t77V1af5Ous/sOlcyp9Ey93v0Ec/2f7OIBmZq2Xqa6OotMdXCWuWWqpbPMLVOl9b32bjFTpfUz6r02Uz4WCAAAUIHlCgAAoALLFQAAQAWWKwAAgAosVwAAABVYrgAAACqwXAEAAFRguQIAAKjAcgUAAFBBTCltLohxGUJYrh4uRj8R9/bDYbhLtzGn1px27n1K6Tin0Kx2R6a6IlMdkKmuyFQHZKorazP14HI1dBCP0kk8zaq9ShfhLJ5n9y6p77V3af1Nus4O2FDJnErP1OvdTzDX7BetIZmatl6mujqLTHVwlrllqqWzzC1TpfW99m4xU6X1M+q9NlM+FggAAFCB5QoAAKACyxUAAEAFlisAAIAKLFcAAAAVWK4AAAAqsFwBAABUYLkCAACowHIFAABQgeUKAACggphS2lwQ4zKEsFw9XIx+Iu7th8Nwl25jTq057dz7lNJxTqFZ7Y5MdUWmOiBTXZGpDshUV9Zm6sHlauggHqWTeJpVe5Uuwlk8z+5dUt9r79L6m3SdHbChkjmVnqnXu59grtkvWkMyNW29THV1Fpnq4Cxzy1RLZ5lbpkrre+3dYqZK62fUe22mfCwQAACgAssVAABABZYrAACACixXAAAAFViuAAAAKrBcAQAAVGC5AgAAqMByBQAAUIHlCgAAoALLFQAAQAUxpbS5IMZlCGG5ergY/UTc2w+H4S7dxpxac9q59yml45xCs9odmeqKTHVAproiUx2Qqa6szdSDy9XQQTxKJ/E0q/YqXYSzeJ7du6S+196l9TfpOjtgQyVzKj1Tr3c/wVyzX7SGZGraepnq6iwy1cFZ5papls4yt0yV1vfau8VMldbPqPfaTPlYIAAAQAWWKwAAgAosVwAAABVYrgAAACqwXAEAAFRguQIAAKjAcgUAAFCB5QoAAKCCH0qK//DyH+Hy8kNW7ZNnIVx+yastrd+m95yUzCmE8e++pPfcyFQfZIoptJTv3PrXb/6R3fcxuPzyoZm797z9sJbuXqbWe4yZiimljX8wxrgMISxXDxfZ/0Z+s/1wmP0t3ea0c2u/pXsds9odmeqKTHVg8fJpePfxm0z1QaY6IFNdWZupB5eroeNXe+nt5fOs2ifPPoXvX19k9y6p36b3WTzPqr1KF9m1Y9ffpOvsN4JDJXMKYfy7L+k91l1OMNfsF60hmZq2XqY2G/Pup8rUQTxKJ/F0rDON+nvfUr5z61+/+Zz9RnCoZE4htPN8/6//yt7G3W/Ru+tMldbL1GatPP89xkz5f64AAAAqsFwBAABUYLkCAACowHIFAABQgeUKAACgAssVAABABZYrAACACixXAAAAFViuAAAAKrBcAQAAVBBTSpsLYlyGEJarh4vRT8S9/XAY7tJtzKk1p517n1I6zik0q92Rqa7IVAcWL5+Gdx+/yVQfZKoDMtWVtZl6cLkaOn61l95ePs+qffLsU/j+9UV275L6bXqfxfOs2qt0kV07dv1Nus5+IzhUMqcQxr/7kt5j3eUEc81+0RqSqWnrZWqzMe9+qkwdxKN0Ek/HOtOov/ct5Tu3/vWbz9lvBIdK5hRCO8/3l18+NHP3W/TuOlOl9TK1WSvPf48xUz4WCAAAUIHlCgAAoALLFQAAQAWWKwAAgAosVwAAABVYrgAAACqwXAEAAFRguQIAAKjAcgUAAFBBTCltLohxGUJYrh4uRj8R9/bDYbhLt1nf0m1OO7f2W7rXMavdkamuyFQHFi+fhncfv8lUH2SqAzLVlbWZenC5Gjp+tZfeXj7Pqn3y7FP4/vVFdu+S+m16n8XzrNqrdJFdO3b9TbrOfiM4VDKnEMa/+5LeY93lBHPNftEakqlp62VqszHvfqpMHcSjdBJPxzrTqL/3LeU7t/71m8/ZbwSHSuYUQjvP95dfPjRz91v07jpTpfUytVkrz3+PMVM+FggAAFCB5QoAAKACyxUAAEAFlisAAIAKLFcAAAAVWK4AAAAqsFwBAABUYLkCAACowHIFAABQgeUKAACggphS2lwQ4zKEsFw9XIx+Iu7th8Nwl25jTq057dz7lNJxTqFZ7Y5MdUWmOrB4+TS8+/hNpvogUx2Qqa6szdSDy9XQ8au99PbyeVbtk2efwvevL7J7l9Rv0/ssnmfVXqWL7Nqx62/SdfYbwaGSOYUw/t2X9B7rLieYa/aL1pBMTVsvU5uNefdTZeogHqWTeDrWmUb9vW8p37n1r998zn4jOFQypxDaeb6//PKhmbvfonfXmSqtl6nNWnn+e4yZ8rFAAACACixXAAAAFViuAAAAKrBcAQAAVGC5AgAAqMByBQAAUIHlCgAAoALLFQAAQAWWKwAAgApiSmlzQYzLEMJy9fDnEMKfM3v/FEL4peAsJfW99i6t/zmltJ9T+BvmVHqmXu9+7LlOMate70em6tW21Hvss8hUH2eZW6ZaOsvcMlVa32vvFjNVWj+X3utnlVIa5Z8Qwrux6nvtPfZZpphVSz9vK72nmlWv99PS3cvU4z3L2HNq7edt5Sxzy1RLZ5lbplq6n5Z6tziruff2sUAAAIAKLFcAAAAVjLlc/XHE+l57l9aX9t5WK/fTa+9t6rfR6/20dPcytbt6mdpdfUu9t9XKz1ta32vvbbn76Xtvq6Wf4VH1fvAvtAAAAOBhPhYIAABQgeUKAACgAssVAABABZYrAACACixXAAAAFfx/jO2AMuneKvUAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "fig = plt.figure(figsize=(12, 12))\n", "for i in range(H * W):\n", " ax = fig.add_subplot(H, W, i + 1)\n", " ax.imshow(mask[i].reshape(H, W))\n", " ax.grid(color='k', linestyle='-', linewidth=1)\n", " ax.set_xticks(torch.arange(0.5, W))\n", " ax.set_yticks(torch.arange(0.5, H))\n", " ax.set_xticklabels([])\n", " ax.set_yticklabels([])\n", "fig.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "747678ef-18d0-4e13-969b-da389f7e3479", "metadata": {}, "source": [ "And for the shifted case" ] }, { "cell_type": "code", "execution_count": 4, "id": "01274104-c52a-4740-85bf-67363300322b", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1cAAANYCAYAAAAolBclAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAABB6klEQVR4nO3dwWpcWbYu6jmv8pTNTSSQqGroguE0zM6efUHCegAJ/AJ6hnioega9gEF6ABkbtnevDmqcjeG4k9cFLqpw1sHM07ixxWqsCs2pHCtiTa3vAzcChkdOzaE/FgMiHbmUkgAAAPh9/q9dHwAAAOApsFwBAAAEsFwBAAAEsFwBAAAEsFwBAAAE+Kml+I9He+W/v/hvVbUf/+O3dPLqWXXvj//xW9pPh1W1f0t/ra5trZ+yd2v99/T39M/yW65uvtYyp5TaZtUyp5Tmc/dbmOuvpZQ/Vf+FtT/kZ+V5+nmqMzXNtTWvU/aeW6Za5tR6ppY5pTT93c/ofVumguplarNe3/+eSqZ6fO5P+Tv2Pz//7/Tr1x+zylRr/dx6T5jX0Uzlln+K/fT18/L+3Yuq2r3ju/Tjy8vq3nvHd+kiX1bVXper6trW+il7t9bflpv0rXxtDljLnFJqm1XLnFKaz91vYa4fSymn1X9h7SAflbN8PtWZmubamtcpe88tUy1zaj1Ty5xSmv7uZ/S+LVNB9TK1Wa/vf08lUz0+96f8HXvz9nP68On7rDLVWj+33hPmdTRTPhYIAAAQwHIFAAAQwHIFAAAQwHIFAAAQwHIFAAAQwHIFAAAQwHIFAAAQwHIFAAAQwHIFAAAQwHIFAAAQIJdSNhfkvEoprdYvTyY/Eff202H6Vr7mmlpz2rmPpZTTmkKz2h2Z6opMdUCmuiJTHTh59Sx9+PRdpvowmqkHl6uh09fPy/t3L6pq947v0o8vL6t77x3fpYt8WVV7Xa6qa1vrp+zdWn9bbqofWkMtc0qpbVYtc0ppPne/hblWP7SGDvJROcvnU52paa6teZ2y99wy1TKn1jO1zCml6e9+Ru/bMhVUL1Ob9fr+91Qy1eNzf8rfsTdvP1cvV0NTZqq1fm69J8zraKZ8LBAAACCA5QoAACCA5QoAACCA5QoAACCA5QoAACCA5QoAACCA5QoAACCA5QoAACCA5QoAACBALqVsLsh5lVJarV+eTH4i7u2nw+pvvjennRv9lu4xZrU7MtUVmeqATHVFpjogU10ZzdSDy9XQQT4qZ/m8qva6XKWLfFndu6W+196t9bflpjpgQy1zaj1Tr3e/hblWP7SGZGq79TLV1VlkqoOzLC1TczrL0jLVWt9r7zlmqrV+Qb1HM+VjgQAAAAEsVwAAAAEsVwAAAAEsVwAAAAEsVwAAAAEsVwAAAAEsVwAAAAEsVwAAAAEsVwAAAAEsVwAAAAFyKWVzQc6rlNJq/fJk8hNxbz8dpm/la66pNaed+1hKOa0pNKvdkamuyFQHZKorMtUBmerKaKYeXK6GDvJROcvnVbXX5Spd5Mvq3i31vfZurb8tN9UBG2qZU+uZer37Lcy1+qE1JFPbrZeprs4iUx2cZWmZmtNZlpap1vpee88xU631C+o9mikfCwQAAAhguQIAAAhguQIAAAhguQIAAAhguQIAAAhguQIAAAhguQIAAAhguQIAAAhguQIAAAhguQIAAAiQSymbC3JepZRW65cnk5+Ie/vpMH0rX3NNrTnt3MdSymlNoVntjkx1RaY6IFNdkakOyFRXRjP14HI1dJCPylk+r6q9LlfpIl9W926p77V3a/1tuakO2FDLnFrP1Ovdb2Gu1Q+tIZnabr1MdXUWmergLEvL1JzOsrRMtdb32nuOmWqtX1Dv0Uz5WCAAAEAAyxUAAEAAyxUAAEAAyxUAAEAAyxUAAEAAyxUAAEAAyxUAAEAAyxUAAEAAyxUAAECAXErZXJDzKqW0Wr88mfxE3NtPh9Xf0m1OOzf6Ld1jzGp3ZKorMtUBmeqKTHVAproymqkHl6uhg3xUzvJ5Ve11uUoX+bK6d0t9r71b62/LTXXAhlrm1HqmXu9+C3OtfmgNydR262Wqq7PIVAdnWVqm5nSWpWWqtb7X3nPMVGv9gnqPZsrHAgEAAAJYrgAAAAJYrgAAAAJYrgAAAAJYrgAAAAJYrgAAAAJYrgAAAAJYrgAAAAJYrgAAAAJYrgAAAALkUsrmgpxXKaXV+uXJ5Cfi3n46TN/K11xTa04797GUclpTaFa7I1NdkakOyFRXZKoDMtWV0Uw9uFwNHeSjcpbPq2qvy1W6yJfVvVvqe+3dWn9bbqoDNtQyp9Yz9Xr3W5hr9UNrSKa2Wy9TXZ1Fpjo4y9IyNaezLC1TrfW99p5jplrrF9R7NFM+FggAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABAgl1I2F+S8Simt1i9PJj8R9/bTYfW3dJvTzo1+S/cYs9odmeqKTHVAproiUx2Qqa6MZurB5WroIB+Vs3xeVXtdrtJFvqzu3VLfa+/W+ttyUx2woZY5tZ6p17vfwlyrH1pDMrXdepnq6iwy1cFZlpapOZ1laZlqre+19xwz1Vq/oN6jmfKxQAAAgACWKwAAgACWKwAAgACWKwAAgACWKwAAgACWKwAAgACWKwAAgACWKwAAgACWKwAAgACWKwAAgAC5lLK5IOdVSmm1fvlLSukvlb3/mFL6teEsLfW99m6t/6WUsl9T+Dvm1HqmXu9+6rluY1a93o9MxdXOqffUZ5GpPs6ytEzN6SxLy1Rrfa+955ip1vql9B6fVSllkj8ppQ9T1ffae+qzbGNWc/p559J7W7Pq9X7mdPcy9XTPMvWc5vbzzuUsS8vUnM6ytEzN6X7m1HuOs1p6bx8LBAAACGC5AgAACDDlcvXnCet77d1a39r7seZyP732fkz9Y/R6P3O6e5naXb1M7a5+Tr0fay4/b2t9r70fy91vv/djzelneFK9H/wHLQAAAHiYjwUCAAAEsFwBAAAEsFwBAAAEsFwBAAAEsFwBAAAEsFwBAAAEsFwBAAAEsFwBAAAE+Kml+I9He+W/v/hvVbUf/+O3dPLqWXXvj//xW9pPh1W1f0t/ra5trZ+yd2v99/T39M/yW65uvtYyp5TaZtUyp5Tmc/dbmOuvpZQ/Vf+FtT/kZ+V5+nmqMzXNtTWvU/aeW6Za5tR6ppY5pTT93c/ofVumguplarNe3/+eSqZ6fO5P+Tv2Pz//7/Tr1x+zylRr/dx6T5jX0UzlUkp1k9PXz8v7dy+qaveO79KPLy+re+8d36WLfFlVe12uqmtb66fs3Vp/W27St/K1OWAtc0qpbVYtc0ppPne/hbl+LKWcVv+FtYN8VM7y+VRnappra16n7D23TLXMqfVMLXNKafq7n9H7tkwF1cvUZr2+/z2VTPX43J/yd+zN28/pw6fvs8pUa/3cek+Y19FM+VggAABAAMsVAABAAMsVAABAAMsVAABAAMsVAABAAMsVAABAAMsVAABAAMsVAABAAMsVAABAAMsVAABAgFxK2VyQ8yqltFq/PJn8RNzbT4fpW/maa2rNaec+llJOawrNandkqisy1QGZ6opMdeDk1bP04dN3merDaKYeXK6GTl8/L+/fvaiq3Tu+Sz++vKzuvXd8ly7yZVXtdbmqrm2tn7J3a/1tual+aA21zCmltlm1zCml+dz9FuZa/dAaOshH5SyfT3Wmprm25nXK3nPLVMucWs/UMqeUpr/7Gb1vy1RQvUxt1uv731PJVI/P/Sl/x968/Vy9XA1NmanW+rn1njCvo5nysUAAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAP7UU/4//+L/T2//n/62svmuo/f/ridE2p5TaZmVOPZkyr94L4szp7s1qszndvfftf21Ody9TT9NUvzP/o/x/jzkOG2w7r7mUsvGv5ZxXKaXV+uVJw3+R32k/HVZ/87057dzot3SPMavdkamuyFQHZKorMtUBmerKaKYeXK6GDvJROcvnVbXX5Spd5Mvq3i31vfZurb8tN9UBG2qZU+uZer37Lcy1+qE1JFPbrZeprs4iUx2cZWmZmtNZlpap1vpee88xU631C+o9min/zxUAAEAAyxUAAEAAyxUAAEAAyxUAAEAAyxUAAEAAyxUAAEAAyxUAAEAAyxUAAEAAyxUAAEAAyxUAAECAXErZXJDzKqW0Wr88mfxE3NtPh+lb+Zpras1p5z6WUk5rCs1qd2SqKzLVAZnqikx1QKa6MpqpB5eroYN8VM7yeVXtdblKF/myundLfa+9W+tvy011wIZa5tR6pl7vfgtzrX5oDcnUdutlqquzyFQHZ1lapuZ0lqVlqrW+195zzFRr/YJ6j2bKxwIBAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAAC5FLK5oKcVyml1frlyeQn4t5+Oqz+lm5z2rnRb+keY1a7I1NdkakOyFRXZKoDMtWV0Uw9uFwNHeSjcpbPq2qvy1W6yJfVvVvqe+3dWn9bbqoDNtQyp9Yz9Xr3W5hr9UNrSKa2Wy9TXZ1Fpjo4y9IyNaezLC1TrfW99p5jplrrF9R7NFM+FggAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABAgl1I2F+S8Simt1i9PJj8R9/bTYfpWvuaaWnPauY+llNOaQrPaHZnqikx1QKa6IlMdkKmujGbqweVq6CAflbN8XlV7Xa7SRb6s7t1S32vv1vrbclMdsKGWObWeqde738Jcqx9aQzK13XqZ6uosMtXBWZaWqTmdZWmZaq3vtfccM9Vav6Deo5nysUAAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAuZSyuSDnVUpptX55MvmJuLefDqu/pducdm70W7rHmNXuyFRXZKoDMtUVmeqATHVlNFMPLldDB/monOXzqtrrcpUu8mV175b6Xnu31t+Wm+qADbXMqfVMvd79FuZa/dAakqnt1stUV2eRqQ7OsrRMzeksS8tUa32vveeYqdb6BfUezZSPBQIAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAATIpZTNBTmvUkqr9cuTyU/Evf10mL6Vr7mm1px27mMp5bSm0Kx2R6a6IlMdkKmuyFQHZKoro5l6cLkaOshH5SyfV9Vel6t0kS+re7fU99q7tf623FQHbKhlTq1n6vXutzDX6ofWkExtt16mujqLTHVwlqVlak5nWVqmWut77T3HTLXWL6j3aKZ8LBAAACCA5QoAACCA5QoAACCA5QoAACCA5QoAACCA5QoAACCA5QoAACCA5QoAACCA5QoAACBALqVsLsh5lVJarV/+klL6S2XvP6aUfm04S0t9r71b638ppezXFP6OObWeqde7n3qu25hVr/cjU3G1c+o99Vlkqo+zLC1TczrL0jLVWt9r7zlmqrV+Kb3HZ1VKmeRPSunDVPW99p76LNuY1Zx+3rn03taser2fOd29TD3ds0w9p7n9vHM5y9IyNaezLC1Tc7qfOfWe46yW3tvHAgEAAAJYrgAAAAJMuVz9ecL6Xnu31rf2fqy53E+vvR9T/xi93s+c7l6mdlcvU7urn1Pvx5rLz9ta32vvx3L32+/9WHP6GZ5U7wf/QQsAAAAe5mOBAAAAASxXAAAAASxXAAAAASxXAAAAASxXAAAAASxXAAAAASxXAAAAASxXAAAAASxXAAAAAX5qKf5Dflaep5+rav+W/pr202F175b6Xnu31n9Pf0//LL/l6uZrLXNqPVOvd7+Fuf5aSvlT9V9Yk6nt1stUV2eRqQ7OsrRMzeksS8tUa32vveeYqdb6BfUezVTTcvU8/ZzO8nlV7XW5qq5tre+1d2v9bbmp7jvUMqfWM/V691uY639WFw/I1HbrZaqrs8hUB2dZWqbmdJalZaq1vtfec8xUa/2Ceo9myscCAQAAAliuAAAAAliuAAAAAliuAAAAAliuAAAAAliuAAAAAliuAAAAAliuAAAAAliuAAAAAuRSyuaCnFcppdX65cnkJ+LefjpM38rXXFNrTjv3sZRyWlNoVrsjU12RqQ7IVFdkqgMy1ZXRTD24XA0d5KNyls+raq/LVbrIl9W9W+p77d1af1tuqgM21DKn1jP1evdbmGv1Q2tIprZbL1NdnUWmOjjL0jI1p7MsLVOt9b32nmOmWusX1Hs0Uz4WCAAAEMByBQAAEMByBQAAEMByBQAAEMByBQAAEMByBQAAEMByBQAAEMByBQAAEMByBQAAEOCnluJ/e/WP9O7dv1fV7h2n9O5/1dX+Vz0xWuaUUtuszKkvLXNtzeuUvZdmTndvVpvN6e69b/9rc7p7mXqapvqdefP2H487EP/StvOaSykb/2LOeZVSWq1fnlT/F/nd9tNh+la+5ppac9q5j6WU05pCs9odmeqKTHVAproiUx04efUsffj0Xab6MJqpB5erodPXz8v7dy+qaveO79KPLy+re+8d36WLfFlVe12uqmtb66fs3Vp/W26qH1pDLXNKqW1WLXNKaT53v4W5Vj+0hg7yUTnL51OdqWmurXmdsvfcMtUyp9Yztcwppenvfkbv2zIVVC9Tm/X6/vdUMtXjc3/K37E3bz9XL1dDU2aqtX5uvSfM62im/D9XAAAAASxXAAAAASxXAAAAASxXAAAAASxXAAAAASxXAAAAASxXAAAAASxXAAAAASxXAAAAAXIpZXNBzquU0mr98mTyE3FvPx1Wf/O9Oe3c6Ld0jzGr3ZGprshUB2SqKzLVgZNXz9KHT99lqg+jmXpwuRo6ff28vH/3oqp27/gu/fjysrr33vFdusiXVbXX5aq6trV+yt6t9bflpvqhNdQyp5TaZtUyp5Tmc/dbmGv1Q2voIB+Vs3w+1Zma5tqa1yl7zy1TLXNqPVPLnFKa/u5n9L4tU0H1MrVZr+9/TyVTPT73p/wde/P2c/VyNTRlplrr59Z7wryOZsrHAgEAAAJYrgAAAAJYrgAAAAJYrgAAAAJYrgAAAAJYrgAAAAJYrgAAAAJYrgAAAAJYrgAAAAJYrgAAAALkUsrmgpxXKaXV+uXJ5Cfi3n46TN/K11xTa04797GUclpTaFa7I1NdkakOyFRXZKoDJ6+epQ+fvstUH0Yz9eByNXT6+nl5/+5FVe3e8V368eVlde+947t0kS+raq/LVXVta/2UvVvrb8tN9UNrqGVOKbXNqmVOKc3n7rcw1+qH1tBBPipn+XyqMzXNtTWvU/aeW6Za5tR6ppY5pTT93c/ofVumguplarNe3/+eSqZ6fO5P+Tv25u3n6uVqaMpMtdbPrfeEeR3NlI8FAgAABLBcAQAABLBcAQAABLBcAQAABLBcAQAABLBcAQAABLBcAQAABLBcAQAABLBcAQAABMillM0FOa9SSqv1y5PJT8S9/XRY/c335rRzo9/SPcasdkemuiJTHZCprshUB05ePUsfPn2XqT6MZurB5Wro9PXz8v7di6raveO79OPLy+ree8d36SJfVtVel6vq2tb6KXu31t+Wm+qH1lDLnFJqm1XLnFKaz91vYa7VD62hg3xUzvL5VGdqmmtrXqfsPbdMtcyp9Uwtc0pp+ruf0fu2TAXVy9Rmvb7/PZVM9fjcn/J37M3bz9XL1dCUmWqtn1vvCfM6mikfCwQAAAhguQIAAAhguQIAAAhguQIAAAhguQIAAAhguQIAAAhguQIAAAhguQIAAAhguQIAAAhguQIAAAiQSymbC3JepZRW65cnk5+Ie/vpMH0rX3NNrTnt3MdSymlNoVntjkx1RaY6IFNdkakOyFRXRjP14HI1dJCPylk+r6q9LlfpIl9W926p77V3a/1tuakO2FDLnFrP1Ovdb2Gu1Q+tIZnabr1MdXUWmergLEvL1JzOsrRMtdb32nuOmWqtX1Dv0Uz5WCAAAEAAyxUAAEAAyxUAAEAAyxUAAEAAyxUAAEAAyxUAAEAAyxUAAEAAyxUAAEAAyxUAAEAAyxUAAECAXErZXJDzKqW0Wr88mfxE3NtPh+lb+Zpras1p5z6WUk5rCs1qd2SqKzLVAZnqikx1QKa6MpqpB5eroYN8VM7yeVXtdblKF/myundLfa+9W+tvy011wIZa5tR6pl7vfgtzrX5oDcnUdutlqquzyFQHZ1lapuZ0lqVlqrW+195zzFRr/YJ6j2bKxwIBAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAAC5FLK5oKcVyml1frlLymlv1T2/mNK6deGs7TU99q7tf6XUsp+TeHvmFPrmXq9+6nnuo1Z9Xo/MhVXO6feU59Fpvo4y9IyNaezLC1TrfW99p5jplrrl9J7fFallEn+pJQ+TFXfa++pz7KNWc3p551L723Nqtf7mdPdy9TTPcvUc5rbzzuXsywtU3M6y9IyNaf7mVPvOc5q6b19LBAAACCA5QoAACDAlMvVnyes77V3a31r78eay/302vsx9Y/R6/3M6e5lanf1MrW7+jn1fqy5/Lyt9b32fix3v/3ejzWnn+FJ9X7wH7QAAADgYT4WCAAAEMByBQAAEMByBQAAEMByBQAAEMByBQAAEMByBQAAEMByBQAAEMByBQAAEOCnluI/5Gflefq5qvZv6a9pPx1W926p77V3a/339Pf0z/Jbrm6+1jKn1jP1evdbmOuvpZQ/Vf+FNZnabr1MdXUWmergLEvL1JzOsrRMtdb32nuOmWqtX1Dv0Uw1LVfP08/pLJ9X1V6Xq+ra1vpee7fW35ab6r5DLXNqPVOvd7+Fuf5ndfGATG23Xqa6OotMdXCWpWVqTmdZWqZa63vtPcdMtdYvqPdopnwsEAAAIIDlCgAAIIDlCgAAIIDlCgAAIIDlCgAAIIDlCgAAIIDlCgAAIIDlCgAAIIDlCgAAIIDlCgAAIEAupWwuyHmVUlqtX55MfiLu7afD9K18zTW15rRzH0sppzWFZrU7MtUVmeqATHVFpjogU10ZzdSDy9XQQT4qZ/m8qva6XKWLfFndu6W+196t9bflpjpgQy1zaj1Tr3e/hblWP7SGZGq79TLV1VlkqoOzLC1TczrL0jLVWt9r7zlmqrV+Qb1HM+VjgQAAAAEsVwAAAAEsVwAAAAEsVwAAAAEsVwAAAAEsVwAAAAEsVwAAAAEsVwAAAAEsVwAAAAFyKWVzQc6rlNJq/fJk8hNxbz8dVn9Ltznt3Oi3dI8xq92Rqa7IVAdkqisy1YGTV8/Sh0/fZaoPo5l6cLkaOn39vLx/96Kqdu/4Lv348rK6997xXbrIl1W11+Wqura1fsrerfW35ab6oTXUMqeU2mbVMqeU5nP3W5hr9UNr6CAflbN8PtWZmubamtcpe88tUy1zaj1Ty5xSmv7uZ/S+LVNB9TK1Wa/vf08lUz0+96f8HXvz9nP1cjU0ZaZa6+fWe8K8jmbKxwIBAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAAC5FLK5oKcVyml1frlyeQn4t5+OkzfytdcU2tOO/exlHJaU2hWuyNTXZGpDshUV2SqAyevnqUPn77LVB9GM/XgcjV0+vp5ef/uRVXt3vFd+vHlZXXvveO7dJEvq2qvy1V1bWv9lL1b62/LTfVDa6hlTim1zaplTinN5+63MNfqh9bQQT4qZ/l8qjM1zbU1r1P2nlumWubUeqaWOaU0/d3P6H1bpoLqZWqzXt//nkqmenzuT/k79ubt5+rlamjKTLXWz633hHkdzZSPBQIAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAATIpZTNBTmvUkqr9cuTyU/Evf10mL6Vr7mm1px27mMp5bSm0Kx2R6a6IlMdkKmuyFQHTl49Sx8+fZepPoxm6sHlauj09fPy/t2Lqtq947v048vL6t57x3fpIl9W1V6Xq+ra1vope7fW35ab6ofWUMucUmqbVcucUprP3W9hrtUPraGDfFTO8vlUZ2qaa2tep+w9t0y1zKn1TC1zSmn6u5/R+7ZMBdXL1Ga9vv89lUz1+Nyf8nfszdvP1cvV0JSZaq2fW+8J8zqaKR8LBAAACGC5AgAACGC5AgAACGC5AgAACGC5AgAACGC5AgAACGC5AgAACGC5AgAACGC5AgAACJBLKZsLcl6llFbrlyeTn4h7++mw+pvvzWnnRr+le4xZ7Y5MdUWmOiBTXZGpDpy8epY+fPouU30YzdSDy9XQ6evn5f27F1W1e8d36ceXl9W9947v0kW+rKq9LlfVta31U/Zurb8tN9UPraGWOaXUNquWOaU0n7vfwlyrH1pDB/monOXzqc7UNNfWvE7Ze26ZaplT65la5pTS9Hc/o/dtmQqql6nNen3/eyqZ6vG5P+Xv2Ju3n6uXq6EpM9VaP7feE+Z1NFM+FggAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABAgl1I2F+S8Simt1i9PJj8R9/bTYfpWvuaaWnPauY+llNOaQrPaHZnqikx1QKa6IlMdkKmujGbqweVq6CAflbN8XlV7Xa7SRb6s7t1S32vv1vrbclMdsKGWObWeqde738Jcqx9aQzK13XqZ6uosMtXBWZaWqTmdZWmZaq3vtfccM9Vav6Deo5nysUAAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAuZSyuSDnVUpptX55MvmJuLefDqu/pducdm70W7rHmNXuyFRXZKoDMtUVmeqATHVlNFMPLldDB/monOXzqtrrcpUu8mV175b6Xnu31t+Wm+qADbXMqfVMvd79FuZa/dAakqnt1stUV2eRqQ7OsrRMzeksS8tUa32vveeYqdb6BfUezZSPBQIAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAATIpZTNBTmvUkqr9ctfUkp/qez9x5TSrw1naanvtXdr/S+llP2awt8xp9Yz9Xr3U891G7Pq9X5kKq52Tr2nPotM9XGWpWVqTmdZWqZa63vtPcdMtdYvpff4rEopk/xJKX2Yqr7X3lOfZRuzmtPPO5fe25pVr/czp7uXqad7lqnnNLefdy5nWVqm5nSWpWVqTvczp95znNXSe/tYIAAAQADLFQAAQIApl6s/T1jfa+/W+tbejzWX++m192PqH6PX+5nT3cvU7uplanf1c+r9WHP5eVvre+39WO5++70fa04/w5Pq/eA/aAEAAMDDfCwQAAAggOUKAAAggOUKAAAggOUKAAAggOUKAAAggOUKAAAggOUKAAAggOUKAAAgwE8txX/Iz8rz9HNV7d/SX9N+Oqzu3VLfa+/W+u/p7+mf5bdc3XytZU6tZ+r17rcw119LKX+q/gtrMrXdepnq6iwy1cFZlpapOZ1laZlqre+19xwz1Vq/oN6jmWparp6nn9NZPq+qvS5X1bWt9b32bq2/LTfVfYda5tR6pl7vfgtz/c/q4gGZ2m69THV1Fpnq4CxLy9SczrK0TLXW99p7jplqrV9Q79FM+VggAABAAMsVAABAAMsVAABAAMsVAABAAMsVAABAAMsVAABAAMsVAABAAMsVAABAAMsVAABAAMsVAABAgFxK2VyQ8yqltFq/PJn8RNzbT4fpW/maa2rNaec+llJOawrNandkqisy1QGZ6opMdUCmujKaqQeXq6GDfFTO8nlV7XW5Shf5srp3S32vvVvrb8tNdcCGWubUeqZe734Lc61+aA3J1HbrZaqrs8hUB2dZWqbmdJalZaq1vtfec8xUa/2Ceo9myscCAQAAAliuAAAAAliuAAAAAliuAAAAAliuAAAAAliuAAAAAliuAAAAAliuAAAAAvzUUvxvr/6R3r3796raveOU3v2vutr/qidGy5xSapuVOfWlZa6teZ2y99LM6e7NarM53b337X9tTncvU0/TVL8zb97+43EH4l/adl5zKWXjX8w5r1JKq/XLk+r/Ir/bfjqs/pZuc9q50W/pHmNWuyNTXZGpDshUV2SqAyevnqUPn77LVB9GM/XgcjV0+vp5ef/uRVXt3vFd+vHlZXXvveO7dJEvq2qvy1V1bWv9lL1b62/LTfVDa6hlTim1zaplTinN5+63MNfqh9bQQT4qZ/l8qjM1zbU1r1P2nlumWubUeqaWOaU0/d3P6H1bpoLqZWqzXt//nkqmenzuT/k79ubt5+rlamjKTLXWz633hHkdzZT/5woAACCA5QoAACCA5QoAACCA5QoAACCA5QoAACCA5QoAACCA5QoAACCA5QoAACCA5QoAACCA5QoAACBALqVsLsh5lVJarV+eTH4i7u2nw/StfM01tea0cx9LKac1hWa1OzLVFZnqgEx1RaY6cPLqWfrw6btM9WE0Uw8uV0Onr5+X9+9eVNXuHd+lH19eVvfeO75LF/myqva6XFXXttZP2bu1/rbcVD+0hlrmlFLbrFrmlNJ87n4Lc61+aA0d5KNyls+nOlPTXFvzOmXvuWWqZU6tZ2qZU0rT3/2M3rdlKqhepjbr9f3vqWSqx+f+lL9jb95+rl6uhqbMVGv93HpPmNfRTPlYIAAAQADLFQAAQADLFQAAQADLFQAAQADLFQAAQADLFQAAQADLFQAAQADLFQAAQADLFQAAQIBcStlckPMqpbRavzyZ/ETc20+H1d98b047N/ot3WPMandkqisy1QGZ6opMdeDk1bP04dN3merDaKYeXK6GTl8/L+/fvaiq3Tu+Sz++vKzuvXd8ly7yZVXtdbmqrm2tn7J3a/1tual+aA21zCmltlm1zCml+dz9FuZa/dAaOshH5SyfT3Wmprm25nXK3nPLVMucWs/UMqeUpr/7Gb1vy1RQvUxt1uv731PJVI/P/Sl/x968/Vy9XA1NmanW+rn1njCvo5nysUAAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAuZSyuSDnVUpptX55MvmJuLefDtO38jXX1JrTzn0spZzWFJrV7shUV2SqAzLVFZnqwMmrZ+nDp+8y1YfRTD24XA2dvn5e3r97UVW7d3yXfnx5Wd177/guXeTLqtrrclVd21o/Ze/W+ttyU/3QGmqZU0pts2qZU0rzufstzLX6oTV0kI/KWT6f6kxNc23N65S955apljm1nqllTilNf/czet+WqaB6mdqs1/e/p5KpHp/7U/6OvXn7uXq5GpoyU631c+s9YV5HM+VjgQAAAAEsVwAAAAEsVwAAAAEsVwAAAAEsVwAAAAEsVwAAAAEsVwAAAAEsVwAAAAEsVwAAAAFyKWVzQc6rlNJq/fJk8hNxbz8dVn/zvTnt3Oi3dI8xq92Rqa7IVAdkqisy1QGZ6spoph5croYO8lE5y+dVtdflKl3ky+reLfW99m6tvy031QEbaplT65l6vfstzLX6oTUkU9utl6muziJTHZxlaZma01mWlqnW+l57zzFTrfUL6j2aKR8LBAAACGC5AgAACGC5AgAACGC5AgAACGC5AgAACGC5AgAACGC5AgAACGC5AgAACGC5AgAACGC5AgAACJBLKZsLcl6llFbrlyeTn4h7++kwfStfc02tOe3cx1LKaU2hWe2OTHVFpjogU12RqQ7IVFdGM/XgcjV0kI/KWT6vqr0uV+kiX1b3bqnvtXdr/W25qQ7YUMucWs/U691vYa7VD60hmdpuvUx1dRaZ6uAsS8vUnM6ytEy11vfae46Zaq1fUO/RTPlYIAAAQADLFQAAQADLFQAAQADLFQAAQADLFQAAQADLFQAAQADLFQAAQADLFQAAQADLFQAAQIBcStlckPMqpbRav/wlpfSXyt5/TCn92nCWlvpee7fW/1JK2a8p/B1zaj1Tr3c/9Vy3Mate70em4mrn1Hvqs8hUH2dZWqbmdJalZaq1vtfec8xUa/1Seo/PqpQyyZ+U0oep6nvtPfVZtjGrOf28c+m9rVn1ej9zunuZerpnmXpOc/t553KWpWVqTmdZWqbmdD9z6j3HWS29t48FAgAABLBcAQAABJhyufrzhPW99m6tb+39WHO5n157P6b+MXq9nzndvUztrl6mdlc/p96PNZeft7W+196P5e633/ux5vQzPKneD/6DFgAAADzMxwIBAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAAC/NRS/If8rDxPP1fV/i39Ne2nw+reLfW99m6t/57+nv5ZfsvVzdda5tR6pl7vfgtz/bWU8qfqv7AmU9utl6muziJTHZxlaZma01mWlqnW+l57zzFTrfUL6j2aqabl6nn6OZ3l86ra63JVXdta32vv1vrbclPdd6hlTq1n6vXutzDX/6wuHpCp7dbLVFdnkakOzrK0TM3pLEvLVGt9r73nmKnW+gX1Hs2UjwUCAAAEsFwBAAAEsFwBAAAEsFwBAAAEsFwBAAAEsFwBAAAEsFwBAAAEsFwBAAAEsFwBAAAEyKWUzQU5r1JKq/XLk8lPxL39dJi+la+5ptacdu5jKeW0ptCsdkemuiJTHZCprshUB2SqK6OZenC5GjrIR+Usn1fVXperdJEvq3u31Pfau7X+ttxUB2yoZU6tZ+r17rcw1+qH1pBMbbdepro6i0x1cJalZWpOZ1laplrre+09x0y11i+o92imfCwQAAAggOUKAAAggOUKAAAggOUKAAAggOUKAAAggOUKAAAggOUKAAAggOUKAAAggOUKAAAgwE8txf/26h/p3bt/r6rdO07p3f+qq/2vemK0zCmltlmZU19a5tqa1yl7L82c7t6sNpvT3Xvf/tfmdPcy9TRN9Tvz5u0/Hncg/qVt5zWXUjb+xZzzKqW0Wr88qf4v8rvtp8P0rXzNNbXmtHMfSymnNYVmtTsy1RWZ6oBMdUWmOnDy6ln68Om7TPVhNFMPLldDp6+fl/fvXlTV7h3fpR9fXlb33ju+Sxf5sqr2ulxV17bWT9m7tf623FQ/tIZa5pRS26xa5pTSfO5+C3OtfmgNHeSjcpbPpzpT01xb8zpl77llqmVOrWdqmVNK09/9jN63ZSqoXqY26/X976lkqsfn/pS/Y2/efq5eroamzFRr/dx6T5jX0Uz5f64AAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAAC5FLK5oKcVyml1frlyeQn4t5+Oqz+5ntz2rnRb+keY1a7I1NdkakOyFRXZKoDJ6+epQ+fvstUH0Yz9eByNXT6+nl5/+5FVe3e8V368eVlde+947t0kS+raq/LVXVta/2UvVvrb8tN9UNrqGVOKbXNqmVOKc3n7rcw1+qH1tBBPipn+XyqMzXNtTWvU/aeW6Za5tR6ppY5pTT93c/ofVumguplarNe3/+eSqZ6fO5P+Tv25u3n6uVqaMpMtdbPrfeEeR3NlI8FAgAABLBcAQAABLBcAQAABLBcAQAABLBcAQAABLBcAQAABLBcAQAABLBcAQAABLBcAQAABLBcAQAABMillM0FOa9SSqv1y5PJT8S9/XSYvpWvuabWnHbuYynltKbQrHZHproiUx2Qqa7IVAdOXj1LHz59l6k+jGbqweVq6PT18/L+3Yuq2r3ju/Tjy8vq3nvHd+kiX1bVXper6trW+il7t9bflpvqh9ZQy5xSaptVy5xSms/db2Gu1Q+toYN8VM7y+VRnappra16n7D23TLXMqfVMLXNKafq7n9H7tkwF1cvUZr2+/z2VTPX43J/yd+zN28/Vy9XQlJlqrZ9b7wnzOpopHwsEAAAIYLkCAAAIYLkCAAAIYLkCAAAIYLkCAAAIYLkCAAAIYLkCAAAIYLkCAAAIYLkCAAAIkEspmwtyXqWUVuuXJ5OfiHv76bD6m+/NaedGv6V7jFntjkx1RaY6IFNdkakOnLx6lj58+i5TfRjN1IPL1dDp6+fl/bsXVbV7x3fpx5eX1b33ju/SRb6sqr0uV9W1rfVT9m6tvy031Q+toZY5pdQ2q5Y5pTSfu9/CXKsfWkMH+aic5fOpztQ019a8Ttl7bplqmVPrmVrmlNL0dz+j922ZCqqXqc16ff97Kpnq8bk/5e/Ym7efq5eroSkz1Vo/t94T5nU0Uz4WCAAAEMByBQAAEMByBQAAEMByBQAAEMByBQAAEMByBQAAEMByBQAAEMByBQAAEMByBQAAEMByBQAAECCXUjYX5LxKKa3WL08mPxH39tNh+la+5ppac9q5j6WU05pCs9odmeqKTHVAproiUx2Qqa6MZurB5WroIB+Vs3xeVXtdrtJFvqzu3VLfa+/W+ttyUx2woZY5tZ6p17vfwlyrH1pDMrXdepnq6iwy1cFZlpapOZ1laZlqre+19xwz1Vq/oN6jmfKxQAAAgACWKwAAgACWKwAAgACWKwAAgACWKwAAgACWKwAAgACWKwAAgACWKwAAgACWKwAAgAC5lLK5IOdVSmm1fnky+Ym4t58Oq7+l25x2bvRbuseY1e7IVFdkqgMy1RWZ6oBMdWU0Uw8uV0MH+aic5fOq2utylS7yZXXvlvpee7fW35ab6oANtcyp9Uy93v0W5lr90BqSqe3Wy1RXZ5GpDs6ytEzN6SxLy1Rrfa+955ip1voF9R7NlI8FAgAABLBcAQAABLBcAQAABLBcAQAABLBcAQAABLBcAQAABLBcAQAABLBcAQAABLBcAQAABLBcAQAABMillM0FOa9SSqv1y19SSn+p7P3HlNKvDWdpqe+1d2v9L6WU/ZrC3zGn1jP1evdTz3Ubs+r1fmQqrnZOvac+i0z1cZalZWpOZ1laplrre+09x0y11i+l9/isSimT/EkpfZiqvtfeU59lG7Oa0887l97bmlWv9zOnu5epp3uWqec0t593LmdZWqbmdJalZWpO9zOn3nOc1dJ7+1ggAABAAMsVAABAgCmXqz9PWN9r79b61t6PNZf76bX3Y+ofo9f7mdPdy9Tu6mVqd/Vz6v1Yc/l5W+t77f1Y7n77vR9rTj/Dk+r94D9oAQAAwMN8LBAAACCA5QoAACCA5QoAACCA5QoAACCA5QoAACCA5QoAACCA5QoAACCA5QoAACDATy3Ff8jPyvP0c1Xt39Jf0346rO7dUt9r79b67+nv6Z/lt1zdfK1lTq1n6vXutzDXX0spf6r+C2sytd16merqLDLVwVmWlqk5nWVpmWqt77X3HDPVWr+g3qOZalqunqef01k+r6q9LlfVta31vfZurb8tN9V9h1rm1HqmXu9+C3P9z+riAZnabr1MdXUWmergLEvL1JzOsrRMtdb32nuOmWqtX1Dv0Uz5WCAAAEAAyxUAAEAAyxUAAEAAyxUAAEAAyxUAAEAAyxUAAEAAyxUAAEAAyxUAAEAAyxUAAEAAyxUAAECAXErZXJDzKqW0Wr88mfxE3NtPh+lb+Zpras1p5z6WUk5rCs1qd2SqKzLVAZnqikx1QKa6MpqpB5eroYN8VM7yeVXtdblKF/myundLfa+9W+tvy011wIZa5tR6pl7vfgtzrX5oDcnUdutlqquzyFQHZ1lapuZ0lqVlqrW+195zzFRr/YJ6j2bKxwIBAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAACWK4AAAAC5FLK5oKcVyml1frlyeQn4t5+Oqz+lm5z2rnRb+keY1a7I1NdkakOyFRXZKoDMtWV0Uw9uFwNHeSjcpbPq2qvy1W6yJfVvVvqe+3dWn9bbqoDNtQyp9Yz9Xr3W5hr9UNrSKa2Wy9TXZ1Fpjo4y9IyNaezLC1TrfW99p5jplrrF9R7NFM+FggAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABAgl1I2F+S8Simt1i9PJj8R9/bTYfpWvuaaWnPauY+llNOaQrPaHZnqikx1QKa6IlMdkKmujGbqweVq6CAflbN8XlV7Xa7SRb6s7t1S32vv1vrbclMdsKGWObWeqde738Jcqx9aQzK13XqZ6uosMtXBWZaWqTmdZWmZaq3vtfccM9Vav6Deo5nysUAAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAuZSyuSDnVUpptX55MvmJuLefDqu/pducdm70W7rHmNXuyFRXZKoDMtUVmeqATHVlNFMPLldDB/monOXzqtrrcpUu8mV175b6Xnu31t+Wm+qADbXMqfVMvd79FuZa/dAakqnt1stUV2eRqQ7OsrRMzeksS8tUa32vveeYqdb6BfUezZSPBQIAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAATIpZTNBTmvUkqr9cuTyU/Evf10mL6Vr7mm1px27mMp5bSm0Kx2R6a6IlMdkKmuyFQHZKoro5l6cLkaOshH5SyfV9Vel6t0kS+re7fU99q7tf623FQHbKhlTq1n6vXutzDX6ofWkExtt16mujqLTHVwlqVlak5nWVqmWut77T3HTLXWL6j3aKZ8LBAAACCA5QoAACCA5QoAACCA5QoAACCA5QoAACCA5QoAACCA5QoAACCA5QoAACCA5QoAACCA5QoAACBALqVsLsh5lVJarV+eTH4i7u2nw/StfM01tea0cx9LKac1hWa1OzLVFZnqgEx1RaY6cPLqWfrw6btM9WE0Uw8uV0Onr5+X9+9eVNXuHd+lH19eVvfeO75LF/myqva6XFXXttZP2bu1/rbcVD+0hlrmlFLbrFrmlNJ87n4Lc61+aA0d5KNyls+nOlPTXFvzOmXvuWWqZU6tZ2qZU0rT3/2M3rdlKqhepjbr9f3vqWSqx+f+lL9jb95+rl6uhqbMVGv93HpPmNfRTPlYIAAAQADLFQAAQADLFQAAQADLFQAAQADLFQAAQADLFQAAQADLFQAAQADLFQAAQADLFQAAQIBcStlckPMqpbRavzyZ/ETc20+H1d98b047N/ot3WPMandkqisy1QGZ6opMdeDk1bP04dN3merDaKYeXK6GTl8/L+/fvaiq3Tu+Sz++vKzuvXd8ly7yZVXtdbmqrm2tn7J3a/1tual+aA21zCmltlm1zCml+dz9FuZa/dAaOshH5SyfT3Wmprm25nXK3nPLVMucWs/UMqeUpr/7Gb1vy1RQvUxt1uv731PJVI/P/Sl/x968/Vy9XA1NmanW+rn1njCvo5nysUAAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAlisAAIAAuZSyuSDnVUpptX75S0rpL5W9/5hS+rXhLC31vfZurf+llLJfU/g75tR6pl7vfuq5bmNWvd6PTMXVzqn31GeRqT7OsrRMzeksS8tUa32vveeYqdb6pfQen1UpZZI/KaUPU9X32nvqs2xjVnP6eefSe1uz6vV+5nT3MvV0zzL1nOb2887lLEvL1JzOsrRMzel+5tR7jrNaem8fCwQAAAhguQIAAAgw5XL15wnre+3dWt/a+7Hmcj+99n5M/WP0ej9zunuZ2l29TO2ufk69H2suP29rfa+9H8vdb7/3Y83pZ3hSvR/8By0AAAB4mI8FAgAABLBcAQAABLBcAQAABLBcAQAABLBcAQAABLBcAQAABLBcAQAABLBcAQAABPippfgP+Vl5nn6uqv1b+mvaT4fVvVvqe+3dWv89/T39s/yWq5uvtcyp9Uy93v0W5vprKeVP1X9hTaa2Wy9TXZ1Fpjo4y9IyNaezLC1TrfW99p5jplrrF9R7NFNNy9Xz9HM6y+dVtdflqrq2tb7X3q31t+Wmuu9Qy5xaz9Tr3W9hrv9ZXTwgU9utl6muziJTHZxlaZma01mWlqnW+l57zzFTrfUL6j2aKR8LBAAACGC5AgAACGC5AgAACGC5AgAACGC5AgAACGC5AgAACGC5AgAACGC5AgAACGC5AgAACJBLKZsLcl6llFbrlyeTn4h7++kwfStfc02tOe3cx1LKaU2hWe2OTHVFpjogU12RqQ7IVFdGM/XgcjV0kI/KWT6vqr0uV+kiX1b3bqnvtXdr/W25qQ7YUMucWs/U691vYa7VD60hmdpuvUx1dRaZ6uAsS8vUnM6ytEy11vfae46Zaq1fUO/RTPlYIAAAQADLFQAAQADLFQAAQADLFQAAQADLFQAAQADLFQAAQADLFQAAQADLFQAAQADLFQAAQADLFQAAQIBcStlckPMqpbRavzyZ/ETc20+H6Vv5mmtqzWnnPpZSTmsKzWp3ZKorMtUBmeqKTHVAproymqkHl6uhg3xUzvJ5Ve11uUoX+bK6d0t9r71b62/LTXXAhlrm1HqmXu9+C3OtfmgNydR262Wqq7PIVAdnWVqm5nSWpWWqtb7X3nPMVGv9gnqPZsrHAgEAAAJYrgAAAAJYrgAAAAJYrgAAAAJYrgAAAAJYrgAAAAJYrgAAAAJYrgAAAAJYrgAAAAJYrgAAAALkUsrmgpxXKaXV+uXJ5Cfi3n46TN/K11xTa04797GUclpTaFa7I1NdkakOyFRXZKoDMtWV0Uw9uFwNHeSjcpbPq2qvy1W6yJfVvVvqe+3dWn9bbqoDNtQyp9Yz9Xr3W5hr9UNrSKa2Wy9TXZ1Fpjo4y9IyNaezLC1TrfW99p5jplrrF9R7NFM+FggAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABAgl1I2F+S8Simt1i9PJj8R9/bTYfW3dJvTzo1+S/cYs9odmeqKTHVAproiUx2Qqa6MZurB5WroIB+Vs3xeVXtdrtJFvqzu3VLfa+/W+ttyUx2woZY5tZ6p17vfwlyrH1pDMrXdepnq6iwy1cFZlpapOZ1laZlqre+19xwz1Vq/oN6jmfKxQAAAgACWKwAAgACWKwAAgACWKwAAgACWKwAAgACWKwAAgACWKwAAgACWKwAAgACWKwAAgACWKwAAgAC5lLK5IOdVSmm1fnky+Ym4t58O07fyNdfUmtPOfSylnNYUmtXuyFRXZKoDMtUVmeqATHVlNFMPLldDB/monOXzqtrrcpUu8mV175b6Xnu31t+Wm+qADbXMqfVMvd79FuZa/dAakqnt1stUV2eRqQ7OsrRMzeksS8tUa32vveeYqdb6BfUezZSPBQIAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAASwXAEAAATIpZTNBTmvUkqr9cuTyU/Evf10WP0t3ea0c6Pf0j3GrHZHproiUx2Qqa7IVAdOXj1LHz59l6k+jGbqweVq6PT18/L+3Yuq2r3ju/Tjy8vq3nvHd+kiX1bVXper6trW+il7t9bflpvqh9ZQy5xSaptVy5xSms/db2Gu1Q+toYN8VM7y+VRnappra16n7D23TLXMqfVMLXNKafq7n9H7tkwF1cvUZr2+/z2VTPX43J/yd+zN28/Vy9XQlJlqrZ9b7wnzOpopHwsEAAAIYLkCAAAIYLkCAAAIYLkCAAAIYLkCAAAIYLkCAAAIYLkCAAAIYLkCAAAIYLkCAAAIYLkCAAAIkEspmwtyXqWUVuuXJ5OfiHv76TB9K19zTa057dzHUsppTaFZ7Y5MdUWmOiBTXZGpDpy8epY+fPouU30YzdSDy9XQ6evn5f27F1W1e8d36ceXl9W9947v0kW+rKq9LlfVta31U/Zurb8tN9UPraGWOaXUNquWOaU0n7vfwlyrH1pDB/monOXzqc7UNNfWvE7Ze26ZaplT65la5pTS9Hc/o/dtmQqql6nNen3/eyqZ6vG5P+Xv2Ju3n6uXq6EpM9VaP7feE+Z1NFM+FggAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABDAcgUAABAgl1I2F+S8Simt1i9/SSn9pbL3H1NKvzacpaW+196t9b+UUvZrCn/HnFrP1OvdTz3Xbcyq1/uRqbjaOfWe+iwy1cdZlpapOZ1laZlqre+19xwz1Vq/lN7jsyqlTPInpfRhqvpee099lm3Mak4/71x6b2tWvd7PnO5epp7uWaae09x+3rmcZWmZmtNZlpapOd3PnHrPcVZL7+1jgQAAAAEsVwAAAAGmXK7+PGF9r71b61t7P9Zc7qfX3o+pf4xe72dOdy9Tu6uXqd3Vz6n3Y83l522t77X3Y7n77fd+rDn9DE+q94P/oAUAAAAP87FAAACAAJYrAACAAJYrAACAAJYrAACAAJYrAACAAP8Hs7VwOxhjYI0AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "fig = plt.figure(figsize=(12, 12))\n", "for i in range(H * W):\n", " ax = fig.add_subplot(H, W, i + 1)\n", " ax.imshow(mask_shifted[i].reshape(H, W))\n", " ax.grid(color='k', linestyle='-', linewidth=1)\n", " ax.set_xticks(torch.arange(0.5, W))\n", " ax.set_yticks(torch.arange(0.5, H))\n", " ax.set_xticklabels([])\n", " ax.set_yticklabels([])\n", "fig.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "dd27ef92-1ba7-405a-9545-251f94f29461", "metadata": {}, "source": [ "We can see that the self-attention maps correspond to the images in the paper (shown in the top of the notebook), illustrating that indeed a custom sparsity pattern is enough to reproduce Swin Transformer without having to ressort to implementing custom code.\n", "\n", "Plus, it is trivial to extend Swin Transformer with other attention patterns (such as local 2d, axial and more, see [the 2d attetnion patterns notebook](https://github.com/fairinternal/xformers/blob/main/docs/source/2d_attention_patterns.ipynb) for more examples." ] }, { "cell_type": "markdown", "id": "699fd7bc-377b-4786-8dc7-732619ddb89e", "metadata": {}, "source": [ "## Using Swin Transformers as a sparse Transformer in your model\n", "\n", "Now that we know that we can represent a Swin Transformer as a particular instantiation of a sparse Transformer, let's use xformers efficient sparse kernels to see\n", "what type of speed / memory trade-offs we get by casting a Swin Transformer as a sparse Transformer.\n", "\n", "To facilitate benchmarking and memory profiling, let's define a function that takes a generic callable and executes it, measuring the execution time and the GPU memory" ] }, { "cell_type": "code", "execution_count": 5, "id": "b072ae40-9bf3-4ab8-9717-acf2e8ffe981", "metadata": {}, "outputs": [], "source": [ "def profile_model(fn, min_run_time=2):\n", " torch.cuda.reset_peak_memory_stats()\n", " torch.cuda.synchronize()\n", " res = benchmark.Timer(\n", " stmt='fn()',\n", " globals={\"fn\": fn},\n", " label=\"profile\",\n", " sub_label=\"\",\n", " description=\"\"\n", " ).blocked_autorange(min_run_time=min_run_time)\n", " torch.cuda.synchronize()\n", " memory = torch.cuda.max_memory_allocated() / 2 ** 20\n", " memory = f\"Memory used: {memory} MB\"\n", " print(res)\n", " print(memory)" ] }, { "cell_type": "markdown", "id": "1edb40c1-98ce-4ae1-a486-42022f3b6b1b", "metadata": {}, "source": [ "Now it comes the core of it. We will implement an `Attention` module following the same API and modules as timm's, but using our `scaled_dot_product_attention` function.\n", "\n", "Note the similarities between this implementation and the one from the [vision transformers notebook](https://github.com/fairinternal/xformers/blob/main/docs/source/vision_transformers.ipynb).\n", "\n", "Note that we are not implementing relative positional embedding for simplicity" ] }, { "cell_type": "code", "execution_count": 6, "id": "8def591e-be74-489a-af6b-45f90e13aadc", "metadata": {}, "outputs": [], "source": [ "from timm.models.layers import Mlp, DropPath\n", "\n", "\n", "# exact the same one as from https://github.com/fairinternal/xformers/blob/main/docs/source/vision_transformers.ipynb\n", "class Attention(torch.nn.Module):\n", " def __init__(\n", " self,\n", " dim,\n", " num_heads=8,\n", " qkv_bias=False,\n", " attn_drop=0.0,\n", " proj_drop=0.0,\n", " attn_mask=None,\n", " **kwargs\n", " ):\n", " super().__init__()\n", " self.num_heads = num_heads\n", "\n", " self.qkv = torch.nn.Linear(dim, dim * 3, bias=qkv_bias)\n", " self.attn_drop = torch.nn.Dropout(attn_drop)\n", " self.proj = torch.nn.Linear(dim, dim)\n", " self.proj_drop = torch.nn.Dropout(proj_drop)\n", " self.attn_mask = attn_mask\n", "\n", " def forward(self, x):\n", " B, N, C = x.shape\n", " qkv = (\n", " self.qkv(x)\n", " .reshape(B, N, 3, self.num_heads, C // self.num_heads)\n", " .permute(2, 0, 3, 1, 4)\n", " )\n", "\n", " qkv = qkv.flatten(1, 2)\n", "\n", " q, k, v = qkv.unbind()\n", " \n", " x = scaled_dot_product_attention(q, k, v, self.attn_mask, dropout=self.attn_drop)\n", " \n", " x = x.reshape(B, self.num_heads, N, C // self.num_heads)\n", "\n", " x = x.transpose(1, 2).reshape(B, N, C)\n", " x = self.proj(x)\n", " x = self.proj_drop(x)\n", " return x\n", " \n", "\n", "# almost copy and paste from timm's implementation, but removing the unneeded elements\n", "# as we don't need to perform the image partitioning anymore\n", "# Note that we call our swin_attention_pattern in the constructor\n", "# to generate the custom sparsity pattern\n", "class SwinTransformerBlock(nn.Module):\n", " r\"\"\" Swin Transformer Block.\n", " Args:\n", " dim (int): Number of input channels.\n", " input_resolution (tuple[int]): Input resulotion.\n", " num_heads (int): Number of attention heads.\n", " window_size (int): Window size.\n", " shift_size (int): Shift size for SW-MSA.\n", " mlp_ratio (float): Ratio of mlp hidden dim to embedding dim.\n", " qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True\n", " qk_scale (float | None, optional): Override default qk scale of head_dim ** -0.5 if set.\n", " drop (float, optional): Dropout rate. Default: 0.0\n", " attn_drop (float, optional): Attention dropout rate. Default: 0.0\n", " drop_path (float, optional): Stochastic depth rate. Default: 0.0\n", " act_layer (nn.Module, optional): Activation layer. Default: nn.GELU\n", " norm_layer (nn.Module, optional): Normalization layer. Default: nn.LayerNorm\n", " \"\"\"\n", "\n", " def __init__(self, dim, input_resolution, num_heads, window_size=7, shift_size=0,\n", " mlp_ratio=4., qkv_bias=True, qk_scale=None, drop=0., attn_drop=0., drop_path=0.,\n", " act_layer=nn.GELU, norm_layer=nn.LayerNorm):\n", " super().__init__()\n", " self.dim = dim\n", " self.input_resolution = input_resolution\n", " self.num_heads = num_heads\n", " self.window_size = window_size\n", " self.shift_size = shift_size\n", " self.mlp_ratio = mlp_ratio\n", " if min(self.input_resolution) <= self.window_size:\n", " # if window size is larger than input resolution, we don't partition windows\n", " self.shift_size = 0\n", " self.window_size = min(self.input_resolution)\n", " assert 0 <= self.shift_size < self.window_size, \"shift_size must in 0-window_size\"\n", " \n", " # create swin_attention_pattern sparsity pattern\n", " attn_mask = AP.swin_attention_pattern(input_resolution[0], input_resolution[1], window_size, shift_size=shift_size)\n", " attn_mask = SparseCS(attn_mask, torch.device(\"cuda\"))\n", "\n", " self.norm1 = norm_layer(dim)\n", " self.attn = Attention(\n", " dim, window_size=(self.window_size, self.window_size), num_heads=num_heads,\n", " qkv_bias=qkv_bias, qk_scale=qk_scale, attn_drop=attn_drop, proj_drop=drop,\n", " attn_mask=attn_mask\n", " )\n", "\n", " self.drop_path = DropPath(drop_path) if drop_path > 0. else nn.Identity()\n", " self.norm2 = norm_layer(dim)\n", " mlp_hidden_dim = int(dim * mlp_ratio)\n", " self.mlp = Mlp(in_features=dim, hidden_features=mlp_hidden_dim, act_layer=act_layer, drop=drop)\n", "\n", " def forward(self, x):\n", " H, W = self.input_resolution\n", " B, L, C = x.shape\n", " assert L == H * W, \"input feature has wrong size\"\n", "\n", " shortcut = x\n", " x = self.norm1(x)\n", "\n", " # W-MSA/SW-MSA\n", " x = self.attn(x) # nW*B, window_size*window_size, C\n", "\n", " # FFN\n", " x = shortcut + self.drop_path(x)\n", " x = x + self.drop_path(self.mlp(self.norm2(x)))\n", "\n", " return x" ] }, { "cell_type": "markdown", "id": "e55578e0-1f4f-4a51-ab6c-eaa526e28b79", "metadata": {}, "source": [ "Let's write a function that given a model, will replace all instances of timm.models.swin_transformer.SwinTransformerBlock with our own implementation, which leverages `scaled_dot_product_attention` and `swin_attention_pattern` from xformers\n", "\n" ] }, { "cell_type": "code", "execution_count": 7, "id": "81e9c20f-69fb-48cc-b4e6-debb404e2240", "metadata": {}, "outputs": [], "source": [ "def replace_attn_with_xformers_one(module):\n", " module_output = module\n", " if isinstance(module, timm.models.swin_transformer.SwinTransformerBlock):\n", " \n", " module_output = SwinTransformerBlock(module.dim, module.input_resolution, module.num_heads, module.window_size, module.shift_size, module.mlp_ratio)\n", " module_output.drop_path = module.drop_path\n", " module_output.norm1 = module.norm1\n", " module_output.norm2 = module.norm2\n", " module_output.mlp = module.mlp\n", " \n", " module_output.attn.qkv = module.attn.qkv\n", " module_output.attn.attn_drop = module.attn.attn_drop\n", " module_output.attn.proj = module.attn.proj\n", " module_output.attn.proj_drop = module.attn.proj_drop\n", " \n", " module_output.train(module.training)\n", " \n", " else:\n", "\n", " for name, child in module.named_children():\n", " module_output.add_module(name, replace_attn_with_xformers_one(child))\n", " del module\n", " return module_output" ] }, { "cell_type": "markdown", "id": "12eaa142-ae39-4a8e-bad9-20b3738e38d2", "metadata": {}, "source": [ "Now it's time to create our Swin Transformer. Nothing unusual here. Note that we will be keeping a copy of the model, which will be the model to use sparse self-attention" ] }, { "cell_type": "code", "execution_count": 8, "id": "d04d2cab-f1ec-4151-b41b-0c3406f0d9f7", "metadata": {}, "outputs": [], "source": [ "model = timm.models.create_model(\"swin_base_patch4_window7_224\").cuda().eval()\n", "\n", "# zero relative positional embedding in original model as we don't implement it here\n", "for n, p in model.named_parameters():\n", " if \"relative_position_bias_table\" in n:\n", " torch.nn.init.zeros_(p)\n", "\n", "model_sparse = copy.deepcopy(model)\n", "model_sparse = replace_attn_with_xformers_one(model_sparse)" ] }, { "cell_type": "markdown", "id": "42d3792a-c219-445e-bc24-e32d3d729a02", "metadata": {}, "source": [ "Let's new create an input tensor verify if both the sparse and the baseline versions produce the same results" ] }, { "cell_type": "code", "execution_count": 9, "id": "52c4fce7-2e5a-4214-8944-cd54832f90f8", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Median absolute difference: 3.70e-05\n", "Max absolute difference: 2.51e-04\n" ] } ], "source": [ "i = torch.rand(32, 3, 224, 224).cuda()\n", "\n", "with torch.no_grad():\n", " r0 = model(i)\n", " r1 = model_sparse(i)\n", "\n", "diff = (r0 - r1).abs()\n", " \n", "print(f\"Median absolute difference: {diff.median().item():.2e}\")\n", "print(f\"Max absolute difference: {diff.max().item():.2e}\")" ] }, { "cell_type": "markdown", "id": "6f7d83f9-b21b-48de-89dd-c255fef30f98", "metadata": {}, "source": [ "The results are almost the same. The reason why they are not equivalent up to float precision is because we currently assume that the number of non-zero elements in the sparse matrix is a multiple of 4, so up to 3 elements in the self-attention might be dropped in order to satisfy this constraint.\n", "This constraint will be lifted in the future.\n", "\n", "Let's new benchmark both the sparse and the baseline versions" ] }, { "cell_type": "markdown", "id": "8ba0279a-ad14-4295-bb37-5778f60650dc", "metadata": {}, "source": [ "### Profiling the baseline (dense) model" ] }, { "cell_type": "code", "execution_count": 10, "id": "909b23f2-2ac7-488e-9447-089496d37346", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Forward only\n", "\n", "profile\n", " Median: 212.33 ms\n", " IQR: 9.49 ms (210.94 to 220.43)\n", " 10 measurements, 1 runs per measurement, 1 thread\n", "Memory used: 1448.72509765625 MB\n", "\n", "Forward + backward\n", "\n", "profile\n", " Median: 626.96 ms\n", " IQR: 12.91 ms (623.15 to 636.06)\n", " 4 measurements, 1 runs per measurement, 1 thread\n", "Memory used: 8615.0703125 MB\n" ] } ], "source": [ "print(\"Forward only\")\n", "with torch.no_grad():\n", " profile_model(lambda : model(i))\n", "print(\"\")\n", "print(\"Forward + backward\")\n", "profile_model(lambda : model(i).sum().backward())" ] }, { "cell_type": "markdown", "id": "97bcc2c6-a2e7-403c-8b43-33c5f818a7c1", "metadata": {}, "source": [ "### Profiling the sparse model" ] }, { "cell_type": "code", "execution_count": 11, "id": "4e17e1aa-6970-4b6f-9adb-4e51b55cb4ac", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Forward only\n", "\n", "profile\n", " Median: 208.51 ms\n", " IQR: 1.29 ms (208.06 to 209.34)\n", " 10 measurements, 1 runs per measurement, 1 thread\n", "Memory used: 1636.5673828125 MB\n", "\n", "Forward + backward\n", "\n", "profile\n", " Median: 607.60 ms\n", " IQR: 9.11 ms (605.09 to 614.20)\n", " 4 measurements, 1 runs per measurement, 1 thread\n", "Memory used: 8770.02001953125 MB\n" ] } ], "source": [ "print(\"Forward only\")\n", "with torch.no_grad():\n", " profile_model(lambda : model_sparse(i))\n", "print(\"\")\n", "print(\"Forward + backward\")\n", "profile_model(lambda : model_sparse(i).sum().backward())" ] }, { "cell_type": "markdown", "id": "6cfa7340-8d7c-4be5-861b-2354324f50ca", "metadata": {}, "source": [ "Those results indicate that the sparse model achieves the same speed as the manually-implemented dense version.\n", "This is very encouraging, as with a generic sparse implementation we are able to achieve comparable speed versus the optimized dense implementation, while being substantially simpler to implement (specially on the windows shift optimizations, see [\\[1\\]](https://github.com/microsoft/Swin-Transformer/issues/52) and [\\[2\\]](https://github.com/microsoft/Swin-Transformer/issues/38) for examples).\n", "\n", "From the memory perspective, the sparse model uses slightly more memory, as it needs to keep the indices of the non-zero elements in memory, while in the baseline dense model the structure is encoded directly in the code. Note that we can further reduce the memory needs by re-using the same sparse pattern over multiple layers." ] }, { "cell_type": "markdown", "id": "95caaf5c-53b4-493e-8a66-1ad09d1917f6", "metadata": {}, "source": [ "# Wrapping up\n", "\n", "In this notebook, we've shown that Swin Transformers can be casted as a sparse transformer, and we've shown that a generic implementation based on the sparse kernels from `xformers` is able to match performances compared to the hand-crafted implementation.\n", "\n", "We hope that this will further illustrate the power of custom sparsity patterns, and we hope xformers will enable new research directions on large sequences.\n", "\n", "Do not hesitate to reach out if you have questions." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.8" } }, "nbformat": 4, "nbformat_minor": 5 }