Spaces:

Tinkering
/

Pytorch-day-prez

Running

File size: 31,981 Bytes

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  <title>PyTorch × Transformers Journey</title>

  <!-- Google Fonts -->
  <link href="https://fonts.googleapis.com/css2?family=Inter:wght@300;400;600;800&family=Fira+Code:wght@400;600&display=swap" rel="stylesheet" />

  <!-- Reveal.js core & dark theme base -->
  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/reveal.js@5/dist/reset.css" />
  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/reveal.js@5/dist/reveal.css" />
  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/reveal.js@5/dist/theme/black.css" id="theme" />

  <!-- Highlight.js -->
  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/[email protected]/styles/github-dark.min.css" />

  <!-- Animations -->
  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/animate.css@4/animate.min.css" />

  <style>
    :root {
      --accent-primary: #ee4c2c; /* PyTorch orange‑red */
      --accent-secondary: #ffb347; /* lighter highlight */
      --bg-gradient-start: #1b1b1b;
      --bg-gradient-end: #242424;
    }
    html, body { font-family: 'Inter', sans-serif; }
    .reveal .slides {
      background: linear-gradient(135deg, var(--bg-gradient-start), var(--bg-gradient-end));
    }
    .reveal h1, .reveal h2, .reveal h3 { color: var(--accent-primary); font-weight: 800; letter-spacing: -0.5px; }
    .reveal pre code { font-family: 'Fira Code', monospace; font-size: 0.75em; }
    .reveal section img, .reveal section svg { border-radius: 1rem; box-shadow: 0 8px 22px rgba(0,0,0,0.4); }
    .fragment.highlight-current-blue.visible { color: var(--accent-secondary) !important; }
    /* slide-density patch */
    .reveal h1 { font-size: 2.6rem; line-height: 1.1; }
    .reveal h2 { font-size: 1.9rem; line-height: 1.15; }
    .reveal h3 { font-size: 1.4rem; line-height: 1.2; }
    .reveal p, .reveal li { font-size: 1.7rem; line-height: 1.35; }
    .reveal pre code { font-size: 0.67em; }
    /* Make <strong> more vibrant and aligned with the accent */
    .reveal strong {
      color: var(--accent-secondary);  /* orange highlight */
      font-weight: 800;
    }

    /* Make <code> more obvious: change background, font, and padding */
    .reveal code {
      background: rgba(255, 255, 255, 0.1);
      color: #ffd080;
      padding: 0.15em 0.4em;
      border-radius: 0.3em;
      font-family: 'Fira Code', monospace;
      font-size: 0.95em;
    }

    @media (max-width: 1024px) { .reveal h1{font-size:2.2rem;} .reveal h2{font-size:1.6rem;} }
    .reveal table td, .reveal table th { font-size: 0.85rem; padding: 4px 8px; }
        body::after {
      content: "";
      position: fixed;
      bottom: 3.5em;
      left: 3.5em;
      width: 270px;         /* desired size */
      height: 117px;
      background-image: url(assets/py2.png);
      background-size: contain;
      background-repeat: no-repeat;
      z-index: 9999;
      box-shadow: 5px 5px 10px #000;
      pointer-events: none;
    }



  
  
  </style>
</head>
<body>
  <div class="reveal">
    <div class="slides">
      <section>
        <img src="assets/screenpage2.png" alt="Full slide image"
        style="
        width:120%;
        height:110%;
        object-fit:cover;
        margin-left:-2.5%;
        margin-top:-2.5%;
        " />      <!-- 1 · Opening -->
        </section>
      <section data-auto-animate>
        <div style="display: flex; align-items: center; justify-content: center; gap: 1.2rem; margin-bottom: 1rem;" class="animate__animated animate__fadeInDown">
          <img src="assets/transparent_PyTorch.png" alt="PyTorch Logo" style="height: 48px;" />
          <span style="color: white; font-size: 2.4rem; font-weight: 700;">×</span>
          <img src="assets/head_logo.svg" alt="Transformers Logo" style="height: 48px;" />
        </div>
      
        <h1 class="animate__animated animate__fadeInDown">PyTorch × Transformers Journey</h1>
        <h3 class="animate__animated animate__fadeInDown animate__delay-1s">Pythonicity, Autodiff & Modularity in Modern AI</h3>
        <p class="animate__animated animate__fadeInUp animate__delay-2s">Pablo Montalvo‑Leroux &nbsp;·&nbsp; ML Engineer @ Hugging Face</p>
      </section>
      
      <section>
        <h2>2016‑2018: Backprop &amp; Birth Pangs</h2>
        <p>The journey began with uncertainty: back in 2016, machine learning was far from standardized. Tools like Theano and CNTK were fading, and many of us—myself included—were jumping framework to framework. It was a time of raw experimentation.</p>
        <ul>
          <li>Frameworks were in flux; few stuck around.</li>
          <li>MLPs evolved to RNNs and LSTMs.</li>
          <li><strong>2017, Attention, then 2018: BERT</strong> arrives, blowing the roof off what's possible.</li>
        </ul>
        <p class="fragment">But reproducing results remained frustratingly difficult.</p>
      </section>
      
      <section>
        <h2>Transformers × PyTorch: Reproducibility</h2>
        <p>That all changed with <code>pytorch-pretrained-bert</code>, the predecessor to Transformers. Suddenly, the magic of BERT was available in an interface that made sense.</p>
      
        <div style="display: flex; gap: 2rem; justify-content: space-between; margin-top: 2rem;">
          <div style="flex: 1; background: #2d2d2d; padding: 1.2rem; border-radius: 1rem; box-shadow: 0 4px 12px rgba(0,0,0,0.3);">
            <p style="font-weight: 800; color: var(--accent-primary); margin-bottom: 0.6rem;">
              🧩 Simpler Interface
            </p>
            <p>No static graphs, just Python functions and PyTorch modules.</p>
          </div>
          <div style="flex: 1; background: #2d2d2d; padding: 1.2rem; border-radius: 1rem; box-shadow: 0 4px 12px rgba(0,0,0,0.3);">
            <p style="font-weight: 800; color: var(--accent-primary); margin-bottom: 0.6rem;">
              ✨ Hackability
            </p>
            <p>Readable, hackable code meant results could be shared, reproduced, improved.</p>
          </div>
          <div style="flex: 1; background: #2d2d2d; padding: 1.2rem; border-radius: 1rem; box-shadow: 0 4px 12px rgba(0,0,0,0.3);">
            <p style="font-weight: 800; color: var(--accent-primary); margin-bottom: 0.6rem;">
              🚀 Community Shift
            </p>
            <p>This shifted the research community towards PyTorch.</p>
          </div>
        </div>
      </section>
      

      <!-- 3 · Static vs Dynamic Graphs -->
      <section>
        <h2>Static vs Dynamic Graphs</h2>
        <p>Static graphs require you to compile, wait, and cross fingers the bug reproduces.</p>
        <p>Dynamic graphs mean you can drop <code>pdb.set_trace()</code> anywhere and continue iterating.</p>
        <p>Nowadays <code>torch.compile</code> gives the best of both worlds: write dynamically, ship something ahead‑of‑time optimised.</p>
      </section>
      

      <!-- 4 · Dynamic Graphs Enabled Contribution -->
      <section>
        <h2>Dynamic Graphs Enabled Contribution</h2>
        <ul>
          <li>Developers debug at line‑rate — no cold‑start recompiles.</li>
          <li>Pull‑requests remained reproducible overnight, which accelerated trust.</li>
          <li>Static‑graph alternatives stalled and the community consolidated around PyTorch.</li>
        </ul>
      </section>
      
      <section>
        <h2>Clone the Paper Tonight → Tweak Tomorrow</h2>
        <p>PyTorch lowered the barrier to implementation — Transformers built on top of that simplicity.</p>
      
        <div style="display: flex; gap: 1.5rem; margin-top: 2rem;">
          <div style="flex: 1; background: #2d2d2d; padding: 1.2rem; border-radius: 1rem; box-shadow: 0 4px 12px rgba(0,0,0,0.3);">
            <p style="font-weight: 800; color: var(--accent-primary); margin-bottom: 0.5rem;">🔍 Live Debugging</p>
            <p>2018: BERT fine-tunes meant <code>print(tensor)</code>, not <em>recompile & hope</em>.</p>
          </div>
      
          <div style="flex: 1; background: #2d2d2d; padding: 1.2rem; border-radius: 1rem; box-shadow: 0 4px 12px rgba(0,0,0,0.3);">
            <p style="font-weight: 800; color: var(--accent-primary); margin-bottom: 0.5rem;">🤝 Fast Review</p>
            <p>Patches were understandable and reproducible — merged quickly, verified quickly.</p>
          </div>
      
          <div style="flex: 1; background: #2d2d2d; padding: 1.2rem; border-radius: 1rem; box-shadow: 0 4px 12px rgba(0,0,0,0.3);">
            <p style="font-weight: 800; color: var(--accent-primary); margin-bottom: 0.5rem;">⚡ Fast Iteration</p>
            <p>Experiments shifted from <em>weeks</em> to <strong>hours</strong> — feedback cycles accelerated.</p>
          </div>
        </div>
      </section>
      
      <!-- 6 · One Model · One File -->
      <section>
        <h2>“One Model · One File” — Why it Matters</h2>
        <pre><code class="language-python" data-trim data-noescape>
# modeling_bert.py  — single source of truth
class BertConfig(PretrainedConfig):
    ...

class BertSelfAttention(nn.Module):
    ...

class BertLayer(nn.Module):
    ...

class BertModel(PreTrainedModel):
    def __init__(self, config):
        super().__init__(config)
        self.embeddings = BertEmbeddings(config)
        self.encoder = nn.ModuleList(
            [BertLayer(config) for _ in range(config.num_hidden_layers)]
        )
        self.init_weights()
        </code></pre>
        <ul>
          <li>All layers, forward pass, and <code>from_pretrained()</code> logic live together.</li>
          <li>No cross‑file inheritance maze — copy to Colab, hack, and run.</li>
          <li>Reviewers diff one file; merge time dropped from days to hours.</li>
        </ul>
      </section>

            <!-- 8 · Paradigms come at a cost -->
      <section>
        <h2>Paradigms Come at a Cost</h2>
        <div style="display: grid; grid-template-columns: repeat(2, 1fr); gap: 1.5rem; margin-top: 2rem;">
          <div style="background: #2d2d2d; padding: 1.2rem; border-radius: 1rem; box-shadow: 0 4px 12px rgba(0,0,0,0.3);">
            <p style="font-weight: 800; color: var(--accent-primary); margin-bottom: 0.5rem;">📈 Community Growth</p>
            <p>The scientific and engineering ML community thrived with Transformers.</p>
          </div>
      
          <div style="background: #2d2d2d; padding: 1.2rem; border-radius: 1rem; box-shadow: 0 4px 12px rgba(0,0,0,0.3);">
            <p style="font-weight: 800; color: var(--accent-primary); margin-bottom: 0.5rem;">🔥 PyTorch Synergy</p>
            <p>Transformers and PyTorch grew together — adoption fed back into both ecosystems.</p>
          </div>
      
          <div style="background: #2d2d2d; padding: 1.2rem; border-radius: 1rem; box-shadow: 0 4px 12px rgba(0,0,0,0.3);">
            <p style="font-weight: 800; color: var(--accent-primary); margin-bottom: 0.5rem;">🛠️ Maintenance Pressure</p>
            <p>We duplicate code on purpose — to preserve clarity, portability, and hackability.</p>
          </div>
      
          <div class="fragment" style="background: #2d2d2d; padding: 1.2rem; border-radius: 1rem; box-shadow: 0 4px 12px rgba(0,0,0,0.3);">
            <p style="font-weight: 800; color: var(--accent-primary); margin-bottom: 0.5rem;">🧬 Pythonic Modularity</p>
            <p>The <strong>Modularity</strong> of python is never far :)</p>
          </div>
        </div>
      </section>
      
      <!-- 8 · Back to Python: Mary Shelley Mode -->
      <section>
        <h2>Back to Python: Modular “Mary Shelley” Mode</h2>
        <p>Compose new blocks via subclass &amp; override.</p>
      <pre><code class="language-python" data-trim>
class GlmMLP(Phi3MLP):
  pass

class GlmAttention(LlamaAttention):
  def __init__(self, config, layer_idx=None):
      super().__init__(config, layer_idx)
      self.o_proj = nn.Linear(config.num_attention_heads * self.head_dim,
                              config.hidden_size, bias=False)

class GlmForCausalLM(LlamaForCausalLM):
  pass
      </code></pre>
      <p>AST expands → full modeling file, still hackable.</p>
    </section>

    <section>
      <h2>Back to Python: It's alive!</h2>
      <p>All the code becomes runnable and a self-contained model definition</p>
    <pre><code class="language-python" data-trim>

  class GlmMLP(nn.Module):
      def __init__(self, config):
          super().__init__()
  
          self.config = config
          self.gate_up_proj = nn.Linear(config.hidden_size, 2 * config.intermediate_size, bias=False)
          self.down_proj = nn.Linear(config.intermediate_size, config.hidden_size, bias=False)
          self.activation_fn = ACT2FN[config.hidden_act]
  
      def forward(self, hidden_states: torch.FloatTensor) -> torch.FloatTensor:
          up_states = self.gate_up_proj(hidden_states)
  
          gate, up_states = up_states.chunk(2, dim=-1)
          up_states = up_states * self.activation_fn(gate)
  
          return self.down_proj(up_states)
  
  
  class GlmAttention(nn.Module):
      """Multi-headed attention from 'Attention Is All You Need' paper"""
  
      def __init__(self, config: GlmConfig, layer_idx: Optional[int] = None):
          super().__init__()
          self.config = config
          self.layer_idx = layer_idx
          self.head_dim = getattr(config, "head_dim", config.hidden_size // config.num_attention_heads)
          self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
          self.scaling = self.head_dim**-0.5
          self.attention_dropout = config.attention_dropout
          self.is_causal = True
  
          self.q_proj = nn.Linear(
              config.hidden_size, config.num_attention_heads * self.head_dim, bias=config.attention_bias
          )
          self.k_proj = nn.Linear(
              config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
          )
          self.v_proj = nn.Linear(
              config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
          )
          self.o_proj = nn.Linear(config.num_attention_heads * self.head_dim, config.hidden_size, bias=False)
  
      def forward(
          self,
          hidden_states: torch.Tensor,
          position_embeddings: Tuple[torch.Tensor, torch.Tensor],
          attention_mask: Optional[torch.Tensor],
          past_key_value: Optional[Cache] = None,
          cache_position: Optional[torch.LongTensor] = None,
          **kwargs: Unpack[FlashAttentionKwargs],
      ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
          input_shape = hidden_states.shape[:-1]
          hidden_shape = (*input_shape, -1, self.head_dim)
  
          query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)
          key_states = self.k_proj(hidden_states).view(hidden_shape).transpose(1, 2)
          value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)
  
          cos, sin = position_embeddings
          query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
  
          if past_key_value is not None:
              # sin and cos are specific to RoPE models; cache_position needed for the static cache
              cache_kwargs = {"sin": sin, "cos": cos, "cache_position": cache_position}
              key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs)
  
          attention_interface: Callable = eager_attention_forward
  
          if self.config._attn_implementation != "eager":
              if self.config._attn_implementation == "sdpa" and kwargs.get("output_attentions", False):
                  logger.warning_once(
                      "`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to "
                      'eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.'
                  )
              else:
                  attention_interface = ALL_ATTENTION_FUNCTIONS[self.config._attn_implementation]
  
          attn_output, attn_weights = attention_interface(
              self,
              query_states,
              key_states,
              value_states,
              attention_mask,
              dropout=0.0 if not self.training else self.attention_dropout,
              scaling=self.scaling,
              **kwargs,
          )
  
          attn_output = attn_output.reshape(*input_shape, -1).contiguous()
          attn_output = self.o_proj(attn_output)
          return attn_output, attn_weights
  
  
  @use_kernel_forward_from_hub("RMSNorm")
  class GlmRMSNorm(nn.Module):
      def __init__(self, hidden_size, eps=1e-6):
          """
          GlmRMSNorm is equivalent to T5LayerNorm
          """
          super().__init__()
          self.weight = nn.Parameter(torch.ones(hidden_size))
          self.variance_epsilon = eps
  
      def forward(self, hidden_states):
          input_dtype = hidden_states.dtype
          hidden_states = hidden_states.to(torch.float32)
          variance = hidden_states.pow(2).mean(-1, keepdim=True)
          hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
          return self.weight * hidden_states.to(input_dtype)
  
      def extra_repr(self):
          return f"{tuple(self.weight.shape)}, eps={self.variance_epsilon}"
  
  
  class GlmRotaryEmbedding(nn.Module):
      def __init__(self, config: GlmConfig, device=None):
          super().__init__()
          # BC: "rope_type" was originally "type"
          if hasattr(config, "rope_scaling") and config.rope_scaling is not None:
              self.rope_type = config.rope_scaling.get("rope_type", config.rope_scaling.get("type"))
          else:
              self.rope_type = "default"
          self.max_seq_len_cached = config.max_position_embeddings
          self.original_max_seq_len = config.max_position_embeddings
  
          self.config = config
          self.rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]
  
          inv_freq, self.attention_scaling = self.rope_init_fn(self.config, device)
          self.register_buffer("inv_freq", inv_freq, persistent=False)
          self.original_inv_freq = self.inv_freq
    </code></pre>
    <p> We keep hackability while reconnecting with Python working paradigms.</p>
  </section>


      <!-- 9 · Logit Debugger -->
      <section>
        <h2>Logit Debugger: Trust but Verify</h2>
        <ul>
          <li>Hook every <code>nn.Module</code>; dump logits layer‑by‑layer</li>
          <li>Spot ε‑level drifts (LayerNorm, FP16 underflow…)</li>
          <li>JSON traces diffable in CI</li>
          <img data-src="assets/visual_debugger.png" alt="Visual debugger" />

        </ul>
      </section>

      <!-- 10 · DTensor & TP API -->
      <section>
        <h2>DTensor & Tensor‑Parallel API</h2>
        <p>Before, changing to Tensor Parallel meant changing the code.</p>

        <pre><code class="language-python" data-trim data-noescape>
          from transformers.modeling_utils import PreTrainedModel
          from megatron.model import ColumnParallelLinear, RowParallelLinear
          
          class MyTPModel(PreTrainedModel):
              def __init__(self, config):
                  super().__init__(config)
                  self.q_proj = ColumnParallelLinear(config.hidden_size, config.hidden_size)
                  self.k_proj = ColumnParallelLinear(config.hidden_size, config.hidden_size)
                  self.v_proj = ColumnParallelLinear(config.hidden_size, config.hidden_size)
                  self.o_proj = RowParallelLinear(config.hidden_size, config.hidden_size)
          
                  </code></pre>
      </section>

      <!-- 11 · Zero‑Config Parallelism -->
      <section>
        <h2>Zero‑Config Tensor Parallelism</h2>
        <p>The <code>tp_plan</code> JSON keeps model code pristine and declarative.</p>
        <pre><code class="language-json" data-trim data-noescape>{
  "layer.*.self_attn.q_proj": "colwise",
  "layer.*.self_attn.k_proj": "colwise",
  "layer.*.self_attn.v_proj": "colwise",
  "layer.*.self_attn.o_proj": "rowwise"
}</code></pre>
<p>Translated to</p>

        <pre><code class="language-python" data-trim data-noescape>
def translate_to_torch_parallel_style(style: str):
    if style == "colwise":
        return ColwiseParallel()
    elif style == "rowwise":
        return RowwiseParallel()
    # …
        </code></pre>
        <p>One JSON → 100 B param model on 8 GPUs. Change the plan, not the code.</p>
      </section>

      <!-- 12 · Cache Allocator -->
      <section>
        <h2>Improvements, Load faster & stronger: Cache Allocator</h2>
        <p>0‑copy weight sharding, single cuda Malloc</p>
        <p>Faster model loads, even for a 50-shards 100B model (when we were sprinting Llama4!)</p>
        <img data-src="assets/fastload.png" alt="SurprisedLewis" />
      </section>

      <!-- 15 · Why Python wins -->
      <section>
        <h2>Why Python Wins</h2>
        <ul>
          <li>Low entry barrier (although hard to master)</li>
          <li>High‑level semantics express low‑level intent</li>
          <li>Seamless C++/Rust extension points</li>
        </ul>
      </section>

      <!-- 16 · Where Python can bite -->
      <section>
        <h2>Where Python can bite 🐍</h2>
        <ul>
          <li>Interpreter overhead on microkernels (token‑by‑token decode)</li>
          <li>GIL can throttle async host‑side work</li>
          <li>Easy to under‑optimise code fresh out of the lab</li>
        </ul>
        <p class="fragment">All of these can be mitigated: Triton, compiled custom ops, compile‑time fallback, <strong>custom kernels</strong></p>
      </section>


      <!-- 17 · Kernel Hub -->
      <section>
        <h2>Kernel Hub: Optimised Ops from the Community</h2>
        <p>Kernel Hub lets any Python program <em>download and hot‑load</em> compiled CUDA/C++ kernels directly from the Hugging Face Hub at runtime.</p>
        <ul>
          <li><strong>Portable</strong> – kernels work from arbitrary paths outside <code>PYTHONPATH</code>.</li>
          <li><strong>Unique</strong> – load multiple versions of the same op side‑by‑side in one process.</li>
          <li><strong>Compatible</strong> – every kernel targets all recent PyTorch wheels (CUDA, ROCm, CPU) and C‑library ABIs.</li>
        </ul>
        <pre><code class="language-python" data-trim data-noescape>
import torch
from kernels import get_kernel

# Download optimised kernels from the Hugging Face Hub
activation = get_kernel("kernels-community/activation")

x = torch.randn(10, 10, dtype=torch.float16, device="cuda")
y = torch.empty_like(x)
activation.gelu_fast(y, x)
print(y)
        </code></pre>
        <p>Same Transformer code — now with a <strong>3× faster</strong> GELU on A100s.</p>
      </section>

      <section>
        <h2>API Design Lessons</h2>
      
        <div style="display: flex; gap: 1.2rem; margin-top: 1.2rem;">
          <div style="flex: 1; background: #2c2c2c; padding: 0.9rem; border-radius: 0.6rem; box-shadow: 0 3px 10px rgba(0,0,0,0.25); font-size: 1.35rem;">
            <p style="font-weight: 700; color: var(--accent-primary); margin-bottom: 0.4rem;">🔍 Make Easy Things Obvious</p>
            <p style="margin-bottom: 0.4rem;">Models load in <code>one line</code> — no boilerplate.</p>
            <pre><code class="language-python" style="font-size: 0.75em;">model = AutoModel.from_pretrained("bert-base-uncased")</code></pre>
          </div>
      
          <div style="flex: 1; background: #2c2c2c; padding: 0.9rem; border-radius: 0.6rem; box-shadow: 0 3px 10px rgba(0,0,0,0.25); font-size: 1.35rem;">
            <p style="font-weight: 700; color: var(--accent-primary); margin-bottom: 0.4rem;">📄 Paper-to-Repo Diff ≈ 0</p>
            <p style="margin-bottom: 0.4rem;">Code reflects architecture directly.</p>
            <pre><code class="language-python" style="font-size: 0.75em;">class LlamaAttention(nn.Module): ...</code></pre>
          </div>
        </div>
      
        <div style="display: flex; gap: 1.2rem; margin-top: 1.2rem;">
          <div style="flex: 1; background: #2c2c2c; padding: 0.9rem; border-radius: 0.6rem; box-shadow: 0 3px 10px rgba(0,0,0,0.25); font-size: 1.35rem;">
            <p style="font-weight: 700; color: var(--accent-primary); margin-bottom: 0.4rem;">🚀 Prototyping → Production</p>
            <p style="margin-bottom: 0.4rem;">Same model runs in vLLM for deployment:</p>
            <pre><code class="language-python" style="font-size: 0.75em;">LLM(model="llama", model_impl="transformers")</code></pre>
          </div>
      
          <div style="flex: 1; background: #2c2c2c; padding: 0.9rem; border-radius: 0.6rem; box-shadow: 0 3px 10px rgba(0,0,0,0.25); font-size: 1.35rem;">
            <p style="font-weight: 700; color: var(--accent-primary); margin-bottom: 0.4rem;">🎛️ Hide Sharding, Show Intent</p>
            <p style="margin-bottom: 0.4rem;">Declarative TP via config:</p>
            <pre><code class="language-json" style="font-size: 0.75em;">"q_proj": "colwise"</code></pre>
          </div>
        </div>
      
        <p style="font-size: 1.35rem; margin-top: 1.6rem;">
          We tune radios without building RF amps. ML should feel the same.
        </p>
        <p class="fragment" style="font-size: 1.35rem;">
          …while empowering those who do build the amps.
        </p>
      </section>
      
      <!-- 14 · Rise of Multimodality -->
      <section>
        <h2>Rise of Multimodality</h2>
        <pre><code class="language-python" data-trim data-noescape>
processor = AutoProcessor.from_pretrained("Qwen/Qwen3-8B")
model = AutoModelForConditionalGeneration.from_pretrained("Qwen/Qwen3-8B")
        </code></pre>
        <p>Same API across text · vision · audio</p>
        <p>More and more models, with specific processing - need to uniformize</p>

      </section>

      <section>
        <h2>Rise of Multimodality: torch-powered processing</h2>
        <p>Torch and torchvision ops have replaced np + PIL defaults in transformers</p>

        <img data-src="assets/normalize_time_torch.webp" width="80%" height="600" alt="Fast load" />

      </section>
      <!-- 19 · Model Growth by Modality -->
      <section>
        <h2>Model Growth by Modality</h2>
        <embed style="border: none;" src="assets/plot_tryout.html" dpi="300" width="70%" height="500px" />
      </section>


      
      <section>
        <h2>Beyond Transformers: Ecosystem Reuse</h2>
        <p><strong>Transformers</strong> makes modeling easy. <strong>vLLM</strong> makes inference fast.</p>
      
        <div style="display: flex; gap: 2rem; margin-top: 2rem;">
          <div style="flex: 1;">
            <p><strong>🔧 Prototype with Transformers:</strong></p>
            <pre><code class="language-python" data-trim data-noescape>
      from transformers import pipeline
      
      pipe = pipeline("text-generation", model="meta-llama/Llama-3.2-1B")
      print(pipe("The future of AI is")[0]["generated_text"])
            </code></pre>
          </div>
          <div style="flex: 1;">
            <img src="assets/vLLM-Full-Logo.png" alt="vLLM Illustration" style="border-radius: 1rem; box-shadow: 0 0 12px #000; width: 100%;" />
          </div>
        </div>
      </section>
      <section>
        <h2>Deploy with vLLM — No Rewrite Needed</h2>
        <p><strong>vLLM</strong> supports <code>transformers</code> models out of the box. </p>
          
        <p>Just specify <code>model_impl="transformers"</code> if needed:</p>
      
        <pre><code class="language-python" data-trim data-noescape>
      from vllm import LLM, SamplingParams
      
      llm = LLM(model="meta-llama/Llama-3.2-1B", model_impl="transformers")
      params = SamplingParams(max_tokens=20)
      outputs = llm.generate("The future of AI is", sampling_params=params)
      print(outputs[0].outputs[0].text)
        </code></pre>
        <p class="fragment">We also support SGLang now, along with thousands of other libraries! </p>

      </section>
      <section>
        <h2 style="margin-bottom: 1rem;">
          Transformers × PyTorch — Enabling the Community
        </h2>
        <img src="assets/transformers_as_ref.png" alt="Transformers as Reference"
          style="
            width: 120%;
            height: 110%;
            object-fit: cover;
            margin-left: -2.5%;
            margin-top: -2.5%;
          " />
      </section>
      
      <section>
        <h2>Takeaways &amp; The Future</h2>
        <div style="display: grid; grid-template-columns: repeat(2, 1fr); gap: 1rem; margin-top: 1.5rem;">
          <div style="background: #2d2d2d; padding: 1rem; border-radius: 0.8rem;">
            <p style="font-weight: 700; font-size: 1.4rem; color: var(--accent-primary); margin-bottom: 0.4rem;">
              🤝 Symbiotic Growth
            </p>
            <p style="display: flex; align-items: center; gap: 0.4rem; font-size: 1.4rem;">
              <img src="assets/transparent_PyTorch.png" alt="PyTorch" style="height: 1.4rem;" />
              <code> PyTorch</code> &amp; <code>transformers</code> grow together
              <img src="assets/head_logo.svg" alt="Transformers" style="height: 1.4rem;" />
            </p>
          </div>
      
          <div style="background: #2d2d2d; padding: 1rem; border-radius: 0.8rem;">
            <p style="font-weight: 700; font-size: 1.4rem; color: var(--accent-primary); margin-bottom: 0.4rem;">
              🧠 Pythonicity × Pragmatism
            </p>
            <p style="font-size: 1.4rem;">High-level code, low-level control — a winning combination for fast iteration.</p>
          </div>
      
          <div style="background: #2d2d2d; padding: 1rem; border-radius: 0.8rem;">
            <p style="font-weight: 700; font-size: 1.4rem; color: var(--accent-primary); margin-bottom: 0.4rem;">
              🚢 Models Ship Faster
            </p>
            <p style="font-size: 1.4rem;">Open-source models are scaling up — and landing in users' hands faster than ever.</p>
          </div>
      
          <div style="background: #2d2d2d; padding: 1rem; border-radius: 0.8rem;">
            <p style="font-weight: 700; font-size: 1.4rem; color: var(--accent-primary); margin-bottom: 0.4rem;">
              📚 Source of Truth for Model Definitions
            </p>
            <p style="font-size: 1.4rem;">We aim to be the canonical reference — while enabling the community to build, remix, and deploy at scale.</p>
          </div>
        </div>
      
        <p style="margin-top: 1.5rem; font-size: 1.3rem;">
          <a href="https://huggingface.co/transformers/contribute" target="_blank">
            hf.co/transformers/contribute
          </a>
        </p>
      </section>
      

    </div>
  </div>

  <!-- Reveal.js core -->
  <script src="https://cdn.jsdelivr.net/npm/reveal.js@5/dist/reveal.js"></script>
  <script src="https://cdn.jsdelivr.net/npm/reveal.js@5/plugin/highlight/highlight.js"></script>
  <script src="https://cdn.jsdelivr.net/npm/reveal.js@5/plugin/notes/notes.js"></script>
  <!-- Plotly for interactive charts -->
  <script src="https://cdn.plot.ly/plotly-2.31.1.min.js"></script>
  <script>
    Reveal.initialize({
      hash: true,
      slideNumber: true,
      transition: 'slide',
      backgroundTransition: 'convex',
      plugins: [ RevealHighlight, RevealNotes ]
    });
  </script>
</body>
</html>