metadata

title: Tokenizer
emoji: 📚
colorFrom: green
colorTo: yellow
sdk: docker
pinned: false

Tokenizer API

A simple API for tokenizing text using spaCy.

Usage Examples

Using curl

curl -X POST "https://theirstory-tokenizer.hf.space:7860/tokenize" \
     -H "Content-Type: application/json" \
     -d '{"text": "Hello world!"}'

Using Python

import requests

response = requests.post(
    "https://theirstory-tokenizer.hf.space:7860/tokenize",
    json={"text": "Hello world!"}
)
tokens = response.json()
print(tokens)

Using JavaScript

const response = await fetch('https://theirstory-tokenizer.hf.space:7860/tokenize', {
    method: 'POST',
    headers: {
        'Content-Type': 'application/json',
    },
    body: JSON.stringify({
        text: 'Hello world!'
    })
});
const tokens = await response.json();
console.log(tokens);

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference