Spaces:
Sleeping
Sleeping
From here, just continue training like you would with a standard language modelling task, using the formatted_chat column. | |
Advanced: Extra inputs to chat templates | |
The only argument that apply_chat_template requires is messages. However, you can pass any keyword | |
argument to apply_chat_template and it will be accessible inside the template. This gives you a lot of freedom to use | |
chat templates for many things. There are no restrictions on the names or the format of these arguments - you can pass | |
strings, lists, dicts or whatever else you want. | |
That said, there are some common use-cases for these extra arguments, | |
such as passing tools for function calling, or documents for retrieval-augmented generation. In these common cases, | |
we have some opinionated recommendations about what the names and formats of these arguments should be, which are | |
described in the sections below. We encourage model authors to make their chat templates compatible with this format, | |
to make it easy to transfer tool-calling code between models. | |
Advanced: Tool use / function calling | |
"Tool use" LLMs can choose to call functions as external tools before generating an answer. When passing tools | |
to a tool-use model, you can simply pass a list of functions to the tools argument: | |
thon | |
import datetime | |
def current_time(): | |
"""Get the current local time as a string.""" | |
return str(datetime.now()) | |
def multiply(a: float, b: float): | |
""" | |
A function that multiplies two numbers | |
Args: | |
a: The first number to multiply | |
b: The second number to multiply | |
""" | |
return a * b | |
tools = [current_time, multiply] | |
model_input = tokenizer.apply_chat_template( | |
messages, | |
tools=tools | |
) | |
In order for this to work correctly, you should write your functions in the format above, so that they can be parsed | |
correctly as tools. Specifically, you should follow these rules: | |
The function should have a descriptive name | |
Every argument must have a type hint | |
The function must have a docstring in the standard Google style (in other words, an initial function description | |
followed by an Args: block that describes the arguments, unless the function does not have any arguments. | |
Do not include types in the Args: block. In other words, write a: The first number to multiply, not | |
a (int): The first number to multiply. Type hints should go in the function header instead. | |
The function can have a return type and a Returns: block in the docstring. However, these are optional | |
because most tool-use models ignore them. | |
Passing tool results to the model | |
The sample code above is enough to list the available tools for your model, but what happens if it wants to actually use | |
one? If that happens, you should: | |
Parse the model's output to get the tool name(s) and arguments. | |
Add the model's tool call(s) to the conversation. | |
Call the corresponding function(s) with those arguments. | |
Add the result(s) to the conversation | |
A complete tool use example | |
Let's walk through a tool use example, step by step. For this example, we will use an 8B Hermes-2-Pro model, | |
as it is one of the highest-performing tool-use models in its size category at the time of writing. If you have the | |
memory, you can consider using a larger model instead like Command-R | |
or Mixtral-8x22B, both of which also support tool use | |
and offer even stronger performance. | |
First, let's load our model and tokenizer: | |
thon | |
import torch | |
from transformers import AutoModelForCausalLM, AutoTokenizer | |
checkpoint = "NousResearch/Hermes-2-Pro-Llama-3-8B" | |
tokenizer = AutoTokenizer.from_pretrained(checkpoint, revision="pr/13") | |
model = AutoModelForCausalLM.from_pretrained(checkpoint, torch_dtype=torch.bfloat16, device_map="auto") |