Question about model embedding of position Learned Tokens

#23

by xiaohuiv1 - opened 10 days ago

10 days ago

Thanks for your great work!
I want to understand how the model embedding of Special Learned Tokens is trained or initialized? because I found you mask the question when pretrain and sft.
Speical Learned Tokens is located in Section 3.1 of the paper.

Thx

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment