Token
- Pidgin definition
- Na small piece of text wey language model dey process. E fit be full word, part of word, or punctuation sign; e no always be one complete word.
English definition
A small piece of text, such as a word, part of a word, or punctuation, that a language model processes. Tokens are not always whole words.
More in AI Foundations
Frequently Asked Questions
What does Token mean?
A small piece of text, such as a word, part of a word, or punctuation, that a language model processes. Tokens are not always whole words.
What is Token in Nigerian Pidgin?
Na small piece of text wey language model dey process. E fit be full word, part of word, or punctuation sign; e no always be one complete word.
Which AI competency does Token belong to?
Token is part of AI Foundations in the aipidgin dictionary.