In the context of large language models, a token is the smallest unit of text the model processes. Tokens can be words, parts of words, punctuation, or spaces — ‘marketing’ might be one token, while ‘unimaginable’ might be split into multiple tokens. LLMs have a ‘context window’ defined by the maximum number of tokens they can process at once. Token limits affect how much text a model can read or generate in a single interaction, which is important when working with long documents or complex prompts.
AI Fundamentals
| LLM
What is Token?
The basic unit of text that a language model processes, roughly equivalent to a word or word fragment.


