AI Glossary · Foundations

Token

The unit of text a model processes — roughly 3-4 characters of English. Prompts, outputs and context windows are all measured in tokens.

Definition

What is Token?

Token is the unit of text a model processes — roughly 3-4 characters of English. Prompts, outputs and context windows are all measured in tokens.

Token matters in enterprise AI programs because it helps business and technology leaders align vocabulary, scope, ownership, and measurable outcomes.

Alignment: The discipline of making an AI system's behavior match human intent, values and organizational policy — both at training time and at runtime.
Artificial Intelligence (AI): Software systems that perform tasks normally requiring human intelligence — perception, reasoning, language, decision-making — by learning patterns from data.
Artificial General Intelligence (AGI): A hypothetical AI capable of performing any intellectual task a human can, across domains. Not a shipping product category today; treat vendor claims with scrutiny.
Benchmark: A standardized task suite (e.g., MMLU, GSM8K, HELM) used to compare model quality on reasoning, coding, safety or domain knowledge.
Chain-of-Thought (CoT): A prompting or training pattern where a model produces intermediate reasoning steps before its final answer, improving accuracy on math, planning and multi-hop tasks.
Classification: A supervised learning task that assigns inputs to discrete categories (spam vs. not spam, churn risk tier). The workhorse of predictive AI.