What are Tokens?

What are tokens and why is everyone trying to save them?

Often, when I am scrolling through Instagram looking for dog videos, I come across a red-faced tech bro shouting something important sounding about TOKENS!!!

“you gotta save those tokens bro” “How to never hit your token limit again broski” “Bro, have you hit 93% of your tokens today? Epic fail!”

OK so I might have exaggerated the frat boy talk just a little, but the point remains… people keep going on about Tokens.

But what are they? Why do they matter? And as always…why should I care about this?

Time to dive straight in and figure out what the heck is going on here.

What Is a Token?

For me, the first thing I think of when I think of a “token” is one of those little plastic things they give you at fairgrounds, to scam you into buying more rides than you want.

But here we are talking about tokens in the context of AI models.

As always, let’s begin with the definition in plain English:

A token is the smallest piece of text that an AI model reads and processes at a time. They come from breaking down bigger chunks of text into smaller pieces.

So specifically, a token might be:

• A word (“under”, “cat”, “hi”)
• Part of a word (“under” “standing”)
• A number (3, 4000)
• Punctuation (!, ,, .)
• An emoji (😊)

(As a rough rule of thumb, one token is about ¾ of a word in English. Or 100 words is roughly 130 tokens)

OK, bit strange all of this token lark.

Let’s look at a quick example before we explain why the AI model needs to do this:

Let’s imagine you input the following sentence into ChatGPT or another LLM:

“I need help in understanding what Tokens are in plain english”

How does ChatGPT process this? First, it breaks this sentence up into tokens (smaller chunks).

This is what the AI model processes before deciding how to respond.

The same thing happens in reverse when ChatGPT responds to your question. First it processes the response using tokens before translating the answer into “human” language for you to read.

Understanding this key fact — that AI models work using tokens and not words — is an important part of working with AI tools like ChatGPT or Claude.

Let’s look into more detail about why we even need tokens in the first place.

Why Do AI Models Work With Tokens?

This was the question that I kept asking myself.

Why can’t we just make things nice and simple and design AI models which process WORDS like a good old-fashioned human would! Why make things complicated?

Ah, but come on, life is never that easy eh.

It all comes down to how AI understands human language.

When two humans talk to each other, we listen for the words, process the meaning based on our understanding of the language, and then reply using words.

AI models work a bit differently.

Remember that at heart, an AI model is really just a complex mathematical model.

Mathematical models ⇒ communicate via maths.

That’s why AI models cannot process words like we can. We need to convert our text first into a format a computer can actually work with: numbers.

Using tokens makes this more efficient.

Imagine if every unique word in the English language has its own number.

The list would be unbelievably long! Imagine every word EVER, including made-up words, human names… arggggghhh too many words!

Tokens solve this infinite-word problem by breaking language into reusable building blocks instead.

The word “tokenisation” might become “token” + “isation”. Now if a new word comes along like “tokenize”, the model already has “token” covered and just needs one new piece.

Tokens are a neat solution for translating human language into computer-friendly numbers.

What is Tokenisation?

Tokenisation describes the process of splitting up text into smaller chunks (or tokens as we will now call them).

You can also think of it as translating “human language” into “ai-model language” if you prefer.

This happens automatically in the background whenever you interact with an LLM like ChatGPT or Claude.

Most of the time, we don’t need to worry too much about what is going on in the background with tokenisation.

But as you start using AI tools like ChatGPT or Claude more and more, you will sadly be unable to avoid thinking about tokens at some point.

Here is why:

Why Should I Care About Tokens?

OK so the model works with tokens. Why does this matter to me, a normal human person who just wants to get things done?

There are two main reasons for a human to care about the concept of tokens.

(and why those tech-bros are banging on about it on TikTok).

1. MONEY! (of course)

Tokens are not just chunks of text that AI uses to process language. They are a form of currency!

AI tools have designed their pricing and subscription models, not around hours used, or words typed but around “tokens processed”.

For example, maybe you have a subscription with Claude but after a long day you see “token limit reached” and you can’t use it anymore.

Grrr!

This means that the AI model has processed the maximum number of tokens allowed, for the subscription that you have selected.

The more you pay → the more tokens you get each day.

This is the biggest reason why a basic understanding of tokens is hugely useful if you want to get more out of your AI tools.

If you understand how tokens work, you can optimise how you work with your AI tools to make sure that you make every token count.

This is why you hear a lot of people talking about “writing good prompts” for AI tools like ChatGPT.

A huge prompt, a giant document pasted in, a massive amount of background context you didn’t need to give, all of that costs tokens before the model has even started answering you.

If you want to squeeze every drop of usefulness out of your AI tools, you better start thinking like them — in tokens.

2. Understanding the “Context Window”

In my opinion, this one is more of a “nice to understand” concept than an absolutely necessary one like the previous reason.

But hey! Let’s do this properly and look at why tokens are important for the so-called “context window” of AI models.

Firstly — what is a context window?

Well, every AI model can only “see” a certain number of tokens at a time.

This is called its context window, or the maximum amount of text the model can “hold in mind” at once.

Imagine a teacher writing class notes onto a whiteboard. As they write more and more, they have to go back and make space by erasing some of their earlier notes.

This is how context windows work. The oldest stuff starts to fall out of view as new information comes in.

Older models had context windows of around 4,000 tokens (a few pages of text). Newer models can handle 128,000 or even 200,000, which sounds enormous, but a long conversation with lots of pasted documents can eat through it faster than you’d think.

This is why ChatGPT might appear to forget something you told it 20 messages ago.

The earlier part of your conversation has been “erased from the board” to make space for the new stuff.

It’s useful to understand where tokens play a role here, so you understand the natural limitations of AI tools when it comes to storing your information in memory.

AI isn’t an infinite memory machine. It’s got to make room somehow!

---

Summary

1. A token is not a word and not a character — it’s a small chunk of text, roughly ¾ of a word on average. Models chop up everything you write into tokens before they can process it.
2. Tokens are how most AI tools are priced. Knowing this helps you understand why shorter, sharper prompts are often better, and how to get the most out of every token.
3. The context window is the maximum number of tokens a model can see at once. When your conversation gets long enough, early parts drop out of view, which is why the AI sometimes seems to forget what you told it.