LLMs: Build Intuition

There’s no denying that Large Language Models (LLMs) have revolutionized not only the tech industry but also everyday life. But have you ever wondered how these seemingly magical systems actually work?

The good news is, you don’t need to deeply dive into their internals to use them effectively in your applications. However, having a rough understanding or intuition about how they work (instead of treating them as a complete black box) can make a huge difference. It helps us understand their limitations, set realistic expectations, and ultimately use them more efficiently and intelligently.

In this multi-part series, I’ll focus on building that intuition—without going into a full technical deep dive (which is both massive and complex on its own). My goal is to give you just enough understanding to confidently design modern, state-of-the-art applications and AI agents, while knowing what to expect from LLMs and what not to.

LLM in a Nutshell

At its core, an LLM is just a very large mathematical equation, where the values of its coefficients are learned from huge amounts of data during training.

For ex-

	f(x) = ax + by 

You can think of this as a very tiny LLM (not large at all, but still a model), where the values of a and b are learned by training on data.

Now, a natural question arises:“But when I talk to ChatGPT, I never give it numbers. So how does a mathematical equation even make sense?”

That’s where a process called tokenization comes in.

Tokenization is the step where text (or any input) is converted into its numerical representation, so that the model can process it mathematically.

Check out this hosted app to play around with tokenization.

TikTokenizer

Here, if you look at the image, my name is getting split into 5 tokens or numbers. (cl100k_base is the tokenizer used in gpt-4o)

Tokenization Example

Now that we have our input ready in numerical form, we can feed it into the mathematical equation (the LLM). The model processes these numbers and produces output—also in numerical form.

Finally, these output numbers are converted back into human-readable text through a process called detokenization, which is simply the reverse of tokenization.

So now, if we stitch all of this together, the whole process behind ChatGPT or any LLM looks like:

LLM Or Text Generation Workflow

Great! Now that we have a high-level picture of what LLMs actually are, we’re ready to go a little deeper into the beauty of this arithmetic computation, a.k.a mathematical equation.

Stay Tuned for Part-2...