How Do Large Language Models (LLMs) Like ChatGPT Work?

Your phone suggests the next word as you type. Large language models are the same idea after growing millions of times larger, so that they now converse, translate and write code that may run on the first attempt in short tasks. The paradox is that they do not "understand" in your sense; they master a single game: guessing the next word.

The Seed: Language as a Probabilistic System

In 1948, in his information theory, Claude Shannon held that language can be modelled probabilistically: every letter and word carries a measurable degree of "surprise". The great turning point came in 2017 with the paper "Attention Is All You Need", which introduced the transformer architecture: each token attends to the tokens that came before it in the context, weighing its relation to each of them at once. Here lies the secret: a word's meaning takes its colour from its context, just as the Arabic word "ayn" shifts between a spring, a spy and the seeing eye.

How Does It Learn?

A vast quantity of text is broken into small units called "tokens", and the model is asked, countless times, to guess the next token in the text. Each time it errs, its internal weights are adjusted slightly. Words become vectors in a multidimensional space where words close in meaning lie close together; Tomas Mikolov and colleagues showed in 2013 that "king − man + woman" comes near "queen" in selected examples. That is, meaning is drawn from the company words keep, as the linguist J. R. Firth summed it up in 1957: "You shall know a word by the company it keeps."

The Alignment Stage: The Human Hand

After initial training the model is encyclopaedic but untidy. So humans step in with what is known as reinforcement learning from human feedback (RLHF): human raters assess and rank its answers, and it comes to favour helpful, polite ones. In this way the raters' values and biases seep into its behaviour, so the question "Who trains the machine?" matters no less than "On what data?"

"Emergent Abilities": Magic or Measurement?

Researchers noticed (2022) that some skills appear suddenly at a certain scale. A later study (2023), however, suggested that much of this "emergence" is an effect of our imprecise way of measuring, not the birth of new understanding. This distinction is the heart of the dispute: are we witnessing the dawn of machine awareness, or a highly polished statistical echo?

Fluency Is Not Understanding

A model may write a polished paragraph on a scientific theory without it being clear that it holds a conception of it in the sense we understand. This is why the researcher Emily Bender and her colleagues described it as a "stochastic parrot". Hence "hallucination", the invention of what has no basis: the model seeks the statistically most likely, not the correct. It is likely inherent to its nature, to be reduced rather than assumed to vanish entirely. It is an assistant that needs a vigilant reviewer, not an infallible authority. There is a lesson here for us before the machine: to say "I do not know" when we do not know, being truthful with others and with ourselves. For the effect of these models on professionals' performance, see the article "Three Myths About Artificial Intelligence".

Three Pillars for Dealing with Them

1. Master the question: the quality of the answer follows the quality of the request.
2. Always verify: especially figures, names and facts.
3. Understand the limits: when is the model in its element, and when does it collapse? The weak representation of Arabic in training data is a question in itself, covered in the article "Arabic Artificial Intelligence".

Large language models are a dazzling station, not the end of the road; as they master our words they pose an older question to us: what does it mean to understand? The coming challenge is not merely to have a machine that speaks as we speak, but to remain responsible for what we say, why, and to whom.