Generative AI and Large Language Models — How They Work

A large language model (LLM) is an AI model pre-trained on very large amounts of text so that it can interpret and produce human language. It is described as the core technology underlying generative AI — systems that produce new output such as text, code or images.

The turning point was 'Attention Is All You Need', submitted to arXiv on 12 June 2017 by eight researchers at Google. The paper's abstract proposes "a new simple network architecture, the Transformer". The transformer made it practical to learn the relationships between words across a sentence efficiently, which opened the path to scaling models up. The 2017 date refers to the first submission (v1); the paper was last revised on 2 August 2023 (v7).

OpenAI applied reinforcement learning from human feedback (RLHF) — refining the model using human ratings — to build a conversational model, and released ChatGPT on 30 November 2022. The service spread quickly, and 2023 is generally summarised as the year generative AI became a mainstream public concern.

An LLM is a transformer-based language model scaled up substantially. The explanation commonly given is that performance has tended to improve as training data and parameter counts grow — a tendency that has been observed, rather than a guarantee.

When an LLM produces content that is not true but presents it as fact, this is called hallucination. It is explained less as a defect of one particular model than as a limit arising from how these systems work.

An LLM does not look a fact up in a database. It predicts the most plausible next piece of text probabilistically — it is optimised for whether output reads naturally, not for whether it is true. A statistically plausible sequence of words can therefore be generated even when it does not match reality.

In addition, as knowledge is compressed into the model's weights during training, the link back to the original source is lost. This is why separate concepts or events can be blended into a single plausible-sounding answer. There is no mechanism inside the model to verify a fact or to signal that it is not confident.

There is no known way to eliminate it entirely, so the approaches in use aim to reduce it. The best known is retrieval-augmented generation (RAG), which connects the model to external trusted material and constrains answers to that material.

RLHF is also used to penalise hallucination and reward accurate answers. None of these removes the structural limit, so checking important facts against the original source remains necessary.