How AI Works

Jump to Examples

In Constructing an AI I described how an AI model is built and how to run one at home. This note goes one level deeper: what actually happens inside the model, and what the code looks like. The goal is not to build a ChatGPT, but to understand the idea so well that the big models stop being a black box.

The whole idea fits into two sentences. An AI model is a mathematical function with a very large number of adjustable numbers (parameters). Learning means changing those numbers, step by step, so that the function makes smaller mistakes on the examples it is shown. Everything else – neurons, layers, attention, transformers – is a clever way of organising those numbers and those steps.

A Bit of History: From One Neuron to the Transformer

The broader history of AI is in Constructing an AI; here I follow only the ideas that are inside every modern model. In 1943 Warren McCulloch and Walter Pitts described a very simple mathematical model of a brain cell: it adds up its inputs and "fires" if the sum is large enough. In 1958 Frank Rosenblatt built the perceptron, a neuron that could learn its own weights from examples. In 1969 Marvin Minsky and Seymour Papert showed in their book Perceptrons that a single neuron cannot learn even a simple rule such as XOR ("one or the other, but not both"). Interest in neural networks dropped for years. Example 2 below solves exactly this problem – with just one extra layer.

The missing piece was a method for training networks with several layers. Backpropagation became widely known in 1986 through a paper by David Rumelhart, Geoffrey Hinton and Ronald Williams. In 1989 Yann LeCun used it to read handwritten numbers with a convolutional network, and in 1997 Sepp Hochreiter and Jürgen Schmidhuber introduced LSTM, a network with a kind of memory for sequences such as text and speech.

For language, a key idea was to turn words into lists of numbers (vectors). In 2003 Yoshua Bengio and colleagues trained a neural network to predict the next word, and in 2013 word2vec showed that such word vectors capture meaning: words used in similar ways get similar vectors. In 2014 researchers added attention to translation models – a way for the model to look back at the most relevant words. In 2017 the paper "Attention Is All You Need" built a whole model out of attention: the transformer. Around 2020, research on scaling laws showed that transformers become steadily better with more parameters, more data and more computing power – which is the road that led to today's large language models.

A short timeline of the ideas:

Some resources for history:

• History of artificial neural networks (Wikipedia)
• Timeline of machine learning (Wikipedia)
• Attention Is All You Need (2017)
• Computer History Museum

The Building Blocks

Every neural network, from the smallest example below to the largest language model, is made from the same few parts. The table goes from the smallest part to the complete learning process.

How a neural network learns:

From a neural network to a language model:

How the code of a real language model is organised:

A few observations. First, the code of a model is small; the knowledge is in the numbers. The same 300 lines can describe a useless model or a very capable one – the difference is in the training data, the size and the computing time. Second, nobody programs the knowledge directly. No one writes "Paris is the capital of France" into the code; the model discovers such patterns because they help it predict text. Third, this is also why nobody fully understands what each parameter does. Researchers study this in a field called interpretability.

Some resources for understanding the details:

• 3Blue1Brown – Neural networks (visual video series)
• Andrej Karpathy – Neural Networks: Zero to Hero (free video course)
• micrograd – backpropagation in about 100 lines of Python
• Jay Alammar – The Illustrated Transformer
• Michael Nielsen – Neural Networks and Deep Learning (free online book)
• Dive into Deep Learning (free online book with code)
• My Learning & Documentation resources
• My Programming & Development resources – Python, JupyterLab and other tools

Examples: An AI from Scratch, Step by Step

The five examples below build up the idea one piece at a time, from one neuron to the skeleton of a GPT-style model. They use only Python and NumPy, so every calculation is visible – nothing is hidden inside a framework. Save each example as a .py file and run it from a terminal, for example python example1.py. I ran all of them with Python 3 and NumPy 2; the outputs shown are from those runs, and yours may differ slightly where random numbers are involved.

Example 1 – One neuron learns a rule

The simplest possible "AI": one neuron with two parameters, w (weight) and b (bias). We never tell it the rule y = 2x + 1; we only show it examples, and it finds the rule itself with gradient descent.

# Learn the rule y = 2x + 1 only from examples
data = [(x, 2 * x + 1) for x in range(-5, 6)]

w, b = 0.0, 0.0          # the two parameters, starting at zero
lr = 0.01                # learning rate: how big each correction is

for epoch in range(1000):
    dw = db = 0.0
    for x, y in data:
        prediction = w * x + b
        error = prediction - y
        dw += 2 * error * x      # how the error changes if w changes
        db += 2 * error          # how the error changes if b changes
    w -= lr * dw / len(data)     # gradient descent: step "downhill"
    b -= lr * db / len(data)
    if epoch % 200 == 0:
        loss = sum((w * x + b - y) ** 2 for x, y in data) / len(data)
        print(f"epoch {epoch:4d}   loss {loss:8.4f}   w = {w:.3f}   b = {b:.3f}")

print(f"learned rule: y = {w:.2f}x + {b:.2f}")

Output:

epoch    0   loss  26.5604   w = 0.400   b = 0.020
epoch  200   loss   0.0003   w = 2.000   b = 0.983
epoch  400   loss   0.0000   w = 2.000   b = 1.000
epoch  600   loss   0.0000   w = 2.000   b = 1.000
epoch  800   loss   0.0000   w = 2.000   b = 1.000
learned rule: y = 2.00x + 1.00

This is the training loop in its smallest form: predict, measure the error, work out which way to change each parameter, take a small step. A language model does exactly the same – only with billions of parameters instead of two.

Example 2 – A small network solves XOR, with backpropagation by hand

This is the problem that one neuron cannot solve (the 1969 book in the history section). With one hidden layer of four neurons, it becomes easy. The code shows all three parts of training: the forward pass, the backward pass (backpropagation, written out step by step) and the update. The @ sign is matrix multiplication.

import numpy as np

# XOR: the output is 1 only when the two inputs are different
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([[0], [1], [1], [0]])

rng = np.random.default_rng(1)
W1 = rng.normal(size=(2, 4)); b1 = np.zeros((1, 4))   # layer 1: 2 inputs -> 4 hidden neurons
W2 = rng.normal(size=(4, 1)); b2 = np.zeros((1, 1))   # layer 2: 4 hidden -> 1 output

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

lr = 1.0
for step in range(5001):
    # 1. forward pass: from input to answer
    h = np.tanh(X @ W1 + b1)
    out = sigmoid(h @ W2 + b2)
    loss = np.mean((out - y) ** 2)

    # 2. backward pass (backpropagation): who is to blame for the error?
    d_out = 2 * (out - y) / len(X) * out * (1 - out)
    dW2 = h.T @ d_out
    db2 = d_out.sum(axis=0, keepdims=True)
    d_h = (d_out @ W2.T) * (1 - h ** 2)
    dW1 = X.T @ d_h
    db1 = d_h.sum(axis=0, keepdims=True)

    # 3. update: move every parameter a little bit downhill
    W1 -= lr * dW1; b1 -= lr * db1
    W2 -= lr * dW2; b2 -= lr * db2

    if step % 1000 == 0:
        print(f"step {step:5d}   loss {loss:.4f}")

print("answers:", out.round(2).ravel(), " expected:", y.ravel())

Output:

step     0   loss 0.2400
step  1000   loss 0.0008
step  2000   loss 0.0003
step  3000   loss 0.0002
step  4000   loss 0.0002
step  5000   loss 0.0001
answers: [0.   0.99 0.99 0.01]  expected: [0 1 1 0]

The answers are close to 0, 1, 1, 0 – the network has learned XOR. The backward pass is the hardest part to read, but it is only the chain rule applied layer by layer, from the output back to the input. In real projects, PyTorch does this part automatically.

Example 3 – The simplest language model: predict the next character

Before neural networks, language could be modelled by simply counting. This example reads a text, counts which character usually follows each group of four characters, and then writes new text by choosing the next character again and again. Save a few pages of your own writing as training.txt – I used the text of my article Constructing a Cloud.

import random
from collections import defaultdict, Counter

text = open("training.txt", encoding="utf-8").read().lower()
CONTEXT = 4    # how many previous characters the model looks at

# 1. "Training": count which character follows each group of characters
counts = defaultdict(Counter)
for i in range(len(text) - CONTEXT):
    context = text[i:i + CONTEXT]
    next_char = text[i + CONTEXT]
    counts[context][next_char] += 1

# 2. "Inference": predict the next character again and again
random.seed(7)
output = "the "
for _ in range(300):
    options = counts.get(output[-CONTEXT:])
    if not options:
        break
    chars = list(options)
    weights = list(options.values())     # more frequent = more likely
    output += random.choices(chars, weights=weights)[0]

print(output)
print("different contexts learned:", len(counts))

Output:

the server the cloud gets current documents: container. create /etc/container. documented addressenting the networks arounder the services ( bind authors and kvm with openvpn ported by good practeriod, the update machine power at learn skills user with ssh (open-sources for even the hardware best cloud?
different contexts learned: 7999

The result looks like my article, has real words and even some grammar, but makes no sense. This shows the core task of every language model – predict the next token – and also its limit: counting only sees the last four characters. A neural language model replaces the counting table with a network that can look much further back and recognise meaning, not only spelling.

Example 4 – Attention in twenty lines

This is the central calculation of the transformer. Each token gets a query, a key and a value; queries are compared with keys, the results become weights, and the values are mixed by those weights. The matrices here are random, so the weights are random too – in a trained model they would show which words really belong together.

import numpy as np

rng = np.random.default_rng(0)
tokens = ["my", "home", "server", "is"]
d = 8                                    # size of each token vector

x = rng.normal(size=(len(tokens), d))    # embeddings: one row (vector) per token
Wq = rng.normal(size=(d, d))             # learned in training; random here
Wk = rng.normal(size=(d, d))
Wv = rng.normal(size=(d, d))

Q = x @ Wq      # query: "what am I looking for?"
K = x @ Wk      # key:   "what do I contain?"
V = x @ Wv      # value: "what do I pass on?"

scores = Q @ K.T / np.sqrt(d)            # how well each query matches each key

# a token may only look at itself and earlier tokens, never at the future
future = np.triu(np.ones_like(scores), k=1).astype(bool)
scores[future] = -np.inf

def softmax(z):
    z = z - z.max(axis=-1, keepdims=True)
    e = np.exp(z)
    return e / e.sum(axis=-1, keepdims=True)

weights = softmax(scores)                # each row now adds up to 1
result = weights @ V                     # mix the values by those weights

for token, row in zip(tokens, weights.round(2)):
    print(f"{token:>7}  looks at: {row}")

Output:

     my  looks at: [1. 0. 0. 0.]
   home  looks at: [1. 0. 0. 0.]
server  looks at: [0.17 0.36 0.47 0.  ]
     is  looks at: [0.05 0.35 0.04 0.56]

Each row shows where one token "looks", and each row adds up to 1. The first token can only look at itself; the last can look at all four. The zeros on the right side are the rule "no looking into the future" – without it, the model could cheat during training by reading the answer.

Example 5 – The skeleton of a GPT, in about 60 lines

This puts everything together in the same order as a real language model: tokenizer, embeddings, transformer blocks (attention plus a small neural network, with residual connections and normalization), output with softmax, and the generation loop with temperature. It uses the same training.txt for its vocabulary.

import numpy as np

rng = np.random.default_rng(42)

# --- 1. Tokenizer: here simply one token per character ---
text = open("training.txt", encoding="utf-8").read().lower()
vocab = sorted(set(text))
to_id = {ch: i for i, ch in enumerate(vocab)}
encode = lambda s: [to_id[c] for c in s]
decode = lambda ids: "".join(vocab[i] for i in ids)

# --- 2. Model size ---
V = len(vocab)   # vocabulary size
D = 32           # size of each token vector (embedding)
T = 64           # context window: max number of tokens the model can see
L = 2            # number of transformer blocks

def p(*shape):   # a new parameter matrix with small random numbers
    return rng.normal(scale=0.02, size=shape)

# --- 3. Parameters (in a real model these are learned in training) ---
params = {
    "tok_emb": p(V, D),          # token -> vector
    "pos_emb": p(T, D),          # position -> vector
    "blocks": [{
        "Wq": p(D, D), "Wk": p(D, D), "Wv": p(D, D), "Wo": p(D, D),
        "W1": p(D, 4 * D), "W2": p(4 * D, D),
    } for _ in range(L)],
    "W_out": p(D, V),            # vector -> score for every token
}

def softmax(z):
    z = z - z.max(axis=-1, keepdims=True)
    e = np.exp(z)
    return e / e.sum(axis=-1, keepdims=True)

def layer_norm(x):
    return (x - x.mean(-1, keepdims=True)) / (x.std(-1, keepdims=True) + 1e-5)

def attention(x, blk):
    Q, K, Vv = x @ blk["Wq"], x @ blk["Wk"], x @ blk["Wv"]
    scores = Q @ K.T / np.sqrt(D)
    scores[np.triu(np.ones_like(scores), k=1).astype(bool)] = -np.inf
    return softmax(scores) @ Vv @ blk["Wo"]

def mlp(x, blk):
    return np.maximum(0, x @ blk["W1"]) @ blk["W2"]   # ReLU in the middle

# --- 4. Forward pass: tokens in, probabilities for the next token out ---
def forward(ids):
    x = params["tok_emb"][ids] + params["pos_emb"][:len(ids)]
    for blk in params["blocks"]:
        x = x + attention(layer_norm(x), blk)   # tokens exchange information
        x = x + mlp(layer_norm(x), blk)         # each token is "thought about"
    logits = layer_norm(x) @ params["W_out"]
    return softmax(logits[-1])                  # only the last position matters

# --- 5. Generation: predict, sample, append, repeat ---
def generate(prompt, n=60, temperature=1.0):
    ids = encode(prompt)
    for _ in range(n):
        probs = forward(ids[-T:])
        probs = softmax(np.log(probs + 1e-12) / temperature)
        ids.append(int(rng.choice(V, p=probs)))
    return decode(ids)

n_params = params["tok_emb"].size + params["pos_emb"].size + params["W_out"].size
n_params += sum(m.size for blk in params["blocks"] for m in blk.values())
print("parameters:", n_params)
print(generate("the cloud "))

Output:

parameters: 30272
the cloud 8oeim-0jkvn3)@?smxd8d3(c["pk)o)djdr?4c:/b2birwmygprvqzb}[(1g

The output is complete nonsense – and that is the most important lesson of this note. The structure is all there, but the parameters are still random. The model has the shape of a GPT but knows nothing, because it has never been trained. To train it, you would add a loss (cross-entropy), calculate the gradients of all 30,272 parameters and repeat the training loop from Example 2 many thousands of times. Writing that by hand is possible but long; with PyTorch, the gradients are calculated automatically.

For comparison: even a "small" model that runs on a home computer has one to a few billion parameters – tens of thousands of times more than this example – and is trained on trillions of tokens instead of one article.

Next steps

1. Watch the code being written. Andrej Karpathy's free course Neural Networks: Zero to Hero starts from backpropagation (similar to Example 2) and ends with building a GPT (similar to Example 5) – and training it.
2. Train a real small model. nanoGPT includes a character-level example small enough to try on an ordinary computer. A graphics card makes it much faster.
3. Use a trained model. To run a ready-made model at home, go back to Constructing an AI.

Is an AI Model Open Source?

After seeing how short the code of a model is, a natural question follows: if I can download a model and run it at home, is it open source? The honest answer is: partly. An AI model is not one thing but three, and each of them can be open or closed.

The three parts of an AI model:

A useful comparison: an open-weight model is like a free program that you may run and adjust, but without the full "source code" of how it was made. You can change its behaviour (with a Modelfile, see Constructing an AI) or train it a little further (fine-tuning), but you cannot rebuild it.

Typical cases (licences change, so always check the current terms):

In 2024 the Open Source Initiative, the organisation that also defines what "open source" means for normal software, published the Open Source AI Definition. In short, it asks for the freedom to use, study, modify and share a model – and for that, the weights and code must be available together with enough information about the training data to build a substantially similar model. By this definition, most "open" models are open-weight, not open source.

How to check the licence of a model yourself:

1. In Ollama, every model in the library has its licence included. Show it with:

   ollama show llama3.2 --license

2. On Hugging Face, each model page has a "License" field on its model card, and you can filter models by licence – for example, show only models under Apache 2.0.
3. Read the important parts: may I use it commercially? May I change it and share the changed version? Are there forbidden uses? For a private home server, almost every open-weight model is fine; for a business or a public website, the details matter.

This is also why the Local AI section of my Digital Resources says that the tools are open source – the models you load into them each have their own licence.

What This Means in Practice

• Why an AI makes things up: it is trained to produce likely text, not true text. When it does not know something, the most likely-sounding continuation can be a confident invention.
• Why it has a knowledge cutoff: its knowledge is frozen in the parameters at the end of training. Newer information can only come in through the text you give it (for example with RAG, see Constructing an AI).
• Why it "forgets" between conversations: the parameters do not change while you use the model. It only "remembers" what is in the current context window.
• Why the training data matters so much: the model learns patterns from its data – including mistakes and biases. Good data is as important as a good architecture.
• Why AI needs so much energy: every token in every answer means a pass through all the parameters. Billions of multiplications per token add up quickly.

A Few More Thoughts

When I started with computers, a program was a list of instructions written by a person: if this, then that. A neural network turns this around. The person writes only the structure and the learning rule; the behaviour comes from the data. The code of a large language model is short enough to read in an afternoon, but the knowledge stored in its billions of numbers cannot be read like code at all.

That is both the fascinating and the uncomfortable part. Understanding the basic mechanism – numbers, a loss, a step downhill, repeated many times – does not make AI less impressive, but it does make it less mysterious. It is a tool built from mathematics and data, with clear strengths and clear limits, and knowing how it works is the best way to use it well.