When someone says “AI” these days, they are almost always thinking of models that can hold a conversation, write text, generate images, summarize documents, write code or answer complex questions. These systems, the most publicly visible ones, are usually built on artificial neural networks. But that expression can be very misleading, because it suggests the machine works “like a brain”. The reality is more interesting and more nuanced: today’s AIs are inspired by certain principles of the biological nervous system, but they are not simulated brains and they do not work the same way. They are mathematical models that learn patterns from huge amounts of data and that, thanks to their scale, can produce behavior that looks intelligent.
1. What exactly is an artificial neural network?
An artificial neural network is a mathematical system made up of simple computing units organized in layers. Each unit, sometimes called an “artificial neuron”, receives numbers, combines them using weights, adds up the results, applies an activation function and outputs a number. That output goes to other neurons in the next layer, and so on until a final result is obtained.
Put very simply, an artificial neuron does something like this:
output = activation_function(weight_1 × input_1 + weight_2 × input_2 + … + bias)
The bias gives AI models more flexibility when learning and helps them draw decision boundaries.
None of this is magic. It is linear algebra, calculus and nonlinear functions repeated millions or billions of times. The power appears when many of these operations are combined in deep layers and adjusted automatically through training.
The word “neural” comes from a historical analogy with the brain. The earliest ideas imitated the way biological neurons appear to connect with one another [1]. But an artificial neuron is not a living cell: it has no membrane, it does not fire real electrical impulses, it does not release neurotransmitters, and it has no consciousness or will of its own. Everything it does is the result of human instructions. It is a mathematical function implemented in digital hardware.
Even so, the architecture is flexible enough to approximate very complex functions [2]. This allows deep neural networks to learn to recognize images, translate languages, generate text, predict proteins [3], drive vehicles or compose music.
2. From text to numbers: tokens, IDs and embeddings
Take a language model. For it, “text” as we perceive it does not exist. Everything has to be converted into numerical representations.
The first step is to split the input into units called tokens. A token can be a whole word, part of a word, a punctuation mark or even single characters. For example:
“The cat sleeps” can be split as [“The”, “cat”, “sleeps”] or similar structures.
The segmentation depends on the vocabulary and the algorithm used by the model [4].
Next, each token is assigned an integer, called a token ID. For example, hypothetically:
“The” → 123 “cat” → 456 “sleeps” → 789
This ID is just a label. It does not mean that the number 456 contains the essence of a cat. It is simply the index the model uses to look up a learned representation.
That representation is called an embedding. Each ID points to a numerical vector, that is, a long list of numbers. In large models, these vectors can have hundreds, thousands or more dimensions.
“cat” → [0.12, -0.83, 0.45, 0.77, …, 0.03]
Words with similar uses tend to occupy nearby or related positions in that space [5]. For example, “dog” and “cat” may be closer to each other than “dog” and “democracy”, although the relationship obviously depends on the context and the model.
It is important to understand this: meaning does not lie in a single number. It is distributed across an entire vector and, above all, in the geometric and statistical relationships among many vectors.
3. Meaning is not fixed: context and internal representations
The word “bank” can mean a financial institution, the side of a river, a blood bank or a bank of fog. Its initial embedding captures a certain general tendency, but it does not fully resolve the ambiguity.
That resolution happens when the model processes the whole sentence. In modern architectures, especially transformers, each token exchanges information with the others through a mechanism called attention [6].
Attention lets the model ask, in mathematical terms: which other parts of the sequence are relevant for interpreting this token?
For example: “I sat on the bank of the river” versus “I went to the bank to withdraw money”.
In the first case, the internal representation of “bank” will tend to be associated with water, grass, a riverside, and so on. In the second, it will be associated with finance, withdrawals, cash, an institution.
So the model does not have a single numerical definition of “bank”. It has multiple contextual representations. Each occurrence of the word is transformed according to its surroundings.
This is crucial for understanding why large models can seem flexible and understanding. They are not applying rigid dictionary rules. They are computing contextual patterns in high-dimensional spaces.
4. Transformers: attention, layers and parallelism
Today’s large language models are usually based on the transformer architecture [6]. A transformer is not a classic “fully connected” neural network, where every neuron simply connects to all the neurons in the next layer. It uses more sophisticated components.
Its main parts include:
- Token embeddings: learned numerical vector representations that turn each token into coordinates in a high-dimensional space where regularities of use are encoded.
- Positional encodings: numerical signals added to the embeddings so the model knows the position and order of each token in the sequence.
- Attention blocks: modules that let each token exchange information with other tokens in the sequence, weighting their relevance through queries, keys and values.
- Feed-forward layers: dense subnetworks inside each transformer block that transform each representation token by token using linear operations and nonlinear functions.
- Residual connections: shortcuts that add a layer’s input to its output.
- Normalizations: operations that rescale and center activations to stabilize and speed up training.
Attention is perhaps the most important concept. It lets each token “look at” other tokens and combine information from them. Technically, queries, keys and values are computed, and relevance weights are assigned between positions in the sequence.
There is no need to go into the full formulas, but the intuitive idea is this:
To understand a word, the model decides which other words in the context matter most.
In a long sentence, this makes it possible to relate elements that are far apart. For example:
“John helped Peter because he was sad”
To interpret “he was sad”, the model has to work out who it refers to. Attention helps assign relative relevance to “John”, “Peter” and the other words.
In addition, transformers process many positions in parallel, which makes them efficient on modern hardware, especially GPUs and TPUs. This is what made it possible to scale models to enormous sizes.
5. How a model generates text: token by token
A language model does not write a complete answer in one go: it generates a sequence of tokens, one after another. That does not mean it never looks ahead. Interpretability studies show that, when writing a rhyming line, the model chooses the rhyming word before it gets there [7], and today’s reasoning models generate intermediate reasoning before giving the final answer [8].
Suppose the input is:
“What is the capital of France?”
The model processes those tokens, builds internal representations and, at the end, produces a probability distribution over every possible token in the vocabulary.
That is, it computes something like this:
“Paris” → 95% “Lyon” → 1% “Marseille” → 1% others → 3%
Then it chooses a token. It may choose the most likely one, or it may use sampling with temperature, top-k, top-p or other criteria [9]. That is why a model can give slightly different answers to the same question.
Once the token has been chosen, it is added to the context:
“What is the capital of France? Paris”
Then it computes the probability of the next token again. It might produce a full stop, add more information or continue with an explanation.
This process repeats until the model emits an end-of-sequence token or reaches a limit.
So the answer is not “stored whole” anywhere. It is built step by step through local probabilistic decisions, conditioned on all the accumulated context.
6. Parameters: the network’s compressed knowledge
When we say a model “knows” things, we should not picture an internal database of stored sentences. What it has are parameters: billions of numbers adjusted during training.
These parameters include:
- attention matrix weights,
- feed-forward layer weights,
- embeddings,
- biases,
- normalization parameters,
- output projections.
During training, the model saw huge amounts of text. In general it did not store that text verbatim, although it can memorize and reproduce some exact passages [10]. What it did was adjust its parameters to minimize prediction errors.
For example, if the model saw:
“The sky is ___” and the next word was often “blue”, “cloudy”, “dark” or “clear”, it learned probabilistic patterns about which tokens tend to follow.
With enough data and enough capacity, these patterns become very rich. The model learns grammar, style, common facts, basic logical structures, code formats, translation, summarization, explanation and many regularities of human language.
But this is not the same as having a model of the world like a person’s. It is an extremely powerful statistical compression of the correlations present in the data.
7. Does AI search through millions of learned facts?
This is a key question. The intuition of “searching through millions of facts” is understandable, but not quite accurate. A pure neural model does not work like a search engine. It has not indexed every sentence it read. It does not do this:
search for relevant document – copy passage – paste answer
What the model does is compute an output from its internal parameters plus the current context.
That kind of searching is done by certain external systems, such as search engines or databases.
The parameters contain learned patterns, but not as separate files. They are distributed throughout the network. Changing a small set of weights can subtly alter many of the model’s abilities.
This explains some curious phenomena. A model can generate sentences it never saw in exactly that form. It can also “hallucinate”, that is, produce false statements with great fluency, because it is optimizing for statistical plausibility, not verified truth [11].
In other words: it does not search through stored data; it generates probable patterns using a compressed internal representation.
That said, many current systems combine neural models with external retrieval. This is known as RAG, Retrieval-Augmented Generation [12]. In these cases, the system does search documents, databases or web pages, and then passes that information to the model so it can generate a better-grounded answer. But the generative core is still a neural model producing tokens conditioned on the expanded context.
References
[1] McCulloch, W. S. and Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 5, 115–133. https://doi.org/10.1007/BF02478259
Historical source. First mathematical model of the neuron as an all-or-nothing logical unit; origin of the analogy between neuron and computing unit.
[2] Hornik, K., Stinchcombe, M. and White, H. (1989). Multilayer feedforward networks are universal approximators. Neural Networks, 2(5), 359–366. https://doi.org/10.1016/0893-6080(89)90020-8
Theoretical framework. Mathematical proof that a network with one hidden layer and enough units can approximate any continuous function. It does not say how many units are needed or how to train them.
[3] Jumper, J. et al. (2021). Highly accurate protein structure prediction with AlphaFold. Nature, 596, 583–589. https://doi.org/10.1038/s41586-021-03819-2
Reliable. Validated in the blind CASP14 competition, with accuracy comparable to experimental methods in most cases.
[4] Sennrich, R., Haddow, B. and Birch, A. (2016). Neural machine translation of rare words with subword units. Proceedings of ACL 2016, 1715–1725. https://doi.org/10.18653/v1/P16-1162
Reliable. Introduces subword segmentation (BPE), used with variations by most current language models.
[5] Mikolov, T., Chen, K., Corrado, G. and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv:1301.3781. https://doi.org/10.48550/arXiv.1301.3781
Reliable. Landmark work on embeddings (word2vec). Published as a preprint, but its results have been widely reproduced.
[6] Vaswani, A. et al. (2017). Attention is all you need. Advances in Neural Information Processing Systems 30 (NIPS 2017). https://doi.org/10.48550/arXiv.1706.03762
Reliable. Original paper on the transformer architecture and the self-attention mechanism.
[7] Lindsey, J. et al. (2025). On the biology of a large language model. Transformer Circuits Thread (Anthropic). https://transformer-circuits.pub/2025/attribution-graphs/biology.html
With reservations. Not peer-reviewed; studies a single closed model (Claude 3.5 Haiku) through a replacement model that approximates it incompletely. Rhyme planning has since been observed in 23 open models (Maar et al., ICLR 2026).
[8] Wei, J. et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35 (NeurIPS 2022). https://doi.org/10.48550/arXiv.2201.11903
Reliable. Shows that generating intermediate steps improves reasoning; the basis for later reasoning models.
[9] Holtzman, A., Buys, J., Du, L., Forbes, M. and Choi, Y. (2020). The curious case of neural text degeneration. ICLR 2020. https://doi.org/10.48550/arXiv.1904.09751
Reliable. Introduces top-p sampling (nucleus sampling) and compares decoding strategies.
[10] Carlini, N. et al. (2021). Extracting training data from large language models. 30th USENIX Security Symposium, 2633–2650. https://doi.org/10.48550/arXiv.2012.07805
Reliable. Extracts hundreds of verbatim sequences from GPT-2’s training data; larger models proved more vulnerable.
[11] Ji, Z. et al. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12). https://doi.org/10.1145/3571730
Reliable. Systematic review of definitions, causes, metrics and mitigation of hallucinations.
[12] Lewis, P. et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems 33 (NeurIPS 2020). https://doi.org/10.48550/arXiv.2005.11401
Reliable. The paper that defines RAG: it combines parametric memory (the model) and non-parametric memory (a document index).