AI hallucination: why ChatGPT makes things up

1. Introduction: The Hallucination Phenomenon in Language Models

The expansion of large language models (LLMs) like ChatGPT has brought artificial intelligence into everyday life. However, the practical application of these tools has exposed a critical structural limitation: the system's ability to generate responses with a high degree of persuasion and grammatical fluency, yet completely devoid of factual truth or logical rigor. This phenomenon is technically termed artificial hallucination.

Artificial hallucination in LLMs refers to the generation of plausible, coherent, and syntactically well-structured statements that are nonetheless incorrect, invented, or without factual grounding in the original training data or provided context. This is not a deliberate intent to deceive—since the architecture lacks intentionality or consciousness—but rather an improper statistical extrapolation resulting from the sequential generation process.

In the Natural Language Processing (NLP) literature, a fundamental taxonomic distinction is established between two manifestations of the phenomenon:

• Knowledge-based hallucinations (factual errors): occur when the model violates real-world facts, generating incorrect information or attributing fictitious properties to entities, events, and historical data.

• Logic-based hallucinations (reasoning failures): occur when the model fails in the chain of reasoning, resulting in internal contradictions, invalid inferences, or inability to decompose steps in complex problems.

The occurrence of these failures is widely documented in technical literature and real-world use cases:

• Invented literary quotations: When asked to provide a poetic quote from William Shakespeare about artificial intelligence, the model constructed a fluid, metrical, and stylistically convincing sentence that was nonetheless completely fabricated.

• Fictitious case law in the judiciary: In 2023, lawyers in the United States used ChatGPT to draft a legal brief; the model cited nonexistent judicial precedents, inventing case names, dates, and jurisprudence in hyperrealistic fashion.

• False claims with scientific veneer: When asked about the smell of rain, the model claimed that scientists proved in 2018 that "rain smells like lime"—citing a fictitious academic study to lend authority to the response.

• Fictitious academic bibliographies: In an experiment reported by researcher Diogo Cortiz, ChatGPT was prompted to list the books of Professor Lúcia Santaella. The model returned a perfectly structured list with five titles pertinent to the field of semiotics; however, none of the listed books actually existed.

• Distortions of historical facts: In dialogue with journalist Flavio Gomes, the model claimed that driver Ayrton Senna died at the Interlagos circuit after colliding with Satoru Nakajima—an incorrect reconstruction of established historical facts.

The unconditioned proliferation of these inaccuracies carries severe risks when LLMs are integrated into critical sectors such as medicine, law, and finance. In these domains, diagnoses induced by hallucinations, null legal citations, or distorted financial analyses can result in irreversible damage, legal sanctions, and erosion of public trust in the technology.

2. Structural Causes: Why Do LLMs Invent Information?

2.1. Probabilistic Next-Token Prediction vs. Factual Verification

The primary cause of hallucinations lies in the fundamental architecture of LLMs themselves, based on the Transformer model. An LLM does not operate as a relational database or a logical rule system. Instead, it is a neural network designed to learn the probability distribution over sequences of linguistic units (tokens).

During pre-training, the model maps billions of parameters in a multidimensional vector space (embedding space). The mathematical representations of semantically correlated words or concepts are positioned close together in this space. Through attention mechanisms (self-attention), the Transformer calculates the contextual weight between tokens in the input window.

Mathematically, the model's inference reduces to maximizing the conditional probability of the next token w_t, given the previous sequence of tokens (w_1, w_2, \dots, w_{t-1}):

P(w_t \mid w_1, w_2, \dots, w_{t-1})

The model selects the next token by evaluating statistical patterns and co-occurrences learned in the training base. It does not execute a binary truth evaluation function T/F (True/False). The neural network manipulates linguistic forms without possessing semantic anchoring to the physical world, common sense, or an internal truth-checking module. Thus, the Transformer achieves syntactic fluency and stylistic elegance without this implying factual accuracy.

2.2. RLHF Incentives and Evaluation Metrics that Encourage Guessing

Analyses conducted by OpenAI in its September 2025 technical report demonstrate that traditional alignment and evaluation methodologies introduce mathematical incentives that force the model toward overconfidence and guessing.

During alignment via Reinforcement Learning from Human Feedback (RLHF) and in standard evaluation benchmarks (such as multiple-choice tests focused strictly on accuracy rate), the system penalizes abstention.

Considering the dynamics of an objective exam, the model's outputs divide into three categories:

1. Accurate responses: factual and correct information.

2. Errors (Hallucinations): incorrect statements generated with assertive tone.

3. Abstentions: explicit acknowledgment of ignorance (e.g., "I don't have that information").

In traditional metrics based on cross-entropy and RLHF reward functions R(s, a), assigning a response like "I don't know" results in zero score (R = 0). In contrast, attempting a random guess offers positive expected utility (E[R] > 0), given the non-zero probability of hitting the correct answer by mere probabilistic coincidence:

Score for Abstention ("I don't know") = 0
Score for Error (Hallucination)       = 0 (or penalty equivalent to abstention)
Expected Utility of Guess             = P(correct) * Reward > 0

These training parameters condition the neural network to take risks and venture incorrect guesses with high confidence rather than abstain. In practical terms, an error masked by assertive tone is infinitely more harmful to the user than honest abstention, yet prevailing training incentives punish abstention.

2.3. Training Data Limitations and Absence of Grounding

The quality of the pre-training corpus imposes direct constraints on model performance. Since the databases contain terabytes of text extracted from the internet, they involuntarily incorporate conspiracy theories, rumors, outdated information, conceptual errors, and bias.

As highlighted by computer scientist Nina da Hora and digital security specialist Bruno Kawakami, the LLM generalizes and interpolates patterns from the accumulation of this data. Without a native external anchoring mechanism (grounding) during the self-generative pre-training phase—that is, without the intrinsic ability to autonomously consult objective reality—the model replicates and synthesizes inconsistencies as if they were facts.

3. Advanced Mitigation Strategies: Beyond RAG

Mitigating hallucinations requires multiscale interventions that act from optimizing decoding during inference to directly altering neural parameters.

3.1. RAG (Retrieval-Augmented Generation) and Its Limitations

Retrieval-Augmented Generation (RAG) acts as the first line of defense by injecting external factual knowledge at inference time. The system queries vector databases through semantic similarity search and appends retrieved documents to the LLM prompt as context.

Despite its widespread adoption, RAG presents structural constraints:

• Retrieval dependency (Garbage In, Garbage Out): If the retriever returns irrelevant, incomplete, or noisy documents, the LLM will process false premises.

• Absence of intrinsic logical verification: RAG provides factual context but does not prevent the model from making deductive reasoning errors when synthesizing the provided context.

• Latency and computational cost: Vector search and context window expansion increase response latency.

3.2. Knowledge Editing (ROME and MEMIT)

Knowledge Editing techniques perform direct parametric interventions in Transformer layers to correct factuality without needing to rebalance or retrain the entire neural network.

• ROME (Rank-One Model Editing): Maps factual associations to specific Multi-Layer Perceptron (MLP) layers and applies rank-one updates to weight matrices to modify specific neural connections associated with an incorrect fact.

• MEMIT (Model Editing via Memory): Extends ROME's approach to mass editing. Instead of using an external memory module, MEMIT calculates and distributes updates directly across multiple parameters throughout several Transformer MLP layers simultaneously, allowing insertion of thousands of facts without degrading the model's residual knowledge.

3.3. Contrastive Decoding

Contrastive Decoding performs factual filtering at the logits level during the token sampling process. The technique contrasts a main expert model (LLM_{\text{expert}}) with a smaller or skeptical model (LLM_{\text{amateur}}).

The final logits vector for next-token selection is calculated by weighted subtraction of logits from both models:

\text{logit}_{\text{final}} = \text{logit}_{\text{expert}} - \alpha \cdot \text{logit}_{\text{amateur}}

The amateur model (or unaligned model) tends to assign high probability to generic tokens, clichés, or those induced by probabilistic noise. By subtracting these logits, the algorithm penalizes high-entropy choices and forces the expert model to select factually and contextually supported tokens. Complementarily, Factual Nucleus Sampling dynamically adapts the p parameter of top-p sampling based on the factual uncertainty of probability distributions.

3.4. Uncertainty Calibration and Uncertainty-Aware Beam Search

Traditional beam search algorithm maintains the k statistically most probable text trajectories. The Uncertainty-Aware Beam Search variation calculates the distribution's entropy and logits variance at each generated token. Beams exhibiting uncertainty spikes—indicative of factual divergence or imminent hallucination—have their scores drastically pruned, preventing unstable sequences from progressing in generation.

3.5. Models Trained and Evaluated to Admit the Unknown (Abstention)

This approach restructures loss functions in training and reward matrices in RLHF. By severely punishing errors with assertive tone and assigning positive rewards for emitting abstention responses when the model's entropy exceeds a safe threshold, the LLM learns to emit uncertainty statements ("I don't have that information in my context") instead of risking false deductions.

3.6. Self-Refinement and Iterative Reasoning

Self-Refinement implements feedback and internal critique cycles during the inference phase:

1. Primary Generation: The model produces an initial draft.

2. Critical Self-Interrogation: An evaluation module within the model itself analyzes the response for unfounded premises or logical contradictions.

3. Revision: The text is rewritten incorporating corrections identified in the critical phase.

Iterative Querying and Reasoning expands this process by forcing the AI to formulate logical verification sub-questions against its own internal knowledge before delivering the final result to the user.

4. Integration and Agentic Systems: The Future of Reliability

Recent studies demonstrate that no single technique in isolation guarantees complete elimination of hallucinations. The state of the art points to the convergence of multiple strategies in orchestrated Agentic Systems.

       +-------------------------------------------------------+
       |                  AGENTIC SYSTEM                       |
       |                                                       |
       |  +--------------------+       +--------------------+  |
       |  |  Structured RAG    |       |  Knowledge Graphs  |  |
       |  |  (Vector Databases)|       |  (Triple Relations)|  |
       |  +---------+----------+       +---------+----------+  |
       |            |                            |             |
       |            +--------------+-------------+             |
       |                           |                           |
       |                           v                           |
       |         +-----------------------------------+         |
       |         | Chain-of-Thought (CoT) + Tool Use |         |
       |         +-----------------+-----------------+         |
       |                           |                           |
       |                           v                           |
       |         +-----------------------------------+         |
       |         |   Contrastive Decoding &          |         |
       |         |    Knowledge Editing (Parametric) |         |
       |         +-----------------------------------+         |
       +-------------------------------------------------------+

Instead of relying exclusively on the Transformer's probabilistic intuition, Agentic Systems use dynamic planning and external tools to isolate text generation from factual verification.

A central piece in this architecture is the integration of Knowledge Graphs. While vector search in traditional RAG relies on spatial proximity of unstructured embeddings—which frequently fails in multi-hop reasoning queries—Knowledge Graphs organize facts into explicit relational triples. For example:

[\text{Socrates}] \xrightarrow{\text{is a}} [\text{Philosopher}] \xrightarrow{\text{influenced}} [\text{Plato}]

By grounding generation in this deterministic network of entities, the agentic system validates the model's logical inferences against strict ontological relationships.

Combining Knowledge Graphs and RAG with explicit reasoning techniques like Chain-of-Thought (CoT), the agent decomposes a complex instruction into auditable deductive steps. During each intermediate step, the agent executes external tool calls (code verifiers, calculators, fact APIs) and submits intermediate results to contrastive decoding. If a logical divergence is detected, the self-refinement cycle is triggered, balancing the LLM's creativity with the required factual rigor.

5. Conclusion and Practical Mitigation Guidelines

Artificial hallucinations should not be interpreted as random software failures, but rather as logical and structural consequences of the stochastic functioning of Transformer networks, amplified by training incentives that historically and mathematically penalized abstention. Overcoming this challenge requires integrated advancement of new alignment metrics, inference decoding optimization, and agentic architectures fed by structured knowledge.

For developers and users seeking to maximize factual accuracy in daily LLM use, the following practical guidelines are recommended:

1. Perform Independent Verification in Reliable Sources: Never use raw LLM output as a primary source or definitive authority in high-responsibility domains such as medical diagnoses, legal brief foundations, or financial investment decisions.

2. Structure Delimited Prompts with Justification Request: Reduce generation entropy by providing explicit context in the prompt. Require the model to present step-by-step reasoning (Chain-of-Thought) before the final answer, which facilitates identification of broken logical links.

3. Explicitly Instruct Abstention in the Command: Destructure the model's overconfidence bias by inserting abstention clauses in the system prompt, such as: "Answer the question using exclusively the provided data. If the information is not present or if there is any factual doubt, respond strictly 'Information not found' and do not attempt to deduce or guess".

4. Maintain Human Oversight (Human-in-the-Loop) in Critical Environments: Implement workflows where human experts validate outputs generated by agentic systems and LLMs before their practical execution in regulated or high operational risk scenarios.