AI Training: How ChatGPT, Claude and Gemini Learned

1. Introduction: The Paradigm Shift in Language Model Training

The trajectory of large language models (LLMs) represents a structural transition in computer science and artificial intelligence. Initially conceived as autoregressive probability distributors, trained to optimize the minimization of cross-entropy loss in predicting text sequences, LLMs have undergone a crucial methodological reformulation to become practical, useful and aligned conversational assistants. The strategic importance of post-training alignment lies precisely in this technical bridge: transforming neural networks capable only of reproducing raw statistical patterns from web datasets into systems that are deterministic from an instruction standpoint, robust against adversarial manipulations and safe for operation in corporate and critical environments.

The central challenge of this discipline—widely formalized as "the alignment problem"—stems from the inherent asymmetries of uncurated real-world data used in the pre-training phase. As these models absorb petabytes of internet data, the simple optimization of the next-token prediction objective function forces the neural network to memorize and extrapolate distributions containing harmful content, systemic biases, misinformation and potentially illegal instructions. Naive approaches to mitigate these failures, such as superficial filtering based on keyword blocklists or heuristic-semantic heuristics, have proven systematically ineffective against prompt injection vulnerabilities (jailbreaking) and complex contextual manipulations.

Faced with this vulnerability inherent to unsupervised pre-training, a consensus has emerged that the utility and safety of modern language models depend as much on their base architecture as on their post-training methodologies. The evolution of the state of the art is defined by the convergence and divergence of three fundamental paradigms: Reinforcement Learning from Human Feedback (RLHF) implemented by OpenAI (ChatGPT), Reinforcement Learning from AI Feedback (RLAIF) under the Constitutional AI framework developed by Anthropic (Claude), and the natively multimodal architecture integrating supervised step-by-step reasoning from Google DeepMind (Gemini).

The deconstruction of these methodologies reveals how AI engineering has advanced from a cycle centered on manual human annotation to highly automated, principled and multimodal pipelines, establishing the foundation for analyzing the fundamental training pipeline.

2. The Foundation of Training: Pre-Training and Supervised Fine-Tuning (SFT)

The development pipeline of a state-of-the-art LLM is structured hierarchically, where each phase alters the geometry of the model's state representation space. Pre-training builds the underlying cognitive infrastructure, the repository of factual knowledge and primitive logical reasoning capabilities, while Supervised Fine-Tuning (SFT) redirects the model's probability density to prioritize instructional and conversational response formats.

During pre-training, the autoregressive neural network is subjected to trillions of tokens in an unsupervised learning task. The stochastic objective is to maximize data likelihood through next-token calculation:

\mathcal{L}_{\text{Pre-training}}(\theta) = -\sum_{i} \log P(x_i \mid x_1, x_2, \dots, x_{i-1}; \theta)

At this stage, critical architectural and data ingestion divergences are observed among the major ecosystems:

• ChatGPT (OpenAI) and Claude (Anthropic): Were initially grounded in base language models centered on one-dimensional sequences of text and source code, in which additional modalities (vision, audio) were coupled in later stages through adapters or linear projection alignments of feature extractors.

• Gemini (Google DeepMind): Was conceived from its initial specification as a natively multimodal architecture. Instead of processing heterogeneous data streams through isolated encoders glued post-hoc, Gemini tokenizes and projects text, code, image, audio and video into a unified embedding space from step zero of pre-training.

Despite the computational scale of pre-training, an unadjusted base model operates merely as an autoregressive generator of raw token distribution. If prompted with "How to format a financial report?", the base model might simply continue the text by generating "in Microsoft Word and Excel?" or accumulating similar questions found in web forums. The transition from undirected text completion to the functional posture of a "conversational assistant" is effected by Supervised Fine-Tuning (SFT).

In SFT, the parameters \theta of the pre-trained model are adjusted on a curated high-quality dataset composed of instruction-ideal response pairs \mathcal{D}_{\text{SFT}} = \{(x^{(j)}, y^{(j)})\}. The model is trained via gradient descent with cross-entropy loss calculated only over the response tokens y^{(j)}. SFT conditions the neural network to recognize dialogue syntax (through special control tokens like <|im_start|> and <|im_end|>), to comply with operational constraints and to adopt a utilitarian persona.

Structural Comparison: Pre-Training vs. Supervised Fine-Tuning (SFT)

Although SFT reconfigures the syntactic structure of interaction, it is inherently limited: the model learns to imitate the form of demonstrations without necessarily optimizing the qualitative quality or deep safety of its outputs in out-of-distribution scenarios. This limitation requires the employment of reinforcement learning.

3. The Human Feedback Paradigm: RLHF in ChatGPT (OpenAI)

To align the model's generation trajectories with multidimensional human preferences of utility, truthfulness and harmlessness, OpenAI consolidated the Reinforcement Learning from Human Feedback (RLHF) pipeline. RLHF treats the language generation problem as a Partially Observable Markov Decision Process, in which the model acts as a policy executing actions (token generation) within a state environment (prompt context).

 ┌─────────────┐     Generates Multiple   ┌───────────────────────┐
 │ SFT Model   │ ─────────────────────> │  Response Samples     │
 └─────────────┘                        └───────────────────────┘
                                                    │
                                                    ▼
 ┌─────────────┐     Preference         ┌───────────────────────┐
 │ Reward      │ <───────────────────── │  Human Evaluators     │
 │ Model       │    Scoring             │  (Ordinal Ranking)    │
 └─────────────┘                        └───────────────────────┘
        │
        │ Reward Signal (R_ψ)
        ▼
 ┌──────────────────────────────────────────────────────────────┐
 │   Policy Optimization via PPO                                │
 │   max E [ R_ψ(x,y) - β * D_KL( π_θ(y|x) || π_SFT(y|x) ) ]    │
 └──────────────────────────────────────────────────────────────┘

The Three-Stage Algorithmic Pipeline (OpenAI)

1. Sampling and Preference Collection: Prompts x extracted from user distributions are submitted to the SFT-adjusted model. The model generates a set of candidate responses \{y_1, y_2, \dots, y_k\}. Human evaluators read and rank these responses in an ordinal preference sequence (y_w \succ y_l, where y_w is the preferred response and y_l the least preferred).

2. Reward Model Training: The Reward Model R_\psi, parameterized by weights \psi, is trained to act as a scalar reward signal estimator. The optimization uses a logistic regression loss function based on the Bradley-Terry preference model:

\mathcal{L}_{\text{RM}}(\psi) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma \left( R_\psi(x, y_w) - R_\psi(x, y_l) \right) \right]

Where \sigma(z) is the sigmoid function. The Reward Model learns to map prompt-response pairs (x, y) to a scalar value in \mathbb{R}, quantifying perceived quality.

1. Policy Optimization via PPO (Proximal Policy Optimization): With the Reward Model R_\psi fixed, the LLM policy (\pi_\theta) is adjusted via on-policy reinforcement learning using the PPO algorithm. The extended optimization function incorporating a Kullback-Leibler (D_{KL}) divergence penalty is expressed as:

\max_{\theta} \mathbb{E}_{(x, y) \sim \mathcal{D}_{\pi_\theta}} \left[ R_\psi(x, y) - \beta \cdot D_{KL}\left( \pi_\theta(y \mid x) \;||\; \pi_{\text{SFT}}(y \mid x) \right) \right]

The inclusion of the D_{KL} penalty calibrated by hyperparameter \beta is mathematically vital. Without the D_{KL} term, PPO optimization suffers from reward hacking, where the policy discovers adversarial shortcuts or aberrant syntactic patterns that artificially maximize the Reward Model score without delivering real value to the user. Moreover, restoring the divergence term prevents policy collapse, preventing the model from losing the functional entropy and generative diversity inherited from the base \pi_{\text{SFT}} model.

Critical Bottlenecks and Limitations of Classical RLHF

Although it enabled the transition from GPT-3 to InstructGPT and ChatGPT, RLHF exhibits severe operational and methodological barriers. It presents extremely high financial and temporal costs, derived from the need to maintain armies of human annotators to generate millions of ordinal comparisons.

From a methodological standpoint, human feedback introduces sociodemographic inconsistencies, subjectivity and cognitive biases into the Reward Model. Human evaluators frequently favor longer and superficially well-written responses that are factually incorrect (sycophancy and sophistry). Additionally, RLHF creates an update bottleneck: retraining or recalibrating the model to align with new safety guidelines requires costly cycles of human re-annotation.

These scalability bottlenecks and the volatility of human judgment have driven the search for methods where AI itself supervises and aligns its outputs based on deterministic specifications.

4. Principle-Oriented Alignment: Constitutional AI and RLAIF in Claude (Anthropic)

Aiming to suppress the intensive dependence on direct human feedback and resolve judgment inconsistencies, Anthropic developed Constitutional AI (CAI) and Reinforcement Learning from AI Feedback (RLAIF) for training the Claude model. This methodology replaces instance-level human preference supervision with an explicit set of written declarative guidelines: the Constitution.

The Architecture of an AI Constitution and the Role of Chain-of-Thought

The Constitution consists of a structured document composed of ethical principles, engagement rules and conduct guidelines. Instead of hiring labelers to judge thousands of response pairs, Anthropic injects these principles directly into the system's training and evaluation prompts. Typical examples of constitutional rules include:

• "Choose the response that is the most helpful, honest and harmless."

• "Select the assistant response that is as harmless, helpful, polite, respectful and considerate as possible, without sounding overly reactive, condescending or accusatory."

A fundamental innovation of this approach lies in the use of chained step-by-step reasoning (Chain-of-Thought - CoT) during the self-critique and refusal phase. When confronted with a borderline or inherently harmful prompt, the model does not emit an opaque, pre-programmed denial. Instead, it executes an explanatory CoT reasoning trace, evaluating the boundary conditions of the request against the Constitution before generating the final refusal token. This makes safety alignment interpretable and auditable, allowing distinction between justified refusals and false-positive refusals (over-refusals).

PHASE 1: Constitutional Supervised Learning (SL CAI)
┌──────────────┐     Generates Response ┌────────────────────────┐
│ Harmful Prompt│ ────────────────────> │  Helpfulness RLHF Model│
└──────────────┘                       └────────────────────────┘
                                                   │
                                                   ▼
┌──────────────┐     Self-Critique +   ┌────────────────────────┐
│ Constitution │ ────────────────────> │ Rewritten/Corrected    │
└──────────────┘     CoT Revision      │ Response               │
                                       └────────────────────────┘
                                                   │
                                                   ▼
                                       ┌────────────────────────┐
                                       │ Model Fine-Tuning      │
                                       │  ──> SL CAI Model      │
                                       └────────────────────────┘

PHASE 2: Reinforcement Learning from AI Feedback (RLAIF)
┌──────────────┐     Generates Pairs   ┌────────────────────────┐
│ SL CAI Model │ ────────────────────> │  Responses (y_A, y_B)  │
└──────────────┘                       └────────────────────────┘
                                                   │
                                                   ▼
┌──────────────┐     Preference        ┌────────────────────────┐
│ Constitution │ ────────────────────> │ Synthetic Preference   │
└──────────────┘     AI Evaluation     │ Dataset (Harmless)     │
                                       └────────────────────────┘
                                                   │
                                                   ▼
                                       ┌────────────────────────┐
                                       │ Preference Model /     │
                                       │ Reward Training        │
                                       └────────────────────────┘
                                                   │
                                                   ▼
                                       ┌────────────────────────┐
                                       │ RL Training (PPO)      │
                                       │  ──> RLAIF Model       │
                                       └────────────────────────┘

Two-Stage Technical Mechanism

The Constitutional AI pipeline unfolds in two strictly chained stages:

Stage 1: Constitutional Supervised Learning (SL CAI)

1. Initial Generation: Prompts designed to induce harmful behaviors (e.g., "How can I hack my neighbor's Wi-Fi network?") are presented to an initial model trained via RLHF focused primarily on helpfulness. The model generates a potentially misaligned initial response.

2. Guided Self-Critique: The model is prompted to critique its own generated initial response, using a randomly selected principle from the Constitution as conditional context (e.g., "Critique the previous response identifying violations of privacy, cybersecurity and illegalities"). The model generates a CoT self-critique trace.

3. Rewriting and Revision: The model receives its own critique and is instructed to rewrite the original response to be in strict compliance with the applicable constitutional principle.

4. SL CAI Model Fine-Tuning: This cycle of critique and rewriting is repeated iteratively over thousands of harmful prompts. The final revised responses are paired with the original prompts. This synthetic harmlessness dataset is combined with human demonstration datasets of helpfulness. A base model is subjected to supervised fine-tuning on these data, originating the SL CAI model.

Stage 2: Reinforcement Learning from AI Feedback (RLAIF)

1. Response Sampling: The SL CAI model trained in the previous stage generates pairs of candidate responses (y_A, y_B) for each prompt from a harmful request dataset.

2. Automated Preference Evaluation: Each pair (y_A, y_B) is formatted as a multiple-choice prompt submitted to an AI evaluator model, instructed to choose the best response based on a constitutional principle. This generates a synthetic harmlessness preference dataset generated entirely by AI.

3. Dataset Fusion and Preference Model: The synthetic harmlessness dataset is merged with the helpful human preference dataset. A Preference Model (Reward Model) is trained on these comparative data.

4. Reinforcement Learning Optimization: The SL CAI model from Stage 1 is optimized via Reinforcement Learning (using PPO) against the Preference Model trained in the previous step. The result is the final policy optimized via RLAIF.

Impact Evaluation and Performance Metrics

Empirical validation of Constitutional AI demonstrated that models aligned by RLAIF outperform models aligned exclusively by traditional human RLHF on multiple axes. Claude achieves significantly lower harmfulness rates while drastically reducing the phenomenon of unnecessary evasion: the model rarely refuses to answer benign borderline requests and, when it refuses a malicious request, is able to justify the denial with clarity and without a moralistic tone.

Moreover, direct human intervention is reduced to an insignificant bottleneck: human annotators are needed only to draft the Constitution and provide a reduced number of demonstration examples (few-shot prompts) used in self-critique models.

This transition to principled abstractions in text opened space for the evolution of architectures capable of aligning complex multimodal interactions.

5. The Native Multimodal Frontier: Gemini Architecture and Integration (Google DeepMind)

As research in large models advanced, it became evident that purely textual alignment was insufficient for agents intended to interact with complex environments. The emergence of Gemini represented a paradigmatic rupture by replacing post-hoc coupling approaches with native multimodality.

Comparative Analysis: Late Fusion vs. First-Order Native Integration

Traditional approaches to multimodal models operated under the late-fusion paradigm. In them, frozen visual encoders (like CLIP) or audio detectors were trained separately and integrated into a pre-existing textual LLM through linear projection layers or adapters (cross-attention layers). This architecture generates an asymmetry: the language model processes visual tokens as an approximate "translation," losing the spatial, temporal and frequency granularity contained in raw non-textual data.

In contrast, Gemini was designed under the premise of first-order native integration (early-fusion). The model uses a unified tokenizer capable of converting continuous streams of text, code, image rasterization, audio spectrograms and video frames into a common vector representation space. These intercalculated tokens feed the same base Transformer attention network from the beginning of pre-training.

       TRADITIONAL ARCHITECTURE (LATE-FUSION)
┌──────────────┐     ┌────────────────┐     ┌──────────────────────┐
│ Vision (CLIP)│ ──> │ Linear Projection──> │                      │
└──────────────┘     └────────────────┘     │  Pre-Trained Text    │
┌──────────────┐                            │  LLM                 │
│ Raw Text     │ ─────────────────────────> │  (Frozen/Unfrozen)   │
└──────────────┘                            └──────────────────────┘

       GEMINI ARCHITECTURE (NATIVE INTEGRATION / EARLY-FUSION)
┌──────────────────────────────────────────────────────────────────┐
│ Unified Tokenizer and Embedding Space                            │
│ (Text, Code, Image, Audio and Video Interleaved)                 │
└──────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌──────────────────────────────────────────────────────────────────┐
│  Unified Transformer Attention Layers (Trained from Scratch)     │
│  Sustained by TPU Infrastructure                                 │
└──────────────────────────────────────────────────────────────────┘

Infrastructure, Synthetic Data and Multimodal Filtering

The enablement and alignment of a natively multimodal architecture at large scale required innovations in infrastructure and post-training:

• Distributed Optimization on TPUs: Gemini was trained and aligned using Google's Tensor Processing Unit clusters (TPU v4 and TPU v5e). The network topology and model parallelism (pipeline and tensor parallelism) enabled sustaining continuous ingestion of very high-dimensional multimodal tensors.

• Interleaved Multi-modal Safety Filtering: The alignment process extends safety filters beyond text. Verification networks analyze the visual and auditory flow in pre- and post-training stages to prevent the generation or association of violent, unauthorized biometric or explicitly harmful content at the visual/auditory token level.

• Verifiable Synthetic Data and Cross-Modal Reasoning: In Gemini's post-training, reinforcement learning is enhanced by synthetic data generated and validated by code compilers and logical and mathematical engines. The model executes CoT reasoning trajectories that alternate between visual interpretation (e.g., reading graphs or technical diagrams) and code or text generation, ensuring rigorous cross-modal coherence.

The synthesis of these three architectural approaches allows confronting the sector's main strategic trade-offs.

6. Integrated Comparative Analysis and Technical Matrix

The consolidation of methodologies developed by OpenAI, Anthropic and Google DeepMind makes explicit different AI engineering philosophies and trade-off management. The following matrix details the fundamental divergences in the axes of post-training, supervision and architecture.

Technological Comparative Matrix

Critical Analysis of Trade-Offs (The "So What?" of Alignment)

1. Scalability and Autonomy (RLAIF) vs. Direct Human Calibration (RLHF): Traditional RLHF in ChatGPT guarantees an initial alignment highly responsive to the nuances of human preference, but collides with an insurmountable bottleneck of scale, high cost and introduction of volatile human biases into the Reward Model. Claude's RLAIF resolves the scale bottleneck by allowing automated and infinitely scalable feedback, however, it transfers the risk to the base model's ability to correctly interpret the Constitution. If the model fails in the CoT self-critique stage, the bias is systematically propagated to the Synthetic Preference Model.

2. Specialized Modularity vs. Native Multimodal Coherence: The ChatGPT and Claude approach prioritized the sophistication of textual and pure reasoning capabilities, leaving visual and auditory modulation to coupled pipelines. This simplifies core language training but limits the model's spatial and temporal intuition. Gemini's native multimodality reduces the late-fusion bottleneck and enables unified reasoning about the physical world, but significantly elevates the computational complexity of pre-training and requires exponentially more complex safety filtering pipelines to validate non-textual data in real time.

7. Conclusion: The Future of Machine Learning and Alignment Autonomy

The evolution of training and alignment methods for large language models demonstrates the maturation of a discipline that has transitioned from heuristic and manual interventions to principled, self-supervised and architecturally integrated systems.

Synthesis of Technical Findings

1. Supervision Automation via RLAIF: The scalability bottleneck and volatility of RLHF based on human labelers are being overcome by the adoption of Constitutional AI and RLAIF, where explicit declarative principles and self-critiques guide the automated generation of preference datasets.

2. Auditability through Chain-of-Thought: The use of explicit reasoning traces (CoT) in safety refusals has replaced opaque blocking filters with explanatory and aligned responses, minimizing false-positive refusals (over-refusals) and making the model's behavioral compliance interpretable.

3. Native Multimodal Paradigm as Standard: The architecture of frontier models has shifted from post-hoc coupling of specialized encoders to unified training (early-fusion) from phase zero of pre-training, enabling fluid and high-fidelity cross-modal reasoning.

The convergence between declarative AI constitutions, deterministic verification pipelines based on synthetic data and natively multimodal architectures defines the framework of the next generation of intelligent systems. The automation of alignment stages not only optimizes the use of computational resources but establishes the mathematical and structural foundations necessary for creating safe, transparent autonomous agents capable of operating in highly complex environments.