Lil'Log · 2026-07-04
<p>The concept of <strong>recursive self-improvement (RSI)</strong> dates back to <a href="https://philpapers.org/rec/GOOSCT">I. J. Good (1965)</a>, where he defined an “ultraintelligent machine” as a system that can surpass humans in all intellectual activities and design better machines to improve itself. <a href="https://www.lesswrong.com/posts/JBadX7rwdcRFzGuju/recursive-self-improvement">Yudkowsky (2008)</a> used the phrase “recursive self-improvement” for a specific feedback loop: an AI uses its current intelligence to improve the cognitive machinery that produces its intelligence.</p>
<p>This feedback loop in modern AI may indicate the model rewriting its own weights directly, or more broadly the model improves the <em>training pipeline</em> and the <em>deployment system</em>, which in turn enables a better successor model with improved performance across economically valuable tasks. The speed of research development in AI has been shown to drastically accelerated in frontier labs (<a href="https://www.anthropic.com/institute/recursive-self-improvement">Anthropic</a>; <a href="https://openai.com/index/how-agents-are-transforming-work/">OpenAI</a>).</p>
Open original
Lil'Log · 2026-06-24
<p>Scaling laws are one of the most critical empirical findings in deep learning. The observation is simple in form: the training loss $L$ decreases predictably as we scale up model size $N$, dataset size $D$, and compute $C$, following a power-law curve, which appears as a straight line on a log-log plot. We can view scaling laws as a framework for describing the relationship between compute, loss, model size and data; at its core, it is about how to allocate precious compute optimally between $N$ and $D$.</p>
Open original
Lil'Log · 2025-05-01
<p><span class="update">Special thanks to <a href="https://scholar.google.com/citations?user=itSa94cAAAAJ&hl=en">John Schulman</a> for a lot of super valuable feedback and direct edits on this post.</span></p>
<p>Test time compute (<a href="https://arxiv.org/abs/1603.08983">Graves et al. 2016</a>, <a href="https://arxiv.org/abs/1705.04146">Ling, et al. 2017</a>, <a href="https://arxiv.org/abs/2110.14168">Cobbe et al. 2021</a>) and Chain-of-thought (CoT) (<a href="https://arxiv.org/abs/2201.11903">Wei et al. 2022</a>, <a href="https://arxiv.org/abs/2112.00114">Nye et al. 2021</a>), have led to significant improvements in model performance, while raising many research questions. This post aims to review recent developments in how to effectively use test-time compute (i.e. “thinking time”) and why it helps.</p>
Open original
Lil'Log · 2024-11-28
<p>Reward hacking occurs when a <a href="(https://lilianweng.github.io/posts/2018-02-19-rl-overview/)">reinforcement learning (RL)</a> agent <a href="https://lilianweng.github.io/posts/2018-01-23-multi-armed-bandit/#exploitation-vs-exploration">exploits</a> flaws or ambiguities in the reward function to achieve high rewards, without genuinely learning or completing the intended task. Reward hacking exists because RL environments are often imperfect, and it is fundamentally challenging to accurately specify a reward function.</p>
<p>With the rise of <a href="https://lilianweng.github.io/posts/2019-01-31-lm/">language models</a> generalizing to a broad spectrum of tasks and RLHF becomes a de facto method for alignment training, reward hacking in RL training of language models has become a critical practical challenge. Instances where the model learns to modify unit tests to pass coding tasks, or where responses contain biases that mimic a user’s preference, are pretty concerning and are likely one of the major blockers for real-world deployment of more autonomous use cases of AI models.</p>
Open original
Lil'Log · 2024-07-07
<p>Hallucination in large language models usually refers to the model generating unfaithful, fabricated, inconsistent, or nonsensical content. As a term, hallucination has been somewhat generalized to cases when the model makes mistakes. Here, I would like to narrow down the problem of hallucination to cases where the model output is fabricated and <strong>not grounded</strong> by either the provided context or world knowledge.</p>
<p>There are two types of hallucination:</p>
<ol>
<li>In-context hallucination: The model output should be consistent with the source content in context.</li>
<li>Extrinsic hallucination: The model output should be grounded by the pre-training dataset. However, given the size of the pre-training dataset, it is too expensive to retrieve and identify conflicts per generation. If we consider the pre-training data corpus as a proxy for world knowledge, we essentially try to ensure the model output is factual and verifiable by external world knowledge. Equally importantly, when the model does not know about a fact, it should say so.</li>
</ol>
<p>This post focuses on extrinsic hallucination. To avoid hallucination, LLMs need to be (1) factual and (2) acknowledge not knowing the answer when applicable.</p>
Open original
Lil'Log · 2024-04-12
<p><a href="https://lilianweng.github.io/posts/2021-07-11-diffusion-models/">Diffusion models</a> have demonstrated strong results on image synthesis in past years. Now the research community has started working on a harder task—using it for video generation. The task itself is a superset of the image case, since an image is a video of 1 frame, and it is much more challenging because:</p>
<ol>
<li>It has extra requirements on temporal consistency across frames in time, which naturally demands more world knowledge to be encoded into the model.</li>
<li>In comparison to text or images, it is more difficult to collect large amounts of high-quality, high-dimensional video data, let along text-video pairs.</li>
</ol>
<blockquote>
<p><br/><b>
🥑 Required Pre-read: Please make sure you have read the previous blog on <a href="https://lilianweng.github.io/posts/2021-07-11-diffusion-models/">“What are Diffusion Models?”</a> for image generation before continue here.
</b><br/><br/></p>
Open original
Lil'Log · 2024-02-05
<p><span class="update">[Special thank you to <a href="https://scholar.google.com/citations?user=FRBObOwAAAAJ&hl=en">Ian Kivlichan</a> for many useful pointers (E.g. the 100+ year old Nature paper “Vox populi”) and nice feedback. 🙏 ]</span><br/></p>
<p>High-quality data is the fuel for modern data deep learning model training. Most of the task-specific labeled data comes from human annotation, such as classification task or <a href="https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/#rl-fine-tuning-with-human-preferences">RLHF</a> labeling (which can be constructed as classification format) for LLM alignment training. Lots of ML techniques in the post can help with data quality, but fundamentally human data collection involves attention to details and careful execution. The community knows the value of high quality data, but somehow we have this subtle impression that “Everyone wants to do the model work, not the data work” (<a href="https://dl.acm.org/doi/abs/10.1145/3411764.3445518">Sambasivan et al. 2021</a>).</p>
Open original
Lil'Log · 2023-10-25
<p>The use of large language models in the real world has strongly accelerated by the launch of ChatGPT. We (including my team at OpenAI, shoutout to them) have invested a lot of effort to build default safe behavior into the model during the alignment process (e.g. via <a href="https://openai.com/research/learning-to-summarize-with-human-feedback">RLHF</a>). However, adversarial attacks or jailbreak prompts could potentially trigger the model to output something undesired.</p>
<p>A large body of ground work on adversarial attacks is on images, and differently it operates in the continuous, high-dimensional space. Attacks for discrete data like text have been considered to be a lot more challenging, due to lack of direct gradient signals. My past post on <a href="https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/">Controllable Text Generation</a> is quite relevant to this topic, as attacking LLMs is essentially to control the model to output a certain type of (unsafe) content.</p>
Open original
Lil'Log · 2023-06-23
<p>Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as <a href="https://github.com/Significant-Gravitas/Auto-GPT">AutoGPT</a>, <a href="https://github.com/AntonOsika/gpt-engineer">GPT-Engineer</a> and <a href="https://github.com/yoheinakajima/babyagi">BabyAGI</a>, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver.</p>
<h1 id="agent-system-overview">Agent System Overview</h1>
<p>In a LLM-powered autonomous agent system, LLM functions as the agent’s brain, complemented by several key components:</p>
<ul>
<li><strong>Planning</strong>
<ul>
<li>Subgoal and decomposition: The agent breaks down large tasks into smaller, manageable subgoals, enabling efficient handling of complex tasks.</li>
<li>Reflection and refinement: The agent can do self-criticism and self-reflection over past actions, learn from mistakes and refine them for future steps, thereby improving the quality of final results.</li>
</ul>
</li>
<li><strong>Memory</strong>
<ul>
<li>Short-term memory: I would consider all the in-context learning (See <a href="https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/">Prompt Engineering</a>) as utilizing short-term memory of the model to learn.</li>
<li>Long-term memory: This provides the agent with the capability to retain and recall (infinite) information over extended periods, often by leveraging an external vector store and fast retrieval.</li>
</ul>
</li>
<li><strong>Tool use</strong>
<ul>
<li>The agent learns to call external APIs for extra information that is missing from the model weights (often hard to change after pre-training), including current information, code execution capability, access to proprietary information sources and more.</li>
</ul>
</li>
</ul>
<figure>
<img src="agent-overview.png" style="width: 100%;" />
<figcaption>Overview of a LLM-powered autonomous agent system.</figcaption>
</figure>
<h1 id="component-one-planning">Component One: Planning</h1>
<p>A complicated task usually involves many steps. An agent needs to know what they are and plan ahead.</p>
Open original
Lil'Log · 2023-03-15
<p><strong>Prompt Engineering</strong>, also known as <strong>In-Context Prompting</strong>, refers to methods for how to communicate with LLM to steer its behavior for desired outcomes <em>without</em> updating the model weights. It is an empirical science and the effect of prompt engineering methods can vary a lot among models, thus requiring heavy experimentation and heuristics.</p>
<p>This post only focuses on prompt engineering for autoregressive language models, so nothing with Cloze tests, image generation or multimodality models. At its core, the goal of prompt engineering is about alignment and model steerability. Check my <a href="https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/">previous post</a> on controllable text generation.</p>
Open original
Lil'Log · 2023-01-27
<p>Many new Transformer architecture improvements have been proposed since my last post on <a href="https://lilianweng.github.io/posts/2020-04-07-the-transformer-family/"><ins>“The Transformer Family”</ins></a> about three years ago. Here I did a big refactoring and enrichment of that 2020 post — restructure the hierarchy of sections and improve many sections with more recent papers. Version 2.0 is a superset of the old version, about twice the length.</p>
<h1 id="notations">Notations</h1>
<table>
<thead>
<tr>
<th>Symbol</th>
<th>Meaning</th>
</tr>
</thead>
<tbody>
<tr>
<td>$d$</td>
<td>The model size / hidden state dimension / positional encoding size.</td>
</tr>
<tr>
<td>$h$</td>
<td>The number of heads in multi-head attention layer.</td>
</tr>
<tr>
<td>$L$</td>
<td>The segment length of input sequence.</td>
</tr>
<tr>
<td>$N$</td>
<td>The total number of attention layers in the model; not considering MoE.</td>
</tr>
<tr>
<td>$\mathbf{X} \in \mathbb{R}^{L \times d}$</td>
<td>The input sequence where each element has been mapped into an embedding vector of shape $d$, same as the model size.</td>
</tr>
<tr>
<td>$\mathbf{W}^k \in \mathbb{R}^{d \times d_k}$</td>
<td>The key weight matrix.</td>
</tr>
<tr>
<td>$\mathbf{W}^q \in \mathbb{R}^{d \times d_k}$</td>
<td>The query weight matrix.</td>
</tr>
<tr>
<td>$\mathbf{W}^v \in \mathbb{R}^{d \times d_v}$</td>
<td>The value weight matrix. Often we have $d_k = d_v = d$.</td>
</tr>
<tr>
<td>$\mathbf{W}^k_i, \mathbf{W}^q_i \in \mathbb{R}^{d \times d_k/h}; \mathbf{W}^v_i \in \mathbb{R}^{d \times d_v/h}$</td>
<td>The weight matrices per head.</td>
</tr>
<tr>
<td>$\mathbf{W}^o \in \mathbb{R}^{d_v \times d}$</td>
<td>The output weight matrix.</td>
</tr>
<tr>
<td>$\mathbf{Q} = \mathbf{X}\mathbf{W}^q \in \mathbb{R}^{L \times d_k}$</td>
<td>The query embedding inputs.</td>
</tr>
<tr>
<td>$\mathbf{K} = \mathbf{X}\mathbf{W}^k \in \mathbb{R}^{L \times d_k}$</td>
<td>The key embedding inputs.</td>
</tr>
<tr>
<td>$\mathbf{V} = \mathbf{X}\mathbf{W}^v \in \mathbb{R}^{L \times d_v}$</td>
<td>The value embedding inputs.</td>
</tr>
<tr>
<td>$\mathbf{q}_i, \mathbf{k}_i \in \mathbb{R}^{d_k}, \mathbf{v}_i \in \mathbb{R}^{d_v}$</td>
<td>Row vectors in query, key, value matrices, $\mathbf{Q}$, $\mathbf{K}$ and $\mathbf{V}$.</td>
</tr>
<tr>
<td>$S_i$</td>
<td>A collection of key positions for the $i$-th query $\mathbf{q}_i$ to attend to.</td>
</tr>
<tr>
<td>$\mathbf{A} \in \mathbb{R}^{L \times L}$</td>
<td>The self-attention matrix between a input sequence of lenght $L$ and itself. $\mathbf{A} = \text{softmax}(\mathbf{Q}\mathbf{K}^\top / \sqrt{d_k})$.</td>
</tr>
<tr>
<td>$a_{ij} \in \mathbf{A}$</td>
<td>The scalar attention score between query $\mathbf{q}_i$ and key $\mathbf{k}_j$.</td>
</tr>
<tr>
<td>$\mathbf{P} \in \mathbb{R}^{L \times d}$</td>
<td>position encoding matrix, where the $i$-th row $\mathbf{p}_i$ is the positional encoding for input $\mathbf{x}_i$.</td>
</tr>
</tbody>
</table>
<h1 id="transformer-basics">Transformer Basics</h1>
<p>The <strong>Transformer</strong> (which will be referred to as “vanilla Transformer” to distinguish it from other enhanced versions; <a href="https://arxiv.org/abs/1706.03762">Vaswani, et al., 2017</a>) model has an encoder-decoder architecture, as commonly used in many <a href="https://lilianweng.github.io/posts/2018-06-24-attention/#born-for-translation">NMT</a> models. Later simplified Transformer was shown to achieve great performance in language modeling tasks, like in encoder-only <a href="https://lilianweng.github.io/posts/2019-01-31-lm/#bert">BERT</a> or decoder-only <a href="https://lilianweng.github.io/posts/2019-01-31-lm/#openai-gpt">GPT</a>.</p>
Open original
Lil'Log · 2023-01-10
<p><span class="update">[Updated on 2023-01-24: add a small section on <a href="#distillation">Distillation</a>.]</span><br/></p>
<p>Large transformer models are mainstream nowadays, creating SoTA results for a variety of tasks. They are powerful but very expensive to train and use. The extremely high inference cost, in both time and memory, is a big bottleneck for adopting a powerful transformer for solving real-world tasks at scale.</p>
<p><strong>Why is it hard to run inference for large transformer models?</strong> Besides the increasing size of SoTA models, there are two main factors contributing to the inference challenge (<a href="https://arxiv.org/abs/2211.05102">Pope et al. 2022</a>):</p>
Open original
Lil'Log · 2022-09-08
<p>Neural networks are <a href="https://lilianweng.github.io/posts/2019-03-14-overfit/">well known</a> to be over-parameterized and can often easily fit data with near-zero training loss with decent generalization performance on test dataset. Although all these parameters are initialized at random, the optimization process can consistently lead to similarly good outcomes. And this is true even when the number of model parameters exceeds the number of training data points.</p>
<p><strong>Neural tangent kernel (NTK)</strong> (<a href="https://arxiv.org/abs/1806.07572">Jacot et al. 2018</a>) is a kernel to explain the evolution of neural networks during training via gradient descent. It leads to great insights into why neural networks with enough width can consistently converge to a global minimum when trained to minimize an empirical loss. In the post, we will do a deep dive into the motivation and definition of NTK, as well as the proof of a deterministic convergence at different initializations of neural networks with infinite width by characterizing NTK in such a setting.</p>
Open original
Lil'Log · 2022-06-09
<p>Processing images to generate text, such as image captioning and visual question-answering, has been studied for years. Traditionally such systems rely on an object detection network as a vision encoder to capture visual features and then produce text via a text decoder. Given a large amount of existing literature, in this post, I would like to only focus on one approach for solving vision language tasks, which is to <em>extend pre-trained <a href="https://lilianweng.github.io/posts/2019-01-31-lm/">generalized language models</a> to be capable of consuming visual signals</em>.</p>
Open original
Lil'Log · 2022-04-15
<p>Here comes the Part 3 on learning with not enough data (Previous: <a href="https://lilianweng.github.io/posts/2021-12-05-semi-supervised/">Part 1</a> and <a href="https://lilianweng.github.io/posts/2022-02-20-active-learning/">Part 2</a>). Let’s consider two approaches for generating synthetic data for training.</p>
<ul>
<li><strong>Augmented data</strong>. Given a set of existing training samples, we can apply a variety of augmentation, distortion and transformation to derive new data points without losing the key attributes. We have covered a bunch of augmentation methods on text and images in a <a href="https://lilianweng.github.io/posts/2021-05-31-contrastive/">previous post</a> on contrastive learning. For the sake of post completeness, I <em>duplicate</em> the section on data augmentation here with some edits.</li>
<li><strong>New data</strong>. Given few or even no data points, we can rely on powerful pretrained models to generate a number of <em>new</em> data points. This is especially true in recent years given the fast progress in large pretrained <a href="https://lilianweng.github.io/posts/2019-01-31-lm/">language models (LM)</a>. Few shot prompting is shown to be effective for LM to learn within context without extra training.</li>
</ul>
<h1 id="data-augmentation">Data Augmentation</h1>
<p>The goal of data augmentation is to modify the input format (e.g. text wording, visual appearance) while the semantic meaning stays unchanged.</p>
Open original
Lil'Log · 2022-02-20
<!-- The performance of supervised learning tasks improves with more high-quality labels available. However, it is expensive to collect a large number of labeled samples. Active learning is one paradigm to deal with not enough labeled data, when there are resources for labeling more data samples but under a limited budget. -->
<p>This is part 2 of what to do when facing a limited amount of labeled data for supervised learning tasks. This time we will get some amount of human labeling work involved, but within a budget limit, and therefore we need to be smart when selecting which samples to label.</p>
Open original
Lil'Log · 2021-12-05
<!-- The performance of supervised learning tasks improves with more high-quality labels available. However, it is expensive to collect a large number of labeled samples. There are several paradigms in machine learning to deal with the scenario when the labels are scarce. Semi-supervised learning is one candidate, utilizing a large amount of unlabeled data conjunction with a small amount of labeled data. -->
<p>When facing a limited amount of labeled data for supervised learning tasks, four approaches are commonly discussed.</p>
Open original
Lil'Log · 2021-09-24
<!-- How to train large and deep neural networks is challenging, as it demands a large amount of GPU memory and a long horizon of training time. This post reviews several popular training parallelism paradigms, as well as a variety of model architecture and memory saving designs to make it possible to train very large neural networks across a large number of GPUs. -->
<p><span class="update">[Updated on 2022-03-13: add <a href="#ec">expert choice routing</a>.]</span><br/>
<span class="update">[Updated on 2022-06-10]: <a href="https://gregbrockman.com/">Greg</a> and I wrote a shorted and upgraded version of this post, published on OpenAI Blog: <a href="https://openai.com/blog/techniques-for-training-large-neural-networks/">“Techniques for Training Large Neural Networks”</a></p>
Open original
Lil'Log · 2021-07-11
<!-- Diffusion models are a new type of generative models that are flexible enough to learn any arbitrarily complex data distribution while tractable to analytically evaluate the distribution. It has been shown recently that diffusion models can generate high-quality images and the performance is competitive to SOTA GAN. -->
<p><span class="update">[Updated on 2021-09-19: Highly recommend this blog post on <a href="https://yang-song.github.io/blog/2021/score/">score-based generative modeling</a> by Yang Song (author of several key papers in the references)].</span><br/>
<span class="update">[Updated on 2022-08-27: Added <a href="#classifier-free-guidance">classifier-free guidance</a>, <a href="#glide">GLIDE</a>, <a href="#unclip">unCLIP</a> and <a href="#imagen">Imagen</a>.</span><br/>
<span class="update">[Updated on 2022-08-31: Added <a href="#ldm">latent diffusion model</a>.</span><br/>
<span class="update">[Updated on 2024-04-13: Added <a href="#prog-distll">progressive distillation</a>, <a href="#consistency">consistency models</a>, and the <a href="#model-architecture">Model Architecture section</a>.</span></p>
Open original
Lil'Log · 2021-05-31
<!-- The main idea of contrastive learning is to learn representations such that similar samples stay close to each other, while dissimilar ones are far apart. Contrastive learning can be applied to both supervised and unsupervised data and has been shown to achieve good performance on a variety of vision and language tasks. -->
<p>The goal of contrastive representation learning is to learn such an embedding space in which similar sample pairs stay close to each other while dissimilar ones are far apart. Contrastive learning can be applied to both supervised and unsupervised settings. When working with unsupervised data, contrastive learning is one of the most powerful approaches in <a href="https://lilianweng.github.io/posts/2019-11-10-self-supervised/">self-supervised learning</a>.</p>
Open original