This post explains the competition from the ground up: what a task looks like, what a submission can and can’t do, and why scores start so low. Then it goes through what I learned in the first 24 hours: replicating the evaluation harness locally, measuring Gemma on it, and comparing it with a model of the same size. I finish with the fine-tuning pipeline I built from that. Every source is listed at the end.

The task

Each task is a real pull request from a Python repository, frozen at the commit just before the fix. The agent gets:

• The pull request description as its problem statement. It is often short and written for maintainers, not as a specification.
• A sandbox with the repository checked out in /workspace, its dependencies installed, and no network.
• Nine tools: run a shell command, read a file (150 lines at a time), edit a file by replacing a string, write a file, check the remaining budget, submit the patch, and three code-graph tools backed by precomputed call graphs and embeddings.

When the agent calls submit_patch, the harness takes git diff of the workspace, applies it to a fresh copy of the repository together with the maintainers’ own tests from the pull request, and runs pytest. The task counts only if every test passes. The score is the fraction of tasks solved.

The 129 public development tasks come from fastapi (67), rich (48), requests (13) and httpx (1). The roughly 120 scored tasks come from private repositories, split evenly between the public and private leaderboards. The organizers kept only tasks that a frontier model could solve or nearly solve.

What a submission can change

A submission is a zip of declarative Google ADK configuration: YAML for the agents, Markdown prompts, sampling settings, a per-task budget file, optional ADK skills, and optional LoRA adapters. No Python from the submission ever runs on the host. That leaves these levers:

• Prompts and structure. One agent, or several: an agent can call another agent as a tool, and agents can be chained in sequences or loops.
• Sampling and budgets. Temperature, output length, and the per-task time, tool-call and turn limits.
• LoRA adapters. Up to eight, rank 128 or less, attached to specific agents, with the whole submission under 3 GiB. This is where post-training comes in.

The model is fixed: gemma-4-31b-it-qat-w4a16-ct, a 4-bit build of Google’s quantization-aware-trained Gemma 4 31B, served by vLLM with a 32k-token context. All ~120 tasks must finish within 12 hours, and each team gets one submission a day.

Why scores start near 10%

Almost every first-day entry scored 0.00 to 0.12. Several things stack up:

1. The official sample configuration gives each task one minute and ten tool calls, which is not enough to finish anything. Many early entries kept it.
2. The tasks are pitched at frontier models, and come from repositories the model has never seen.
3. Grading is all or nothing. The hidden tests check exact function names, error messages and output formats described in the pull request. A fix that is almost right scores zero.
4. The time budget is tight. Twelve hours for about 120 tasks leaves around five minutes each on four L4s.
5. Scores move in big steps. With about 60 tasks on the public leaderboard, one task is worth about 1.7 points, so 0.10 and 0.12 differ by a single task.

My first submission, a single agent with a directive prompt and a five-minute, sixty-call budget, scored 0.10, tied for second. Scoring took about eight of the twelve hours, so there is little room to give each task more time.

Replicating the harness

The organizers published a detailed description of their harness but not the harness itself, so I rebuilt it: the nine tools with the same output limits, the same prompt construction, the same “please continue” nudges, patch extraction, and verification in a fresh copy of the repository. Two checks keep it honest:

• Oracle run. Applying each task’s real fix must pass its tests. 110 of 129 public tasks pass locally; the other 19 need older library versions than my environments have.
• Baseline run. Applying no fix must fail. Three tasks passed without any change and were dropped, leaving 107 development tasks.

The model runs wherever I can get it: vLLM on a Kaggle TPU v5e-8 with the unquantized QAT weights (free, 20 hours a week, but often a two-hour queue), or an OpenAI-compatible API such as OpenRouter for quick checks.

What Gemma 4 does as an agent

On the TPU, the single-agent baseline solved 9 of 29 and 11 of 30 development tasks in two runs. Variants I expected to help did not:

The traces were more informative than the scores. Gemma either solves a task quickly (6 to 38 tool calls) or gets stuck and burns the whole budget. Two patterns dominate the stuck runs:

• Failed edits. In one run 48 of 77 edit_file calls failed because the text to replace contained backslashes the file didn’t: Gemma copied \" where the source had ".
• Loops. One run issued the same failing shell command 39 times in a row.

A third problem is structural: several tasks per run crashed when the conversation grew past about 24,500 tokens. vLLM requires the prompt plus the space reserved for the reply to fit in 32,768 tokens, and my configuration reserved 8,192 for the reply (the official sample reserves 16,384, which caps the conversation even lower). ADK’s context compaction only starts at 32,768, so it never gets a chance to run first.

To see whether this is Gemma or my harness, I ran Qwen 3.8 27B, an open model of similar size, through the identical harness and prompt. On the tasks both ran cleanly, Qwen solved 5 of 13 and Gemma 2 of 13, and Qwen made almost no failed edits. The tools work; Gemma’s behavior with them is the bottleneck, and prompt changes alone didn’t fix it. That points to changing the weights.

The fine-tuning pipeline

The goal is supervised fine-tuning on successful agent runs, rendered exactly as the host renders them:

• Data. Gemma’s own runs that passed, and public agent trajectories from an open-weight teacher: nebius/SWE-rebench-openhands-trajectories, generated by Qwen3-Coder-480B under Apache 2.0. I converted them to my nine tools and kept resolved runs that don’t install packages, don’t leave scratch files in the patch, don’t depend on file views my read limit would cut, and fit in 28,000 tokens. I left out public trajectory sets generated by proprietary models, whose providers’ terms restrict training competing models on their outputs. Whether the competition allows distillation from other models at all is still an open question on the forum; the organizers have said they are discussing it.
• Rendering. Each conversation is rebuilt exactly the way the host’s ADK harness would send it, and passed through Gemma 4’s own chat template the way vLLM applies it. Only the tokens the model generates itself receive loss.
• Training. QLoRA on the unquantized QAT weights, rank 64, adapting attention and MLP projections in the language model only. Gemma 4 31B’s full-attention layers share one projection for keys and values; I leave that projection out on those layers, because a server may apply it differently from training.
• Checking. After training, the adapter is served on top of the exact 4-bit competition model in vLLM, as the host does, and must change the model’s output. Some Gemma 4 adapters have been reported to load in vLLM without taking effect.

The first training set has 843 conversations and 18 million tokens, about six million of them trained. I expect one epoch to take two to three and a half hours on a single rented H100.

What I’m doing next

1. Train the first adapter and compare it with the baseline on 45 development tasks, served the way the host serves it.
2. Test the context fix, a smaller output reservation so conversations can grow to about 28,600 tokens, against the baseline on the same tasks.
3. If the adapter helps, collect more of Gemma’s own successful runs on a larger pool of public tasks with runnable environments, retrain, and repeat. Later, train on pairs of passing and failing attempts at the same task.

I’ll write up what holds up.

Sources

Competition

• Google DeepMind, The Gemma 4 Developer Agent Competition on Kaggle: overview, rules, evaluation, data description and harness guide; and the paper track.
• Kaggle discussion: Clarification on distillation from external LLMs.
• Google, Gemma 4 on Kaggle Models (gemma-4-31b-it-qat-w4a16-ct and gemma-4-31b-it-qat-q4_0-unquantized).

Software

• Google, Agent Development Kit (ADK) and its agent config format.
• vLLM, its Gemma 4 recipe, and vllm-tpu for TPU serving.
• LiteLLM and OpenRouter.

Data

• Nebius, SWE-rebench and SWE-rebench-openhands-trajectories.
• Jimenez et al., SWE-bench.
• Pan et al., SWE-Gym; Yang et al., SWE-smith.

The final weights are on the Hub as osjayaprakash/llm-0.5b-fineweb-edu, under MIT. It loads as a plain LlamaForCausalLM, so nothing custom is needed to run it.

The setup

The model is a standard decoder-only transformer, written in plain PyTorch:

Parameters

489.3M (448.3M excluding embeddings)

Layers and width

24 layers, d_model 1280

Attention

Grouped-query, 20 query heads sharing 4 key/value heads

Feed-forward

SwiGLU, d_ff 3840

Normalization and positions

Pre-norm RMSNorm, RoPE

Vocabulary

32,000-token byte-level BPE, trained on the same corpus

Context

2048 tokens

Batch

524,288 tokens per step

The training data is 2.03B tokens of FineWeb-Edu, with 50M held out for validation. The GPU was a 32 GB RTX 5090 rented on Vast.ai at $0.479 an hour. It trained at 48,000 tokens a second, about 85% of the card’s theoretical peak.

Results

Each doubling of the tokens paid, and each paid a little less than the one before. The first took validation loss down 0.18 and average accuracy up 2 points; the second, 0.14 and 1.2 points. Neither curve has bent toward a floor. At 4.2B tokens the model has seen about 8.5 tokens per parameter, still under half the Chinchilla-optimal ratio. Tokens were the limit from start to finish. I stopped because the budget ran out, not because the model stopped learning.

Training was stable throughout: across all 8,000 steps there were no loss spikes and no non-finite gradients, and gradient clipping stopped triggering after step 540.

Against public models

I ran every model through the same lm-evaluation-harness tasks on the same machine. Before evaluating my checkpoints, I converted them to the Hugging Face Llama format and confirmed the converted model gives the same outputs as the original.

At a matched 2.1B tokens, my model scored 8.5 points above Pythia-410M. Finishing at 4.19B puts it 9.7 points above that checkpoint and 4.8 points behind the fully trained Pythia-410M, which saw 72 times as much data.

SmolLM2-360M beats Qwen2.5-0.5B with fewer parameters and less than a quarter of the tokens. That gap comes from how its training data was chosen and mixed. I haven’t worked on data selection at all yet, so it’s the most promising next step once tokens stop being the bottleneck.

On MMLU (5-shot) the model finished at 0.272, up from 0.254 at half the tokens. Chance is 0.25 on a four-way multiple choice, so this is the first sign of that kind of knowledge appearing, and not much more than a sign: social sciences carries it at 0.320 while humanities is still at chance. I leave MMLU out of the average.

Restarting the learning-rate schedule has a cost

Each phase used a cosine schedule: warm the learning rate up, then decay it to a tenth of its peak. To continue after phase 1, I warmed the learning rate back up to 2e-4 and decayed it again.

Raising the learning rate on a model that had just been annealed undid part of that annealing. Validation loss rose from 3.323 to 3.383 and took 600 steps to get back below 3.323, about a third of phase 2. The shaded areas in the chart are that cost.

Phase 3 repeated it and showed what governs the size of the bill. It rose from 3.139 to 3.193, then needed 1,100 steps to recover, nearly twice phase 2’s 600. The restart itself wasn’t worse; phase 3’s cosine simply runs over 4,000 steps instead of 2,000, so it holds the learning rate near its peak for twice as long. The cost tracks how long the schedule stays hot, not the act of restarting.

Both phases finished well ahead, so both restarts were worth doing. But together they spent 1,700 of the run’s 8,000 steps, 21% of the budget, re-earning ground already taken. A warmup-stable-decay schedule avoids this entirely: hold the learning rate constant and decay it only at the very end. It’s the better choice whenever the total training budget isn’t fixed in advance, which for me it usually isn’t.

What moved and what didn’t

LAMBADA, which asks the model to predict the last word of a passage, rose elevenfold from 0.021 to 0.230, making it the most useful single sign of progress. SciQ nearly doubled. HellaSwag sat nearly still for two phases and then finally began to move in phase 3, from 0.298 to 0.326.

ARC-Challenge ended at 0.265 and WinoGrande at 0.502, both at or near chance after the whole run, and Pythia’s checkpoints at the same token counts behave the same way. At this scale those two cost evaluation time without informing a single decision. I’d drop them from the tracking suite and watch validation perplexity and LAMBADA instead.

Averaging the weights didn’t help

Phase 3 also kept an exponential moving average of the weights, on the usual reasoning that averaging away the noise the last steps leave in each parameter is worth a few tenths of a point. It wasn’t. The averaged weights scored 0.443 against the final checkpoint’s 0.445, and were slightly worse on validation perplexity too, 42.5 against 42.1. That’s within noise, but it is certainly not a gain.

The reason makes sense in hindsight. Averaging helps when training ends while the learning rate is still high and the weights are still being jostled around. This run ends at 1e-5 after a full cosine decay, so the last thousand steps are already taking tiny, quiet steps. The decay had done the averaging already. The two techniques are substitutes, not complements.

So: if you can afford to decay the learning rate properly, do that and skip the averaging. Keep it for runs that end while the rate is still high, which is what happens with a constant-rate schedule, an early stop, or a budget that runs out mid-decay. Finding this out cost one gigabyte of disk and no measurable training time, which is a fair price for knowing.

Training health

I also logged per-layer gradient norms, activation sizes, and prediction entropy throughout. None of them needed action this time, but each would have caught a problem long before it showed up in the loss curve.

The first layer carries the largest gradients throughout, four to six times those of the early-middle layers, and the second half of the network runs higher than the first. All 24 layers shrink together, and each learning-rate restart shows up as a faint band at steps 2000 and 4000. No layer ever drifted away from the rest across the whole run. A layer whose gradients grew or collapsed on its own would be the first sign of an unstable run, and nothing here ever needed me to intervene.

The residual stream tells a different story: it’s banded by depth, not by time. Each layer adds its output to a running sum that only gets normalised on the way into the next block, so the stream grows steadily from layer 0 at the bottom to layer 23 at the top. That gradient is the signature of a healthy pre-norm network. What you don’t want is a single row suddenly brightening, which is what a layer heading for numerical trouble looks like well before the loss notices.

Prediction entropy drops steeply during warmup, from 7.44 nats to about 3.5 by the end of phase 1, and then stops falling. It spends phases 2 and 3 wandering around 3 nats, bouncing half a nat either way because I measure it on a single batch. That plateau is worth noticing: the model kept getting better at predicting the right token long after it stopped getting more confident on average. Loss and confidence are not the same thing, and only one of them was still improving.

The update sizes shrink as the learning rate decays and step back up at each restart, which is the loss curve’s story told from the optimizer’s side. None of this needed action. That’s rather the point of logging it: it’s cheap, and the run where one of these goes wrong is the run where you want it already there.

Cost

What I’d do next

The run is done, and it never hit a wall I could fix with engineering. Ranked by what would actually pay:

Change the data mix, don’t just add more of it. The cheap token gains are largely spent, and the corpus is the binding constraint now. The benchmarks that stayed flat are flat because FineWeb-Edu contains none of what they test: no code, little reference text, nothing requiring multi-step reasoning. SmolLM2-360M beating Qwen2.5-0.5B on a quarter of the tokens is the evidence that mixture outweighs volume from here.

If you do add tokens, use one warmup-stable-decay schedule. Another doubling would still pay, since the curve is straight to the end. Running it as a single schedule recovers the fifth of the budget my two restarts cost.

Skip the weight averaging if you can decay properly, per the measurement above.

Drop ARC-Challenge and WinoGrande from the tracking suite at this scale. Both sat at chance for the whole run.

What I wouldn’t bother with at this budget: architecture changes, a different optimizer, or a hyperparameter sweep. Nothing in the diagnostics ever suggested the model was the limiting factor. No instability, no dead layers, no gradient pathology, 85% of the card’s peak throughput from the first step to the last. It was the data all along.

Getting the model

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("osjayaprakash/llm-0.5b-fineweb-edu")
tok = AutoTokenizer.from_pretrained("osjayaprakash/llm-0.5b-fineweb-edu")

It’s a base model: no instruction tuning, no chat template, no alignment. It continues text and nothing else, and at this size it produces fluent, plausible, frequently wrong prose. Ask it about mitochondria and it will tell you they are the powerhouse of the cell, three times, before moving on. Treat it as a reference point for what a small budget buys rather than as something to build on. The model card carries the full numbers and limitations.

This post explains the competition from the ground up: what the environment looks like, how scoring works and why it’s so harsh, what the constraints of a Kaggle submission mean in practice, and how the strongest public agent, Tufa Labs’ Duck harness, plays. I finish with the weaknesses I found reading its code and what I plan to try. Every source is listed at the end.

The environment

An ARC-AGI-3 game is a small interactive program. At every step the agent receives a frame: a grid of up to 64×64 cells, each holding one of 16 colors, plus some metadata (the current state, which actions are allowed, how many levels are complete). The agent replies with an action, the game updates, and a new frame comes back.

There are at most seven actions, and their meaning is never explained:

• RESET starts or restarts the game.
• ACTION1 to ACTION5 and ACTION7 are simple actions. In many games they behave like up, down, left, right and “interact”, but the agent has to find that out.
• ACTION6 is a click: it takes an (x, y) coordinate on the grid.

Each game has several levels of increasing difficulty, and a game is in one of three states: NOT_FINISHED, WIN or GAME_OVER. Some games have a visible timer or move budget that ends the level when it runs out. Nothing tells the agent what the goal is. It has to work out the rules and the objective from how the frames change, then solve the level, then carry what it learned into the next one.

The competition ships 25 public games for development, and they give a sense of the range:

• They have between 6 and 10 levels each (median 7), 183 levels in total.
• 13 are tagged as using both keyboard-style actions and clicks, 7 as click-only and 4 as keyboard-only.
• Many actions return an animation rather than a single frame. When I stepped through every public game with random actions, 14 of the 25 regularly returned multi-frame responses, with up to 42 frames for one action, and in most of those the intermediate frames showed things the final frame didn’t: a sliding block, a falling piece, a flash.

The scored games are different. Evaluation uses 110 private games the agent has never seen: 55 decide the public leaderboard and the other 55 the private one that sets the final ranking. Nothing learned about a specific public game helps directly, which is the point.

The arc-agi toolkit runs games locally and offline, and the ARC-AGI-3-Agents repository provides a template agent with two methods to implement: choose_action, which picks the next action from the latest frame, and is_done, which decides when to stop.

How scoring works

Scoring rewards two things: completing levels, and doing it with about as few actions as a person would. The scoring methodology works in three steps.

1. Per level. Each completed level is compared with a human baseline, the number of actions first-time human players needed. The level scores (human actions / agent actions)², so matching the human count earns 1. Beating it earns little extra: the methodology caps a level at 1.15, and Kaggle caps overall scores at 100%. An uncompleted level scores 0.
2. Per game. Level scores are averaged with weights equal to the 1-indexed level number, so level 1 has weight 1, level 2 weight 2, and so on. The denominator is the sum of the weights of all levels in the game, finished or not.
3. Overall. The total is the average across games, reported as a percentage.

Two features of this formula explain most of what follows.

The square punishes exploration. An agent that needs three times as many actions as a person keeps only a ninth of that level’s score. An agent that needs ten times as many keeps one percent. The human baselines are small: across the public games, level 1 needs a median of 30 actions (as few as 7 in one game, at most 78), and a whole game needs a median of 638. An agent that probes each action a few times to see what it does has already spent a good part of the budget for the first level.

The weights reward depth. In a 7-level game the weights sum to 28, so level 1 is worth 1/28 of the game, under 4%, and the last level is worth a quarter. Solving the first three levels at exactly human efficiency scores (1 + 2 + 3) / 28, about 21% of that game. Solving only level 1 at three times the human action count scores (1/9) / 28, about 0.4%.

So an agent that reliably solves a level or two of most games, slowly, lands where the leaderboard is now: around one percent. Moving the score means reaching deeper levels, and doing it efficiently.

What a Kaggle submission can use

The competition is notebook-only, and the constraints shape every design decision:

• Hardware. One NVIDIA RTX Pro 6000 with 96 GB of memory. The competition started on H100s and moved to the RTX Pro 6000 in May; the machines are reserved for this competition’s notebooks.
• Time. At most 9 hours for all 110 games, raised from 6 hours with the hardware change. That’s under five minutes of wall-clock per game if they ran one after another, so agents run many games in parallel.
• No internet. Models, wheels and code have to be attached as Kaggle datasets or models, and any model has to run locally. Frontier APIs are out.
• Open weights. Prize winners must release their code and use open models, so the practical choice is an open-weight model small enough to serve on one GPU.
• One submission a day. Each submission reruns the notebook against the private games, and the score comes back hours later.

There are also two milestone prizes, on June 30 and September 30, for the leaderboard leaders on those dates who have published their notebooks.

The combination is unusual. Unlike most Kaggle competitions there’s no training set to fit and no labels, and unlike most agent benchmarks the agent can’t call a frontier model. It has to get as much reasoning as possible out of a mid-sized open model in a fixed amount of GPU time.

What has worked so far

Four earlier systems set the direction.

Stochastic Goose. Dries Smit’s agent won the ARC-AGI-3 agent preview competition, which ran on a handful of preview games before the official release.

ARCgentica. Symbolica’s agent scored 36% the day after the official public games were released, using frontier models. It lets the model call itself recursively, in the style of Recursive Language Models, so sub-agents can each work on part of a game with a limited number of actions. Its main lesson is that these models are good at solving problems by writing code, as in CodeAct.

RGB Agent. The RGB Agent wraps a general-purpose coding harness, OpenCode, around the model instead of building an ARC-specific one. It keeps the full game history in a log file the model can search and read, which lets it analyze before acting and queue only a few actions at a time. It was the first to show near-human action efficiency on the three preview games.

The Duck. Tufa Labs’ Duck harness, the successor to Stochastic Goose, won the first milestone with a leaderboard score of 1.21% using Qwen 3.6 27B in FP8. Tufa published the notebook, the code and a blog post, and most public notebooks since then are forks of it. It’s worth understanding in detail.

How the Duck plays

The Duck gives the model one tool: a Python interpreter preloaded with the game’s state. There’s no fixed perception pipeline and no hand-written planner. The model writes code to inspect the board and to act, and the harness runs it.

The game as Python variables. Each tool call starts a fresh interpreter with these variables already defined:

• current_frame and previous_frame, which expose .ascii (the grid as letters, one per color), .segmentation (the board split into connected single-color objects), the step and the level.
• history and transitions, the list of past actions and the frames before and after them.
• valid_actions, the actions allowed right now.
• action(...), which executes one action or a whole list of them in the real game.

Actions get readable names (UP, DOWN, LEFT, RIGHT, SPACE, MOUSE), and because action can be called in a loop, the model can write a search, such as a breadth-first search to a target, and execute its result in one call. Each call has a 30-second limit and output is capped at 4,096 characters.

A world model in notes. The prompt asks the model to keep a running note with labeled sections: “World model:”, “Goal model:”, “Action model:”, “Plan:” and so on. The harness pulls those sections out of each reply and repeats them in the next prompt, so the model’s understanding survives even when older messages are dropped.

A bounded context. The full system prompt is always included. Older turns are evicted to keep the input around 32K tokens, which lets a game run for hours at a steady speed.

One image per turn. At the start of each turn the model also sees the current frame as an image, upscaled 4× to 256×256 pixels. Tufa found this works best with Qwen’s 16×16-pixel image patches. They tried showing several frames or whole animations and found the model couldn’t make use of them.

Prompting did a lot of the work. Tufa’s write-up is candid that the prompt carries a lot of hard-won guidance. It tells the model not to treat an energy bar as the objective and not to invent goals like moving a block to a fixed position. It keeps the model from hallucinating robots or treating the game as an Atari title, and from printing whole boards into its own context. Custom tools didn’t help: in Tufa’s words, hand-crafting tools “seems to hinder the creative abilities of the model.” The largest gains came from better base models and from adding the image.

On the 25 public games, over 20 runs each, the Duck averaged 1.60% ± 0.45. That spread is the other thing to know about this competition: the same notebook can score very differently from one run to the next, and Tufa reported their own best submission scoring as low as 0.77% on a rerun. With one submission a day, a single leaderboard score says little about whether a change helped.

Where I see room

I read through the public Duck notebooks and harness code to find what I’d change first. Three things stood out, and none of them needs a better model.

Knowledge is thrown away too often. The harness clears the world, goal and action notes whenever a level is completed, and also whenever the game ends in GAME_OVER. In the public notebooks a game over restarts the current level rather than the whole game, so the agent is back on the same level it was just learning, having forgotten what it found. Clearing notes at a level change is more defensible, since layouts change, but what each action does usually carries over. Because later levels carry most of the weight, anything that speeds up level 3 onward is worth more than it looks.

Time is split evenly regardless of progress. The public notebooks give every game the same fixed wall-clock budget. In the configurations I looked at, 28 games run at once with a little over two hours each, which in four waves just fits 110 games into 9 hours. A game stuck on level 1 after an hour holds its slot as long as a game that’s clearing a level every twenty minutes. Moving time from stalled games to progressing ones, or starting queued games sooner, is a scheduling change that doesn’t touch the agent at all.

The model never sees animations. The harness keeps only the last frame of each action. In more than half the public games, the frames in between show how the action worked: which way something moved, whether a projectile hit anything, where a piece landed before snapping into place. Tufa found that passing whole animations as images didn’t help a model of this size. A compact text summary of what changed during the animation, added to the action’s result, might.

There are also two smaller points. The prompt never tells the model how many actions it has spent on the current level, even though efficiency is squared in the score. And every run is sampled without a fixed seed, which adds to the run-to-run noise that makes changes hard to evaluate.

What I’m doing next

My plan is to test those changes one at a time against an unmodified baseline, on the public games, before spending any daily submissions on them:

1. Keep the world, goal and action notes through a game over, and carry the action and goal notes into each new level, marked as needing a quick recheck.
2. Stop a game after it has gone about an hour without completing a level, and let games that are still progressing use the time freed up.
3. Add a short summary of each action’s animation to its result.

Because a single run is so noisy, I’ll compare games one by one between paired runs rather than looking only at the averages, and repeat runs where the result is close. I’ll write up what holds up.

Sources

Competition and environment

• ARC Prize, ARC Prize 2026 - ARC-AGI-3 on Kaggle: overview, rules, evaluation and data description.
• ARC Prize, ARC-AGI-3 and documentation, including the scoring methodology.
• ARC Prize, arc-agi toolkit and ARC-AGI-3-Agents on GitHub.
• Kaggle discussion: Upgraded accelerators and Update on code requirements.

Agents

• Tufa Labs (Bessis, Cottaar, Pressman, Smit, Tešnar, Viel), Tufa Labs’ winning solution for ARC-AGI-3 Milestone 1, the duck-harness repository and the blog post.
• Dries Smit, 1st place in the ARC-AGI-3 agent preview competition.
• Symbolica, ARC-AGI-3.
• RGB Agent (blog.alexisfox.dev).

Background

• Wang et al., Executable Code Actions Elicit Better LLM Agents (CodeAct).
• Zhang, Kraska and Khattab, Recursive Language Models.

MCP (Model Context Protocol)

MCP is a protocol for connecting a model to tools, databases, and data sources it never saw in training.

Without it, Claude works from training data with a cutoff date plus whatever you paste into the conversation. With it, Claude reaches:

• Live data such as weather and stock prices
• Company systems such as a customer or product database
• Tools such as code interpreters, calculators, and mail systems
• Files on your own machine

How a request flows

1. You run an MCP server that fronts a tool or data source
2. Claude sends it a request: fetch this, run that
3. The server does the work and returns the result
4. Claude uses the result in its answer

MCP 2.0

MCP 2.0 revises the original design rather than replacing it. The headline change is a stateless architecture: each request carries everything the server needs, so the server keeps no session for you between calls.

Compare it to asking a librarian for a book. Under MCP 1.0 the librarian tracked your whole visit history to answer. Under 2.0 they answer the question in front of them.

Dropping per-client state buys three things:

• Speed. No session to look up or maintain.
• Reliability. One failed request cannot corrupt the next.
• Simpler servers. Less bookkeeping to write and get wrong.

Build new servers on 2.0. Existing 1.0 servers keep working, so migrate them when you have a reason.

Skills

A Skill is a set of written instructions for a task you repeat. It lives in a file and gives Claude the steps you want, in the order you want them.

A Skill holds:

1. Instructions in markdown
2. A description that says when to use it
3. The steps to follow

A skill called write-tech-news might say: search for this week’s tech stories, confirm each one in a second source, draft the post, commit it, push to GitHub. Ask for the Skill and Claude runs those steps.

Skill against MCP

MCP gives Claude reach: a database it can query, an API it can call. A Skill gives Claude a procedure: the steps you want followed. One is a new capability, the other is a job description.

Sub-agents

A Sub-agent is a separate Claude instance scoped to one kind of work. You start it with its own instructions, it works through the task on its own context, and it reports back.

Common ones:

• A code reviewer that reads diffs and flags bugs
• A researcher that searches and summarizes
• A deploy agent that ships to servers
• An explorer that maps an unfamiliar codebase

Sub-agent against Skill

A Skill runs inside your conversation and spends your context. A Sub-agent runs beside it with a context of its own and hands back a result. A Skill says “here are the steps, follow them.” A Sub-agent says “go handle this and report when it’s done.”

All four side by side

Picking one

Reach for MCP when Claude needs something it cannot see: a live feed, a database, a file on disk.

Reach for a Skill when you find yourself typing the same five instructions every week.

Reach for a Sub-agent when a task would flood your main conversation, or when two pieces of work can run at once.

Using them together

A weekly tech-news post uses all three. MCP 2.0 pulls the articles. The Skill holds the steps for turning them into a post. A Sub-agent runs the whole job while your main session does something else.

Andrej Karpathy made this point in A Recipe for Training Neural Networks, and it runs through every guide on the subject. This post collects what those guides, the published logbooks of large training runs, and a handful of papers say about finding these bugs in language models specifically. Where I can, I show what each check looked like in my own 0.5B pretraining run. Everything I cite is listed at the end.

Language models raise the stakes in two ways. Each attempt is expensive: one phase of my run took more than six hours, and the runs behind PaLM and OPT took weeks on hundreds to thousands of accelerators. And some failures only appear at scale or late in training, long after the cheap checks have passed. So almost everything below is about finding problems before the expensive part starts, or in its first few minutes, and then noticing the rest as early as possible.

Check the data and the tokenizer

Read the data after it’s been processed. Karpathy’s first step is to spend real time with the raw data, and to look at it again at the last point before it enters the model. For an LLM that means decoding a batch of token IDs back into text. Tokenizer bugs, broken document boundaries and garbled encodings are obvious when you read them and invisible in a loss curve.

Deduplicate. Web text repeats itself far more than you’d guess. Lee et al. found a single 61-word sentence repeated more than 60,000 times in C4. Removing duplicates cut the rate at which models reproduced training text verbatim by a factor of ten, and reached the same accuracy in fewer steps. Duplicates also cause instability: the OLMo 2 team traced many of their loss spikes to documents containing the same short phrase repeated dozens of times, and filtering them out nearly eliminated the spikes.

Check for overlap with your benchmarks. Lee et al. also found that over 4% of some standard validation sets appeared in the training data. The GPT-3 paper flagged a benchmark example as contaminated if it shared any 13-token sequence with the training set, then compared scores on the clean subset. It also admits that a bug in their filtering left some overlap in place, and that retraining to fix it was too expensive. Run the check before training, while fixing it is still cheap.

Look for tokens the model never learns. A tokenizer trained on different data from the model can contain tokens that almost never appear in training. Their embeddings stay close to their random starting point, and the model behaves erratically when it meets them. The best-known case is SolidGoldMagikarp, one of about 140 GPT-2 and GPT-3 tokens, many of them Reddit usernames, that made the models evade, insult or hallucinate. Land and Bartolo show how to find such tokens automatically from the tokenizer and the model’s output weights, and found them across many open models. I trained my tokenizer on a million documents from the same corpus as the model, which keeps the mismatch small, but a quick scan for rarely used tokens is still worth doing.

Know where documents start and end. Most pretraining pipelines pack documents end to end and cut fixed-length windows out of the stream, so a window can span the end of one document and the start of the next. Mine marks each document with start and end tokens, and attention can still reach across that boundary. That’s a common and reasonable choice, but it’s a choice. Make it deliberately, and make sure evaluation formats text the same way (more on that below).

Check the model and the loss

Know what the loss should be at step 0. A freshly initialized model should be close to guessing uniformly, so its loss should be about the natural log of the vocabulary size. Hugging Face’s LLM course makes the same point for classifiers: work out the loss of random guessing and compare. With my 32,000-token vocabulary that’s ln(32,000) ≈ 10.37. A much higher starting loss usually means the output layer’s initialization is off; a much lower one can mean the model is seeing the answer. I only started logging at step 10, where the loss was already 8.82, so I can’t show my own step-0 value. Log it.

Test that the model can’t see the future. An autoregressive model must predict each token from earlier tokens only. If the attention mask leaks, training loss looks wonderful and the model is useless at generation. Karpathy suggests checking this with gradients: the gradient of an output should only reach inputs it is allowed to depend on. My test does the same check more simply: change the last token and confirm no earlier prediction moves.

def test_causal_masking():
    """Changing a future token must not affect earlier logits."""
    model = Transformer(tiny_cfg()).eval()
    x = torch.randint(0, 100, (1, 8))
    y = x.clone()
    y[0, -1] = (x[0, -1] + 1) % 100
    with torch.no_grad():
        a, _ = model(x, x)
        b, _ = model(y, y)
    assert torch.allclose(a[:, :-1], b[:, :-1], atol=1e-5)

The related bug is an off-by-one in the targets. Each target should be the input shifted left by one token. If inputs and targets line up exactly, the model only has to copy its input, and the loss falls far below anything plausible almost immediately.

Overfit a single batch. Every guide I read agrees on this one. A model that can’t drive its loss to near zero on one repeated batch has a bug in the model, the loss, or the optimizer step. Pierce Freeman’s debugging tips suggest building up from one example to two, then five, so you see the capacity you expect before you scale up. The LLM course adds a practical warning: throw the model away afterwards and build a fresh one, or your real run starts from weights that have memorized one batch.

Scale the initialization with depth. Each transformer block adds its output to a running sum, so with standard initialization the sum grows with the number of layers. GPT-2 and Karpathy’s build-nanogpt shrink the initial weights of each block’s output projection by 1/√(2 × layers) to compensate; I do the same, giving 0.02/√48 for my 24 layers. The BLOOM team found the usual 0.02 standard deviation too large at 100B+ parameters and used a much smaller value, as described in Stas Bekman’s ML Engineering book. Takase et al. show the other side: if the embeddings start too small relative to the rest, normalization layers amplify their gradients and training spikes. Scaling the embeddings up, or normalizing them, fixed it.

Be deliberate about weight decay. Weight decay on normalization gains and biases pulls them toward zero for no benefit. My optimizer decays only weight matrices. OLMo 2 goes further and also leaves the embeddings undecayed. Either way, print the parameter groups once and check each parameter landed where you meant.

Write it twice and compare. Freeman’s tip is to write a slow, obvious version of any tricky tensor code next to the fast one and check they agree. I used the same idea when converting checkpoints to the Hugging Face Llama format for evaluation: the export script runs both models on the same random tokens and fails if their outputs differ. The largest difference was 0.0039, which is bf16 rounding noise. Without that check, a transposed weight would have quietly lowered every benchmark score.

A check like mine tells you that two implementations differ, not where. For that, Transformers has model_addition_debugger_context, written for people porting models into the library. Wrap one forward pass in it and it records the shape, dtype, mean, standard deviation, minimum and maximum of every module’s inputs and outputs in a JSON file, optionally with the full tensors saved alongside. Trace both implementations on the same input and compare the files: the first module whose numbers disagree is where to look. By default it keeps only the first and last layers; pass do_prune_layers=False to record all of them.

Check the training loop

Normalize the loss correctly across gradient accumulation. When a batch is too big for memory, you split it into micro-batches and add up their gradients. For this to match the full batch, each micro-batch’s loss has to be weighted by its share of the batch’s tokens. In October 2024, Hugging Face and Unsloth described a bug in the widely used Transformers trainer: it averaged each micro-batch’s mean loss, which gives short, padded sequences too much weight. Training with accumulation produced a noticeably higher loss than the same batch without it. My loop simply divides each micro-batch’s loss by the number of micro-batches, 64, as nanoGPT does. That’s correct here only because every window has exactly 2,048 tokens and nothing is padded. For fine-tuning on variable-length examples, the same line would be the bug.

Give each GPU its own random stream. With several GPUs, each one must draw different data. I seed each rank’s data sampler with the base seed plus its rank. Hugging Face’s Smol Training Playbook describes the reverse mistake in the SmolLM3 run: every tensor-parallel rank shared one seed, so their slices of the weights started out correlated. The run fell behind a comparison run and had to be restarted after a trillion tokens.

Make runs and evaluations repeatable. Fix the random seed so two runs of the same config match. Evaluate on the same data every time: my validation loss resets its sampler before each evaluation, so a change in validation loss reflects the model, not a different set of windows.

Use bf16 rather than fp16. fp16 can’t represent large values, and a single overflow turns the loss into NaN. fp16 training needs a loss scaler, which my loop enables only when fp16 is selected. The BLOOM team credits much of their final run’s stability to switching to bf16 after their earlier fp16 attempts diverged, and the MIT course checklist gives the same advice for anyone seeing NaNs.

Log gradient norms before clipping. Clipping caps the gradient norm, so a norm logged after clipping can never show you a problem above the cap. Google’s Deep Learning Tuning Playbook suggests logging the unclipped norm and setting the clip threshold around the 90th percentile of what you observe. If most steps are clipped, the clip is doing the job the learning rate should be doing.

Run every code path once before the long run. A crash in the evaluation code or in checkpoint saving usually shows up only when that code first runs, which may be hours in. The LLM course recommends running evaluation on its own before training starts, and more generally walking through the pipeline in the order the trainer does: one item from the dataset, decoded; one batch from the loader; a forward pass; the loss; a backward pass; one optimizer step. Only then start the full run. The same logic applies to anything that happens on a schedule. Evaluate, sample, save a checkpoint and load it back in the first few minutes, not for the first time at step 2,000.

Reproduce confusing GPU errors on the CPU. CUDA runs asynchronously, so an error is often reported by a later operation than the one that caused it, with an unhelpful message. The LLM course’s advice is to move the model and one batch to the CPU and run the same step there, where the error points at the real line. My unit tests all run a tiny model on the CPU for the same reason: they’re fast, and when they fail they fail clearly.

Log more than the loss

Loss is one number summarizing millions of parameters. By the time it looks wrong, the cause usually started much earlier. Freeman recommends logging per-layer gradient and weight statistics so you can see a problem at its source. My training loop records these every 50 steps:

In my run these stayed quiet. There were no non-finite gradients in 4,000 steps, and clipping triggered on 21 of the 412 logged steps: 14 during the 200-step warmup and none after step 540. The update size followed the schedule: its median fell from 0.18 around step 550 to 0.03 by step 2050, then more than doubled when the learning rate restarted for phase 2. The largest output logit stayed between 19 and 23 for the whole run. My post on the 0.5B run has a heatmap of the per-layer gradient norms.

None of these needed action this time. That’s the point: they cost almost nothing to log, and if something had gone wrong they would have shown where.

Throughput is worth watching too. It isn’t a model metric, but a drop usually means something else is wrong. My run held 85% of the GPU’s peak throughput throughout. In the Smol Training Playbook, drops in throughput lined up with spikes in disk read latency.

Track down NaNs and overflows

A non-finite value tells you that something broke, but not where. The Transformers debugging guide describes a tool for that, DebugUnderflowOverflow. With the Trainer you turn it on with debug="underflow_overflow"; in your own loop you wrap the model in it. It hooks every module’s forward pass, checks inputs, outputs and weights, and when it finds an inf or NaN it stops and prints the last several modules it ran, each with the smallest and largest absolute values it saw. Reading back up that list shows where values started to grow.

The guide’s worked example is a T5 model training in fp16. The largest value coming out of one feed-forward block was about 62,700. fp16 can’t represent anything above 65,504, and the dropout that followed scales its surviving values up, which pushed them past the limit. Its fixes are to run that one block in fp32 by switching autocast off inside it, or to turn mixed precision off entirely. When the problem is inside a module rather than between modules, a companion function, detect_overflow, can be dropped in after any intermediate step.

Two options help when the failure comes later in training. trace_batch_nums records those same minimum and maximum values for chosen batches even when nothing has overflowed yet, so you can compare a batch just before the failure with one from much earlier and see which layer drifts first. abort_after_batch_num stops the run once you have what you need.

The same guide explains a NaN that catches many people out: a model pretrained in bf16, as many TPU-trained models were, often overflows when run or fine-tuned in fp16, because its activations were never kept inside fp16’s range. Use bf16 or fp32 for it instead. bf16 has the same range as fp32, which is why overflows like the T5 one largely disappear with it. What bf16 gives up is precision.

I never needed this tool, because my non-finite count stayed at zero, but it’s the next step if that count ever goes above zero.

Understand loss spikes

A loss spike is a sudden jump in training loss, sometimes recovering by itself and sometimes sending the run off the rails. They’re the most documented failure in large-scale pretraining, and the published accounts agree on more than you might expect.

They aren’t just bad data. Training the 540B-parameter PaLM model hit about 20 spikes despite gradient clipping. Each time, the team restarted from a checkpoint about 100 steps before the spike and skipped the next 200 to 500 batches. When they replayed the same batches from a different checkpoint, no spike appeared. The cause was the combination of that data with that state of the model.

Gradient norms warn first. In OLMo 2, gradient-norm spikes came before loss spikes and became more frequent as models grew. That’s another reason to log the unclipped norm.

Many of them are reproducible at small scale. Wortsman et al. showed that small models trained at high learning rates show the same two failures as large ones. In the first, attention logits grow until attention collapses onto a single token; normalizing queries and keys (qk-layernorm) fixes it. In the second, output logits drift upward; a small extra loss term that penalizes their overall size, called z-loss, fixes it. PaLM and Wortsman et al. weight z-loss at 1e-4; OLMo 2 uses 1e-5. The same paper suggests longer warmup and a smaller AdamW epsilon, whose default can become as large as the gradients themselves at scale. The practical upshot: sweep the learning rate on a small model, and see where it breaks, before committing to the big one.

They can come from the data loader. The SmolLM3 run in the Smol Training Playbook had spikes because its loader read sequences in order within each document, so one very long, unusual file could fill a whole batch. Shuffling at the sequence level fixed it.

The fixes that recur. Meta’s OPT-175B logbook records weeks of restarts. Along the way, the team lowered gradient clipping from 2.5 to 1.0, raised weight decay to 0.1, lowered Adam’s β2 to 0.95, and cut the learning rate when gradient and activation norms began to drift. Those are the same values my run started with: clipping at 1.0, weight decay 0.1, β2 0.95, and bf16. They’re now common defaults for GPT-style pretraining. The Tuning Playbook adds two cheap checks: sweep warmup length over several orders of magnitude, and run a few hundred steps at twice your chosen learning rate to see how close to the edge you are.

My spike detector flags any logged loss more than four standard deviations above its moving average. It didn’t fire in 4,000 steps, which at my scale and learning rate is what the literature above would predict.

Read what the model writes

Freeman suggests saving the model’s predictions at regular intervals, and Karpathy suggests watching how they change. For a language model the easiest version is to generate text from a few fixed prompts every few hundred steps, always with the same settings, and read it. Here’s what mine produced with greedy decoding:

The first three rows show the usual progression: pure repetition, then fluent sentences that loop, then sentences that stay on topic. The last two are more useful. The model still can’t state a simple fact, which matches its chance-level MMLU score. And it has never produced code for the code prompt. That’s not a bug in training; my corpus is educational web text with almost no code. The samples show that more directly than any benchmark score, and code is first on my list of data to add.

Evaluate the way you trained

Format evaluation text like training text. My training documents all start with a start-of-text token, so my evaluations add it too. Leave it out and every benchmark is measured on inputs the model never saw in training. Check the same for end-of-text tokens, chat templates after fine-tuning, and whitespace handling in the tokenizer.

Verify every conversion. Benchmark harnesses usually load models in a standard format, so you’re often evaluating a converted copy rather than the model you trained. The export check described above is what makes my scores trustworthy.

Pick benchmarks that move at your scale. Small models sit at chance on many popular benchmarks. HellaSwag and ARC-Challenge barely moved in my run, while LAMBADA rose from 0.02 to 0.21, so that’s the one to watch between checkpoints. A benchmark stuck at chance only adds noise to an average.

Choose baselines carefully. Comparing against a public model at the same number of training tokens is a useful control, but check where that checkpoint was in its own training. Pythia’s 1-billion-token checkpoint is still inside its learning-rate warmup, which makes it a weak comparison; its 2-billion-token checkpoint is a fairer one.

Long runs: resuming, repeating data, and hardware

Resume the whole state, not just the weights. A resumed run needs the optimizer state, the step count for the learning-rate schedule, and the data position. Restarting AdamW without its state resets its running averages, and the first steps after the resume take much larger updates than they should. Restarting the data sampler with its original seed replays the exact windows the run began with. I reseed the sampler from the step number on resume to avoid that.

Restarting a learning-rate schedule has a cost. Karpathy warns against trusting a schedule’s defaults. I learned a version of that when I extended my run: warming a cosine schedule back up to continue training undid part of the previous annealing and cost more than 500 steps of progress. A warmup-stable-decay schedule, which holds the learning rate constant and decays it only at the end, avoids this.

An epoch isn’t always a pass over the data. My loader samples random windows rather than walking through the corpus in order. After 1.03 “epochs” of tokens, the arithmetic of random sampling says about 36% of the corpus has never been seen, while about 28% has been seen twice or more. That’s fine, but it means a train/validation gap can open before you reach the epoch count you expect. How much repetition hurts is well studied: Muennighoff et al. found that up to about four epochs of repeated data are nearly as good as fresh data, with returns falling off quickly after that.

Plan for hardware failure. Most of OPT’s early restarts came from hardware problems, not modeling ones. At my scale the equivalent is simpler but just as real. My rented machine’s disk disappears when the instance is destroyed, so I copied the phase 1 weights to another machine as soon as it finished. I ran training inside tmux so a dropped SSH connection wouldn’t kill it. And I checked that there was enough free disk space before starting a phase, since running out mid-save can corrupt the checkpoint you most need.

Debug the distributed setup

My run used a single GPU, so this section is what I’d check first rather than what I saw. Distributed training adds failure modes that have nothing to do with the model.

Test communication on its own. Before debugging a multi-GPU training script, check that the GPUs can talk to each other at all. The Transformers debugging guide links a small script, torch-distributed-gpu-test.py, that you launch the same way as your training job; it prints OK for each process that can communicate and allocate memory. Setting NCCL_DEBUG=INFO makes the communication library log what it’s doing, which is usually enough to spot a wrong network interface or a failed connection.

Catch shape mismatches before they hang. Collective operations like gather and reduce expect every process to contribute a tensor of the same shape. If one process sends something different, the job doesn’t fail; it waits, and the only symptom is a timeout, often many minutes later. Accelerate’s debug mode, turned on with accelerate launch --debug or ACCELERATE_DEBUG_MODE=1, checks shapes across processes before each such operation and raises an error that lists every process’s shape instead. It costs very little, so it’s worth leaving on while a new setup is being brought up.

Check that every rank does the same number of steps. A related hang in hand-written loops comes from one process skipping or adding a step, for example because its share of the data ran out first. The processes then wait at different synchronization points. Sampling random windows, as my loader does, sidesteps this because no process can run out.

Rule out the framework. For DeepSpeed, the Transformers guide’s first advice is to rerun without it: if the error goes away, it’s in the DeepSpeed setup, not the model. Two of its specific notes are worth remembering. A process that dies at startup with no traceback has usually been killed by the operating system for using too much CPU memory, often because optimizer or parameter offloading to the CPU is switched on. And repeated OVERFLOW! messages in fp16 mean the loss scaler can’t find a workable scale; the guide suggests a higher starting scale.

Match the symptom to the cause

The MIT checklist and Josh Tobin’s Troubleshooting Deep Neural Networks both map what you see in the curves to the likely cause. Adapted for pretraining, with what I added from the sources above:

One entry needs a pretraining caveat. In the first pass over the corpus, training and validation loss should sit close together because the model hasn’t seen either. A gap only means something once the data starts repeating, which with random sampling happens sooner than the epoch count suggests.

Not every jump is a bug, either. My validation loss rose when phase 2 started, because I deliberately raised the learning rate again. Knowing what each change should do to the curves is what lets you tell an expected bump from a real problem.

Change one thing at a time

Karpathy’s advice for the model itself is to start from something known to work and resist being creative early. Copy a standard architecture, use Adam at a sensible learning rate, and add complexity one piece at a time, checking that each piece helps. I did exactly that: a standard Llama-style decoder and AdamW with a peak learning rate of 3e-4. It’s also why my results can be compared with Pythia’s at all. If I had changed the architecture and the data and the schedule at once, I couldn’t say which change mattered.

Small runs are how you afford that discipline. Wortsman et al. show that many large-scale failures can be reproduced in small models, and the Tuning Playbook is built around short, cheap experiments that each answer one question. Test a change small, then scale it.

The short version

Data and tokenizer:

• A decoded batch reads as the text you expected.
• Exact and near duplicates are removed, including long runs of repeated text.
• No benchmark example shares a 13-token sequence with the training data.
• No tokens in the vocabulary are so rare the model never learns them.

Model and loss:

• Loss at step 0 is close to ln(vocabulary size).
• Changing a future token doesn’t change earlier predictions, and targets are shifted by one.
• The model drives its loss to near zero on one batch.
• Output projections are scaled down with depth, and the parameter groups for weight decay are what you intended.
• Anything written twice, such as a checkpoint conversion, gives matching outputs; if not, trace both to find the first layer that differs.
• After the overfitting test, the real run starts from a fresh model.

Training loop:

• Gradient accumulation weights each micro-batch by its share of tokens.
• Each GPU has its own random stream, and runs are repeatable from a seed.
• Training uses bf16, and gradient norms are logged before clipping.
• Evaluation, sampling, and saving and reloading a checkpoint have all run once in the first few minutes.
• For multi-GPU runs, the communication test passes and shape checks are on while the setup is new.

During the run:

• Log per-layer gradient norms, update sizes, output-logit size and non-finite counts, not just the loss.
• Generate from fixed prompts on a schedule and read the output.
• Know in advance how you’ll respond to a loss spike, and which tool you’ll reach for if a NaN appears.
• Before reacting to a change in the curves, ask whether you caused it.

Evaluation and long runs:

• Evaluation text is formatted like training text.
• Checkpoints save the optimizer, schedule and data position, and live somewhere that survives the machine.

Sources

Guides:

• Andrej Karpathy, A Recipe for Training Neural Networks, 2019.
• Josh Tobin, Troubleshooting Deep Neural Networks, Full Stack Deep Learning, 2019.
• Pierce Freeman, Debugging Tips for Neural Network Training, 2022.
• Varun Godbole, George E. Dahl, Justin Gilmer, Christopher J. Shallue and Zachary Nado, Deep Learning Tuning Playbook, Google Research, 2023.
• Stas Bekman, Machine Learning Engineering Open Book, 2023 onward.
• MIT MAS.S60, How to AI (Almost) Anything, debugging checklist slides, spring 2025.
• Loubna Ben Allal, Lewis Tunstall, Nouamane Tazi et al., The Smol Training Playbook, Hugging Face, 2025.

Training logs and reports:

• Susan Zhang, Stephen Roller, Naman Goyal et al., OPT-175B chronicles and logbook, Meta AI, 2022.
• BigScience, BLOOM-176B training chronicles, 2022.
• Aakanksha Chowdhery et al., PaLM: Scaling Language Modeling with Pathways, 2022.
• Team OLMo, 2 OLMo 2 Furious, Allen Institute for AI, 2024.
• Andrej Karpathy, build-nanogpt, 2024.

Papers:

• Tom Brown et al., Language Models are Few-Shot Learners, 2020.
• Katherine Lee et al., Deduplicating Training Data Makes Language Models Better, 2021.
• Jessica Rumbelow and mwatkins, SolidGoldMagikarp (plus, prompt generation), 2023.
• Niklas Muennighoff et al., Scaling Data-Constrained Language Models, 2023.
• Mitchell Wortsman et al., Small-scale proxies for large-scale Transformer training instabilities, 2023.
• Sho Takase, Shun Kiyono, Sosuke Kobayashi and Jun Suzuki, Spike No More: Stabilizing the Pre-training of Large Language Models, 2023.
• Sander Land and Max Bartolo, Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models, 2024.

Hugging Face documentation:

• Debugging, Transformers documentation. Earlier versions, such as v4.13 and v4.49, cover the same overflow tools.
• Model debugging toolboxes, Transformers documentation.
• Debugging the training pipeline, Hugging Face LLM course, chapter 8.
• Debugging distributed operations, Accelerate documentation.

Bug reports:

• Hugging Face, Fixing Gradient Accumulation, 2024.
• Unsloth, blog post on the gradient accumulation bug, 2024.

This post works through those techniques in the order they build on each other. Where I can, it uses my own 0.5B model on an RTX 5090 as the worked example. Every source is listed at the end.

Two phases: prefill and decode

A request to a language model runs in two phases. Prefill processes the whole prompt in one pass, all its tokens in parallel, like a training forward pass. Decode then generates the response one token at a time, and each new token needs a full pass through the model.

The two phases have different bottlenecks, and serving systems report them separately:

• Time to first token is mostly prefill. It’s what a user feels as the pause before the answer starts.
• Time per output token is decode. It sets how fast the answer streams.
• Throughput is total tokens per second across all requests, which sets the cost.
• Goodput, a term from the DistServe paper, is the rate of requests that meet their latency targets. High throughput with most requests missing their targets isn’t useful.

Why decoding is limited by memory

To generate one token, the GPU has to read every weight in the model from memory. For one request at a time, each weight is used for only about two arithmetic operations: a multiply and an add. That’s far too little work per byte to keep the GPU’s arithmetic units busy.

kipply’s Transformer Inference Arithmetic puts numbers on this. An A100 can do about 312 trillion bf16 operations a second but read only about 1.5 TB of memory a second, a ratio of roughly 208 operations per byte. Until each weight is used for about that many operations, the GPU spends its time waiting on memory, and processing one token takes about as long as processing a couple of hundred. Horace He’s Making Deep Learning Go Brrrr gives the general framing: every operation is limited by compute, by memory bandwidth, or by overhead, and knowing which tells you what to optimize.

My model. In bf16 my 489.3M weights take 0.98 GB. The RTX 5090 reads memory at about 1.8 TB/s, so one request at a time can never exceed about 1,800 tokens a second, however fast the arithmetic is. Its rated bf16 throughput of about 210 trillion operations a second is about 117 times its memory bandwidth in bytes, so it takes a batch of roughly 100 requests decoding together before arithmetic becomes the limit.

That’s the central lesson of serving: batch many requests together, so each weight read from memory does work for all of them. The rest of this post is about how to fit a large batch in memory, keep it full, and waste as little of it as possible.

The KV cache

Attention needs the keys and values of every earlier token. Recomputing them for the whole sequence at each step would make generating n tokens cost work proportional to n², so every serving system stores them instead. That’s the KV cache.

My own generate function, which I only use to print sample text during training, doesn’t have one: it runs the whole sequence through the model for every new token. For a few 64-token samples every few hundred steps that’s fine. For serving, the KV cache is the first thing to add.

The cache has a cost. Per token it takes 2 (keys and values) × layers × key/value heads × head size × bytes per value. kipply gives this for standard multi-head attention; with fewer key/value heads, the count of key/value heads replaces the count of attention heads. For my model in bf16 that’s 2 × 24 × 4 × 64 × 2 = 24 KB per token, or about 50 MB for a full 2,048-token context.

With standard multi-head attention, every one of my model’s 20 attention heads would keep its own keys and values, five times as much. At 32K tokens, one sequence’s cache would be about 4 GB, four times the size of the model itself. For a batch of requests, the KV cache, not the weights, is what fills the GPU.

Share key/value heads. Noam Shazeer’s multi-query attention (MQA) observed that decoding is limited by reloading these cached keys and values, and let all attention heads share a single key/value head, shrinking the cache by the number of heads with little loss in quality. Grouped-query attention (GQA) is the middle ground: a few key/value heads, each shared by a group of query heads. Its authors found it reaches quality close to multi-head attention at speed close to multi-query attention, and that an existing multi-head model can be converted with about 5% of its original training compute. My model uses GQA with 4 key/value heads, which is why its cache is 24 KB per token rather than 120 KB.

Managing the cache: PagedAttention

Early serving systems reserved one contiguous block of memory per request, sized for the longest response it might produce. Most of that went unused. The vLLM paper (Kwon et al., 2023) measured that in existing systems only 20 to 38% of the memory reserved for the KV cache held actual keys and values; the rest was lost to over-reservation and fragmentation.

PagedAttention borrows the operating system’s answer to the same problem: virtual memory. It stores each request’s cache in small fixed-size blocks that don’t need to be contiguous, allocated as the sequence grows. Waste falls to nearly nothing, and requests can share blocks, for example when sampling several responses to the same prompt. The paper reports 2 to 4 times the throughput of the best earlier systems at the same latency. The launch blog post put it at up to 24 times Hugging Face Transformers, with existing systems wasting 60 to 80% of their cache memory against under 4% for vLLM.

Batching: static to continuous

With static batching, a batch starts together and finishes together: short responses wait for the longest one, and new requests wait for the whole batch. The Orca paper (Yu et al., 2022) replaced that with iteration-level scheduling: the batch is re-formed at every decoding step, finished requests leave immediately, and new ones join. This is now called continuous batching. Orca also batches only the operations that can be batched, such as the matrix multiplications, and runs attention per request, so sequences of different lengths can share a batch. On GPT-3 175B it reported 36.9 times the throughput of NVIDIA’s FasterTransformer at the same latency.

Anyscale’s benchmark separates the two ideas. Serving OPT-13B on one A100 with widely varying response lengths, naive static batching produced about 81 tokens a second. Systems with continuous batching alone reached about 300 to 350, roughly 4 times more. vLLM, with continuous batching and paged memory, reached about 1,865, 23 times the naive baseline. Both ideas matter, and the memory management is worth more than the scheduling.

Prefill and decode get in each other’s way

Continuous batching mixes new requests’ prefills with ongoing requests’ decodes. A long prompt’s prefill takes much longer than a decode step, so every request in the batch stalls while it runs, and output streaming stutters.

Split long prefills into chunks. Sarathi-Serve (Agrawal et al., 2024) breaks each prompt into roughly equal chunks and adds one chunk per step alongside the ongoing decodes, so no step is ever much longer than the rest. Compared with vLLM, it reports 2.6 times the serving capacity for Mistral-7B on one A100, 3.7 times for Yi-34B on two, and 5.6 times for Falcon-180B.

Or run them on different GPUs. Prefill is limited by compute and decode by memory bandwidth, so the best hardware and parallelism differ. DistServe (Zhong et al., 2024) runs the two phases on separate GPUs, tunes each separately, and moves the KV cache between them. It reports serving 7.4 times more requests, or meeting latency targets 12.6 times tighter, than existing systems. Splitwise (Patel et al., 2024) makes the same split across machines, reporting 1.4 times the throughput at 20% lower cost, or 2.35 times at the same cost and power.

Reuse shared prefixes

Many requests start the same way: a system prompt, few-shot examples, the earlier turns of a chat, the fixed preamble of an agent loop. Their KV cache for that shared start is identical. SGLang (Zheng et al., 2024) keeps cached prefixes in a tree, called RadixAttention, so any new request that shares a prefix picks up its cache instead of recomputing it. It reports up to 6.4 times the throughput of earlier systems on workloads such as agents, retrieval-augmented generation and multi-turn chat.

Make attention itself cheaper

FlashAttention (Dao et al., 2022) computes attention in tiles held in the GPU’s small, fast on-chip memory, never writing the full attention matrix to main memory. That makes attention’s extra memory grow linearly with sequence length instead of quadratically, and cuts the traffic to main memory, which is the resource decoding is short of. FlashAttention-2 roughly doubled its speed by splitting the work better across the GPU. It’s the same kernel that makes long-sequence training fit, as the training post shows.

Make the weights smaller

If decoding is limited by reading the weights, fewer bytes per weight means faster decoding, as well as more room for the KV cache.

• LLM.int8() (Dettmers et al., 2022) runs models of up to 175B parameters in 8-bit with no loss in quality, halving memory. The difficulty was a few feature dimensions with very large values that appear in large models; it keeps those in 16-bit and runs about 99.9% of the multiplication in 8-bit.
• GPTQ (Frantar et al., 2022) quantizes weights to 3 or 4 bits after training, using approximate second-order information to correct for the error, with negligible accuracy loss. It quantizes a 175B model in about 4 GPU-hours and lets it run on a single GPU, about 3.25 times faster than fp16 on an A100.
• AWQ (Lin et al., 2023) observes that about 1% of weights matter most, and that they’re best identified by the size of the activations they multiply rather than by their own size. Scaling those channels up before quantizing to 4 bits protects them, with no retraining.

For my model, 4-bit weights would take about 0.25 GB instead of 0.98 GB, which would raise the ceiling on single-request decoding speed about fourfold.

Speculative decoding

Decoding one token barely uses the GPU’s arithmetic, and checking several candidate tokens in one pass costs about the same as generating one. Speculative decoding exploits that. A small, fast draft model proposes the next few tokens, and the large model checks them all in a single pass, keeping the longest run it agrees with. A carefully designed acceptance rule makes the output follow exactly the same distribution as the large model alone, so quality is unchanged.

It was developed independently by two groups. Leviathan et al. at Google reported 2 to 3 times faster decoding for T5-XXL with identical outputs, and Chen et al. at DeepMind 2 to 2.5 times for the 70B-parameter Chinchilla, with no retraining in either case. It helps most at small batch sizes, where the GPU has arithmetic to spare.

Spreading a model across chips

When a model is too big for one accelerator, it has to be split, and the best split depends on the batch size. Pope et al. (2022) built a model of the trade-offs for serving PaLM on TPUs. Layouts that keep weights in place and move activations win at small batches and low latency; layouts that move the weights instead win at large batches. With these choices they served the 540B-parameter PaLM at 29 ms per generated token using 8-bit weights, reached 76% utilization of the hardware during prefill of large batches, and used multi-query attention to support contexts up to 32 times longer.

Rules of thumb

• Decoding is limited by memory bandwidth. One request at a time can’t go faster than bandwidth divided by the size of the weights.
• Batching is the main lever: each weight read should serve as many requests as possible.
• At long contexts and large batches, the KV cache, not the weights, fills the GPU. Estimate it before choosing a batch size.
• Fewer key/value heads, through grouped-query or multi-query attention, shrinks the cache almost for free.
• Allocate the cache in pages and schedule at every step, not once per batch.
• Keep long prefills from stalling decodes, by chunking them or by running them on separate GPUs.
• Quantized weights and speculative decoding both help most when batches are small.
• Report time to first token and time per output token separately, and measure goodput, not just throughput.

Sources

Background:

• kipply, Transformer Inference Arithmetic, 2022.
• Horace He, Making Deep Learning Go Brrrr From First Principles, 2022.
• Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal and Jeff Dean, Efficiently Scaling Transformer Inference, 2022.

Attention and the KV cache:

• Noam Shazeer, Fast Transformer Decoding: One Write-Head is All You Need, 2019.
• Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra and Christopher Ré, FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, 2022.
• Tri Dao, FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning, 2023.
• Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón and Sumit Sanghai, GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, 2023.

Serving systems:

• Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim and Byung-Gon Chun, Orca: A Distributed Serving System for Transformer-Based Generative Models, 2022.
• Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang and Ion Stoica, Efficient Memory Management for Large Language Model Serving with PagedAttention, 2023, and the vLLM launch post, 2023.
• Cade Daniel, Chen Shen, Eric Liang and Richard Liaw, How continuous batching enables 23x throughput in LLM inference while reducing p50 latency, Anyscale, 2023.
• Lianmin Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs, 2023.
• Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki and Ricardo Bianchini, Splitwise: Efficient Generative LLM Inference Using Phase Splitting, 2023.
• Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin and Hao Zhang, DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving, 2024.
• Amey Agrawal et al., Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve, 2024.

Faster decoding:

• Tim Dettmers, Mike Lewis, Younes Belkada and Luke Zettlemoyer, LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, 2022.
• Elias Frantar, Saleh Ashkboos, Torsten Hoefler and Dan Alistarh, GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, 2022.
• Yaniv Leviathan, Matan Kalman and Yossi Matias, Fast Inference from Transformers via Speculative Decoding, 2022.
• Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre and John Jumper, Accelerating Large Language Model Decoding with Speculative Sampling, 2023.
• Ji Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, 2023.

This post follows the playbook’s structure and keeps its numbers, and uses my own 0.5B model, trained on a single RTX 5090, as the worked example. That run never needed more than one GPU, which makes it a useful baseline: it shows exactly where memory goes before any of the techniques below are needed.

The playbook frames everything around three constraints that trade against each other. Memory is a hard limit: if a step doesn’t fit, it doesn’t run. Compute efficiency is how much of the GPU’s arithmetic you actually use. Communication is the time GPUs spend exchanging data instead of computing. Almost every technique buys one of these with another.

Start with the memory budget

Four things take GPU memory during training: the model’s weights, their gradients, the optimizer’s state, and the activations saved during the forward pass for use in the backward pass. On top of those, CUDA itself takes 1 to 2 GB, and some memory is always lost to buffers and fragmentation.

For the first three, the playbook counts bytes per parameter. In the common mixed-precision setup, the model computes in bf16 but keeps an fp32 “master” copy of the weights for the optimizer to update:

Some frameworks also accumulate gradients in fp32, because bf16 loses small values; that adds 4 bytes, for 20. Mixed precision doesn’t save memory on these items at all. What it saves is compute time and activation memory.

At 16 to 20 bytes per parameter, the numbers grow quickly:

From 7B parameters up, these alone don’t fit on an 80 GB H100, before a single activation is stored.

My run. My model keeps its weights in fp32 and lets PyTorch’s autocast run the matrix multiplications in bf16, so there’s no separate bf16 copy: 4 bytes of weights, 4 of gradients and 8 of Adam state, which is also 16 bytes per parameter. For 489.3M parameters that’s 7.8 GB. The full checkpoint I save for resuming is 5.9 GB, which is 12 bytes per parameter: weights plus Adam’s two moments, since gradients aren’t saved. Checking the checkpoint size against this arithmetic is a quick way to confirm you’re saving what you think you’re saving.

Profile it. The playbook uses PyTorch’s profiler with memory tracking to watch a training step. Activations climb through the forward pass. During the backward pass, gradients accumulate while activations are freed. Then the optimizer step runs. One detail catches people out: the optimizer’s state is only allocated during the first optimizer step, so a run can finish step 1 and then run out of memory on step 2.

Activations grow with sequence length

Activation memory depends on the batch, not just the model. The playbook’s estimate for a standard transformer in mixed precision is

L · s · b · h · (34 + 5 · a · s / h) bytes,

where L is the number of layers, s the sequence length, b the micro-batch size, h the hidden size and a the number of attention heads. It grows linearly with batch size. The second term, which is the attention matrix stored for the backward pass, grows with the square of the sequence length. For short sequences activations are minor; from about 2,000 to 4,000 tokens they become the largest item in memory.

The chart shows why my run fit at all. My micro-batch is 4 sequences of 2,048 tokens. Storing the attention matrix would take about 49 GB, far more than the GPU has. The attention kernel I use, PyTorch’s scaled_dot_product_attention, never writes that matrix out, in the style of FlashAttention, which leaves about 8.6 GB. Add 7.8 GB of weights, gradients and optimizer state and the estimate is 16.4 GB, against the 19.6 GB peak I measured. The remaining 3.2 GB is most likely the output logits, 4 × 2,048 × 32,000 values, plus working memory. The formula also assumes a standard feed-forward block and full multi-head attention, which mine doesn’t quite match, so treat it as an estimate.

Activation recomputation trades compute for memory: throw some activations away in the forward pass and recompute them in the backward pass. Recomputing everything, keeping only each layer’s input, amounts to an extra forward pass and typically costs 30 to 40% more compute. Selective recomputation drops only the attention activations, which are large but cheap to recompute. For GPT-3 175B, the playbook cites a 70% cut in activation memory for 2.7% more compute. FlashAttention already recomputes attention scores in the backward pass, so anyone using it is doing selective recomputation already.

Recomputation changes how efficiency should be counted. Hardware FLOPs utilization counts the recomputed work; model FLOPs utilization (MFU) counts only the work the model needs. The playbook prefers MFU, because what matters is how long training takes.

Batch size and gradient accumulation

The playbook measures batch size in tokens. Large pretraining runs typically use between 4 million and 60 million tokens per batch: Llama 1 used about 4 million, and DeepSeek about 60 million. Sequence lengths of 2,000 to 8,000 tokens are typical, with longer contexts added near the end of training.

When the batch doesn’t fit in memory, gradient accumulation runs several micro-batches in sequence and adds up their gradients before the optimizer step. Memory stays constant, but the micro-batches run one after another, so it’s slower. Across GPUs, the global batch size is

global batch = micro-batch × accumulation steps × data-parallel GPUs.

The playbook’s advice is to use more GPUs before more accumulation, since GPUs run in parallel and accumulation doesn’t.

My run used a micro-batch of 4 sequences, 64 accumulation steps and one GPU: 4 × 64 × 2,048 = 524,288 tokens per step. On 8 GPUs, the same batch would need only 8 accumulation steps per GPU.

Data parallelism

Data parallelism is the simplest way to use more GPUs: every GPU holds a full copy of the model, processes a different slice of the batch, and the GPUs average their gradients with an all-reduce before the optimizer step. The playbook describes three optimizations that make it efficient:

• Overlap communication with the backward pass. Start averaging a layer’s gradients as soon as they’re computed, instead of waiting for the whole backward pass to finish.
• Bucket the gradients. Send them in a few large messages rather than many small ones.
• Skip the averaging during accumulation. Only average after the last micro-batch. My training loop does this with PyTorch’s require_backward_grad_sync flag.

Data parallelism alone doesn’t reduce memory per GPU, since every GPU still holds everything. And it stops scaling well at around 512 GPUs, where the time for the all-reduce to go around the ring of GPUs can no longer be hidden behind computation.

ZeRO removes the duplication. Each stage shards one more item across the N data-parallel GPUs, with Ψ the parameter count and k = 12 bytes of optimizer state per parameter (fp32 master weights plus two Adam moments):

For a 7B model on 8 GPUs, that’s 112 GB per GPU with plain data parallelism, about 39 GB with ZeRO-1, 26 GB with ZeRO-2 and 14 GB with ZeRO-3, before activations. ZeRO-2 communicates no more than plain data parallelism, since an all-reduce is itself a reduce-scatter followed by an all-gather, and the playbook calls it usually the better choice over ZeRO-1. ZeRO-3 has to gather each layer’s weights before using them, in both the forward and backward passes, which raises communication by about half. Fetching the next layer’s weights while the current one computes hides most of that.

ZeRO can’t shard activations, since each GPU’s are different, and every layer still has to fit on one GPU while it runs. Those limits are what the next techniques address.

Tensor and sequence parallelism

Tensor parallelism splits individual weight matrices across GPUs. A linear layer can be split by columns, where each GPU computes part of the output, or by rows, where each GPU computes a partial sum. The transformer’s feed-forward block pairs a column split with a row split, so it needs only one all-reduce in the forward pass. Attention splits naturally by heads.

That communication sits in the middle of every layer’s computation, so it can’t be fully hidden, and it needs the fastest links available. The playbook’s benchmarks show it clearly: going from 8 to 16 GPUs, which crosses from one node to two, cost tensor parallelism about 43% of its throughput, against 14% for pipeline parallelism. The rule is to keep tensor parallelism within a node, typically 8 GPUs. It also can’t usefully exceed the number of attention heads. My model has 20 query heads but only 4 key/value heads, so splitting it more than 4 ways would mean duplicating key/value heads.

Tensor parallelism leaves some activations whole: those of the normalization and dropout layers. Sequence parallelism splits those along the sequence dimension instead. It replaces each all-reduce with a reduce-scatter and an all-gather, which cost the same in total, and cuts the remaining activation memory by the tensor-parallel degree. With both, the playbook fits 16,000-token sequences for a 70B model across 16 GPUs.

Context parallelism

At very long sequence lengths, 128,000 tokens and up, even full recomputation leaves too much activation memory. Context parallelism splits the sequence itself across GPUs, for every layer. Most of the model doesn’t care: each token’s feed-forward and normalization computations are independent. Only attention needs every token’s keys and values.

Ring Attention handles that by passing key and value chunks around a ring of GPUs. Each GPU computes attention for its own queries against the chunk it holds, sends that chunk on, receives the next one, and repeats until it has seen them all, overlapping the transfer with the computation. A causal mask makes the naive split unbalanced, because early tokens attend to far less than late ones. Zig-zag attention fixes that by giving each GPU a mix of early and late tokens.

Pipeline parallelism

Pipeline parallelism splits the model by layers: GPU 1 holds the first few layers, GPU 2 the next, and so on. It reduces weight memory per GPU, and it only sends activations between neighbouring stages, which makes it well suited to slower links between nodes. Its cost is the bubble: while the first micro-batch travels forward through the pipeline and back, most GPUs sit idle.

With p pipeline stages and m micro-batches per step, the idle fraction in the simplest schedule, all forward passes then all backward passes, is (p − 1) / m. More micro-batches shrink the bubble, but this schedule has to keep activations for all of them. The 1F1B schedule alternates one forward and one backward pass once the pipeline is full, which gives the same bubble while keeping activations for only p micro-batches. Interleaved schedules give each GPU several non-adjacent chunks of layers; with v chunks per GPU the bubble shrinks to (p − 1) / (v · m), at the cost of v times more communication. Llama 3.1 used an interleaved 1F1B schedule.

The newest schedules go further. Zero Bubble splits the backward pass into the part that computes gradients for the layer’s input, which the previous stage is waiting for, and the part that computes gradients for the weights, which can be scheduled later to fill gaps. DualPipe, used for DeepSeek-V3, feeds micro-batches into both ends of the pipeline at once and overlaps nearly all communication with computation.

Expert parallelism

Mixture-of-experts models replace the single feed-forward block with many smaller “experts” and send each token to a few of them. Expert parallelism places different experts on different GPUs and routes tokens to them with an all-to-all exchange. No matrix is split, so it’s lighter than tensor parallelism. DeepSeek-V3, for example, has 256 experts and limits each token to experts on at most 4 nodes, which bounds how far any token has to travel. Expert parallelism only splits the experts, so the rest of the model is usually combined with data parallelism.

Choosing a configuration

The playbook calls the combination 5D parallelism: data, tensor, sequence or context, pipeline and expert, with ZeRO on top of data parallelism.

Its procedure for choosing:

1. Fit the model in memory. Under about 10B parameters, one technique is often enough: tensor parallelism, or ZeRO-3 with full recomputation, across 8 GPUs. From 10B to 100B, combine tensor parallelism of 8 with pipeline parallelism or ZeRO-3. On more than about 512 GPUs, pure data parallelism or ZeRO-3 becomes inefficient, so add tensor or pipeline parallelism. Add context parallelism for long sequences and expert parallelism for mixture-of-experts models. With few GPUs, use full recomputation and more accumulation.
2. Reach the target batch size by adjusting data parallelism and accumulation.
3. Maximize throughput. Scale tensor parallelism up to the node size, then data parallelism with ZeRO-3, and switch to pipeline parallelism when data-parallel communication becomes the bottleneck. Change one dimension at a time.

Pipeline parallelism and ZeRO-3 both split weights across GPUs, but differently. Pipeline parallelism keeps whole layers and sends activations, and prefers many accumulation steps. ZeRO-3 keeps slices of every layer and sends weights, and prefers large micro-batches and long sequences. They’re rarely used together, while ZeRO-1 or 2 combine easily with pipelines; DeepSeek-V3 used pipeline parallelism with ZeRO-1.

Across several thousand benchmarked configurations on 1 to 64 nodes of 8 H100s, efficiency fell as the number of nodes grew, especially for small models, because the fixed global batch of 1 million tokens left less work per GPU. The best MFU in the data behind the playbook’s summary chart was about 44 to 47% for models from 1B to 9B parameters and about 34% for 80B. The playbook’s most striking finding is that implementation quality mattered as much as the choice of technique: tensor parallelism beat pipeline parallelism until the pipeline code was optimized, after which pipeline parallelism won.

My single-GPU run reached about 85% MFU, measured against the RTX 5090’s rated bf16 throughput, at 48,000 tokens a second and about 3.7 billion floating-point operations per token. The two figures come from different hardware, so they aren’t directly comparable. The gap does show what the playbook is about, though: most of the lost efficiency at scale is time spent communicating, and on one GPU there’s nothing to communicate.

Inside the GPU

Past the choice of parallelism, the playbook’s last section is about making each GPU faster.

Fuse operations. A sequence of small element-wise operations, such as a normalization, reads and writes GPU memory at every step. A fused kernel does them all in one pass. torch.compile does much of this automatically, and my run used it. Custom kernels in Triton or CUDA go further at more effort.

Use FlashAttention. It computes attention in tiles held in the GPU’s fast on-chip memory, never writing the full attention matrix out, which is the saving in the chart above. FlashAttention-2 roughly doubled its speed with better partitioning of work, and FlashAttention-3 adds support for newer hardware and fp8.

Choose number formats deliberately. The formats differ in how their bits are split between exponent, which sets the range, and mantissa, which sets the precision:

bf16 keeps fp32’s range with less precision, which is why it has largely replaced fp16 for training. fp8 doubles the throughput of bf16 on H100s but is harder to keep stable; DeepSeek-V3 was the first large public model trained with it, using separate scaling factors for small tiles of each matrix. The playbook still called fp8 training experimental in early 2025.

Rules of thumb

• Weights, gradients and Adam state cost 16 bytes per parameter in standard mixed precision, 20 with fp32 gradient accumulation.
• Activations grow linearly with batch size and, without FlashAttention, with the square of sequence length.
• A run that survives step 1 and then runs out of memory has just allocated its optimizer state.
• Training costs about 6 floating-point operations per parameter per token: 2 in the forward pass and 4 in the backward pass.
• Use more GPUs before more gradient accumulation.
• Keep tensor parallelism inside a node, and no wider than the number of attention heads.
• Prefer ZeRO-2 to ZeRO-1; use ZeRO-3 when the weights themselves don’t fit.
• Measure MFU, and change one parallelism dimension at a time.

Sources

• Nouamane Tazi, Ferdinand Mom, Haojun Zhao, Phuc Nguyen, Mohamed Mekkouri, Leandro von Werra and Thomas Wolf, The Ultra-Scale Playbook: Training LLMs on GPU Clusters, Hugging Face, 2025.
• Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra and Christopher Ré, FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, 2022.
• Tri Dao, FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning, 2023.

The loop runs in an isolated workspace, which keeps a bad step off your working tree. Tasks come from wherever you keep them: a file, an issue tracker, email. It suits the work that has a clear finish line and no interesting decisions in it, such as writing tests, fixing lint, updating docs, and migrating an API across a codebase. You still decide what gets built and whether the result is right.

What I learned writing a few:

• One skill = one job. Don’t try to do everything in one skill
  • Bad: “My coding helper” (does linting, formatting, testing, docs, reviews)
  • Good: “Markdown linter for my blog posts” (only fixes markdown)
• If you have 3 different workflows, make 3 separate skills
• A skill that does five jobs fires at the wrong time and steps on your other skills
• Spend your effort on the description. Claude reads it first, and it decides whether your skill loads at all
  • Bad: “helps with formatting”
  • Good: “Fixes markdown syntax and formatting for blog posts when user asks to review a draft”
• A vague description means the skill sits there unused
• Say when the skill applies and what problem it solves
• Make it condition-based (when X happens, use this skill)
  • Example: “Use this when user provides git diffs to fix merge conflicts”

Start with one problem you already have:

• Pick one actual thing you need to do, not something you imagine might be useful
  • Example: “I always struggle formatting my blog post frontmatter” is better than “I want to help with writing someday”
• Test whether Claude reaches for the skill on its own
  • Example: Does it load when you paste a blog post draft, without you naming it?
• Fix the hardest part first
  • Example: If detecting the frontmatter format is tricky, solve that before worrying about date formatting
• Change the instructions based on what Claude does, not what you assumed it would do
  • Example: If Claude keeps adding extra blank lines, adjust the instructions instead of guessing why

Get the structure right:

• Skills are just folders with a SKILL.md file
• The file needs YAML frontmatter at the top
• Two required fields: name and description
• Everything else is markdown instructions

File organization matters:

• Put templates (files Claude copies) in templates/ folder
  • Example: templates/blog-post-template.md is a blank post structure Claude fills in
• Put reference docs (files Claude reads) in references/ folder
  • Example: references/markdown-style-guide.md tells Claude your formatting rules (but isn’t copied)
• Folder names tell Claude what each file does
• Don’t mix templates and references in the same folder
  • Bad: Everything in one folder with confusing names
  • Good: Clear folders so Claude knows what to do with each file

Keep your skills in version control:

• Keep your skills in a Git repo (like on GitHub)
  • Example: ~/.claude/skills/ as a git repo synced with GitHub
• If they’re just on one computer, they drift and become inconsistent
  • Problem: You edit a skill on laptop but forget the changes on desktop
• Committing to a personal repo keeps all your machines synced
  • Example: git push after tweaking your skill, then git pull on your work computer
• You start trusting a skill once you can see its history
  • Example: When something breaks, you can see exactly what changed and when

Write your tests first:

• Know what success looks like before you code
  • Example: “Input: messy blog post → Output: clean frontmatter with correct date format”
• This catches skills nobody needs before you write them
  • Example: If you never paste markdown to Claude, don’t build a markdown skill
• It also gives you something to check against when the skill misbehaves
  • Example: When Claude adds wrong formatting, you have specific test cases to check against

Quick start process with a real example:

1. Pick one workflow you do regularly
   • Example: “I format blog post dates from random formats into YYYY-MM-DD”
2. Write down the exact steps
   • Example: Check for multiple date formats → Convert to ISO format → Update frontmatter
3. Create a skill folder with SKILL.md
   • Example: ~/.claude/skills/blog-date-formatter/
4. Add clear name and description
   • Example: name: “blog-date-formatter”, description: “Converts messy blog post dates to ISO format YYYY-MM-DD when user pastes content”
5. Write instructions in markdown
   • Example: “Look for dates in these formats: ‘July 11, 2026’ or ‘7/11/26’ → Always convert to 2026-07-11”
6. Test it with real examples
   • Example: Paste an actual blog post you wrote and see if Claude fixes the dates right
7. Commit to GitHub
   • Example: git commit -m "Add blog date formatter skill"

A skill is a file of instructions you stopped having to retype. Build one for the workflow you run most, and see whether Claude picks it up on its own.

Sources:

• Skill authoring best practices - Claude Platform Docs
• The Complete Guide to Building Skills for Claude
• How to create a Claude Code skill
• 7 Rules for Creating an Effective Claude Code Skill
• Teach Claude your way of working using skills