We spent the last several years building post-training infrastructure inside Meta, across multiple generations of Llama.
Llama 3 gave the industry a strong foundation. It showed what open models could become when strong research, infrastructure, data and all other things came together. It taught us what happens when model iteration becomes faster: researchers try more ideas, push models harder, and expect infrastructure to keep up with a pace that was almost impossible a few years earlier. Its success also created enormous excitement and raised the ambition for what would come next.
Our team built the research-to-production platform behind it — post-training, evaluation, batch inference, and data generation — serving hundreds of researchers iterating on Llama 3's checkpoints. It powered the models behind Meta AI, carried the line through 3.1 and 3.2 including its first multimodal capabilities, and its async RL infrastructure accelerated the experiments behind Llama 3.3, which brought a 70B dense model on par with the 405B.
Llama 4 was a different kind of lesson.
Llama 4 pursued multiple model families, architectures, and frontier capabilities in parallel, multiplying the complexity across the entire development stack. From the outside, frontier models are often judged by a single public outcome: did the model achieve top results on public leaderboards? From the inside, however, a release is the product of many interdependent systems working together—including model architecture, data recipes, pre-training, mid-training, post-training, evaluation, inference and deployment, and the reliability and scalability of the underlying distributed infrastructure.
Those systems do not all rise or fall together: some can represent major breakthroughs even when others fall short—and even when the overall release does not meet expectations.
Below, we share some of our reflections on the complexity of Llama 4, the challenges it surfaced, and the lessons we carried forward.
Llama 4 marked a major jump in complexity, combining native multimodality, a mixture-of-experts architecture, three model families, and more than five frontier capabilities. Its largest model, Behemoth, has 288B active parameters across 16 experts and nearly two trillion total parameters.
Scaling post-training to that size required Meta to substantially up-level the underlying infrastructure. That is the part we are proud of.
The infrastructure did more than increase the number of supported model lines and variants by over 5× compared with Llama 3—it expanded what the research team could attempt. It enabled trillion-parameter-scale experimentation. It supported a major architecture shift from dense models to sparse MoE models. It helped power multiple model families and many different capabilities across text, vision, reasoning, coding, multilinguality, and long context. It gave researchers the ability to run, compare, debug, and scale experiments that would otherwise have been impossible on that timeline.
From 405B to two trillion parameters#
For Llama 3, we built the system later described in the LlamaRL paper: a fully distributed asynchronous RL framework that scaled from 8B to 405B models and ran up to 10.7× faster than DeepSpeed-Chat-style baselines on a 405B policy. The basic idea was to stop making generation and training take turns. We ran them in parallel, synchronized multi-terabyte model weights in seconds, and used importance-weighted corrections — AIPO in the paper — so the trainer could learn from slightly stale samples without going unstable. That system became the starting point for Llama 4.
Then Llama 4 changed the problem. Behemoth, the largest model in the family, was a mixture-of-experts model with 288B active parameters, 16 experts, and nearly two trillion parameters in total. The broader program added native multimodality, multiple model families, and capabilities across reasoning, coding, vision, long context, and multilinguality. On our side, the number of model lines and variants the infrastructure had to support grew more than 5× over Llama 3.
At the beginning of 2025, on a timeline with very little room for error, our mandate was the largest-scale RL training Meta had attempted: fully asynchronous RL on a roughly two-trillion-parameter MoE model across 6000 to nearly 10000 GPUs, which is later highlighted in the Meta Tech Blog. At the time, we weren't aware of any publicly described RL system operating at comparable model and cluster scale.
And RL at that scale isn’t one model running on a big cluster. A real pipeline includes a policy trainer, high-throughput generation, reward models, reference models, evaluators, and sometimes judges or rule-based scorers — each with its own memory footprint, compute profile, parallelism strategy, and communication pattern. Fitting a two-trillion-parameter model into memory is hard enough. The real job was keeping that entire learning loop moving.
What starts breaking at 5,000 GPUs#
At two trillion parameters and more than 5,000 GPUs, much of what had worked for us at 405B simply stopped working. The optimizations we needed didn’t exist yet, so we built them.
Weight synchronization was the first wall. Online RL needs the trainer to push updated weights to the samplers at every training step, and at this scale the two sides don’t even shard the model the same way: training ran 5D parallelism — data, tensor, expert, pipeline, and context — while the samplers used their own sharding, tuned for generation throughput. The standard recipe is to gather the weights onto one trainer rank, then broadcast them out to the sampler ranks. Across 5,000 GPUs, that gather-and-broadcast pattern fell over immediately: it ran out of GPU memory and stacked up latency, bottlenecked by the communication bandwidth available across nodes and racks in the datacenter. We redesigned the transfer path to skip the gathers and scatters wherever possible, so weight shards move directly between the ranks that hold them over RDMA. That brought full weight synchronization for the two-trillion-parameter model down to about three seconds.
Fast synchronization across mismatched sharding bought us more than low overhead. The easiest way to keep weight transfer cheap is to force the trainer and samplers onto identical sharding, but identical sharding is rarely what makes generation fast. Because our sync stayed fast even when the two sides sharded differently, the samplers were free to run a much faster parallelism configuration of their own — and that alone improved end-to-end RL training speed by more than 2x.
Stragglers were the other big one. Long-generation reasoning tasks were just taking off at the time, and one unusually long sample could leave the rest of the batch waiting. We built prompt caching and dynamic batching to deal with it, which relieved memory pressure and improved training efficiency at the same time. Both techniques started showing up in open-source RL frameworks about half a year later.
Then there were the problems that only became visible once the system got big enough. At smaller scale, most attention goes to GPU memory and GPU-to-GPU communication. At 5,000 GPUs, CPU memory and host-network bandwidth became critical bottlenecks — moving large batches of trajectories, rewards, and token-level metadata could overwhelm parts of the system that had never limited us before. Job initialization was similar. A startup sequence that looks fine on a few hundred GPUs gets painfully slow across several thousand, and at that scale a restart burns real cluster capacity before any useful training begins. We redesigned the initialization path, changed how data moved through the system, and removed bottlenecks that simply didn’t exist at smaller scale.
None of this makes for a model demo. Together, it’s what made the training program run.
When training and inference disagree#
The finding we’re proudest of sounds simple: trainer–inference mismatch.
Online RL assumes the system generating a token and the trainer computing that token’s probability are consistent enough for the learning update to stay valid. With a dense model, the small numerical differences between training and inference stacks are usually manageable. Mixture-of-experts models make that assumption fragile: a small discrepancy in the token router can send the same token to different experts on the two sides. The model has the same nominal weights everywhere but performs different computation in practice — and that difference can invalidate the assumptions behind the RL update. In our training runs, it became a major source of instability.
We identified trainer–inference mismatch as a first-order correctness problem for MoE reinforcement learning and built a fix that worked at full training scale. At the time, we found no public work describing router discrepancy this way in a production RL system. Today, trainer–inference consistency is much more widely discussed as a requirement for stable MoE RL; in early 2025, we were learning it the hard way, during real training runs, with a frontier-model program depending on the result.
That kind of experience is hard to reproduce on a benchmark or a small cluster. Some failure modes only show up when thousands of GPUs, heterogeneous model components, several parallelism strategies, and a hard deadline all meet at once — and very few teams have personally debugged an online RL system under those conditions.
What we took forward#
We’re proud of the infrastructure we helped build — systems that worked at frontier scale. It supported the largest-scale RL training Meta had attempted at the time, met extreme timeline and scale requirements, supported three model families across 5+ frontier capabilities, and made trillion-parameter-scale experimentation practical.
That work shaped how we think about AI infrastructure. At frontier scale, infrastructure is not just a support layer. It determines how fast researchers can test ideas, how early teams can catch failures, and how much compute turns into learning instead of waste. The better the infrastructure, the more ambitious the research loop can become.
That is the experience we are bringing into Goaly.
Why Goaly#
Most AI companies will never train a two-trillion-parameter model on 8,000 to 10,000 GPUs, but a smaller version of the same problems is already showing up everywhere. A failed run eats a real share of a small team’s compute budget. A systems problem looks like a model problem and sends researchers hunting in the wrong part of the stack. An invisible bottleneck convinces a team it needs more hardware when what it actually needs is better visibility. And diagnosing any of it is usually reactive — something breaks, and a few experienced engineers spend days stitching together metrics, traces, and logs to reconstruct why.
We believe the next generation of AI companies will need frontier-style iteration, but they won’t all have frontier-lab-scale infrastructure teams. They won’t have unlimited GPUs, months of buffer, or dozens of senior AI infra engineers babysitting every unstable run at 3 AM. Yet they still need to train, evaluate, and adapt proprietary models on their own data.
Goaly exists to make that possible.
We’re building the AI performance layer for model teams: model-, cloud-, and hardware-agnostic infrastructure that helps teams understand training behavior, detect wasted compute early, diagnose why runs go wrong, and make model iteration faster, cheaper, and more reliable.
Our lesson from Llama was simple: building frontier-scale AI isn’t just about bigger models. It’s about the systems that make iteration affordable and turn every GPU-hour into actual model improvements.
We recently published our first technical blog on foundational work in post-training efficiency, and we plan to share more about our core products once we emerge from stealth.